Data versioning: the reason you cannot reproduce last month’s result

Someone asks you to reproduce the 89% from last month’s report. You pull the same code, run the same pipeline, and get 84%. Nothing in the repository changed. The model is identical. The only thing that moved is the data, and because nobody versioned it, you cannot say what it looked like when the 89% was real. You are now defending a result you cannot recreate.

Code is versioned. Data usually is not.

Every team versions code. Git, commits, tags, the whole habit is automatic. Then the same team points that carefully versioned code at a dataset that lives in a folder or a table that gets overwritten in place, and treats it as if it stands still. It does not. Rows get added. A labeling error gets fixed. A field gets renamed. Someone reruns an export against a source that has since changed. The dataset drifts continuously, and none of that drift is recorded.

So a result is only half-captured. You know the exact code that ran. You have no idea what data it ran on. Reproducibility needs both, and the missing half is the one nobody pinned down. A result you cannot reproduce is a result you cannot fully trust, because you cannot separate a real improvement from a dataset that quietly shifted beneath you.

The data really does move

A migration lead who has moved close to three billion records showed me how alive a dataset is. He phases migrations: core entities first, a human reconciles, then he reruns the tool for the incremental records before go-live. Between the first run and the reconciliation and the incremental run, the data is different every time by design. A source file says “California” and gets mapped to “CA”. A duplicate customer gets resolved under a last-write-wins rule a human designed. Each of those steps changes what the dataset contains.

Now imagine grading a model at two of those moments without recording which was which. Same query, different underlying records, different score, and no way to explain the gap. That is the everyday reality of an unversioned dataset. It is not static background. It is a moving thing you keep taking undated snapshots of.

What versioning actually buys you

Version your data and the fog clears. Every result points at a specific dataset state you can restore. When the score changes, you can ask the useful question, whether the model changed or the data changed, and actually answer it, because you can rerun the old model on the old data and the new model on the old data and compare. Right now most teams cannot run that comparison at all, so every change is a guess.

It also protects the number you report upward. Set your accuracy baseline on a versioned dataset and the baseline is falsifiable, which is the point of a baseline. Set it on a dataset that gets overwritten and the baseline is a memory, impossible to check and easy to quietly drift away from. When someone eventually asks how you know the feature improved, a dataset version is the difference between showing them and telling them.

You do not need heavy tooling

This is not a platform purchase. Snapshot the dataset used for each significant run and give it an identifier. Record which dataset version produced which result, next to the code commit. Keep enough history that you can restore a past state and rerun. Even a disciplined convention beats the common situation, which is a live table nobody can rewind. Say you version the eval set and last month’s 89% becomes reproducible on demand. The 84% you got today is then a real signal about a real change, not a mystery you shrug at.

How we approach it at Density Labs

In the AI Opportunity Assessment, our fixed two week, $2,500 engagement, we check whether you can reproduce a past result. If the data is unversioned, that is a finding, because it means no metric you have reported is fully verifiable and no baseline is truly falsifiable. We look at whether dataset states are captured alongside code, so a result can be traced to the exact data that produced it.

If you cannot recreate last month’s number, you never really knew it. Version the data, not just the code.