Data Versioning
Data versioning applies the discipline of source control to datasets and models so any result can be traced to the exact inputs that produced it.
Why code versioning is not enough
Git tracks code well but handles large binary datasets poorly: it stores full copies, bloats history, and struggles past a few hundred megabytes. Yet reproducing a model result requires pinning not just the code but the exact data it trained on. Data-versioning tools such as DVC fill this gap by versioning data alongside code without storing the data in the code repository.
Pointers, not payloads
The central trick is content addressing. The tool computes a hash of each data file and stores that hash, plus the file's path, in a small text metafile that is committed to Git. The heavy bytes live in separate remote storage keyed by their hash. Checking out a commit restores the pointers; a pull fetches the matching bytes. The Git history stays small while pointing unambiguously at exact data.
Pipelines as tracked stages
- A stage declares its command, its input dependencies, and its outputs
- The tool hashes inputs to decide whether a stage must rerun
- Outputs are versioned as data, closing the loop from raw input to result
- A single lock file captures the exact hashes of every artifact in a run
Reproducibility guarantee
Because every input and output is content-addressed, checking out an old commit and rerunning the pipeline regenerates the same artifacts, provided the code and environment are also pinned. This turns "which data produced this figure?" from an archaeology problem into a lookup. For a research organization publishing physics simulations, this is the difference between a claim that can be independently rebuilt and one that cannot.
Relationship to experiment tracking
Data versioning answers "what were the exact inputs?"; experiment tracking answers "what were the parameters and metrics of each run?" They compose: a tracked experiment records the data version hash it used, so a metric can be traced back to a specific dataset revision. See content-addressable storage and experiment tracking.