Data Provenance and Lineage
Knowing where data came from and every transformation it passed through is what lets you trust, debug, and correct any analysis built on it.
The Data's Biography
Data provenance, also called lineage, is the recorded history of a dataset: its origin, every transformation applied, and how it reached its current form. Just as a result needs provenance, so does the data feeding it. Without lineage, a value in a table is an orphan, its meaning, reliability, and units unknown.
What Lineage Captures
- Source: which instrument, simulation, or database produced the raw data, and when.
- Transformations: every filtering, cleaning, unit conversion, and aggregation step.
- Derivations: which inputs combined to produce each derived quantity.
- Versions: which version of the data a given analysis used.
Why It Is Hard
Data pass through many hands and tools, spreadsheets, scripts, manual edits, and lineage breaks wherever a step is undocumented or done by hand. A single unrecorded manual correction can make a dataset impossible to regenerate. Capturing lineage means scripting transformations rather than performing them interactively, so the history is recorded automatically.
What Lineage Buys
When an error is found in a source, lineage tells you exactly which downstream results are affected and must be revised, turning a vague worry into a bounded fix. It also lets a suspicious result be traced back step by step to find where a value went wrong, and it makes an analysis regenerable from raw inputs.
In Scientific Pipelines
A physics result may draw on material property tables, cross-section libraries, and prior simulation outputs, each with its own provenance. Kronos work records which data versions fed a frozen result, so if an input is later corrected, the affected results are identifiable and the recomputation is traceable rather than guesswork.