Data Provenance
The recorded history of how a dataset was produced: its sources, the code that transformed it, and the parameters used.
Where Did This Number Come From
Provenance is the documented history of a piece of data: where it originated, what code processed it, with what parameters, and in what order. Good provenance lets anyone trace a final number back through every transformation to its raw source. Without it, a dataset is a set of values whose meaning and trustworthiness cannot be checked, and whose errors cannot be traced.
What Provenance Records
- The raw inputs and their sources, with versions or identifiers.
- Each processing step, the code that performed it, and its version.
- The parameters and configuration used at each step.
- The order of operations, forming a chain from raw input to final result.
Why It Matters for V&V
Provenance is what makes a result auditable. When a figure is questioned, provenance lets a reviewer trace it back and check every step. When a bug is found in a processing routine, provenance identifies exactly which results were affected and must be regenerated. When a result is reproduced, provenance is the recipe that says what to run and in what order. It is the connective tissue of a reproducible workflow.
Capturing It
Provenance can be captured manually, in careful lab-notebook records, but it is far more reliable when captured automatically by the workflow itself. Workflow systems that record each step, its inputs, and its outputs build provenance as a side effect of running. Version control supplies the code versions; the workflow supplies the execution history. Together they answer, for any number, the question of exactly how it was produced.
Provenance also underpins honest correction. If a raw dataset is later found to be flawed, provenance identifies every downstream result derived from it, so the correction can be propagated completely rather than leaving stale conclusions in place. A record without provenance cannot be corrected reliably, only distrusted wholesale.