Reproducibility Runbooks
A runbook records exactly how to reproduce a result, from environment to inputs to commands, so anyone can regenerate it later.
Why runbooks exist
A result is only reproducible if someone else, or the same person months later, can regenerate it. That requires knowing the exact code version, the inputs, the environment, and the steps. A runbook captures all of this so reproduction is a matter of following instructions, not reverse-engineering what was done.
What a runbook records
- The exact code version and its dependencies.
- The input files and parameters used.
- The computing environment, pinned so it can be recreated.
- The commands to run and the expected outputs.
Pinning the environment
Software environments drift as libraries update, and a drifted environment can change results. Runbooks pin the environment, often with containers or lock files, so the same inputs produce the same outputs on a cold machine years later. This is the difference between reproducible in principle and reproducible in fact.
Two tiers of reproduction
Runbooks state whether a result reproduces exactly, byte for byte, or within a numerical tolerance, matching the two tiers used in automated verification. Stating the standard is part of the runbook.
For Kronos
The analyses behind the Hyperion breeder and the burner each carry a runbook, forming the backbone of the 81-analysis record and the licensing evidence package.
The payoff
Runbooks turn reproducible science from an aspiration into a routine, letting reviewers, collaborators, and regulators check the work independently.