Reproducibility in Fusion ML
Making results repeatable and auditable, the standard that separates credible work from a lucky run.
Why it is non-negotiable
A machine-learning result that cannot be reproduced is not a result. In fusion, where models may inform design and control decisions, reproducibility is the basis of trust: another team, or the same team later, must be able to obtain the same numbers from the same data and code.
What must be captured
- The exact dataset and version, with all cleaning steps
- Code, dependencies, and configuration
- Random seeds for training and data splitting
- Trained model weights and evaluation scripts
- The precise train, validation, and test splits used
Sources of irreproducibility
Undocumented preprocessing, data that changes without versioning, test-set tuning, and unstated random seeds all make results unrepeatable. Nondeterminism in hardware and libraries can cause small variations; these should be characterized so a reproduction knows what agreement to expect.
Reporting honestly
Reproducible reporting distinguishes what was tuned from what was held out, reports uncertainty across seeds and folds rather than a single best run, and states the data range so extrapolation is visible. Cherry-picking the best of many runs, or reporting the run that happened to look best on the test set, is the opposite of reproducible practice.
In a design setting
For work that feeds a design, such as the Kronos program's frozen physics basis, reproducibility means any cited figure can be traced to the data, code, and configuration that produced it, and regenerated on demand. Two-tier reproduction, exact byte-level where possible and tolerance-level where nondeterminism intrudes, is a practical standard. Results stated this way can be checked; results stated any other way must be taken on faith, which has no place in engineering.