Reproducible Training Runbooks

Every training run is deterministic and re-executable: pin the data, seed, code, and container, and two runs produce the same model — reproducibility is a hard gate.

If you cannot reproduce it, you cannot certify it

A model that cannot be regenerated cannot be trusted to run a fusion machine, because there is no way to prove what it is. Kronos enforces reproducibility as a hard gate: a registered training job, re-executed from its pinned inputs, must reproduce the model within a declared tolerance. Non-reproducible training is a defect, not an inconvenience.

Reproducibility is engineered end to end. The dataset is content-addressed; random seeds are fixed for every source of nondeterminism; the code is a specific commit; the environment is a container image pinned by digest; and hardware nondeterminism (nondeterministic GPU kernels, reduction order) is either disabled or bounded by the tolerance. A two-tier standard applies: byte-identical where achievable, tolerance-identical otherwise.

What a runbook pins

• Dataset hash and split hashes
• All seeds: init, shuffling, augmentation, dropout, framework RNGs
• Code commit and container image digest
• Deterministic kernel flags where available
• Declared reproduction tolerance (Tier-1 byte / Tier-2 numeric)

def make_deterministic(seed):
    random.seed(seed); np.random.seed(seed)
    torch.manual_seed(seed)
    torch.use_deterministic_algorithms(True)
    torch.backends.cudnn.benchmark = False

# repro gate: retrain from manifest, compare to registered artifact
assert diff(retrained, registered) <= REPRO_TOL

A two-tier reproduction standard is applied because full bit-identity is not always achievable on accelerators: Tier-1 requires byte-identical artifacts where deterministic kernels exist, and Tier-2 requires numeric agreement within a declared tolerance where reduction order or vendor libraries introduce bounded nondeterminism. Each model records which tier it met.

This runbook discipline is what makes scheduled retraining trustworthy and what lets the reproducibility gate pass or fail objectively. It also underpins forensic replay: after any incident, the exact model that held authority can be rebuilt from its manifest and interrogated.