Checkpointing and Fault Tolerance

How long-running Kronos simulations and training runs survive node failures without losing days of compute.

Long jobs will hit failures

L0 runs jobs that last hours to days across many nodes, and at that scale hardware failures are not exceptional but expected. A single node fault should never cost a whole campaign. Checkpointing and fault tolerance are what let long-running simulations and training runs survive failures and resume, rather than restart from zero.

Checkpointing

A checkpoint is a saved snapshot of a job's state, solver fields and iteration count, or model weights and optimizer state, written periodically to storage. If a node fails, the job restarts from the last checkpoint instead of the beginning. The checkpoint interval trades storage and I/O overhead against the amount of work lost on failure.

• Periodic state snapshots to durable storage
• Restart from last checkpoint after a fault
• Interval tuned to failure rate and job length
• Asynchronous writes to hide checkpoint cost

Tolerance strategies by workload

Different workloads tolerate faults differently. Monte Carlo is naturally resilient: histories are independent, so a lost batch is simply re-run and only its contribution is redone. Coupled solves and synchronous training are fragile: a lost rank stalls the whole job, so they lean on checkpoints and, where possible, elastic recovery that re-forms the job around a replacement node.

Checkpoints double as reproducibility anchors. A checkpoint plus the recorded environment and seeds lets a run be resumed or re-derived exactly, which ties fault tolerance to the broader reproducibility discipline. The same snapshot that survives a crash also lets an auditor replay from a known state.

Fault tolerance keeps both machines' study cadences reliable. A multi-day breeder gyrokinetic run or a burner stability campaign can complete despite node churn, and the scheduler's use of preemptible backfill depends on it: a preempted sweep job resumes from its checkpoint elsewhere with no scientific loss.