Compute Scheduling and Orchestration
How L0 packs Monte Carlo, coupled solves, and GPU training onto a shared pool without starving the reproducible core.
Three workload shapes, one queue discipline
L0 runs three very different shapes on one pool: throughput-parallel Monte Carlo, latency-bound coupled solves, and GPU training. A single fair-share scheduler cannot treat them alike. Kronos orchestration classifies each job by shape and reproducibility tier, then places it where it will neither waste the pool nor collide with sensitive work.
Backfill for the wide frontier
Monte Carlo and hyperparameter sweeps are ideal backfill: they are independent, checkpointable, and preemptible. The scheduler uses them to fill gaps left by large coupled solves, keeping utilization high without delaying tightly scheduled reproducible runs. A breeder TBR sweep can be paused, migrated, and resumed with no scientific loss.
Reservations for the reproducible core
Certified reproducible runs, the coupled MHD-thermomechanics solves that seed twin surrogates, get reserved placement on pinned nodes. They are never co-scheduled with noisy neighbors that could perturb timing or numerics. The scheduler treats reproducibility as a hard constraint, not a preference.
- Shape-aware placement: throughput, coupled, or training
- Preemptible backfill for sweeps and Monte Carlo
- Hard reservations for certified deterministic runs
- Gang scheduling for multi-node coupled and distributed-training jobs
Coupled solves and distributed training both need gang scheduling: all ranks start together or not at all, because a partial allocation deadlocks a synchronous job. Orchestration allocates these atomically and holds them until the full set of nodes and the required fabric topology are available.
The scheduler is machine-blind by design. A breeder equilibrium campaign and a burner end-cell study compete for the pool on equal terms, arbitrated by shape, tier, and priority. This keeps both machines' study cadences moving without either one monopolizing the substrate.