Experiment Scheduler with Priority & Preemption

The scheduler allocates machine time by priority, preempts running work for safety, and prevents low-priority starvation.

Scarce, exclusive machine time

Both machines are single, exclusive resources: only one campaign owns the breeder or the burner at a time. The scheduler decides what runs next from a queue of approved experiment plans, using priority classes, and can preempt a running plan when a higher-priority need arises, above all a safety action.

Priority classes

Preemption is graceful

Preemption never yanks the machine. A preempted breeder shot is brought to a safe stop through its saga compensations before the higher-priority plan starts; the preempted plan is re-queued with its progress checkpointed. A P0 safety action is the exception that runs immediately through the abort path, which itself leaves the machine safe.

def admit(next_plan, running):
    if next_plan.priority < running.priority:      # lower number = higher priority
        safe_stop(running)             # run compensations, checkpoint, requeue
        return start(next_plan)
    return enqueue(next_plan)

# starvation guard: aging raises effective priority of long-waiting P3 plans

Anti-starvation and fairness

• Waiting plans age, raising their effective priority so opportunistic work eventually runs.
• Priority inheritance prevents a low-priority plan holding a shared calibration from blocking a high-priority one (see priority inheritance).
• Machine-conditioning and recovery windows are scheduled as first-class, non-preemptible maintenance.

Coordination with campaigns

The scheduler consumes plans emitted by the parameter-sweep orchestrator and hands each admitted plan to the campaign engine. Every admit, preempt, and requeue decision is logged for audit and reproducibility.

Machine state, not just the queue

Admission is not purely a queue decision; the scheduler also gates on machine readiness. A breeder plan is not admitted while the machine is in RECOVERY after a fault, and a burner plan is not admitted until the plug and throat fields are established and stable. The scheduler treats conditioning windows, vacuum readiness, and post-fault clearances as hard preconditions, so a high-priority plan waits for a safe machine rather than forcing an unsafe start. This keeps priority strictly about ordering approved-and-ready work, never about overriding the machine's physical state, which remains the domain of the envelope and interlocks.