Rollback and Safe-State Fallback
Every deployed model has a certified predecessor and a physics-based safe state it can fall to within a bounded time, so a bad model is always reversible.
Reversibility as a precondition of deployment
A model is only allowed to act if there is a defined, tested way to stop it acting. Kronos maintains two fallback targets for every model that holds authority: the previous certified model version, and a model-free safe state driven entirely by the deterministic L1 control plane. Rollback to either is bounded in time and does not depend on the failing model behaving well.
Rollback is triggered automatically by monitor breaches — an uncertainty spike, an envelope violation attempt, a drift alarm, or a divergence beyond threshold — and can also be triggered manually by an operator at any time. Because the L1 failsafe path is certified independently of any ML component, the safe-state fallback holds even if every model is wrong at once.
The fallback ladder
- Level 0: revert to previous PROD model version (same task, known-good)
- Level 1: hand control to model-free deterministic controller
- Level 2: L1 safe-state sequence (breeder controlled ramp-down; burner beam-notch / plug de-energize)
- All levels: append rollback event to the registry, do not delete history
def guard(model, state, monitors):
if monitors.breach(state):
registry.log_rollback(model, cause=monitors.first_breach())
if registry.prev_prod(model.name):
return activate(registry.prev_prod(model.name))
return L1.safe_state() # model-free, certified path
return model.act(state)
Rollback is never a data-loss event: the quarantined model, the inputs that tripped it, and the full timeline are preserved for the postmortem that drives the next incident-driven retrain. The existence of a guaranteed rollback is precisely what makes canarying a new controller acceptable at all.