Offline Batch RL from Logged Pulses

Control policies are learned from the logged L0 pulse archive using conservative offline RL, never by exploring on the live machine, where exploration is a fault.

No exploration on hardware

Classical reinforcement learning explores by trying actions and observing outcomes. On a fusion machine, an exploratory action is a candidate disruption or a coil overstress. Kronos therefore uses offline RL exclusively: policies are learned from the fixed dataset of logged breeder and burner pulses, plus twin-generated rollouts, with no live exploration whatsoever.

Offline RL has a characteristic hazard: the policy can favor actions that are rare or absent in the data, where value estimates are unfounded optimism. Kronos uses conservative algorithms (such as CQL and implicit-Q-learning style objectives) that penalize out-of-distribution actions, keeping the learned policy near the support of the logged behavior. The policy is pulled toward what has actually been observed to be safe.

Offline-RL safeguards

• Conservative value estimation penalizing OOD actions
• Behavior-cloning regularization toward logged policies
• Uncertainty-aware action selection; abstain when unsure
• Twin-based evaluation before any real-data claim
• Standard gate + shadow + canary chain before authority

# Conservative Q-learning objective (schematic)
# minimize TD error + push down Q on OOD actions,
# pull up Q on actions actually seen in the dataset
L = td_error(Q, batch) \
  + alpha * (logsumexp(Q(s, all_actions))         # OOD penalty
             - Q(s, batch.action).mean())         # in-data bonus
# policy stays within logged support -> safe to evaluate

Offline RL is where preference-shaped rewards from control-policy RLHF are actually optimized into a policy. Everything learned here is evaluated first on the twin, then walked through the same validation and canary chain as any controller. The dataset it learns from carries the full data lineage, so the provenance of a learned policy reaches back to specific pulses.