Preference-Based RL for Control Policies

Human preference feedback shapes control-policy behavior where a scalar reward is hard to specify, but only in simulation and only behind the full validation and canary chain.

When reward is easier to judge than to write

Some control objectives are hard to encode as a clean scalar: a good breeder shape trajectory or a graceful burner ramp is partly a matter of expert judgment about smoothness, margin, and robustness. Preference-based RL lets experts compare pairs of simulated trajectories and rank them; a reward model learns the implicit objective; and a policy is optimized against it. This complements, it does not replace, the physics-based cost terms.

This is used with extreme caution. Human-preference-shaped control policies are trained and evaluated only in simulation and on the digital twin. They inherit the entire deployment chain — validation gates, shadow, canary, rollback — before any authority, and the physics safety envelope always dominates the learned reward. A learned preference can shape behavior inside the envelope; it can never widen it.

Guardrails specific to control RLHF

• Preferences over simulated / twin trajectories only, never live rollouts
• Physics constraints as hard barriers, not soft reward terms
• Reward model uncertainty penalized to avoid reward hacking
• Conservative offline-RL objective (see offline RL)
• Full gate + shadow + canary before authority, per the standard pipeline

# Preference-based reward, constrained
r_pref = reward_model(traj)                     # learned from rankings
r_safe = -LARGE if envelope_violated(traj) else 0
r_unc  = -lambda_u * reward_model.epistemic_std(traj)  # anti-hacking
return r_pref + r_safe + r_unc     # safety term dominates by design

For the burner especially, the honest gates cap what any policy can be trained to claim: a policy cannot be rewarded into operating a plug regime that is 166 to 830x beyond any existing device, because such trajectories cannot be validated and are barred by the honest-gate checks. RLHF here is a tool for polishing behavior inside proven bounds, not for reaching past them.