Reward Modeling for Control and Copilots

The reward model that RLHF optimizes against is itself a versioned, validated artifact — miscalibrated reward is a hazard, so it passes its own gates.

The reward model is a model too

In any RLHF system the reward model is the objective. If it is wrong, the policy optimized against it is confidently wrong. Kronos therefore treats the reward model as a first-class MLOps artifact: versioned, lineage-tracked, and gated, exactly like a controller or surrogate. A reward model that overfits or is miscalibrated is a defect that can propagate into every policy trained on it.

Reward models are validated on held-out human preferences and, critically, on their uncertainty. A reward model must know where it is unsure, because RL optimizers exploit exactly the regions where the reward is high but wrong. Reward hacking is the failure mode where a policy finds inputs the reward model scores highly but humans would reject; penalizing reward-model uncertainty is the primary defense.

Reward-model validation

• Held-out preference accuracy and agreement with expert consensus
• Calibration of reward and of its epistemic uncertainty
• Robustness to adversarial / out-of-distribution trajectories
• Stability across reward-model seeds (ensemble disagreement)
• Sanity on golden pairs where the correct ranking is known

# Bradley-Terry reward model loss over preference pairs
def reward_loss(r, better, worse):
    return -torch.log(torch.sigmoid(r(better) - r(worse))).mean()

# Deploy as ensemble; use disagreement as epistemic signal
def reward_with_uncertainty(models, x):
    rs = torch.stack([m(x) for m in models])
    return rs.mean(0), rs.std(0)     # std -> penalty in RL objective

Reward models feed both copilot RLHF and control-policy RLHF, but the bar is higher for control because the consequences are physical. A reward model used anywhere near control is ensembled and its disagreement is treated as a first-class uncertainty that flows into the validation gates.