Reward Modeling
A reward model is a learned proxy for what we want, standing in for a true objective that is hard to specify.
When the objective must be learned
Reward modeling addresses tasks where the desired behavior is clear to a human but impossible to write as code: helpfulness, aesthetic quality, safe manipulation. Rather than hand-code the reward, we learn one from human evaluations, comparisons, or demonstrations, then optimize a policy against it.
Sources of supervision
- Comparisons: humans rank outputs; fit a reward under a Bradley-Terry preference model
- Ratings: absolute scores, though these are noisier and less consistent than comparisons
- Demonstrations: infer reward from expert behavior via inverse RL
The over-optimization trap
A learned reward is a proxy, and optimizing a proxy hard enough breaks it, an instance of Goodhart's law. As the policy pushes into regions where the reward model was never trained, its errors become exploitable and measured reward rises while true quality falls. Countermeasures include penalizing divergence from a trusted policy (a KL leash), keeping the reward model uncertain and conservative out of distribution, and periodically collecting fresh human data on the policy's current outputs.
# Guarded objective: pursue proxy reward but stay near a base policy
# maximize E[ r_model(x) ] - beta * KL( policy || base_policy )Recursive and scalable variants
When outputs exceed what a human can evaluate directly, reward modeling can be made recursive: use AI assistance or decomposition so evaluators can judge complex behavior, an approach sometimes called scalable oversight. Reward modeling is the technical core of RLHF and a central tool in alignment: get the objective right, because RL will optimize exactly what you actually measure, not what you intended.