Reinforcement Learning from Human Feedback
RLHF trains a reward model from human preference comparisons, then optimizes a policy against it.
When reward is hard to write down
For many tasks, especially open-ended text generation, there is no programmable reward that captures what people actually want. RLHF learns the objective from human judgments instead of hand-coding it. It has three stages: supervised fine-tuning, reward-model training, and policy optimization.
Preference-based reward modeling
Humans find it far easier to compare two outputs than to score one on an absolute scale. RLHF collects pairs (or rankings) and fits a reward model under the Bradley-Terry assumption: the probability that response A is preferred to B is sigmoid(r(A) - r(B)). Training maximizes the likelihood of the observed preferences, yielding a scalar reward for any output.
# Reward-model loss on a preference (chosen > rejected)
loss = -log(sigmoid(r(chosen) - r(rejected)))Policy optimization with a KL leash
A policy is then optimized to maximize the learned reward, typically with PPO. A crucial term penalizes the KL divergence from the original supervised model. Without it the policy drifts into strange, high-reward-model but low-quality regions, a form of reward hacking. The KL penalty keeps outputs close to the trusted starting distribution while nudging them toward preferred behavior.
Strengths and pitfalls
- Optimizes hard-to-specify goals directly from human judgment
- The reward model is imperfect and can be over-optimized (Goodhart's law)
- Human labels carry bias and inconsistency that propagate into the policy
RLHF is the standard alignment recipe for instruction-following language models. Simpler alternatives such as direct preference optimization skip the explicit reward model and the RL loop, optimizing the policy on preferences with a single supervised-style objective, while inheriting the same dependence on preference-data quality.