Distributional Reinforcement Learning
Instead of predicting the mean return, distributional RL models the full distribution of possible returns, improving stability and risk awareness.
Modeling the whole distribution
Standard value-based RL predicts the expected return, a single number per state-action pair. Distributional reinforcement learning instead models the entire probability distribution of returns. The same action can yield very different outcomes across runs, and capturing that spread carries information the mean discards.
The distributional Bellman equation
The Bellman equation has a distributional form: the distribution of returns from a state equals the immediate reward plus a discounted, shifted copy of the next state's return distribution. Algorithms learn this distribution directly rather than collapsing it to its mean at every step.
How it is represented
- Categorical (C51): a fixed set of return values with learned probabilities.
- Quantile regression: learn the return's quantiles, adapting the support to the data.
- Implicit quantile networks: sample quantiles to approximate the full distribution flexibly.
Why it helps
Even when only the mean is used to choose actions, learning the full distribution often improves performance and stability. The richer target provides a stronger training signal and better-shaped representations. Distributional value learning was a key component of the Rainbow agent that combined several DQN improvements.
Risk-sensitive control
Knowing the distribution enables risk-sensitive decisions: an agent can prefer actions with acceptable worst cases rather than the best average, avoiding rare catastrophic outcomes. For control of high-value or safety-critical physical systems, where a rare failure matters far more than a slightly lower average, reasoning about the tail of the return distribution rather than just its mean is directly useful.