The Advantage Function

The advantage function measures how much better an action is than the policy's average, sharpening the learning signal for policy gradients.

Better than average

The advantage function is A^pi(s, a) = Q^pi(s, a) - V^pi(s). It answers a focused question: how much better or worse is taking action a in state s compared with what the policy would do on average from s? Positive advantage means the action beats the baseline; negative means it lags.

Why subtract the value

Kronos motion — learning physics

In policy gradients, weighting each action by its raw return or Q value mixes in the overall goodness of the state, which is irrelevant to which action to prefer there. Subtracting V^pi removes that state-dependent offset, leaving only the relative merit of the action. This is a variance-reducing baseline that keeps the gradient unbiased.

Estimating the advantage

One-step TD advantage: r + gamma V(s') - V(s), the TD error itself.
n-step: use several rewards before bootstrapping, trading bias for variance.
Generalized advantage estimation (GAE): an exponentially weighted average of n-step advantages controlled by a parameter lambda.

Generalized advantage estimation

GAE is the standard modern estimator. It blends short- and long-horizon advantage estimates with a decay lambda, much as TD(lambda) blends returns, giving a tunable bias-variance trade-off. It is a key component of PPO and other actor-critic methods.

Role in algorithms

The advantage is the weight of choice in nearly all modern policy optimization. A2C and A3C are named for it (Advantage Actor-Critic), and dueling DQN architectures explicitly separate a value stream from an advantage stream. Sharpening the credit signal this way is one of the most effective and general tricks in the field.