Double Q-Learning and Overestimation

The max in Q-learning systematically overestimates values; double Q-learning corrects it by decoupling action selection from evaluation.

The overestimation bias

Q-learning forms its target with a maximum over next-state action values. When those values are noisy estimates, the maximum tends to pick out actions whose values happen to be overestimated, so the target is biased upward. This maximization bias compounds through bootstrapping and can produce badly inflated value estimates.

Why the max is to blame

Taking the maximum of noisy estimates is not the same as the maximum of the true values. Even if every estimate is unbiased individually, the max of several of them is biased high, because the estimate that comes out on top is disproportionately likely to be one with positive error. The single estimator both chooses and evaluates the best action, and the two roles reinforce the error.

Double Q-learning

Double Q-learning breaks this coupling by keeping two independent value estimators. One selects the action believed best; the other evaluates that action's value. Because the estimator that picks the action is not the one that scores it, the upward bias is largely removed. The two estimators are updated on alternating experiences so they stay independent.

In deep RL

Double DQN applies the idea cheaply within the deep Q-network: the online network selects the next action and the target network evaluates it, reusing the target network that DQN already maintains. This small change reliably reduces overestimation and improves stability with almost no extra cost.

Broader relevance

The same insight motivates the twin critics of TD3 and SAC, which take the minimum of two Q estimates to counter overestimation in continuous control. Recognizing and correcting maximization bias is one of the most broadly useful lessons in value-based RL.