Deep Q-Networks (DQN)

DQN scales Q-learning to high-dimensional inputs using a neural network, experience replay, and a slowly updated target network.

Q-learning meets deep learning

The deep Q-network (DQN) uses a neural network to approximate the action-value function Q(s, a; w), enabling Q-learning on raw high-dimensional inputs such as images. It became widely known for learning to play many Atari games from pixels using a single architecture, and it demonstrated that deep function approximation could be stabilized enough to work.

The instability problem

Kronos motion — learning physics

Naively combining Q-learning with a neural network hits the deadly triad — approximation, bootstrapping, and off-policy learning — and tends to diverge. DQN introduced two mechanisms that made training reliable.

Experience replay

The agent stores transitions (s, a, r, s') in a large buffer and trains on random minibatches sampled from it. This breaks the strong temporal correlation between consecutive samples and reuses each experience many times, improving both stability and sample efficiency. Prioritized replay later biased sampling toward high-error transitions.

Target network

The bootstrapped target uses a separate target network whose weights are a delayed copy of the online network, updated only periodically. Freezing the target for a while stops the estimate from chasing a moving target and greatly reduces oscillation and divergence.

Refinements

Double DQN: decouples action selection from evaluation to curb overestimation.
Dueling DQN: separates state value and advantage streams.
Prioritized replay: samples informative transitions more often.
Rainbow: combines several improvements into one agent.

DQN handles discrete action spaces; continuous control needs actor-critic methods such as DDPG or SAC.