Q-Learning
A model-free reinforcement learning method that learns the value of actions to derive an optimal policy.
Definition
Q-learning learns a function Q(state, action) estimating the expected long-run reward of taking an action in a state and acting optimally thereafter. The optimal policy is then to pick the highest-value action in each state.
A key difficulty is that neural Q-functions can be unstable, since the target values shift as the network learns. Techniques such as a separate target network and experience replay, which reuses past transitions, were the innovations that made deep Q-learning work reliably.
Deep Q-learning's breakthrough on Atari games demonstrated learning skilled control directly from pixels, but it also exposed instabilities that required careful engineering to tame. The method suits problems with discrete actions; for continuous control, policy-based methods are usually preferred. Its core idea, bootstrapping value estimates from later value estimates, is powerful but can diverge without the stabilizing tricks that made it practical.
It updates Q toward the observed reward plus the discounted best next-state value, a rule derived from the Bellman equation, and it works without a model of the environment's dynamics.
Key features
- Off-policy: learns the optimal policy while exploring differently.
- Exploration via epsilon-greedy action selection.
- Deep Q-networks approximate Q with a neural network.
Why it matters
Q-learning is a foundational reinforcement learning algorithm, and its deep variant achieved landmark results in game playing, demonstrating learned control from raw observations.
Fusion connection
Value-based methods are studied for discrete control decisions in simulated fusion environments, where the agent learns which adjustment maximizes long-run plasma performance.