Temporal-Difference Learning

Temporal-difference learning updates value estimates from one step of experience by bootstrapping off its own next estimate.

Learning from one step

Temporal-difference (TD) learning is a central idea in reinforcement learning: update a value estimate using a single observed transition, correcting toward the immediate reward plus the discounted estimate of the next state. It combines the model-free sampling of Monte Carlo with the bootstrapping of dynamic programming.

The TD(0) update

Kronos motion — next scientists

After observing a transition (s, r, s'), TD(0) updates V(s) <- V(s) + alpha [ r + gamma V(s') - V(s) ]. The bracketed term is the TD error: the gap between the current estimate and a slightly better one-step estimate. alpha is the learning rate.

Bootstrapping

The key move is bootstrapping: the update uses V(s'), the agent's own estimate of the next state, rather than waiting for the full return. This lets TD learn online, step by step, during an episode and even in continuing tasks that never end — something Monte Carlo cannot do.

Bias-variance trade-off

Monte Carlo: unbiased, high variance, needs complete episodes.
TD: some bias from bootstrapping, lower variance, updates every step.
In practice TD often learns faster and is more broadly applicable.

The TD error as a signal

The TD error is more than a bookkeeping term. It resembles reward-prediction signals observed in biological learning, and it drives most modern algorithms: SARSA and Q-learning are TD control methods, and the critic in actor-critic uses the TD error to guide the policy. TD(lambda) interpolates between TD and Monte Carlo.