Actor-Critic Methods

Actor-critic methods pair a policy that acts with a value estimator that critiques, combining low-variance updates with direct policy optimization.

Two networks, two jobs

Actor-critic methods combine value-based and policy-based reinforcement learning. The actor is a parameterized policy that selects actions. The critic is a learned value function that evaluates them. The critic's estimate replaces the noisy Monte Carlo return that made REINFORCE unstable.

How they interact

Kronos motion — actor critic

The critic computes a TD error, delta = r + gamma V(s') - V(s), which estimates the advantage of the action just taken. The actor updates its parameters in the direction grad log pi(a | s) * delta, making beneficial actions more likely. The critic updates its own parameters to reduce the same TD error.

Why it works well

Lower variance: the critic's bootstrapped estimate is far less noisy than a full return.
Online: updates happen every step, not only at episode end.
Flexible: handles continuous actions and stochastic policies naturally.
General: it instantiates generalized policy iteration with the two processes run in parallel.

The bias-variance balance

Bootstrapping through the critic introduces bias in exchange for lower variance — the reverse of REINFORCE. Getting the balance right is why practical actor-critic methods use techniques like GAE, target networks, and careful learning-rate ratios between actor and critic.

The family

Actor-critic is a template rather than a single algorithm. A2C and A3C add parallel workers, PPO adds a clipped trust region, DDPG and TD3 make it deterministic and off-policy, and SAC adds entropy for exploration.