Trust Region Policy Optimization

TRPO improves a policy while limiting how far it moves, using a constraint on policy change to guarantee monotonic improvement.

Safe steps in policy space

Trust Region Policy Optimization (TRPO) is a policy-gradient method that addresses a basic fragility: a single overlarge update can destroy a good policy. TRPO improves the policy while constraining how much it is allowed to change per step, keeping updates within a trust region where the improvement estimate remains reliable.

The constraint

TRPO maximizes a surrogate objective — the expected advantage under the new policy relative to the old — subject to a constraint that the average KL divergence between the new and old policies stays below a small threshold. The KL constraint measures how much the action distribution has shifted, bounding the step in a way that raw parameter distance cannot.

Monotonic improvement

The method rests on a theoretical result: if the policy change is small enough, the surrogate objective lower-bounds the true improvement, so optimizing it within the trust region guarantees the true return does not decrease. This monotonic-improvement property is TRPO's main appeal over plain policy gradients.

The cost

Enforcing the KL constraint requires estimating and inverting a curvature (Fisher information) matrix, done approximately with conjugate gradient and a line search. This makes TRPO more complex and computationally heavier than first-order methods, which limited its everyday adoption.

The successor

PPO was designed to capture TRPO's stability with a much simpler first-order objective: instead of a hard KL constraint it clips the probability ratio, achieving a similar limit on policy change without second-order machinery. PPO largely replaced TRPO in practice, but TRPO clarified why constraining the step matters and what property good policy updates should preserve.