Inverse Reinforcement Learning

Inverse RL infers the reward function that best explains observed expert behavior, rather than optimizing a given reward.

Inferring the goal

Inverse reinforcement learning (IRL) turns the usual problem around. Instead of being given a reward and finding a policy, IRL is given demonstrations of good behavior and infers the reward function that would make that behavior optimal. It is useful when the goal is hard to specify by hand but easy to demonstrate.

Why infer the reward

Many desirable behaviors are difficult to encode as an explicit reward — smooth, natural, human-like control, or nuanced judgments of what counts as good. It is often easier to show examples than to write the objective. Recovering the reward, rather than just copying actions, yields a compact, transferable description of the goal that generalizes to new situations the demonstrations did not cover.

The ambiguity problem

IRL is underdetermined: many reward functions, including trivial ones, can explain the same behavior. Methods resolve this with additional principles, most notably maximum-entropy IRL, which prefers the reward that makes the observed trajectories most likely while assuming the least beyond the data.

Relation to imitation learning

Plain imitation learning (behavioral cloning) copies the expert's actions directly and can fail when the agent drifts into states the expert never visited. IRL instead recovers the underlying intent, so the agent can act sensibly in novel states. Adversarial imitation methods blend the two, learning a reward and a policy together.

Uses and caution

IRL underpins learning from human demonstration in robotics and preference-based reward learning for aligning models with human intent. A caution: an inferred reward inherits the biases and limitations of the demonstrations, so demonstrations must be representative and the recovered reward checked before it is trusted to drive behavior.