Imitation Learning

Imitation learning trains a policy to mimic expert demonstrations, bypassing reward design when good examples are available.

Learning from demonstration

Imitation learning trains an agent to reproduce expert behavior from a dataset of demonstrations, sidestepping the difficulty of designing a reward. When examples of good behavior are plentiful and a reward is hard to write, imitation can be far faster than learning from scratch by trial and error.

Behavioral cloning

Kronos motion — design envelope

The simplest approach, behavioral cloning, treats imitation as supervised learning: given states as inputs and the expert's actions as labels, train a policy to predict the expert action. It is easy and effective when demonstrations cover the relevant situations well.

Compounding errors

Behavioral cloning has a well-known weakness: covariate shift. Small prediction errors move the agent into states the expert never visited, where the policy was never trained, causing larger errors and a cascade of drift away from good behavior. The agent's mistakes take it off the distribution it learned from.

Fixing the drift

DAgger: iteratively collect the expert's action on states the learner actually visits, expanding coverage where the policy goes.
Inverse RL: recover the reward behind the demonstrations so the agent can recover in novel states — see inverse RL.
Adversarial imitation: match the distribution of the learner's trajectories to the expert's using a discriminator.

Combining with RL

Imitation and reinforcement learning complement each other. Demonstrations can seed a policy or warm-start a value function, giving RL a strong starting point that skips the slow early exploration phase; RL then refines the policy beyond mere imitation using reward. This hybrid is common when a competent baseline controller or human operator can supply examples to bootstrap a simulated learner.