GAIL: Generative Adversarial Imitation Learning
GAIL imitates an expert by matching state-action distributions through an adversarial game, skipping explicit reward recovery.
Imitation as distribution matching
Behavior cloning matches actions pointwise and suffers from compounding error. Inverse RL recovers a reward but is expensive. Generative Adversarial Imitation Learning (GAIL) takes a direct route: make the learner's state-action distribution match the expert's, using the adversarial framework of generative adversarial networks.
The adversarial game
A discriminator D is trained to tell expert state-action pairs from the learner's; the policy (generator) is trained to fool it. The discriminator's output becomes the reward: the policy is rewarded for producing pairs the discriminator judges expert-like, typically r = -log(1 - D(s,a)) or log D(s,a). Policy optimization uses a standard RL algorithm such as TRPO or PPO.
# GAIL alternation
# 1) update D to classify expert vs policy pairs
d_loss = -(log(D(expert_sa)) + log(1 - D(policy_sa)))
# 2) reward policy by how expert-like D judges it, then RL step
reward = -log(1 - D(s, a))Why it beats cloning
Because the policy is trained by RL, it visits its own states during training and learns to recover from drift, eliminating behavior cloning's covariate shift. And because it matches occupancy distributions rather than recovering an explicit reward, it avoids the inner RL loop that makes classical inverse RL slow. GAIL can match expert performance from relatively few demonstrations.
Costs and descendants
- Requires environment interaction (unlike behavior cloning), and inherits GAN training instability
- Sensitive to reward form and discriminator overfitting
- AIRL extends it to recover a transferable, disentangled reward function
GAIL formalized the link between imitation, inverse RL, and adversarial learning. Its descendants (AIRL, discriminator-actor-critic) improve sample efficiency and reward transfer, making adversarial imitation a mainstay when demonstrations exist but a reward does not.