Self-Play
In self-play an agent improves by competing against copies of itself, generating an ever-stronger curriculum.
An opponent that grows with you
Self-play trains an agent by having it play against itself or past versions of itself. The opponent is always matched to the agent's current skill, producing an automatic curriculum: as the agent improves, so does its adversary, and both are pushed to keep advancing. No external teacher or fixed opponent pool is required.
Why it produces strong play
Self-play was central to landmark results: TD-Gammon in backgammon, and AlphaGo Zero and AlphaZero in Go, chess, and shogi, which reached superhuman strength starting only from the rules. In these systems self-play generates the training data, and search converts the current network into a stronger target, so quality compounds each iteration.
Stability concerns
- Cycling: strategies can chase each other in circles (rock-paper-scissors dynamics) without net progress
- Overfitting to self: an agent may exploit quirks of its own past selves rather than learn general skill
- Forgetting: beating only the latest version can erase the ability to beat older ones
Opponent sampling
Remedies keep and sample from a population of past checkpoints rather than always facing the newest version, which preserves robustness and prevents strategy collapse. Fictitious self-play and Policy-Space Response Oracles formalize this by best-responding to a mixture over past policies, with connections to computing Nash equilibria in two-player zero-sum games.
Scope
Self-play applies naturally to symmetric competitive games and to any setting expressible as an agent versus an equally capable adversary, including adversarial training and some multi-agent tasks. Its power is turning a hard exploration problem into a sequence of achievable ones.