Curriculum Learning
Curriculum learning orders training examples from easy to hard, mirroring how humans learn structured material.
Order of learning matters
Curriculum learning presents training examples in a meaningful order, typically easy examples first and hard ones later, rather than the usual random shuffle. The motivation is that a good ordering can guide optimization toward better regions of the loss surface, speed convergence, and improve final performance, much as a syllabus builds difficulty gradually.
Defining difficulty
- Heuristic: sentence length, image clutter, noise level, or human-judged difficulty
- Model-based: the current model loss or confidence on each example
- Self-paced learning: the model itself selects which examples to include as it improves
- Transfer-based: difficulty estimated by a separate teacher model
Pacing the curriculum
Beyond ranking examples, a pacing function decides how quickly harder examples enter training. Introduce hard examples too fast and the benefit vanishes; too slow and training wastes time on trivial cases. Common schedules start with the easiest fraction and gradually widen the pool until all data is used, often annealing back to uniform sampling near the end.
Evidence and caveats
Curricula help most when the task is hard, the data is noisy, or optimization is unstable; on clean, easy problems a well-tuned shuffle is often just as good, so the benefit is not universal. Anti-curricula (hard first) sometimes win, and hard-example mining, which emphasizes difficult cases, appears to contradict easy-first ordering. These are reconciled by noting that focusing on hard examples helps a competent model, while easy-first helps a model still finding its footing.
Curriculum ideas appear across reinforcement learning, where task difficulty is scheduled, and connect to multi-task learning when tasks are ordered rather than examples.