Multi-Task Learning
Multi-task learning trains one model on several related tasks so shared structure improves all of them.
Learning many things at once
Multi-task learning trains a single model to perform several related tasks simultaneously, sharing representations across them. The premise is that related tasks depend on overlapping structure, so learning them together acts as a mutual inductive bias: each task regularizes the others, and features learned for one help another, especially tasks with little data of their own.
Sharing architectures
- Hard parameter sharing: a shared backbone with a small task-specific head per task; strong regularization, low overfitting risk
- Soft parameter sharing: separate models whose parameters are encouraged to stay similar via a penalty
- Mixture-of-experts and gating: shared experts routed per task or per example
Balancing the losses
The combined objective is a weighted sum of per-task losses, and the weights matter. Tasks with larger gradients or faster convergence can dominate, starving others. Remedies include uncertainty-based weighting that learns a weight per task, gradient normalization to equalize learning speeds, and methods that resolve conflicting gradients by projecting them apart.
When it helps and when it hurts
Multi-task learning helps most when tasks are genuinely related and complementary; it can hurt through negative transfer when tasks conflict, pulling shared parameters in incompatible directions. Diagnosing this requires comparing multi-task performance against strong single-task baselines per task, not just the average. Auxiliary tasks are a common special case: extra objectives added only to improve a primary task, discarded at deployment.
It relates to curriculum learning in ordering what a model learns, and to transfer learning, which shares knowledge sequentially rather than jointly.