The Bias-Variance Trade-off
Prediction error splits into a part from wrong assumptions and a part from sensitivity to data; reducing one tends to raise the other.
Decomposing Error
The expected error of a predictor can be split into three parts: bias, error from a model too simple to capture the truth; variance, error from a model too sensitive to the particular training sample; and irreducible noise, error no model can remove. Understanding this decomposition explains why the most flexible model is rarely the best.
The Two Failure Modes
- High bias, low variance: a rigid model that misses the pattern the same way every time; this is underfitting.
- Low bias, high variance: a flexible model that chases each sample's noise and swings wildly between datasets; this is overfitting.
- The sweet spot: enough flexibility to capture the signal, enough restraint to ignore the noise.
The Trade-off
Making a model more flexible generally lowers bias but raises variance; making it simpler does the reverse. Total error is minimized at an intermediate complexity, not at either extreme. The art of model building is finding that balance for the data and question at hand, rather than reflexively reaching for the most powerful method.
Levers That Move the Balance
More training data reduces variance without adding bias, which is why data quantity often beats model cleverness. Regularization trades a little bias for a large cut in variance. Averaging many models, as ensembles do, reduces variance directly. Each lever repositions the balance rather than escaping the trade-off.
Why It Matters
The decomposition reframes model selection as variance control, not fit maximization. A model that fits the training data perfectly has driven bias to near zero at the cost of ruinous variance, which is exactly overfitting. Keeping the trade-off in view is what turns model tuning from guesswork into reasoning about where error is coming from.