Bias-Variance Tradeoff

Prediction error splits into bias, variance, and noise; reducing one of the first two often raises the other.

Decomposing error

For squared-error loss, a model's expected error on a new point decomposes into three parts: bias squared, variance, and irreducible noise. Bias is error from wrong assumptions (too simple a model); variance is error from sensitivity to the particular training sample; noise is randomness no model can remove.

The two failure modes

• High bias, low variance: underfitting. The model is too simple, misses real structure, and errs consistently.
• Low bias, high variance: overfitting. The model is too flexible, chases noise, and changes wildly with the data.
• The sweet spot minimizes their sum, not either alone.

How model complexity moves them

As you make a model more flexible, more tree depth, more features, higher polynomial degree, bias falls and variance rises. Training error keeps dropping, but test error follows a U-shape: it improves, bottoms out, then worsens as overfitting sets in. The bottom of that U is the target.

# symptoms in numbers
# underfit:  train error high, val error high (close together)
# overfit:   train error low,  val error high (large gap)
# good fit:  train error low,  val error low  (small gap)

Levers you control

Reduce variance with more data, regularization, simpler models, or bagging. Reduce bias with a richer model, better features, or boosting. Diagnose which problem you have from learning curves and the train-versus-validation gap, then pull the matching lever.

This tradeoff is the organizing idea of practical machine learning: nearly every design choice is a move along the bias-variance axis.