Bias-Variance Trade-off
The tension between a model too simple to fit the pattern and one so flexible it fits the noise.
Definition
The bias-variance trade-off decomposes a model's expected error into bias, error from overly simple assumptions, and variance, error from sensitivity to the particular training sample. Reducing one often raises the other.
Recent findings on very large models complicate the classic picture: beyond the point where a model can fit the training data exactly, test error can fall again, a phenomenon called double descent. The trade-off remains a vital guide but is not the whole story at extreme scale.
The trade-off is a way of reasoning about generalization, not a formula to compute, since bias and variance are rarely measured directly. Its practical value is diagnostic: a large train-test gap points to variance and calls for more data or regularization, while poor performance on both points to bias and calls for a more capable model or better features. This diagnosis guides the next experiment.
The two extremes
- High bias, low variance: underfitting, misses real structure.
- Low bias, high variance: overfitting, chases noise.
- The best model minimizes their sum, not either alone.
Managing it
Adding capacity lowers bias but raises variance; regularization and more data lower variance. Ensembles reduce variance by averaging, which is why random forests generalize well.
Why it matters
This trade-off is the conceptual core of generalization and guides nearly every modeling decision about complexity and data.
Fusion connection
Kronos sizes each surrogate's complexity to the available simulation data, avoiding both an oversimple model that misses physics and an overflexible one that fits noise.