Overfitting and Generalization
A model that memorizes its training data fails on new data; the goal is to capture the signal without fitting the noise.
Fitting the Noise
Overfitting occurs when a model learns the idiosyncrasies and noise of its training data rather than the underlying pattern. Such a model scores well on data it has seen and poorly on data it has not. The complementary failure, underfitting, is a model too rigid to capture the real structure at all. The aim is the middle: generalization, good performance on unseen data.
How to Recognize It
- A large gap between training performance and held-out performance signals overfitting.
- Poor performance on both signals underfitting.
- Small gap with good performance on both is the target.
- Wildly complex fits that wiggle through every point are a visual warning sign.
Why More Flexibility Is Not Free
Every added parameter gives a model more freedom to bend toward the data, including its noise. Beyond the point where added flexibility captures real structure, it begins capturing accident, and out-of-sample error rises even as in-sample error keeps falling. This is why in-sample fit alone is a dangerous guide.
Defenses
Practitioners fight overfitting with more or cleaner data, with regularization that penalizes complexity, with simpler model classes, and with honest held-out evaluation that never lets the test data influence training. Early stopping, which halts training when held-out error starts to rise, is a common practical guard for iterative learners.
In Scientific Surrogates
A surrogate model fit to simulation outputs can overfit the specific runs it saw and mislead outside them. Kronos surrogate and data-driven work is evaluated on held-out cases and its valid input range is stated, so a fast approximation is not trusted beyond where it was actually tested against the underlying physics.