Regularization
Techniques that constrain a model's complexity to improve generalization and reduce overfitting.
Definition
Regularization adds a preference for simpler models, usually by penalizing large parameter values in the loss. This trades a small increase in training error for better performance on unseen data.
Regularization can be viewed through a Bayesian lens: an L2 penalty corresponds to a Gaussian prior on the weights, and L1 to a Laplace prior. Seeing it this way clarifies that regularization encodes prior belief about plausible models, not an arbitrary trick to suppress numbers.
Different regularizers encode different beliefs: L1 assumes many features are irrelevant and should be zeroed, while L2 assumes all features contribute a little. Choosing between them, or combining them as in elastic net, is therefore a statement about the expected structure of the solution. Early stopping regularizes implicitly by limiting how far the optimizer moves from its initialization.
Common forms
- L2 (ridge): penalizes the sum of squared weights, shrinking them smoothly.
- L1 (lasso): penalizes absolute weights, driving many to exactly zero for feature selection.
- Dropout: randomly disables units during training.
- Early stopping: halt training when validation error stops improving.
Why it matters
Regularization is the primary lever against overfitting. It embeds the principle that, absent strong evidence, a simpler explanation should be preferred, improving robustness when deployment data differs slightly from training data.
Fusion connection
Regularized surrogates for Hyperion behave more smoothly across the design space, avoiding wild extrapolations between the simulated points they were trained on.