Regularization
Regularization constrains a model to prefer simpler solutions, trading a little training fit for better generalization.
Penalizing complexity
Regularization is any technique that discourages a model from fitting the training data too closely, in exchange for better performance on new data. Most often it adds a penalty on model complexity to the loss, so the optimizer balances fitting the data against staying simple. It is the primary tool against overfitting.
Norm penalties
- L2 (ridge): penalizes the sum of squared weights; shrinks all weights smoothly toward zero.
- L1 (lasso): penalizes the sum of absolute weights; drives some weights to exactly zero, selecting features.
- Elastic net: a weighted mix of L1 and L2.
- A strength hyperparameter (alpha or lambda) sets how hard the penalty bites.
See ridge, lasso, and elastic net for the linear-model cases.
Beyond norm penalties
- Early stopping: halt training when validation error stops improving.
- Dropout: randomly zero units during neural-network training.
- Data augmentation: expand the training set with label-preserving transforms.
- Tree constraints: max depth, min samples per leaf, pruning.
The Bayesian view
A penalty on weights is equivalent to a prior belief about them: L2 corresponds to a Gaussian prior, L1 to a Laplace prior. Regularized fitting is then maximum a posteriori estimation. This view explains why regularization pulls parameters toward a preferred, simple region unless the data insists otherwise.
Set the strength by cross-validation. Too much regularization causes underfitting; too little lets overfitting return. The right amount sits at the bottom of the validation-error curve.