Ridge Regression

Ridge regression adds an L2 penalty to least squares, shrinking weights to reduce variance and tame correlated features.

The penalty

Ridge regression minimizes squared error plus a penalty on the squared size of the weights: L(w) = sum (y_i - x_i . w)^2 + alpha * sum w_j^2. The hyperparameter alpha controls the strength. As alpha grows, weights shrink toward zero, trading a little bias for a large drop in variance.

Why shrinkage helps

When features are correlated, ordinary least squares can produce huge, unstable weights that cancel each other. The L2 penalty makes X^T X + alpha*I always invertible and well-conditioned, giving a stable closed form w = (X^T X + alpha*I)^{-1} X^T y. Predictions become far less sensitive to noise in the training data.

from sklearn.linear_model import Ridge
model = Ridge(alpha=1.0)
model.fit(X_train, y_train)
# tune alpha by cross-validation, never on the test set

Ridge versus lasso

• Ridge (L2) shrinks all weights smoothly but rarely sets any to exactly zero.
• Lasso (L1) drives some weights to exactly zero, performing feature selection.
• Ridge handles correlated groups gracefully; lasso arbitrarily picks one of a group.

Choosing alpha

alpha is a hyperparameter, selected by cross-validation over a log-spaced grid. Standardize features first so the penalty treats them on equal footing. Ridge is the default regularizer when you expect many small effects rather than a few dominant ones.

Ridge is a concrete instance of regularization, the general principle of constraining a model to improve generalization.