Least Squares Estimation
Least squares chooses model parameters that minimize the total squared gap between predictions and observations.
The principle
Given a model with parameters θ and predictions ŷi(θ), least squares minimizes S(θ) = Σ (yi − ŷi(θ))². Squaring makes all errors positive, penalizes large misses heavily, and produces a smooth objective that calculus can minimize.
The normal equations
For a linear model y = Xβ + ε, setting the gradient of S to zero gives the normal equations XᵀX β = Xᵀy, whose solution is β̂ = (XᵀX)⁻¹ Xᵀy when XᵀX is invertible. This is the matrix form behind every linear least-squares fit.
Geometric view
Least squares projects the observation vector y orthogonally onto the space spanned by the model's columns. The fitted values are that projection and the residuals are perpendicular to it. This geometry explains why residuals are uncorrelated with the predictors by construction.
Numerical practice
Solving the normal equations directly can be ill-conditioned when predictors are nearly collinear. Stable implementations use the QR decomposition or the singular value decomposition instead of inverting XᵀX. This is a case where the textbook formula and good numerical code differ.
Beyond ordinary least squares
- Weighted least squares handles unequal error variances.
- Ridge and lasso add penalties to control variance and select predictors.
- Nonlinear least squares fits curved models by iterative optimization.
Under independent normal errors, least squares is exactly maximum likelihood, which is why the method is both intuitive and statistically principled.