Linear Regression
Linear regression fits a straight-line relationship between predictors and a response by minimizing squared error.
The model
Simple linear regression models a response y as y = β₀ + β₁x + ε, where ε is random noise with mean zero. The coefficients β₀ (intercept) and β₁ (slope) are chosen to fit observed pairs (xi, yi). Multiple regression extends this to many predictors.
Fitting by least squares
The standard fit minimizes the sum of squared residuals Σ (yi − ŷi)². For simple regression this gives β₁ = Cov(x, y)/Var(x) and β₀ = ȳ − β₁ x̄. Squared error yields a closed-form solution and, under normal noise, coincides with the maximum likelihood estimate.
Interpreting coefficients
The slope β₁ is the expected change in y per unit change in x, holding other predictors fixed. This 'holding fixed' clause is essential in multiple regression and easy to forget: a coefficient describes a partial effect within the model, not a raw association.
Goodness of fit
The coefficient of determination R² is the fraction of variance in y explained by the model, between 0 and 1. A high R² indicates a close fit but says nothing about whether the linear form is correct — residual plots are needed to check for curvature or unequal spread.
Assumptions and cautions
- Linearity: the true relationship is approximately linear.
- Independent errors with constant variance.
- No extreme leverage points distorting the line.
- Extrapolation beyond the observed range of x is unreliable.
Regression quantifies association; a coefficient is not a causal effect unless the design or a causal model supports that reading.