Logistic Regression

Logistic regression models the probability of a binary outcome as a smooth function of predictors.

Why not linear regression

For a yes/no response, a straight line can predict probabilities below 0 or above 1, which is meaningless. Logistic regression instead models the probability through the logistic (sigmoid) function, keeping predictions in [0, 1].

The model

It sets P(y = 1 | x) = σ(β₀ + β₁x), where σ(z) = 1/(1 + e^{−z}). Equivalently, the log-odds are linear: log(p/(1 − p)) = β₀ + β₁x. Each coefficient is the change in log-odds per unit change in a predictor, and its exponential is an odds ratio.

Fitting

There is no closed-form solution. The coefficients are found by maximizing the likelihood — equivalently, minimizing cross-entropy loss — using iterative methods such as Newton-Raphson or gradient descent. The log-likelihood is concave, so the optimum is unique when the classes are not perfectly separable.

import math
def sigmoid(z):
    return 1/(1+math.exp(-z))
# log-odds 0.4 -> probability
print(round(sigmoid(0.4),4))  # 0.5987

Interpretation and decision

The model outputs a calibrated probability; a decision threshold (often 0.5, but chosen to balance error costs) turns it into a class label. Because it is a Bernoulli model with a linear log-odds, logistic regression is the simplest and most interpretable probabilistic classifier.

Cautions

Perfectly separable data drive coefficients to infinity, requiring regularization. And as with linear regression, coefficients describe partial associations within the model, not causal effects.

Logistic regression extends naturally to more than two classes. The multinomial (softmax) version models several categories at once, and it is exactly the output layer of many neural network classifiers. Understanding the two-class case makes that generalization straightforward: softmax replaces the single sigmoid with a normalized set of exponentials, one per class.