Logistic Regression by Gradient Descent

Classify data with a sigmoid model trained on cross-entropy loss, deriving the clean gradient by hand.

The model

Logistic regression predicts a class probability p = sigma(w . x + b), where sigma is the logistic function. It is a linear model squashed to (0,1). We fit it by minimizing binary cross-entropy, the negative log-likelihood of the labels.

The gradient

A pleasant surprise: the gradient of cross-entropy through the sigmoid simplifies to (p - y) x, the same clean form as linear regression's residual times input. This is why logistic regression trains stably - the sigmoid nonlinearity and the loss are matched.

import numpy as np
rng=np.random.default_rng(2)
X=np.vstack([rng.normal(-1,1,(200,2)),rng.normal(1,1,(200,2))])
y=np.r_[np.zeros(200),np.ones(200)]
w=np.zeros(2); b=0.0; sig=lambda z:1/(1+np.exp(-z))
for it in range(3000):
    p=sig(X@w+b); g=p-y
    w-=0.1*(X.T@g)/len(y); b-=0.1*g.mean()
print('weights:',np.round(w,3),'acc:',round(((sig(X@w+b)>0.5)==y).mean(),3))

Reading the output

The decision boundary is the line w . x + b = 0; points are classified by which side they fall on. Unlike a hard threshold, logistic regression gives calibrated-ish probabilities, so you can rank predictions by confidence and choose an operating threshold to suit the cost of each error type.

Extensions

Add an L2 penalty to prevent weights blowing up on separable data, where the unregularized likelihood pushes them to infinity. For more than two classes, the softmax generalization (multinomial logistic regression) applies. Because the loss is convex, gradient descent finds the global optimum - a rare comfort in machine learning.