The Softmax Cross-Entropy Gradient

Turn logits into class probabilities and derive why the combined gradient collapses to predicted-minus-target.

Softmax

Softmax converts a vector of logits z into a probability distribution: p_i = exp(z_i) / sum_j exp(z_j). It is the multi-class generalization of the sigmoid. Subtract max(z) before exponentiating to avoid overflow - a standard numerical safeguard.

Cross-entropy loss

For a one-hot target y, the loss is L = -sum_i y_i log(p_i) = -log(p_correct). It punishes assigning low probability to the true class and is minimized when p matches y.

The gradient

Composing softmax with cross-entropy, the Jacobian mess cancels and the gradient with respect to the logits is simply dL/dz = p - y. Each logit is pushed by how far its predicted probability is from the target - the same elegant form as sigmoid cross-entropy and linear-regression residuals.

import numpy as np
def softmax(z):
    e=np.exp(z-z.max()); return e/e.sum()
z=np.array([2.0,1.0,0.1]); y=np.array([1,0,0.])
p=softmax(z)
loss=-np.log(p[y.argmax()])
grad=p-y
print('probs:',np.round(p,3))
print('loss:',round(loss,3),'grad:',np.round(grad,3))

Why it matters

This clean gradient is why softmax-cross-entropy is the default output layer for classification: backpropagation starts from p - y with no awkward terms, gradients stay well-scaled, and training is stable. Understanding this cancellation demystifies the last layer of nearly every classifier network you will build or read about.