Vanishing and Exploding Gradients

In deep and recurrent networks, gradients can shrink to nothing or blow up as they propagate, stalling or destabilizing training.

The core mechanism

Backpropagation computes a gradient as a product of many factors, one per layer or time step, via the chain rule. If those factors are consistently less than one, their product decays exponentially with depth and the gradient vanishes; if consistently greater than one, the product grows exponentially and the gradient explodes. Either way, deep or long networks become hard to train because early layers receive useless gradient signal.

Symptoms

Kronos motion — neural operator

Vanishing: early layers barely update, loss plateaus, long-range dependencies are never learned.
Exploding: loss spikes to NaN, weights swing wildly, training diverges.

Causes

Saturating activations like sigmoid and tanh have small derivatives that compound toward zero. Poor weight initialization sets the per-layer scaling too small or too large. In recurrent networks, repeatedly multiplying by the same recurrent matrix amplifies whichever tendency its eigenvalues have. Very deep plain networks suffer purely from the length of the multiplication chain.

Remedies for vanishing

Use non-saturating activations such as ReLU whose positive-side derivative is one. Add residual connections, which give gradients a direct additive path that bypasses the multiplicative chain, the key to training very deep networks. Apply normalization layers to keep activations well scaled, and use variance-preserving initialization such as He or Xavier. Gated recurrent cells (LSTM, GRU) create protected pathways for gradient flow across time.

Remedies for exploding

The standard fix is gradient clipping: if the gradient's norm exceeds a threshold, rescale it down before the update. This caps step size without changing direction. Careful initialization, normalization, and a modest learning rate also reduce the risk. Exploding gradients are usually easier to detect and fix than vanishing ones.

python

# He initialization preserves variance through ReLU layers
import numpy as np
W = np.random.randn(n_out, n_in) * np.sqrt(2.0 / n_in)

Why it shaped modern architectures

Much of deep learning's progress is a story of taming these gradients. ReLU, batch and layer normalization, residual connections, and gated recurrence all exist in large part to keep gradients flowing at a usable scale through many layers, which is what made networks with dozens or hundreds of layers trainable.

Chain-rule products decay or grow exponentially.
Saturating activations and bad init are common causes.
Residuals, normalization, and ReLU fight vanishing.
Clipping fights exploding.