Vanishing Gradient
The failure of gradients to propagate through deep or recurrent networks, stalling learning in early layers.
Definition
The vanishing gradient problem occurs when the gradients used to update early layers become extremely small after passing back through many layers, so those layers learn very slowly or not at all. The mirror problem, exploding gradients, makes updates blow up.
The complementary failure, exploding gradients, is usually handled by gradient clipping, which caps the update magnitude. Together, careful initialization, normalization, and skip connections keep the scale of both activations and gradients roughly constant through depth.
The residual connection, which adds a layer's input to its output, was a decisive fix: it gives gradients a direct path backward and lets networks reach hundreds of layers deep. This single architectural idea, combined with normalization, turned very deep learning from a fragile experiment into routine practice, and it appears in nearly every modern deep architecture including the transformer.
Causes and fixes
- Saturating activations like sigmoid shrink gradients; ReLU helps.
- Careful weight initialization keeps signal scale stable.
- Batch normalization stabilizes activation distributions.
- Residual (skip) connections give gradients a direct path.
- Gated units like the LSTM address it in sequences.
Why it matters
This problem long limited the depth of trainable networks. The techniques that overcame it are precisely what made modern deep learning possible.
Fusion connection
Kronos deep surrogates use residual connections and normalization so that gradient flow stays healthy, allowing enough depth to capture nonlinear plasma behavior.