Residual Connections

A residual connection adds a layer's input to its output, letting networks learn refinements to the identity and making very deep stacks trainable.

Learning a residual

A residual connection, the defining idea of ResNet, reroutes a layer's input directly to its output by addition: the block computes y = x + F(x), where F is the layer's transformation. Instead of learning the full desired mapping, the block only needs to learn the difference, or residual, from the identity. If the best thing a block can do is pass its input through unchanged, it can simply learn F(x) near zero, which is far easier than learning an identity mapping from scratch through nonlinear layers.

Solving the degradation problem

Before residual connections, making networks deeper past a point made them worse, not just harder to train but higher in training error, a degradation not explained by overfitting. Residual connections resolved this. Because the identity path is always available, adding layers cannot hurt: extra blocks can learn to do nothing. This let networks grow to hundreds and even a thousand layers and keep improving, a leap that reshaped deep learning.

The gradient highway

The addition also helps gradients. During backpropagation the derivative flows through the identity path with a factor of one, so it reaches early layers undiminished even when the transformation branch attenuates it. This shortcut prevents the vanishing gradients that otherwise plague deep stacks and is why residual connections appear in nearly every modern deep architecture.

def residual_block(x):
    return x + F(x)      # F: conv-norm-relu-conv, or attention/MLP

• Enables training of networks hundreds of layers deep
• Gives gradients a direct, undiminished path to early layers
• Each block learns a refinement rather than a full transformation
• Foundational to transformers, ResNets, and most modern designs

Reach and variants

Residual connections are everywhere: every transformer block wraps its attention and MLP sublayers in them, paired with normalization. Their continuous-time limit motivates neural ODEs, which read the update as a differential equation step. A denser alternative that concatenates rather than adds features is described in dense connectivity.