ResNet and Skip Connections

Residual connections let a layer learn a correction to its input rather than a full transformation, making very deep networks trainable.

The degradation problem

Before ResNet, simply stacking more layers eventually made networks worse, not better, even on training data. This was not overfitting but an optimization failure: very deep plain networks were hard to train because gradients degraded across many layers. The finding was counterintuitive, since a deeper network can in principle represent anything a shallower one can by making extra layers act as identity.

The residual block

Kronos motion — error correction

ResNet's fix is the skip connection. Instead of learning a target mapping H(x) directly, a block learns a residual F(x) = H(x) - x and outputs F(x) + x, adding the input back via a shortcut. If the best thing a block can do is nothing, it need only drive F(x) toward zero, which is easy. Learning a correction to the input is easier than learning the full mapping from scratch.

python

import torch.nn as nn
class ResBlock(nn.Module):
    def __init__(self, c):
        super().__init__()
        self.conv = nn.Sequential(
            nn.Conv2d(c, c, 3, padding=1), nn.BatchNorm2d(c), nn.ReLU(),
            nn.Conv2d(c, c, 3, padding=1), nn.BatchNorm2d(c))
    def forward(self, x):
        return nn.functional.relu(x + self.conv(x))  # skip connection

Gradient flow

The additive shortcut gives gradients a direct path backward that bypasses the block's weight layers. During backpropagation the gradient splits: part flows through the transformation and part flows straight through the identity path, which is always one. This prevents the exponential shrinking of vanishing gradients and is why residual networks train reliably at depths of 50, 100, or more than 1,000 layers.

Impact and reach

ResNet made very deep convolutional networks practical and won the 2015 ImageNet competition, becoming a standard vision backbone. The idea proved universal. Transformers wrap every attention and feedforward sublayer in a residual connection, forming a residual stream that each layer reads from and writes to. Nearly every modern deep architecture, in vision or language, relies on residual connections.

Related designs

DenseNet extends the idea by connecting each layer to all subsequent layers via concatenation rather than addition, encouraging feature reuse. Highway networks use learned gates on the shortcut. U-Net uses long skip connections across an encoder-decoder to carry spatial detail. All share the insight that giving information and gradients shortcut paths makes deep networks easier to optimize.

Deeper plain networks degraded due to optimization, not overfitting.
Blocks learn a residual correction plus an identity shortcut.
Shortcuts give gradients a direct backward path.
Residual streams underpin ResNets and transformers alike.