Highway Networks

Highway networks use learned gates to control how much of each layer's input passes through unchanged versus transformed, an early route to training very deep networks.

Gated depth

Highway networks were among the first architectures to train networks of many dozens of layers. They borrow the gating idea from LSTMs and apply it to depth. Each layer computes both a transformed version of its input and a learned transform gate that decides, per element, how much of the transformed output to use versus how much of the original input to carry straight through. When the gate is closed, the input passes unchanged, giving information a clear path through many layers.

The gating equations

A highway layer outputs y = H(x)*T(x) + x*(1 - T(x)), where H is the transformation, T is the transform gate produced by a sigmoid, and the carry portion is one minus the gate. If T is near zero the layer acts as an identity; if T is near one it acts as a normal transformation. The gate is learned, so the network decides for itself which layers should transform and which should mostly pass information along.

T = torch.sigmoid(W_t @ x + b_t)      # transform gate
H = relu(W_h @ x + b_h)               # transformation
y = H * T + x * (1 - T)               # carry the rest

• Learned gates modulate transformation versus carry, per element
• Enables training of networks tens of layers deep
• Bias the transform gate negative at init so layers start near identity
• A direct precursor to the residual connection

Relation to residual networks

Highway networks and residual connections both create shortcut paths that let information and gradients skip transformations, and both made great depth trainable. The difference is that a highway layer gates the shortcut with a learned, input-dependent weight, while a residual connection uses a fixed, ungated identity add. The residual's simpler form proved easier to optimize and scale, so it largely superseded gating for depth, but highway networks established the core insight that a learnable pass-through path is what unlocks very deep training.