Batch Normalization

Batch normalization standardizes layer activations across a mini-batch, stabilizing and accelerating the training of deep networks.

The idea

Batch normalization normalizes the inputs to a layer so that, across the current mini-batch, each feature has roughly zero mean and unit variance. It then applies a learnable scale and shift so the network can undo the normalization if that is best. Introduced in 2015, it made deep networks train faster, tolerate higher learning rates, and depend less on careful initialization.

The computation

Kronos motion — neural operator

For each feature, the layer computes the mean and variance over the batch, subtracts the mean, divides by the standard deviation (with a small epsilon for stability), then scales by a learned parameter gamma and adds a learned bias beta. Gamma and beta let the layer represent any mean and variance, so normalization restricts nothing the network could otherwise learn; it only reparameterizes the problem into a better-conditioned form.

python

import numpy as np
def batch_norm(x, gamma, beta, eps=1e-5):
    mu = x.mean(axis=0)
    var = x.var(axis=0)
    xhat = (x - mu) / np.sqrt(var + eps)
    return gamma * xhat + beta

Why it helps

Batch norm was originally motivated as reducing internal covariate shift, the change in each layer's input distribution as earlier layers update. Later analysis argued its main benefit is smoothing the loss landscape, making gradients more predictable so larger steps are safe. Whatever the precise reason, the practical effect is robust: faster convergence and reduced sensitivity to hyperparameters.

Training versus inference

During training the statistics come from the current batch. At inference there may be no batch, so the layer uses running averages of the mean and variance accumulated during training. This train-test difference must be handled carefully; a mismatch is a common source of bugs. It also means batch norm behaves oddly with very small batches, where per-batch statistics are noisy.

Limitations and alternatives

Because it depends on batch statistics, batch norm degrades with tiny batches and does not fit sequence models cleanly, where sequence length and padding complicate the batch dimension. Layer normalization, which normalizes across features within a single example rather than across the batch, avoids these issues and is the standard choice in transformers. Group and instance normalization serve other niches. Batch norm remains dominant in convolutional vision networks.

Standardizes activations across the mini-batch.
Learnable scale and shift preserve expressiveness.
Smooths the loss landscape and speeds training.
Uses running statistics at inference; weak with tiny batches.