Normalization Variants Compared
Batch, layer, instance, and group normalization differ only in which axes they compute statistics over, and that choice determines where each works best.
One idea, several axes
Normalization layers all do the same thing: subtract a mean, divide by a standard deviation, then apply a learned scale and shift. What separates the popular variants is only the set of axes over which the mean and variance are computed. Understanding a normalization method reduces to answering: which dimensions are pooled to estimate the statistics, and which are kept separate?
The four common choices
- Batch norm: statistics over the batch and spatial axes, per channel; depends on batch size
- Layer norm: statistics over the feature or channel axis, per sample; batch-independent
- Instance norm: statistics per sample and per channel over spatial positions; used in style transfer
- Group norm: statistics per sample over groups of channels; a middle ground
Batch norm's use of batch statistics couples samples together and makes its behavior differ between training and inference, which is powerful with large batches but fragile with small ones. The other three compute statistics within a single sample, so they behave identically regardless of batch size.
Pre-norm versus post-norm
Where the normalization sits relative to the residual connection also matters. Post-norm, the original transformer placement, applies normalization after the residual add. Pre-norm applies it to the sublayer input before attention or the MLP. Pre-norm keeps a clean residual path and trains deep stacks more stably, which is why it dominates modern large models, often paired with RMSNorm.
Choosing a variant
The choice follows the data and the training regime. Convolutional vision with large batches historically favored batch norm; small-batch vision favors group norm; sequence models favor layer norm or RMSNorm because sequence length and batch vary and batch statistics would be unreliable. All variants serve the same purpose within a residual network: keeping activation scales controlled so gradients stay well behaved through great depth.