Dropout

Dropout randomly disables neurons during training, forcing the network to build redundant, robust features and reducing overfitting.

Random deactivation

Dropout is a regularization method that, during each training step, randomly sets a fraction of a layer's activations to zero. Which units are dropped changes every step, so the network never relies on any single unit being present. A common drop probability is 0.5 for fully connected layers and lower, often 0.1, for large modern models. At test time nothing is dropped and the full network is used.

Why it reduces overfitting

Because any unit may vanish, the network cannot depend on fragile co-adaptations where units only work in specific combinations. Each unit must contribute usefully on its own, spreading the representation across many features. This redundancy makes the model generalize better to unseen data. Dropout can also be viewed as training an ensemble of many thinned subnetworks that share weights, then averaging them at test time.

Inverted dropout and scaling

Dropping units reduces the expected sum of activations, so a correction is needed to keep the scale consistent between training and testing. Modern implementations use inverted dropout: during training, surviving activations are divided by the keep probability, so their expected total matches the full network. Test time then requires no change at all, which keeps inference simple and fast.

import numpy as np
def dropout(x, p=0.5, training=True):
    if not training:
        return x
    mask = (np.random.rand(*x.shape) > p) / (1 - p)  # inverted dropout
    return x * mask

Where to apply it

Dropout works best in wide fully connected layers, which have the most capacity to overfit. It is used more sparingly in convolutional layers, where spatial correlation weakens standard dropout and variants like spatial dropout, which drops whole feature maps, work better. In transformers, dropout is applied to attention weights and feedforward activations. It generally goes after the activation function within a block.

Interactions and current status

Dropout and batch normalization can interfere, since both alter activation statistics, so many convolutional networks rely on batch norm and use little or no dropout. In large models trained on abundant data, heavy dropout is less necessary because the data itself limits overfitting, though modest dropout still helps. It remains a simple, cheap, and widely used tool in the regularization toolkit.

• Randomly zeroes activations during training.
• Prevents fragile co-adaptation of units.
• Acts like averaging an ensemble of subnetworks.
• Inverted dropout keeps inference unchanged.