Optimization

Mini-Batch Gradient Descent

The middle ground between full-batch and single-sample updates: average gradients over a small batch for stable, hardware-friendly steps.

Between two extremes

Full-batch gradient descent uses all n samples per step (low variance, high cost). Pure stochastic gradient descent uses one sample (cheap, high variance). Mini-batch gradient descent averages the gradient over B samples, where B is typically 32 to 1024, capturing most of the variance reduction while keeping each step cheap.

Variance scaling

The variance of the averaged gradient falls roughly as 1/B for independent samples, so the gradient standard deviation shrinks as 1/sqrt(B). Doubling the batch cuts noise by about 30 percent but doubles per-step cost; there are diminishing returns to very large batches.

Why it dominates in practice

Vectorized hardware (GPUs, TPUs) processes a batch nearly as fast as a single sample, so mini-batches use compute efficiently.
Moderate noise still helps escape saddle points and flat regions.
Batch statistics enable techniques like batch normalization.

Batch size and learning rate

A common heuristic is the linear scaling rule: when you multiply the batch size by k, multiply the learning rate by k as well, often with a warmup period. Very large batches can generalize worse without careful tuning, a phenomenon linked to sharper minima.

Epochs and shuffling

One epoch is a full pass through the dataset, split into ceil(n/B) mini-batches. Reshuffling the data each epoch decorrelates successive batches and improves convergence. Sampling without replacement within an epoch is standard and slightly outperforms sampling with replacement.

python

import numpy as np
def minibatches(X, y, B):
    idx = np.random.permutation(len(X))
    for s in range(0, len(X), B):
        j = idx[s:s+B]
        yield X[j], y[j]

Mini-batching is the default regime for training the neural surrogates that accelerate parameter sweeps in physics and engineering simulation.