Bootstrap Resampling

The bootstrap estimates the uncertainty of a statistic by resampling the observed data with replacement.

The idea

When the sampling distribution of a statistic has no clean formula, the bootstrap approximates it computationally. Treat the observed sample as a stand-in for the population, draw many new samples of the same size with replacement, recompute the statistic on each, and study the spread of those values.

Why it works

The empirical distribution of the data is the best nonparametric estimate of the true distribution. Resampling from it mimics drawing fresh samples from the population. The variation across bootstrap replicates approximates the variation the statistic would show across real repeated samples.

import random, statistics
data = [4.1, 5.2, 3.9, 6.0, 4.8, 5.5, 4.0]
means = []
for _ in range(10000):
    resample = [random.choice(data) for _ in data]
    means.append(statistics.mean(resample))
lo, hi = sorted(means)[250], sorted(means)[9750]
print(round(lo,3), round(hi,3))  # ~95% percentile interval

Confidence intervals

The simplest bootstrap interval takes the 2.5th and 97.5th percentiles of the replicated statistics for a 95% interval. More refined methods (bias-corrected and accelerated) adjust for skew and bias in the statistic, and are preferred when the sampling distribution is asymmetric.

Strengths and limits

• Works for statistics with no analytic standard error, like a median or a correlation.
• Makes few distributional assumptions.
• Struggles with very small samples and with extremes such as the maximum.
• Requires the observed sample to represent the population reasonably well.

The bootstrap trades mathematical derivation for computation, which is an easy trade when repeated resampling is cheap.