Bagging

Bagging trains models on bootstrap resamples and averages them, cutting variance without raising bias.

Bootstrap aggregating

Bagging, short for bootstrap aggregating, trains many copies of a model on different bootstrap samples of the data, then averages their predictions (or takes a majority vote). Each bootstrap sample draws n points with replacement, so it omits some points and duplicates others. Averaging the resulting models reduces variance while leaving bias roughly unchanged.

Why it reduces variance

If models were independent, averaging m of them would divide their variance by m. Bootstrap samples overlap, so the models are correlated and the reduction is smaller, but still substantial. Bagging therefore helps most with high-variance, low-bias learners, above all deep decision trees.

• Bootstrap: sample n points with replacement to build each training set.
• Train an independent model on each sample, in parallel.
• Aggregate by averaging (regression) or voting (classification).
• About 37% of points are left out of each sample: the out-of-bag set for free validation.

from sklearn.ensemble import BaggingClassifier
from sklearn.tree import DecisionTreeClassifier
bag = BaggingClassifier(DecisionTreeClassifier(),
                        n_estimators=200, oob_score=True)
bag.fit(X_train, y_train)

Out-of-bag estimate

Each point is out of bag for roughly a third of the models. Predicting it with only those models gives an out-of-bag error estimate that approximates cross-validation without a separate holdout.

Bagging and random forests

Random forests are bagging applied to trees plus random feature selection at each split, which further decorrelates the trees. Bagging is the parallel, variance-reducing counterpart to boosting, which is sequential and reduces bias.