Random Forests
A random forest averages many decorrelated decision trees, cutting variance for strong, robust tabular predictions.
Many trees, decorrelated
A random forest is an ensemble of decision trees whose predictions are averaged (regression) or voted (classification). Averaging many high-variance trees cancels their individual errors, so the forest generalizes far better than any single tree. Two sources of randomness keep the trees from agreeing too much.
Bagging plus feature sampling
- Each tree trains on a bootstrap sample: n points drawn with replacement from the training set.
- At each split, only a random subset of features is considered.
- These two devices decorrelate the trees so their errors are more independent.
This is bagging with an added feature-subsampling twist. Because errors that are independent average away, decorrelation is what makes the forest strong.
from sklearn.ensemble import RandomForestClassifier
rf = RandomForestClassifier(n_estimators=500, max_features='sqrt')
rf.fit(X_train, y_train)Free diagnostics
The points left out of each bootstrap sample (out-of-bag) provide an unbiased error estimate without a separate validation set. Feature importances aggregate impurity reductions across all trees, though permutation importance is more reliable when features are correlated.
Practical notes
Forests are robust, need little tuning, and rarely overfit as you add trees (more trees only stabilize the average). They cost more memory and prediction time than one tree and are less interpretable. On many tabular problems gradient boosting edges out random forests in accuracy, but forests remain a dependable, low-fuss baseline that is easy to parallelize.