Deep Ensembles
Deep ensembles quantify uncertainty by training several networks independently and measuring how much their predictions disagree.
Disagreement as uncertainty
A deep ensemble trains several copies of a network from different random initializations, and possibly different data orderings, then combines their predictions. Where the members agree, the model is confident; where they disagree, uncertainty is high. This simple idea often gives better-calibrated uncertainty than more elaborate Bayesian schemes.
Why independent training works
Different initial weights lead the optimizer to different solutions that all fit the training data well but extrapolate differently. On inputs far from the data, these solutions diverge, so their spread grows, capturing epistemic uncertainty. Near the data they converge, so the spread is small. The ensemble thus reports low confidence exactly where it should.
Making predictions
For regression, the ensemble mean is the prediction and the variance across members estimates the uncertainty; each member can also output its own noise estimate to add aleatoric uncertainty. For classification, the averaged probabilities give both the prediction and a measure of disagreement. Combining the members is cheap once they are trained.
Practical notes
- Five to ten members usually capture most of the benefit
- Members train independently, so training parallelizes perfectly
- Prediction cost scales with the number of members
- Diversity can be increased by varying architecture or data subsets
Why they are popular
Deep ensembles need no change to the base model, no special loss, and no approximate inference; any network can be ensembled. They are a strong default for uncertainty in scientific machine learning, where the main drawbacks, the memory and compute of several models, are usually acceptable given the value of trustworthy error bars on a surrogate.