Calibration of Predictive Uncertainty
A model is calibrated when its stated probabilities match observed frequencies; miscalibration makes uncertainty estimates untrustworthy.
What calibration means
A predictive model is calibrated if events it assigns 90 percent probability actually happen about 90 percent of the time, and its 90 percent intervals contain the truth about 90 percent of the time. Calibration is a property of the uncertainty, separate from the accuracy of the point prediction: a model can be accurate but overconfident, or inaccurate but well calibrated.
Diagnosing calibration
- Reliability diagram: predicted probability vs observed frequency, ideally on the diagonal
- Coverage plot: nominal interval level vs empirical coverage across the data
- Probability integral transform (PIT) histogram: uniform if calibrated for regression
- Expected calibration error: a scalar summary of the gap
Sharpness
Calibration alone is not enough. A model that always predicts the climatological distribution is perfectly calibrated but useless. The aim is calibration plus sharpness: intervals as narrow as possible while still achieving their nominal coverage. Proper scoring rules balance the two automatically.
Recalibration
A miscalibrated model can often be corrected post-hoc without retraining: temperature scaling for classifiers, isotonic or Platt scaling for probabilities, and conformal methods for regression intervals. These fit a small correction on held-out data to align stated and observed frequencies.
Why it matters for decisions
Downstream reliability analysis, optimization under uncertainty, and safety gates all consume the model's stated probabilities. If those probabilities are systematically wrong, every decision built on them is biased. Verifying calibration on independent data is a non-negotiable step before a surrogate's uncertainty is used to make engineering claims.