Probability Calibration
A calibrated model outputs probabilities that match observed frequencies, so a 0.7 prediction is right about 70 percent of the time.
Confidence you can trust
A classifier can be accurate yet still report meaningless probabilities. Calibration is the property that predicted probabilities match empirical frequencies: among all cases where the model says 0.7, roughly 70 percent should be positive. Calibration matters whenever a downstream decision uses the probability itself, for thresholding, ranking, cost-sensitive choices, or combining models.
Measuring calibration
A reliability diagram bins predictions by confidence and plots the observed frequency in each bin against the mean predicted probability; a perfectly calibrated model lies on the diagonal. The expected calibration error summarizes the average gap across bins. Proper scoring rules like the Brier score and log loss reward both accuracy and calibration together.
Post-hoc calibration methods
- Platt scaling: fit a logistic regression mapping raw scores to probabilities; good for small validation sets
- Isotonic regression: fit a non-decreasing step function; more flexible but needs more data
- Temperature scaling: a single-parameter softmax rescaling for neural nets, described in its own page
Calibration is always fit on a held-out set distinct from training data, otherwise it merely memorizes training scores.
Which models need it
Models differ in their tendencies. Modern deep networks are typically overconfident; naive Bayes pushes probabilities toward the extremes; boosted trees are often miscalibrated as well. Well-regularized logistic regression is usually close to calibrated by construction. Calibration does not change the ranking of predictions, so it cannot fix a poor classifier, only correct the meaning of its confidence values.
Under distribution shift, calibration degrades even if accuracy holds, so it should be monitored, not set once.