Conformal Prediction
Conformal prediction wraps any model in prediction intervals with a guaranteed coverage rate, without assuming a data distribution.
Guarantees without distribution assumptions
Most uncertainty methods rely on modeling assumptions that may be wrong. Conformal prediction gives a distribution-free guarantee: for a chosen confidence level, its prediction sets contain the true value at least that fraction of the time, provided only that the data are exchangeable. It works around any underlying model, treating it as a black box.
How it works
A held-out calibration set is scored by a nonconformity measure, typically the size of the model's error on each calibration point. The quantile of these scores at the desired confidence level becomes a threshold. For a new input, the prediction set is all outputs whose nonconformity falls below that threshold. For regression this yields an interval; for classification, a set of plausible labels.
A regression example
Fit any regressor, then on a calibration set compute the absolute residuals. Take the quantile of those residuals corresponding to the target coverage, say the ninetieth percentile for ninety percent coverage. The interval for a new prediction is the point estimate plus or minus that quantile. The coverage guarantee holds regardless of whether the regressor is any good.
Properties
- Coverage is guaranteed in finite samples, not just asymptotically
- Works with any model: linear, tree, or neural network
- Interval width reflects difficulty when adaptive scores are used
- Requires data exchangeability, which distribution shift can break
Why it complements other methods
Conformal prediction turns a model's uncertainty estimate into intervals with a stated coverage rate. It pairs well with Bayesian networks or ensembles: those provide a heuristic notion of confidence, and conformal calibration converts it into a guarantee. The main caveat is that the guarantee assumes the new data resemble the calibration data, so extrapolation still demands caution.