Validation Metrics
Validation metrics quantify agreement between a model's predicted distribution and experimental data, going beyond a single error number.
Comparing distributions, not points
A predictive model outputs a distribution, and the data have their own uncertainty, so validation must compare distributions rather than points. A validation metric measures the discrepancy between the model's predicted distribution and the observed data distribution, providing a quantitative, reportable measure of predictive credibility.
The area validation metric
The area metric computes the area between the model's predicted cumulative distribution function and the empirical CDF of the data. It is expressed in the units of the quantity, is interpretable as an average distributional disagreement, and does not reward a model for being confidently wrong, unlike some likelihood-based scores.
Other metrics
- Coverage: fraction of observations falling inside predicted intervals vs the nominal level
- Continuous ranked probability score (CRPS): a proper score for the full distribution
- Reliability and sharpness: calibration of intervals traded against their width
- Statistical tests (Kolmogorov-Smirnov) for distributional equality
Calibration vs sharpness
A good predictive distribution is both calibrated (its stated probabilities match observed frequencies) and sharp (as concentrated as calibration allows). A metric that rewards only one can be gamed; proper scoring rules such as CRPS reward both simultaneously, which is why they are preferred for ranking models.
Reporting discipline
Validation metrics should be computed on data not used in calibration and reported alongside the domain of applicability, since a metric value is meaningful only within the conditions tested. Extrapolation beyond the validated domain carries model-form uncertainty that no in-domain metric can certify. Honest validation states both the metric and where it does and does not apply.