Validating a Surrogate Model
A surrogate is only trustworthy after validation against data it never saw, on metrics that expose both accuracy and honest uncertainty.
Fit is not accuracy
A surrogate can match its training data perfectly and still fail everywhere else - the classic overfitting trap. Validation asks the only question that matters: how well does the surrogate predict outputs it was not trained on? This requires data held back from training, or fresh runs of the expensive model at new points.
Holdout and cross-validation
The simplest check reserves a test set never used in fitting and reports error on it. When data is scarce - the usual case with expensive models - k-fold cross-validation reuses the data by rotating which fold is held out, and leave-one-out cross-validation takes this to the limit. For Gaussian processes, leave-one-out error has an efficient closed form.
Accuracy metrics
- Root-mean-square error - overall magnitude of prediction error
- Mean absolute error - robust to outliers
- Coefficient of determination (R-squared) - fraction of variance explained
- Maximum error - the worst-case point, critical for safety limits
Validating the uncertainty
For a probabilistic surrogate, accuracy of the mean is not enough - the error bars must be honest too. Coverage tests check that stated intervals contain the truth at the claimed rate. Proper scoring rules such as the continuous ranked probability score and the negative log predictive density reward predictions that are both accurate and well-calibrated.
Where to test
Average error can hide local failure. Validation should probe the regions that matter - near operating limits, in nonlinear zones, and at the edges of the training domain where extrapolation risk is highest. Out-of-distribution tests reveal how badly the surrogate degrades outside its training envelope.
Kronos discipline
No Kronos surrogate informs a design decision until it is validated against held-out full-fidelity runs of the machines, on both accuracy and calibration, with attention to behavior near operating boundaries. A surrogate that fails validation is refit with more data or replaced; it is never used on faith.