Leave-One-Out CV

Leave-one-out cross-validation estimates a surrogate's predictive error by testing on each training point held out in turn.

The idea

Leave-one-out cross-validation (LOO) removes one training point, refits the surrogate on the rest, predicts the held-out point, and repeats for every point. The collected prediction errors estimate how the surrogate will perform on unseen data drawn from the same region, using only the data already gathered.

Why it suits surrogates

Kronos motion — operating point

Expensive-model surrogates are trained on small datasets where reserving a separate test set is costly. LOO reuses every point for both training and testing, giving a nearly unbiased error estimate without extra model runs. For Kriging and other kernel models, a closed-form shortcut computes all LOO residuals from a single fit, avoiding N refits.

python

# Gaussian process LOO residuals in closed form
# K = covariance matrix (with nugget), y = targets
import numpy as np
Kinv = np.linalg.inv(K)
alpha = Kinv @ y
loo_resid = alpha / np.diag(Kinv)   # e_i = (y_i - mu_-i)
loo_var = 1.0 / np.diag(Kinv)

What LOO reveals

Predictive accuracy: root-mean-square LOO error
Calibration: standardized LOO residuals should have unit variance if the predictive variance is right
Local weakness: large residuals flag regions needing more points

Calibration check

Dividing each LOO residual by its predicted standard deviation gives standardized residuals. If the surrogate's uncertainty is honest, these should be roughly standard normal. Systematic over- or under-dispersion signals a miscalibrated emulator whose variance needs correction before use.

Cautions

LOO estimates error within the sampled region only; it says nothing about extrapolation beyond the data. It can be optimistic when training points are clustered, and it does not detect missing inputs. Use LOO to tune and calibrate the surrogate, but verify decision-critical predictions against fresh true-model runs, especially near boundaries.