Heteroscedastic Noise Surrogates
When observation noise varies across input space, the surrogate must model an input-dependent noise level rather than a single constant.
Constant vs varying noise
Homoscedastic models assume the same noise variance everywhere. Real data often violate this: a simulation may be noisier in stiff regimes, or an experiment may have larger scatter at extreme operating points. Heteroscedastic surrogates model the noise variance as a function of the input, sigma^2(x), alongside the mean.
Why it matters
Ignoring heteroscedasticity produces intervals that are too wide where data are clean and too narrow where data are noisy, so coverage is wrong locally even if it looks acceptable on average. Correct local uncertainty is essential when decisions depend on the reliability of specific operating points.
Approaches
- Heteroscedastic Gaussian processes with a second GP modeling log-variance
- Neural networks with two output heads: mean and variance, trained by negative log-likelihood
- Quantile regression, which captures varying spread without a variance model
- Input-dependent nugget terms in kriging
Training subtleties
Jointly learning mean and variance can be unstable: the model may explain residuals as noise instead of signal, inflating variance and underfitting the mean. Warm-starting the mean, using a variance floor, or alternating optimization improves stability. The negative log-likelihood objective naturally balances the two if regularized carefully.
Verification
Validate that predicted intervals achieve their nominal coverage in different input regions, not just overall. A reliability diagram stratified by input region reveals whether the noise model is capturing real structure. In simulation surrogates, comparing predicted noise against repeated stochastic runs at fixed inputs is a direct check.