Tuning Gaussian-Process Hyperparameters

GP hyperparameters - length scales, variance, and noise - are learned by maximizing the marginal likelihood, which balances fit against simplicity.

What needs tuning

A Gaussian process has a handful of hyperparameters that control its behavior: one or more length scales, the signal variance, and the observation noise variance. These are not set by hand; they are learned from the training data. The quality of the surrogate depends heavily on getting them right.

The marginal likelihood

Kronos motion — which application

The standard objective is the log marginal likelihood - the probability of the observed data under the GP, with the function values integrated out. Maximizing it over the hyperparameters is the training step. Its expression has two competing terms: a data-fit term that rewards matching the observations and a complexity-penalty term (a log-determinant) that discourages overly flexible models.

Built-in Occam's razor

That trade-off is automatic regularization. A model that is too smooth fits the data poorly; a model that is too flexible is penalized by the complexity term. The marginal likelihood balances them without a separate validation set, which is why GPs resist overfitting on small datasets - a distinctive advantage over methods that need external regularization tuning.

Optimization mechanics

Gradient-based optimizers use analytic derivatives of the log marginal likelihood
The objective is often non-convex, so multiple restarts guard against local optima
Priors on hyperparameters (MAP estimation) stabilize small-data fits
Fully Bayesian treatment integrates over hyperparameters via sampling for extra robustness

Pitfalls

The likelihood surface can have multiple modes - for example, one explaining the data as signal, another as noise. Poor initialization can land in a bad mode. Very small noise can make the kernel matrix ill-conditioned, so a jitter term is added for numerical stability. Length scales far outside the data range signal an identifiability problem.

In practice

For Kronos GP surrogates of the machines, hyperparameters are fit by marginal-likelihood maximization with multiple restarts and mild priors, then cross-checked by leave-one-out error. This yields length scales that both fit the data and reveal which physics inputs the response depends on most.