Gaussian Process Kernels
The kernel of a Gaussian process encodes assumptions about smoothness, scale, and structure, and choosing it is the main modeling decision.
The kernel is the model
In a Gaussian process almost all of the modeling assumptions live in the kernel, the function that sets the covariance between the function's values at two inputs. It determines how smooth the sampled functions are, how quickly correlations decay with distance, and what periodic or structured behavior is expected. Choosing a kernel is choosing what kind of function you believe generated the data.
Common kernels
- Squared exponential (RBF): infinitely smooth functions, controlled by a length scale
- Matern: a tunable smoothness parameter gives rougher, more realistic samples
- Periodic: functions that repeat with a fixed period
- Linear: recovers Bayesian linear regression as a special case
Hyperparameters and their meaning
Each kernel carries hyperparameters with clear interpretations. The length scale sets how far apart inputs must be before their outputs decorrelate; a short length scale gives wiggly functions, a long one gives flat trends. The signal variance sets the vertical amplitude of variation. A separate noise variance accounts for measurement error in the observations.
Building compound kernels
Kernels can be added and multiplied to compose structure. A sum of a smooth trend kernel and a periodic kernel models a rising signal with seasonal oscillation. A product combines behaviors, such as a periodic pattern whose amplitude decays over distance. This algebra lets a modeler encode rich prior knowledge without leaving the Gaussian process framework.
Choosing well
The right kernel encodes real prior knowledge; the wrong one forces the data to fight the assumptions. When knowledge is thin, the Matern family is a safe default because its smoothness parameter can itself be selected. Hyperparameters are then learned by maximizing the marginal likelihood, letting the data refine the human choice of kernel form.