Sparse Gaussian Processes
Sparse Gaussian processes summarize a large dataset with a small set of inducing points, cutting the cubic cost of exact inference.
The scaling wall
Exact Gaussian process inference inverts a matrix whose size is the number of data points, costing time that grows as the cube of that number. Past a few thousand observations this is prohibitive. Sparse Gaussian processes recover most of the accuracy at a fraction of the cost by compressing the data into a small set of representative points.
Inducing points
The central device is a set of inducing points, a handful of pseudo-inputs, far fewer than the data, that act as a summary of the full training set. The GP is approximated so that all information flows through these inducing points. Their locations and associated values are treated as parameters and optimized, so the summary adapts to the data rather than being placed arbitrarily.
Variational formulation
The most widely used approach casts the approximation as variational inference. It defines an approximate posterior anchored at the inducing points and minimizes its divergence from the true posterior. This gives a principled objective, a lower bound on the marginal likelihood, that jointly tunes the inducing locations, the kernel hyperparameters, and the noise level.
What it buys
- Cost scales with the number of inducing points, not the dataset size
- Enables mini-batch training on very large datasets
- Retains calibrated predictive uncertainty, unlike many fast approximations
- Number of inducing points trades accuracy against speed
When to use it
Sparse GPs are the practical choice when the appeal of a Gaussian process, honest uncertainty and few assumptions, is needed at a data scale that exact inference cannot reach. The main tuning decision is how many inducing points to allow: too few underfit and overstate uncertainty, too many erode the speed advantage.