Bayesian Neural Networks

Bayesian neural networks place distributions over weights, turning point predictions into predictive distributions with quantified uncertainty.

From weights to distributions

A standard neural network learns a single set of weights. A Bayesian neural network treats the weights as random variables with a prior and infers a posterior p(weights | data). Predictions integrate over this posterior, so regions unsupported by data produce wide predictive distributions rather than confident but wrong answers.

The intractable posterior

Kronos motion — operating point

The weight posterior is high-dimensional and non-Gaussian, so exact inference is impossible. Practical BNNs use approximations: variational inference (see Variational Inference for BNNs), the Laplace approximation (see Laplace Approximation), or sampling methods such as stochastic-gradient MCMC.

Predictive integration

The predictive distribution is p(y|x, data) = integral p(y|x, w) p(w|data) dw, approximated by averaging network outputs over sampled weights. The mean gives the point prediction; the spread separates into aleatoric noise (from the output likelihood) and epistemic uncertainty (from posterior weight spread).

Priors matter

Weight priors act as regularization and shape extrapolation behavior
Overly tight priors underfit; overly broad priors give diffuse, uninformative predictions
Function-space priors are more interpretable than weight-space priors but harder to specify

Practical status

Exact BNNs are costly and sensitive to approximation quality, so simpler surrogates for uncertainty, such as deep ensembles and Monte Carlo dropout, are often preferred in practice and can be viewed as approximate Bayesian inference. Whichever method is used, calibration must be verified empirically; a model being Bayesian does not guarantee its uncertainties are well calibrated on real data.