Variational Inference for BNNs

Variational inference approximates an intractable weight posterior with a simpler distribution by maximizing a lower bound on the evidence.

The variational idea

Rather than sample the true posterior p(w|data), variational inference posits a tractable family q(w; params), often independent Gaussians per weight, and finds the member closest to the true posterior in KL divergence. This turns inference into optimization, which scales to large networks with stochastic gradients.

The ELBO

Kronos motion — uncertainty

Minimizing KL(q || posterior) is equivalent to maximizing the evidence lower bound: ELBO = E_q[log p(data|w)] - KL(q(w) || prior(w)). The first term rewards fitting the data; the second penalizes departing from the prior. The gap between the ELBO and the true log evidence is exactly the KL divergence we cannot compute directly.

Bayes by backprop

The reparameterization trick writes a sampled weight as w = mu + sigma * epsilon with epsilon a standard normal, making the sampling differentiable so gradients flow to mu and sigma. This 'Bayes by backprop' procedure trains the variational parameters with ordinary stochastic gradient descent.

Strengths and weaknesses

Scales to modern architectures and large datasets
Mean-field (independent) q ignores weight correlations, often underestimating uncertainty
The KL term can dominate or vanish, requiring careful weighting and initialization

Practice

Mean-field variational BNNs are fast but tend to be overconfident because the factorized approximation cannot capture posterior correlations. Structured variational families or combining VI with ensembling improve calibration. As with any approximate inference, validate predictive coverage on held-out data before trusting the uncertainties for decisions.