Variational Laplace for Bayesian neural networks
  We develop variational Laplace for Bayesian neural networks (BNNs) which
exploits a local approximation of the curvature of the likelihood to estimate
the ELBO without the need for stochastic sampling of the neural-network
weights. The Variational Laplace objective is simple to evaluate, as it is (in
essence) the log-likelihood, plus weight-decay, plus a squared-gradient
regularizer. Variational Laplace gave better test performance and expected
calibration errors than maximum a-posteriori inference and standard
sampling-based variational inference, despite using the same variational
approximate posterior. Finally, we emphasise care needed in benchmarking
standard VI as there is a risk of stopping before the variance parameters have
converged. We show that early-stopping can be avoided by increasing the
learning rate for the variance parameters.
