Bayesian Deep Learning via Subnetwork Inference
  The Bayesian paradigm has the potential to solve core issues of deep neural
networks such as poor calibration and data inefficiency. Alas, scaling Bayesian
inference to large weight spaces often requires restrictive approximations. In
this work, we show that it suffices to perform inference over a small subset of
model weights in order to obtain accurate predictive posteriors. The other
weights are kept as point estimates. This subnetwork inference framework
enables us to use expressive, otherwise intractable, posterior approximations
over such subsets. In particular, we implement subnetwork linearized Laplace as
a simple, scalable Bayesian deep learning method: We first obtain a MAP
estimate of all weights and then infer a full-covariance Gaussian posterior
over a subnetwork using the linearized Laplace approximation. We propose a
subnetwork selection strategy that aims to maximally preserve the model's
predictive uncertainty. Empirically, our approach compares favorably to
ensembles and less expressive posterior approximations over full networks. Our
proposed subnetwork (linearized) Laplace method is implemented within the
laplace PyTorch library at https://github.com/AlexImmer/Laplace.
