Radial Bayesian Neural Networks: Beyond Discrete Support In Large-Scale
  Bayesian Deep Learning
  We propose Radial Bayesian Neural Networks (BNNs): a variational approximate
posterior for BNNs which scales well to large models while maintaining a
distribution over weight-space with full support. Other scalable Bayesian deep
learning methods, like MC dropout or deep ensembles, have discrete support-they
assign zero probability to almost all of the weight-space. Unlike these
discrete support methods, Radial BNNs' full support makes them suitable for use
as a prior for sequential inference. In addition, they solve the conceptual
challenges with the a priori implausibility of weight distributions with
discrete support. The Radial BNN is motivated by avoiding a sampling problem in
'mean-field' variational inference (MFVI) caused by the so-called 'soap-bubble'
pathology of multivariate Gaussians. We show that, unlike MFVI, Radial BNNs are
robust to hyperparameters and can be efficiently applied to a challenging
real-world medical application without needing ad-hoc tweaks and intensive
tuning. In fact, in this setting Radial BNNs out-perform discrete-support
methods like MC dropout. Lastly, by using Radial BNNs as a theoretically
principled, robust alternative to MFVI we make significant strides in a
Bayesian continual learning evaluation.
