Sampling-free Variational Inference for Neural Networks with
  Multiplicative Activation Noise
  To adopt neural networks in safety critical domains, knowing whether we can
trust their predictions is crucial. Bayesian neural networks (BNNs) provide
uncertainty estimates by averaging predictions with respect to the posterior
weight distribution. Variational inference methods for BNNs approximate the
intractable weight posterior with a tractable distribution, yet mostly rely on
sampling from the variational distribution during training and inference.
Recent sampling-free approaches offer an alternative, but incur a significant
parameter overhead. We here propose a more efficient parameterization of the
posterior approximation for sampling-free variational inference that relies on
the distribution induced by multiplicative Gaussian activation noise. This
allows us to combine parameter efficiency with the benefits of sampling-free
variational inference. Our approach yields competitive results for standard
regression problems and scales well to large-scale image classification tasks
including ImageNet.
