Sampling-Free Variational Inference of Bayesian Neural Networks by
  Variance Backpropagation
  We propose a new Bayesian Neural Net formulation that affords variational
inference for which the evidence lower bound is analytically tractable subject
to a tight approximation. We achieve this tractability by (i) decomposing ReLU
nonlinearities into the product of an identity and a Heaviside step function,
(ii) introducing a separate path that decomposes the neural net expectation
from its variance. We demonstrate formally that introducing separate latent
binary variables to the activations allows representing the neural network
likelihood as a chain of linear operations. Performing variational inference on
this construction enables a sampling-free computation of the evidence lower
bound which is a more effective approximation than the widely applied Monte
Carlo sampling and CLT related techniques. We evaluate the model on a range of
regression and classification tasks against BNN inference alternatives, showing
competitive or improved performance over the current state-of-the-art.
