Closed Form Variational Objectives For Bayesian Neural Networks with a
  Single Hidden Layer
  In this note we consider setups in which variational objectives for Bayesian
neural networks can be computed in closed form. In particular we focus on
single-layer networks in which the activation function is piecewise polynomial
(e.g. ReLU). In this case we show that for a Normal likelihood and structured
Normal variational distributions one can compute a variational lower bound in
closed form. In addition we compute the predictive mean and variance in closed
form. Finally, we also show how to compute approximate lower bounds for other
likelihoods (e.g. softmax classification). In experiments we show how the
resulting variational objectives can help improve training and provide fast
test time predictions.
