Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adam
  Uncertainty computation in deep learning is essential to design robust and
reliable systems. Variational inference (VI) is a promising approach for such
computation, but requires more effort to implement and execute compared to
maximum-likelihood methods. In this paper, we propose new natural-gradient
algorithms to reduce such efforts for Gaussian mean-field VI. Our algorithms
can be implemented within the Adam optimizer by perturbing the network weights
during gradient evaluations, and uncertainty estimates can be cheaply obtained
by using the vector that adapts the learning rate. This requires lower memory,
computation, and implementation effort than existing VI methods, while
obtaining uncertainty estimates of comparable quality. Our empirical results
confirm this and further suggest that the weight-perturbation in our algorithm
could be useful for exploration in reinforcement learning and stochastic
optimization.
