Fast yet Simple Natural-Gradient Descent for Variational Inference in
  Complex Models
  Bayesian inference plays an important role in advancing machine learning, but
faces computational challenges when applied to complex models such as deep
neural networks. Variational inference circumvents these challenges by
formulating Bayesian inference as an optimization problem and solving it using
gradient-based optimization. In this paper, we argue in favor of
natural-gradient approaches which, unlike their gradient-based counterparts,
can improve convergence by exploiting the information geometry of the
solutions. We show how to derive fast yet simple natural-gradient updates by
using a duality associated with exponential-family distributions. An attractive
feature of these methods is that, by using natural-gradients, they are able to
extract accurate local approximations for individual model components. We
summarize recent results for Bayesian deep learning showing the superiority of
natural-gradient approaches over their gradient counterparts.
