The Bayesian Learning Rule
  We show that many machine-learning algorithms are specific instances of a
single algorithm called the Bayesian learning rule. The rule, derived from
Bayesian principles, yields a wide-range of algorithms from fields such as
optimization, deep learning, and graphical models. This includes classical
algorithms such as ridge regression, Newton's method, and Kalman filter, as
well as modern deep-learning algorithms such as stochastic-gradient descent,
RMSprop, and Dropout. The key idea in deriving such algorithms is to
approximate the posterior using candidate distributions estimated by using
natural gradients. Different candidate distributions result in different
algorithms and further approximations to natural gradients give rise to
variants of those algorithms. Our work not only unifies, generalizes, and
improves existing algorithms, but also helps us design new ones.
