Scalable Natural Gradient Langevin Dynamics in Practice
  Stochastic Gradient Langevin Dynamics (SGLD) is a sampling scheme for
Bayesian modeling adapted to large datasets and models. SGLD relies on the
injection of Gaussian Noise at each step of a Stochastic Gradient Descent (SGD)
update. In this scheme, every component in the noise vector is independent and
has the same scale, whereas the parameters we seek to estimate exhibit strong
variations in scale and significant correlation structures, leading to poor
convergence and mixing times. We compare different preconditioning approaches
to the normalization of the noise vector and benchmark these approaches on the
following criteria: 1) mixing times of the multivariate parameter vector, 2)
regularizing effect on small dataset where it is easy to overfit, 3) covariate
shift detection and 4) resistance to adversarial examples.
