Adaptively Preconditioned Stochastic Gradient Langevin Dynamics
  Stochastic Gradient Langevin Dynamics infuses isotropic gradient noise to SGD
to help navigate pathological curvature in the loss landscape for deep
networks. Isotropic nature of the noise leads to poor scaling, and adaptive
methods based on higher order curvature information such as Fisher Scoring have
been proposed to precondition the noise in order to achieve better convergence.
In this paper, we describe an adaptive method to estimate the parameters of the
noise and conduct experiments on well-known model architectures to show that
the adaptively preconditioned SGLD method achieves convergence with the speed
of adaptive first order methods such as Adam, AdaGrad etc. and achieves
generalization equivalent of SGD in the test set.
