A Contour Stochastic Gradient Langevin Dynamics Algorithm for
  Simulations of Multi-modal Distributions
  We propose an adaptively weighted stochastic gradient Langevin dynamics
algorithm (SGLD), so-called contour stochastic gradient Langevin dynamics
(CSGLD), for Bayesian learning in big data statistics. The proposed algorithm
is essentially a \emph{scalable dynamic importance sampler}, which
automatically \emph{flattens} the target distribution such that the simulation
for a multi-modal distribution can be greatly facilitated. Theoretically, we
prove a stability condition and establish the asymptotic convergence of the
self-adapting parameter to a {\it unique fixed-point}, regardless of the
non-convexity of the original energy function; we also present an error
analysis for the weighted averaging estimators. Empirically, the CSGLD
algorithm is tested on multiple benchmark datasets including CIFAR10 and
CIFAR100. The numerical results indicate its superiority to avoid the local
trap problem in training deep neural networks.
