Stochastic Gradient Langevin Dynamics Algorithms with Adaptive Drifts
  Bayesian deep learning offers a principled way to address many issues
concerning safety of artificial intelligence (AI), such as model
uncertainty,model interpretability, and prediction bias. However, due to the
lack of efficient Monte Carlo algorithms for sampling from the posterior of
deep neural networks (DNNs), Bayesian deep learning has not yet powered our AI
system. We propose a class of adaptive stochastic gradient Markov chain Monte
Carlo (SGMCMC) algorithms, where the drift function is biased to enhance escape
from saddle points and the bias is adaptively adjusted according to the
gradient of past samples. We establish the convergence of the proposed
algorithms under mild conditions, and demonstrate via numerical examples that
the proposed algorithms can significantly outperform the existing SGMCMC
algorithms, such as stochastic gradient Langevin dynamics (SGLD), stochastic
gradient Hamiltonian Monte Carlo (SGHMC) and preconditioned SGLD, in both
simulation and optimization tasks.
