A Diffusion Approximation Theory of Momentum SGD in Nonconvex
  Optimization
  Momentum Stochastic Gradient Descent (MSGD) algorithm has been widely applied
to many nonconvex optimization problems in machine learning, e.g., training
deep neural networks, variational Bayesian inference, and etc. Despite its
empirical success, there is still a lack of theoretical understanding of
convergence properties of MSGD. To fill this gap, we propose to analyze the
algorithmic behavior of MSGD by diffusion approximations for nonconvex
optimization problems with strict saddle points and isolated local optima. Our
study shows that the momentum helps escape from saddle points, but hurts the
convergence within the neighborhood of optima (if without the step size
annealing or momentum annealing). Our theoretical discovery partially
corroborates the empirical success of MSGD in training deep neural networks.
