On the diffusion approximation of nonconvex stochastic gradient descent
  We study the Stochastic Gradient Descent (SGD) method in nonconvex
optimization problems from the point of view of approximating diffusion
processes. We prove rigorously that the diffusion process can approximate the
SGD algorithm weakly using the weak form of master equation for probability
evolution. In the small step size regime and the presence of omnidirectional
noise, our weak approximating diffusion process suggests the following dynamics
for the SGD iteration starting from a local minimizer (resp.~saddle point): it
escapes in a number of iterations exponentially (resp.~almost linearly)
dependent on the inverse stepsize. The results are obtained using the theory
for random perturbations of dynamical systems (theory of large deviations for
local minimizers and theory of exiting for unstable stationary points). In
addition, we discuss the effects of batch size for the deep neural networks,
and we find that small batch size is helpful for SGD algorithms to escape
unstable stationary points and sharp minimizers. Our theory indicates that one
should increase the batch size at later stage for the SGD to be trapped in flat
minimizers for better generalization.
