Dynamic of Stochastic Gradient Descent with State-Dependent Noise
  Stochastic gradient descent (SGD) and its variants are mainstream methods to
train deep neural networks. Since neural networks are non-convex, more and more
works study the dynamic behavior of SGD and the impact to its generalization,
especially the escaping efficiency from local minima. However, these works take
the over-simplified assumption that the covariance of the noise in SGD is (or
can be upper bounded by) constant, although it is actually state-dependent. In
this work, we conduct a formal study on the dynamic behavior of SGD with
state-dependent noise. Specifically, we show that the covariance of the noise
of SGD in the local region of the local minima is a quadratic function of the
state. Thus, we propose a novel power-law dynamic with state-dependent
diffusion to approximate the dynamic of SGD. We prove that, power-law dynamic
can escape from sharp minima exponentially faster than flat minima, while the
previous dynamics can only escape sharp minima polynomially faster than flat
minima. Our experiments well verified our theoretical results. Inspired by our
theory, we propose to add additional state-dependent noise into (large-batch)
SGD to further improve its generalization ability. Experiments verify that our
method is effective.
