Noise and Fluctuation of Finite Learning Rate Stochastic Gradient
  Descent
  In the vanishing learning rate regime, stochastic gradient descent (SGD) is
now relatively well understood. In this work, we propose to study the basic
properties of SGD and its variants in the non-vanishing learning rate regime.
The focus is on deriving exactly solvable results and discussing their
implications. The main contributions of this work are to derive the stationary
distribution for discrete-time SGD in a quadratic loss function with and
without momentum; in particular, one implication of our result is that the
fluctuation caused by discrete-time dynamics takes a distorted shape and is
dramatically larger than a continuous-time theory could predict. Examples of
applications of the proposed theory considered in this work include the
approximation error of variants of SGD, the effect of minibatch noise, the
optimal Bayesian inference, the escape rate from a sharp minimum, and the
stationary covariance of a few second-order methods including damped Newton's
method, natural gradient descent, and Adam.
