Dynamics of Stochastic Momentum Methods on Large-scale, Quadratic Models
  We analyze a class of stochastic gradient algorithms with momentum on a
high-dimensional random least squares problem. Our framework, inspired by
random matrix theory, provides an exact (deterministic) characterization for
the sequence of loss values produced by these algorithms which is expressed
only in terms of the eigenvalues of the Hessian. This leads to simple
expressions for nearly-optimal hyperparameters, a description of the limiting
neighborhood, and average-case complexity.
  As a consequence, we show that (small-batch) stochastic heavy-ball momentum
with a fixed momentum parameter provides no actual performance improvement over
SGD when step sizes are adjusted correctly. For contrast, in the non-strongly
convex setting, it is possible to get a large improvement over SGD using
momentum. By introducing hyperparameters that depend on the number of samples,
we propose a new algorithm sDANA (stochastic dimension adjusted Nesterov
acceleration) which obtains an asymptotically optimal average-case complexity
while remaining linearly convergent in the strongly convex setting without
adjusting parameters.
