On the Hyperparameters in Stochastic Gradient Descent with Momentum
  Following the same routine as [SSJ20], we continue to present the theoretical
analysis for stochastic gradient descent with momentum (SGD with momentum) in
this paper. Differently, for SGD with momentum, we demonstrate it is the two
hyperparameters together, the learning rate and the momentum coefficient, that
play the significant role for the linear rate of convergence in non-convex
optimization. Our analysis is based on the use of a hyperparameters-dependent
stochastic differential equation (hp-dependent SDE) that serves as a continuous
surrogate for SGD with momentum. Similarly, we establish the linear convergence
for the continuous-time formulation of SGD with momentum and obtain an explicit
expression for the optimal linear rate by analyzing the spectrum of the
Kramers-Fokker-Planck operator. By comparison, we demonstrate how the optimal
linear rate of convergence and the final gap for SGD only about the learning
rate varies with the momentum coefficient increasing from zero to one when the
momentum is introduced. Then, we propose a mathematical interpretation why the
SGD with momentum converges faster and more robust about the learning rate than
the standard SGD in practice. Finally, we show the Nesterov momentum under the
existence of noise has no essential difference with the standard momentum.
