Optimal Adaptive and Accelerated Stochastic Gradient Descent
  Stochastic gradient descent (\textsc{Sgd}) methods are the most powerful
optimization tools in training machine learning and deep learning models.
Moreover, acceleration (a.k.a. momentum) methods and diagonal scaling (a.k.a.
adaptive gradient) methods are the two main techniques to improve the slow
convergence of \textsc{Sgd}. While empirical studies have demonstrated
potential advantages of combining these two techniques, it remains unknown
whether these methods can achieve the optimal rate of convergence for
stochastic optimization. In this paper, we present a new class of adaptive and
accelerated stochastic gradient descent methods and show that they exhibit the
optimal sampling and iteration complexity for stochastic optimization. More
specifically, we show that diagonal scaling, initially designed to improve
vanilla stochastic gradient, can be incorporated into accelerated stochastic
gradient descent to achieve the optimal rate of convergence for smooth
stochastic optimization. We also show that momentum, apart from being known to
speed up the convergence rate of deterministic optimization, also provides us
new ways of designing non-uniform and aggressive moving average schemes in
stochastic optimization. Finally, we present some heuristics that help to
implement adaptive accelerated stochastic gradient descent methods and to
further improve their practical performance for machine learning and deep
learning.
