Weighted AdaGrad with Unified Momentum
  Integrating adaptive learning rate and momentum techniques into SGD leads to
a large class of efficiently accelerated adaptive stochastic algorithms, such
as Nadam, AccAdaGrad, \textit{etc}. In spite of their effectiveness in
practice, there is still a large gap in their theories of convergences,
especially in the difficult non-convex stochastic setting. To fill this gap, we
propose \emph{weighted AdaGrad with unified momentum}, dubbed AdaUSM, which has
the main characteristics that (1) it incorporates a unified momentum scheme
which covers both the heavy ball momentum and the Nesterov accelerated gradient
momentum; (2) it adopts a novel weighted adaptive learning rate that can unify
the learning rates of AdaGrad, AccAdaGrad, Adam, and RMSProp. Moreover, when we
take polynomially growing weights in AdaUSM, we obtain its
$\mathcal{O}(\log(T)/\sqrt{T})$ convergence rate in the non-convex stochastic
setting. We also show that the adaptive learning rates of Adam and RMSProp
correspond to taking exponentially growing weights in AdaUSM, which thereby
provides a new perspesctive for understanding Adam and RMSProp. Lastly,
comparative experiments of AdaUSM against SGD with momentum, AdaGrad, AdaEMA,
Adam, and AMSGrad on various deep learning models and datasets are also
provided.
