A Novel Convergence Analysis for Algorithms of the Adam Family
  Since its invention in 2014, the Adam optimizer has received tremendous
attention. On one hand, it has been widely used in deep learning and many
variants have been proposed, while on the other hand their theoretical
convergence property remains to be a mystery. It is far from satisfactory in
the sense that some studies require strong assumptions about the updates, which
are not necessarily applicable in practice, while other studies still follow
the original problematic convergence analysis of Adam, which was shown to be
not sufficient to ensure convergence. Although rigorous convergence analysis
exists for Adam, they impose specific requirements on the update of the
adaptive step size, which are not generic enough to cover many other variants
of Adam. To address theses issues, in this extended abstract, we present a
simple and generic proof of convergence for a family of Adam-style methods
(including Adam, AMSGrad, Adabound, etc.). Our analysis only requires an
increasing or large "momentum" parameter for the first-order moment, which is
indeed the case used in practice, and a boundness condition on the adaptive
factor of the step size, which applies to all variants of Adam under mild
conditions of stochastic gradients. We also establish a variance diminishing
result for the used stochastic gradient estimators. Indeed, our analysis of
Adam is so simple and generic that it can be leveraged to establish the
convergence for solving a broader family of non-convex optimization problems,
including min-max, compositional, and bilevel optimization problems. For the
full (earlier) version of this extended abstract, please refer to
arXiv:2104.14840.
