A decreasing scaling transition scheme from Adam to SGD
  Adaptive gradient algorithm (AdaGrad) and its variants, such as RMSProp,
Adam, AMSGrad, etc, have been widely used in deep learning. Although these
algorithms are faster in the early phase of training, their generalization
performance is often not as good as stochastic gradient descent (SGD). Hence, a
trade-off method of transforming Adam to SGD after a certain iteration to gain
the merits of both algorithms is theoretically and practically significant. To
that end, we propose a decreasing scaling transition scheme to achieve a smooth
and stable transition from Adam to SGD, which is called DSTAdam. The
convergence of the proposed DSTAdam is also proved in an online convex setting.
Finally, the effectiveness of the DSTAdam is verified on the CIFAR-10/100
datasets. Our implementation is available at:
https://github.com/kunzeng/DSTAdam.
