Adaptive Gradient Methods with Dynamic Bound of Learning Rate
  Adaptive optimization methods such as AdaGrad, RMSprop and Adam have been
proposed to achieve a rapid training process with an element-wise scaling term
on learning rates. Though prevailing, they are observed to generalize poorly
compared with SGD or even fail to converge due to unstable and extreme learning
rates. Recent work has put forward some algorithms such as AMSGrad to tackle
this issue but they failed to achieve considerable improvement over existing
methods. In our paper, we demonstrate that extreme learning rates can lead to
poor performance. We provide new variants of Adam and AMSGrad, called AdaBound
and AMSBound respectively, which employ dynamic bounds on learning rates to
achieve a gradual and smooth transition from adaptive methods to SGD and give a
theoretical proof of convergence. We further conduct experiments on various
popular tasks and models, which is often insufficient in previous work.
Experimental results show that new variants can eliminate the generalization
gap between adaptive methods and SGD and maintain higher learning speed early
in training at the same time. Moreover, they can bring significant improvement
over their prototypes, especially on complex deep networks. The implementation
of the algorithm can be found at https://github.com/Luolc/AdaBound .
