FastAdaBelief: Improving Convergence Rate for Belief-based Adaptive
  Optimizers by Exploiting Strong Convexity
  AdaBelief, one of the current best optimizers, demonstrates superior
generalization ability compared to the popular Adam algorithm by viewing the
exponential moving average of observed gradients. AdaBelief is theoretically
appealing in that it has a data-dependent $O(\sqrt{T})$ regret bound when
objective functions are convex, where $T$ is a time horizon. It remains however
an open problem whether the convergence rate can be further improved without
sacrificing its generalization ability. %on how to exploit strong convexity to
further improve the convergence rate of AdaBelief. To this end, we make a first
attempt in this work and design a novel optimization algorithm called
FastAdaBelief that aims to exploit its strong convexity in order to achieve an
even faster convergence rate. In particular, by adjusting the step size that
better considers strong convexity and prevents fluctuation, our proposed
FastAdaBelief demonstrates excellent generalization ability as well as superior
convergence. As an important theoretical contribution, we prove that
FastAdaBelief attains a data-dependant $O(\log T)$ regret bound, which is
substantially lower than AdaBelief. On the empirical side, we validate our
theoretical analysis with extensive experiments in both scenarios of strong and
non-strong convexity on three popular baseline models. Experimental results are
very encouraging: FastAdaBelief converges the quickest in comparison to all
mainstream algorithms while maintaining an excellent generalization ability, in
cases of both strong or non-strong convexity. FastAdaBelief is thus posited as
a new benchmark model for the research community.
