Rapidly Adapting Moment Estimation
  Adaptive gradient methods such as Adam have been shown to be very effective
for training deep neural networks (DNNs) by tracking the second moment of
gradients to compute the individual learning rates. Differently from existing
methods, we make use of the most recent first moment of gradients to compute
the individual learning rates per iteration. The motivation behind it is that
the dynamic variation of the first moment of gradients may provide useful
information to obtain the learning rates. We refer to the new method as the
rapidly adapting moment estimation (RAME). The theoretical convergence of
deterministic RAME is studied by using an analysis similar to the one used in
[1] for Adam. Experimental results for training a number of DNNs show promising
performance of RAME w.r.t. the convergence speed and generalization performance
compared to the stochastic heavy-ball (SHB) method, Adam, and RMSprop.
