A Qualitative Study of the Dynamic Behavior for Adaptive Gradient
  Algorithms
  The dynamic behavior of RMSprop and Adam algorithms is studied through a
combination of careful numerical experiments and theoretical explanations.
Three types of qualitative features are observed in the training loss curve:
fast initial convergence, oscillations, and large spikes in the late phase. The
sign gradient descent (signGD) flow, which is the limit of Adam when taking the
learning rate to 0 while keeping the momentum parameters fixed, is used to
explain the fast initial convergence. For the late phase of Adam, three
different types of qualitative patterns are observed depending on the choice of
the hyper-parameters: oscillations, spikes, and divergence. In particular, Adam
converges much smoother and even faster when the values of the two momentum
factors are close to each other. This observation is particularly important for
scientific computing tasks, for which the training process usually proceeds
into the high precision regime.
