LaProp: Separating Momentum and Adaptivity in Adam
  We identity a by-far-unrecognized problem of Adam-style optimizers which
results from unnecessary coupling between momentum and adaptivity. The coupling
leads to instability and divergence when the momentum and adaptivity parameters
are mismatched. In this work, we propose a method, Laprop, which decouples
momentum and adaptivity in the Adam-style methods. We show that the decoupling
leads to greater flexibility in the hyperparameters and allows for a
straightforward interpolation between the signed gradient methods and the
adaptive gradient methods. We experimentally show that Laprop has consistently
improved speed and stability over Adam on a variety of tasks. We also bound the
regret of Laprop on a convex problem and show that our bound differs from that
of Adam by a key factor, which demonstrates its advantage.
