An Optimistic Acceleration of AMSGrad for Nonconvex Optimization
  We propose a new variant of AMSGrad, a popular adaptive gradient based
optimization algorithm widely used for training deep neural networks. Our
algorithm adds prior knowledge about the sequence of consecutive mini-batch
gradients and leverages its underlying structure making the gradients
sequentially predictable. By exploiting the predictability and ideas from
optimistic online learning, the proposed algorithm can accelerate the
convergence and increase sample efficiency. After establishing a tighter upper
bound under some convexity conditions on the regret, we offer a complimentary
view of our algorithm which generalizes the offline and stochastic version of
nonconvex optimization. In the nonconvex case, we establish a non-asymptotic
convergence bound independently of the initialization. We illustrate the
practical speedup on several deep learning models via numerical experiments.
