An Adaptive Gradient Method with Energy and Momentum
  We introduce a novel algorithm for gradient-based optimization of stochastic
objective functions. The method may be seen as a variant of SGD with momentum
equipped with an adaptive learning rate automatically adjusted by an 'energy'
variable. The method is simple to implement, computationally efficient, and
well suited for large-scale machine learning problems. The method exhibits
unconditional energy stability for any size of the base learning rate. We
provide a regret bound on the convergence rate under the online convex
optimization framework. We also establish the energy-dependent convergence rate
of the algorithm to a stationary point in the stochastic non-convex setting. In
addition, a sufficient condition is provided to guarantee a positive lower
threshold for the energy variable. Our experiments demonstrate that the
algorithm converges fast while generalizing better than or as well as SGD with
momentum in training deep neural networks, and compares also favorably to Adam.
