Convergence Analysis of a Momentum Algorithm with Adaptive Step Size for
  Non Convex Optimization
  Although ADAM is a very popular algorithm for optimizing the weights of
neural networks, it has been recently shown that it can diverge even in simple
convex optimization examples. Several variants of ADAM have been proposed to
circumvent this convergence issue. In this work, we study the ADAM algorithm
for smooth nonconvex optimization under a boundedness assumption on the
adaptive learning rate. The bound on the adaptive step size depends on the
Lipschitz constant of the gradient of the objective function and provides safe
theoretical adaptive step sizes. Under this boundedness assumption, we show a
novel first order convergence rate result in both deterministic and stochastic
contexts. Furthermore, we establish convergence rates of the function value
sequence using the Kurdyka-Lojasiewicz property.
