Generalized AdaGrad (G-AdaGrad) and Adam: A State-Space Perspective
  Accelerated gradient-based methods are being extensively used for solving
non-convex machine learning problems, especially when the data points are
abundant or the available data is distributed across several agents. Two of the
prominent accelerated gradient algorithms are AdaGrad and Adam. AdaGrad is the
simplest accelerated gradient method, which is particularly effective for
sparse data. Adam has been shown to perform favorably in deep learning problems
compared to other methods. In this paper, we propose a new fast optimizer,
Generalized AdaGrad (G-AdaGrad), for accelerating the solution of potentially
non-convex machine learning problems. Specifically, we adopt a state-space
perspective for analyzing the convergence of gradient acceleration algorithms,
namely G-AdaGrad and Adam, in machine learning. Our proposed state-space models
are governed by ordinary differential equations. We present simple convergence
proofs of these two algorithms in the deterministic settings with minimal
assumptions. Our analysis also provides intuition behind improving upon
AdaGrad's convergence rate. We provide empirical results on MNIST dataset to
reinforce our claims on the convergence and performance of G-AdaGrad and Adam.
