Local Quadratic Convergence of Stochastic Gradient Descent with Adaptive
  Step Size
  Establishing a fast rate of convergence for optimization methods is crucial
to their applicability in practice. With the increasing popularity of deep
learning over the past decade, stochastic gradient descent and its adaptive
variants (e.g. Adagrad, Adam, etc.) have become prominent methods of choice for
machine learning practitioners. While a large number of works have demonstrated
that these first order optimization methods can achieve sub-linear or linear
convergence, we establish local quadratic convergence for stochastic gradient
descent with adaptive step size for problems such as matrix inversion.
