First-order and second-order variants of the gradient descent in a
  unified framework
  In this paper, we provide an overview of first-order and second-order
variants of the gradient descent method that are commonly used in machine
learning. We propose a general framework in which 6 of these variants can be
interpreted as different instances of the same approach. They are the vanilla
gradient descent, the classical and generalized Gauss-Newton methods, the
natural gradient descent method, the gradient covariance matrix approach, and
Newton's method. Besides interpreting these methods within a single framework,
we explain their specificities and show under which conditions some of them
coincide.
