Efficient Natural Gradient Descent Methods for Large-Scale Optimization
  Problems
  We propose an efficient numerical method for computing natural gradient
descent directions with respect to a generic metric in the state space. Our
technique relies on representing the natural gradient direction as a solution
to a standard least-squares problem. Hence, instead of calculating, storing, or
inverting the information matrix directly, we apply efficient methods from
numerical linear algebra to solve this least-squares problem. We treat both
scenarios where the derivative of the state variable with respect to the
parameter is either explicitly known or implicitly given through constraints.
We apply the QR decomposition to solve the least-squares problem in the former
case and utilize the adjoint-state method to compute the natural gradient
descent direction in the latter case. As a result, we can reliably compute
several natural gradient descents, including the Wasserstein natural gradient,
for a large-scale parameter space with thousands of dimensions, which was
believed to be out of reach. Finally, our numerical results shed light on the
qualitative differences among the standard gradient descent method and various
natural gradient descent methods based on different metric spaces in
large-scale nonconvex optimization problems.
