Kronecker-factored Quasi-Newton Methods for Deep Learning
  Second-order methods have the capability of accelerating optimization by
using much richer curvature information than first-order methods. However, most
are impractical for deep learning, where the number of training parameters is
huge. In Goldfarb et al. (2020), practical quasi-Newton methods were proposed
that approximate the Hessian of a multilayer perceptron (MLP) model by a
layer-wise block diagonal matrix where each layer's block is further
approximated by a Kronecker product corresponding to the structure of the
Hessian restricted to that layer. Here, we extend these methods to enable them
to be applied to convolutional neural networks (CNNs), by analyzing the
Kronecker-factored structure of the Hessian matrix of convolutional layers.
Several improvements to the methods in Goldfarb et al. (2020) are also proposed
that can be applied to both MLPs and CNNs. These new methods have memory
requirements comparable to first-order methods and much less per-iteration time
complexity than those in Goldfarb et al. (2020). Moreover, convergence results
are proved for a variant under relatively mild conditions. Finally, we compared
the performance of our new methods against several state-of-the-art (SOTA)
methods on MLP autoencoder and CNN problems, and found that they outperformed
the first-order SOTA methods and performed comparably to the second-order SOTA
methods.
