WoodFisher: Efficient Second-Order Approximation for Neural Network
  Compression
  Second-order information, in the form of Hessian- or Inverse-Hessian-vector
products, is a fundamental tool for solving optimization problems. Recently,
there has been significant interest in utilizing this information in the
context of deep neural networks; however, relatively little is known about the
quality of existing approximations in this context. Our work examines this
question, identifies issues with existing approaches, and proposes a method
called WoodFisher to compute a faithful and efficient estimate of the inverse
Hessian.
  Our main application is to neural network compression, where we build on the
classic Optimal Brain Damage/Surgeon framework. We demonstrate that WoodFisher
significantly outperforms popular state-of-the-art methods for one-shot
pruning. Further, even when iterative, gradual pruning is considered, our
method results in a gain in test accuracy over the state-of-the-art approaches,
for pruning popular neural networks (like ResNet-50, MobileNetV1) trained on
standard image classification datasets such as ImageNet ILSVRC. We examine how
our method can be extended to take into account first-order information, as
well as illustrate its ability to automatically set layer-wise pruning
thresholds and perform compression in the limited-data regime. The code is
available at the following link, https://github.com/IST-DASLab/WoodFisher.
