Picking Winning Tickets Before Training by Preserving Gradient Flow
  Overparameterization has been shown to benefit both the optimization and
generalization of neural networks, but large networks are resource hungry at
both training and test time. Network pruning can reduce test-time resource
requirements, but is typically applied to trained networks and therefore cannot
avoid the expensive training process. We aim to prune networks at
initialization, thereby saving resources at training time as well.
Specifically, we argue that efficient training requires preserving the gradient
flow through the network. This leads to a simple but effective pruning
criterion we term Gradient Signal Preservation (GraSP). We empirically
investigate the effectiveness of the proposed method with extensive experiments
on CIFAR-10, CIFAR-100, Tiny-ImageNet and ImageNet, using VGGNet and ResNet
architectures. Our method can prune 80% of the weights of a VGG-16 network on
ImageNet at initialization, with only a 1.6% drop in top-1 accuracy. Moreover,
our method achieves significantly better performance than the baseline at
extreme sparsity levels.
