Structured Directional Pruning via Perturbation Orthogonal Projection
  Structured pruning is an effective compression technique to reduce the
computation of neural networks, which is usually achieved by adding
perturbations to reduce network parameters at the cost of slightly increasing
training loss. A more reasonable approach is to find a sparse minimizer along
the flat minimum valley found by optimizers, i.e. stochastic gradient descent,
which keeps the training loss constant. To achieve this goal, we propose the
structured directional pruning based on orthogonal projecting the perturbations
onto the flat minimum valley. We also propose a fast solver sDprun and further
prove that it achieves directional pruning asymptotically after sufficient
training. Experiments using VGG-Net and ResNet on CIFAR-10 and CIFAR-100
datasets show that our method obtains the state-of-the-art pruned accuracy
(i.e. 93.97% on VGG16, CIFAR-10 task) without retraining. Experiments using
DNN, VGG-Net and WRN28X10 on MNIST, CIFAR-10 and CIFAR-100 datasets demonstrate
our method performs structured directional pruning, reaching the same minimum
valley as the optimizer.
