Slimming Neural Networks using Adaptive Connectivity Scores
  In general, deep neural network (DNN) pruning methods fall into two
categories: 1) Weight-based deterministic constraints, and 2) Probabilistic
frameworks. While each approach has its merits and limitations there are a set
of common practical issues such as, trial-and-error to analyze sensitivity and
hyper-parameters to prune DNNs, which plague them both. In this work, we
propose a new single-shot, fully automated pruning algorithm called Slimming
Neural networks using Adaptive Connectivity Scores (SNACS). Our proposed
approach combines a probabilistic pruning framework with constraints on the
underlying weight matrices, via a novel connectivity measure, at multiple
levels to capitalize on the strengths of both approaches while solving their
deficiencies. In \alg{}, we propose a fast hash-based estimator of Adaptive
Conditional Mutual Information (ACMI), that uses a weight-based scaling
criterion, to evaluate the connectivity between filters and prune unimportant
ones. To automatically determine the limit up to which a layer can be pruned,
we propose a set of operating constraints that jointly define the upper pruning
percentage limits across all the layers in a deep network. Finally, we define a
novel sensitivity criterion for filters that measures the strength of their
contributions to the succeeding layer and highlights critical filters that need
to be completely protected from pruning. Through our experimental validation we
show that SNACS is faster by over 17x the nearest comparable method and is the
state of the art single-shot pruning method across three standard Dataset-DNN
pruning benchmarks: CIFAR10-VGG16, CIFAR10-ResNet56 and ILSVRC2012-ResNet50.
