Data-Efficient Structured Pruning via Submodular Optimization
  Structured pruning is an effective approach for compressing large pre-trained
neural networks without significantly affecting their performance, which
involves removing redundant regular regions of weights. However, current
structured pruning methods are highly empirical in nature, do not provide any
theoretical guarantees, and often require fine-tuning, which makes them
inapplicable in the limited-data regime. We propose a principled data-efficient
structured pruning method based on submodular optimization. In particular, for
a given layer, we select neurons/channels to prune and corresponding new
weights for the next layer, that minimize the change in the next layer's input
induced by pruning. We show that this selection problem is a weakly submodular
maximization problem, thus it can be provably approximated using an efficient
greedy algorithm. Our method is one of the few in the literature that uses only
a limited-number of training data and no labels. Our experimental results
demonstrate that our method outperforms popular baseline methods in various
one-shot pruning settings.
