Statistical Mechanical Analysis of Neural Network Pruning
  Deep learning architectures with a huge number of parameters are often
compressed using pruning techniques to ensure computational efficiency of
inference during deployment. Despite multitude of empirical advances, there is
a lack of theoretical understanding of the effectiveness of different pruning
methods. We inspect different pruning techniques under the statistical
mechanics formulation of a teacher-student framework and derive their
generalization error (GE) bounds. It has been shown that Determinantal Point
Process (DPP) based node pruning method is notably superior to competing
approaches when tested on real datasets. Using GE bounds in the aforementioned
setup we provide theoretical guarantees for their empirical observations.
Another consistent finding in literature is that sparse neural networks (edge
pruned) generalize better than dense neural networks (node pruned) for a fixed
number of parameters. We use our theoretical setup to prove this finding and
show that even the baseline random edge pruning method performs better than the
DPP node pruning method. We also validate this empirically on real datasets.
