Deep Neural Compression Via Concurrent Pruning and Self-Distillation
  Pruning aims to reduce the number of parameters while maintaining performance
close to the original network. This work proposes a novel
\emph{self-distillation} based pruning strategy, whereby the representational
similarity between the pruned and unpruned versions of the same network is
maximized. Unlike previous approaches that treat distillation and pruning
separately, we use distillation to inform the pruning criteria, without
requiring a separate student network as in knowledge distillation. We show that
the proposed {\em cross-correlation objective for self-distilled pruning}
implicitly encourages sparse solutions, naturally complementing magnitude-based
pruning criteria. Experiments on the GLUE and XGLUE benchmarks show that
self-distilled pruning increases mono- and cross-lingual language model
performance. Self-distilled pruned models also outperform smaller Transformers
with an equal number of parameters and are competitive against (6 times) larger
distilled networks. We also observe that self-distillation (1) maximizes class
separability, (2) increases the signal-to-noise ratio, and (3) converges faster
after pruning steps, providing further insights into why self-distilled pruning
improves generalization.
