Deconstructing the Structure of Sparse Neural Networks
  Although sparse neural networks have been studied extensively, the focus has
been primarily on accuracy. In this work, we focus instead on network
structure, and analyze three popular algorithms. We first measure performance
when structure persists and weights are reset to a different random
initialization, thereby extending experiments in Deconstructing Lottery Tickets
(Zhou et al., 2019). This experiment reveals that accuracy can be derived from
structure alone. Second, to measure structural robustness we investigate the
sensitivity of sparse neural networks to further pruning after training,
finding a stark contrast between algorithms. Finally, for a recent dynamic
sparsity algorithm we investigate how early in training the structure emerges.
We find that even after one epoch the structure is mostly determined, allowing
us to propose a more efficient algorithm which does not require dense gradients
throughout training. In looking back at algorithms for sparse neural networks
and analyzing their performance from a different lens, we uncover several
interesting properties and promising directions for future research.
