Winning the Lottery with Continuous Sparsification
  The search for efficient, sparse deep neural network models is most
prominently performed by pruning: training a dense, overparameterized network
and removing parameters, usually via following a manually-crafted heuristic.
Additionally, the recent Lottery Ticket Hypothesis conjectures that, for a
typically-sized neural network, it is possible to find small sub-networks
which, when trained from scratch on a comparable budget, match the performance
of the original dense counterpart. We revisit fundamental aspects of pruning
algorithms, pointing out missing ingredients in previous approaches, and
develop a method, Continuous Sparsification, which searches for sparse networks
based on a novel approximation of an intractable $\ell_0$ regularization. We
compare against dominant heuristic-based methods on pruning as well as ticket
search -- finding sparse subnetworks that can be successfully re-trained from
an early iterate. Empirical results show that we surpass the state-of-the-art
for both objectives, across models and datasets, including VGG trained on
CIFAR-10 and ResNet-50 trained on ImageNet. In addition to setting a new
standard for pruning, Continuous Sparsification also offers fast parallel
ticket search, opening doors to new applications of the Lottery Ticket
Hypothesis.
