Pruning neural networks without any data by iteratively conserving
  synaptic flow
  Pruning the parameters of deep neural networks has generated intense interest
due to potential savings in time, memory and energy both during training and at
test time. Recent works have identified, through an expensive sequence of
training and pruning cycles, the existence of winning lottery tickets or sparse
trainable subnetworks at initialization. This raises a foundational question:
can we identify highly sparse trainable subnetworks at initialization, without
ever training, or indeed without ever looking at the data? We provide an
affirmative answer to this question through theory driven algorithm design. We
first mathematically formulate and experimentally verify a conservation law
that explains why existing gradient-based pruning algorithms at initialization
suffer from layer-collapse, the premature pruning of an entire layer rendering
a network untrainable. This theory also elucidates how layer-collapse can be
entirely avoided, motivating a novel pruning algorithm Iterative Synaptic Flow
Pruning (SynFlow). This algorithm can be interpreted as preserving the total
flow of synaptic strengths through the network at initialization subject to a
sparsity constraint. Notably, this algorithm makes no reference to the training
data and consistently competes with or outperforms existing state-of-the-art
pruning algorithms at initialization over a range of models (VGG and ResNet),
datasets (CIFAR-10/100 and Tiny ImageNet), and sparsity constraints (up to
99.99 percent). Thus our data-agnostic pruning algorithm challenges the
existing paradigm that, at initialization, data must be used to quantify which
synapses are important.
