Characterizing Sparse Connectivity Patterns in Neural Networks
  We propose a novel way of reducing the number of parameters in the
storage-hungry fully connected layers of a neural network by using pre-defined
sparsity, where the majority of connections are absent prior to starting
training. Our results indicate that convolutional neural networks can operate
without any loss of accuracy at less than half percent classification layer
connection density, or less than 5 percent overall network connection density.
We also investigate the effects of pre-defining the sparsity of networks with
only fully connected layers. Based on our sparsifying technique, we introduce
the `scatter' metric to characterize the quality of a particular connection
pattern. As proof of concept, we show results on CIFAR, MNIST and a new dataset
on classifying Morse code symbols, which highlights some interesting trends and
limits of sparse connection patterns.
