Neural network relief: a pruning algorithm based on neural activity
  Current deep neural networks (DNNs) are overparameterized and use most of
their neuronal connections during inference for each task. The human brain,
however, developed specialized regions for different tasks and performs
inference with a small fraction of its neuronal connections. We propose an
iterative pruning strategy introducing a simple importance-score metric that
deactivates unimportant connections, tackling overparameterization in DNNs and
modulating the firing patterns. The aim is to find the smallest number of
connections that is still capable of solving a given task with comparable
accuracy, i.e. a simpler subnetwork. We achieve comparable performance for
LeNet architectures on MNIST, and significantly higher parameter compression
than state-of-the-art algorithms for VGG and ResNet architectures on
CIFAR-10/100 and Tiny-ImageNet. Our approach also performs well for the two
different optimizers considered -- Adam and SGD. The algorithm is not designed
to minimize FLOPs when considering current hardware and software
implementations, although it performs reasonably when compared to the state of
the art.
