Activation Density driven Energy-Efficient Pruning in Training
  Neural network pruning with suitable retraining can yield networks with
considerably fewer parameters than the original with comparable degrees of
accuracy. Typical pruning methods require large, fully trained networks as a
starting point from which they perform a time-intensive iterative pruning and
retraining procedure to regain the original accuracy. We propose a novel
pruning method that prunes a network real-time during training, reducing the
overall training time to achieve an efficient compressed network. We introduce
an activation density based analysis to identify the optimal relative sizing or
compression for each layer of the network. Our method is architecture agnostic,
allowing it to be employed on a wide variety of systems. For VGG-19 and
ResNet18 on CIFAR-10, CIFAR-100, and TinyImageNet, we obtain exceedingly sparse
networks (up to $200 \times$ reduction in parameters and over $60 \times$
reduction in inference compute operations in the best case) with accuracy
comparable to the baseline network. By reducing the network size periodically
during training, we achieve total training times that are shorter than those of
previously proposed pruning methods. Furthermore, training compressed networks
at different epochs with our proposed method yields considerable reduction in
training compute complexity ($1.6\times$ to $3.2\times$ lower) at near
iso-accuracy as compared to a baseline network trained entirely from scratch.
