Network Trimming: A Data-Driven Neuron Pruning Approach towards
  Efficient Deep Architectures
  State-of-the-art neural networks are getting deeper and wider. While their
performance increases with the increasing number of layers and neurons, it is
crucial to design an efficient deep architecture in order to reduce
computational and memory costs. Designing an efficient neural network, however,
is labor intensive requiring many experiments, and fine-tunings. In this paper,
we introduce network trimming which iteratively optimizes the network by
pruning unimportant neurons based on analysis of their outputs on a large
dataset. Our algorithm is inspired by an observation that the outputs of a
significant portion of neurons in a large network are mostly zero, regardless
of what inputs the network received. These zero activation neurons are
redundant, and can be removed without affecting the overall accuracy of the
network. After pruning the zero activation neurons, we retrain the network
using the weights before pruning as initialization. We alternate the pruning
and retraining to further reduce zero activations in a network. Our experiments
on the LeNet and VGG-16 show that we can achieve high compression ratio of
parameters without losing or even achieving higher accuracy than the original
network.
