NodeDrop: A Condition for Reducing Network Size without Effect on Output
  Determining an appropriate number of features for each layer in a neural
network is an important and difficult task. This task is especially important
in applications on systems with limited memory or processing power. Many
current approaches to reduce network size either utilize iterative procedures,
which can extend training time significantly, or require very careful tuning of
algorithm parameters to achieve reasonable results. In this paper we propose
NodeDrop, a new method for eliminating features in a network. With NodeDrop, we
define a condition to identify and guarantee which nodes carry no information,
and then use regularization to encourage nodes to meet this condition. We find
that NodeDrop drastically reduces the number of features in a network while
maintaining high performance, reducing the number of parameters by a factor of
114x for a VGG like network on CIFAR10 without a drop in accuracy.
