DiffPrune: Neural Network Pruning with Deterministic Approximate Binary
  Gates and $L_0$ Regularization
  Modern neural network architectures typically have many millions of
parameters and can be pruned significantly without substantial loss in
effectiveness which demonstrates they are over-parameterized. The contribution
of this work is two-fold. The first is a method for approximating a
multivariate Bernoulli random variable by means of a deterministic and
differentiable transformation of any real-valued multivariate random variable.
The second is a method for model selection by element-wise multiplication of
parameters with approximate binary gates that may be computed deterministically
or stochastically and take on exact zero values. Sparsity is encouraged by the
inclusion of a surrogate regularization to the $L_0$ loss. Since the method is
differentiable it enables straightforward and efficient learning of model
architectures by an empirical risk minimization procedure with stochastic
gradient descent and theoretically enables conditional computation during
training. The method also supports any arbitrary group sparsity over parameters
or activations and therefore offers a framework for unstructured or flexible
structured model pruning. To conclude experiments are performed to demonstrate
the effectiveness of the proposed approach.
