FlipOut: Uncovering Redundant Weights via Sign Flipping
  Modern neural networks, although achieving state-of-the-art results on many
tasks, tend to have a large number of parameters, which increases training time
and resource usage. This problem can be alleviated by pruning. Existing
methods, however, often require extensive parameter tuning or multiple cycles
of pruning and retraining to convergence in order to obtain a favorable
accuracy-sparsity trade-off. To address these issues, we propose a novel
pruning method which uses the oscillations around $0$ (i.e. sign flips) that a
weight has undergone during training in order to determine its saliency. Our
method can perform pruning before the network has converged, requires little
tuning effort due to having good default values for its hyperparameters, and
can directly target the level of sparsity desired by the user. Our experiments,
performed on a variety of object classification architectures, show that it is
competitive with existing methods and achieves state-of-the-art performance for
levels of sparsity of $99.6\%$ and above for most of the architectures tested.
For reproducibility, we release our code publicly at
https://github.com/AndreiXYZ/flipout.
