Stochastic Model Pruning via Weight Dropping Away and Back
  Deep neural networks have dramatically achieved great success on a variety of
challenging tasks. However, most successful DNNs have an extremely complex
structure, leading to extensive research on model compression.As a significant
area of progress in model compression, traditional gradual pruning approaches
involve an iterative prune-retrain procedure and may suffer from two critical
issues: local importance judgment, where the pruned weights are merely
unimportant in the current model; and an irretrievable pruning process, where
the pruned weights have no chance to come back. Addressing these two issues,
this paper proposes the Drop Pruning approach, which leverages stochastic
optimization in the pruning process by introducing a drop strategy at each
pruning step, namely, drop away, which stochastically deletes some unimportant
weights, and drop back, which stochastically recovers some pruned weights. The
suitable choice of drop probabilities decreases the model size during the
pruning process and helps it flow to the target sparsity. Compared to the
Bayesian approaches that stochastically train a compact model for pruning, we
directly aim at stochastic gradual pruning. We provide a detailed analysis
showing that the drop away and drop back approaches have individual
contributions. Moreover, Drop Pruning can achieve competitive compression
performance and accuracy on many benchmark tasks compared with state-of-the-art
weights pruning and Bayesian training approaches.
