Dynamic Collective Intelligence Learning: Finding Efficient Sparse Model
  via Refined Gradients for Pruned Weights
  With the growth of deep neural networks (DNN), the number of DNN parameters
has drastically increased. This makes DNN models hard to be deployed on
resource-limited embedded systems. To alleviate this problem, dynamic pruning
methods have emerged, which try to find diverse sparsity patterns during
training by utilizing Straight-Through-Estimator (STE) to approximate gradients
of pruned weights. STE can help the pruned weights revive in the process of
finding dynamic sparsity patterns. However, using these coarse gradients causes
training instability and performance degradation owing to the unreliable
gradient signal of the STE approximation. In this work, to tackle this issue,
we introduce refined gradients to update the pruned weights by forming dual
forwarding paths from two sets (pruned and unpruned) of weights. We propose a
novel Dynamic Collective Intelligence Learning (DCIL) which makes use of the
learning synergy between the collective intelligence of both weight sets. We
verify the usefulness of the refined gradients by showing enhancements in the
training stability and the model performance on the CIFAR and ImageNet
datasets. DCIL outperforms various previously proposed pruning schemes
including other dynamic pruning methods with enhanced stability during
training.
