Network Pruning That Matters: A Case Study on Retraining Variants
  Network pruning is an effective method to reduce the computational expense of
over-parameterized neural networks for deployment on low-resource systems.
Recent state-of-the-art techniques for retraining pruned networks such as
weight rewinding and learning rate rewinding have been shown to outperform the
traditional fine-tuning technique in recovering the lost accuracy (Renda et
al., 2020), but so far it is unclear what accounts for such performance. In
this work, we conduct extensive experiments to verify and analyze the uncanny
effectiveness of learning rate rewinding. We find that the reason behind the
success of learning rate rewinding is the usage of a large learning rate.
Similar phenomenon can be observed in other learning rate schedules that
involve large learning rates, e.g., the 1-cycle learning rate schedule (Smith
et al., 2019). By leveraging the right learning rate schedule in retraining, we
demonstrate a counter-intuitive phenomenon in that randomly pruned networks
could even achieve better performance than methodically pruned networks
(fine-tuned with the conventional approach). Our results emphasize the
cruciality of the learning rate schedule in pruned network retraining - a
detail often overlooked by practitioners during the implementation of network
pruning. One-sentence Summary: We study the effective of different retraining
mechanisms while doing pruning
