Reproducibility Study: Comparing Rewinding and Fine-tuning in Neural
  Network Pruning
  Scope of reproducibility: We are reproducing Comparing Rewinding and
Fine-tuning in Neural Networks from arXiv:2003.02389. In this work the authors
compare three different approaches to retraining neural networks after pruning:
1) fine-tuning, 2) rewinding weights as in arXiv:1803.03635 and 3) a new,
original method involving learning rate rewinding, building upon Lottery Ticket
Hypothesis. We reproduce the results of all three approaches, but we focus on
verifying their approach, learning rate rewinding, since it is newly proposed
and is described as a universal alternative to other methods.
  We used CIFAR10 for most reproductions along with additional experiments on
the larger CIFAR100, which extends the results originally provided by the
authors. We have also extended the list of tested network architectures to
include Wide ResNets. The new experiments led us to discover the limitations of
learning rate rewinding which can worsen pruning results on large
architectures.
  Results: We were able to reproduce the exact results reported by the authors
in all originally reported scenarios. However, extended results on larger Wide
Residual Networks have demonstrated the limitations of the newly proposed
learning rate rewinding -- we observed a previously unreported accuracy
degradation for low sparsity ranges. Nevertheless, the general conclusion of
the paper still holds and was indeed reproduced.
