How I Learned to Stop Worrying and Love Retraining
  Many Neural Network Pruning approaches consist of several iterative training
and pruning steps, seemingly losing a significant amount of their performance
after pruning and then recovering it in the subsequent retraining phase. Recent
works of Renda et al. (2020) and Le & Hua (2021) demonstrate the significance
of the learning rate schedule during the retraining phase and propose specific
heuristics for choosing such a schedule for IMP (Han et al., 2015). We place
these findings in the context of the results of Li et al. (2020) regarding the
training of models within a fixed training budget and demonstrate that,
consequently, the retraining phase can be massively shortened using a simple
linear learning rate schedule. Going a step further, we propose similarly
imposing a budget on the initial dense training phase and show that the
resulting simple and efficient method is capable of outperforming significantly
more complex or heavily parameterized state-of-the-art approaches that attempt
to sparsify the network during training. These findings not only advance our
understanding of the retraining phase, but more broadly question the belief
that one should aim to avoid the need for retraining and reduce the negative
effects of 'hard' pruning by incorporating the sparsification process into the
standard training.
