Lottery Ticket Preserves Weight Correlation: Is It Desirable or Not?
  In deep model compression, the recent finding "Lottery Ticket Hypothesis"
(LTH) (Frankle & Carbin, 2018) pointed out that there could exist a winning
ticket (i.e., a properly pruned sub-network together with original weight
initialization) that can achieve competitive performance than the original
dense network. However, it is not easy to observe such winning property in many
scenarios, where for example, a relatively large learning rate is used even if
it benefits training the original dense model. In this work, we investigate the
underlying condition and rationale behind the winning property, and find that
the underlying reason is largely attributed to the correlation between
initialized weights and final-trained weights when the learning rate is not
sufficiently large. Thus, the existence of winning property is correlated with
an insufficient DNN pretraining, and is unlikely to occur for a well-trained
DNN. To overcome this limitation, we propose the "pruning & fine-tuning" method
that consistently outperforms lottery ticket sparse training under the same
pruning algorithm and the same total training epochs. Extensive experiments
over multiple deep models (VGG, ResNet, MobileNet-v2) on different datasets
have been conducted to justify our proposals.
