Dynamical Isometry: The Missing Ingredient for Neural Network Pruning
  Several recent works [40, 24] observed an interesting phenomenon in neural
network pruning: A larger finetuning learning rate can improve the final
performance significantly. Unfortunately, the reason behind it remains elusive
up to date. This paper is meant to explain it through the lens of dynamical
isometry [42]. Specifically, we examine neural network pruning from an unusual
perspective: pruning as initialization for finetuning, and ask whether the
inherited weights serve as a good initialization for the finetuning? The
insights from dynamical isometry suggest a negative answer. Despite its
critical role, this issue has not been well-recognized by the community so far.
In this paper, we will show the understanding of this problem is very important
-- on top of explaining the aforementioned mystery about the larger finetuning
rate, it also unveils the mystery about the value of pruning [5, 30]. Besides a
clearer theoretical understanding of pruning, resolving the problem can also
bring us considerable performance benefits in practice.
