The Generalization-Stability Tradeoff In Neural Network Pruning
  Pruning neural network parameters is often viewed as a means to compress
models, but pruning has also been motivated by the desire to prevent
overfitting. This motivation is particularly relevant given the perhaps
surprising observation that a wide variety of pruning approaches increase test
accuracy despite sometimes massive reductions in parameter counts. To better
understand this phenomenon, we analyze the behavior of pruning over the course
of training, finding that pruning's benefit to generalization increases with
pruning's instability (defined as the drop in test accuracy immediately
following pruning). We demonstrate that this "generalization-stability
tradeoff" is present across a wide variety of pruning settings and propose a
mechanism for its cause: pruning regularizes similarly to noise injection.
Supporting this, we find less pruning stability leads to more model flatness
and the benefits of pruning do not depend on permanent parameter removal. These
results explain the compatibility of pruning-based generalization improvements
and the high generalization recently observed in overparameterized networks.
