A Gradient Flow Framework For Analyzing Network Pruning
  Recent network pruning methods focus on pruning models early-on in training.
To estimate the impact of removing a parameter, these methods use importance
measures that were originally designed to prune trained models. Despite lacking
justification for their use early-on in training, such measures result in
surprisingly low accuracy loss. To better explain this behavior, we develop a
general framework that uses gradient flow to unify state-of-the-art importance
measures through the norm of model parameters. We use this framework to
determine the relationship between pruning measures and evolution of model
parameters, establishing several results related to pruning models early-on in
training: (i) magnitude-based pruning removes parameters that contribute least
to reduction in loss, resulting in models that converge faster than
magnitude-agnostic methods; (ii) loss-preservation based pruning preserves
first-order model evolution dynamics and is therefore appropriate for pruning
minimally trained models; and (iii) gradient-norm based pruning affects
second-order model evolution dynamics, such that increasing gradient norm via
pruning can produce poorly performing models. We validate our claims on several
VGG-13, MobileNet-V1, and ResNet-56 models trained on CIFAR-10/CIFAR-100. Code
available at https://github.com/EkdeepSLubana/flowandprune.
