Powerpropagation: A sparsity inducing weight reparameterisation
  The training of sparse neural networks is becoming an increasingly important
tool for reducing the computational footprint of models at training and
evaluation, as well enabling the effective scaling up of models. Whereas much
work over the years has been dedicated to specialised pruning techniques,
little attention has been paid to the inherent effect of gradient based
training on model sparsity. In this work, we introduce Powerpropagation, a new
weight-parameterisation for neural networks that leads to inherently sparse
models. Exploiting the behaviour of gradient descent, our method gives rise to
weight updates exhibiting a "rich get richer" dynamic, leaving low-magnitude
parameters largely unaffected by learning. Models trained in this manner
exhibit similar performance, but have a distribution with markedly higher
density at zero, allowing more parameters to be pruned safely. Powerpropagation
is general, intuitive, cheap and straight-forward to implement and can readily
be combined with various other techniques. To highlight its versatility, we
explore it in two very different settings: Firstly, following a recent line of
work, we investigate its effect on sparse training for resource-constrained
settings. Here, we combine Powerpropagation with a traditional weight-pruning
technique as well as recent state-of-the-art sparse-to-sparse algorithms,
showing superior performance on the ImageNet benchmark. Secondly, we advocate
the use of sparsity in overcoming catastrophic forgetting, where compressed
representations allow accommodating a large number of tasks at fixed model
capacity. In all cases our reparameterisation considerably increases the
efficacy of the off-the-shelf methods.
