Signing the Supermask: Keep, Hide, Invert
  The exponential growth in numbers of parameters of neural networks over the
past years has been accompanied by an increase in performance across several
fields. However, due to their sheer size, the networks not only became
difficult to interpret but also problematic to train and use in real-world
applications, since hardware requirements increased accordingly. Tackling both
issues, we present a novel approach that either drops a neural network's
initial weights or inverts their respective sign. Put simply, a network is
trained by weight selection and inversion without changing their absolute
values. Our contribution extends previous work on masking by additionally
sign-inverting the initial weights and follows the findings of the Lottery
Ticket Hypothesis. Through this extension and adaptations of initialization
methods, we achieve a pruning rate of up to 99%, while still matching or
exceeding the performance of various baseline and previous models. Our approach
has two main advantages. First, and most notable, signed Supermask models
drastically simplify a model's structure, while still performing well on given
tasks. Second, by reducing the neural network to its very foundation, we gain
insights into which weights matter for performance. The code is available on
GitHub.
