Plant 'n' Seek: Can You Find the Winning Ticket?
  The lottery ticket hypothesis has sparked the rapid development of pruning
algorithms that aim to reduce the computational costs associated with deep
learning during training and model deployment. Currently, such algorithms are
primarily evaluated on imaging data, for which we lack ground truth information
and thus the understanding of how sparse lottery tickets could be. To fill this
gap, we develop a framework that allows us to plant and hide winning tickets
with desirable properties in randomly initialized neural networks. To analyze
the ability of state-of-the-art pruning to identify tickets of extreme
sparsity, we design and hide such tickets solving four challenging tasks. In
extensive experiments, we observe similar trends as in imaging studies,
indicating that our framework can provide transferable insights into realistic
problems. Additionally, we can now see beyond such relative trends and
highlight limitations of current pruning methods. Based on our results, we
conclude that the current limitations in ticket sparsity are likely of
algorithmic rather than fundamental nature. We anticipate that comparisons to
planted tickets will facilitate future developments of efficient pruning
algorithms.
