Bespoke vs. Pr\^et-\`a-Porter Lottery Tickets: Exploiting Mask
  Similarity for Trainable Sub-Network Finding
  The observation of sparse trainable sub-networks within over-parametrized
networks - also known as Lottery Tickets (LTs) - has prompted inquiries around
their trainability, scaling, uniqueness, and generalization properties. Across
28 combinations of image classification tasks and architectures, we discover
differences in the connectivity structure of LTs found through different
iterative pruning techniques, thus disproving their uniqueness and connecting
emergent mask structure to the choice of pruning. In addition, we propose a
consensus-based method for generating refined lottery tickets. This lottery
ticket denoising procedure, based on the principle that parameters that always
go unpruned across different tasks more reliably identify important
sub-networks, is capable of selecting a meaningful portion of the architecture
in an embarrassingly parallel way, while quickly discarding extra parameters
without the need for further pruning iterations. We successfully train these
sub-networks to performance comparable to that of ordinary lottery tickets.
