How many winning tickets are there in one DNN?
  The recent lottery ticket hypothesis proposes that there is one sub-network
that matches the accuracy of the original network when trained in isolation. We
show that instead each network contains several winning tickets, even if the
initial weights are fixed. The resulting winning sub-networks are not instances
of the same network under weight space symmetry, and show no overlap or
correlation significantly larger than expected by chance. If randomness during
training is decreased, overlaps higher than chance occur, even if the networks
are trained on different tasks. We conclude that there is rather a distribution
over capable sub-networks, as opposed to a single winning ticket.
