Universality of Winning Tickets: A Renormalization Group Perspective
  Foundational work on the Lottery Ticket Hypothesis has suggested an exciting
corollary: winning tickets found in the context of one task can be transferred
to similar tasks, possibly even across different architectures. This has
generated broad interest, but methods to study this universality are lacking.
We make use of renormalization group theory, a powerful tool from theoretical
physics, to address this need. We find that iterative magnitude pruning, the
principal algorithm used for discovering winning tickets, is a renormalization
group scheme, and can be viewed as inducing a flow in parameter space. We
demonstrate that ResNet-50 models with transferable winning tickets have flows
with common properties, as would be expected from the theory. Similar
observations are made for BERT models, with evidence that their flows are near
fixed points. Additionally, we leverage our framework to study winning tickets
transferred across ResNet architectures, observing that smaller models have
flows with more uniform properties than larger models, complicating transfer
between them.
