One ticket to win them all: generalizing lottery ticket initializations
  across datasets and optimizers
  The success of lottery ticket initializations (Frankle and Carbin, 2019)
suggests that small, sparsified networks can be trained so long as the network
is initialized appropriately. Unfortunately, finding these "winning ticket"
initializations is computationally expensive. One potential solution is to
reuse the same winning tickets across a variety of datasets and optimizers.
However, the generality of winning ticket initializations remains unclear.
Here, we attempt to answer this question by generating winning tickets for one
training configuration (optimizer and dataset) and evaluating their performance
on another configuration. Perhaps surprisingly, we found that, within the
natural images domain, winning ticket initializations generalized across a
variety of datasets, including Fashion MNIST, SVHN, CIFAR-10/100, ImageNet, and
Places365, often achieving performance close to that of winning tickets
generated on the same dataset. Moreover, winning tickets generated using larger
datasets consistently transferred better than those generated using smaller
datasets. We also found that winning ticket initializations generalize across
optimizers with high performance. These results suggest that winning ticket
initializations generated by sufficiently large datasets contain inductive
biases generic to neural networks more broadly which improve training across
many settings and provide hope for the development of better initialization
methods.
