Ensembling Off-the-shelf Models for GAN Training
  The advent of large-scale training has produced a cornucopia of powerful
visual recognition models. However, generative models, such as GANs, have
traditionally been trained from scratch in an unsupervised manner. Can the
collective "knowledge" from a large bank of pretrained vision models be
leveraged to improve GAN training? If so, with so many models to choose from,
which one(s) should be selected, and in what manner are they most effective? We
find that pretrained computer vision models can significantly improve
performance when used in an ensemble of discriminators. Notably, the particular
subset of selected models greatly affects performance. We propose an effective
selection mechanism, by probing the linear separability between real and fake
samples in pretrained model embeddings, choosing the most accurate model, and
progressively adding it to the discriminator ensemble. Interestingly, our
method can improve GAN training in both limited data and large-scale settings.
Given only 10k training samples, our FID on LSUN Cat matches the StyleGAN2
trained on 1.6M images. On the full dataset, our method improves FID by 1.5x to
2x on cat, church, and horse categories of LSUN.
