KNN-Diffusion: Image Generation via Large-Scale Retrieval
  While the availability of massive Text-Image datasets is shown to be
extremely useful in training large-scale generative models (e.g. DDPMs,
Transformers), their output typically depends on the quality of both the input
text, as well as the training dataset. In this work, we show how large-scale
retrieval methods, in particular efficient K-Nearest-Neighbors (KNN) search,
can be used in order to train a model to adapt to new samples. Learning to
adapt enables several new capabilities. Sifting through billions of records at
inference time is extremely efficient and can alleviate the need to train or
memorize an adequately large generative model. Additionally, fine-tuning
trained models to new samples can be achieved by simply adding them to the
table. Rare concepts, even without any presence in the training set, can be
then leveraged during test time without any modification to the generative
model. Our diffusion-based model trains on images only, by leveraging a joint
Text-Image multi-modal metric. Compared to baseline methods, our generations
achieve state of the art results both in human evaluations as well as with
perceptual scores when tested on a public multimodal dataset of natural images,
as well as on a collected dataset of 400 million Stickers.
