Data Determines Distributional Robustness in Contrastive Language Image Pre-training (CLIP)
Contrastively trained image-text models such as CLIP, ALIGN, and BASIC have
demonstrated unprecedented robustness to multiple challenging natural
distribution shifts. Since these image-text models differ from previous
training approaches in several ways, an important question is what causes the
large robustness gains. We answer this question via a systematic experimental
investigation. Concretely, we study five different possible causes for the
robustness gains: (i) the training set size, (ii) the training distribution,
(iii) language supervision at training time, (iv) language supervision at test
time, and (v) the contrastive loss function. Our experiments show that the more
diverse training distribution is the main cause for the robustness gains, with
the other factors contributing little to no robustness. Beyond our experimental
results, we also introduce ImageNet-Captions, a version of ImageNet with
original text annotations from Flickr, to enable further controlled experiments
of language-image training.