Beyond Separability: Analyzing the Linear Transferability of Contrastive
  Representations to Related Subpopulations
  Contrastive learning is a highly effective method for learning
representations from unlabeled data. Recent works show that contrastive
representations can transfer across domains, leading to simple state-of-the-art
algorithms for unsupervised domain adaptation. In particular, a linear
classifier trained to separate the representations on the source domain can
also predict classes on the target domain accurately, even though the
representations of the two domains are far from each other. We refer to this
phenomenon as linear transferability. This paper analyzes when and why
contrastive representations exhibit linear transferability in a general
unsupervised domain adaptation setting. We prove that linear transferability
can occur when data from the same class in different domains (e.g., photo dogs
and cartoon dogs) are more related with each other than data from different
classes in different domains (e.g., photo dogs and cartoon cats) are. Our
analyses are in a realistic regime where the source and target domains can have
unbounded density ratios and be weakly related, and they have distant
representations across domains.
