Contrastive Learning Inverts the Data Generating Process
  Contrastive learning has recently seen tremendous success in self-supervised
learning. So far, however, it is largely unclear why the learned
representations generalize so effectively to a large variety of downstream
tasks. We here prove that feedforward models trained with objectives belonging
to the commonly used InfoNCE family learn to implicitly invert the underlying
generative model of the observed data. While the proofs make certain
statistical assumptions about the generative model, we observe empirically that
our findings hold even if these assumptions are severely violated. Our theory
highlights a fundamental connection between contrastive learning, generative
modeling, and nonlinear independent component analysis, thereby furthering our
understanding of the learned representations as well as providing a theoretical
foundation to derive more effective contrastive losses.
