The Power of Contrast for Feature Learning: A Theoretical Analysis
  Contrastive learning has achieved state-of-the-art performance in various
self-supervised learning tasks and even outperforms its supervised counterpart.
Despite its empirical success, theoretical understanding of why contrastive
learning works is still limited. In this paper, (i) we provably show that
contrastive learning outperforms autoencoder, a classical unsupervised learning
method, for both feature recovery and downstream tasks; (ii) we also illustrate
the role of labeled data in supervised contrastive learning. This provides
theoretical support for recent findings that contrastive learning with labels
improves the performance of learned representations in the in-domain downstream
task, but it can harm the performance in transfer learning. We verify our
theory with numerical experiments.
