Provable Guarantees for Self-Supervised Deep Learning with Spectral
  Contrastive Loss
  Recent works in self-supervised learning have advanced the state-of-the-art
by relying on the contrastive learning paradigm, which learns representations
by pushing positive pairs, or similar examples from the same class, closer
together while keeping negative pairs far apart. Despite the empirical
successes, theoretical foundations are limited -- prior analyses assume
conditional independence of the positive pairs given the same class label, but
recent empirical applications use heavily correlated positive pairs (i.e., data
augmentations of the same image). Our work analyzes contrastive learning
without assuming conditional independence of positive pairs using a novel
concept of the augmentation graph on data. Edges in this graph connect
augmentations of the same data, and ground-truth classes naturally form
connected sub-graphs. We propose a loss that performs spectral decomposition on
the population augmentation graph and can be succinctly written as a
contrastive learning objective on neural net representations. Minimizing this
objective leads to features with provable accuracy guarantees under linear
probe evaluation. By standard generalization bounds, these accuracy guarantees
also hold when minimizing the training contrastive loss. Empirically, the
features learned by our objective can match or outperform several strong
baselines on benchmark vision datasets. In all, this work provides the first
provable analysis for contrastive learning where guarantees for linear probe
evaluation can apply to realistic empirical settings.
