Towards the Generalization of Contrastive Self-Supervised Learning
  Recently, self-supervised learning has attracted great attention, since it
only requires unlabeled data for training. Contrastive learning is one popular
method for self-supervised learning and has achieved promising empirical
performance. However, the theoretical understanding of its generalization
ability is still limited. To this end, we define a kind of
$(\sigma,\delta)$-measure to mathematically quantify the data augmentation, and
then provide an upper bound of the downstream classification error based on the
measure. We show that the generalization ability of contrastive self-supervised
learning depends on three key factors: alignment of positive samples,
divergence of class centers, and concentration of augmented data. The first two
factors can be optimized by contrastive algorithms, while the third one is
priorly determined by pre-defined data augmentation. With the above theoretical
findings, we further study two canonical contrastive losses, InfoNCE and
cross-correlation loss, and prove that both of them are indeed able to satisfy
the first two factors. Moreover, we empirically verify the third factor by
conducting various experiments on the real-world dataset, and show that our
theoretical inferences on the relationship between the data augmentation and
the generalization of contrastive self-supervised learning agree with the
empirical observations.
