Contrastive Principal Component Learning: Modeling Similarity by
  Augmentation Overlap
  Traditional self-supervised contrastive learning methods learn embeddings by
pulling views of the same sample together and pushing views of different
samples away. Since views of a sample are usually generated via data
augmentations, the semantic relationship between samples is ignored. Based on
the observation that semantically similar samples are more likely to have
similar augmentations, we propose to measure similarity via the distribution of
augmentations, i.e., how much the augmentations of two samples overlap. To
handle the dimensional and computational complexity, we propose a novel
Contrastive Principal Component Learning (CPCL) method composed of a
contrastive-like loss and an on-the-fly projection loss to efficiently perform
PCA on the augmentation feature, which encodes the augmentation distribution.
By CPCL, the learned low-dimensional embeddings theoretically preserve the
similarity of augmentation distribution between samples. Empirical results show
our method can achieve competitive results against various traditional
contrastive learning methods on different benchmarks.
