Similarity Contrastive Estimation for Self-Supervised Soft Contrastive
  Learning
  Contrastive representation learning has proven to be an effective
self-supervised learning method. Most successful approaches are based on the
Noise Contrastive Estimation (NCE) paradigm and consider different views of an
instance as positives and other instances as noise that positives should be
contrasted with. However, all instances in a dataset are drawn from the same
distribution and share underlying semantic information that should not be
considered as noise. We argue that a good data representation contains the
relations, or semantic similarity, between the instances. Contrastive learning
implicitly learns relations but considers the negatives as noise which is
harmful to the quality of the learned relations and therefore the quality of
the representation. To circumvent this issue we propose a novel formulation of
contrastive learning using semantic similarity between instances called
Similarity Contrastive Estimation (SCE). Our training objective can be
considered as soft contrastive learning. Instead of hard classifying positives
and negatives, we propose a continuous distribution to push or pull instances
based on their semantic similarities. The target similarity distribution is
computed from weak augmented instances and sharpened to eliminate irrelevant
relations. Each weak augmented instance is paired with a strong augmented
instance that contrasts its positive while maintaining the target similarity
distribution. Experimental results show that our proposed SCE outperforms its
baselines MoCov2 and ReSSL on various datasets and is competitive with
state-of-the-art algorithms on the ImageNet linear evaluation protocol.
