Intriguing Properties of Contrastive Losses
  We study three intriguing properties of contrastive learning. First, we
generalize the standard contrastive loss to a broader family of losses, and we
find that various instantiations of the generalized loss perform similarly
under the presence of a multi-layer non-linear projection head. Second, we
study if instance-based contrastive learning (with a global image
representation) can learn well on images with multiple objects present. We find
that meaningful hierarchical local features can be learned despite the fact
that these objectives operate on global instance-level features. Finally, we
study the phenomenon of feature suppression among competing features shared
across augmented views, such as "color distribution" vs "object class". We
construct datasets with explicit and controllable competing features, and show
that, for contrastive learning, a few bits of easy-to-learn shared features can
suppress, and even fully prevent, the learning of other sets of competing
features. In scenarios where there are multiple objects in an image, the
dominant object would suppress the learning of smaller objects. Existing
contrastive learning methods critically rely on data augmentation to favor
certain sets of features over others, and could suffer from learning saturation
for scenarios where existing augmentations cannot fully address the feature
suppression. This poses open challenges to existing contrastive learning
techniques.
