Understanding the Behaviour of Contrastive Loss
  Unsupervised contrastive learning has achieved outstanding success, while the
mechanism of contrastive loss has been less studied. In this paper, we
concentrate on the understanding of the behaviours of unsupervised contrastive
loss. We will show that the contrastive loss is a hardness-aware loss function,
and the temperature {\tau} controls the strength of penalties on hard negative
samples. The previous study has shown that uniformity is a key property of
contrastive learning. We build relations between the uniformity and the
temperature {\tau} . We will show that uniformity helps the contrastive
learning to learn separable features, however excessive pursuit to the
uniformity makes the contrastive loss not tolerant to semantically similar
samples, which may break the underlying semantic structure and be harmful to
the formation of features useful for downstream tasks. This is caused by the
inherent defect of the instance discrimination objective. Specifically,
instance discrimination objective tries to push all different instances apart,
ignoring the underlying relations between samples. Pushing semantically
consistent samples apart has no positive effect for acquiring a prior
informative to general downstream tasks. A well-designed contrastive loss
should have some extents of tolerance to the closeness of semantically similar
samples. Therefore, we find that the contrastive loss meets a
uniformity-tolerance dilemma, and a good choice of temperature can compromise
these two properties properly to both learn separable features and tolerant to
semantically similar samples, improving the feature qualities and the
downstream performances.
