Contrastive Attraction and Contrastive Repulsion for Representation
  Learning
  Contrastive learning (CL) is effective in learning data representations
without label supervision, where the encoder needs to contrast each positive
sample over multiple negative samples via a one-vs-many softmax cross-entropy
loss. However, conventional CL is sensitive to how many negative samples are
included and how they are selected. Proposed in this paper is a doubly CL
strategy that contrasts positive samples and negative ones within themselves
separately. We realize this strategy with contrastive attraction and
contrastive repulsion (CACR) makes the query not only exert a greater force to
attract more distant positive samples but also do so to repel closer negative
samples. Theoretical analysis reveals the connection between CACR and CL from
the perspectives of both positive attraction and negative repulsion and shows
the benefits in both efficiency and robustness brought by separately
contrasting within the sampled positive and negative pairs. Extensive
large-scale experiments on standard vision tasks show that CACR not only
consistently outperforms existing CL methods on benchmark datasets in
representation learning, but also provides interpretable contrastive weights,
demonstrating the efficacy of the proposed doubly contrastive strategy.
