Energy-Based Contrastive Learning of Visual Representations
  Contrastive learning is a method of learning visual representations by
training Deep Neural Networks (DNNs) to increase the similarity between
representations of positive pairs and reduce the similarity between
representations of negative pairs. However, contrastive methods usually require
large datasets with significant number of negative pairs per iteration to
achieve reasonable performance on downstream tasks. To address this problem,
here we propose Energy-Based Contrastive Learning (EBCLR) that combines
contrastive learning with Energy-Based Models (EBMs) and can be theoretically
interpreted as learning the joint distribution of positive pairs. Using a novel
variant of Stochastic Gradient Langevin Dynamics (SGLD) to accelerate the
training of EBCLR, we show that EBCLR is far more sample-efficient than
previous self-supervised learning methods. Specifically, EBCLR shows from X4 up
to X20 acceleration compared to SimCLR and MoCo v2 in terms of training epochs.
Furthermore, in contrast to SimCLR, EBCLR achieves nearly the same performance
with 254 negative pairs (batch size 128) and 30 negative pairs (batch size 16)
per positive pair, demonstrating the robustness of EBCLR to small number of
negative pairs.
