Provable Stochastic Optimization for Global Contrastive Learning: Small
  Batch Does Not Harm Performance
  In this paper, we study contrastive learning from an optimization
perspective, aiming to analyze and address a fundamental issue of existing
contrastive learning methods that either rely on a large batch size or a large
dictionary. We consider a global objective for contrastive learning, which
contrasts each positive pair with all negative pairs for an anchor point. From
the optimization perspective, we explain why existing methods such as SimCLR
requires a large batch size in order to achieve a satisfactory result. In order
to remove such requirement, we propose a memory-efficient Stochastic
Optimization algorithm for solving the Global objective of Contrastive Learning
of Representations, named SogCLR. We show that its optimization error is
negligible under a reasonable condition after a sufficient number of iterations
or is diminishing for a slightly different global contrastive objective.
Empirically, we demonstrate that on ImageNet with a batch size 256, SogCLR
achieves a performance of 69.4% for top-1 linear evaluation accuracy using
ResNet-50, which is on par with SimCLR (69.3%) with a large batch size 8,192.
We also attempt to show that the proposed optimization technique is generic and
can be applied to solving other contrastive losses, e.g., two-way contrastive
losses for bimodal contrastive learning.
