Do More Negative Samples Necessarily Hurt in Contrastive Learning?
Recent investigations in noise contrastive estimation suggest, both
empirically as well as theoretically, that while having more "negative samples"
in the contrastive loss improves downstream classification performance
initially, beyond a threshold, it hurts downstream performance due to a
"collision-coverage" trade-off. But is such a phenomenon inherent in
contrastive learning? We show in a simple theoretical setting, where positive
pairs are generated by sampling from the underlying latent class (introduced by
Saunshi et al. (ICML 2019)), that the downstream performance of the
representation optimizing the (population) contrastive loss in fact does not
degrade with the number of negative samples. Along the way, we give a
structural characterization of the optimal representation in our framework, for
noise contrastive estimation. We also provide empirical support for our
theoretical results on CIFAR-10 and CIFAR-100 datasets.