Semantically-Conditioned Negative Samples for Efficient Contrastive
  Learning
  Negative sampling is a limiting factor w.r.t. the generalization of
metric-learned neural networks. We show that uniform negative sampling provides
little information about the class boundaries and thus propose three novel
techniques for efficient negative sampling: drawing negative samples from (1)
the top-$k$ most semantically similar classes, (2) the top-$k$ most
semantically similar samples and (3) interpolating between contrastive latent
representations to create pseudo negatives. Our experiments on CIFAR-10,
CIFAR-100 and Tiny-ImageNet-200 show that our proposed \textit{Semantically
Conditioned Negative Sampling} and Latent Mixup lead to consistent performance
improvements. In the standard supervised learning setting, on average we
increase test accuracy by 1.52\% percentage points on CIFAR-10 across various
network architectures. In the knowledge distillation setting, (1) the
performance of student networks increase by 4.56\% percentage points on
Tiny-ImageNet-200 and 3.29\% on CIFAR-100 over student networks trained with no
teacher and (2) 1.23\% and 1.72\% respectively over a \textit{hard-to-beat}
baseline (Hinton et al., 2015).
