A Generalized Supervised Contrastive Learning Framework
  Based on recent remarkable achievements of contrastive learning in
self-supervised representation learning, supervised contrastive learning
(SupCon) has successfully extended the batch contrastive approaches to the
supervised context and outperformed cross-entropy on various datasets on
ResNet. In this work, we present GenSCL: a generalized supervised contrastive
learning framework that seamlessly adapts modern image-based regularizations
(such as Mixup-Cutmix) and knowledge distillation (KD) to SupCon by our
generalized supervised contrastive loss. Generalized supervised contrastive
loss is a further extension of supervised contrastive loss measuring
cross-entropy between the similarity of labels and that of latent features.
Then a model can learn to what extent contrastives should be pulled closer to
an anchor in the latent space. By explicitly and fully leveraging label
information, GenSCL breaks the boundary between conventional positives and
negatives, and any kind of pre-trained teacher classifier can be utilized.
ResNet-50 trained in GenSCL with Mixup-Cutmix and KD achieves state-of-the-art
accuracies of 97.6% and 84.7% on CIFAR10 and CIFAR100 without external data,
which significantly improves the results reported in the original SupCon (1.6%
and 8.2%, respectively). Pytorch implementation is available at
https://t.ly/yuUO.
