Universum-inspired Supervised Contrastive Learning
  Mixup is an efficient data augmentation method which generates additional
samples through respective convex combinations of original data points and
labels. Although being theoretically dependent on data properties, Mixup is
reported to perform well as a regularizer and calibrator contributing reliable
robustness and generalization to neural network training. In this paper,
inspired by Universum Learning which uses out-of-class samples to assist the
target tasks, we investigate Mixup from a largely under-explored perspective -
the potential to generate in-domain samples that belong to none of the target
classes, that is, universum. We find that in the framework of supervised
contrastive learning, universum-style Mixup produces surprisingly high-quality
hard negatives, greatly relieving the need for a large batch size in
contrastive learning. With these findings, we propose Universum-inspired
Contrastive learning (UniCon), which incorporates Mixup strategy to generate
universum data as g-negatives and pushes them apart from anchor samples of the
target classes. Our approach not only improves Mixup with hard labels, but also
innovates a novel measure to generate universum data. With a linear classifier
on the learned representations, our method achieves 81.68% top-1 accuracy on
CIFAR-100, surpassing the state of art by a significant margin of 5% with a
much smaller batch size, typically, 256 in UniCon vs. 1024 in SupCon using
ResNet-50.
