On the Marginal Benefit of Active Learning: Does Self-Supervision Eat
  Its Cake?
  Active learning is the set of techniques for intelligently labeling large
unlabeled datasets to reduce the labeling effort. In parallel, recent
developments in self-supervised and semi-supervised learning (S4L) provide
powerful techniques, based on data-augmentation, contrastive learning, and
self-training, that enable superior utilization of unlabeled data which led to
a significant reduction in required labeling in the standard machine learning
benchmarks. A natural question is whether these paradigms can be unified to
obtain superior results. To this aim, this paper provides a novel algorithmic
framework integrating self-supervised pretraining, active learning, and
consistency-regularized self-training. We conduct extensive experiments with
our framework on CIFAR10 and CIFAR100 datasets. These experiments enable us to
isolate and assess the benefits of individual components which are evaluated
using state-of-the-art methods (e.g.~Core-Set, VAAL, simCLR, FixMatch). Our
experiments reveal two key insights: (i) Self-supervised pre-training
significantly improves semi-supervised learning, especially in the few-label
regime, (ii) The benefit of active learning is undermined and subsumed by S4L
techniques. Specifically, we fail to observe any additional benefit of
state-of-the-art active learning algorithms when combined with state-of-the-art
S4L techniques.
