Training Data Subset Search with Ensemble Active Learning
  Deep Neural Networks (DNNs) often rely on very large datasets for training.
Given the large size of such datasets, it is conceivable that they contain
certain samples that either do not contribute or negatively impact the DNN's
optimization. Modifying the training distribution in a way that excludes such
samples could provide an effective solution to both improve performance and
reduce training time. In this paper, we propose to scale up ensemble Active
Learning (AL) methods to perform acquisition at a large scale (10k to 500k
samples at a time). We do this with ensembles of hundreds of models, obtained
at a minimal computational cost by reusing intermediate training checkpoints.
This allows us to automatically and efficiently perform a training data subset
search for large labeled datasets. We observe that our approach obtains
favorable subsets of training data, which can be used to train more accurate
DNNs than training with the entire dataset. We perform an extensive
experimental study of this phenomenon on three image classification benchmarks
(CIFAR-10, CIFAR-100 and ImageNet), as well as an internal object detection
benchmark for prototyping perception models for autonomous driving. Unlike
existing studies, our experiments on object detection are at the scale required
for production-ready autonomous driving systems. We provide insights on the
impact of different initialization schemes, acquisition functions and ensemble
configurations at this scale. Our results provide strong empirical evidence
that optimizing the training data distribution can provide significant benefits
on large scale vision tasks.
