Rebuilding Trust in Active Learning with Actionable Metrics
  Active Learning (AL) is an active domain of research, but is seldom used in
the industry despite the pressing needs. This is in part due to a misalignment
of objectives, while research strives at getting the best results on selected
datasets, the industry wants guarantees that Active Learning will perform
consistently and at least better than random labeling. The very one-off nature
of Active Learning makes it crucial to understand how strategy selection can be
carried out and what drives poor performance (lack of exploration, selection of
samples that are too hard to classify, ...).
  To help rebuild trust of industrial practitioners in Active Learning, we
present various actionable metrics. Through extensive experiments on reference
datasets such as CIFAR100, Fashion-MNIST, and 20Newsgroups, we show that those
metrics brings interpretability to AL strategies that can be leveraged by the
practitioner.
