Revisiting Uncertainty-based Query Strategies for Active Learning with
  Transformers
  Active learning is the iterative construction of a classification model
through targeted labeling, enabling significant labeling cost savings. As most
research on active learning has been carried out before transformer-based
language models ("transformers") became popular, despite its practical
importance, comparably few papers have investigated how transformers can be
combined with active learning to date. This can be attributed to the fact that
using state-of-the-art query strategies for transformers induces a prohibitive
runtime overhead, which effectively nullifies, or even outweighs the desired
cost savings. For this reason, we revisit uncertainty-based query strategies,
which had been largely outperformed before, but are particularly suited in the
context of fine-tuning transformers. In an extensive evaluation, we connect
transformers to experiments from previous research, assessing their performance
on five widely used text classification benchmarks. For active learning with
transformers, several other uncertainty-based approaches outperform the
well-known prediction entropy query strategy, thereby challenging its status as
most popular uncertainty baseline in active learning for text classification.
