Investigating the Effectiveness of Representations Based on Pretrained
  Transformer-based Language Models in Active Learning for Labelling Text
  Datasets
  Active learning has been shown to be an effective way to alleviate some of
the effort required in utilising large collections of unlabelled data for
machine learning tasks without needing to fully label them. The representation
mechanism used to represent text documents when performing active learning,
however, has a significant influence on how effective the process will be.
While simple vector representations such as bag-of-words and embedding-based
representations based on techniques such as word2vec have been shown to be an
effective way to represent documents during active learning, the emergence of
representation mechanisms based on the pre-trained transformer-based neural
network models popular in natural language processing research (e.g. BERT)
offer a promising, and as yet not fully explored, alternative. This paper
describes a comprehensive evaluation of the effectiveness of representations
based on pre-trained transformer-based language models for active learning.
This evaluation shows that transformer-based models, especially BERT-like
models, that have not yet been widely used in active learning, achieve a
significant improvement over more commonly used vector representations like
bag-of-words or other classical word embeddings like word2vec. This paper also
investigates the effectiveness of representations based on variants of BERT
such as Roberta, Albert as well as comparing the effectiveness of the [CLS]
token representation and the aggregated representation that can be generated
using BERT-like models. Finally, we propose an approach Adaptive Tuning Active
Learning. Our experiments show that the limited label information acquired in
active learning can not only be used for training a classifier but can also
adaptively improve the embeddings generated by the BERT-like language models as
well.
