Cost-Sensitive BERT for Generalisable Sentence Classification with
  Imbalanced Data
  The automatic identification of propaganda has gained significance in recent
years due to technological and social changes in the way news is generated and
consumed. That this task can be addressed effectively using BERT, a powerful
new architecture which can be fine-tuned for text classification tasks, is not
surprising. However, propaganda detection, like other tasks that deal with news
documents and other forms of decontextualized social communication (e.g.
sentiment analysis), inherently deals with data whose categories are
simultaneously imbalanced and dissimilar. We show that BERT, while capable of
handling imbalanced classes with no additional data augmentation, does not
generalise well when the training and test data are sufficiently dissimilar (as
is often the case with news sources, whose topics evolve over time). We show
how to address this problem by providing a statistical measure of similarity
between datasets and a method of incorporating cost-weighting into BERT when
the training and test sets are dissimilar. We test these methods on the
Propaganda Techniques Corpus (PTC) and achieve the second-highest score on
sentence-level propaganda classification.
