Task Adaptive Pretraining of Transformers for Hostility Detection
  Identifying adverse and hostile content on the web and more particularly, on
social media, has become a problem of paramount interest in recent years. With
their ever increasing popularity, fine-tuning of pretrained Transformer-based
encoder models with a classifier head are gradually becoming the new baseline
for natural language classification tasks. In our work, we explore the gains
attributed to Task Adaptive Pretraining (TAPT) prior to fine-tuning of
Transformer-based architectures. We specifically study two problems, namely,
(a) Coarse binary classification of Hindi Tweets into Hostile or Not, and (b)
Fine-grained multi-label classification of Tweets into four categories: hate,
fake, offensive, and defamation. Building up on an architecture which takes
emojis and segmented hashtags into consideration for classification, we are
able to experimentally showcase the performance upgrades due to TAPT. Our
system (with team name 'iREL IIIT') ranked first in the 'Hostile Post Detection
in Hindi' shared task with an F1 score of 97.16% for coarse-grained detection
and a weighted F1 score of 62.96% for fine-grained multi-label classification
on the provided blind test corpora.
