Noisy Text Data: Achilles' Heel of popular transformer based NLP models
  In the last few years, the ML community has created a number of new NLP
models based on transformer architecture. These models have shown great
performance for various NLP tasks on benchmark datasets, often surpassing SOTA
results. Buoyed with this success, one often finds industry practitioners
actively experimenting with fine-tuning these models to build NLP applications
for industry use cases. However, for most datasets that are used by
practitioners to build industrial NLP applications, it is hard to guarantee the
presence of any noise in the data. While most transformer based NLP models have
performed exceedingly well in transferring the learnings from one dataset to
another, it remains unclear how these models perform when fine-tuned on noisy
text. We address the open question by Kumar et al. (2020) to explore the
sensitivity of popular transformer based NLP models to noise in the text data.
We continue working with the noise as defined by them -- spelling mistakes &
typos (which are the most commonly occurring noise). We show (via experimental
results) that these models perform badly on most common NLP tasks namely text
classification, textual similarity, NER, question answering, text summarization
on benchmark datasets. We further show that as the noise in data increases, the
performance degrades. Our findings suggest that one must be vary of the
presence of noise in their datasets while fine-tuning popular transformer based
NLP models.
