ShufText: A Simple Black Box Approach to Evaluate the Fragility of Text
  Classification Models
  Text classification is the most basic natural language processing task. It
has a wide range of applications ranging from sentiment analysis to topic
classification. Recently, deep learning approaches based on CNN, LSTM, and
Transformers have been the de facto approach for text classification. In this
work, we highlight a common issue associated with these approaches. We show
that these systems are over-reliant on the important words present in the text
that are useful for classification. With limited training data and
discriminative training strategy, these approaches tend to ignore the semantic
meaning of the sentence and rather just focus on keywords or important n-grams.
We propose a simple black box technique ShutText to present the shortcomings of
the model and identify the over-reliance of the model on keywords. This
involves randomly shuffling the words in a sentence and evaluating the
classification accuracy. We see that on common text classification datasets
there is very little effect of shuffling and with high probability these models
predict the original class. We also evaluate the effect of language model
pretraining on these models and try to answer questions around model robustness
to out of domain sentences. We show that simple models based on CNN or LSTM as
well as complex models like BERT are questionable in terms of their syntactic
and semantic understanding.
