Is BERT Really Robust? A Strong Baseline for Natural Language Attack on
  Text Classification and Entailment
  Machine learning algorithms are often vulnerable to adversarial examples that
have imperceptible alterations from the original counterparts but can fool the
state-of-the-art models. It is helpful to evaluate or even improve the
robustness of these models by exposing the maliciously crafted adversarial
examples. In this paper, we present TextFooler, a simple but strong baseline to
generate natural adversarial text. By applying it to two fundamental natural
language tasks, text classification and textual entailment, we successfully
attacked three target models, including the powerful pre-trained BERT, and the
widely used convolutional and recurrent neural networks. We demonstrate the
advantages of this framework in three ways: (1) effective---it outperforms
state-of-the-art attacks in terms of success rate and perturbation rate, (2)
utility-preserving---it preserves semantic content and grammaticality, and
remains correctly classified by humans, and (3) efficient---it generates
adversarial text with computational complexity linear to the text length. *The
code, pre-trained target models, and test examples are available at
https://github.com/jind11/TextFooler.
