On Adversarial Examples for Biomedical NLP Tasks
  The success of pre-trained word embeddings has motivated its use in tasks in
the biomedical domain. The BERT language model has shown remarkable results on
standard performance metrics in tasks such as Named Entity Recognition (NER)
and Semantic Textual Similarity (STS), which has brought significant progress
in the field of NLP. However, it is unclear whether these systems work
seemingly well in critical domains, such as legal or medical. For that reason,
in this work, we propose an adversarial evaluation scheme on two well-known
datasets for medical NER and STS. We propose two types of attacks inspired by
natural spelling errors and typos made by humans. We also propose another type
of attack that uses synonyms of medical terms. Under these adversarial
settings, the accuracy of the models drops significantly, and we quantify the
extent of this performance loss. We also show that we can significantly improve
the robustness of the models by training them with adversarial examples. We
hope our work will motivate the use of adversarial examples to evaluate and
develop models with increased robustness for medical tasks.
