Uncovering More Shallow Heuristics: Probing the Natural Language
  Inference Capacities of Transformer-Based Pre-Trained Language Models Using
  Syllogistic Patterns
  In this article, we explore the shallow heuristics used by transformer-based
pre-trained language models (PLMs) that are fine-tuned for natural language
inference (NLI). To do so, we construct or own dataset based on syllogistic,
and we evaluate a number of models' performance on our dataset. We find
evidence that the models rely heavily on certain shallow heuristics, picking up
on symmetries and asymmetries between premise and hypothesis. We suggest that
the lack of generalization observable in our study, which is becoming a topic
of lively debate in the field, means that the PLMs are currently not learning
NLI, but rather spurious heuristics.
