It's All in the Heads: Using Attention Heads as a Baseline for
  Cross-Lingual Transfer in Commonsense Reasoning
  Commonsense reasoning is one of the key problems in natural language
processing, but the relative scarcity of labeled data holds back the progress
for languages other than English. Pretrained cross-lingual models are a source
of powerful language-agnostic representations, yet their inherent reasoning
capabilities are still actively studied. In this work, we design a simple
approach to commonsense reasoning which trains a linear classifier with weights
of multi-head attention as features. To evaluate this approach, we create a
multilingual Winograd Schema corpus by processing several datasets from prior
work within a standardized pipeline and measure cross-lingual generalization
ability in terms of out-of-sample performance. The method performs
competitively with recent supervised and unsupervised approaches for
commonsense reasoning, even when applied to other languages in a zero-shot
manner. Also, we demonstrate that most of the performance is given by the same
small subset of attention heads for all studied languages, which provides
evidence of universal reasoning capabilities in multilingual encoders.
