Cross-Lingual Ability of Multilingual BERT: An Empirical Study
  Recent work has exhibited the surprising cross-lingual abilities of
multilingual BERT (M-BERT) -- surprising since it is trained without any
cross-lingual objective and with no aligned data. In this work, we provide a
comprehensive study of the contribution of different components in M-BERT to
its cross-lingual ability. We study the impact of linguistic properties of the
languages, the architecture of the model, and the learning objectives. The
experimental study is done in the context of three typologically different
languages -- Spanish, Hindi, and Russian -- and using two conceptually
different NLP tasks, textual entailment and named entity recognition. Among our
key conclusions is the fact that the lexical overlap between languages plays a
negligible role in the cross-lingual success, while the depth of the network is
an integral part of it. All our models and implementations can be found on our
project page: http://cogcomp.org/page/publication_view/900 .
