Understanding Cross-Lingual Syntactic Transfer in Multilingual Recurrent
  Neural Networks
  It is now established that modern neural language models can be successfully
trained on multiple languages simultaneously without changes to the underlying
architecture. But what kind of knowledge is really shared among languages
within these models? Does multilingual training mostly lead to an alignment of
the lexical representation spaces or does it also enable the sharing of purely
grammatical knowledge? In this paper we dissect different forms of
cross-lingual transfer and look for its most determining factors, using a
variety of models and probing tasks. We find that exposing our LMs to a related
language does not always increase grammatical knowledge in the target language,
and that optimal conditions for lexical-semantic transfer may not be optimal
for syntactic transfer.
