Cross-lingual Transfer of Monolingual Models
  Recent studies in zero-shot cross-lingual learning using multilingual models
have falsified the previous hypothesis that shared vocabulary and joint
pre-training are the keys to cross-lingual generalization. Inspired by this
advancement, we introduce a cross-lingual transfer method for monolingual
models based on domain adaptation. We study the effects of such transfer from
four different languages to English. Our experimental results on GLUE show that
the transferred models outperform the native English model independently of the
source language. After probing the English linguistic knowledge encoded in the
representations before and after transfer, we find that semantic information is
retained from the source language, while syntactic information is learned
during transfer. Additionally, the results of evaluating the transferred models
in source language tasks reveal that their performance in the source domain
deteriorates after transfer.
