On Negative Interference in Multilingual Models: Findings and A
  Meta-Learning Treatment
  Modern multilingual models are trained on concatenated text from multiple
languages in hopes of conferring benefits to each (positive transfer), with the
most pronounced benefits accruing to low-resource languages. However, recent
work has shown that this approach can degrade performance on high-resource
languages, a phenomenon known as negative interference. In this paper, we
present the first systematic study of negative interference. We show that,
contrary to previous belief, negative interference also impacts low-resource
languages. While parameters are maximally shared to learn language-universal
structures, we demonstrate that language-specific parameters do exist in
multilingual models and they are a potential cause of negative interference.
Motivated by these observations, we also present a meta-learning algorithm that
obtains better cross-lingual transferability and alleviates negative
interference, by adding language-specific layers as meta-parameters and
training them in a manner that explicitly improves shared layers'
generalization on all languages. Overall, our results show that negative
interference is more common than previously known, suggesting new directions
for improving multilingual representations.
