Should we Stop Training More Monolingual Models, and Simply Use Machine
  Translation Instead?
  Most work in NLP makes the assumption that it is desirable to develop
solutions in the native language in question. There is consequently a strong
trend towards building native language models even for low-resource languages.
This paper questions this development, and explores the idea of simply
translating the data into English, thereby enabling the use of pretrained, and
large-scale, English language models. We demonstrate empirically that a large
English language model coupled with modern machine translation outperforms
native language models in most Scandinavian languages. The exception to this is
Finnish, which we assume is due to inferior translation quality. Our results
suggest that machine translation is a mature technology, which raises a serious
counter-argument for training native language models for low-resource
languages. This paper therefore strives to make a provocative but important
point. As English language models are improving at an unprecedented pace, which
in turn improves machine translation, it is from an empirical and environmental
stand-point more effective to translate data from low-resource languages into
English, than to build language models for such languages.
