Towards Lingua Franca Named Entity Recognition with BERT
  Information extraction is an important task in NLP, enabling the automatic
extraction of data for relational database filling. Historically, research and
data was produced for English text, followed in subsequent years by datasets in
Arabic, Chinese (ACE/OntoNotes), Dutch, Spanish, German (CoNLL evaluations),
and many others. The natural tendency has been to treat each language as a
different dataset and build optimized models for each. In this paper we
investigate a single Named Entity Recognition model, based on a multilingual
BERT, that is trained jointly on many languages simultaneously, and is able to
decode these languages with better accuracy than models trained only on one
language. To improve the initial model, we study the use of regularization
strategies such as multitask learning and partial gradient updates. In addition
to being a single model that can tackle multiple languages (including code
switch), the model could be used to make zero-shot predictions on a new
language, even ones for which training data is not available, out of the box.
The results show that this model not only performs competitively with
monolingual models, but it also achieves state-of-the-art results on the
CoNLL02 Dutch and Spanish datasets, OntoNotes Arabic and Chinese datasets.
Moreover, it performs reasonably well on unseen languages, achieving
state-of-the-art for zero-shot on three CoNLL languages.
