Load What You Need: Smaller Versions of Multilingual BERT
  Pre-trained Transformer-based models are achieving state-of-the-art results
on a variety of Natural Language Processing data sets. However, the size of
these models is often a drawback for their deployment in real production
applications. In the case of multilingual models, most of the parameters are
located in the embeddings layer. Therefore, reducing the vocabulary size should
have an important impact on the total number of parameters. In this paper, we
propose to generate smaller models that handle fewer number of languages
according to the targeted corpora. We present an evaluation of smaller versions
of multilingual BERT on the XNLI data set, but we believe that this method may
be applied to other multilingual transformers. The obtained results confirm
that we can generate smaller models that keep comparable results, while
reducing up to 45% of the total number of parameters. We compared our models
with DistilmBERT (a distilled version of multilingual BERT) and showed that
unlike language reduction, distillation induced a 1.7% to 6% drop in the
overall accuracy on the XNLI data set. The presented models and code are
publicly available.
