{'fraction_non_alphanumeric': 0.03219827742849875, 'fraction_numerical': 0.013773303775847658, 'mean_word_length': 5.220162748643761, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Cross-Lingual Language Model Meta-Pretraining

The success of pretrained cross-lingual language models relies on two essential abilities, i.e., generalization ability for learning downstream tasks in a source language, and cross-lingual transferability for transferring the task knowledge to other languages. However, current methods jointly learn the two abilities in a single-phase cross-lingual pretraining process, resulting in a trade-off between generalization and cross-lingual transfer. In this paper, we propose cross-lingual language model meta-pretraining, which learns the two abilities in different training phases. Our method introduces an additional meta-pretraining phase before cross-lingual pretraining, where the model learns generalization ability on a large-scale monolingual corpus. Then, the model focuses on learning cross-lingual transfer on a multilingual corpus. Experimental results show that our method improves both generalization and cross-lingual transfer, and produces better-aligned representations across different languages.


Introduction
Pretraining-finetuning language models (LMs) has become an emerging paradigm in natural language processing (NLP) (Bommasani et al., 2021). Nonetheless, most resources like training data are English-centric, making it hard to apply this paradigm to other languages. Recently, pretrained cross-lingual LMs have shown to be effective for cross-lingual transfer, and become a promising way to build multilingual NLP applications (Conneau and Lample, 2019;Conneau et al., 2020;Chi et al., 2021b).
Cross-lingual LMs are typically pretrained with the unsupervised language modeling tasks on an unlabeled multilingual text corpus (Devlin et al., 2019;Conneau et al., 2020;Xue et al., 2021). The pretrained cross-lingual LMs achieve zero-shot cross-lingual transfer, which enables the models cross-lingual transfer generalization task performance (source) task performance (target) cross-lingual transferability Figure 1: Cross-lingual pretraining methods typically employ a single-phase cross-lingual pretraining process, resulting in a trade-off between generalization and cross-lingual transfer.
to be finetuned on the downstream task training data of one source language but directly applied to other target languages (Wu and Dredze, 2019;K et al., 2020). Pretrained cross-lingual LMs can be further improved by utilizing cross-lingual supervisions from parallel corpora (Conneau and Lample, 2019;Chi et al., 2021b). Moreover, tokenlevel alignments are also proven to be beneficial for cross-lingual transfer (Cao et al., 2020;Chi et al., 2021c).
The success of pretrained cross-lingual LMs relies on two essential abilities. The first ability is the generalization ability, which helps the model to learn downstream tasks in a source language. The second ability is cross-lingual transferability that transfers the downstream task knowledge to target languages. However, current methods jointly learn the two abilities in a single-phase cross-lingual pretraining process, resulting in a trade-off between generalization and cross-lingual transfer. The trade-off is typically achieved by a rebalanced language distribution (Conneau and Lample, 2019) that controls the frequency of a language to be used in cross-lingual pretraining. As shown in Figure 1, sampling the source language more frequently encourages generalization to downstream tasks, which provides better results on the source language. Nonetheless, it is accompanied by the weakening of cross-lingual transferability, resulting in the low performance on target languages. On the contrary, increasing the probabilities of the target languages improves crosslingual transfer but harms generalization.
In this paper, we propose cross-lingual language model meta-pretraining (METAXLM), a novel paradigm for learning cross-lingual LMs that learns generalization ability and cross-lingual transferability in different training phases. Our method introduces an additional meta-pretraining phase before cross-lingual pretraining, which can be understood as the pretraining of pretraining. Specifically, METAXLM first learns generalization ability on a large-scale monolingual corpus in the meta-pretraining phase, and then focuses on learning cross-lingual transfer on a multilingual corpus in the cross-lingual pretraining phase. Finally, the model is finetuned on downstream tasks for various application scenarios of pretrained cross-lingual LMs.
We conduct extensive experiments on ten downstream tasks under three different application scenarios, including cross-lingual understanding tasks for cross-lingual transfer, multilingual classification for supervised finetuning, and crosslingual alignment tasks for feature-based applications. Experimental results show that our method provides substantial improvements over the baseline in all of the three application scenarios. Moreover, comprehensive analysis shows that our method successfully breaks the trade-off of the single-phase pretraining, which improves both generalization and cross-lingual transfer, and produces better-aligned representations at both sentence level and word level.
Our contributions can be summarized as follows: • We propose cross-lingual language model meta-pretraining, a novel paradigm for learning cross-lingual language models.
• We conduct extensive experiments on ten downstream tasks, and demonstrate the effectiveness of our method in three application scenarios.
• We show that our method improves both generalization and cross-lingual transfer, and produces better-aligned representations across languages, by only using additional monolingual data.

Related Work
Pretrained cross-lingual language models Cross-lingual language model pretraining is first proposed by Conneau and Lample (2019), which aims to build universal cross-lingual encoders. They show that learning masked language modeling (MLM; Devlin et al. 2019) on multilingual text improves the results on cross-lingual classification. Similarly, MBERT (Devlin et al., 2019) learns a multilingual version of BERT, which is also proven to be effective for cross-lingual transfer (Wu and Dredze, 2019;Pires et al., 2019;K et al., 2020;Zheng et al., 2021). Several studies demonstrate that pretraining cross-lingual language models at scale leads to significant performance gains (Conneau et al., 2020;Xue et al., 2021). Besides, Conneau and Lample (2019) introduce parallel corpora into cross-lingual pretraining, and show that the cross-lingual transferability can be improved by using parallel corpora during pretraining. Recent studies also present various objectives that leverage parallel corpora to improve cross-lingual LM pretraining (Huang et al., 2019;Chi et al., 2021b;Wei et al., 2021;Kale et al., 2021;Chi et al., 2021a,d). Moreover, the word alignments implied in parallel corpora are also proven to be beneficial for cross-lingual transfer Cao et al., 2020;Chi et al., 2021c). Additionally, several pretrained cross-lingual models are designed for producing cross-lingual sentence embeddings (Artetxe and Schwenk, 2019;, cross-lingual natural language generation (Liu et al., 2020;Lin et al., 2020;Ren et al., 2021), or the cross-lingual transfer of natural language generation (Chi et al., 2020;Maurya et al., 2021;. Cross-lingual transferability of monolingual models Artetxe et al. (2020) demonstrate the Transformer body of a pretrained monolingual LM can also serve as a LM for another language. Similarly, Reimers and Gurevych (2020) propose a transferring framework that extends a monolingual model to new languages using parallel data.  propose a knowledge distillation method, which leverages parallel corpora to extend an existing sentence embedding model to new  languages. de Souza et al. (2021) show that a pretrained monolingual LM performs well when directly finetuning it on a new language. However, these studies focus on extending an existing monolingual model to new languages, while our goal is to pretrain a cross-lingual LM that performs crosslingual transfer and produces universal representations. We provide related analyses in Section 5.3 and 5.4. Figure 2 illustrates the overview of our method, which contains three phases: meta-pretraining, cross-lingual pretraining, and model finetuning.

Method
In this section, we first introduce the metapretraining phase and cross-lingual pretraining phase. Then, we present three application scenarios of cross-lingual LMs for model finetuning.

Meta-Pretraining
Meta-pretraining can be understood as the pretraining of pretraining. A pretrained model typically provides good initialization for downstream task models. Similarly, a meta-pretrained model provides initialization for the next model pretraining phase. In the meta-pretraining phase, we pretrain a monolingual language model on a large-scale unlabeled corpus. Following Liu et al.
(2019), we use a Transformer (Vaswani et al., 2017) encoder as the backbone network, and pretrain the model with the masked language modeling (MLM; Devlin et al. 2019) task. The MLM task is to predict the masked words of an input text sequence. Each text sequence contains several sentences that consist of at most 512 to-kens. Note that a pretrained monolingual language model such as RoBERTa (Liu et al., 2019) can also be directly used as a meta-pretrained model.

Cross-Lingual Pretraining
With the meta-pretrained model, the cross-lingual pretraining phase focuses on learning crosslingual transfer. We pretrain a cross-lingual language model on a multilingual unlabeled text corpus with the MLM task. For each batch, we employ the rebalanced language sampling distribution (Conneau and Lample, 2019) to increase the probabilities of low-resource languages. Formally, considering a n-language multilingual text corpus with a size of N i for the i-th language, the rebalanced probability of the i-th language is where α controls the rebalanced distribution.
We use the same model architecture with the meta-pretrained model so that the parameters of the Transformer body can be directly initialized. For the vocabulary, following Conneau et al.
(2020), we utilize a shared vocabulary across different languages. Thus, the resulting vocabulary is different from the meta-pretrained model. To alleviate this issue, we employ a vocabulary matching strategy. For each word in the cross-lingual vocabulary, we first directly search the word in the monolingual vocabulary, and then search the normalized word if the original one is not found.
Note that only a small part of words can be initialized because most of the words in the vocabulary are in different languages with the metapretrained model. An extension of the mapping strategy is to initialize more word embeddings with easily accessible bilingual dictionaries. The method is straightforward, just first searching the word translation in the dictionaries, and then applying the mapping strategy mentioned above. Using bilingual dictionaries is not strictly an unsupervised setting, but it is an intuitive way to utilize more knowledge from the meta-pretrained model.

Model Finetuning
The pretrained cross-lingual LM can be applied to a wide range of downstream tasks, where the word prediction layer is replaced with various task layers. Following Devlin et al. (2019), we apply the same task layer structures with BERT for the downstream tasks. Besides, we present three application scenarios of the pretrained model as follows.
Cross-lingual transfer is a basic application of pretrained cross-lingual LMs. In this scenario, LMs are finetuned with training data in a source language but evaluated in various target languages. When the source language is different from the target languages, the setting is also known as zeroshot cross-lingual transfer.
Supervised finetuning is a common application scenario for monolingual pretrained LMs. In this scenario, a pretrained cross-lingual LM is directly used as a monolingual model for a specific target language.
Feature-based applications directly use the produced contextualized sentence representations without model finetuning. For example, the extracted representations can be used for crosslingual sentence retrieval and word alignment (Hu et al., 2020;Jalili Sabet et al., 2020;Chi et al., 2021c).

Experiments
In this section, we present extensive experiments on various downstream tasks. In specific, we first evaluate the models on natural language understanding (NLU) tasks for both the cross-lingual transfer and the supervised finetuning scenarios. The two scenarios evaluate cross-lingual transfer and generalization, respectively. Then, we evaluate the models for feature-based applications on the cross-lingual alignment tasks at both sentence level and word level.

Setup
Data We use multilingual raw sentences extracted from Wikipedia dumps as the training data, including 67.4GB unlabeled text in 94 languages. The sentences are split into subword pieces by sentencepiece (Kudo and Richardson, 2018) using the vocabulary with 250K subwords, which is provided by XLM-R (Conneau et al., 2020). For bilingual dictionaries, we use the MUSE dictionaries (Lample et al., 2018) that consist of 45 Englishcentric bilingual lexicons. More details about the training data are in Appendix.
Training The backbone of METAXLM is a base-size Transformer (Vaswani et al., 2017)  Baseline We compare our proposed method with the single-phase cross-lingual pretraining process, i.e., pretraining the language models from scratch. We use the unsupervised XLM (Conneau and Lample, 2019) model as the baseline model, which learns the MLM task on a multilingual unlabeled text corpus. We reimplement XLM under the same pretraining setup with METAXLM for a fair comparison, i.e. pretraining with the same pretraining task, batch size, steps, learning rates, random seeds, etc. It means that METAXLM and XLM only differ in whether to use a metapretraining phase. Note that our models are pretrained with significantly fewer training data and training steps than the current state-of-the-art pretrained models because of the GPU limitation.

Cross-lingual Understanding
We evaluate METAXLM on XTREME (Hu et al., 2020) cross-lingual understanding tasks under the cross-lingual transfer setting. We use seven XTREME datasets that provide training sets in English, and dev/test sets in multiple languages. The datasets are (1) (7) the PAWS-X (Yang et al., 2019) dataset for paraphrase adversary. Table 1 presents the evaluation results on XTREME cross-lingual understanding. For each task, the presented results are first averaged over all the test languages, and then averaged over five runs with different random seeds. The detailed results on all test languages can be found in Appendix. It can be observed that METAXLM outperforms XLM on six out of seven downstream tasks, improving the average XTREME score from 49.2 to 55.7. It is worth mentioning that METAXLM only uses more English training data than the baseline model, but it still improves the downstream task performance for not only English but also other languages. It demonstrates that our method successfully utilizes the pretrained monolingual LM to learn a better cross-lingual LM, which provides substantial gains on downstream cross-lingual understanding tasks. Besides, it can be found that simply using bilingual dictionaries for initialization provides further gains over METAXLM, improving the average score from 55.7 to 59.3. On XNLI and PAWS-X, METAXLM+Dict even outperforms MBERT, which is pretrained with significantly a larger batch size and more training steps (Devlin et al., 2019).

Multilingual Classification
We investigate whether our method is effective for the supervised finetuning scenario. Under this setting, the models use the same language for both finetuning and testing. Thus, this setting evaluates whether our method improves generalization for the languages other than the meta-pretraining language. We use the multilingual Amazon Reviews dataset (Prettenhofer and Stein, 2010) for evaluating multilingual classification. The dataset consists of Amazon reviews written in four languages that are collected from the domains of Book, DVD, and Music. Each review is assigned with a label of 'positive' or 'negative'.  gual meta-pretrained model, which improves the generalization for other languages. We also see that using bilingual dictionaries also improves the results over METAXLM in the supervised finetuning scenario, which is consistent with the crosslingual transfer scenario.

Cross-Lingual Alignment
To evaluate whether our method encourages the alignment of the representations from different languages, we conduct experiments on the crosslingual sentence retrieval task and the word alignment task for cross-lingual alignment at sentence level and word level, respectively.
Cross-lingual sentence retrieval The crosslingual sentence retrieval task has been used for evaluating the cross-lingual sentence representations produced by the pretrained cross-lingual LMs (Hu et al., 2020). The goal of the task is to extract translation pairs from a bilingual document. Following Chi et al. (2021b), we use parallel corpora in 36 English-centric language pairs from the Tatoeba (Artetxe and Schwenk, 2019) as the test sets. The sentences representations are the average of the hidden vectors. Then, the corresponding sentence pairs can be extracted by the nearest neighbor searching method using the cosine similarity as the distance measure. Figure 3 reports the averaged accuracy@1 scores on Tatoeba where the sentences representations are obtained from different layers. Surprisingly, METAXLM greatly improves the retrieval accuracy over the baseline model, showing that our model produces more similar sentence representations for the translation pairs than the baseline model. It indicates that our method produces universal sentence representations that are betteraligned while only using additional unlabeled English text.
Word alignment We evaluate the word-level representations with the word alignment task. The goal of the word alignment task is to extract the corresponding word pairs given an input translation pair. We use the optimal-transport alignment method (Chi et al., 2021c) for the task. The method uses the hidden vectors as word representations, and produces the resulting aligned word pairs with optimal transport. Following Jalili Sabet et al. (2020), we use parallel corpora with golden alignments as test sets, which contain 1,244 translation pairs collected from EuroParl 1 , WPT2003 2 , and WPT2005 3 . The resulting word pairs are evaluated by the alignment error rate (AER; Och and Ney 2003). A lower AER score indicates the word representations produces by the model are better-aligned across different languages. Figure 4 illustrates the AER scores over different layers of the pretrained models, where layer-0 stands for the word embedding layer. It can be observed that METAXLM produces the word alignments with greatly lower AER scores than XLM on all layers except the embedding layer. The results are consistent with the results of sen- tence retrieval, demonstrating that our model produces better-aligned representations at both sentence level and word level. Furthermore, we observe that both METAXLM and XLM obtain the best performance around layer-8, while the performance of METAXLM is more stable than XLM.

Effects of Meta-Pretraining
To explore how meta-pretraining improves the cross-lingual pretraining, we analyze the crosslingual effectiveness of XLM and METAXLM at various pretraining steps. We evaluate the pretrained models with the pretraining steps ranging from 2K to 40K on the MLQA cross-lingual question answering. Note that the models are pretrained with different steps but finetuned under the same setup. Figure 5 illustrates the model performance curves along with the cross-lingual pretraining, including (a) the F1 scores averaged over the six non-English MLQA dev sets, (b) the F1 scores on the English MLQA dev set, and (c) the cross-lingual transfer gap (Hu et al., 2020). The cross-lingual transfer gap measures the difference between the score in English and the average score in the other languages. For a certain performance of English, the reduction of gap score indicates the growth of cross-lingual transferability.

Lagging phenomenon of cross-lingual transfer
We first analyze the performance curves of XLM to study the mechanism of the single-phase crosslingual pretraining. From Figure 5 (a) and (b), we observe that the results on both English and the other languages all remain at a low level in the first 14K steps. Although the performance of XLM on English has shown a rapid improvement after the 14K step, the performance on non-English sets shows a similar trend till the 20K step. In Figure 5 (c), the transfer gap of XLM has a fast increase from the 14K step to the 20K step, while remaining a relatively stable transfer gap after the 20K step. The above observations suggest a lagging phenomenon of cross-lingual transfer in a single-phase pretraining process, i.e., the learning of cross-lingual transfer lags behind the learning of generalization. The observations are also consistent with (Dufter and Schütze, 2020).
METAXLM vs. XLM Comparing to XLM, the results of METAXLM on non-English dev sets keep rising at the very beginning. Different from XLM starting with a low transfer gap, METAXLM starts with a high transfer gap but continually reduces the gap during training. For the results on the English dev set, METAXLM remains a high performance because of the meta-pretraining on English data. Combining the observations, it shows that our meta-pretraining method improves the cross-lingual transfer in two aspects. First, the meta-pretrained model has already obtained very good generalization ability, which enables the model to focuses on learning cross-lingual transfer. Besides, the meta-pretrained model provides good initialization for learning pretraining tasks on other languages, leading to better generalization for not only the meta-pretraining language, but also the other languages.

Effects of Transfer Direction
In addition to English-source cross-lingual transfer, we explore the cross-lingual transferability of   METAXLM for other transfer directions. The pretrained models are finetuned on Amazon Reviews under the cross-lingual transfer setting. Specifically, we use the four languages as the source languages respectively, and test the models on all the four languages. Figure 6 shows the results of the pretrained models for 16 transfer directions. We see that the transferability of the pretrained models varies among different transfer directions. METAXLM obtains consistent gains for 15 out of 16 transfer directions, indicating that using English as meta-pretraining language is beneficial for not only English-source cross-lingual transfer but also other transfer directions. Moreover, the transfer between similar languages such as English-French can be more effective than the dissimilar directions such as German-Japanese.

Meta-Pretraining vs. Knowledge Distillation
Inspired by  that uses knowledge distillation (KD; Hinton et al. 2015) to improve resulting multilingual sentence embeddings. We explore whether KD is beneficial for crosslingual LM pretraining. We pretrain a XLM model with an auxiliary KD task, where the English RoBERTa-base (Liu et al., 2019)   uation results of KD. XLM+KD only obtains slight improvements over XLM. Compared with METAXLM, XLM+KD uses more computational cost, with one more network forwarding for each step, while obtaining much lower results. Thus, our method is a more practical way to utilize monolingual knowledge to improve cross-lingual pretraining.

Effects of Parameter Freezing
We investigate whether freezing the initialized meta-pretrained parameters improves the cross-lingual effectiveness. Inspired by Mono-Trans (Artetxe et al., 2020), which freezes the Transformer body to extend an English BERT to a new language, we pretrain a variant of METAXLM with the Transformer body parameters frozen, i.e., only updating the embedding layers and the language modeling head during the pretraining phase.
The evaluation results are shown in Table 4.
It shows an interesting fact that METAXLM+freezing outperforms XLM on three of the downstream tasks, while preserving a reasonable high performance on the other two tasks. It indicates that the Transformer body of the meta-pretrained monolingual LM can also serve as a Transformer body of a cross-lingual LM. This finding is also consistent with the findings in MonoTrans. Despite the cross-lingual transferability of the Transformer body, freezing the body does not perform better than the standard METAXLM on the downstream tasks. Thus, it is a better choice to update all parameters during the cross-lingual pretraining phase.

Effects of Initialization
We conduct experiments to study the effects of initializing different components of the model. After the meta-pretraining phase, we pretrain several variants of METAXLM where we initialize the Transformer body only, the word embeddings only, or both.  results on cross-lingual understanding. It shows that the model produces the best results when both the Transformer body and embedding layers are initialized. Differently, initializing only one of the components can provide slight improvements on TyDiQA and PAWS-X, but harms the results on the other three tasks. This suggests that the joint initialization of the two components encourages the model to extract languageinvariant knowledge, which facilitates the learning of cross-lingual LMs. On the contrary, extracting language-invariant knowledge can be difficult when only one of the components is initialized.

Conclusion
In this paper, we propose cross-lingual language model meta-pretraining, which provides a novel paradigm for learning cross-lingual language models. Extensive experiments on ten downstream tasks demonstrate the effectiveness of our method in three application scenarios. Moreover, we show that our method improves both generalization and cross-lingual transfer, and produces universal representations that are betteraligned at both sentence level and word level, by only using additional monolingual data. Nonetheless, we should point out that our method introduces an additional meta-pretraining phase, which potentially requires more computational resources. Fortunately, pretrained monolingual models are currently easily accessible, which can be directly used as meta-pretrained models. Besides, the meta-pretraining phase is independent of the cross-lingual pretraining phase. Thus, our method can also be easily applied to other pretrained cross-lingual models. For future work, we would like to apply our method to language model pretraining at a larger scale. Exploring crosslingual language model meta-pretraining for natural language generation is also an interesting research topic.

A Pretraining Data
Following Chi et al. (2021c), we sample raw sentences from Wikipedia dump 4 using a rebalanced language sampling distribution (Conneau and Lample, 2019) with α = 0.7. Table 6 shows the statistics of Wikipedia dump we use.   Table 7 shows the hyperparameters for crosslingual pretraining. Both the baseline model and our model are pretrained using the same hyperparameters. Table 8 shows the hyperparameters for model finetuning.