{'fraction_non_alphanumeric': 0.0344882707908724, 'fraction_numerical': 0.016464852335851016, 'mean_word_length': 4.651309846431798, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Cross-lingual Adaption Model-Agnostic Meta-Learning for Natural Language Understanding

Meta learning with auxiliary languages has demonstrated promising improvements for cross-lingual natural language processing. However, previous studies sample the meta-training and meta-testing data from the same language, which limits the ability of the model for cross-lingual transfer. In this paper, we propose XLA-MAML, which performs direct cross-lingual adaption in the meta-learning stage. We conduct zero-shot and few-shot experiments on Natural Language Inference and Question Answering. The experimental results demonstrate the effectiveness of our method across different languages, tasks, and pretrained models. We also give analysis on various cross-lingual specific settings for meta-learning including sampling strategy and parallelism.


Introduction
There are around 7,000 languages spoken around the world, while most natural language processing (NLP) studies only consider English with largescale training data. Such setting limits the ability of these NLP techniques when sufficient labeled data is lacking for effective finetuning. To address this problem and extend the global reach of NLP, recent research of Cross-lingual Natural Language Understanding (XL-NLU) focuses on multi-lingual pretrained representations such as multi-lingual BERT (mBERT), which cover more than 100 languages. These cross-lingual pretrained models allow zeroshot or few-shot transfer among languages for specific tasks, where a model for a task is trained in monolingual English data and directly applied to a target low resource language. With the support of strong multi-lingual representations, the further transfer among languages of XL-NLU could be formalized similar to the few-shot learning task.
Meta learning, or learning to learn, aims to create models that can learn new skills or adapt to new tasks rapidly from few training examples, which has demonstrated promising result for fewshot learning. Model-Agnostic Meta-Learning (MAML) (Finn et al., 2017) has been recently shown beneficial for various few-shot learning and cross-domain learning NLP tasks. The algorithm allows fast adaptation to a new task using only a few data points.
From MAML perspective, we can re-formulate cross-lingual transfer task as shown in Figure 1. For model f θ with parameters θ, we can consider the task distribution P(τ ) formed by drawing from different auxiliary language bags of data samples {(x, y) l }. For meta training, we simulate the crosslingual transfer process and sample a meta dataset {S, Q} from different bags {(x, y) l }. The support set S simulates the transfer source training data and updates the model parameters to θ i , which is called the inner step. Then the query set Q simulates the transfer target testing data, model parameters are trained by optimizing for the performance of f (θ ) with respect to θ across tasks sampled from the target bags {(x, y) l }, which is referred as the meta step. The algorithm minimize the prediction error on target language given a small support set for learning. Nooralahzadeh et al. (2020) proposed a metalearning based cross-lingual transfer model, which uses data of non-target low-resource languages as auxiliary data. However, they sample the inner tasks and meta-update tasks from the same auxiliary language, that the meta learning cannot perform transfer for cross-lingual adaption. In this paper, to address this problem, we propose Cross-lingual Adaption Model-Agnostic Meta-Learning (XLA-MAML), which performs crosslingual adaption in the meta learning stage to transfer knowledge across different languages.
In conclusion, our contribution is 3-fold: • We propose a cross-lingual adaption metalearning approach, which directly performs meta learning adaption across different languages, while previous studies sample the support and query set from the same language.
• We conduct zero-shot and few-shot experiments on two novel benchmarks of XL-NLU, XNLI and MLQA. Experimental results demonstrate the effectiveness of our model compared with previous MAML studies, especially under zero-shot settings.
• We analyze the XLA-MAML performance under a number of cross-lingual meta learning settings, including sampling strategy and parallelism. The additional results on XLM-R suggest that XLA-MAML benefits even with a stronger encoder.

Meta Learning
Meta-learning algorithms train over a variety of learning tasks τ including potentially unseen tasks on a distribution of tasks P(τ ) and optimized for the best performance of a base-learner model f θ with parameters θ. Each task τ is associated with a dataset sampleD, containing both feature vectors {x} and labels {y}. The optimal model parameters are: Where one dataset D is considered as one data sample. The dataset sample D consists of a support set S and a query set Q, where the support set S is used for meta-training which simulates fast adaptation and the query set Q is used for metatesting that evaluates performance and compute a loss with respect to model parameter initialization. A loop of meta-training to update the parameter initialization and meta-testing to update the loss is called an episode. Meta learning consists of updating the parameters of the base-learner f θ by episode loops until it reaches stop criterion. The optimal model parameters in Equation 1 could be rewritten as: There are three common approaches of meta-learning: metric-based, model-based, and optimization-based. Each approach uses different ways to model P θ (y|x). Metric based meta learning methods predict probability over a set of known labels y is a weighted sum of labels of support set samples. The weight is generated by a kernel function k θ , measuring the similarity between two data samples, by modeling (Koch et al., 2015;Sung et al., 2018;Vinyals et al., 2016;Snell et al., 2017). Model based meta learning methods depend on a model designed specifically for fast learning, which updates its parameters rapidly with a few training steps. This rapid parameter update can be achieved by its internal architecture or controlled by another meta-learner model. Memory-Augmented Neural Networks (Santoro et al., 2016) uses external memory storage to facilitate the learning process of neural networks. Meta Network (Munkhdalai and Yu, 2017) uses fast weights, which predicts the parameters of another neural network and the generated weights.
In this work we focus on optimisation-based method MAML that optimisation-based methods are compatible with any model that learns through gradient descent. Optimized based adjust the optimization algorithm so that the model can perform fast adaption (Ravi and Larochelle, 2017;Finn et al., 2017;Nichol et al., 2018). Formally, given a task τ i associated with a dataset sample D = {S, Q}, MAML optimizes the parameters towards the optimal target in Equation 2 via two steps. In the meta-training step, or inner step for short, the algorithm updates the parameters to θ on the support set S. The updated parameters θ is computed by loss L (0) and gradient descent steps on S with an update step size α, The meta-testing step, or meta step for short, takes in the updated model f (θ ) and optimizes the parameters θ based on the loss L (1) of f (θ ) across Q. While the objective is computed using the updated model parameters θ , the meta-optimization Compute adapted parameters with gradient descent: Perform zero-shot or few-shot evaluation on target language test set {xtgt}; is performed over the model parameters θ, that the optimal model parameters are, The model parameters θ are updated with a metaupdate step size β, 3 Cross-lingual Adaption Model-Agnostic Meta-Learning

Meta Task Formulation
Cross-lingual Natural Language Understanding aims to transfer knowledge from high resource languages to target low resource languages, that the meta task formulation of MAML needs to adapt to such changes. To perform cross-lingual adaption, we no longer randomly form meta learning tasks τ i from one distribution D = {(x, y)}, where D stands for the available feature vectors x and ground truth labels y pairs. We split D into {D l i } l i ∈L by language l of language set L, and form the associated dataset D of meta learning tasks τ i as D = (S, Q) with the following two steps: 1. Sample two subset of languages for support set S and query set Q: 2. Sample support set S with K data-points and query set Q with N data points from language subsets of D: Where the language subset size is fixed as a hyperparameter. Thus the sampled tasks τ i can simulate the process of cross-lingual transfer and allow the meta learning perform cross-lingual fast adaption.
Due to the characteristics of low-resource crosslingual transfer, we further explore two settings: • The available resource amount for the crosslingual transfer is relatively low. To take advantage of all the data, we change the sampling strategy from random sampling to covering all the given data.
• Parallel data is often beneficial for crosslingual learning, we explore the effectiveness of parallelism in the inner-step and meta-step.

Training Procedure
We show the pipeline of training procedure in Algorithm 1. We first initialize our model f (θ) with multilingual pretrained Masked Language Models (mMLM) such as mBERT (Devlin et al., 2019), which jointly trains monolingual masked language models across multiple languages. These models have been proven strong performance on crosslingual transfer and serves as effective model initialization. In the second step we finetune our model on English monolingual data as a second pretraining step. This step allows the model to take benefits  of the high resource data and serves as a baseline model. We formulate meta-learning tasks as formerly stated. The meta dataset sample D aims to perform fast adaption under cross-lingual transfer setting. We perform the inner step and update the model to f (θ i ) on the support set S loss L (0) for hyperparameter set inner steps by gradient descent. We perform the meta step and optimize the parameters θ based on the loss L (1) of f (θ ) on the query set Q.
We perform either zero-shot evaluation or fewshot learning on the target languages. For zero-shot evaluation, we directly evaluate the performance of the models on unseen test set {x tgt } of the target languages. For few-shot learning, we use a small development set of the target language to finetune the model as the standard supervised learning, and then evaluate the performance on the test set.

Datasets
We show the dataset details in Table 1 and Table 2.

Natural Language Inference
Natural Language Inference (NLI) is the task of determining whether a given pair of hypothesis and premise is true (entailment), false (contradiction), or undetermined (neutral). It can be considered as a sentence pair classification task.
MNLI The Multi-Genre Natural Language Inference (MNLI)  dataset has 433k sentence pairs annotated with textual entailment information. The dataset covers a range of genres of spoken and written text and supports cross-genre evaluation.
In our experiment MNLI is applied as the high resource English dataset in the second pretraining step.
The dataset supports both zero-shot evaluation and few-shot training for cross-lingual transfer. Under the zero-shot evaluation setting, the model is directly tested on unseen target language with only the high-resource language English data available from MNLI and low-resource auxiliary languages data from the development set of XNLI. Under fewshot learning setting, the model is further finetuned by the development set of target language data of XNLI, and then tested on the target test data.

Question Answering
Question Answering (QA) is the task of answering a question based on a given support document. While there are various QA task settings including multi-choice QA, cloze style QA and so on, in this paper we focus on reading comprehension dataset, where the answer to every question is a segment of text (a span) from the corresponding reading passage.
SQuAD The Stanford Question Answering Dataset (SQuAD) (Rajpurkar et al., 2016(Rajpurkar et al., , 2018 consisting of 100,000+ questions posed by crowdworkers on a set of Wikipedia articles. For each question, three candidate answers spans are given, which are segment slices of the paragraph. The evaluation of model prediction is based on Exact Match (EM), which measures the percentage of predictions that match any one of the ground truth answers exactly, and F1 score, which measures the average overlap between the prediction and ground truth answer.
In our experiment, SQuAD 1.1 (Rajpurkar et al., 2016) is applied as the high resource English dataset in the second pretraining step. There are no un-answerable questions in the training data.  Split ar  bg  de  el  en  es  fr  hi  ru  sw  th  tr  ur  vi  zh XNLI Dev 2,500 2,500 2,500 2,500 2,500 2,500 2,500 2,500 2,500 2,500 2,500 2,500 2,500 2,500 2,500 Test 5,000 5,000 5,000 5,000 5,000 5,000 5,000 5,000 5,000 5,000 5,000 5,000 5,000 5,000 5,000 MLQA Dev 517  -512  -1148 500  -507  -----511 504  Test 5,335  -4,517  -11,590 5,254  -4,918  -----5,495 5,137   Table 2: The statistics of the multilingual datasets by languages. Dev is short for development set. The data of XNLI is parallel across languages generated by human translation from the English Data. The data of MLQA is partially parallel generated by automatic matching of target language Wikipedia articles to English Wikipedia, and then verified by crowd-sourced human annotation.
5K extractive QA instances (12K in English) in SQuAD format in seven languages -English(en), Arabic(ar), German(de), Spanish(es), Hindi(hi), Vietnamese(vi) and Simplified Chinese(zh). The development set of languages are highly but not completely parallel, detailed statistics are shown in Table 2. There are other cross-lingual question answering datasets such as Cross-lingual Question Answering Dataset (XQuAD) (Artetxe et al., 2020) and Information-Seeking Question Answering in Typologically Diverse Languages (TyDiQA) (Clark et al., 2020). XQuAD does not have a development set that it does not support few-shot training scenario. TyDiQA has a relatively small size of English training data, which has a mis-match with our experimental settings of English as a high resource language.
For both NLI and QA, we train with the same model described in Devlin et al. (2019), except for the pretrained layers are changed to mMLMs.

Training setup and hyper-parameters
We implement our algorithm with the libraries Pytorch (Paszke et al., 2019), Transformers (Wolf et al., 2020a) and higher (Grefenstette et al., 2019). All datasets are loaded with the Datasets (Wolf et al., 2020b) library.
We use the cache of pretrained models of Transformers to initialize the model. For MNLI finetuning, we train the model with 32 batch size and 128 max sequence length for 3 epochs. We use AdamW (Loshchilov and Hutter, 2019) optimizer with 2e-5 learning rate. For SQuAD finetuning, we train the model with 12 batch size, 384 max sequence length and 128 document stride for 2 epochs. We use AdamW optimizer with 3e-5 learning rate. For both datasets we only use the training data to finetune the model.
For XLA-MAML, for both datasets we use the Figure 2: Differences in performance in terms of accuracy scores on the test set for zero-shot XLA-MAML on XNLI using the mBERT model. Rows correspond to auxiliary and columns to target languages. Numbers on the off-diagonal indicate performance differences between XLA-MAML and the baseline model. The coloring scheme indicates the differences in performance (e.g., blue for large improvement).
same data preprocessing parameters as the English model. We use batch size 8 for both inner-step update and meta-step update. We use learning rate of 1e-5 with SGD optimizer for the inner-step update, and use learning rate of 1e-5, weight decay of 0.01 with Adam optimizer (Kingma and Ba, 2015) and a linear learning rate scheduler for the metastep update. NLI models with memory usage of around 10G use 1 minute for each 100 meta steps trained on GeForce GTX 1080Ti and QA models with memory usage of around 22G use 2 minutes for each 100 meta steps trained on TITAN RTX. Since the low resource languages only have development set, to avoid fitting to the test set and select a stopping criterion of the algorithm, we observe the performance of a few runs on unseen language development set to fix a stopping iteration

Results
NLI We employ zero-shot and few-shot learning for XNLI dataset with XLA-MAML and report the results in Table 3 and Figure 2. In Table 3, the scores are reported by 3 run average on XNLI test set. For fair comparison to X- MAML Nooralahzadeh et al. (2020), that they rely on a weaker MNLI pretrained model, we use their public finetuning code and load a MNLI pretrained model trained with same hyperparameters with our baseline, which achieved higher performance than the scores reported in Nooralahzadeh et al. (2020). We compare results of XLA-MAML and X-MAML only using one auxiliary language development set as auxiliary data. To simulate fast adaption from high resource en to low resource language, we use en for the inner-step update and aux. lang for the meta-step update. Here the we use the development set of XNLI en for inner step which is a subset of MNLI development set. The experimental results show no statistical significant difference in drawing samples from XNLI or MNLI English development set. The results show that our method can achieve comprehensive improvement over the languages. Nooralahzadeh et al. (2020) reported hi to be the most beneficial auxiliary language for their algorithm, XLA-MAML achieves further improvement compared to X-MAML with the sample cross-lingual data resource. To be noted, under our setting hi is not the most beneficial auxiliary language. We show the zero-shot evaluation results of all auxiliary language and target language pairs in Figure 2. We observe the auxiliary languages beneficial for zero-shot scores for up to 5.4% accuracy for some language pairs such as ur → hi. Significant improvement was achieved for all auxiliary languages except for en, where is reasonable that no cross-lingual adaption is performed and also sw, which matches with the results reported by Nooralahzadeh et al. (2020), where X-MAML lowered the scores with sw as auxiliary language. We consider this caused by the typological commonalities of sw.
We also report their result of two auxiliary language pairs. We reproduce the best auxiliary language pairs results reported in Nooralahzadeh et al. (2020). We show the corresponding results, which the two languages either as inner-step update or meta-step update. We also show the best language pairs results for XLA-MAML.
For few-shot learning learning, we show results of two settings. In the one auxiliary and two auxiliary setting, since we have a development set on the target language for finetuning, we use tgt. lang as the inner-step and aux. lang as the meta-step. Under few-shot setting, with the supervision of target language, the effect of auxiliary languages  are limited for all MAML methods. The search of best language for few-shot learning setting is enormous. We compare with the best auxiliary language setting of X-MAML and achieve higher or comparable results. Under few-shot setting, with the supervision of target language, the effect of auxiliary languages are limited for all MAML methods.
In addition, we report the results from Devlin et al. QA We employ zero-shot and few-shot learning for MLQA dataset with XLA-MAML and report results in Table 4.We use similar settings as NLI for QA results. The code is not available for X-MAML QA and the paper did not report results on mBERT, that we reproduce their results with our own implementation. We report results for one-auxiliary language hi . Few-shot learning was not investigated by X-MAML in Nooralahzadeh et al. (2020), that we show our results comparing to few-shot baseline. Similar to NLI, we report TRANSLATE-TEST and TRANSLATE-TRAIN Lewis et al. (2020)    in Table 5. As we can see our model out-performs the zero-shot baseline for around 2% acc. in average, and also exceeds XMAML baseline with hi, demonstrating the generalization ability of our model on stronger mMLM baselines.

Effects of Sampling Strategy
We report the results for sampling strategy in Table 6. Since the development set for XLA-MAML is relatively small, unlike in few-shot learning the data for MAML is larger, we consider it important to cover all of the development data during the meta learning update. We use compare two sampling strategies, the first strategy randomly samples a batch of examples at each meta learning episode, the second strategy preprocesses the batches that  they cover all of the examples. We show results on one auxiliary language and one auxiliary language bi-pair. There is a significant performance drop when the samples are randomly drawn at each episode, reflects the importance of making full use of all data for low-resource transfer learning.

Effects of Parallel Data
Parallel data is reported useful for cross-lingual representation alignment  and mMLM pretraining (Lample and Conneau, 2019). The XNLI dataset is completely parallel, that it is possible for us to investigate the effectiveness of parallelism of inner-step and meta-step. The results on one auxiliary language and one auxiliary language bi-pair are shown in Table 7. We can see that parallelism gains all-round but margin improvement. Taking account that the parallel data is relatively tiny, we consider parallelism useful for XLA-MAML training.  Gu et al. (2018) demonstrated the effectiveness of Meta Learning for Neural Machine Translation over strong baselines of cross-lingual transfer learning. By considering language pairs as tasks, MAML has shown strong results with as few as 600 parallel sentences. Nooralahzadeh et al. (2020) proposed X-MAML, which is an MAML based method with auxiliary languages data as tasks, which uses the data of auxiliary languages in both the inner-step and meta-step. While they achieved improvement over mMLM baselines, their meta learning procedure cannot perform cross-lingual adaption. Our experimental results show how our method gains superiority over X-MAML.

Conclusion
In this paper, we propose XLA-MAML, a crosslingual adaption-based meta learning method using auxiliary languages to improve cross-lingual transfer. The algorithm directly performs cross-lingual adaption by using different languages in the support set and the query set during the meta learning procedure, which encourages the meta-learning algorithm to capture cross-lingual shared knowledge among the auxiliary language and source language, and further improve target language performance. The experiments on cross-lingual NLI and QA on mBERT and XLM-R show the effectiveness of our method, especially under zero-shot setting. Under few-shot setting, while the supervision of the target language weakens the effect of auxiliary languages, our method still achieves comparable or better results. We also investigate various XL-specific settings for meta learning, including the sampling strategy and the usage of parallelism.

A Experimental Details
For NLI, we follow the standard sentence pair classification input method for pretrained MLMs (Devlin et al., 2019). We add a special separator token [SEP] to the end of each sentence sequence and concatenate them together. A special classification token [CLS] is added in front of every input example. The token sequence is then encoded by the mMLM. At the output, the [CLS] representation is fed into an output layer for classification. inputs and outputs into the pretrained model and finetune all the parameters end-to-end.
For QA, we formulate the task into a sequence tagging task (Devlin et al., 2019). For encoding, we represent the input question and passage as a single packed sequence. Similar to NLI, we add a special separator token [SEP] to the end of the question and the support passage and then concatenate them together. A unused special classification token [CLS] is added in front of every input example to align the input with the MLM training phrase. For finetuning, we introduce a start vector S min R H and an end vector E min R H . The probability of a token to be the start token is calculated by the softmax of a dot product between S and the token representation T i : P (i = start) = e S * T i e S * T i and the end token probability P (j = end) vice versa. The final score of a candidate span (i, j) is defined as T i * S +T j * E, where j ≥ i. The training objective is the sum of the log-likelihoods of the correct start and end positions. During evaluation, if the best prediction is an invalid answer where j < i, we search for the k-best prediction until the prediction is valid.