{'fraction_non_alphanumeric': 0.029756450446105618, 'fraction_numerical': 0.014371835061490234, 'mean_word_length': 5.184312555920072, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Cross-lingual Retrieval for Iterative Self-Supervised Training

Recent studies have demonstrated the cross-lingual alignment ability of multilingual pretrained language models. In this work, we found that the cross-lingual alignment can be further improved by training seq2seq models on sentence pairs mined using their own encoder outputs. We utilized these findings to develop a new approach -- cross-lingual retrieval for iterative self-supervised training (CRISS), where mining and training processes are applied iteratively, improving cross-lingual alignment and translation ability at the same time. Using this method, we achieved state-of-the-art unsupervised machine translation results on 9 language directions with an average improvement of 2.4 BLEU, and on the Tatoeba sentence retrieval task in the XTREME benchmark on 16 languages with an average improvement of 21.5% in absolute accuracy. Furthermore, CRISS also brings an additional 1.8 BLEU improvement on average compared to mBART, when finetuned on supervised machine translation downstream tasks.


Introduction
Pretraining has demonstrated success in various natural language processing (NLP) tasks. In particular, self-supervised pretraining can learn useful representations by training with pretext task, such as cloze and masked language modeling, denoising autoencoder, etc. on large amounts of unlabelled data [23,26,6,40,20,17]. Such learnt "universal representations" can be finetuned on task-specific training data to achieve good performance on downstream tasks.
More recently, new pretraining techniques have been developed in the multilingual settings, pushing the state-of-the-art on cross-lingual understanding, and machine translation. Since the access to labeled parallel data is very limited, especially for low resource languages, better pretraining techniques that utilizes unlabeled data is the key to unlock better machine translation performances [32,19,4].
In this work, we propose a novel self-supervised pretraining method for multilingual sequence generation: Cross-lingual Retrievial for Iterative Self-Supervised training (CRISS). CRISS is developed based on the finding that the encoder outputs of multilingual denoising autoencoder can be used as language agnostic representation to retrieve parallel sentence pairs, and training the model on these retrieved sentence pairs can further improve its sentence retrieval and translation capabilities in an iterative manner. Using only unlabeled data from many different languages, CRISS iteratively mines for parallel sentences across languages, trains a new better multilingual model using these mined sentence pairs, mines again for better parallel sentences, and repeats.
In summary, we present the following contributions in this paper: • We present empirical results that show the encoder outputs of multilingual denoising autoencoder (mBART) represent language-agnostic semantic meaning.
• We present empirical results that show finetuning mBART on only one pair of parallel bi-text will improve cross-lingual alignment for all language directions. • We introduce a new iterative self-supervised learning method that combines mining and multilingual training to improve both tasks after each iteration. • We significantly outperform the previous state of the art on unsupervised machine translation and sentence retrieval. • We show that our pretraining method can further improve the performance of supervised machine translation task compared to mBART.
This paper is organized as follows. Section 2 is devoted to related work. Section 3 introduces improvable language agnostic representation emerging from pretraining. Section 4 describes the details of cross-lingual retrieval for iterative self-supervised training (CRISS). Section 5 evaluates CRISS with unsupervised and supervised machine translation tasks as well as sentence retrieval tasks. Section 6 iterates over ablation studies to understand the right configurations of the approach. Then we conclude by Section 7.

Related work
Emergent Cross-Lingual Alignment On understanding cross-lingual alignment from pretrained models, [38] [24] present empirical evidence that there exists cross-lingual alignment structure in the encoder which is trained with multiple languages on a shared masked language modeling task.
Analysis from [35] shows that shared subword vocabulary has negligible effect while model depth matters more for cross-lingual transferability. In English language modeling, retrieval-based data augmentation has been explored by [13] and [9]. Our work combines this idea with the emergent cross-lingual alignment to retrieve sentences in another language instead of retrieving paraphrases in the same language.
Multilingual Pretraining Methods With large amount of unlabeled data, various self-supervised pretraining approaches have been proposed to initialize models or parts of the models for downstream tasks (e.g. machine translation, classification, inference and so on) [6,23,26,40,20,27,32,7,28,17,19]. Recently these approaches have been extended from single language training to crosslingual training [34,15,4,19]. In the supervised machine learning literature, data augmentation [39,14,1,31] has been applied to improve learning performance. To the best our knowledge, little work has been explored on self-supervised data augmentation for pretraining. This work pretrains multilingual model with self-supervised data augmenting procedure using the power of emergent crosslingual representation alignment discovered by the model itself in an iterative manner.
Unsupervised Machine Translation Several authors have explored unsupervised machine translation techniques to utilize monolingual data for machine translation. A major line of work that does not use any labeled parallel data typically works as follow: They first train an initial language model using a noisy reconstruction objective, then finetune it using on-the-fly backtranslation loss [16,32,19,18,8]. Our work differs from this line of work in that we do not use backtranslation but instead retrieve the target sentence directly from a monolingual corpus. More recently, [29] and [37] start incorporating explicit cross-lingual data into pretraining. However, since the quality of cross-lingual data extracted is low, additional mechanisms such as sentence editing, or finetuning with iterative backtranslation is needed. To the best of our knowledge, our approach is the first one that achieves competitive unsupervised machine translation results without using backtranslation.

Self-Improvable Language Agnostic Representation from Pretraining
We start our investigation with the language agnostic representation emerging from mBART pretrained models [19]. Our approach is grounded by the following properties that we discovered in mBART models: (1) the mBART encoder output represents the semantics of sentences, (2) the representation is language agnostic, and (3) the language agnostics can be improved by finetuning the mBART models with bitext data of a small number of language pairs (or even only 1 pair of languages) in an iterative manner. We explain these findings in details in this section and the next section on the iterative procedure.

Cross-lingual Language Representations
We use mBART [19] seq2seq pre-training scheme to initialize cross-lingual models for both parallel data mining and multilingual machine translation. The mBART training covers N languages: D = {D 1 , ..., D N } where each D i is a collection of monolingual documents in language i. mBART trains a seq2seq model to predict the original text X given g(X) where g is a noising function, defined below, that corrupts text. Formally we aim to maximize L θ : where x is an instance in language i and the distribution P is defined by the Seq2Seq model. In this paper we used the mbart.cc25 checkpoint [19] open sourced in the Fairseq library [21] 1 . This model is pretrained using two types of noise in g -random span masking and order permutation -as described in [19].
With a pretrained mBART model, sentences are then encoded simply by extracting L2-normalized average-pooled encoder outputs. To understand the language agnostic of the mBART sentence representation, we study sentence retrieval tasks. For each language pair, we go through each sentence in the source language, find the closest sentence to that sentence in the target language (using cosine similarity), and report the average top-1 retrieval accuracy for each language pair. We use the TED58 dataset which contains multi-way translations of TED talks in 58 languages [25] 2 .

Case Study
The sentence retrieval accuracy on this TED dataset is depicted in Figure 1. The average retrieval accuracy is 57% from the mBART model which is purely trained on monolingual data of 25 languages without any parallel data or dictionary; the baseline accuracy for random guessing is 0.04%. We also see high retrieval accuracy for language pairs with very different token distribution such as Russian-German (72%) or Korean-Romanian (58%). The high retrieval accuracy suggests that mBART model trained by monolingual data of multiple languages is able to generate language agnostic representation that are aligned at the semantic level in the vector space.
Moreover the cross-lingual semantic alignment over multiple languages not only emerges from the monolingual training but also can be improved by a relatively small amount of parallel data of just one direction. Figure 2 shows the sentence retrieval accuracy of an mBART model that are finetuned with Japanese-English in the IWSLT17 parallel dataset (223, 000 sentence pairs) [3]. Even with parallel data of one direction, the retrieval accuracy improved for all language directions by 27% (absolute) on average.
Inspired by these case studies, we hypothesize that the language agnostic representation of the pretrained model can be self-improved by parallel data mined by the model itself without any supervised parallel data. We will devote section 4 to details of the self-mining capability and the derived Cross-lingual Retrieval Iterative Self-Supervised Training (CRISS) procedure.
4 Cross-lingual Retrieval for Iterative Self-Supervised Training (CRISS) Input: (1) monolingual data sets D i and D j for language i and j respectively, (2) a pretrained model Θ, 3: Set k, M , τ to be the desired KNN size, the desired mining size, and the desired minimum score threshold respectively 4: for each x in D i , each y in D j do 5: x, y ← Embed(Θ, x), Embed(Θ, y) 6: Using FAISS [12] 7: end for 8: In this section, we make use the of the language agnostic representation from mBART described in Section 3 to mine parallel data from a collection of monolingual data. The mined parallel data is then used to further improve the cross-lingual alignment performance of the pretrained model. The mining and improving processes are repeated multiple times to improve the performance of retrieval, unsupervised and supervised multilingual translation tasks as depicted in Figure 3 using Algorithm 2.
Unsupervised parallel data mining To mine parallel data without supervised signals, we employ the margin function formulation [2,1,30] based on K nearest neighbors (KNN) to score and rank pairs of sentences from any two languages. Let x and y be the vector representation of two sentences in languages i and j respectively, we score the semantic similarity of x and y using a ratio margin function [1] defined as the following: where N x is the KNN neighborhood of x in the monolingual dataset of y's language; and N y is the KNN neighborhood of y in the monolingual dataset x's language. The margin scoring function can be interpreted as a cosine score normalized by average distances (broadly defined) to the margin regions established by the cross-lingual KNN neighborhoods of the source and target sentences respectively. The KNN distance metrics are defined by cos(x, y). We use FAISS [12] -a distributed dense vector similarity search library -to simultaneously search for all neighborhoods in an efficient manner at billion-scale. Thus we obtain the mining Algorithm 1.
Iteratively mining and multilingual training Building on the unsupervised mining capability of the pretrained model (Algorithm 1), we can conduct iteratively mining and multilingual training procedure (Algorithm 2) to improve the pretrained models for both mining and downstream tasks. In Algorithm 2, we repeat mining and multilingual training T times. On multilingual training, we simply augment each mined pair (x, y) of sentences by adding a target language token at the beginning of y to form a target language token augmented pair (x, y ). We then aggregate all mined pairs {(x, y) } of the mined language pairs into a single data set to train a standard seq2seq machine translation transformer models [33] from the pretrained mBART model. To avoid overfitting to the noisy mined data, at each iteration we always refine from the original monolingual trained mBART model but only update the mined dataset using the improved model (line 7 of Algorithm 2).

Experiment Evaluation
We pretrained an mBART model with Common Crawl dataset constrained to the 25 languages as in [19] for which we have evaluation data. We also employ the same model architecture, the same BPE prepossessing and adding the same set of language tokens as in [19]. We keep the same BPE vocab and the same model architecture throughout pretraining and downstream tasks.
On mining monolingual data, we use the same text extraction process as described in [30] to get the monolingual data to mine from, which is a curated version of the Common Crawl corpus. We refer the reader to [30,36] for a detailed description of how the monolingual data are preprocessed. We subsample the resulting common crawl data to 100 million sentences in each language. For low resources, we may have fewer than 100 million monolingual sentences. The statistics of monolingual data used are included in the supplementary materials.
We find that a small number of pivot languages are enough to achieve high translation and sentence retrieval accuracy. Our default configuration mines sentences to and from English, Hindi, Spanish, and Chinese, for a total of 90 languages pairs (180 language directions) instead of all 300 language pairs (600 directions) for the 25 languages. We set the K = 5 for the KNN neighborhood retrieval for the margin score functions (Equation 2). In each iteration, we tune the margin score threshold based on validation BLEU on a sampled validation set of size 2000.
With the mined 180 directions parallel data, we then train the multilingual transformer model for maximum 20, 000 steps using label-smoothed cross-entropy loss as described in Algorithm 2. We sweep for the best maximum learning rate using validation BLEUs.
After pretraining, the same model are evaluated with three tasks: sentence retrieval, unsupervised machine translation, and supervised machine translation tasks. For supervised machine translation, we use CRISS model to initialize the weights to train models for supervised machine translation.

Unsupervised Machine Translation
We evaluate CRISS model using unsupervised neural machine translation benchmarks that cover both low resource and high resource language directions. For English-French we use WMT '14, for English-German and English-Romanian we use WMT'16 test data, and for English-Nepali and English-Sinhala we use Flores test set [10]. For decoding in both the unsupervised and supervised machine translation tasks, we use beam-search with beam size 5, and report the final results in BLEU [22]. To be consistent with previous literature, we used multi-bleu.pl 3 for evaluation.
As shown in Table 1  We use the Tatoeba dataset [2] to evaluate the cross-lingual alignment quality of CRISS model following the evaluation procedure specified in the XTREME benchmark [11].
As shown in Table 2, compared to other pretraining approaches that don't use parallel data, CRISS outperforms state-of-the-art by a large margin, improving all 16 languages by an average of 21.5% in absolute accuracy. Our approach even beats the state-of-the-art supervised approach [2] in Kazakh, improving accuracy from 17.39% to 77.9%. This shows the potential of our work to improve translation for language pairs with little labeled parallel training data.

Supervised Machine Translation
For the supervised machine translation task, we use the same benchmark data as in mBART [19]. We finetune models learned from CRISS iteration 1, 2, and 3 on supervised training data of each bilingual direction. For all directions, we use 0.3 dropout rate, 0.2 label smoothing, 2500 learning rate warm-up steps, 3e − 5 maximum learning rate. We use a maximum of 40K training steps, and final models are selected based on best valid loss. As shown in Table 3, CRISS improved upon mBART on low resource directions such as Gujarati-English (17.7 BLEU improvement), Kazakh-English (5.9 BLEU improvement), Nepali-English (4.5 BLEU improvement

Ablation Studies
We conduct ablation studies to understand the performance implications of bilingual finetuning versus multilingual finetuning and that of the number of pivot languages to mine parallel data.

Multilingual Finetuning versus Bilingual Finetuning
Multilingual training is known to help improve translation quality for low resource language directions. Since we only use mined pseudo-parallel data at finetuning step, every language direction is essentially low-resource. We ran experiments to confirm that finetuning multilingually help with translation quality for every direction. Here, we use mBART25 encoder outputs to mine parallel data for 24 from-English and 24 to-English directions. For the bilingual config, we finetune 48 separate models using only bilingual pseudo-parallel data for each direction. For the multilingual config, we combine all 48 directions data and finetune a single multilingual model. As we can see in Figure 5 and Figure 6 multilingual finetuning performs better in general 4 and particularly on to-English directions. In this section, we explore how choosing the number of language directions to mine and finetune affect the model performance. [2] found that using two target languages was enough to make the sentence embedding space aligned. Unlike their work which requires parallel training data, we are not limited by the amount of labeled data, and can mine for pseudo-parallel data in every language direction. However, our experiments show that there is limited performance gain by using more than 2 pivot languages. We report the translation quality and sentence retrieval accuracy of 1st iteration CRISS trained with 1 pivot language (English), 2 pivot languages (English, Spanish), and 4 pivot languages (English, Spanish, Hindi, Chinese).   Note that the embedding space are already well aligned even with 1 pivot language because of the emergent alignment in multilingual denoising autoencoder training, and because we jointly train the from-English and to-English directions in the same model. We observe that the average translation quality and sentence retrieval accuracy improve slightly as we increase the number of pivot languages. Since using more pivot languages will increase the computational cost linearly in the mining stage, we used 4 pivot languages in the final version.

Conclusion
We introduced a new self-supervised training approach that iteratively combine mining and multilingual training procedures to achieve state-of-the-art performances in unsupervised machine translation and sentence retrieval. The proposed approach achieved these results even though we artificially limited the amount of unlabeled data we used. Future work should explore (1) a thorough analysis and theoretical understanding on how the language agnostic representation arises from denoising pretrainining, (2) whether the same approach can be extended to pretrain models for non-seq2seq applications, e.g. unsupervised structural discovery and alignment, and (3) whether the learned cross-lingual representation can be applied to other other NLP and non-NPL tasks and how. We will release our code and pretrained models, and invite the research community to try them out for cross-lingual tasks.

Broader Impact
Our work advances the state-of-the-art in unsupervised machine translation. For languages where labelled parallel data is hard to obtain, training methods that better utilize unlabeled data is key to unlocking better translation quality. This technique can help to remove language barriers across the world, particularly for the community speaking low resource languages.