{'fraction_non_alphanumeric': 0.0324717347443755, 'fraction_numerical': 0.017206593322928166, 'mean_word_length': 4.8612226684515845, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Complete Multilingual Neural Machine Translation

Multilingual Neural Machine Translation (MNMT) models are commonly trained on a joint set of bilingual corpora which is acutely English-centric (i.e. English either as source or target language). While direct data between two languages that are non-English is explicitly available at times, its use is not common. In this paper, we first take a step back and look at the commonly used bilingual corpora (WMT), and resurface the existence and importance of implicit structure that existed in it: multi-way alignment across examples (the same sentence in more than two languages). We set out to study the use of multi-way aligned examples in order to enrich the original English-centric parallel corpora. We reintroduce this direct parallel data from multi-way aligned corpora between all source and target languages. By doing so, the English-centric graph expands into a complete graph, every language pair being connected. We call MNMT with such connectivity pattern complete Multilingual Neural Machine Translation (cMNMT) and demonstrate its utility and efficacy with a series of experiments and analysis. In combination with a novel training data sampling strategy that is conditioned on the target language only, cMNMT yields competitive translation quality for all language pairs. We further study the size effect of multi-way aligned data, its transfer learning capabilities and how it eases adding a new language in MNMT. Finally, we stress test cMNMT at scale and demonstrate that we can train a cMNMT model with up to 12,432 language pairs that provides competitive translation quality for all language pairs.


Introduction
Multilingual machine translation (Dong et al., 2015;Firat et al., 2016a;Johnson et al., 2017;Aharoni et al., 2019), which can serve multiple language pairs with a single model, has attracted much attention. In contrast to bilingual MT systems which can only serve one single language pair, multilingual models can serve O(N 2 ) language pairs (N being the number of languages in a multilingual model) (Zhang et al., 2020). The amount of available training data can differ a lot across language pairs and the majority of available MT training data is English-centric (Tiedemann, 2018;Arivazhagan et al., 2019b) which in practice means that most non-English language pairs do not see a single training example when training multilingual models (see Figure 1a). As a consequence, the actual performance of language pairs that do not include English on the source or target side lags behind the ones with large amounts of training data. Further, when increasing the number of languages, it gets (a) impractical to gather training data for each language pair and (b) challenging to find the right mix during training. Which is why models tasked with direct translation between non-English pairs either resort to bridging (pivoting) through a pivot language (Habash and Hu, 2009), or make use of synthetic parallel data (via back-translation) (Firat et al., 2016b; or study the problem under zero-shot settings (Johnson et al., 2017;Ha et al., 2016).
In this study, we make use of the potential pre- To make use of this data, the model samples a source and target language from the set of multiway aligned corpus during training, which allows the model to see language pairs where originally no training data existed (missing connections in Figure 1a). As our experiments support, this method enables us to get access to training data for all tested language pairs (generating a complete graph (Figure 1b)). We will show that it is possible to generate a complete graph for at least a 6-language WMT setup. Some of the WMT training data is multi-way parallel by construction. Nevertheless, we show that we also find many training examples where the source and target origin from different sources. We further show on our 112 languages internal dataset, that we can find sufficient training data for over 12,000 language pairs by only providing 111 English-centric training corpora. This result indicates that it is possible to generate direct training data for many language pairs without the need for crawling new training examples. Our experiments suggest that before falling back to methods like zero-shot translation, you should investigate the structure of your pre-existing training data.
To address the problem of finding the right mix of examples from different language pairs during training, we further introduce a hierarchical sampling strategy that is language-specific (as opposed to being language pair specific). In addition to fixing some chronic issues of MNMT (i.e. low quality for out of English translation (Firat et al., 2016a;Johnson et al., 2017;Arivazhagan et al., 2019b)), the proposed sampling strategy efficiently ensures all source-target pairs are covered.
Experiments demonstrate that we can train a cM-NMT model on a 30-language-pair WMT setup that outperforms bilingual and multilingual baselines as well as bridging on all non-English language pairs. We further show that the performance of the English language pairs stay stable and do not suffer from the changes in both the training data and the new training data sampling strategy. Furthermore, we share experiments at scale by demonstrating that we can train a cMNMT model that can serve 12,432 language pairs. Our contribution is three-fold: • We show that we can find a lot of training examples for all language pairs in a multilingual mix by only pivoting pre-existing Englishcentric training data. We further show that many of the extracted examples originate from different data sources and this method could scale to many more datasets. We also support these findings with experiments on our internal dataset, where we were able to find training data for all 12,432 language pairs.
• We demonstrate that cMNMT outperforms bilingual baselines, multilingual baselines as well as bridging on all non-English language pairs while keeping translation performance on English-centric language pairs.
• We introduce a new sampling strategy that is purely based on the target language instead of language pairs and does scale to MNMT models which hundreds of languages.

A Peek at Multi-way Aligned Examples in Bilingual Corpora
We choose six languages Czech (cs), English (en), French (fr), German (de), Spanish (es) and Russian (ru) from the public WMT datasets. The selection of the languages was driven by the fact that the WMT 2013 evaluation campaign (Bojar et al., 2013)  Some of the extracted non-English training examples are multi-way parallel by construction. The UN corpus is a 6-way parallel corpus, and three of the languages (English, French and Spanish) are in our 6-language mix. A portion of the Europarl corpus is again multi-way aligned. Nevertheless, a good amount of the extracted data is coming from different sources. Table 2 shows the number of non-English bilingual training examples separated by the two sources they originated from. Table 3 shows how many translations are available for each sentence in the WMT training data. The majority (123 million) of the multi-way aligned examples do only have translations into two languages. As our original bilingual training data is English-centric, all of the 123 million training examples consist of an English sentence and a translation into one of our five other languages. A total of 13 million multi-way aligned examples are available in at least three languages. Further, Figure 2 shows the average number of translations conditioned by the language. Both Spanish and German have, on average more than three translations. In comparison, the majority of the multi-way aligned examples with Czech or English on the target side are bilingual (having only two translations). Our study resurfaced the inherent multi-way aligned information in the commonly used set of parallel corpora instead of discarding this information.

Complete Multilingual NMT
We call MNMT models that are trained for all possible source-target pairs as complete MNMT as all languages are connected via training data (also see Figure 1). Before going into details of how the missing pairs' data gathered, we recap MNMT first.
Multilingual NMT Framework MNMT (Firat et al., 2016b;Johnson et al., 2017) is an extension of bilingual NMT which uses a single model to translate between multiple languages. The model parameters are trained on a joint set of bilingual corpora from different language pairs. Given the data imbalance across the different corpora, it is common to oversample the language pairs with less training data (Lee et al., 2016;Johnson et al., 2017). For a given language pair p, let D(p) be the size of the available parallel corpus, the sample probability with a temperature T is defined as As a result, T = 1 corresponds to the actual data distribution, and T = 100 corresponds to (almost) an equal number of samples for each language pair). In addition to being able to translate language pairs that the model was trained with, the model can also translate between language pairs never seen explicitly during training which is often referred as zero-shot translation (Johnson et al., 2017;Ha et al., 2016).
Using multi-way aligned data in MNMT Instead of only relying on bilingual corpora, bilingual examples from different language pairs with identical target sentences can be combined into a single multi-way aligned training example. An example is given in Table 4. By comparing the English sides of the Spanish-English and the German-English corpora, we extract a multi-way aligned example that contains translations into all three languages.    While we can extract direct training data for any source-target pair among the languages considered, the total number of language pairs increases quadratically. The vanilla language pair based sampling strategy in Eq. (1) with adjustable temperature is capable of balancing low-high resource language pairs during training. However, we noticed a critical failure mode, which is further amplified in complete MNMT. The language-pair based sampling strategy (regardless of the temperature being used) over-represents English in English-centric models. Notice half of the languages have English on the source side, with the other half on the target side. This over-representation yields a schedule of examples for the encoder (resp. for the decoder) to see English examples half of the time throughout the entire training process. As a result, trained models end up favouring English either on the source and/or target. Although the implications on the encoder could be minimal, over-exposing English examples to the decoder curtail the learning signal when the target language is non-English. We hypothesise that this imbalance in the learning signal with respect to the target language is one of the roots of poor translation quality of multilingual models when translating out of English (Firat et al., 2016a;Johnson et al., 2017;Arivazhagan et al., 2019b).
To alleviate the over-representation of English with the language-pair based sampling strategy, we propose a hierarchical sampling strategy with two levels: i) we choose a target language (based on a temperature-based schedule), ii) uniformly sample a source language. Formally, for a given target language l, let D(l) be the size of the available training examples with target language l, the sample probability with a temperature T is defined as During training, the scheduler samples a batch of training examples based on the target language only, as opposed to source-target language pair specific sampling. After choosing a target language, for each multi-way aligned example, we randomly (uniformly) pick one of the translations as the source sentence.

Experiments
We use a public transformer implementation with the transformer-big model size (Vaswani et al., 2017) for all multilingual setups. All bilingual models use a vocabulary of 32,000 subwords, while all multilingual models use a vocabulary of 64,000 subword units. All multilingual models are trained for 500,000 updates using an average batch size of around 33,000 sentences (∼1 million tokens). All bilingual models are trained for 400,000 steps as they converged earlier using a batch size of around 8,000 sentences (∼260,000 tokens). Due to the data imbalance across languages, we use a temperature-based data sampling strategy to oversample low-resource language pairs in standard MNMT models (Equation 1) and low-resource target languages in cMNMT models (Equation 2). We use a temperature of T = 5 in both cases. All multilingual models add a token at the beginning of the input sentence to specify the required target language. All BLEU (Papineni et al., 2002) scores are calculated with sacreBLEU (Post, 2018). 1

Baselines on WMT
We train several baselines: (i) bilingual models, (ii) multilingual models based on English-centric data, and (iii) bridging non-English language pairs.
Bilingual Baselines We train two bilingual baselines (using either transformer-base or transformerbig) for each language pair. In addition to training baselines on the original English-centric WMT data, we also train models for non-English language pairs on the extracted direct data (see Table 1). We experimented with several dropout rates for both setups and found that dropout=0.1 works best for transformer-base while dropout=0.3 works best for transformer-big. As can be seen from Table 5 and Table 6, the experiments suggest that the translation quality of the non-English language pairs is far behind the ones for English-centric language pairs. As an example, the translation quality between German and Russian reaches 6-7 BLEU only.   Bridging (Pivoting) Baselines The quality of MNMT is still behind the one from bilingual baselines for most of the language pairs (comparing Table 6 and Table 7). Nevertheless, having a single NMT model for each language pair is impractical, especially when increasing the number of language pairs. An alternative approach is called bridging (Cohn and Lapata, 2007;Wu and Wang, 2007;Utiyama and Isahara, 2007). For the bridging approach, we compromise and train only English-centric models. To enable the translation between non-English language pairs, the source sentence cascades through the source→English and English→target systems to generate the target sentence. This simple process has several limitations: (i) translation errors accumulate in the pipeline, (ii) decoding time gets doubled since inference has to be run twice, (iii) bridging through a morphologically low language (i.e. English), important information could be lost (i.e. gender). The BLEU scores (

Complete MNMT Models on WMT
Without adding new training data and taking into account the multi-way property of the data, we train a complete multilingual NMT system (cMNMT, see Section 3). We compare the performance of cMNMT with the best baseline model that is based on bridging (Table 8) and report BLEU and delta BLEU numbers in Table 9. The BLEU scores for the non-English language pairs go up from at least 1.4 BLEU for Russian→Spanish up to 5.0 BLEU for Czech→Russian. We changed the sampling strategy for our cMNMT models to be conditioned on the target language only (Section 3). As a result, English has been seen less often as the target language when compared to a standard MNMT setup. Interestingly, this seems to affect only the performance of Russian→English, which shows a decrease of 1 BLEU point. The other language pairs with English as the target language are keeping their translation quality.
When comparing our cMNMT model to the English-centric baseline (Table 7), we see an average BLEU increase of 14.6 BLEU for all non-English language pairs. It is worth noticing that every language pair has now at least 22 absolute BLEU points. Interestingly, the absolute BLEU scores in each row (translations into the same language) are much closer, suggesting a more universal input representation.  Training Data Sampling Strategy In Section 3, we did introduce our new training data sampling strategy that is based on the target language only. This change was mainly driven by the fact that having a language-pair conditioned schedule is not scalable when building a system of 12,432 language pairs. Instead of finding a good sampling weight for each of the 12,432 language pairs, we only need to find a suitable mix for the 112 target languages. Further, we have more control over how often each target language will be seen during training. To see the impact of this change, we train an MNMT system on the joint set of the 30 different bilingual corpora with a standard language-pair based temperature scheduling scheme and compare it to a cMNMT model. We used temperature 5 in both setups. ∆BLEU numbers for each language-pair can be seen in Figure 3. The language-conditioned temperature scheduling increases BLEU scores for 29 out of 30 language-pairs with larger gains for the lowresource language-pairs. This experiment suggests that a target language based temperature scheduling is not only simpler but also performs better on average.

Separate Multi-way Aligned Examples
We test the transfer learning capability of cMNMT by training a cMNMT model only on the 13 million multi-way aligned examples that have translations in at least three languages (see Table 3).
In other words, we remove all training examples that are only available in English and one additional language. If no transfer learning is happening, the English-centric scores will decrease while the BLEU numbers of the non-English language pairs are not affected. Experimental results can be seen in Table 10. Interestingly, we find that the performance of all language pairs is similarly affected. This indicates that transfer learning is happening between the language pairs and that non-English language pairs benefit from having more Englishcentric data.  To further study this effect, we reverse that experiment and remove all examples that have translations into more than two languages. This experiment investigates if the non-English language pairs in a standard MNMT model can benefit from having training examples with identical English sides. Experimental results can be found in Table 11. The BLEU scores for English-centric language pairs drop by 0.9 points on average while the performance of non-English language pairs decreases by 1.6 BLEU on average.  Leave N-Out We further investigate the transfer learning capability of our approach by training several cMNMT models on different amounts of training data. We start with a cMNMT model trained on English-centric bilingual training data only. This setup ensures that all languages have been seen on both the source and target side during training. We further group the remaining multi-way aligned training examples by target language and add one after another to the training data. Important to mention: We retrained all configurations from scratch. Experimental results are summarized in Figure 4. We report average BLEU scores grouped by the target language. We can see that adding training data x → y for a target language y, gives a significant boost in translation quality for that target language. These results demonstrate that even though we can translate between language pairs without seeing a single example during training, adding supervision during training significantly increases BLEU scores.
Adding a New Language We further investigate how a cMNMT model behaves when fine-tuned (Freitag and Al-Onaizan, 2016) to a new language. We chose Italian as the new language as the test set newstest2009 is multi-way in Czech, English, French, German, Italian and Spanish and thus we can report BLEU scores between all language pairs. We run two experiments with two different sets of fine-tuning data. First, we fine-tuned the cM-NMT model (  Scaling cMNMT: 12,432 Language Pairs We run additional experiments on a 112 language inhouse dataset (Arivazhagan et al., 2019b) to see if our approach scales to 12,432 language pairs. Our in-house dataset does not only contain more languages than the WMT setup, but also has a much wider range of available training resources. While for the high resource languages, we have access to billions of training examples, most of the low resource languages have less than 1 million training examples. We refer the reader to the description in Arivazhagan et al. (2019b) for more details regarding the dataset. Figure 5 shows the training data sizes and the average translations per multi-way example. Although a deeper and wider architecture does improve the quality of multilingual models for this dataset, we use the same experimental setup as used in our WMT experiments (see Section 4) to run an MNMT and cMNMT model on our in-house data. Experimental results can be seen in Table 13. cMNMT outperforms MNMT for non-English languages by 10.1 BLEU points on average while keeping the translation quality for language pairs that include English as source or target. These results demonstrate that our proposed approach does scale far behind the six language WMT setup. en→ x x →en x → y all +0.34 -0.05 +10.1 low resource +0.23 -0.15 +8.82 mid resource +0.35 -0.05 +11.02 high resource +0.45 +0.04 +9.73

Related Work
Direct models To translate between languages with little training data, three general approaches emerged, i. bridging through a third language (pivot-based MT) (Cheng et al., 2016;Currey and Heafield, 2019), ii. generating pseudo-parallel data between direct language pairs and training the direct pairs with that (zero-resource MT) (Firat et al., 2016b; and, iii. zero-shot methods where the model is asked to translate a direct pair only at test time (Johnson et al., 2017;Ha et al., 2016;Arivazhagan et al., 2019a).
Although pivot-based approaches perform sufficiently good when cascaded with strong bilingual models (Gu et al., 2019), their practicality is limited due to compounding errors from pipelining and doubled inference cost. The zero-resource approaches, combined with iterative-back translation (Hoang et al., 2018) are quite powerful but their inefficiency is worth noting. For N languages, one needs to devise a training routine that could sample N 2 − N pairs, generate pseudo-parallel data. The added time to generate pseudo-parallel data for every pair grows quadratically, making it challenging for systems considering a large number of languages. Recently, by devising a practical sub-sampling approach, (Zhang et al., 2020) demonstrated zero-resource techniques could be scaled to massively multilingual setup. We find the study by (Zhang et al., 2020) closest to our work, having the goal of any-to-any multilingual translation. But compared to sampling language pairs with no parallel data and generating pseudoparallel data on-the-fly, our approach makes use of existing multi-way alignment information before training. Lastly, zero-shot approaches attempt to measure the generalization performance of the MNMT models, but to date, the zero-shot quality still trails behind the pivot and zero-resource methods (Al-Shedivat and Parikh, 2019). Our proposed cMNMT, naturally fills the gap between these three approaches, the multi-way data can be extracted offline, and efficiently be mixed with the original data using a hierarchical data sampler. It does not require extra steps to generate pseudo-parallel data, and (as expected) it handily outperforms zero-shot approaches.
N-way data In this paper, we only made use of multi-way aligned data to sample bilingual pairs out of it. But there exist several approaches that make use of the multi-view structure in the data, such as Dabre et al. (2019), who explored the use of small multi-parallel corpora a for one-tomany NMT. Another approach is multi-source NMT (Zoph and Knight, 2016). Although multisource NMT is a promising direction, it has practical problems such as lacking multiple sources at inference time (Nishimura et al., 2018). We believe research in this direction will be the key to improve mid/high-resource NMT and address several robustness issues to the input noise. Aulamo et al. (2020) recently released MultiParaCrawl where the authors extracted direct data for non-English lan-guage pairs from the English-centric Paracrawl corpus.
Sampling scheduling Several approaches proposed to address data sampling for multi-task models, some relying on temperature-based heuristics (Lee et al., 2016;Devlin et al., 2018;Arivazhagan et al., 2019b), others relying on adaptive schedules that incorporate the model gains, baselines or quality expectations into the data schedulers (Kiperwasser and Ballesteros, 2018;Jean et al., 2019;Wang et al., 2020). We believe data sampling is a critical research area for not only MNMT but also multi-task learning in general. We reveal a critical failure mode of the commonly used temperature sampling strategy, and how it causes the poor translation quality while translating out of English.

Conclusion
In this work, we introduced complete Multilingual Neural Machine Translation (cMNMT) that exploits the multi-way alignment information in the underlying training data to improve translation quality for language pairs where training data is scared or not available. Standard MNMT models are trained on a joint set of different training corpora for a variety of language pairs. cM-NMT combines the different corpora and constructs multi-way aligned training examples that consist of translations of the same sentence into multiple languages. In combination with a novel temperaturebased sampling approach that is conditioned on the target language only, we show that cMNMT is superior to the standard MNMT model and the even better-performing bridging approach. Experimental results on a public WMT 30 language pairs dataset and an in-house 12,432 language pairs dataset demonstrated an average BLEU increase of more than 10 BLEU points for non-English language pairs. This approach leads to a single NMT model that can serve 12,432k language pairs with reasonable quality which also surpasses the translation quality of the bridging approach, which is nowadays used in most modern MT services.