{'fraction_non_alphanumeric': 0.0347985347985348, 'fraction_numerical': 0.014811275680840898, 'mean_word_length': 4.755728689275894, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
NICT‘s Submission To WAT 2020: How Effective Are Simple Many-To-Many Neural Machine Translation Models?

In this paper we describe our team‘s (NICT-5) Neural Machine Translation (NMT) models whose translations were submitted to shared tasks of the 7th Workshop on Asian Translation. We participated in the Indic language multilingual sub-task as well as the NICT-SAP multilingual multi-domain sub-task. We focused on naive many-to-many NMT models which gave reasonable translation quality despite their simplicity. Our observations are twofold: (a.) Many-to-many models suffer from a lack of consistency where the translation quality for some language pairs is very good but for some others it is terrible when compared against one-to-many and many-to-one baselines. (b.) Oversampling smaller corpora does not necessarily give the best translation quality for the language pair associated with that pair.


Introduction
Neural machine translation (NMT) (Sutskever et al., 2014;Bahdanau et al., 2014) is an end-to-end machine translation (MT) modeling approach that is known to give state-of-the-art translations for a variety of language pairs. Although, it is known to work particularly well for language pairs with an abundance of parallel corpora it tends to perform rather poorly for language pairs that lack large parallel corpora. Fortunately, multilingual neural machine translation (MNMT)  methods can be applied in order to significantly improve the translation quality for such language pairs. The underlying reason for improvement is that sharing parameters among several language pairs enables transfer learning which is proven to improve translation quality regardless of the language pair.
For the 7 th Workshop on Asian Translation (WAT, 2020), our team (NICT-5) decided to focus on the Indic languages task and the NICT-SAP task. Both tasks showcase resource poor Indic and South-East Asian Languages and thus multilingual NMT solutions can be applied to great effect in these tasks. It is common to train one-to-many or many-to-one NMT models for multilingual tasks but many-to-many models are often not showcased. This sparked out curiosity and we decided to investigate how well a many-to-many model would perform in the case of the two sub-tasks we chose.
The many-to-many models we trained used the Transformer (Vaswani et al., 2017) architecture using the simple black-box token-prepending technique (Johnson et al., 2017). In essence we simply concatenated the individual parallel corpora while prepending an artificial token such as 2xx where xx indicates the target language. Typically, the smallest corpus in the multilingual dataset is oversampled to match the size of the largest one but we tried settings with and without oversampling. Furthermore, following (Chu et al., 2017) we additionally prepended source sentences with tokens such as 2dom where dom indicates the domain of the corpus. We only did this when we knew that the test (and train) sets would involve multiple domains. An evaluation of our models showed that their performance is not consistent because they sometimes outperform the one-to-many and manyto-one models and sometimes underperform them. Furthermore the performance also depends on the language pair. As a secondary observation, we noticed that when parallel corpora sizes are not too different, oversampling smaller corpora negatively affects the final translation quality. We hope that our many-to-many models will serve as baselines which can be significantly improved upon in the future. Although there are many-to-one and oneto-many models that may be better, many-to-many models have zero-shot translation (Johnson et al., 2017) capabilities and thus should be focused on in the interest of a one-for-all NMT model.

Related Work
Our work in this paper focuses on neural machine translation and multilingualism.
Neural machine translation (NMT) (Sutskever et al., 2014;Bahdanau et al., 2014) is now the defacto machine translation paradigm that is used in research as well as engineering applications. While the initial architectures used recurrent neural networks, the more recent architectures use self-attention and feed-forward networks (Vaswani et al., 2017) which enable faster training and decoding. The main advantage of NMT is that the translation models are small (can fit on low-memory and low-computation devices) and the training approach is end-to-end (rather than modular). Large translation models and modular (multiple components) design were the key features of the predecessor of NMT aka statistical machine translation (SMT) (Koehn et al., 2007) which involved error compounding as the input sentences were processed by multiple components that were prone to making mistakes.
Another advantage of NMT is that its inner working is non-symbolic which enables it to incorporate multiple languages without any need to modify the basic architecture. While we use the multilingual NMT approach proposed by (Johnson et al., 2017) for multilingualism and the derivative multidomain NMT approach by (Chu et al., 2017) we refer readers to  and (Chu and Wang, 2018) for overviews on multilingualism and domain-adaptation, respectively. We do not describe the multilingual or multi-domain NMT modeling techniques in this paper as they are the same as described in Johnson et al. (2017) and Chu et al. (2017).
Specific to WAT, multilingual multi-domain approaches have been shown to improve translation quality for low-resource languages (Banerjee et al., 2018;Dabre et al., 2018;Philip et al., 2018).

Experiments
In this section we describe the tasks, datasets, implementation details, evaluation methodology and actual models trained.

Tasks
We participated in the NICT-SAP and Indic multilingual tasks. Our team name is "NICT-5". The NICT-SAP task involves two domains: Wikinews and Software Documentation (loosely speaking a   (En). For both tasks, the objective was to train a single multilingual and multi-domain NMT model. The desired models could be one-to-many, many-to-one or many-tomany. English is either the source or the target language for both tasks.

Datasets
We used some corpora from the many listed in the official task descriptions 12 . In particular the following parallel corpora were used: NICT-SAP Task: We used parallel corpora from the Asian Language Treebank (ALT) (Thu et al., 2016), KDE, GNOME and Ubuntu. The last three corpora were taken from OPUS 3 . Where the ALT corpus is for the ALT domain test set domain 4 , the other three are for the IT or Software Documentation domain 5 (Buschbeck and Exel, 2020). Detailed statistics for corpora can be found in Table 1.  dataset 7 . Detailed statistics for corpora can be found in Table 2.
With the exception of character splitting Thai, we do not perform any explicit pre-processing of any of the corpora used.

Implementation and Models Trained
We performed necessary modifications to the ten-sor2tensor v1.14 8 implementation of the transformer model. Tensor2tensor has an internal subword segmentation and we chose the option to train separate subword vocabularies of size 32,000. As we only train many-to-many models, our vocabularies are multilingual. We modified the original code to enable oversampling of smaller corpora during data pre-processing. We also modified the code to prepend the source sentences with a token 2xx to indicate the target language to be generated where xx is one of en, bg, hi, mr, ml, ta, te, gu, th, ms, id as applicable. Additionally for the NICT-SAP task, we prepend the source sentences with a token like 2it or 2alt to distinguish between the IT and Wikinews domains.
As for the models trained, we trained transformer big models on single Tesla V100 GPUs using the hyperparameter settings corresponding to "transformer big single gpu". Some important hyperparameters are: 6-layer encoder and decoder models with 16 attention heads, 1024-4096 hidden-filter sizes. We trained the models till convergence on development set BLEU score (Papineni et al., 2002). The development set BLEU score is the average of the BLEU scores of individual language pairs pairs. Evaluation on development set is done every 1000 batches (of 2048 tokens) and training stops when the BLEU score does not improve for 10 consecutive evaluations. Before evaluation, the model 7 http://preon.iiit.ac.in/˜jerin/ resources/datasets/pib_v0.2.tar 8 https://github.com/tensorflow/ tensor2tensor/ parameters are saved as a checkpoint and the last 10 checkpoints are averaged to give a single model which is then decoded using a beam of size 4 and a length penalty of 0.6 9 .
We trained a total of 4 models, 2 models per task; one with oversampling the smaller corpora to match the size of the largest corpora and one without.

Results
Tables 3 and 4 contain results for the NICT-SAP and Indic tasks for translation to and from English. We primarily report BLEU scores for our translations and mark scores that are either better than or not better than (check captions) the organizers translations. We used the same data as the organizers did. The organizers trained one-to-many or many-to-one models whereas we only trained many-to-many-models. For other scores such as RIBES, JPO adequacy and AM-FM scores kindly check the workshop overview paper (Nakazawa et al., 2020) or the evaluation page 10 as applicable.

NICT-SAP results
From table 3, it is clear that our many-to-many models outperform the organizer's one-to-many or many-to-one models. Upon further investigation it seems that our models are better for the IT domain translation. Furthermore, models with and without oversampling of smaller corpora do not exhibit significant difference in performance for the IT domain (in most cases) but in the case of the Wikinews domain, models without oversampling tend to be significantly better (in most cases). Apart from Thai and Hindi to English translation, all other translation directions show reasonable BLEU scores indicating that multilingual NMT models  Table 3: Results of our single multi-domain (ALT and IT) many-to-many NMT models for the NICT-SAP task. We indicate in the column labelled "Os" whether oversampling of the smaller corpora was done. Scores marked with "*" are those that do not beat organizers baselines.  Table 4: Results of our single many-to-many NMT models for the Indic task. We indicate in the column labelled "Os" whether oversampling of the smaller corpora was done. Scores marked with "+" are those that beat organizers baselines.

Translation Direction
are a reasonable solution for the involved language pairs. As for the reasons why Thai to English translation quality is poor (below or around 15 BLEU), it is clear that the parallel corpus for that pair is the smallest which has a strong negative impact. Table 4 presents results that are rather disappointing. Most of our translations were unable to beat those of the organizers'. However, there might be a simple explanation for this. Note that the Indic task involves almost twice as many translation directions as the NICT-SAP task. We used big transformer models for both tasks and so, the problem is not representation capacity but rather a lack of it. This lack of capacity likely comes from our naive multilingual solution coupled with rather small parallel corpora for each language pair. Future work will focus on expanding the amount of parallel corpora via popular techniques such as backtranslation (Sennrich et al., 2016). Another observation, just as in several cases of the ALT domain translations of the NICT-SAP task, we see that using oversampling is detrimental to translation quality.

Conclusion
In this paper we have described our submissions to the NICT-SAP and Indic multilingual tasks in WAT 2020. In general our many-to-many models have mixed performance where they sometimes outperform one-to-many or many-to-one models (organzier's models) and sometimes do not. Furthermore we observed that our models trained without oversampling smaller corpora tended to perform better than their counterparts that used oversampling of smaller corpora. This is especially true when parallel corpora for those pairs/domains contain approximately 100,000 sentences or fewer. This shows that investigation into improved data balancing methods might be necessary rather than relying on naive approaches we used in this paper. In the future we hope to improve upon our results by leveraging sophisticated methods involving better corpora balancing and monolingual corpora through back-translation.