{'fraction_non_alphanumeric': 0.04072140974626634, 'fraction_numerical': 0.029404165159423287, 'mean_word_length': 4.617416545718433, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
To Optimize, or Not to Optimize, That Is the Question: TelU-KU Models for WMT21 Large-Scale Multilingual Machine Translation

We describe TelU-KU models of large-scale multilingual machine translation for five Southeast Asian languages: Javanese, Indonesian, Malay, Tagalog, Tamil, and English. We explore a variation of hyperparameters of flores101_mm100_175M model using random search with 10% of datasets to improve BLEU scores of all thirty language pairs. We submitted two models, TelU-KU-175M and TelU-KU- 175M_HPO, with average BLEU scores of 12.46 and 13.19, respectively. Our models show improvement in most language pairs after optimizing the hyperparameters. We also identified three language pairs that obtained a BLEU score of more than 15 while using less than 70 sentences of the training dataset: Indonesian-Tagalog, Tagalog-Indonesian, and Malay-Tagalog.


Introduction
This paper describes our participation in the WMT21 shared task of large-scale multilingual machine translation. Specifically, we chose small track #2, which involves thirty language pairs, and used the neural machine translation (NMT) method. We call our models TelU-KU (Telkom University -Kumamoto University) as we use our university name in our submissions.
NMT has been widely used in machine translation research for many languages. Currently NMT has become the state of the art of machine translation with a large number of parallel corpus (Bojar et al., 2017;Nakazawa et al., 2017;Chu and Wang, 2018;Sutskever et al., 2014). Meanwhile, for low resources cases, the NMT tends to give poor translation results (Duh et al., 2013;Sennrich and Zhang, 2019;Zoph et al., 2016;Koehn and Knowles, 2017). In order to get better translation results of NMT for low resources languages, some approaches are applied, such as using a large number of monolingual corpora (Artetxe et al., 2018a,b;Lample et al., 2018b,a), applying transfer learning approach to share lexical and sentence level representation (Gu et al., 2018), using sub-word representation (Durrani et al., 2019), and hyperparameter optimization (HPO) (Sennrich and Zhang, 2019;Rubino et al., 2020).
HPO is an important part of building an NMT system in many real-world applications. In other words, selecting effective hyperparameters is critical to building a strong NMT system. However, in many cases, hyperparameters are often set manually based on intuition and heuristics mechanisms, tedious and error-prone processes that can lead to unreliable experimental results and poor performance of shared tasks or production systems. This is because HPO requires rigorous testing and resources, which makes it a high-cost process. To deal with this problem, tablelookup has been proposed as a benchmark procedure (Zhang and Duh, 2020). Their study provides evaluation protocols and a benchmark dataset for comparing the HPO methods. Moreover, like other NMT models, transformers require setting various hyperparameters, but researchers often use default parameters, even when their data conditions differ substantially from the data conditions previously used to determine those default values.
In low-resource languages cases, the performance of the transformer is highly dependent on the hyperparameter settings (Araabi and Monz, 2020). The experimental results show that the best-suited combination of hyperparameters and regularization methods can produce substantial improvement for low-resource languages data. On the other side, grid search and manual search are the most frequently used strategies for HPO. However, according to the experiment, random search is actually better than grid search in several conditions (Bergstra and Bengio, 2012). Random searches are actually better suited to running on a cluster of computers than grid searches when a group of computers fails. Random search also allows the experimenter to change the "resolution" on the fly. In addition, they have advantages in high-dimensional searching spaces.
In this work, we experimented with two models, that is, TelU-KU-175M and TelU-KU-175M_HPO. Both models are based on a pre-trained model of flores101_mm100_175M. The TelU-KU-175M is our model that manually fine-tuning the hyperparameters, whereas the TelU-KU-175M_HPO is based on hyperparameter optimization. We used a random search method while using 10% of datasets to find the best hyperparameter optimization. In addition, we also included the M2M-100 175M model to compare with our results. This model uses the same pre-trained model as ours but without fine-tuning. Fine-tuning is a common practice in NLP to train a pretrained model for several epochs on a downstream dataset and has proven to improve performance.
Our experimental results show improvement in most language pairs after optimizing the hyperparameters. The TelU-KU-175M is able to improves the average BLEU scores by 0.35-0.59 over M2M-100 175M. Meanwhile, the TelU-KU-175M_HPO improve the scores by 1.08-1.41 over the baseline. We also identified three language pairs that obtained a BLEU score of more than 15 while using less than 70 sentences of the training dataset: Id-Tl, Tl-Id, and Ms-Tl. This paper is organized as follows. Section 2 explains the experiment. Section 3 shows the obtained results. Section 4 discusses the effect of HPO and the multilingual model. Section 5 provides the conclusion and future direction of this work.

Experiments
In this section, we first describe languages overview of the Southeast Asian language. Then, we discuss data and preprocessing. Finally, we discuss the model and architecture of our model submission.
Indonesian and Malay are considered closely related languages due to being mutually intelligible in morphology, and both languages belong to the Malayo-Polynesian language family (Susanto et al., 2012). The base of formal Indonesian is from Malayo-Riau (Abas, 1987). The main difference is the influence of the vocabulary. Indonesian is largely influenced by Dutch, whereas English influences Malay. The Tagalog language has the same language family as Indonesian and Malay. However, it has different morphology characteristics, and the vocabulary is influenced by several countries, such as Spain, America, and Malay. Javanese is one of the Indonesian ethnic languages used by more than 42% of Indonesia's population, mostly from the central and eastern parts of Java (Novitasari et al., 2020). Javanese is also used in Suriname and New Caledonia. Currently, the Javanese is influenced by Indonesian. This is because Indonesian is used in formal documents and as a daily conversation. Last, Tamil belongs to Dravidian, a unique family where the language is mostly spoken in a southern state (Tamil Nadu) of India (Kumar and Singh, 2019). Table 1 shows the general characteristic of the selected languages.

Data and preprocessing
We used a dataset provided by the WMT21 organizers. Thus, our system was considered a constrained system. We used three types of datasets, that is, training, evaluation, and hidden test datasets. The training dataset is a parallel corpus from Opus monolingual and Wikipedia, as shown in Table 2. The evaluation dataset is a parallel corpus from Flores101 (Goyal et al., 2021). The evaluation dataset consists of two evaluations, that is, dev and devtest, as much as, 997 and 1,012 sentences, respectively. Last, the hidden test dataset is an unknown parallel corpus provided by the organizer through the Dynabench leaderboard. 1 We tokenized all the training and evaluation datasets by SentencePiece tokenizer (Sennrich et al., 2016). This tokenizer is an unsupervised text tokenizer and detokenizer, where the vocabulary size is predetermined prior to the neural model training. We preprocessed dataset according to the guideline, 2 that is, encode and binarize.

Models & Architectures
We use an NMT system with big Transformer architecture (Ng et al., 2019;Vaswani et al., 2017)  ory. 3 We experimented with the following two models: • TelU-KU-175M is a pre-trained flo-res101_mm100_175M with fine-tuning. We manually tune the hyperparameters, as shown in Table 3, column 4.
• TelU-KU-175M_HPO is a pre-trained flores101_mm100_175M with HPO. The hyperparameters and their ranges are shown in Table 3. Some of these hyperparameters are based on (Ravikumar, 2020). Figure 1 shows the logical flow of our approach. We run 30 iterations of random searches for two epochs. Due to costly training, we only run the optimization using only 10% training, dev, and devtest. From those 30 models, we select the best model based on the results from the devtest. Then, we use the hyperparameter from the best models to fine-tune the flo-res101_mm100_175M model. The hyperparameter optimization results are shown in Table 3, column 5.

Results
We evaluate the generated texts of our models using the sentence-piece BLEU (spBLEU). The spBLEU uses a SentencePiece tokenizer with 256k tokens, and then the BLEU score is computed on the SentencePiece tokenized text. The results are shown in Table 4.   We also compare our results with the pretrained model without fine-tuning (M2M-100 175M) as the baseline. Our models: the TelU-KU-175M and TelU-KU-175M_HPO, improved the BLEU scores on 16-17 language pairs over the M2M-100 175M model. In terms of the improvement in each language pair, TelU-KU-175M and TelU-KU-175M_HPO improved BLEU scores by 0.04-5.39 and 0.01-12.22, respectively. The BLEU scores also decreased on several language pairs between 0.1 to 3.25 and 0.02 to 6.5 for TelU-KU-175M and TelU-KU-175M_HPO, respectively.

Discussion
This section discusses the effect of HPO and NMT of the multilingual model against our models' evaluation results.

Effect of HPO
The main objective of HPO in this task is to explore a high-dimensional search space in NMT. As we mentioned in Section 2.3, we run HPO using random search for 30 iterations using 10% of the dataset. The best hyperparameter values were determined based on devtest. The best configurations based on the average BLEU scores for each iteration were saved for running the pretrained model, M2M-100 175M, using full datasets (TelU-KU-175M_HPO). The detail of the best configurations is shown in Table 3. The evaluation results were conducted by translating the dev, devtest, and test datasets. We show our detailed evaluation results in Table 4, while Table 5 is our average evaluation among other participants.
We found that fine-tuning of 10% dataset does not lead to the best results for all language pairs translation compared to the manual hyperparameter tuning (TelU-KU-175M) and fine-tuning using 100% dataset (TelU-KU-175M_HPO). The best HPO using 10% of datasets resulted in an average BLEU of 12.75 for dev, 12.33 for devtest, and 12.39 for test datasets. Nevertheless, these values are higher compared to the baseline (M2M-100 175M) in Table 5. This means that fine-tuning with only 10% dataset using a basic method, random search (Bergstra and Bengio, 2012), indeed increases the BLEU scores.  The BLEU scores improved even more after using the full dataset with the same hyperparameter values (TelU-KU-175M_HPO). This means that the number of datasets influences the performance. We left for future work discussing the effect of the number of datasets in HPO for NMT.
We also study the hyperparameter importance of all optimized hyperparameters using Hyanova 4 , a python implementation of a functional analysis of variance (fANOVA) algorithm (Hutter et al., 2014). The algorithm partitions the observed variation of a response value into components against its inputs (Klein and Hutter, 2019). In this study, the response value is the BLEU score, while hyperparameters are the inputs. The higher the fANOVA values, the more important the hyperparameter. Table 6 shows that LR is the most important hyperparameter, while BS is the least important. This means that the LR influence the achieved BLEU scores. From our observation, within our selected range in Table 3, the higher the learning rate, the higher the average BLEU score. Therefore, it is important to tune the LR within a higher range.
Furthermore, we investigate the statistical significance across language pairs from Table 4 using Wilcoxon signed-rank test with α = 0.05. We show the p-values of all models in Table 7. Unfortunately, all the results demonstrate statistically non-significant as all of the p-values were more than 0.05. Although the average BLEU can be increased by optimizing the hyperparameter values, this finding shows that HPO might not contribute much to the performance.
One of the possible causes is the utilization of random search, which is categorized as an uninformed search. This category does not learn from previous results, and therefore, each solution is independent of the other. Moreover, uninformed search is proven to be inferior to the informed search, i.e.,   bayesian optimization or evolutionary algorithm (Fatyanosa andAritsugi, 2020, 2021). This work only calculates the statistical significance across language pairs and leaves the calculation per language pair for future studies.

Effect of multilingual model
The TelU-KU-175M and TelU-KU-175M_HPO models produced 16-17 language pairs that have higher BLEU scores compared to M2M-100 175M, as shown in Table 4 with blue colors. Among them, we identified seven language pairs that obtained a BLEU score below 10: Id-Ta, Ta-Id, Ta-Jv, Ms-Ta, Ta-Ms, Ta-Tl, and Jv-Tl. Most of these language pairs were related to the Tamil language. We found that most of the Tamil translation results had an English sentence as unknown words (see Tables 8 and 9 in appendix). The translation results leading to not the same as a reference file. As a result, these language pairs had a lower BLEU score.
Surprisingly, we identified three language pairs that obtained a BLEU score of more than 15: Id-Tl, Tl-Id, and Ms-Tl, while using less than 70 sentences of the training dataset. This could be because of the attention mechanism in NMT of multilingual models. The attention mechanism, which was initially called a softalignment model in (Bahdanau et al., 2015), aligns a source phrase to a target word. Training this attention-based model is done by maximizing the conditional log-likelihood. After training, the model can do translation from any of the source languages to any of the target languages included in the parallel training corpora (Firat et al., 2016).
The Id-Tl, for example, obtained a BLEU score of 17.94 using only 56 sentences of training datasets. The NMT system that trained with fewer training datasets, e.g., below 1M, usually obtained lower BLEU scores. However, the Id-Tl results indicated that this language pair obtained an advantage from the attention mechanism of the multilingual model using 30 language pairs. In this study, these 30 language pairs are considered as low-resource languages and mostly have the same language family. Table 2 shows that our models used a small number of training datasets, e.g., below 1M, in all language pairs. Whereas, Table 1 shows that most languages have the same language family: Malayo-Polynesian. Therefore, we argue that low-resource language with the same language family should be considered in the NMT of the multilingual model. For example, if we want to improve the Tamil language performance, we should considered to add other languages with the same (or closely) language family as Tamil, e.g., Kannada, Bengali, Hindi.

Conclusion
We described our team submission for WMT21. Our results show improvement in most language pairs after optimizing the hyperparameters. Furthermore, we also found three language pairs that obtained a BLEU score of more than 15 while using less than 70 sentences of the training dataset. In this study, we used 30 language pairs that are considered as lowresource language and mostly have the same language family. This result indicated that low-resource language with the same language family should be considered in the NMT of the multilingual model.
As future work, we plan to use a more sophisticated optimization algorithm, specifically informed searches such as bayesian optimization or evolutionary algorithm. Additionally, we want to try other percentages of the optimized dataset to see the effect of the number of training data on the performance. We also plan to use a specific tokenizer for Tamil, e.g., Indic NLP library (Kunchukuttan, 2020), iNLTK (Arora, 2020). The Tamil language needs a particular pre-processing due to its writing system that differs from other languages. Last, we plan to clean the dataset in pre-processing steps, considering that the dataset used in this work is noisy. We expect this will maximize the attention mechanism in the NMT of a multilingual model. Therefore, our model could produce better translation results.

Lang pairs
Model Text

English Source
We now have 4-month-old mice that are non-diabetic that used to be diabetic, he added.

TelU-KU-175M
We now have 4-month-old mice that are non-diabetic that are used to be diabetes, he added. TelU-KU- 175M_HPO We now have 4-month-old mice that are non-diabetic that used to be diabetic, he added.

Id-En
Reference We now have 4-month-old mice that are non-diabetic that used to be diabetic, he added.

M2M-100_175M
"Now there is a age of 4 months of non-diabetes that has long been diabetes," she said.

TelU-KU-175M
This day there is a 4 months old nondiabetes who had previously been diabetes, his debut.

TelU-KU-175M_HPO
In this period there is a four-month-old nondiabetes that was formerly diabetes, his tambahnya.

TelU-KU-175M
This time there is a four-month-old nondiabetes that had previously been diabetes, his debut.

TelU-KU-175M_HPO
This time there is a four-month-old nondiabetes that was formerly diabetes, his tambahnya.

Jv-En
Reference We now have 4-month-old mice that are non-diabetic that used to be diabetic, he added.

M2M-100_175M
Now we have four-year-old age, which are not diagnosed with diabetes, diabetes.

TelU-KU-175M
We have four-year-old children who have diabetes, ujare. TelU-KU-175M_HPO Today we have a four-year-old mice that is not-to-be-diabetes who now has diabetes, then.

Ms-En
Reference We now have 4-month-old mice that are non-diabetic that used to be diabetic, he added.

M2M-100_175M
"We now have a four-month old baby that has no diabetes," he added. TelU-KU-175M We now have a four-month boy that has no diabetes, he added. TelU-KU- 175M_HPO We now have a four-month-old mice child who has no diabetes, he added.

Ms-Ta
Reference

Ta-En
Reference We now have 4-month-old mice that are non-diabetic that used to be diabetic, he added.

M2M-100_175M
He said, We have now one of the four-year-old rolls, and it was not until it was long," he further said. TelU-KU-175M He said to me, I have a four-thirds light now, but it was not until it was lost." TelU-KU-175M_HPO He said, "I am now one of the four-member elephants, and that it was not until it had been passed."

Tl-En
Reference We now have 4-month-old mice that are non-diabetic that used to be diabetic, he added.

M2M-100_175M
We have 4 small areas that do not have diabetic dating diabetic, he said. TelU-KU-175M May we have 4 new gullies that are not diabetic dating diabetesic, he says. TelU-KU-175M_HPO There are four-year-old, non-diabetic, former diabetic, he added.

Tl-Ta
Reference