{'fraction_non_alphanumeric': 0.030309544282029234, 'fraction_numerical': 0.014545715104614504, 'mean_word_length': 4.847088395475493, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Transfer Learning with Shallow Decoders: BSC at WMT2021’s Multilingual Low-Resource Translation for Indo-European Languages Shared Task

This paper describes the participation of the BSC team in the WMT2021’s Multilingual Low-Resource Translation for Indo-European Languages Shared Task. The system aims to solve the Subtask 2: Wikipedia cultural heritage articles, which involves translation in four Romance languages: Catalan, Italian, Occitan and Romanian. The submitted system is a multilingual semi-supervised machine translation model. It is based on a pre-trained language model, namely XLM-RoBERTa, that is later fine-tuned with parallel data obtained mostly from OPUS. Unlike other works, we only use XLM to initialize the encoder and randomly initialize a shallow decoder. The reported results are robust and perform well for all tested languages.


Introduction
We present the work carried out by the BSC Team in the context of WMT2021's first edition of the Multilingual Low-Resource Translation Shared Task. The tasks addresses the issue of multilinguality in machine translation (MT) for lowresource languages, focusing on two language families: North Germanic and Romance. We take part in the Subtask 2, which involves translation in four Romance languages: Catalan, Italian, Occitan and Romanian.

Background
Machine translation for low-resource languages is characterised by the lack of sufficient parallel data of a given language pair, either because the combination is infrequent or because the languages involved are themselves low-resource. Several works have attempted to overcome this pitfall, using different techniques. A common solution is to employ back-translation (Sennrich et al., 2016), while other research focuses on using other languages as pivots to compensate for the lack of data (Firat et al., 2016;Zoph et al., 2016). Artetxe et al. (2018); Lample et al. (2018) make use of monolingual data only.
Our approach is based on multilinguality. Previous works such as Vergés Boncompte and Ruiz Costa-Jussà (2020); Tubay and Costa-Jussa (2018) have shown that the use of multilingual MT is beneficial, as it generalizes better by sharing parameters among all the languages involved, especially if the languages belong to the same linguistic family. At the same time, training of multilingual MT models from scratch usually requires large parallel corpora and may not be feasible in a low-resource and zeroresource translation scenarios.
Pre-training of large language models from scratch on monolingual data and then fine-tuning them for the specific downstream tasks, has proved to be an extremely successful approach for many natural language processing problems. Crosslingual language models such as XLM and XLM-RoBERTa (Conneau and Lample, 2019;Conneau et al., 2020), that combine unsupervised (monolingual data) and supervised (parallel data) training objectives, perform especially well both on cross-lingual NLU tasks and in machine translation. The idea of combining the power of pre-trained cross-lingual language models with a multilingual machine translation setting naturally follows from there.
This idea was explored in  where a denoising seq2seq auto-encoder (mBART) based on BART  was pre-trained on extensive monolingual corpora in many languages. A similar approach is implemented in (Lin et al., 2020) where alignment information is used to pre-train a multilingual MT transformer on existing public parallel datasets. Both approaches require either a computationally intensive pre-training on monolingual data or access to extensive large-scale parallel data.
Initializing an encoder and decoder of a bilingual MT seq2seq transformer with a pre-trained crosslingual language model was famously proposed in (Conneau and Lample, 2019). A natural next step is to initialize a multilingual MT seq2seq transformer with a shared encoder and a shared decoder by the XLM-like language encoder which was first performed in (Ma et al., 2020).
We reuse this idea by initializing the encoder with a pre-trained XLM-Roberta (Conneau et al., 2020), as in (Ma et al., 2020). However, unlike (Ma et al., 2020), we only initialize the encoder, with the motivation of being able to instantiate a shallower decoder, following previous works that for a given compute budget suggest that it is more efficient to use deeper encoders and shallower decoders (Kasai et al., 2020). The encoder-only initialization was already implemented in Fairseq (Ott et al., 2019). 1 3 Experimental Framework

Fine-Tuning Data
To train our MT system, we use all parallel data available in OPUS 2 for the targeted language pairs, namely ca-it, ca-oc, ca-ro.
We further include a small dataset ca-oc, the Catalan -Occitan Gencat Crawling, specifically obtained for the occasion, by leveraging parallel data from a crawling of the Catalan Government Internet domains and subdomains. We use the Cor-pusCleaner 3 pipeline to process the WARC files obtained from the crawling. This allows us to maintain the metadata and retrieve the original url per each document. We then extract the content of the same URLs in both languages and align them at document level using vecalign 4 . The final dataset of 503 sentences was obtained by manually reviewing 1,237 automatically aligned sentences. Although smaller than expected, one motivation to crawl this brand new dataset is to contribute to the development of MT resources for Occitan, which is a severely under-resourced language. We are The resulting statistics of the corpora used to train our system can be seen in Table 1. As expected, the number of aligned sentences is much larger for Italian and Romanian as target languages, since Occitan is such a low-resource language. Nonetheless, we must bear in mind that Catalan and Occitan belong to the same sub-branch in the family tree of the Romance languages, as shown in Figure 1. Thus, their considerable typological closeness makes up for the reduced amount of aligned sentences available for this language pair.

Preprocessing
We start by preprocessing our data with a filtering and a tokenization step.
To ensure that there is no train-test overlap, we filter all of our training data by removing all sentences from the validation and test sets present in our train set. To build our system, we use SentencePiece BPE tokenization with the original shared vocabulary of 250,000 tokens of XLM-R model (Conneau et al., 2020;Kudo and Richardson, 2018), and we only keep sentences with a maximum size of 512 tokens.
The final number of parallel sentences used for training is shown in Table 1.

System Description
We base our system on XLM-RoBERTa (Conneau et al., 2020) and then fine-tune it with the collected parallel data. As described earlier, the seq2seq multilingual transformer with shared encoders, shared decoders and shared embedding tables is initialized by XLM-R BASE pretrained language model on the encoder side, whereas a shallow decoder of 3 layers is initialized randomly. Sharing of embedding tables for all directions (ca-it, it-ca, ca-ro, ro-ca, ca-oc, oc-ca) was initially implemented due   to memory constraints but eventually turned out to work well. The token indicating the required target language is prepended to the target sentences, thus the model is aware to what language it has to translate to, as in (Wu et al., 2016).
It is important to note that the data used to train XLM-RoBERTa does not contain any Occitan text, as can be seen in Table 2. Thus the only knowledge that the multilingual transformer has about the language directions including Occitan comes from the XLM-R language model being pre-trained on text in related languages, such as Catalan.
We use default Fairseq parameters for finetuning, first of all, the Adam optimizer (Kingma and Ba, 2017) with β 1 = 0.9 and β 2 = 0.98. The polynomial decay learning rate schedule starts from 5e−04, warmed up to over 1000 updates and gradually decays to 0 over around 60k updates. The model was fine-tuned for 2 days on 4 NVIDIA V100 GPUs. The final learning rate was around 3e−04 with a batch size of 3, 072 sentences.
During inference we use the beam search generation algorithm with a beam size of 5. Since the languages between which we are translating are typologically close, we do not assign any length penalty, and we use the best checkpoint for generating.

Results
Here we report the official evaluation. 6 We submitted our results a bit later than the deadline due to some bottlenecks in our in-house computational resources. Table 3 reports the results obtained by our system on the evaluation test set, together with the two official baselines provided by the organisers (M2M-100 and mt5-devFinetuned).
Out of 7 competing systems and 2 baselines, our system was ranked 5th in Average, 3rd in the Catalan-to-Occitan direction, 4th in the Catalanto-Romanian direction and 6th in the Catalan-to-Italian direction.

Human Evaluation
The organizers of the workshop have recently released the results of a human evaluation for the  language pairs ca-it and ca-oc. 7 A sentence level evaluation has been performed taking into account the document as context. Each sentence is evaluated in a Likert-like scale [1,5] answering the question of direct assessments. A second human evaluation is performed where 60 selected terms (mostly named entities, dates and locations) are annotated as being either well translated, not translated or mistranslated, by majority voting among the annotators. The results can be found in Tables 4 and 5, respectively.

Little sisters over big cousins
As seen in Table 3 the average results of our system are above both baselines, although it is the results for Catalan-Occitan that give the greater leverage, because in the other two scenarios M2M has a higher BLEU. Actually, the score for Catalan-Occitan is substantially higher than the score obtained for the other two pairs, although the finetuning data used in this model is, at least, ten times smaller than the data used in the other two models. These results are replicated in most of the other competing systems 8 . The reason for this apparent anomaly is clearly due to the linguistic similarity between the Catalan and Occitan, which in medieval times were practically one and the same language. This result confirms the intuition that when two languages are similar enough, less data is needed.
That said, we also hypothesize a positive impact of the curated dataset (Catalan -Occitan Gencat Crawling) added to the rest of parallel data obtained in the OPUS repository, but there is no definitive proof of it. Furthermore, we can also hypothesize that the presence of Spanish in the multilingual corpus, being a high-resource language and also 7 http://www.statmt.org/wmt21/ multilingualHeritage-translation-manual. html 8 http://statmt.org/wmt21/ multilingualHeritage-translation-results. html linguistically close to Catalan and Occitan (more so than to the other two Romance languages involved in the task), has a beneficial impact on the results. Indeed, low-resource languages can greatly benefit from their similarity to other languages present in the multilingual training. In such scenarios, less data can lead to satisfactory results, and with a smaller carbon print, since the models use less computational power for training.

Human Evaluation results
The human evaluation on two of the test sets shown in in table 4 validates the relative position of our system in the global ranking. Interestingly, human scores correlate well with BLEU for Catalan-Italian, and less well for Catalan-Occitan. In the latter case, human scores tend to be lower than the corresponding BLEU. The reason for this may again have to do with linguistic similarity between Catalan and Occitan: "Catalanish" Occitan may be deemed acceptable by subword-based BLEU, but not by human evaluators. The performance of our system as evaluated for term translation, shown in in table 5 is consistent with the other evaluations regarding the position of the system in the overall ranking.

Vocabulary
One of the shortcomings of our approach is the big vocabulary size (250k tokens), inherited from XLM. This big vocabulary size was required by XLM to cover a very diverse set of languages. However, this makes it sometimes challenging to fit the embedding tables in memory, which is especially inefficient taking into account that a large proportion of tokens are not used (since we focus on a tiny subset of languages). Thus, either pruning the vocabulary, or using pre-trained models specifically trained for the Romance languages family (with a reduced vocabulary size) would be better alternatives.

Shallow decoders and transfer learning
While recent works have suggested that allocating more computation to (deeper) encoders (Kasai et al., 2020) at the expense of allocating less computation to (shallower) decoders is more efficient, this approach is not yet standard in the machine translation literature, especially when applying transfer learning. This method has the advantage of not reusing pre-trained weights for the decoder, although a middle ground is perhaps worth exploring.  Namely, use just some of the pre-trained weights to initialize the decoder layers. For example reuse the first N layers of XLM in the decoder, even if there is no 1-to-1 mapping between layers because there are less in the fine-tuned model.

Conclusions
We have showed that our approach is a simple, yet effective method for multilingual machine translation between linguistically similar languages. The encoder-only initialization allows for having a shallow decoder, which is computationally wise. As future work, we plan to further explore transfer learning techniques in the context of shallow decoders as well as applying different vocabulary pruning techniques.

Code availability
We release 9 with an open license the scripts used for this work for the sake of reproducibility.