{'fraction_non_alphanumeric': 0.03302385055873686, 'fraction_numerical': 0.014176905542892089, 'mean_word_length': 4.966169154228855, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 2, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Multi-Domain Adaptation in Neural Machine Translation Through Multidimensional Tagging

While NMT has achieved remarkable results in the last 5 years, production systems come with strict quality requirements in arbitrarily niche domains that are not always adequately covered by readily available parallel corpora. This is typically addressed by training domain specific models, using fine-tuning methods and some variation of back-translation on top of in-domain monolingual corpora. However, industrial practitioners can rarely afford to focus on a single domain. A far more typical scenario includes a set of closely related, yet succinctly different sub-domains. At Booking.com, we need to translate property descriptions, user reviews, as well as messages, (for example those sent between a customer and an agent or property manager). An editor might need to translate articles across a set of different topics. An e-commerce platform would typically need to translate both the description of each item and the user generated content related to them. To this end, we propose MDT: a novel method to simultaneously fine-tune on several sub-domains by passing multidimensional sentence-level information to the model during training and inference. We show that MDT achieves results competitive to N specialist models each fine-tuned on a single constituent domain, while effectively serving all N sub-domains, therefore cutting development and maintenance costs by the same factor. Besides BLEU (industry standard automatic evaluation metric known to only weakly correlate with human judgement) we also report rigorous human evaluation results for all models and sub-domains as well as specific examples that better contextualise the performance of each model in terms of adequacy and fluency. To facilitate further research, we plan to make the code available upon acceptance.


Introduction
Neural machine translation (NMT) has achieved remarkable results in recent years. A strong testament to its success and efficacy is the increasingly widespread industrial adoption of NMT solutions Johnson et al. (2017); Levin et al. (2017a); Crego et al. (2016). Model parameter estimation in NMT architectures (Bahdanau et al., 2015;Gehring et al., 2017;Vaswani et al., 2017) is still largely a supervised learning problem which requires large amounts of translated sentence pairs (parallel data). Obviously, acquiring a sufficient number of high quality parallel sentences in order to train a functional domain-specific NMT system can be prohibitively expensive; especially, if one needs to develop such systems for several domains across different language pairs. On the other hand, large quantities of untranslated in-domain content (monolingual data) are often readily available.
Various domain adaptation strategies have been developed to address the low-resource setting of niche domains (Chu and Wang, 2018). Some of the more popular approaches involve Figure 1: Schematic diagram of MDT in our setting. We use generic parallel data to train a base source-target and a reverse target-source models. We then back-translate target language monolingual in-domain data using the reverse model, and mix it with upsampled indomain parallel data to fine-tune the base model. The data is tagged with two special tokens: <SYNTHETIC={0,1}>, and <DOMAIN={reviews,messaging,descriptions}>.
generating synthetic in-domain data with the help of existing monolingual corpora, and using that data to fine-tune the more general NMT systems Sennrich et al. (2016a).
In real-world scenarios practitioners often need to deploy translation engines for several closely related, yet different sub-domains. For example, an online travel marketplace needs to translate offering descriptions, user-generated reviews and customer service communications, all related to travel, but all having different linguistic nuances. This fragmentation is further compounded by the company's need to provide services across many distinct languages. It can be very expensive or outright impossible to develop and maintain separate translation pipelines for every combination of language and sub-domain.
We propose a new method for training models which are simultaneously fine-tuned on several closely related, yet succinctly different sub-domains. We show that those models achieve competitive (and often superior) results to single domain fine-tuned baselines while effectively serving N use cases, therefore cutting development and maintenance costs by a factor of N .

Related Work
Our work builds on a growing body of domain adaptation research, mainly related to fine-tuning through tagged back-translation.

Domain tagging
There are a number of research directions related to using tags (or special tokens) within NMT, primarily as a way to pass additional information to the model. Practically speaking, these are attractive approaches as they usually do not require any special modifications to off-theshelf translation software. The majority of use cases tag sentences on the source side: Kobus et al. (2017) Britz et al. (2017) prepends the domain tag to the training input on the target side, thus forcing the decoder to predict the domain based on the source sentence alone.

Back-Translation for Domain Adaptation
Back-translation (BT) is a form of semi-supervised learning that can be used to fine-tune both statistical Bertoldi and Federico (2009); Bojar and Tamchyna (2011) and neural (Sennrich et al., 2016a) machine translation models to new domains. The idea behind this technique is to augment limited parallel in-domain data with a synthetic corpus produced by translating monolingual data from the target language using a target-to-source translation system. A synthetic corpus produced via back-translation will have machine-generated source sentences "translated to" human-written in-domain targets. BT model fine-tuning then becomes a three-stage process: first, genuine parallel data is used to train a reverse model in the target-to-source direction; second, that reverse model is used to translate target-side in-domain monolingual data into the source language; third, synthetic data is used in combination with few truly parallel in-domain samples to fine-tune the base source-to-target model. This simple approach works surprisingly well in practice Bojar et al. (2018); Barrault et al. (2019). Recent research showed that the details of how we generate the synthetic BT data matter a lot (Edunov et al., 2018;Imamura et al., 2018). Specifically, the authors find that randomized sampling and noising is preferable to plain beam search. Edunov et al. (2018) hypothesise that the improvement is due to randomization contributing to the source-side diversification of the synthetic data. Caswell et al. (2019), on the other hand, suggest that synthetic data adds both helpful and harmful signals, which sampling and noising BT strategies help the model to separate. The TaggedBT technique which they introduce achieves competitive results by simply tagging synthetic data with a special token indicating that the data is machine-generated.

Multidimensional tagging
As discussed in Section 2, introducing special tokens in the training data has been independently useful at passing content-specific information (e.g. domain, voice, gender, etc.) and data-specific information (e.g. whether a given data point is synthetic). The current work extends this idea into the multidimensional setting. Whenever several meaningful dimensions describing the data are available at inference and training time, we can encode that information with special tokens indicating the values along each of the dimensions (Figure 1).  MDT (our method) is competitive (and on average superior) against the strong fine-tuning baseline (top10 from (Edunov et al., 2018)) despite having significantly lower training and deployment costs.
A real-world multi-domain adaptation setting lends itself very naturally to the MDT approach. For example, domain or topic is one such dimension, whether or not the data is synthetic is another. The definition of a synthetic sample may also differ between applications. Back-translation as used in this work is an obvious way of generating such samples, but so can be pseudo-alignment (Imankulova et al., 2017;Schwenk et al., 2019). A hybrid dataset may include samples from all three origins (genuine, machine translated and pseudo-aligned) and a tag can help the model differentiate between them. Lastly, multilingual models where the source languages are not trivially different, can be boosted with a language tag 1 . It is therefore clear that although our experiments only cover a two-dimensional setting with the attributes mentioned above (data domain and source), multidimensional tagging can be extended to cover other data aspects.

Experimental Setup
This section describes our data sources, model architecture, and synthetic data generation and mixing strategies that we employ in our experiments. Our principal goal is to evaluate MDT fine-tuning approach as a scalable alternative to state-of-the-art domain fine-tuning for NMT.

Data
We run our experiments on three language pairs (Arabic-English, German-English and Russian-English) which span three different scripts. Our parallel data sources include a large generic corpus which is a mixture of publicly available and in-house data 2 , as well as three much smaller domain-specific parallel datasets (Table 1). The monolingual data which we use to create back-translated models contains 1M proprietary text segments for each language and domain. All three domains ("Reviews", "Messaging" and "Descriptions") are travel-related, and in fact could be considered as sub-domains of a more general "Travel" domain. Nevertheless, they all exhibit distinct linguistic characteristics which makes it challenging to treat them as a single domain. Appendix C provides examples of sentences from different data sources.

Synthetic data generation
We generate all synthetic data using a target-source reverse model trained purely on the generic parallel corpus. According to prior experiments we found top10 3 method from Edunov et al. (2018) to be the best-performing domain adaptation method, and we use it as the main approach to benchmark against. Because we do have limited in-domain parallel data, our fine-tuning parallel data is not purely synthetic, but a mix of synthetic and genuine (which we upsample to reach 1:1 composition).
top10 Following Edunov et al. (2018) we use our reverse target-source models to translate monolingual data back to English, but at the generation stage we sample from the next token distribution instead of using beam search to approximate MAP translation. At each sampling step we only consider top 10 most probable candidates.
MDT As described in Section 3, we extend the idea of tagged BT Caswell et al. (2019) to multi-attribute setting by prepending source-side tags which qualify various aspects of the data. Specifically, in this experiment we tag the data according to two characteristics: (1) whether it is synthetically generated or genuine, (2) which sub-domain it belongs to. Both types of tags are treated just like any other tokens, i.e. their learned embeddings are stored in the shared source-side embeddings table.

Model architecture
Prior to feeding parallel data into the sequence-to-sequence models, all text is preprocessed using the byte-pair encoding (BPE) tokenization scheme (Sennrich et al., 2016b). Our models follow the transformer-base architecture from Vaswani et al. (2017) as implemented in OpenNMTtf 4 v1.25 (Klein et al., 2017) with early stopping based on development sets of 5000 sentences per each use case.

Evaluation
The context of this work is a real-world industrial setting which involves translating large volumes of customer-facing text. Therefore our main evaluation criteria are human-based assessments. The human evaluation was performed by professional translators on a 4-point adequacy Likert scale using 250 samples per language, per domain. Appendix A provides details of the scoring guidelines that human evaluators follow. Additionally we report case-sensitive BLEU score (Papineni et al., 2002) as implemented by sacreBLEU 5 Post (2018). Table 2 summarizes our multi-domain adaptation results. On average MDT does not only match, but in fact outperforms the strong top10 (Edunov et al., 2018) baseline. As mentioned in Section 4.4, given the production quality requirement of our systems we consider human scoring the gold standard for evaluating translations, not the BLEU score alone. Most human and BLEU scores do rank-wise agree, but there are some exceptions. Specifically the German-English MDT model does better than the respective top10 models on Messaging and Descriptions domains according to the human evaluators, however it is not reflected in the BLEU scores.

Ablation experiment
In order to assess the role of tags, we perform an ablation experiment for German language, in which we compare the MDT performance to that of a model trained without the tags (but on the same mix of training data). It appears that the tags indeed on average improve the performance (Table 3). The models without tags perform worse on "Reviews" and "Messaging" domains according to human evaluations, and on all three domains according to the BLEU score evaluations.
Human score BLEU score Average -.10 -1.26 Table 3: The effect of tags removal on human and BLEU score in German-English MDT model.

Conclusions
In this work we introduce multidimensional tagging and demonstrate that it can be a scalable solution for multi-domain adaptation in a realistic resource-constrained setting. Somewhat surprisingly we find that MDT models in fact outperform on average our best alternative fine-tuning technique (top10 from Edunov et al. (2018)), even though the alternative method trains a custom model for each sub-topic. Although the present work offers limited empirical evaluations of MDT (two dimensions: 3 sub-domains and 2 data sources; three language pairs), we think that the technique can prove useful in a broader setting. We believe it to be particularly well suited to many real-world scenarios in which practitioners develop solutions for multiple related domains, while leveraging data from different sources, both genuine and synthetic. All experimental results reported in this work follow rigorous human evaluations in addition to the standard BLEU scores assessments.

A Human evaluations criteria
Each reported human evaluation reading is based on a random test set of 250 text samples which are evaluated by professional translators. Even though all translators were aligned and calibrated during previous evaluations, all sentences from the sample are always sent to the same individual translator to preserve consistency. We use an internally built tool ( Figure 2) which allows scoring on a four-point Likert scale, a modified version of the "Accuracy" dimension of the Fluency/Adequacy framework White et al. (1994); Callison-Burch et al. (2007); Levin et al. (2017b). We observed that fluency is almost never an issue in neural machine translation, so we do not score it explicitly. The following are the scoring guidelines for the four-point accuracy scale that are given to the translators: 4 All aspects of the review are comprehensible.

3
The fundamental information provided is accurately conveyed in the translation. Minor errors in non-essential supplementary information that are vague or obscured, but do not contend with the core of the meaning in the description, are allowed.

2
The fundamental information provided is obscured/distorted. The translation either indicates different factual information to what is present in the source, or the translation introduces incorrect information. 1 The translation does not make any sense, and/or does not even allude to the core of the source text.

B Reproducibility
Prior to feeding parallel data into the sequence-to-sequence models, all text is pre-processed using byte-pair encoding (BPE) tokenization scheme (Sennrich et al., 2016b). For all language pairs the BPE vocabulary size is set to 32k. For EN-DE language pair the vocabulary is learned jointly, while for EN-RU and EN-AR we use separate 32k vocabularies due to different alphabets in source and target. All our models follow the transformer-base architecture as described Figure 2: A screenshot of the internal human evaluation tool used by the language specialists.
in Vaswani et al. (2017) and implemented in OpenNMT-tf software (Klein et al., 2017) 6 . We trained the models using Adam Kingma and Ba (2014) optimizer with β 1 = 0.9 and β 2 = 0.998 with label smoothing set to 0.1 and noam decay with an initial learning rate of 2.0. While no hyper-parameter tuning is done, early stopping is based on a dev set of 5000 sentences. Furthermore, we use an effective batch size of 25,000 tokens accumulated over different GPUs and keep training until validation loss does not decrease for two consecutive steps. We select the checkpoint with minimum sentence level validation loss -therefore completely ignoring BLEU at model selection. We report both BLEU and human evaluation results using beam width equal to four on a separate test set. Training our base models took around 5 days using 8 NVIDIA V100 GPUs. Fine-tuning (both the single-domain baseline and the multi-domain MDT variant) took around 16 hours on a single GPU of the same model showing that there is no noticeable difference in training time. Inference time is the same for all models and only depends on sequence length.

C Text samples
The table below provides a few typical text samples from each domain for each of the three source languages. We also show English reference (human) translation as well as translation outputs from each of the three engines: base model, domain fine-tuned model (top10) and MDT (our method).