{'fraction_non_alphanumeric': 0.03258136731530911, 'fraction_numerical': 0.010573445841226107, 'mean_word_length': 4.78406374501992, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
When Does Translation Require Context? A Data-driven, Multilingual Exploration

Although proper handling of discourse phenomena significantly contributes to the quality of machine translation (MT), common translation quality metrics do not adequately capture them. Recent works in context-aware MT attempt to target a small set of these phenomena during evaluation. In this paper, we propose a new metric, P-CXMI, which allows us to identify translations that require context systematically and confirm the difficulty of previously studied phenomena as well as uncover new ones that have not been addressed in previous work. We then develop the Multilingual Discourse-Aware (MuDA) benchmark, a series of taggers for these phenomena in 14 different language pairs, which we use to evaluate context-aware MT. We find that state-of-the-art context-aware MT models find marginal improvements over context-agnostic models on our benchmark, which suggests current models do not handle these ambiguities effectively. We release code and data to invite the MT research community to increase efforts on context-aware translation on discourse phenomena and languages that are currently overlooked.


Introduction
In machine translation (MT), information from previous utterances has been found crucial to adequately translate a number of discourse phenomena including anaphoric pronouns, lexical cohesion, and discourse markers (Guillou et al., 2018;Läubli et al., 2018;Toral et al., 2018). However, while generating proper translations of these phenomena is important, they represent only a small portion of the words in natural language data. Because of this, common metrics such as BLEU (Papineni et al., 2002) do not provide a clear picture of whether they are appropriately captured or not. * Table 1: Some representative works on contextual machine translation that perform evaluation on discourse phenomena, contrasted to our work. For a more complete review see Maruf et al. (2021) Recent work on neural machine translation (NMT) models that attempt to incorporate extrasentential context (Tiedemann and Scherrer, 2017;Miculicich et al., 2018;Maruf and Haffari, 2018, inter alia) often perform targeted evaluation of certain discourse phenomena, mostly focusing on lexical cohesion (Voita et al., 2019b,a) and pronoun translation (Müller et al., 2018;Bawden et al., 2018;Lopes et al., 2020). However, only a limited set of discourse phenomena for a few language pairs have been studied (see summary in Table 1). The difficulty of broadening these studies stems from the reliance of previous work on introspection and domain knowledge to identify the relevant discourse phenomena, which then requires engineering language-specific methods to create test suites or manually designing data for evaluation.
In this paper, we fill this gap by proposing a data-driven, semi-automatic methodology for identifying salient phenomena that require context for translation, and we apply this method to create a multilingual benchmark testing these discourse phenomena. This is done through several steps. First, we develop P-CXMI ( §2) as a metric to identify when context is helpful in MT, or more broadly text generation in general. Then, we perform a systematic analysis of words with high P-CXMI to find categories of translations where context is useful ( §3). This allows us to identify novel discourse phenomena that to our knowledge have not been addressed previously (e.g. consistency of formality and verb forms), without requiring languagespecific domain knowledge. Finally, we design a series of methods to automatically tag words belonging to the identified classes of ambiguities ( §4) and we evaluate existing translation models for different categories of ambiguous translations ( §5).
We perform our study on a parallel corpus spanning 14 language pairs, over which we compare translation ambiguity and model performance. Somewhat surprisingly, we find that the contextaware methods, while improving on translation evaluation metrics, only perform slightly better in our benchmark, hinting that improvements on these metrics might be due to factors other than context usage (Kim et al., 2019). We also find that DeepL, a commercial document-level translation system, does better in our benchmark than its sentencelevel ablation and Google Translate. We hope that the released benchmark and code, as well as our findings, will spur targeted evaluation of discourse phenomena in MT to cover more languages and a wider variety of phenomena in the future.

Cross-Mutual Information
While document-level MT models can be compared using standard automatic performance metrics such as BLEU (Papineni et al., 2002), these metrics do not provide a clear picture of whether models are performing better due to improvements in the handling of context or due to other improvements in training or modelling (Kim et al., 2019). Another common evaluation paradigm is contrastive evaluation, which evaluates contextual models on their ability to distinguish between correct and incorrect translations of very specific discourse phenomena, such as anaphora resolution (Müller et al., 2018) and lexical cohesion (Bawden et al., 2018). However, such evaluation provides only a limited measure of context usage with respect to a fixed number of ambiguous translations defined by the creators of the dataset, and hence cannot capture other unanticipated ways in which the model might be using context. We are therefore interested in devising a metric that is able to capture all context usage by a model, beyond a predefined set.
Conditional Cross-Mutual Information (CXMI)  is a phenomena-agnostic metric for measuring the influence of context on model predictions. CXMI is defined as where X and Y are a source and target sentence, respectively, C is the context, H q M T A is the entropy of a context-agnostic MT model, and H q M T C refers to a context-aware MT model. This quantity can be estimated over a held-out set with N sentence pairs and the respective context as: Importantly, the authors find that training a single model q M T as both the context-agnostic and context-aware model ensures that non-zero CXMI values are due to context and not other factors (see the original paper for a more detailed explanation).

Context Usage Per Sentence and Word
CXMI measures the context usage by a model by comparing the log-likelihood ratio of samples across the whole corpus. However, for our purposes, we are interested in measuring how much the context is helpful for single sentences or even just particular words in a sentence.
Pointwise Mutual Information (P-MI) (Church and Hanks, 1990) is a measure of the association between two random variables for specific outcomes of them. Mutual information can be seen as the expected value of P-MI over all possible outcomes of the variables. Taking inspiration from this, we define the Pointwise Cross-Mutual Information (P-CXMI) for a given triplet of source, target and context (x, y, C) as Intuitively, P-CXMI measures how much more (or less) likely a target sentence y is when it is given context C, compared to not being given that context. Note that this is estimated according to the models q M T A and q M T C since, just like CXMI, this measure depends on their learned distributions.
We can also apply P-CXMI at word-level (as opposed to sentence-level) to measure how much (T-V) Roger. I got'em. Two-Six, this is Two-Six , we're mobile. Formality 了解 捕捉した。 2-6 こちら移動中だ。 (Honorifics) Our tools today don't look like shovels and picks. They look like the stuff we walk around with. Pronouns As ferramentas de hoje não se parecem com pás e picaretas. Elas se parecem com as coisas que usamos.
Louis XIV had a lot of people working for him. They made his silly outfits, like this.
They're the ones who know what society is going to be like in another generation. I don't.
Ellipsis Ancak onlar başka bir nesilde toplumun nasıl olacagını biliyorlar. Ben bilmiyorum. Table 2: Examples of high P-CXMI tokens and corresponding linguistic phenomena. Contextual sentences are italicized. The high P-CXMI target token is highlighted in pink, source and contextual target tokens related to the high P-CXMI token are highlighted in blue and green respectively. more likely a particular word in a sentence is when it is given the context, by leveraging the autoregressive property of the neural decoder. Given the triplet (x, y, C) and the word index i, we can measure the P-CXMI for that particular word as: Note that nothing constrains the form of C or even x and P-CXMI can, in principle, be applied to any conditional language modelling problem.
Using this metric, we now ask: what kind of words tend to see their likelihood increase when given the context? Such words should have a high P-CXMI, which we examine in the following §3.

Which Translation Phenomena Benefit from Context?
To identify salient translation phenomena that require context, we perform a thematic analysis (Braun and Clarke, 2006), examining words with high P-CXMI across different language pairs and manually identifying patterns and categorizing them into phenomena where context is useful for translation. To do so, we systematically examined (1) the mean P-CXMI per POS tag, (2) the vocabulary items with the highest P-CXMI, and (3) the individual tokens with the highest P-CXMI. We follow the experimental setup in . See also D for more details on the training.

Data
To compare linguistic phenomena that arise during document-level translation across various language pairs, we need a dataset that is document-level, rich in context-dependent discourse phenomena, and parallel in multiple languages. We, therefore, perform our study on transcripts of TED talks and their translations (Qi et al., 2018). We choose to study translation between English and Arabic, German, Spanish, French, Hebrew, Italian, Japanese, Korean, Dutch, Portuguese, Romanian, Russian, Turkish and Mandarin Chinese. These 14 target languages are chosen for their high availability of TED talks and of linguistic tools, as well as for the diversity of language types represented in our comparative study (Table 7 in Appendix A). In total, our dataset contains 113,711 parallel training sentences from 1,368 talks, 2,678 development sentences from 41 talks, and 3,385 testing sentences from 43 talks.

Analysis Procedure
We adopt a top-down approach and start our analysis by studying POS tags with high mean P-CXMI. In Appendix B, we report the mean P-CXMI for selected POS tags on our test data for translation from English to the 14 target languages. Some types of ambiguity, such as dual form pronouns ( §3.3), can be linked to a single POS tag and be identified at this step, whereas others require finer inspection. Next, we inspect the vocabulary items with high mean P-CXMI. At this step, we can detect phenomena that are reflected by certain lexical items that consistently benefit from context for translation.
Finally, we examine individual tokens that obtain the highest P-CXMI. In doing so, we identify patterns that do not depend on lexical features, but rather on syntactic constructions for example. In Table 2, we provide selected examples of tokens that have high P-CXMI and the discourse phenomenon we have identified from them.

Identified Phenomena
Through our thematic analysis of P-CXMI, we identified various types of translation ambiguity. Unlike previous work, our methodology requires no prior knowledge of the language pair to find relevant discourse phenomena and easily scales to various languages.
Firstly, we find relatively high P-CXMI for proper nouns (PROPN) for the majority of languages. As in the first row of Table 2, proper nouns and named entities may have multiple possible translations (another translation for "Avelile" is " 艾薇儿"), but the same entity should be referred to by the same word in a translated document for lexical cohesion (Carpuat, 2009).
We find high P-CXMI for second-person pronouns (PRON.2) in languages in which they encode politeness distinction (Appendix A, "Pronouns Politeness"), such as German, Spanish, French, Italian, Portuguese, Russian, and Chinese. While English uses the same second-person pronouns for everyone, these languages perform T-V distinction where this pronoun depends on the level of formality and relationship between the speaker and addressee. We also find high P-CXMI for verbs in Japanese and Korean. These two languages use honorifics and speech levels for politeness distinction, where prefixes and suffixes of certain words and the verb lemma depend on the interlocutors' relationship and on the formality of the discourse. Moreover, Japanese has relatively high P-CXMI on auxiliary verbs (AUX) and particles (PART) which also often control the formality and tone of the sentence, and further confirms the significance of this phenomenon.
In English, only the 3rd person singular pronoun may contain gender distinction and it assigns gender based solely on semantic rules (Appendix A, "Gendered Pronouns", "Gender Assignment"). We find that several languages have high P-CXMI on pronouns (PRON), and these languages use gendered pronouns for pronouns other than the 3rd person singular or assign gender using formal rules (German, French, Hebrew, Italian, Portuguese, Russian, and Chinese). When translating a gender-neutral English pronoun to a gendered target pronoun, context is therefore needed to resolve anaphora and determine the gender. We also find that Arabic obtains high P-CXMI on dual form pro-nouns (0.139 on PRON.3.Dual words), whereas duality not explicitly expressed in English pronouns which leads to ambiguity.
We also find high P-CXMI for certain verb forms, such as the future form in Italian and imperfect form in Spanish, French, Italian, and Romanian. While English verbs may have five forms (e.g. write, writes, wrote, written, writing), other languages often have a more fine-grained system of verb forms. For example, English has only a single form for the past tense, while the French past tense consists of five verb forms. Verbs must be translated using the verb form that reflects the tone, mood and cohesion of the document.
Finally, among the individual tokens with the highest P-CXMI, we find that many are due to ellipsis in the English sentence that does not occur on the target side. For example, in the last row of Table 2, the English text does not repeat the verb know in the second sentence as it can be understood from the previous sentence. However, in Turkish, there is no natural way to translate the verb-phrase ellipsis and must infer that "don't" refers to "don't know", and translate the verb accordingly.

Cross-phenomenon MT Evaluation
After identifying a set of linguistic phenomena where context is useful to resolve ambiguity during translation, we develop a series of methods to automatically tag tokens belonging to these classes of ambiguous translations and propose the Multilingual Discourse-Aware (MuDA) benchmark for context-aware MT models.

MT Evaluation Framework
Given a pair of parallel source and target documents (X, Y ), our MuDA tagger assigns a set of discourse phenomena tags {t 1 i , · · · , t n i } to each target token y i ∈ Y . Then, using the compare-mt toolkit (Neubig et al., 2019), we compute the mean word f-measure of system outputs compared to the reference for each tag. This allows us to identify which discourse phenomena models can translate more or less accurately.

Automatic Tagging of Discourse Phenomena
In this section, we describe how we devise automatic taggers for each discourse phenomenon identified through our thematic analysis.

Lexical Cohesion
To tag words that require lexical cohesion, we first extract word alignments from a parallel corpus denote the source and target document pair. We use the AWESOME aligner (Dou and  to obtain: where each x i and y j are the lemmatized content source and target words and ↔ denotes a bidirectional word alignment. Then, for each target word y j that is aligned to source word x i , if the alignment pair x i , y j occurred at least 3 times already in the current document, we tag y j for lexical cohesion. Formality For languages that use distinct pronouns for different levels of formality (T-V distinction), we tag the target pronouns containing formality distinction in their various forms. Some languages such as Spanish and Italian often drop the subject pronoun, and T-V distinction is instead reflected in the verb form. For these languages, we use spaCy (Honnibal and Montani, 2017) and Stanza (Qi et al., 2020) to find POS tags (for other phenomena that use POS tags, we do the same), which we use to detect verbs with a second-person subject in the source text, and conjugated in the second (T) or third (V) person in the target text. For languages such as Japanese and Korean with a more complex honorifics system, we construct a word list of common honorifics-related words to tag. For details of formality-related pronouns and honorifics-related words, see Appendix C.
Pronoun Choice To find pronouns in English that have multiple translations in the target language, we manually construct a list P = { p s , p t } for each language (provided in Appendix C), where each p s is an English pronoun and p t the list of possible translations of p s in the language . Then, for each aligned token pair x i , y j , if x i , y j are both pronouns with x i , p t |y j ∈ p t ∈ P , we tag y j as an ambiguous pronoun.
Verb Form For each target language, we define each target token y j , if y j is a verb and its verb form is in V , we tag y j as an ambiguous verb.
Ellipsis To detect translation ambiguity due to ellipsis, we look for instances where the ellipsis occurs on the source side, but not on the target side, which means that the ellipsis must be resolved during translation. Existing ellipsis detection models only operate monolingually, and are often limited to just one type of ellipsis such as verb-phrase ellipsis or noun-phrase ellipsis. We, instead, are interested in capturing ellipsis in general. To detect when ellipsis occurs in the source but not in the target, we first train an English ellipsis detection model. To do so, we extract an ellipsis dataset from the English data in the Penn Treebank (Marcus et al., 1993) and train a BERT text classification model (Devlin et al., 2019), which achieves 0.77 precision and 0.73 recall (see Appendix C for training details). Then, for each sentence pair where the source sentence is predicted to contain an ellipsis, we tag the word y j in the target sentence Y m if: (1) y j is a verb, noun, proper noun or pronoun; (2) y j has occurred in the previous target sentences of the same document; (3) y j is not aligned to any source words, that is,

Evaluation of Automatic Tags
We apply the MuDA tagger to the reference translations of our data from TED talks. In Appendix B, we report the mean P-CXMI for each language and MuDA tag. Overall, we find higher P-CXMI on tokens with a tag compared to those without, which suggests that our tags are indeed able to capture high P-CXMI words. The tags that obtain the highest P-CXMI depends on the target language, hinting that different language pairs contain different translation ambiguities.
In Figure 1, we can see that the frequency of  tags can significantly vary across languages. Overall, ellipses are infrequent, as only 4.5% of the English sentences have been marked for ellipsis which gives an upper bound for the number of ellipsis tags. We do remark that languages from a different family than English have a relatively higher number of ellipsis tags. Korean and especially Japanese have more formality tags than those with T-V distinction, which is aligned with our intuition that register is more often important when translating to languages with honorific systems.

Manual Evaluation
To verify the correctness of our automatic tagger, authors of the paper who are native speakers of French and Japanese manually verified the MuDA tags on 50 randomly selected utterances as well as all words tagged with ellipsis in our corpus. In Table 3, we report the tags' precision.
For both languages, ellipsis is the least precise. False positives often come from when a single source word is translated into two or more words, but our automatic aligner does not align all target words to the corresponding source word. Another common mistake is when the reference is not a word-by-word translation of the source expression, so many target words are not aligned to any source words even if there was no ellipsis. We believe that the ellipsis tagger is still useful in selecting difficult examples that require context for translation, and despite the low precision, we find a significantly high P-CXMI for ellipsis words for many target languages (Appendix B).

A Cross-lingual, Cross-Model
Exploration of Context-aware MT Next, we use our MuDA benchmark to perform an initial exploration of context usage across 14 languages pairs and 4 models, including those we trained ourselves and commercial systems.

Trained Models
In all of our experiments, we base ourselves on the Transformer architecture (Vaswani et al., 2017). For all the language pairs, we train a base model (transformer-small) and incorporate the target side context by concatenating it to the current target sentence (Tiedemann and Scherrer, 2017).
To evaluate potentially stronger models, we additionally train a large model (transformer-large) that was pretrained on a large corpora, for German, French, Japanese and Chinese. Further training details can be found in Appendix D.

Commercial Models
To assess if commercially available machine translation engines are able to leverage context and therefore do well in the MuDA Benchmark, we consider two engines: 2 (1) the Google Cloud Translation v2 API. In early experiments, we assessed that this model only does sentence-level translation, but included it due to its widespread usage and recognition; (2) the DeepL v2 API. This model advertises its usage of context as part of their translations and our experiments confirm this. Early experimentation with other providers (Amazon and Azure) indicated that these are not context-aware so we refrained from evaluating them.
To obtain the translations for each provider, we feed the whole document into an API request. To resegment the translation, we include special marker tokens in the source document that are preserved during translation and split the translation on those tokens. For DeepL, we also evaluate a sentencelevel version, where we feed each sentence in a document separately, to compare with its documentlevel counterpart. Table 4 shows the results for our base models, trained either without context (no-context) or with context, and for the latter with either predicted context (context) or reference context (context-gold) during decoding. Results are reported with respect to standard MT metrics such as BLEU (Papineni et al., 2002) and COMET (Rei et al., 2020), as well as the MuDA benchmark.

Results and Discussion
First, we find that although BLEU scores are highest for context-gold models for most language pairs, COMET is more often higher for the no-context models. Moreover, in terms of     mean word f-measure overall words, we do not find significant differences between the three systems. It is therefore difficult to see which system performs the best using only corpus-level metrics. For words that do require context for translation tagged with MuDA, our context-aware models obtain only marginally higher word f-measure than context-agnostic models, which suggests that these models are not sensitive enough to context or unable to use the relevant information from context to translate ambiguous words well. For the pretrained large models (Table 5), context-aware models obtain slightly higher scores on corpus-level metrics. We do find that improvements are more pronounced for certain tags such as ellipsis, lexical cohesion and formality.
Among commercial engines (Table 6), DeepL seems to outperform Google on most metrics and language pairs. Also, the sentence-level ablation of DeepL performs worse that its document-level system for most MuDA tags, which further suggests DeepL is able to handle context to some extent.
Overall, current context-aware MT systems are unable to obtain considerable improvements over their context-agnostic counterparts on challenging MuDA data.

Related Work
Because examples requiring context to translate accurately is sparse in document-level datasets, standard MT metrics such as BLEU or COMET are   not sensitive enough to gains in context-aware MT. Thus, several works resort to measuring the performance of context-aware models targeted to discourse phenomena that require context. Some works have attempted to do an automatic evaluation of discourse without relying on a contrastive dataset. The first example of discourse phenomena evaluations was done by Hardmeier et al. (2010), which evaluated the automatically precision and recall of pronoun translation in statistical MT systems. Jwalapuram et al. (2019) proposed evaluating models on pronoun translation based on a pairwise comparison between translations that were generated with and without context, and later Jwalapuram et al. (2020) extended this work to include more languages and phenomena in their automatic evaluation/test set creation. While these works rely on prior domain knowledge and intuitions to identify context-aware phenomena, we instead take a systematic, data-driven approach and find additional phenomena in doing so.
Most works have focused on evaluating performance in discourse phenomena through the use of contrastive datasets instead. Müller et al. (2018) automatically create a dataset for anaphoric pronoun resolution to evaluate MT models in EN → DE. Bawden et al. (2018) manually creates a dataset for both pronoun resolution and lexical choice in EN → FR. Voita et al. ( , 2019b creates a dataset for anaphora resolution, deixis, ellipsis and lexical cohesion in EN → RU. However,  suggest that the task of translating and disambiguating between two contrastive choices are inherently different, which motivates our approach in measuring direct translation performance through evaluation of word f-measure.

Conclusions and Future Work
In this work, we investigate the types of ambiguous translations where MT models benefit from context using our proposed P-CXMI metric. Our datadriven thematic analysis helps us identify contextsensitive discourse phenomena that has not been addressed in prior works on context-aware MT, such as ellipsis, formality and verb forms, for 14 language pairs. The advantages of our approach is that it is systematic, and does not require domain knowledge in language to identify these phenomena, so we believe that our methodology can be easily extended to other language pairs. P-CXMI can also be used to identify types of context-dependent words for tasks outside MT. Based on our findings, we then construct the MuDA benchmark that tags words in a given parallel corpus and evaluate models on 5 context-dependent discourse phenomena. We find that ellipsis is the most challenging to tag with high precision and leave improvements to model cross-lingual ellipsis for future work.
Our evaluation of models on MuDA reveals that both context-aware and commercial translation systems do not considerably improve on many of the discourse-aware translations compared to models that do not use context, and we encourage the development of models that address these ambiguities. Table 7 summarizes the properties of the languages analyzed in this work. C Tagger Details C.1 Formality Words Table 9 gives the list of words related to formality for each target language. Table 10 provides English pronouns and the list of possible target pronouns. Table 11 lists verb forms that may require disambiguation during translation.

C.4 Ellipsis Classifier
We train a BERT text classification model (Devlin et al., 2019) on data from the Penn Treebank, where we labeled each sentence containing the tag '*?*' as containing ellipsis (Bies et al., 1995). We obtain 248,596 sentences total, with 2,863 tagged as ellipsis. Then, our model using HuggingFace Transformers (Wolf et al., 2020). To address the imbalance in labels, we up-weight the loss for samples tagged as ellipsis by a factor of 100.

D Training details
The transformer-small model has hidden size of 512, feedforward size of 1024, 6 layersa and 8 attention heads. The transformer-large model has hidden size of 1024, feedforward size of 4096, 6 layers, 16 attention heads.
As in Vaswani et al. (2017), we train using the Adam optimizer with β 1 = 0.9 and β 2 = 0.98 and use an inverse square root learning rate scheduler, with an initial value of 10 −4 for large model and 5 × 10 −4 for the base and multi models, with a linear warm-up in the first 4000 steps.