{'fraction_non_alphanumeric': 0.031958019909013105, 'fraction_numerical': 0.012747173550740957, 'mean_word_length': 4.80203841630733, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Can Your Context-Aware MT System Pass the DiP Benchmark Tests? : Evaluation Benchmarks for Discourse Phenomena in Machine Translation

Despite increasing instances of machine translation (MT) systems including contextual information, the evidence for translation quality improvement is sparse, especially for discourse phenomena. Popular metrics like BLEU are not expressive or sensitive enough to capture quality improvements or drops that are minor in size but significant in perception. We introduce the first of their kind MT benchmark datasets that aim to track and hail improvements across four main discourse phenomena: anaphora, lexical consistency, coherence and readability, and discourse connective translation. We also introduce evaluation methods for these tasks, and evaluate several baseline MT systems on the curated datasets. Surprisingly, we find that existing context-aware models do not improve discourse-related translations consistently across languages and phenomena.


Introduction and Related Work
The advances in neural machine translation (NMT) systems have led to great achievements in terms of state-of-the-art performance in automatic translation tasks. There have even been claims that their translations are no worse than what an average bilingual human may produce (Wu et al., 2016) or even that the translations are on par with professional translators (Hassan et al., 2018).
However, these claims only hold under a narrow set of controlled circumstances. When translations are evaluated monolingually or at the document level, human translations are preferred over MT outputs. Läubli et al. (2018) conduct extensive experiments for Chinese-English translations with professional translators, and find that although there is no statistical difference in adequacy between human and MT output at a sentence level, there is a statistically strong prefer-ence for human translations both in terms of adequacy and fluency when evaluated at the document level. Crucially, the document (or discourse) level phenomena (e.g., coreference, coherence) may not seem lexically significant but contribute significantly to readability and understandability of the translated texts (Guillou, 2012).
Meanwhile, there have been numerous attempts to model extra sentential context for MT -previously within the statistical MT (Carpuat et al., 2013;Hardmeier et al., 2013), and recently within the NMT framework. The NMT framework such as the Transformer (Vaswani et al., 2017) provides more flexibility to incorporate larger context. This has spurred a great deal of interest in developing context-aware NMT systems that take advantage of source or target contexts, e.g., (Maruf and Haffari, 2018), (Miculicich et al., 2018) and (Voita et al., 2018(Voita et al., , 2019, to name a few. Despite the increasing interest in contextual MT, there is no framework for a principled comparison of results: there are no standard corpora and no agreed-upon evaluation measures. The selection of training datasets has mostly been arbitrary and much smaller in size compared to the standard ones (e.g., WMT datasets).
More critically, the lack of appropriate evaluation measures has been the key impediment in advancing contextual MT as it is important to measure if the context improves translations in terms of discourse phenomena, rather than mere improvements in lexical matching as is done with BLEU (Papineni et al., 2002). Indeed, recent studies also propose targeted datasets for evaluating phenomena like coreference (Guillou et al., 2014;Guillou and Hardmeier, 2016;Lapshinova-Koltunski et al., 2018;Bawden et al., 2018;Voita et al., 2018), and in the case of (Voita et al., 2019), testsets for ellipsis and lexical cohesion. The WMT-2019 tasks have also included docu-ment level translation and several adjoining usersubmitted testsets targeted towards specific phenomena including subject-verb agreement, coreference, and others (Bojar et al., 2018(Bojar et al., , 2019. In this work, we cover four diverse discourse phenomena using automatic data extraction methods, and also propose automatic evaluation methods for these tasks. Our targeted evaluation datasets are called the DiP benchmark tests (for Discourse Phenomena), that will allow us to compare models across discourse task strengths.
Our analysis of state-of-the-art (SOTA) NMT models proves that testing a system on a single language pair is not sufficient as we observe significant differences in system behavior and quality across languages. Our methods for automatically extracting testsets can be applied to multiple languages, and find cases that are difficult to translate without having to resort to synthetic data. Moreover, they can be easily updated to reflect current challenges, since datasets can become outdated as systems improve over the years.
Our aim is to push the improvement of translation systems towards human-like output. Our main contributions in this paper are as follows: • Benchmark datasets for four discourse phenomena: anaphora, coherence & readability, lexical consistency, and discourse connectives.
• Automatic evaluation methods and agreements with human judgments.
• Benchmark evaluation and analysis of three SOTA context-aware systems contrasted with baselines, for French/German/Russian-English language pairs. We open-source our framework at https://ntunlpsg.github.io/project/discomt/DIP/.

Machine Translation Models
We first introduce the baseline MT systems that we will be benchmarking in this work and report their BLEU scores in our proposed setup.

Model Architectures
We test the performance of three context-aware NMT models introduced by Voita et al. (2018), Miculicich et al. (2018) and Zhang et al. (2018) on our DiP benchmark testsets. 1 Alongside, we also evaluate a sentence-level model, and a simple concatenation-based model (Tiedemann and Scherrer, 2017) to contrast with the three elaborate context-aware models. SEN2SEN: Our SEN2SEN baseline is a standard 6-layer base Transformer model (Vaswani et al., 2017) which translates sentences independently.
CONCAT: Our CONCAT model is a 6-layer base Transformer whose input is two sentences (previous and current sentence) merged, with a special character serving as a separator.
ANAPH: Voita et al. (2018) incorporate the source context by encoding it with a separate encoder, then fusing it in the last layer of a standard Transformer encoder using a gate. They claim that their model explicitly captures anaphora resolution. HAN: Miculicich et al. (2018) introduce a hierarchical attention network (HAN) into the Transformer framework to dynamically attend to the context at two levels: word and sentence. They achieve the highest BLEU when hierarchical attention is applied separately to both the encoder and the decoder. SAN: Zhang et al. (2018) use a separate Transformer encoder to encode the context in the source side, which is then incorporated into the source encoder and target decoder using gates. We refer to this model as source attention network (SAN).
For the context-aware models, we use the implementations from official author repositories. As the official code for ANAPH (Voita et al., 2018) has not been released, we implement the model in the Fairseq framework (Ott et al., 2019). 2 For training the SEN2SEN and CONCAT models we used the Transformer implementation from Fairseq. We confirmed with the authors of HAN and SAN that our configurations were correct, and we took the best configuration directly from the ANAPH paper. Further details about the training settings and hyperparameters can be found in Appendix A.4.

Training Data
It is essential to provide the models with training data that contains adequate amounts of discourse phenomena, if we expect them to learn such phenomena. To construct such datasets, we first manually investigated the standard WMT corpora consisting of UN (Ziemski et al., 2016), Europarl (Tiedemann, 2012)    as well as the standard IWSLT dataset (Cettolo et al., 2012). We analyzed 100 randomly selected pairs of consecutive English sentences from each dataset, where the first sentence was treated as the context. Table 1 shows the percentage of cases containing the respective discourse phenomena.
In accordance with intuition, data sources based on narrative texts such as IWSLT exhibit increased amounts of discourse phenomena compared to strictly formal texts such as the UN corpus. On the other hand, the UN corpus consists of largely unrelated sentences, where only lexical consistency is well-represented due to the usage of very specific and strict naming of political concepts. We decided to exclude the UN corpus and combine the other datasets that have more discourse phenomena. We evaluate the models on the WMT-14 testset which consists of news articles. Table 2 shows the statistics of the resulting datasets.

BLEU Scores
The BLEU scores on the WMT-14 testset for each of the five trained models for De-En, Fr-En and Ru-En translation tasks are given in Table 3.
We observe a variability in BLEU scores across the models. In contrast to increases in BLEU for selected language-pairs and datasets reported in the published work, incorporating context within elaborate context-dependent models decreases BLEU scores for Fr-En and De-En. CONCAT, the simple concatenation-based model, achieves the best BLEU out of all of the tested models. This shows that context knowledge is indeed helpful for improving the BLEU. For Ru-En task, dedicated context-aware mod-  els improve the performance. In particular, ANAPH achieves the highest score of all -interestingly, it has been trained and tested on En-Ru in the original paper (Voita et al., 2018). This shows that complex architectures might only be useful for certain types of languages (such as highly inflected languages, like Russian).

Benchmark Testset Generation
We extract the testsets for the evaluated discourse phenomena automatically, based on existing errors in system outputs. This ensures that the data can (i) provide hard contexts for translation without being artificial, (ii) be generated for multiple source languages, and (iii) be updated as frequently as possible; making them adaptable to errors in newer (and possibly more accurate) systems, and making the tasks harder over time.
We use the system outputs released by WMT for the most recent years (Bojar et al., 2017(Bojar et al., , 2018(Bojar et al., , 2019 to build our testsets. For De-En, Fr-En and Ru-En, these consist of translation outputs from 51, 31 and 41 unique systems respectively. Since the data comes from a wide variety of systems, our testsets representatively aggregate different types of errors from several (arguably SOTA) models. Also note that the MT models we are benchmarking are not a part of these system submissions to WMT, so there is no potential bias in the testsets.
In this paper, we focus on translations from French, German, and Russian to English. We include French since Fr-En is a popular translation pair that results in some of the highest BLEU scores. WMT discontinued French from 2016 onwards, so the benchmark testsets for French are smaller and based on relatively older 2013-2015 (Bojar et al., 2013(Bojar et al., , 2014(Bojar et al., , 2015 data. Other source languages that are part of WMT can be extracted as needed; the testsets can also be expanded if older data were to be considered. The following sections describe the dataset, evaluation and verification procedures, and analysis of each of the discourse phenomena we benchmark. Anaphora are references to entities that occur elsewhere in a text; mishandling them can result in ungrammatical sentences or the reader inferring the wrong antecedent, leading to misunderstanding of the text (Guillou, 2012). We focus specifically on the aspect of incorrect pronoun translations.

Pronoun Testset
To obtain hard contexts for pronoun translation, we look for source texts that lead to erroneous pronoun translations in recent WMT submissions. We align the WMT system translations with their references, and collect the cases in which the translated pronouns do not match the reference. This process requires the pronouns in the target language to be separate morphemes as in English.
Our anaphora testset is an updated version of the one proposed by Jwalapuram et al. (2019), who also provide a list of cases where the translations can be considered wrong (rather than acceptable variants). We filter the system translations based on their list. The corresponding source texts are extracted as a test suite for pronoun translation. This gives us a pronoun benchmark testset with 1478 samples for Fr-en, 2245 samples for De-En and 2368 samples for Ru-En.

Pronoun Evaluation
Targeted evaluation of pronouns in MT has been challenging as it is not fair to expect an exact match with the reference. Evaluation methods like APT (Miculicich Werlen and Popescu-Belis, 2017) or AutoPRF (Hardmeier and Federico, 2010) are specific to language pairs or lists of pronouns, requiring extensive manual intervention. They have also been criticised for failing to produce evaluations that are consistent with human judgments (Guillou and Hardmeier, 2018). Jwalapuram et al. (2019) propose a model based evaluation measure for pronouns that generalizes well across language pairs and pronouns. They train a pairwise ranking model that scores "good" pronoun translations (like in the reference) higher than the "poor" pronoun translations (like in the MT output) in context, and show that their model is good at making this distinction, along with having high agreements with human judgements. However, they do not rank multiple system translations against each other, which is our main goal; the absolute scores produced by their model are not useful since it is trained in a pairwise fashion.
We devise a way to use their model to score and rank system translations in terms of pronouns. First, we re-train their model with more up-to-date WMT data. 3 We obtain a score for each benchmarked MT system (SEN2SEN, CONCAT, etc.) translation using the model, plus the corresponding reference sentence. We then normalize the score for each translated sentence by calculating the difference with the reference. To get an overall score for an MT system, the assigned scores are summed across all sentences in the testset.
where ρ i (.|θ) denotes the score given to sentence i by the pronoun model with parameters θ. The systems are ranked based on this overall score, where a lower score indicates a better performance.
User study. To confirm that our normalizationbased ranking of systems agrees with human judgments, we conducted a user study. Participants are asked to rank given translation candidates in terms of their pronoun usage. We include the reference in the candidates, as a control. We ask participants to rank system translations directly rather than a synthetically constructed contrastive pair (as was done by Jwalapuram et al. (2019)) to ensure that our evaluations, which will be conducted on actual translated texts, are reliable. We first conducted the study in a bilingual setup, in the presence of the source for German-English. Participants were shown a source context of two sentences and the source sentence in bold, followed by three candidate translations of the source sentence, one of which is the reference.
The other two were translations with different pronoun errors produced by MT systems. Participants annotate 100 such samples. See Appendix A.1 for the user study interface.
We then conducted the study in a monolingual setup without the source, i.e., native speakers are shown the reference context in English, and the two candidate English translations and the reference translation as possible options for the sentence that follows ( Figure 1). To facilitate comparison, the data used for the German-English and only-English studies is the same.
The results are analysed to check (i) how often the reference is preferred over the system translations (our control), and (ii) how often the users agree in preference over the system translations (i.e., human judgment for translation quality). There were two participants in the bilingual setup, with the control experiment yielding an agreement of 0.72 according to Gwet's AC1 (Gwet, 2008). 4 There were four participants in the monolingual setup, with the control yielding an AC1 agreement of 0.82, which is higher than the bilingual setup. We therefore use the monolingual setup to evaluate the rankings obtained from our modified evaluation method. We obtain an agreement of 0.91, justifying the use of our modified pronoun model for evaluation.

Results and Analysis
The ranking results obtained from evaluating the MT systems on our pronoun benchmark testset using our evaluation measure are given in Table 4. We also report common pronoun errors for each model based on our manual analysis.
Overall, we observe that surprisingly, SEN2SEN is translating pronouns comparatively well -outperforming all other models in De-En and Fr-En, and only giving way to ANAPH in Ru-En. The success of the SEN2SEN model can be explained by its tendency to use it as the default pronoun, which statistically appears most often due to the lack of grammatical gender in English. More variability in pronouns occurs in the outputs of the context-aware models, but this does not contribute to a greater success.  Table 4: Pronoun evaluation: Rankings of the different models for each language pair, obtained by summing the evaluation score for each sample in the pronoun benchmark. Each set of rankings is followed by the results of the manual analysis on a subset of the translation data. The percentages for the following types of errors are reported: Anaphora -instances of Gender Copy, Named Entity and Language specific errors.
Specifically, we observed the following types of errors in our manual analysis on a subset of the translation data: (i) Gender copy. Translating from Fr/De/Ru to En often requires the 'flattening' of gendered pronouns to it, since Fr/De/Ru assign gender to all nouns. In many cases the machine translated pronouns tend to (mistakenly) agree with the source language. For example, diese Wohnung in Earls Court..., und sie hatte... is translated to : apartment in Earls Court, and she had..., a version which upholds the female gender expressed in sie, instead of translating it to it. This was the most common error, except for Ru-En, where Named Entity errors were slightly more prevalent.
(ii) Named entity. A particularly hard problem is to infer gender from a named entity, e.g., Lady Liberty...She is meant to...-she is wrongly translated to it. Such examples demand higher inference abilities such as world knowledge (e.g., distinguish male/female names).
(iii) Language specific phenomena. Pronouns can be ambiguous in the source language. For example in German, the pronoun sie can mean both she and you, depending on capitalization, sentence structure, and context. This type of error often appears in the context-aware models, while being relatively rare for the SEN2SEN model. Pitler and Nenkova (2008) define coherence as the ease with which a text can be understood, and view readability as an equivalent property that indicates whether it is well-written. It has been shown that NMT systems generate more fluent sentences than their phrase-based counterparts (Castilho et al., 2017). However, when the output is evaluated at the document-level, it has also been shown that it lacks coherence (Läubli et al., 2018).

Coherence Testset
Our coherence and readability benchmarking is conducted at the document level; we try to find documents that can be considered hard to translate. To do this, we use the coherence model recently proposed by Moon et al. (2019), that achieves state-of-the-art results in most coherence assessment tasks. The model has a Siamese framework, trained in a pairwise ranking fashion with positive and negative documents. The network models both syntax and inter-sentence coherence relations, along with global topic structures.
The coherence model is originally trained on WSJ articles, where a negative document is formed by shuffling sentences of an original (positive) document. It needed to be re-trained with MT data to better capture the coherence issues that are present in MT outputs. It has been shown in some studies that MT outputs are incoherent (Smith et al., 2015(Smith et al., , 2016Läubli et al., 2018). We thus re-train the coherence model with reference translations as positive and MT outputs as negative documents. We use the older WMT submissions from 2011-2015 for this re-training, to ensure that the training data does not overlap with the data used for extracting our benchmark testset.
The model takes a system translation (multisentential) and its reference as input and produces a score for each. Similar to Eq. 1, we consider the difference between the scores produced by the model for the reference and the translated text as the coherence score for the translated text.
For a given source text (document) in the WMT testsets, we obtain the coherence scores for each of the translations (i.e., WMT submissions) and average them. The source texts are then sorted based on the mean coherence scores of their translations. The texts that have lower mean coherence scores can be considered to have been hard to translate coherently. We threshold the scores to extract approximately the bottom 30% of the texts as a tradeoff between getting hard enough contexts and a reasonably-sized testset. These source texts form our benchmark testset for coherence and readability. This yields 38 documents for Fr-En, 128 documents for De-En and 180 documents for Ru-En.

Coherence Evaluation
Coherence and readability is also a hard task to evaluate, as it can be quite subjective. We resort to model-based evaluation here as well, to capture the different aspects of coherence in translations.
We use our re-trained coherence model to score the benchmarked MT system translations and modify the scores for use in the same way as the anaphora evaluation (Eq. 1) to obtain a relative ranking. As mentioned before ( §3), the benchmarked MT systems do not overlap with the WMT system submissions, so there is no potential bias in evaluation since the testset extraction and the evaluation processes are independent. To confirm that the model does in fact produce rankings that humans would agree with, and to validate our model re-training, we conduct a user study. User study. The participants are shown three candidate English translations of the same source text, and asked to rank the texts on how coherent and readable they are (Figure 2). To optimize an-  Rankings of the different models for each language pair, obtained by summing evaluation scores for each document in the coherence benchmark testsets.
notation time, participants are only shown the first four sentences of the document; they annotate 100 such samples. We also include the reference as one of the candidates for control, and to confirm that we are justified in re-training the evaluation model to assign a higher score to the reference. Three participants took part in the study. Our control experiment results in an AC1 agreement of 0.84. The agreement between the human judgements and the coherence evaluation model's rankings is 0.82. The high agreement validates our proposal to use the modified coherence model to evaluate the benchmarked MT systems.

Results and Analysis
From the rankings in Table 5, we see that SEN2SEN is the most coherent model for De-En and Ru-En. For Fr-En however, we observe an advantage of the context-aware model -SAN, which ranks high for De-En as well. We identified the following types of coherence and readability errors (more examples in Appendix A.6).
(i) Inconsistency. As in Somasundaran et al. (2014), we observe that inconsistent translation of words across sentences (in particular named entities) breaks the continuity of meaning.
(ii) Translation error. Errors at various levels spanning from ungrammatical fragments to model hallucinations introduce fragments which bear little relation to the whole text (Smith et al., 2016). An example of this: Reference: There is huge applause for the Festival Orchestra, who appear on stage for the first time in casual leisurewear in view of the high heat.
Translation: There is great applause for the solicitude orchestra , which is on the stage for the first time, with the heat once again in the wake of an empty leisure clothing.

Lexical Consistency
Lexical consistency in translation was first defined as 'one translation per discourse' by Carpuat (2009), i.e., the translation of a particular source word consistently to the same target word in that context. Guillou (2013) analyze different humangenerated texts and conclude that human translators tend to maintain lexical consistency, which supports the important elements in a text. The consistent usage of lexical items in a discourse can be formalized by computing the lexical chains (Morris and Hirst, 1991; Lotfipour-Saedi, 1997).

Lexical Consistency Testset
To extract a testset for lexical consistency evaluation, we first align the translations from WMT submissions with their references. In order to get a reasonable lexical chain formed by a consistent translation, we consider translations of blocks of 3-5 sentences in which the (lemmatized) word we are considering occurs at least twice in the reference. For each such word, we check if the corresponding system translation produces the same (lemmatized) word at least once, but fewer than the number of times the word occurs in the reference. In such cases, the system translation has failed to be lexically consistent in translation (see Figure 3 for an example). We limit the errors considered to nouns and adjectives. The source texts of these cases form the benchmark testset. This gives us a testset with 172 sets of sentences for Fr-En, 312 sets for De-En and 358 sets for Ru-En.
One possible issue with this method could be that reference translations may contain forced consistency, i.e., human translators introduce consistency to make the text more readable, despite inconsistent word usage in the source. It may not be reasonable to expect consistency in a system translation if there is none in the source. To confirm, we conducted a manual analysis where we compared the lexical chains of nouns and adjectives in Russian and French source texts against the lexical chains in the English reference. We find that in a majority (77%) of the cases, the lexical chains in the source are reflected accurately in the reference, and there are relatively few cases where humans force consistency. Considering the fact that the same data is used for BLEU calculations, we presume that this should not be a significant issue.

Lexical Consistency Evaluation
For lexical consistency, we adopt a simple evaluation method. For each block of 3-5 sentences, we either have a consistent translation of the word in focus, or the translation is inconsistent. We simply count the instances of consistency and rank the systems based on the percentage of accuracy.
It is possible that the word used in the system translation is not the same as the word in the reference, but the MT output is still consistent (e.g., a synonym used consistently). We tried to use alignments coupled with similarity obtained from ELMo (Peters et al., 2018) and BERT (Devlin et al., 2018) embeddings to evaluate such cases to avoid unfairly penalizing the system translations, but we found this to be noisy and unreliable. Thus, we match with the reference, as it can be argued that such words are salient and therefore must be translated exactly to convey the correct meaning.

Results and Analysis
The rankings of the MT systems based on accuracy on the lexical consistency benchmark testsets are given in Table 6, along with our findings from a manual analysis on a subset of the translations.
The overall low quality of Russian translations contributes to the prevalence of Random translations, and the necessity to transliterate named entities increases NE errors, compared to other languages. CONCAT and SEN2SEN are again successful on average, taking the first or second place in all tested languages, while ANAPH leads the board again for Ru-En. Our manual inspection of the lexical chains shows the following tendencies: (i) Synonym & related word. Words are exchanged for their synonyms (poll -survey), hypernyms/hyponyms (ambulance -car) or related concepts (wine -vineyard).
(ii) Named entity. Models tend to distort proper names and translate them inconsistently. For example, the original name Fchtorf (name of a town) gets translated to feeding-community. (iii) Omission. Occurs when words are omitted altogether from the lexical chain.  Table 6: Lexical consistency evaluation: Rankings of the different models for each language pair, ranked by their Accuracy. Accuracy here is defined as the percentage of samples in the benchmark dataset translations in which the models maintain lexical consistency. Each set of rankings is followed by the results of the manual analysis on a subset of the translation data for Synonyms, Related words, Omissions, Named Entity, Random translation.

Discourse Connectives
Discourse connectives are used to link the contents of texts together by signaling coherence relations that are essential to the understanding of the texts (Prasad et al., 2014). Failing to translate a discourse connective correctly can result in texts that are hard to understand or ungrammatical.

Discourse Connective Testset
Finding errors in discourse connective translations can be quite tricky, since there are often many acceptable variants. To mitigate confusion, we limit the errors we consider in discourse connectives to the setting where the reference contains a connective but the translations fail to produce any. Although there is an accepted list of explicit discourse connectives, it would not be appropriate to simply extract such cases, since those words may not always act in the capacity of a discourse connective. In order to identify the discourse connectives, we build a simple explicit connective classifier (a neural model) using annotated data from the Figure 4: Connective study interface. Participants are shown the reference with the connective and another option without the connective, and asked to choose the best option that follows the given context. Penn Discourse Treebank or PDTB (Prasad et al., 2018). The classifier achieves an average crossvalidation F1 score of 93.92 across the 25 sections of PDTBv3, proving that it generalizes well. See Appendix A.3 for more details about the model.
After identifying the explicit connectives in the reference and the system translations, we align them and extract the source texts of cases with missing connective translations. We only use the classifier on the reference text, but consider all possible markers in the system translations to give them the benefit of the doubt. This gives us a discourse connective benchmark testset with 109 samples for Fr-En, 109 samples for De-En and 117 samples for Ru-En.

Discourse Connective Evaluation
There has been some work on semi-automatic evaluation of translated discourse connectives in Meyer et al. (2012) and Hajlaoui and Popescu-Belis (2013); however, it is limited to only En-Fr, based on a dictionary list of equivalent connectives, and requires using potentially noisy alignments and other heuristics. In the interest of evaluation simplicity, we expect the model to produce the same connective as the reference. Since the nature of the challenge is that connectives tend to be omitted altogether, we report both the accuracy of connective translations with respect to the reference, and the percentage of cases where any candidate connective is produced. User study. To confirm that the presence of the connective conveys some information and contributes to better readability and understanding of the text, we conduct a user study. As presented in Figure 4, participants are shown two previous sentences from the reference for context, and asked to choose between two candidate options for the sentence that may follow. These options consist of the reference translation with the connective highlighted, and the same text with the connective deleted. We also conducted a study using system translations with missing connectives directly; see Appendix A.3 for discussion.
Participants are asked to choose the sentence which more accurately conveys the intended meaning. There were two participants who annotated 200 such samples. The reference with the connective was chosen over the version without the connective with an AC1 agreement of 0.98. See Appendix A.3 for connective-wise results. Note that participants may prefer the version with the connective due to loss of grammaticality or loss of sense information when the connective is missing. Although indistinguishable in this setting, we argue that since both affect translation quality, it is reasonable to expect a translation for the connectives.

Results and Analysis
The rankings of MT systems based on their accuracy of connective translations are given in Table 7, along with our findings from a manual analysis on a subset of the translations. The ranking shows that SEN2SEN models are on average the most accurate and omit the connectives less often. ANAPH continues its high performance in Ru-En, and while SAN leads the board for De-En in terms of accuracy, it has a low percentage of cases overall in which any connective is produced.
In benchmark outputs, we observed mostly omissions of connectives (disappears in the translation), synonymous translations (e.g., Naldo is also a great athlete on the bench -Naldo's "great sport" on the bank, too.), and mistranslations.

Discussion
Our benchmark evaluation on various discourse phenomena across different MT systems and language pairs reveals gaps in evaluation results that are typically reported. A lack of comprehensive evaluation makes it difficult to determine which models perform conclusively better than others.
Our results re-emphasize the gap between BLEU scores and translation quality at the discourse level. The overall BLEU scores for Fr-En are higher than the BLEU scores for De-En; however, we see that both the lexical consistency and the discourse connective accuracies are higher for De-En. Similarly, for Ru-En, both SAN and HAN have higher BLEU scores than the SEN2SEN and CONCAT models, but are unable to outperform  these simpler models consistently in the discourse tasks, often ranking last.
We also reveal a gap in performance consistency across language pairs. Models may be tuned for a particular language pair, such as ANAPH which was trained for En-Ru. For the same language pair (Ru-En), we show results consistent with what is reported; the model leads the board for anaphora and lexical consistency, while ranking second for coherence and readability, and discourse connectives. However, it is not so successful in other languages, ranking at the bottom for anaphora in De-En and discourse connectives in Fr-En, and close to bottom for coherence in Fr-En and De-En. SAN performs highly in coherence for Fr-En and De-En, in contrast to its performance on other tasks and languages; the authors originally report improved results for Fr-En.
In general, our findings match the conclusions from Kim et al. (2019) regarding the lack of satisfactory performance gains in context-aware models. Given no comprehensive evaluation across language pairs, the best bet for training an MT model is to use the baseline SEN2SEN and CON-CAT models, which perform more or less reliably across different tasks. Our results emphasize the need for standard benchmarking datasets and evaluation measures across language pairs, that will provide a better picture of MT system performance.
Although some of the testsets we provide are limited in size, it is a consequence of favouring precision to maintain data quality and limiting data to recent years. However, since the extraction is automatic, the datasets can be extended as submissions are added to the upcoming evaluation campaigns, while also increasing the difficulty of the tasks as MT systems improve. We hope that the discourse benchmark testsets and evaluation procedures we provide can contribute towards a more comprehensive MT evaluation framework, and prove useful in obtaining a more complete idea of a system's translation quality.

Conclusions
We presented the first of their kind discourse phenomena based benchmarking testsets called the DiP tests, designed to be challenging for NMT systems. We show that complex context-aware models are not consistent in their performance. Our main goal is to motivate the benchmarking of MT systems with more indicative performance yardsticks. We will release the document-level training corpora and discourse benchmark testsets for public use, and also propose to accept translations from MT systems to maintain a leaderboard for the described phenomena.  is based on a model that is trained on WMT11-15 data and tested on WMT-2017 data. We retrain the model with more up-to-date data from WMT13-18, and test the model on WMT-19 data. Note that this training data is taken from WMT submissions, which do not overlap with the benchmarked MT models; there is therefore no conflict in using this trained model to evaluate the benchmarked model translations. Results are shown in Table 8. Their model scores the translations in context; we provide the previous two sentences from the reference translation as context according to their settings.   Figure 5.
User Study. The bilingual (German-English) user study interface for pronoun translation ranking is shown in Figure 6. Results. The total assigned scores (difference between reference score and translation score) obtained for each system after summing the over the samples in the respective testsets are given in Table 10. The models are ranked based on these scores from lowest score (best performing) to highest score (worst performing).

A.2 Coherence
Re-trained model. We re-train the pairwise coherence model in Moon et al. (2019) to suit the MT setting, with reference translations as the positive documents and the MT outputs as the negative documents. The results are shown in Table 9.
Training data Test data Accuracy WMT11-15 WMT17-18 77.35 Table 9: Results of the re-trained coherence model.
Results. The total assigned scores (difference between reference score and translation score) obtained for each system after summing the over the samples in the respective testsets are given in Table 11. The models are ranked based on these scores from lowest score (best performing) to highest score (worst performing).

A.3 Discourse Connectives
Connective Classification model. We build an explicit connective classifier to identify candidates that are acting in the capacity of a discourse connective. The model consists of an LSTM layer (Hochreiter and Schmidhuber, 1997) followed by a linear layer for binary classification, initialized by ELMo embeddings (Peters et al., 2018). We  use annotated data from the Penn Discourse Treebank (PDTBv3) (Prasad et al., 2018) and conduct cross-validation experiments across all 25 sections. Our classifier achieves an average crossvalidation precision of 95.58, recall of 92.35 and F1 of 93.92, which shows that it generalizes very well. The high precision also provides certainty that the model is classifying discourse connectives reliably.
User Study. For discourse connectives, we conducted two user studies. The first study in which   the participants chose between the reference and its noisy version with the connective deleted was reported in the main paper. We present the connective-wise breakdown in Table 12.
In the second study, the participants were shown the reference along with the system translation that was missing the connective (Figure 7). In this study, the setup has no artificially constructed data; the idea is to check if there is a possibility that the system translation is structured in such a way as to require no connective. However, the AC1 agreement for preferring the reference was 0.82 (2 annotators; different annotators from the first study) for this study as well, which is still quite high. Table 13 has the connectivewise breakdown; here we see that the results are slightly different for certain connectives, but overall the strong preference for the reference with the connective is retained. Our assumption that connectives must be translated is validated through both studies.
Note that for both studies, participants were also given options to choose 'Neither' in case they didn't prefer either choice, or 'Invalid' in case there was an issue with the data itself (e.g., transliteration issues, etc.); data that was marked as such was excluded from further consideration.

A.4 Model Parameters
Parameters used to train SEN2SEN, CONCAT, ANAPH, and SAN models are displayed in Table  15, and parameters for HAN in Table 14.

A.5 Datasets
Our trainset is a combination of Europarl (Tiedemann, 2012), IWSLT (Cettolo et al., 2012) and News Commentary datasets, the development set is a combination of WMT-2016 and older WMT data (excluding 2014). We test on WMT-2014 data. We tokenize the data using the Moses software 5 , lowercase the text, and   apply BPE encodings 6 from Sennrich et al. (2016). We learn the BPE encodings with the command learn-joint-bpe-and-vocab -s 40000.

Phenomenon
Example Anaphora Gender Copy S: Mir wurde diese Wohnung in Earls Court gezeigt, und sie hatte ... T: I was shown this apartment in Earls Court , and she had .. Named Entity T: ... Lady Liberty is stepping forward. It is meant to be carrying the torch of liberty R: She is meant to be carrying the torch of Liberty.

Lexical Consistency
Synonym T: Watch the Tory party conference. The convention is supposed to be about foreign policy, (...). R: Under tight security -the Tory party conference. The party conference was to address foreign policy (...).

Related Word T:
In the collision of the car with a taxi, a 27-year-old passer was fatally injured. R: A 27-year old passenger was fatally injured when the ambulance collided with a taxi.
Named Entity T: The Feeding-Community farmer , however , also had the ready-filled specialities. The demand for the good "made in Feed orf" was correspondingly high. R: But the Fchtorf farmer also had bottled specialties with him.
There was a lot of demand for the good "made in Fchtorf" beverage.
Omission T: (...) during the single-family home attempt, it stayed by the royal highlands thanks to the burglar alarm. They got off when the culprits turned hand on Friday just before 20 a.m. R: It is thanks to the alarm system that the attempt in the Knigswieser Strae at the single family home (...). On Friday just before 20.00 the alarm rang when the offenders took action.

Coherence
Ungrammatical T: "They didn't play badly for long periods -like Stone Hages , like Hip Horst -Senser.
Only the initial phase, we've been totally wasted", annoyed the ASV coach. R: "Over long periods, they had -as in Steinhagen, as against Hllhorst -not played badly. We only overslept the initial phase", said the ASV coach annoyed.
Hallucination T: Before appointing Greece , Jeffrey Pyett was the US ambassador to Kiev. When it came to the Maidan and the coup in 2014 , it was a newspaper. R: Before his appointment, Geoffrey Ross Pyatt was an ambassador in Kiyv. During his mission, the Maydan events and state coup happened, reminds Gazeta.Ru Inconsistency T: The one-in-house airline crashed on Sunday afternoon at a parking lot near Essen-Mosquitos. Essen Mill is a small airport that's used a lot by airline pilots. R: On Sunday afternoon, the single-seated aircraft crashed (..) a parking lot near the airport Essen-Mlheim Essen-Mlheim is a small airport, which is frequently used by pilots with light private planes.

Discourse Connectives
Omission T: Two people died driving their car against a tree . R: Two people died after driving their car into a tree.
Synonym T: Naldo's "great sport" on the bank, too. R: Naldo is also a great athlete on the bench