{'fraction_non_alphanumeric': 0.034967963482557894, 'fraction_numerical': 0.009171238326563089, 'mean_word_length': 4.747292418772563, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Estimating post-editing effort: a study on human judgements, task-based and reference-based metrics of MT quality

Devising metrics to assess translation quality has always been at the core of machine translation (MT) research. Traditional automatic reference-based metrics, such as BLEU, have shown correlations with human judgements of adequacy and fluency and have been paramount for the advancement of MT system development. Crowd-sourcing has popularised and enabled the scalability of metrics based on human judgments, such as subjective direct assessments (DA) of adequacy, that are believed to be more reliable than reference-based automatic metrics. Finally, task-based measurements, such as post-editing time, are expected to provide a more de- tailed evaluation of the usefulness of translations for a specific task. Therefore, while DA averages adequacy judgements to obtain an appraisal of (perceived) quality independently of the task, and reference-based automatic metrics try to objectively estimate quality also in a task-independent way, task-based metrics are measurements obtained either during or after performing a specific task. In this paper we argue that, although expensive, task-based measurements are the most reliable when estimating MT quality in a specific task; in our case, this task is post-editing. To that end, we report experiments on a dataset with newly-collected post-editing indicators and show their usefulness when estimating post-editing effort. Our results show that task-based metrics comparing machine-translated and post-edited versions are the best at tracking post-editing effort, as expected. These metrics are followed by DA, and then by metrics comparing the machine-translated version and independent references. We suggest that MT practitioners should be aware of these differences and acknowledge their implications when decid- ing how to evaluate MT for post-editing purposes.


Introduction
Assessing the quality of the output of machine translation (MT) systems has been a widely explored topic in the last two decades. As with other applications outputting language (e.g. text summarisation), quality assessment of MT is challenging and highly dependent on the purpose of the transla-tion. Therefore, the quality of machine translated (MT'ed) texts may depend on their usage. Table 1 shows two machine translations into English, and their respective post-edited versions. In Example 1, the MT'ed version has a different meaning from that of the original sentence: readers may be led to believe that the product is good and they should buy it, whilst the correct recommendation is against buying the product. Although this sentence would be problematic for an end user, it is rather simple to correct by a post-editor (only one word needs to be added). Example 2, on the other hand, shows a sentence where an end user can understand the MT'ed version with little effort, even though it contains multiple errors. A post-editor, however, would need to perform at least five word-level edit operations in order to transform the machine translation into the post-edited version.
These examples illustrate how sensitive MT evaluation is to purpose. Nirenburg [1] argues that MT can be classified into two groups according to its purpose: dissemination or assimilation. MT for dissemination is expected to be either ready as is or adequate for post-editing, since the purpose is publication. In contrast, MT for assimilation has the purpose of communication: the MT'ed text does not need to be grammatically correct as long as the reader can understand its message. In this paper we will focus on MT for dissemination; more specifically, when MT is used for postediting (PE). PE is the task of editing MT'ed texts, a common practice among translation providers, where the aim is to improve productivity and, consequently, reduce translation costs. However, when MT'ed sentences contain too many problems, it may be easier to translate from scratch than to post-edit MT (this is indeed often reported by translators [2]).
Some recent work has claimed that state-of-the-art MT can be used for dissemination without supervision, with a well-known example claiming to have achieved human parity for Chinese-to-English translation [3]. However, in-depth revisions of this work [4,5] suggest that we are still far from achieving human performance and, therefore, PE will still remain a key task in the translation industry. Therefore, finding ways of estimating the quality of MT'ed texts in terms of PE effort is a highly desirable feature (it would, for example, al-  low accurate budgeting of a translation job) and it is also a relevant topic for research in MT. According to Krings [6], PE effort has three dimensions: temporal, cognitive and technical. The temporal dimension is the one most easily related to professional productivity or throughput: one just has to directly measure the time spent by the post-editor in transforming the MT output into an adequate PE'ed version. PE time for a sentence may be expected to increase roughly linearly with the total number of words in a sentence; therefore, PE time is normalised by dividing it by the number of words in the MT'ed sentence. The measured ratio between PE time and the number of words in the segment (PETpW) can be directly used to assess the effort of post-editing a segment. The main drawback of extracting PE time is that it is relatively more expensive and requires posteditors to avoid breaks during the editing of a given sentence.
Previous work has proposed several ways to address this issue. For example, the shared task on quality estimation (QE) of MT organised yearly as part of WMT conferences started with the purpose of training models to predict perceived PE effort, moving later to predicting more accurate measurements such as actual PE time and the translation edit rate (TER) [7] observed when comparing a MT'ed sentence and its post-edited (PE'ed) counterparts, called humantargeted TER or HTER [8]. HTER gives an indirect indication of the effort needed to transform a MT'ed sentence into its PE'ed version.
Despite its popularity, HTER has been subject to criticism: Graham et al. [9] criticise this metric and contend that subjective direct assessments (DA) of adequacy are more reliable than HTER measurements [10,11]. They define adequacy as the degree to which the MT'ed segment expresses the meaning of the reference segment in the target language. Adequacy is therefore assessed in the target language, monolingually. Their DA is a combination of many independent human judgements of adequacy for a given sentence (in a 0%-100% scale) into a single score -standardised to zero mean and unit standard deviation after low-quality assessments are filtered out.
As discussed above, MT should be evaluated according to its purpose. Nevertheless, previous work has disregarded this assumption by using DA as gold standard for tasks where PE effort is the aspect of quality to be assessed [9,12,13]. Although we agree that DA may be a useful and reasonably cheap way of assessing subjectively the adequacy of MT output, in this paper we provide an in-depth analysis of ways to assess PE effort, with a focus on reducing PE time, a highly desirable feature by the translation industry.
We propose to assess the usefulness of metrics according to their ability to rank translations based on the time that would be required to post-edit them. This has a very practical application in the translation industry, where knowing which segments are easier to post-edit and which are the most difficult would allow a project manager to select post-editors accordingly, perhaps sending segments estimated as "easier" to less experienced (or cheaper) translators and/or sending the "most difficult" segments to experienced translators or to be translated from scratch.
Our main contributions are: • a comprehensive review of task-specific (PE-based) metrics (e.g. HTER), reference-based metrics (e.g. TER) and DA, where the goal is to rank MT'ed segments according to PE time; • the release of a dataset with source, MT'ed, reference, and PE'ed texts; detailed information about five independent post-editing jobs for each MT'ed text, and DA annotations; 1 • a new ranking score for MT'ed segments called splitaveraged, time-ratio assessment (SATRA). In Section 2 we present the dataset created and used for this paper. Section 3 presents our ranking analysis using all evaluation metrics available in our dataset. In section 4 we discuss related work. The paper ends with concluding remarks (Section 5).

Dataset and annotators
We extend the dataset made available by the WMT 2016 shared task on document-level quality estimation [14]. This dataset contains 1,047 segments totalling 26,875 words, MT'ed by 41 different systems -with an average of 26 segments per system-extracted from the test sets of English-Spanish WMT translation shared tasks between 2008 and 2012. Existing MT'ed segments were crowd-annotated (via Amazon Mechanical Turk) using DA scores made available by Graham et al. [12]. 2 Although the aim of the work presented in [12] was to generate DA scores at the document level, they first assessed each segment independently. Each segment has then a DA score and these are the values used in our experiments. The DA value of each segment is obtained by averaging the assessment of various annotators (previous work recommend at  Table 2: Statistics (mean and standard deviation) of task-specific (PE-based) metrics in our dataset least 15 annotations per segment [15], and the study in [12] follows the same protocol).
For the post-editing task, we hired five professional translators with experience in PE (hereafter referred to as ANN0, ANN1, ANN2, ANN3 and ANN4), who generated PE'ed versions for each segment of this dataset. The annotators used the PET tool [16], which records the edit operations performed during the PE task, including the time elapsed to post-edit a segment. We use PE time normalised by the length of the target segment in words, PETpW).
We then calculated the following task-specific (PEbased) metrics: • HTER, HBLEU and HMETEOR, respectively the TER, BLEU [17], and METEOR [18] scores of the MT'ed segment using the PE'ed version as reference, 3 and • Keys/char: ratio between the number of keys pressed by an annotator and the number of characters in the MT'ed segment. Since we have access to the references from the WMT datasets, we also calculated standard reference-based BLEU, METEOR and TER scores. Table 2 shows some statistics of the task-specific (PEbased) metrics extracted for this dataset. We show statistics per annotator and also the averaged values for all annotators (ALL). Statistics for DA, and reference-based metrics are shown in Table 3. All averages in both tables are weighted by the number of MT'ed words, as post-editing time -the measurement we want to track-is expected to grow linearly with sentence length. As may be seen, the values of quality indicators show a rather wide range.

Comparing the ranking ability of metrics
In order to analyse the performance of evaluation metrics as a proxy for PETpW, we propose experiments that look at how 3 These metrics were calculated using the Asiya toolkit [19].
these metrics rank MT'ed segments. We argue that looking at the rankings gives a reliable perspective of the usefulness of the metrics for the PE task, since it gives us the relative differences, in terms of effort, among the segments to be PE'ed.

Ranking correlation
In this experiment, we try to identify which metric produces rankings that are closest to PETpW rankings. Firstly, we calculate Spearman's ρ rank correlation coefficient between PETpW and all metrics. In addition to ρ, we also compute a new ranking score called split-averaged, time-ratio assessment (SATRA) for a ranking R as follows: , the average measured PETpW for segments R m to R n (those ranked m-th to n-th), where T (R j ) is the total PE time and L(R j ) the total length in (MT'ed) words for segment R j (ranked j-th). The value of SATRA(R) should be close to 1 for a random ranking (the average PETpW above any split of the ranking and that below the split should roughly be the same), smaller than 1 for a good ranking (one that would rank easier-to-post-edit segments better than hard-to-postedit ones), and the minimum possible for a ranking based on the measured PETpW. These two scores are used to measure how close two ranked distributions are. 4 Table 4 shows Spearman's ρ correlation coefficients and SATRA scores between all metrics and the individual PETpW of all annotators and the averaged values of all annotators (ALL). The last line of the table provides the scores obtained by an oracle using the actual PETpW as the ranking metric; this helps to interpret SATRA scores as, unlike Spearman's ρ, they do not have a fixed lower bound. DA shows moderate Spearman's ρ values across all annotators and for ALL, which are considerably smaller than those achieved by HTER, HBLEU, HMETEOR and Keys/char. SATRA shows similar results: DA presents larger (worse) values than the task-specific PE-based metrics. Following previous work [21], we calculate the statistical significance difference  Table 4: Spearman's ρ (↑) and SATRA (S, ↓) scores for all metrics using PETpW as gold standard. The best results are shown in bold and * means no statistically significant difference between the metrics according to Williams test with p < 0.01.  Table 5: Spearman's ρ (↑) and SATRA (↓) scores for all metrics using PETpW as gold standard for the leave-one-out experiment between all metrics using Williams' test over the Spearman's ρ scores (p < 0.01). 5 The large majority of the results are statistically different. The best overall metric is Keys/char, which achieves the highest Spearman's ρ scores and the lowest SATRA scores for all annotators individually and for ALL. The only annotator where the Keys/char metric is not so salient is ANN3. Our hypothesis is that this annotator may have interacted more with the mouse, 6 instead of with the keyboard. HTER, HBLEU and HMETEOR do not show significant differences among them. This is in line with the results reported by previous work [9] that found no difference between these metrics when correlating them to DA. Finally, independentreference-based metrics show the worst ranking scores with respect to PETpW.
In a real-world scenario, the PETpW of one annotator could be estimated based on the PETpW of other annotator(s). In order to simulate this case and evaluate whether the results from Table 4 would still stand, we performed leaveone-out experiments. In this case, SATRA and Spearman's ρ scores are calculated between each one of the studied metrics for one annotator and the averaged PETpW of all other annotators. For example, for ANN0, the PETpW is the average PETpW of ANN1 to ANN4, and its correlation with DA and the HTER, HBLEU, HMETEOR, Keys/char and PETpW for ANN0 post-edits. Table 5 shows the results of this experiment. As expected, the difference between Spearman's ρ and SATRA scores for HTER, HBLEU and HMETEOR and for 5 Williams test is calculated using mt-qe-eval: https://github. com/ygraham/mt-qe-eval. 6 PET records keyboard actions, but not mouse actions.
Keys/char is lower than in Table 4, since we are not dealing with the individual PETpW of each annotator. SATRA scores for PE-based metrics are better (lower) than DA (except for ANN1), and similarly for Keys/char (except for ANN1). In addition, Keys/char is not the best metric overall anymore, although it still shows the best SATRA in three out of five cases. In general, PE-based approaches still outperform DA in most cases. For reference, the last row of Table 5 shows Spearman's ρ and SATRA for the PETpW of each annotator versus the leave-one-out PETpW. It is worth mentioning that, with only five annotators, it is difficult to devise a model that would be a good estimator of quality for new annotators. In fact, after doing an analysis using the distribution-agnostic Kolmogorov-Smirnov test 7 over the PETpW distributions (considering p < 0.05), we identified three clusters of annotators where their PETpW measurements come from the same distribution. Basically, ANN0, ANN2 and ANN4 could be clustered together, whilst ANN1 and ANN3 would have their own clusters. This may be impacting our results, but a deeper analysis of the effect of such clusters is left for future work.

Analysis of tails
The experiments in this section aim to obtain a closer view of how the metrics studied perform for the best and the worst segments, by performing an analysis of the tails of the PETpW distribution. In other words, we want to analyse how the task-specific PE-based metrics, reference-based metrics, Figure 1: Number of segments shared between the 500 best sentences acording to PETpW and the other metrics and DA perform on the best and worst segments according to PETpW. Our experiment consists in counting the number of common segments between different cuts of the PETpW ranking and the rankings obtained with each different metric.
Best segments: firstly, we look at the first 500 sentences in the PETpW ranking, that is, the 500 easiest-to-post-edit sentences, and compare to the first 500 sentences in the rankings according to other metrics. We split the rankings in sets of 50 to show the performance of the metrics and the differences among them. Figure 1 shows the results of this experiment for all annotators individually and for the ALL case. For clarity we only show four metrics: Key/char, HTER, DA and BLEU. One can clearly identify three groups of metrics: • BLEU, TER and METEOR rankings behave similarly and show the lowest number of segments in common to the PETpW ranking; • HTER, HBLEU, HMETEOR, and Keys/char rankings are the best, sharing the largest number of segments with the PETpW ranking; • DA ranking is better than the reference-based metrics, but worse than the task-specific PE-based metrics. These findings are in agreement with those obtained when ranking all segments. Figure 2 shows scatter plots for DA vs. PETpW and HTER vs. PETpW averaged over all five annotators. The top two graphs show the scatter plots for the entire dataset. In this case, both metrics look similar in comparison to PETpW, although DA seems to show more outliers. The bottom two graphs show the scatter plots for the best 500 segments according to PETpW. In this case, HTER shows a clear tendency, where the majority of the values have a low HTER Worst segments: a similar trend is shown when we analyse the 500 worst segments (due to space constraints, Figure  3 only shows results for ALL), although the gap between DA and task-specific PE-based metrics is smaller. One hypothesis is that, for the worst segments, where the quality is very low, differences in adequacy track differences in PE time better.

Related work
In what follows we present previous work on human taskbased evaluation that targeted PE effort and on the use of DA for the same purpose.
PE time is a straightforward indicator of MT quality: segments that take longer to be PE'ed are considered worse than segments that can be quickly corrected. Koponen et al. [22] argue that PE time is the most effective way of measuring cognitive aspects of the PE task and relate them to the quality of the translations. Plitt and Masselot [23] use PETpW (actually, its converse: words per hour) to measure the gain in productivity when post-editing MT'ed text -in a real translation workflow-over the productivity when performing translation from scratch.
Perceived PE effort: humans are asked to give a score for the MT'ed sentences according to a Likert [24] scale representing perceived PE effort [25]. This type of score can be given with or without actual post-editing and it represents a judgement on how difficult it would be (or it was) to fix the given MT'ed sentence. Perceived PE effort scores were used in the WMT 2012 [20] and WMT 2014 [26] QE shared task editions.
Eye-tracking: previous work have also relied on eyetracking to evaluate PE effort. O'Brien [27] measures fixation time and correlates it with GTM (a similarity metric between the machine translation and the reference sentence based on precision, recall and F -measure [28]). Low GTM scores show correlation with high fixation time. PE pauses (extracted from keystroke logs) can also be viewed as an indirect measure of cognitive effort [29]. Long pauses are associated with segments that demand more cognitive PE effort.
Edit distance and n-gram-based scores: PE effort can also be evaluated indirectly, by using a metric that takes into account edit operations. HTER [8] is an example of such a metric, which computes the minimum number of edits to transform the machine translation into the PE'ed version. Task-specific, PE-based (human or H-) variants of commonly-used reference-based similarity measures have also been studied, such as HBLEU and HMETEOR. However, HTER is the most widely used as an indirect measurement of PE effort [14,26,30,31,32,33] DA: Graham et al. [10,11] propose the use of DA for MT evaluation. According to the authors, the biggest advantage of their approach in comparison to early practices of adequacy judgements is that they can reliably crowd-source the annotations. Graham et al. [9] also express a strong criticism of HTER on the grounds that it does not show high Pearson r correlation scores with DA. In another work [12], the same authors also criticise a variant of HTER for document-level QE, suggesting that DA is a more adequate metric to compare different QE systems. Recently, Bentivogli et al. [13] evaluate HTER and mTER (multi-reference TER) against DA scores and conclude that mTER is a better proxy for PE effort because it shows higher correlation scores with DA than HTER. However, our analysis on a real-world measurement of productivity (PETpW) show that PE-based metrics (including HTER) are the most adequate metrics to approximate PETpW, outperforming DA. Therefore, we argue that if mTER and HTER were compared using their correlations to PETpW, the results could be different (this analysis is left for future work).

Concluding remarks
The advancement and adoption of MT depends more than ever on the availability of reliable metrics to evaluate its quality. Averaged subjective direct assessment (DA) of MT quality, which is performed independently of purpose and may easily be crowd-sourced, has become very popular. However, in an important application of MT, namely dissemination via post-editing, it is only natural to use actual measurements that are obtained after performing post-editing. It is also natural for quality estimation models to target such metrics.
The results of our experiments on a dataset that includes PE indicators collected for five translators show that DA judgements provide a reasonable approximation of relevant, measurable aspects of MT usefulness in a dissemination task, such as PE time; however, as expected, task-specific metrics comparing MT'ed and PE'ed text -such as HTER or the number of keystrokes per raw MT character -are better trackers of PETpW. DA does however perform better than metrics such as BLEU, TER or METEOR with respect to an independent reference translation.
These results lead us to recommend that MT practitioners should use task-specific metrics wherever this is possible, and non-expert subjective judgements such as DA only when specific, measurable metrics are not available or feasible for a task.
Acknowledgements: Work supported by the Spanish government through project EFFORTUNE (TIN2015-69632-R) and grant PRX16/00043 for MLF, and by the European Commission through project GoURMET (No. 825299). CS and MLF are both first authors with equal contribution.