{'fraction_non_alphanumeric': 0.025672057798804417, 'fraction_numerical': 0.01540323467928265, 'mean_word_length': 4.945922372437854, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Is Machine Translation Getting Better over Time?

Recent human evaluation of machine translation has focused on relative preference judgments of translation quality, making it difficult to track longitudinal improvements over time. We carry out a large-scale crowd-sourcing experiment to estimate the degree to which state-of-theart performance in machine translation has increased over the past five years. To facilitate longitudinal evaluation, we move away from relative preference judgments and instead ask human judges to provide direct estimates of the quality of individual translations in isolation from alternate outputs. For seven European language pairs, our evaluation estimates an average 10-point improvement to state-of-theart machine translation between 2007 and 2012, with Czech-to-English translation standing out as the language pair achieving most substantial gains. Our method of human evaluation offers an economically feasible and robust means of performing ongoing longitudinal evaluation of machine translation.


Introduction
Human evaluation provides the foundation for empirical machine translation (MT), whether human judges are employed directly to evaluate system output, or via the use of automatic metricsvalidated through correlation with human judgments. Achieving consistent human evaluation is not easy, however. Annual evaluation campaigns conduct large-scale human assessment but report ever-decreasing levels of judge consistency -when given the same pair of translations to repeat-assess, even expert human judges will worryingly often contradict both the preference judg-ment of other judges and their own earlier preference (Bojar et al., 2013). For this reason, human evaluation has been targeted within the community as an area in need of attention, with increased efforts to develop more reliable methodologies.
One standard platform for human evaluation is WMT shared tasks, where assessments have (since 2007) taken the form of ranking five alternate system outputs from best to worst (Bojar et al., 2013). This method has been shown to produce more consistent judgments compared to fluency and adequacy judgments on a five-point scale (Callison-Burch et al., 2007). However, relative preference judgments have been criticized for being a simplification of the real differences between translations, not sufficiently taking into account the large number of different types of errors of varying severity that occur in translations (Birch et al., 2013). Relative preference judgments do not take into account the degree to which one translation is better than another -there is no way of knowing if a winning system produces far better translations than all other systems, or if that system would have ranked lower if the severity of its inferior translation outputs were taken into account.
Rather than directly aiming to increase human judge consistency, some methods instead increase the number of reference translations available to automatic metrics. HTER (Snover et al., 2006) employs humans to post-edit each system output, creating individual human-targeted reference translations which are then used as the basis for computing the translation error rate. HyTER, on the other hand, is a tool that facilitates creation of very large numbers of reference translations (Dreyer and Marcu, 2012). Although both approaches increase fairness compared to automatic metrics that use a single generic reference translation, even human post-editors will inevitably vary in the way they post-edit translations, and the process of creating even a single new reference trans-lation for each system output is often too resourceintensive to be used in practice.
With each method of human evaluation, a tradeoff exists between annotation time and the number of judgments collected. At one end of the spectrum, the WMT human evaluation collects large numbers of quick judgments (approximately 3.5 minutes per screen, or 20 seconds per label) (Bojar et al., 2013). 1 In contrast, HMEANT (Lo and Wu, 2011) uses a more time-consuming fine-grained semantic-role labeling analysis at a rate of approximately 10 sentences per hour (Birch et al., 2013). But even with this detailed evaluation methodology, human judges are inconsistent (Birch et al., 2013).
Although the trend appears to be toward more fine-grained human evaluation of MT output, it remains to be shown that this approach leads to more reliable system rankings -with a main reason to doubt this being that far fewer judgments will inevitably be possible. We take a counterapproach and aim to maintain the speed by which assessments are collected in shared task evaluations, but modify the evaluation set-up in two main ways: (1) we structure the judgments as monolingual tasks, reducing the cognitive load involved in assessing translation quality; and (2) we apply judge-intrinsic quality control and score standardization, to minimize noise introduced when crowd-sourcing is used to leverage numbers of assessments and to allow for the fact that human judges will vary in the way they assess translations. Assessors are regarded as reliable as long as they demonstrate consistent judgments across a range of different quality translations.
We elicit direct estimates of quality from judges, as a quantitative estimate of the magnitude of each attribute of interest (Steiner and Norman, 1989). Since we no longer look for relative preference judgments, we revert back to the original fluency and adequacy criteria last used in WMT 2007 shared task evaluation. Instead of fivepoint fluency/adequacy scales, however, we use a (100-point) continuous rating scale, as this facilitates more sophisticated statistical analyses of score distributions for judges, including workerintrinsic quality control for crowd-sourcing. The latter does not depend on agreement with experts, and is made possible by the reduction in information-loss when a continuous scale is used. In addition, translations are assessed in isolation from alternate system outputs, so that judgments collected are no longer relative to a set of five translations. This has the added advantage of eliminating the criticism made of WMT evaluations that systems sometimes gain advantage from luckof-the-draw comparison with low quality output, and vice-versa (Bojar et al., 2011).
Based on our proposed evaluation methodology, human judges are able to work quickly, on average spending 18 and 13 seconds per single segment adequacy and fluency judgment, respectively. Additionally, when sufficiently large volumes of such judgments are collected, mean scores reveal significant differences between systems. Furthermore, since human evaluation takes the form of direct estimates instead of relative preference judgments, our evaluation introduces the possibility of large-scale longitudinal human evaluation. We demonstrate the value of longitudinal evaluation by investigating the improvement made to stateof-the-art MT over a five year time period (between 2007 and 2012) using the best participating WMT shared task system output. Since it is likely that the test data used for shared tasks has varied in difficulty over this time period, we additionally propose a simple mechanism for scaling system scores relative to task difficulty.
Using the proposed methodology for measuring longitudinal change in MT, we conclude that, for the seven European language pairs we evaluate, MT has made an average 10% improvement over the past 5 years. Our method uses non-expert monolingual judges via a crowd-sourcing portal, with fast turnaround and at relatively modest cost.

Monolingual Human Evaluation
There are several reasons why the assessment of MT quality is difficult. Ideally, each judge should be a native speaker of the target language, while at the same time being highly competent in the source language. Genuinely bilingual people are rare, however. As a result, judges are often people with demonstrated skills in the target language, and a working knowledge -often self-assessedof the source language. Adding to the complexity is the discipline that is required: the task is cognitively difficult and time-consuming when done properly. The judge is, in essence, being asked to decide if the supplied translations are what they would have generated if they were asked to do the same translation.
The assessment task itself is typically structured as follows: the source segment (a sentence or a phrase), plus five alternative translations and a "reference" translation are displayed. The judge is then asked to assign a rank order to the five translations, from best to worst. A set of pairwise preferences are then inferred, and used to generate system rankings, without any explicit formation of stand-alone system "scores".
This structure introduces the risk that judges will only compare translations against the reference translation. Certainly, judges will vary in the degree they rely on the reference translation, which will in turn impact on inter-judge inconsistency. For instance, even when expert judges do assessments, it is possible that they use the reference translation as a substitute for reading the source input, or do not read the source input at all. And if crowd-sourcing is used, can we really expect high proportions of workers to put the additional effort into reading and understanding the source input when a reference translation (probably in their native language) is displayed? In response to this potential variability in how annotators go about the assessment task, we trial assessments of adequacy in which the source input is not displayed to human judges. We structure assessments as a monolingual task and pose them in such a way that the focus is on comparing the meaning of reference translations and system outputs. 2 We therefore ask human judges to assess the degree to which the system output conveys the same meaning as the reference translation. In this way, we focus the human judge indirectly on the question we wish to answer when assessing MT: does the translation convey the meaning of the source? The fundamental assumption of this approach is that the reference translation accurately captures the meaning of the source; once that assumption is made, it is clear that the source is not required during the evaluation.
Benefits of this change are that the task is both easier to describe to novice judges, and easier to answer, and that it requires only monolingual speakers, opening up the evaluation to a vastly larger pool of genuinely qualified workers.
With this set-up in place for adequacy, we also re-introduce a fluency assessment. Fluency ratings can be carried out without the presence of a reference translation, reducing any remnant bias towards reference translations in the evaluation setup. That is, we propose a judgment regime in which each task is presented as a two-item fluency and adequacy judgment, evaluated separately, and with adequacy restructured into a monolingual "similarity of meaning" task. When fluency and adequacy were originally used for human evaluation, each rating used a 5point adjective scale (Callison-Burch et al., 2007). However, adjectival scale labels are problematic and ratings have been shown to be highly dependent on the exact wording of descriptors (Seymour et al., 1985). Alexandrov (2010) provides a summary of the extensive problems associated with the use of adjectival scale labels, including bias resulting from positively-and negatively-worded items not being true opposites of one another, and items intended to have neutral intensity in fact proving to have specific conceptual meanings.
It is often the case, however, that the question could be restructured so that the rating scale no longer requires adjectival labels, by posing the question as a statement such as The text is fluent English and asking the human assessor to specify how strongly they agree or disagree with that statement. The scale and labels can then be held constant across experimental set-ups for all attributes evaluated -meaning that if the scale is still biased in some way it will be equally so across all set-ups.

Assessor Consistency
One way of estimating the quality of a human evaluation regime is to measure its consistency: whether or not the same outcome is achieved if the same question is asked a second time. In MT, annotator consistency is commonly measured using Cohen's kappa coefficient, or some variant thereof (Artstein and Poesio, 2008). Originally developed as a means of establishing assessor independence, it is now commonly used in the reverse sense, with high numeric values being used as evidence of agreement. Two different measurements can be made -whether a judge is consistent with other judgments performed by themselves (intraannotator agreement), and whether a judge is consistent with other judges (inter-annotator agreement).
Cohen's kappa is intended for use with categor-ical judgments, but is also commonly used with five-point adjectival-scale judgments, where the set of categories has an explicit ordering. One particular issue with five-point assessments is that score standardization cannot be applied. As such, a judge who assigns two neighboring intervals is awarded the same "penalty" for being "different" as the judge who chooses the extremities. The kappa coefficient cannot be directly applied to many-valued interval or continuous data. This raises the question of how we should evaluate assessor consistency when a continuous rating scale is in place. No judge, when given the same translation to judge twice on a continuous rating scale, can be expected to give precisely the same score for each judgment (where repeat assessments are separated by a considerable number of intervening ones). A more flexible tool is thus required. We build such a tool by starting with two core assumptions: A: When a consistent assessor is presented with a set of repeat judgments, the mean of the initial set of assessments will not be significantly different from the mean score of repeat assessments.
B: When a consistent judge is presented with a set of judgments for translations from two systems, one of which is known to produce better translations than the other, the mean score for the better system will be significantly higher than that of the inferior system.
Assumption B is the basis of our quality-control mechanism, and allows us to distinguish between Turkers who are working carefully and those who are merely going through the motions. We use a 100-judgment HIT structure to control same-judge repeat items and deliberately-degraded system outputs (bad reference items) used for workerintrinsic quality control (Graham et al., 2013). bad reference translations for fluency judgments are created as follows: two words in the translation are randomly selected and randomly re-inserted elsewhere in the sentence (but not as the initial or final words of the sentence).
Since adding duplicate words will not degrade adequacy in the same way, we use an alternate method to create bad reference items for adequacy judgments: we randomly delete a short sub-string of length proportional to the length of the original translation to emulate a missing phrase. Since  this is effectively a new degradation scheme, we tested against experts. For low-quality translations, deleting just two words from a long sentence often made little difference. The method we eventually settled on removes a sequence of k words, as a function of sentence length n: To filter out careless workers, scores for bad reference pairs are extracted, and a difference-of-means test is used to calculate a worker-reliability estimate in the form of a p-value. Paired tests are then employed using the raw scores for degraded and corresponding system outputs, using a reliability significance threshold of p < 0.05. If a worker does not demonstrate the ability to reliably distinguish between a bad system and a better one, the judgments from that worker are discarded. This methodology means that careless workers who habitually rate translations either high or low will be detected, as well as (with high probability) those that click (perhaps via robots) randomly. It also has the advantage of not filtering out workers who are internally consistent but whose scores happen not to correspond particularly well to a set of expert assessments.
Having filtered out users who are unable to reliably distinguish between better and worse sets of translations (p ≥ 0.05), we can now examine how well Assumption A holds for the remaining users, i.e. the extent to which workers apply consistent scores to repeated translations. We compute mean scores for the initial and repeat items and look for even very small differences in the two distributions for each worker. Table 1 shows numbers of workers who passed quality control, and also that Si Si+5 1 bad reference its corresponding system output 1 system output a repeat of it 1 reference its corresponding system output Above in reverse for Si and Si+5 4 system outputs 4 system outputs

Five Years of Machine Translation
To estimate the improvement in MT that took place between 2007 and 2012, we asked workers on Amazon's Mechanical Turk (MTurk) to rate the quality of translations produced by the bestreported participating system for each of WMT 2007 and WMT 2012 (Callison-Burch et al., 2007;Callison-Burch et al., 2012). Since it is likely that the test set has changed in difficulty over this time period, we also include in the evaluation the original test data for 2007 and 2012, translated by a single current MT system. We use the latter to calibrate the results for test set difficulty, by calculating the average difference in rating, ∆, between the 2007 and 2012 test sets. This is then added to the difference in rating for the best-reported systems in 2012 and 2007, to arrive at an overall evaluation of the 5-year gain in MT quality for a given language pair, separately for fluency and adequacy. Experiments were carried out for each of German, French and Spanish into and out of English, and also for Czech-to-English. English-to-Czech was omitted because of a low response rate on MTurk. For language pairs where two systems tied for first place in the shared task, a random selection of translations from both systems was made.

HIT structure
To facilitate quality control, we construct each HIT on MTurk as an assessment of 100 translations. Each individual translation is rated in isolation from other translations with workers required to iterate through 100 translations without the opportunity to revisit earlier assessments. A 100-translation HIT contains the following items: 70 randomly selected system outputs made up of roughly equal proportions of translations for each evaluated system, 10 bad reference translations (each based on one of the 70 system outputs), 10 exact repeats and 10 reference translations. We divide a 100-translation HIT into 10 sets of 10 translations. Table 2 shows how the content of each set is determined. Translations are then randomized only within each set (of 10 translations), with the original sequence order of the sets preserved. In this way, the order of quality control items is unpredictable but controlled so pairs are separated by a minimum of 40 intervening assessments (4 sets of translations). The HIT structure results in 80% of assessed translations corresponding to genuine outputs of a system (including exact repeat assessments), which is ultimately what we wish to obtain, with 20% of assessments belonging to quality control items (bad reference or reference translations).

Assessment set-up
Separate HITs were provided for evaluation of fluency and adequacy. For fluency, a single system output was displayed per screen, with a worker required to rate the fluency of a translation on a 100point visual analog scale with no displayed point scores. A similar set-up was used for adequacy but with the addition of a reference translation (displayed in gray font to distinguish it from the system output being assessed). The Likert-type statement that framed the judgment was Read the text below and rate it by how much you agree that: • [for fluency] the text is fluent English • [for adequacy] the black text adequately expresses the meaning of the gray text.
In neither case was the source language string provided to the workers. Tasks were published on MTurk, with no region restriction but the stipulation that only native speakers of the target language should complete HITs, and with a qualification of an MTurk prior HIT-approval rate of at least 95%. Instructions were always presented in the target language. Workers were paid US$0.50 per fluency HIT, and US$0.60 per adequacy HIT. 3 Close to one thousand individual Turkers contributed to this experiment (some did both fluency and adequacy assessments), providing a total of more than 220,000 translations, of which 140,000 were provided by workers meeting the quality threshold.
In general, it cost approximately US$30 to assess each system, with low-quality workers approximately doubling the cost of the annotation. We rejected HITs where it was clear that randomclicking had taken place, but did not reject solely on the basis of having not met the quality control threshold, to avoid penalizing well-intentioned but low-quality workers.
Overall change in performance Table 3 shows the overall gain made in five years, from WMT 07 to WMT 12. Mean scores for the two top-performing systems from each shared task (BEST 07 , BEST 12 ) are included, as well as scores for the benchmark current MT system on the two test sets (CURR 07 , CURR 12 ). For each language pair, a 100-translation HIT was constructed by randomly selecting translations from the pool of (3003 + 2007) × 2 that were available, and this results in apparently fewer assessments for the 2007 test set. In fact, numbers of evaluated translations are relative to the size of each test set. Average z scores for each system are also presented, based on the mean and standard deviation of all assessments provided by an individual worker, with positive values representing deviations above the mean of workers. In addition, we include mean BLEU (Papineni et al., 2001) and METEOR (Banerjee and Lavie, 2005) automatic scores for the same system outputs.
The CURR benchmark shows fluency scores that are 5.9 points higher on the 2007 data set than they are on the 2012 test data, with a larger difference in adequacy of 8.3 points. As such, the 2012 test data is more challenging than the 2007 test data. Despite this, both fluency and adequacy scores for the best system in 2012 have increased by 4.5 and 2.0 points respectively, amounting to estimated average gains of 10.4 points in fluency and 10.3 points in adequacy for state-of-the-art MT across the seven language pairs. Looking at the standardized scores, it is apparent that the presence of the CURR translations for the 2007 test set pushes the mean score for the 2007 best systems below zero. The presence in the HITs of reference translations also shifts stan-dardized system evaluations below zero, because they are not attributable to any of the systems being assessed. 4 Results for automatic metrics lead to similar conclusions: that the test set has indeed increased in difficulty; and that, in spite of this, substantial improvements have been made according to automatic metrics, +13.5 using BLEU, and +7.1 on average using METEOR. Table 4 shows mean fluency and adequacy scores by language pair for translation into English. Relative gains in both adequacy and fluency for the to-English language pairs are in agreement with the estimates generated through the use of the two automatic metrics. Most notably, Czech-to-English translation appears to have made substantial gains across the board, achieving more than double the gain made by some of the other language pairs; results for best participating 2007 systems show that this may in part be caused by the fact that Czechto-English translation had a lower 2007 baseline to begin with (BEST 07 F:40.8; A:41.7) in comparison to, for example, Spanish-to-English translation (BEST 07 F:56.7; A:59.0).

Language pairs
Another notable result is that although the test data for each year's shared task is parallel across five languages, test set difficulty increases by different degrees according to human judges and automatic metrics, with BLEU scores showing substantial divergence across the to-English language pairs. Comparing BLEU scores achieved by the benchmark system for Spanish to English and Czech-to-English, for example, the benchmark system achieves close scores on the 2007 test data with a difference of only |52.3 − 51.2| = 1.1, compared to the score difference for the benchmark scores for translation of the 2012 test data of |25.0 − 38.3| = 13.3. This may indicate that the increase in test set difficulty that has taken place over the years has made the shared task disproportionately more difficult for some language pairs than for others. It does seem that some language pairs are harder to translate than others, and the differential change may be a consequence of the fact that increasing test set complexity for all languages in parallel has a greater impact on translation difficulty for language pairs that are intrinsically harder to translate between.    Table 5, * denotes significance at p < 0.05; * * significance at p < 0.01; and * * * significance at p < 0.001. Table 5 shows results for translation out-of English, and once again human evaluation scores are in agreement with automatic metrics with Englishto-Spanish translation achieving most substantial gains for the three out-of-English language pairs, an increase of 12.4 points for fluency, and 11.8 points with respect to adequacy, while Englishto-French translation achieves a gain of 8.8 for CURR07  fluency and 7.4 points for adequacy. English-to-German translation achieves the lowest gain of all languages, with apparently no improvement in fluency, as the human fluency evaluation of the benchmark system on the supposedly easier 2007 data receives a substantially lower score than the same system over the 2012 data. This result demonstrates why fluency, evaluated without a reference translation, should not be used to evaluate MT systems without an adequacy assessment, since it is entirely possible for a low-adequacy translation to achieve a high fluency score. For all language pairs, Figure 1 plots the net gain in fluency, adequacy and F 1 against increase in test data difficulty.

Conclusion
We carried out a large-scale human evaluation of best-performing WMT 2007 and 2012 shared task systems in order to estimate the improvement made to state-of-the-art machine translation over this five year time period. Results show significant improvements have been made in machine translation of European language pairs, with Czechto-English recording the greatest gains. It is also clear from our data that the difficulty of the task has risen over the same period, to varying degrees for individual language pairs. Researchers interested in making use of the dataset are invited to contact the first author.