{'fraction_non_alphanumeric': 0.03611407249466951, 'fraction_numerical': 0.009328358208955223, 'mean_word_length': 4.777136258660508, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Iqmt: A Framework for Automatic Machine Translation Evaluation

We present the IQMT Framework for Machine Translation Evaluation Inside QARLA. IQMT offers a common workbench in which existing evaluation metrics can be utilized and combined. It provides i) a measure to evaluate the quality of any set of similarity metrics (KING), ii) a measure to evaluate the quality of a translation using a set of similarity metrics (QUEEN), and iii) a measure to evaluate the reliability of a test set (JACK). The first release of the IQMT package is freely available for public use. Current version includes a set of 26 metrics from 7 different well-known metric families, and allows the user to supply its own metrics. For future releases, we are working on the design of new metrics that are able to capture linguistic aspects of translation beyond lexical ones.


Introduction
In the last years, it has been repeatedly argued that current Machine Translation (MT) evaluation metrics do not capture well possible improvements attained by means of incorporating linguistic knowledge to MT systems (Och et al., 2003). One of the possible reasons is that most of the current metrics do not take into account any information at linguistic levels further than lexical. This is the case of metrics such as BLEU (Papineni et al., 2001), NIST (Doddington, 2002), WER (Nießen et al., 2000), PER (Leusch et al., 2003), GTM (Melamed et al., 2003), ROUGE (Lin and Och, 2004a), and METEOR (Banerjee and Lavie, 2005). More sophisticated metrics are required. However, little work has been done in that direction. For instance, metrics such as ROUGE and METEOR may consider stemming. We may also find the WNM metric (Babych and Hartley, 2004), a variant of BLEU which weights n-grams according to their statistical salience estimated out from a monolingual corpus. Additionally, METEOR may perform a lookup for synonymy in WordNet (Fellbaum, 1998). But all these are still attempts at the lexical level. To our knowledge, the only attempt so far to exploit information at an upper level has been done by Liu and Gildea (2005) who introduced a series of syntax-based features based on syntactic tree matching. Doubtless the design of a metric that is able to capture all the linguistic aspects that distinguish 'correct' translations from 'incorrect' ones is an ambitious and difficult goal. Instead of building such a sophisticated metric we suggest to follow a 'divide and conquer' strategy, and design a set of specialized metrics, devoted to the evaluation of partial aspects of MT quality. The new challenge is how to combine their outputs into a single measure. In a recent work, Kulesza and Shieber (2004) tried to combine some aspects of different metrics by applying machine learning techniques to build classifiers that distinguished between human-generated ('good') and machine-generated ('bad') translations. They used features inspired in metrics like BLEU, NIST, WER and PER. Our approach is based on QARLA , a probabilistic framework originally designed for the evaluation of text summarization systems. QARLA automatically identifies the features that distinguish human translations from automatic ones. It permits metric combinations, without any a-priori weighting of their relative importance. Besides, no training or adjustment of parameters is required, there is no need for human assessments, and it does not depend on the scale properties of the metrics being evaluated. The methodology which is closest to QARLA is ORANGE (Lin and Och, 2004b). However, ORANGE does not permit metric combinations. Our initial experiments applying QARLA to MT evaluation are discussed in . As a result of our experience, in this work we present the IQMT 1 Framework for MT Evaluation, a public workbench in which similarity metrics may be robustly combined. The rest of the paper is organized as follows. Section 2 describes the fundamentals of QARLA. The IQMT architecture is deployed in Section 3. Section 4 presents a case of study on the evaluation of the Europarl Corpus Spanish-to-English translation task. Finally, ongoing work is outlined in Section 5.

Fundamentals
IQMT is based on the QARLA Framework . QARLA uses similarity to models (human references) as a building block. The main assumption is that all human references are equally optimal and, while they are likely to be different, the best similarity metric is the one that identifies and uses the features that are common to all human references, grouping and separating them from automatic translations. Therefore, one of the main characterisitics of QARLA that differentiates it from other approaches, is that, besides considering the similarity of automatic translations to human references, QARLA additionally considers the distribution of similarities among human references. The input for QARLA is a set of test cases , a set of similarity metrics ¡ , and sets of models ¢ for each test case. With such a testbed, QARLA provides three measures: . This measure has some interesting properties: (i) it is able to combine different similarity metrics into a single evaluation measure.
(ii) it is not affected by the scale properties of individual metrics, i.e. it does not require metric normalisation and it is not affected by metric weighting.
(iii) Peers (automatic translations) which are very far from the set of models (human references) all receive QUEEN=0. In other words, QUEEN does not distinguish between very poor translation strategies.
(iv) The value of QUEEN is maximised for peers that "merge" with the models under all metrics in ¡ .
(v) The universal quantifier on the metric parameter 8 implies that adding redundant metrics does not bias the result of QUEEN.
However, the main drawback of QUEEN is that it requires the use of multiple references (at least three), when in most cases only a single reference translation is available.

KING
Based on QUEEN, QARLA provides a mechanism to determine the quality of a set of metrics, the KING measure: KING represents the probability that, for a given set of human references ¢ , and a set of metrics ¡ , the QUEEN quality of a human reference is greater than the QUEEN quality of any automatic translation in . Therefore, KING measures the ability of a set of metrics to discern between automatic and human translations.

JACK
Again based on QUEEN, QARLA provides a mechanism to determine the reliability of the test set, the JACK measure:
The set of similarity metrics is a dynamic component in our framework. We have started by addapting existing MT evaluation metrics. These metrics are transformed into similarity metrics by considering just a single reference when computing its value. Current version integrates 26 variants from 7 families of metrics (BLEU, NIST, WER and PER, GTM, ROUGE, and METEOR) 2 . New metrics will be added in the future. But the main advantage of the 'IQ XML' representation schema is that it allows users to supply their own metrics in a transparent and unified manner. For every new metric, the user is responsible for generating an IQ XML similarity file for each pair system-reference+w , reference-reference+w , and system-systemw .

IQeval
IQeval allows us to calculate the KING, QUEEN, and JACK measures. Several options must be specified:  This information is specified according to a configuration file. The IQsetup component generates, as a by-pass product, a default configuration file for IQeval. This configuration file contains a series of predefined sets. It must be edited in order to define new sets. Other options are available: £ the subset of sentences per system to evaluate (all sentences are considered by default). £ level of granularity (sentence/system) (system level results are printed by default).
IQeval also permits to obtain individual metric scores outside QARLA.

Finding an Optimal Metric Set
The optimal set is defined by the combination of metrics exhibiting the highest KING value. However, exploring all possible combinations is not viable 3 . IQeval provides an implementation of a simple algorithm which performs an approximate search in order to find a suboptimal set of metrics: 1. Individual metrics are ranked by their KING value.
2. Following that order, metrics are individually added to the set of optimal metrics only if the global KING increases.

A case of study: Europarl
For a robust estimation of the KING, QUEEN, and JACK probabilities, the ideal scenario would consist of a large number of human references per sentence, and automatic outputs generated by heterogeneous MT systems. Unfortunately, this kind scenario is rarely found. Generally, few references are available (one in most cases), and MT systems are similar to each other. Thus, we have tested our system under a more realistic scenario. We utilize the data from the 'Openlab 2006' Initiative 4 promoted by the TC-STAR 5 Consortium. SMT systems are built as described in (Giménez and Màrquez, 2005). As to 'SYSTRAN', we used the freely available on-line version 8 . Let us note that evaluation is unfair to 'SYSTRAN' because SMT systems have been trained using in-domain data. However, we include 'SYS-TRAN' for the sake of heterogeneity. We use all the 26 currently available metric variants.

Evaluating with Standard Metrics
First we analyze the individual behaviour of standard metrics. We use one representative from each family, the metric variant with highest KING value in the given test set. See, in Table 1, the poor level of agreement between metrics. For instance, according to '1-PER' and '1-WER' the word-based SMT system ('WB') is best. However, according to the rest of metrics the phrase-based systems ('PB' and 'PB++') are best, obtaining very similar scores. Also note that, contrary to our expectations, the 'SYSTRAN' system outperforms the word-based system according to five metrics and the two phrase-based systems according to two metrics. Therefore, the key question is "which metric should I trust?".

Evaluating with IQMT
Inside the IQMT Framework systems are evaluated according to their human-likeness. Thus, we must trust the metric (or set of metrics) with highest descriptive power (highest KING), i.e. the metric which best identifies the features that distinguish between human translations and automatic translations. We apply the algorithm described in Subsection 3.3. In the case of the 'Openlab 2006' data, we can count only on three human references per sentence. In order to increase the number of samples for QUEEN estimation we can use reference similarities between manual translation pairs from other sentences, assuming that the distances between manual references are relatively stable across examples. The optimal set is: NIST-2, NIST-3, NIST-4, and 1-WER© It attains a KING measure of 0.38, which means that in 38% of the cases this metric set is able to identify human references with respect to all automatic translations. Interestingly, the optimal set contains metrics working at all levels of granularity from 1-grams to 4-grams. We use this metric set to compute the QUEEN measure for all systems. See results at the system level in Table 2. As expected, phrase-based systems attain best results, significantly better than the word-based system and 'SYSTRAN'.  'PB++' slightly outperforms 'PB', although not very significantly. Interestingly, the 'SYSTRAN' systems performs significantly better than the 'WB' system. This means that, in this test set, translations produced by 'SYSTRAN' are more human-like than those produced by the word-based SMT system, even though 'SYSTRAN' is not designed for the specific domain. Moreover, the QUEEN measure at the sentence level allows the user to perform a detailed error analysis by inspecting particular cases. Table 3 shows an interesting case of error analysis, in which all systems attain a QUEEN score under 0.2 except the 'PB++' system which scores 0.83. The QUEEN measure identifies the features which characterize human translations. QUEEN favours those automatic translations which share these features that are common to all references. In this case the 'PB++' system output is rewarded for providing exact translations, according to all references, for 'gestión de las crisis' ('crisis management') and 'esperan señales' ('they expect signs'). On the other hand, the automatic translations which do not share these common features are penalized.

MT System 1-PER 1-WER BLEU-3 GTM-2 MTR-exact NIST-3 RG-L
Finally, the quality of the given test set of systems, references and metrics (JACK measure), considering the optimal metric set, is 0.77. This means that, in most cases (77%), system outputs are heterogeneously distributed closely around human references according to all metrics, and consequently, the test set is representative and reliable.

Ongoing work
Currently, we are devoting our main efforts to the development of syntax-based metrics. We are experimenting with metrics based on dependency trees provided by MINI-PAR (Lin, 1998). We are also developing metrics based on shallow parsing annotation, i.e. part-of-speech, lemma and chunk information, provided by the SVMTool 9 (Giménez and Màrquez, 2004), the Freeling 10 package (Carreras et al., 2004) and the Phreco software, respectively. On the 9 The SVMTool may be freely downloaded at http://www.lsi.upc.es/˜nlp/SVMTool/ . 10 Freeling Suite of Language Analyzers may be freely downloaded at http://www.lsi.upc.es/˜nlp/freeling/ one hand we intend to capture morphosyntactic and syntactic similarities between automatic and reference translations. On the other hand we aim to capture partial aspects such as the proportion of correctly translated nouns and verbs, or the proportion of correctly translated noun and verb phrases. In the future, we intend to move on to other linguistic levels, such as Full Parsing or Semantic Role Labeling.
source los ciudadanos esperan de nosotros algo más que la simple gestión de las crisis ; esperan señales y una política sostenible en estosámbitos . systems WB the citizens expect of us something more than the simple management of the crisis and a sustainable policy in these areas . expectantly signals SYSTRAN the citizens wait for of us something more than the simple management of the crises; they wait for signals and a sustainable policy in these scopes.

PB
the citizens expect us any more than simply managing crises ; they hope signals and a sustainable policy in these areas .

PB++
the citizens expect us something more than simply crisis management ; they expect signs and a sustainable policy in these areas .

R0
the public expect more than just crisis management ; they expect signs , and a sustainable policy in these fields .

R1
citizens expect something more of us than just simple crisis management ; they expect signs and sustainable policies in these areas .

R2
the citizens expect from us something more than a simple crisis management ; they expect signs and a sustainable policy in these matters .