{'fraction_non_alphanumeric': 0.042716594719405684, 'fraction_numerical': 0.019626543519726935, 'mean_word_length': 4.572867132867133, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Automatic Evaluation Method for Machine Translation Using Noun-Phrase Chunking

As described in this paper, we propose a new automatic evaluation method for machine translation using noun-phrase chunking. Our method correctly determines the matching words between two sentences using corresponding noun phrases. Moreover, our method determines the similarity between two sentences in terms of the noun-phrase order of appearance. Evaluation experiments were conducted to calculate the correlation among human judgments, along with the scores produced using automatic evaluation methods for MT outputs obtained from the 12 machine translation systems in NTCIR-7. Experimental results show that our method obtained the highest correlations among the methods in both sentence-level adequacy and fluency.


Introduction
High-quality automatic evaluation has become increasingly important as various machine translation systems have developed. The scores of some automatic evaluation methods can obtain high correlation with human judgment in document-level automatic evaluation (Coughlin, 2007). However, sentence-level automatic evaluation is insufficient. A great gap exists between language processing of automatic evaluation and the processing by humans. Therefore, in recent years, various automatic evaluation methods particularly addressing sentence-level automatic evaluations have been proposed. Methods based on word strings (e.g., BLEU (Papineni et al., 2002), NIST(NIST, 2002), METEOR (Banerjee and Lavie., 2005), ROUGE-L (Lin and Och, 2004), and IMPACT (Echizen-ya and Araki, 2007)) calculate matching scores using only common words between MT outputs and references from bilingual humans. However, these methods cannot determine the correct word correspondences sufficiently because they fail to focus solely on phrase correspondences. Moreover, various methods using syntactic analytical tools (Pozar and Charniak, 2006;Mutton et al., 2007;Mehay and Brew, 2007) are proposed to address the sentence structure. Nevertheless, those methods depend strongly on the quality of the syntactic analytical tools.
As described herein, for use with MT systems, we propose a new automatic evaluation method using noun-phrase chunking to obtain higher sentence-level correlations. Using noun phrases produced by chunking, our method yields the correct word correspondences and determines the similarity between two sentences in terms of the noun phrase order of appearance. Evaluation experiments using MT outputs obtained by 12 machine translation systems in NTCIR-7 (Fujii et al., 2008) demonstrate that the scores obtained using our system yield the highest correlation with the human judgments among the automatic evaluation methods in both sentence-level adequacy and fluency. Moreover, the differences between correlation coefficients obtained using our method and other methods are statistically significant at the 5% or lower significance level for adequacy. Results confirmed that our method using noun-phrase chunking is effective for automatic evaluation for machine translation.

Automatic Evaluation Method using Noun-Phrase Chunking
The system based on our method has four processes. First, the system determines the corre-spondences of noun phrases between MT outputs and references using chunking. Secondly, the system calculates word-level scores based on the correct matched words using the determined correspondences of noun phrases. Next, the system calculates phrase-level scores based on the noun-phrase order of appearance. The system calculates the final scores combining word-level scores and phrase-level scores.

Correspondence of Noun Phrases by Chunking
The system obtains the noun phrases from each sentence by chunking. It then determines corresponding noun phrases between MT outputs and references calculating the similarity for two noun phrases by the PER score (Su et al., 1992). In that case, PER scores of two kinds are calculated. One is the ratio of the number of match words between an MT output and reference for the number of all words of the MT output. The other is the ratio of the number of match words between the MT output and reference for the number of all words of the reference. The similarity is obtained as an F -measure between two PER scores. The high score represents that the similarity between two noun phrases is high. Figure 1 presents an example of the determination of the corresponding noun phrases. (2) Determination of corresponding noun phrases 1.0000 0.3714 0.7429 Figure 1: Example of determination of corresponding noun phrases.
In Fig. 1, "the amount", "the crowning fall" and "the end" are obtained as noun phrases in MT output by chunking, and "it", "the end part", "the amount" and "crowning drop" are obtained in the reference by chunking. Next, the system determines the corresponding noun phrases from these noun phrases between the MT output and reference. The score between "the end" and "the end part" is the highest among the scores between "the end" in the MT output and "it", "the end part", "the amount", and "crowning drop" in the reference. Moreover, the score between "the end part" and "the end" is the highest among the scores between "the end part" in reference and "the amount", "the crowning fall", "the end" in the MT output. Consequently, "the end" and "the end part" are selected as noun phrases with the highest mutual scores: "the end" and "the end part" are determined as one corresponding noun phrase. In Fig. 1, "the amount" in the MT output and "the amount" in reference, and "the crowning fall" in the MT output and "crowning drop" in the reference also are determined as the respective corresponding noun phrases. The noun phrase for which the score between it and other noun phrases is 0.0 (e.g., "it" in reference) has no corresponding noun phrase. The use of the noun phrases is effective because the frequency of the noun phrases is higher than those of other phrases. The verb phrases are not used for this study, but they can also be generated by chunking. It is difficult to determine the corresponding verb phrases correctly because the words in each verb phrase are often fewer than the noun phrases.

Word-level Score
The system calculates the word-level scores between MT output and reference using the corresponding noun phrases. First, the system determines the common words based on Longest Common Subsequence (LCS). The system selects only one LCS route when several LCS routes exist. In such cases, the system calculates the Route Score (RS) using the following Eqs. (1) and (2): words in corresponding 2 noun phrase words in non 1 corresponding noun phrase (2) In Eq. (1), β is a parameter for length weighting of common parts; it is greater than 1.0. Figure 2 portrays an example of determination of the common parts. In the first process of Fig. 2, LCS is 7. In this example, several LCS routes exist. The system selects the LCS route which has ",", "the amount of", "crowning", "is", and "." as the common parts. The common part is the part for which the common words appear continuously. In contrast, IMPACT selects a different LCS route that includes ", the", "amount of", "crowning", "is", and "." as the common parts. In IMPACT, using no analytical knowledge, the LCS route is determined using the information of the number of words in the common parts and the position of the common parts. The RS for LCS route selected using our method is 32 (= 1 2.0 + (2 + 2 + 1) 2.0 + 2 2.0 + 1 2.0 + 1 2.0 ) when β is 2.0. The RS for LCS route selected by IMPACT is 19 (= (1 + 1) 2.0 + (2 + 1) 2.0 + 2 2.0 + 1 2.0 + 1 2.0 ). In the LCS route selected by IMPACT, the weight of "the" in the common part ", the" is 1 because "the" in the reference is not included in the corresponding noun phrase. In the LCS route selected using our method, the weight of "the" in "the amount of" is 2 because "the" in MT output and "the" in the reference are included in the corresponding noun phrase "NP1". Therefore, the system based on our method can select the correct LCS route. Moreover, the word-level score is calculated using the common parts in the selected LCS route as the following Eqs. (3), (4), and (5). (1) First process for determination of common parts : LCS = 7 (2) Second process for determination of common parts : LCS=3 Equation (3) represents recall and Eq. (4) represents precision. Therein, m signifies the word number of the reference in Eq. (3), and n stands for the word number of the MT output in Eq. (4). Here, RN denotes the repetition number of the determination process of the LCS route, and i, which has initial value 0, is the counter for RN . In Eqs. (3) and (4), α is a parameter for the repetition process of the determination of LCS route, and is less than 1.0. Therefore, R wd and P wd becomes small as the appearance order of the common parts between MT output and reference is different. Moreover, length(c) represents the number of words in each common part; β is a parameter related to the length weight of common parts, as in Eq. (1). In this case, the weight of each common word in the common part is 1. The system calculates score wd as the wordlevel score in Eq. (5). In Eq. (5), γ is determined as P wd /R wd . The score wd is between 0.0 and 1.0.
The system calculates score wd multi using R wd multi and P wd multi which are, respectively, maximum R wd and P wd when multiple references are used as the following Eqs. (6), (7) and (8). In Eq. (8), γ is determined as P wd multi /R wd multi . The score wd multi is between 0.0 and 1.0.

Phrase-level Score
The system calculates the phrase-level score using the noun phrases obtained by chunking. First, the system extracts only noun phrases from sentences. Then it generalizes each noun phrase as each word. Figure 3 presents examples of generalization by noun phrases.  Figure 3 presents three corresponding noun phrases between the MT output and the reference. The noun phrase "it", which has no corresponding noun phrase, is expressed as "NP" in the reference. Consequently, the MT output is generalized as "NP1 NP2 NP3"; the reference is generalized as "NP NP3 NP1 NP2". Subsequently, the system obtains the phraselevel score between the generalized MT output and reference as the following Eqs. (9), (10), and (11).
In Eqs. (9) and (10), cnpp denotes the common noun phrase parts; m cnp and n cnp respectively signify the quantities of common noun phrases in the reference and MT output. Moreover, m no cnp and n no cnp are the quantities of noun phrases except the common noun phrases in the reference and MT output. The values of m no cnp and n no cnp are processed as 1 when no non-corresponding noun phrases exist. The square root used for m no cnp and n no cnp is to decrease the weight of the noncorresponding noun phrases. In Eq. (11), γ is determined as P np /R np . In Fig. 3, R np and P np are 0.7071 (= 1×2 2.0 +0.5×1 2.0 (3×1) 2.0 ) when α is 0.5 and β is 2.0. Therefore, score np is 0.7071.
The system obtains score np multi calculating the average of score np when multiple references are used as the following Eq. (12).

Final Score
The system calculates the final score by combining the word-level score and the phraselevel score as shown in the following Eq. (13).
Therein, δ represents a parameter for the weight of score np : it is between 0.0 and 1.0. The ratio of score wd to score np is 1:1 when δ is 1.0. Moreover, score wd multi and score np multi are used for Eq. (13) in multiple references. In Figs. 2 and 3, the final score between the MT output and the reference is 0.4185 (= 0.2164+0.7×0.7071

1+0.7
) when δ is 0.7. The system can realize high-quality automatic evaluation using both word-level information and phraselevel information.

Experimental Procedure
We calculated the correlation between the scores obtained using our method and scores produced by human judgment. The system based on our method obtained the evaluation scores for 1,200 English output sentences related to the patent sentences. These English output sentences are sentences that 12 machine translation systems in NTCIR-7 translated from 100 Japanese sentences. Moreover, the number of references to each English sentence in 100 English sentences is four. These references were obtained from four bilingual humans. Table 1 presents types of the 12 machine translation systems.
Moreover, three human judges evaluated 1,200 English output sentences from the perspective of adequacy and fluency on a scale of 1-5. We used the median value in the evaluation results of three human judges as the final scores of 1-5. We calculated Pearson's correlation efficient and Spearman's rank correlation efficient between the scores obtained using our method and the scores by human judgments in terms of sentence-level adequacy and fluency.
Additionally, we calculated the correlations between the scores using seven other methods and the scores by human judgments to compare our method with other automatic evaluation methods. The other seven methods were IMPACT, ROUGE-L, BLEU 1 , NIST, NMG-WN (Ehara, 2007;Echizen-ya et al., 2009), METEOR 2 , and WER (Leusch et al., 2003). Using our method, 0.1 was used as the value of the parameter α in Eqs. (3)-(10) and 1.1 was used as the value of the parameter β in Eqs.
(1)-(10). Moreover, 0.3 was used as the value of the parameter δ in Eq. (13). These val-  (Utiyama and Isahara, 2003). Moreover, we obtained the noun phrases using a shallow parser (Sha and Pereira, 2003) as the chunking tool. We revised some erroneous results that were obtained using the chunking tool.

Experimental Results
As described in this paper, we performed comparison experiments using our method and seven other methods. Tables 2 and 3 respectively show Pearson's correlation coefficient for sentence-level adequacy and fluency. Tables 4 and 5 respectively show Spearman's rank correlation coefficient for sentence-level adequacy and fluency. In Tables 2-5, bold typeface signifies the maximum correlation coefficients among eight automatic evaluation methods. Underlining in our method signifies that the differences between correlation coefficients obtained using our method and IMPACT are statistically significant at the 5% significance level. Moreover, "Avg." signifies the average of the correlation coefficients obtained by 12 machine translation systems in respective automatic evaluation methods, and "All" are the correlation coefficients using the scores of 1,200 output sentences obtained using the 12 machine translation systems.

Discussion
In Tables 2-5, the "Avg." score of our method is shown to be higher than those of other methods. Especially in terms of the sentence-level adequacy shown in Tables 2 and 4, "Avg." of our method is about 0.03 higher than that of IMPACT. Moreover, in system No. 8 and "All" of Tables 2 and 4, the differences between correlation coefficients obtained using our method and IMPACT are statistically significant at the 5% significance level. Moreover, we investigated the correlation of machine translation systems of every type. Table 6 shows "All" of Pearson's correlation coefficient and Spearman's rank correlation coefficient in SMT (i.e., system Nos. 1-2, system Nos. 4-8 and system Nos. 10-11) and RBMT (i.e., system Nos. 3 and 12). The scores of 900 output sentences obtained by 9 machine  Table 6 because EBMT is only system No. 9. In Table 6, our method obtained the highest correlation among the eight methods, except in terms of the adequacy of RBMT in Pearson's correlation coefficient. The differences between correlation coefficients obtained using our method and IMPACT are statistically significant at the 5% significance level for adequacy of SMT.
To confirm the effectiveness of noun-phrase chunking, we performed the experiment using a system combining BLEU with our method. In this case, BLEU scores were used as score wd in Eq. (13). This experimental result is shown as "BLEU with our method" in Tables 2-5. In the results of "BLEU with our method" in Tables 2-5, underlining signifies that the differences between correlation coefficients obtained using BLEU with our method and BLEU alone are statistically significant at the 5% significance level. The coefficients of correlation for BLEU with our method are higher than those of BLEU in any machine translation system, "Avg." and "All" in Tables 2-5. Moreover, for sentence-level adequacy, BLEU with our method is significantly better than BLEU in almost all machine translation systems and "All" in Tables 2 and 4. These results indicate that our method using noun-phrase chunking is effective for some methods and that it is statistically significant in each machine translation system, not only "All", which has large sentences.
Subsequently, we investigated the precision of the determination process of the corresponding noun phrases described in section 2.1: in the results of system No. 1, we calculated the precision as the ratio of the number of the correct corresponding noun phrases for the number of all noun-phrase correspondences obtained using the system based on our method. Results show that the precision was 93.4%, demonstrating that our method can determine the corresponding noun phrases correctly.
Moreover, we investigated the relation be- tween the correlation obtained by our method and the quality of chunking. In "Our method" shown in Tables 2-5, noun phrases for which some erroneous results obtained using the chunking tool were revised. "Our method II" of Tables 2-5 used noun phrases that were given as results obtained using the chunking tool. Underlining in "Our method II" of Tables 2-5 signifies that the differences between correlation coefficients obtained using our method II and IMPACT are statistically significant at the 5% significance level. Fundamentally, in both "Avg." and "All" of Tables 2-5, the correlation coefficients of our method II without the revised noun phrases are lower than those of our method using the revised noun phrases. However, the difference between our method and our method II in "Avg." and "All" of Tables 2-5 is not large. The performance of the chunking tool has no great influence on the results of our method because score wd in Eqs.
(3), (4), and (5) do not depend strongly on the performance of the chunking tool. For example, in sentences shown in Fig. 2, all common parts are the same as the common parts of Fig. 2 when "the crowning fall" in the MT output and "crowning drop" in the reference are not determined as the noun phrases. Other common parts are determined correctly because the weight of the common part "the amount of" is higher than those of other common parts by Eqs. (1) and (2). Consequently, the determination of the common parts except "the amount of" is not difficult.
In other language sentences, we already performed the experiments using Japanese sentences from Reuters articles (Oyamada et al., 2010). Results show that the correlation coefficients of IMPACT with our method, for which IMPACT scores were used as score wd in Eq. (13), were highest among some methods. Therefore, our method might not be languagedependent. Nevertheless, experiments using various language data are necessary to elucidate this point.

Conclusion
As described herein, we proposed a new automatic evaluation method for machine transla-  tion. Our method calculates the scores for MT outputs using noun-phrase chunking. Consequently, the system obtains scores using the correctly matched words and phrase-level information based on the corresponding noun phrases. Experimental results demonstrate that our method yields the highest correlation among eight methods in terms of sentencelevel adequacy and fluency.
Future studies will improve our method, enabling it to achieve high correlation in sentence-level fluency. Future studies will also include experiments using data of various languages.