{'fraction_non_alphanumeric': 0.039503386004514675, 'fraction_numerical': 0.014338266031268289, 'mean_word_length': 4.867794947265146, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Phrase Translation Model Enhanced with Association based Features

In this paper, we propose to enhance the phrase translation model with association measures as new feature functions. These features are estimated on counts of phrase pair co-occurrence and their marginal counts. Four feature functions, namely, Dice coefficient, log-likelihood-ratio, hyper-geometric distribution and link probability are exploited and compared. Experimental results demonstrate that the performance of the phrase translation model can be improved by enhancing it with these association based feature functions. Moreover, we study the correlation between the features to predict the usefulness of a new association feature given the existing features.


Introduction
Phrase-based translation is one of the dominant current approaches to statistical machine translation (SMT). A phrase translation model, incorporated in a data structure known as a phrase table, is the most important component of a phrasebased SMT system, since the translations are generated by concatenating target phrases stored in the phrase table. The pairs of source and corresponding target phrases are extracted from the word-aligned bilingual training corpus (Och and Ney, 2003;Koehn et al., 2003). These phrase pairs together with useful feature functions, are called collectively a phrase translation model or phrase table. A phrase translation model embedded in a state-of-the-art phrase-based SMT system normally exploits the feature functions involving conditional translation probabilities and lexical weights (Koehn et al., 2003). The phrase-based conditional translation probabilities are estimated from the relative frequencies of the source and target phrase in a given phrase pair. To avoid over-training, lexical weights are used to validate the quality of a phrase translation pair. They check how well the words in the phrase pair translate to each other. Such a phrase translation model has demonstrated its success in phrasebased SMT. Zhao et al. (2004) improved the phrase table by using models employing tf.idf term weights to estimate the similarity between the sourcelanguage and target-language in a phrase pair; this measure will, for instance, assign higher similarity to a phrase pair if both of them contain a very rare word (in their respective languages). The similarity measure is used as an additional feature. The translation probability can be also discriminatively trained as in (Tillmann and Zhang, 2006;Liang et al., 2006). Deng et al. (2008) improved the algorithm of phrase extraction. To balance precision and recall, an information retrieval process implemented with a loglinear model is used to formulate the phrase pair extraction problem.
The quality of a phrase-pair can also be checked by other measures, such as measures of source and target phrase co-occurrence; we will call these association scores.  used phrase pair association based significance testing to discard most of the phrase pairs in a phrase table without loss of performance. However, if one discards phrase pairs from a phrase table, one must heuristically choose a threshold on the value of the association as a criterion for discarding. Chen et al. (2005) used word-based association scores as rescoring features to re-rank the nbest lists of translations. The improvement reported in (Chen et al., 2005) confirmed that association based features are helpful for improving the translation performance. However, this work used the features for re-ranking rather than for decoding. Furthermore, the association features were based on words rather than on phrases, even though the latter is the basic translation unit. Smadja et al. (1996) used association scores to produce translations for the collocations in one language from parallel corpora. This also proved that association scores are a good measure for phrase pairs in parallel corpora.
In this paper, we propose to enhance the phrase translation model with phrase pair cooccurrence based association scores as new feature functions. Association features are estimated from the counts of how often a given sourcelanguage phrase is in a training corpus sentence that is aligned with a sentence containing a given target-language phrase, and also on the overall counts of the source phrase and the target phrase in this corpus. Therefore, they can provide complementary information to the existing link-based estimates, such as phrase translation probabilities and lexical weights. Also, compared with discriminatively-learned scores as introduced in (Tillmann and Zhang, 2006;Liang et al., 2006), association features are easy to calculate.
We systematically explore the subset of phrase features that can be derived from standard association measures. We first exploit and compare four widely used feature functions, namely, Dice coefficient, log-likelihood-ratio, hyper-geometric distribution and link probability. We demonstrate that the performance of the phrase translation model can be improved by incorporating such feature functions. Moreover, we study the correlation between features to predict the usefulness of a new feature given the existing features, and thus introduce the technique of using correlation as a possible cheap way to search for useful new features.
This paper is organized as follows. Section 2 reviews the basic idea of a standard phrase-based SMT system. Section 3 proposes the idea of enhancing the translation model with new association-based features. Section 4 describes the experimental setting and results on NIST Chinese-to-English translation tasks. Section 5 discusses and analyzes the results obtained. And Section 6 concludes the paper.

SMT Baseline
Phrase-based statistical machine translation systems are usually modeled through a log-linear framework (Och and Ney, 2002) by introducing the hidden word alignment variable ܽ (Brown et al., 1993).  (Och, 2003).
The SMT system we applied in our experiments is Portage, a state-of-the-art phrase-based translation system (Ueffing et al., 2007). The models and feature functions which are employed by the decoder are: • Phrase translation model(s) estimated on the word alignments generated using IBM model 2 and HMMs (Vogel et al., 1996), with standard phrase conditional translation probabilities and lexical weights being employed; • Distortion model, which assigns a penalty based on the number of source words which are skipped when generating a new target phrase; • Language model(s) trained using SRILM toolkit (Stolcke, 2002) with modified Kneser-Ney smoothing method (Chen et al, 1996); • Word/phrase penalties.

Traditional phrase-table features
A typical phrase translation model exploits features estimating phrase conditional translation probabilities and lexical weights. The phrase conditional translation probabilities are estimated from relative frequencies. Normally, twodirections, i.e., direct and inverse phrase translation probabilities are exploited. The direct phrase-based translation probability is computed as follows: where ݈݅݊݇൫݂ ሚ , ݁൯ is the frequency of source phrase ݂ ሚ and target phrase ݁̃ linked to each other.
Most of the longer phrases are seen only once in the training corpus. Therefore, relative frequencies overestimate the phrase translation probabilities. To overcome the overestimation problem, lexical weights are introduced.
There are various approaches for estimating the lexical weights. The first approach is intro-duced in (Koehn et al., 2003), Given a phrase pair ൫݂ ሚ , ݁൯ and a word alignment ܽ between the source word position ݅ ൌ 1, … , ݊ and the target word position ݆ ൌ 0,1, … , ݉, the lexical weight is computed by where ‫‬ሺ݂ |݁ ሻ is the lexical translation probability, which is computed from the relative frequencies in the same word-aligned training corpus.
The second approach calculates lexical weights based on a statistical lexicon for the constituent words in the phrase (Vogel et al., 2003).
where ‫‬ሺ݂ |݁ ሻ is word probability estimated using IBM alignment model 1, without considering the position alignment probabilities.
The third approach assumes that all source words are conditionally independent, so that: To compute ‫‬ሺ݂ |݁ሻ, Zens and Ney (2004) describe a "noisy-or" combination: where ‫‬ሺ݂ |݁ሻ is the probability that ݂ is not in the translation of ݁̃, and ‫‬ሺ݂ |݁ ሻ is a lexical probability. Zens and Ney (2004) obtain the lexical probability from smoothed relative-frequency estimates from the same word aligned training corpus. Foster et al. (2006) uses IBM1 probabilities to perform further smoothing.

Additional phrase-table association features
On top of these traditional feature functions which are based on the phrase-pair link count, we propose to exploit additional feature functions for the phrase table based on phrase-pair co-occurrence information. All of these consider a co-occurrence of a source phrase and a target phrase to have occurred if they are found in sentences that have been aligned with each other (no alignment between the words in each is necessary). We study four commonly used association-based features, namely, Dice coefficient, log-likelihood-ratio, hyper-geometric distribution, and link probability in this paper. Let ‫ܥ‬ሺ݂ ሚ , ݁ሻ denotes the co-occurrence count 1 of source phrase ݂ ሚ and target phrase ݁̃ in the parallel training corpus. It is worth noting that phrase ݂ ሚ and phrase ݁̃ are not necessarily linked to each other. ‫ܥ‬ሺ݂ ሚ ሻ and ‫ܥ‬ሺ݁ሻ denote the count of ݂ ሚ in the source sentences and the count of ݁̃ in the target sentences, respectively. ݂ ሚ and ݁̃ mean that corresponding phrases do not occur in the respective sentences. N represents the total number sentence pairs in the parallel training corpus. For each phrase, only one occurrence per sentence is taken into account. The statistics can be organized in a contingency table, e.g. in Table  1. When collecting the statistics of the data, we only need to count ‫ܥ‬ሺ݂ ሚ , ݁ሻ, ‫ܥ‬ሺ݂ ሚ ሻ , ‫ܥ‬ሺ݁ሻ and N; the other counts could be easily calculated accordingly. Then, we may compute the following association features: 1) Dice coefficient (Dice, 1945) as in Equation (7). It compares the co-occurrence count of phrase pair ݂ ሚ and ݁̃ with the sum of the independent occurrence counts of ݂ ሚ and ݁.
2) Log-likelihood-ratio (Dunning, 1993) as in Equation (8)  . 3) Hyper-geometric distribution is the probability of the phrase-pair globally cooccurring ‫ܥ‬ሺ݂ ሚ , ݁ሻ times among N sentencepairs by chance, given the marginal frequencies ‫ܥ‬ሺ݂ ሚ ሻ and ‫ܥ‬ሺ݁ሻ. We computed this prob-1 There are at least three ways to count the number of co-occurrence of ݂ ሚ and ݁, if one or both of them have more than one occurrence in a given sentence pair (Melamed, 1998). We choose to count the cooccurrence as 1 if they both occur; otherwise 0. ability assuming a binomial distribution. Without simplification, this probability can be expressed by: However, because this probability is very small, we follow  in computing the negative of the natural logs of the p-value associated with the hypergeometric distribution as the feature functions: Therefore, the higher the value of ‫ܪ‬ ሺ݂ ሚ , ݁ሻ, the stronger the association between phrase ݂ ሚ and phrase ݁.

4)
Link probability (Moore, 2005) is the conditional probability of the phrase pair being linked given that they co-occur in a given sentence pair. It is estimated as: where ݈݅݊݇ሺ݂ ሚ , ݁ሻ is the number of times that ݂ ሚ and ݁̃ are linked in sentence pairs.

Phrase-table smoothing
As (Foster et al., 2006) shows, the phrase table can be improved by applying smoothing techniques. A motivation for this is our observation that the phrase pairs which co-occur only once: ‫ܥ‬൫݂ ሚ , ݁൯ ൌ 1 are amazingly frequent in the phrase table even when the training corpus is very large. To compensate for this over-confidence in rare events, we apply absolute discounting.
The optimal values of ݀ ଵ and ݀ ଶ are determined by heuristic tuning on the dev set.

Experimental setting
Since the values of association scores are strongly related to the corpus size, i.e., the total number of events, we carried out the experiments in two data conditions. The first one is the large data condition which is distributed for NIST 2 2008 evaluation Chinese-to-English track. In particular, all the allowed bilingual corpora except the UN corpus have been used for estimating the phrase table. The second one is the small data condition where only the FBIS 3 corpus is used to train the phrase-table. However, we used the same two language models for both of these conditions. The first one is a 5-gram LM which is estimated on the target side of the texts used in the large data condition. The second LM is a 5gram LM trained on the so-called English Gigaword corpus, one of the allowed monolingual resources for the NIST task.
We carried out experiments for translating Chinese to English. We use the same development and test sets for the two data conditions. We first created a development set which used mainly data from the NIST 2005 test set, and also some balanced-genre web-text from the NIST training material. Evaluation was performed on the NIST 2004, 2006 and 2008 test sets. Table 2 gives figures about training, development and test corpora; |S| is the number of the sentences, and |W| is the size of running words. In our baseline system, we applied smoothing technique "RF+ZN-IBM1" proposed by Foster et al. (2006) to estimate the phrase translation probabilities and lexical weights. This smoothing technique achieved the best performance in our experiments which was consistent with the results reported in (Foster et al., 2006

Results
Our evaluation metric is IBM BLEU (Papineni et al., 2002), which performs case-insensitive matching of n-grams up to n = 4.
First, the results without performing count smoothing on association features are reported in Table 3. The row LD shows the results of large data track, and SD for small data track. ∆ is the average improvement of BLEU-score over the baseline for three test sets. Note that three association features, namely, Dice coefficient ("dc"), log-likelihood-ratio ("llr"), and hyper-geometric distribution ("hg") improved the performance on all three test sets for both data tracks. However, link probability ("lp") failed to give noticeable and consistent improvement. Log-likelihoodratio showed the biggest improvement (0.44 BLEU-point on average for three test sets on the large data condition and 0.52 BLEU-point on small data condition), the runner-up was Dice coefficient (about 0.32 BLEU-point on large data and 0.37 on small data  Table 3: Results (BLEU% score) of systems without counts smoothing. "dc" is Dice coefficient; "llr" is log-likelihood-ratio; "hg" is hypergeometric distribution; and "lp" is link probability. LD is large data track, and SD is small data track. ∆ is the average improvement of BLEUscore over the baseline for three test sets. For count smoothing, first we need to set the values of ݀ ଵ and ݀ ଶ for each association feature.
By observing translation performance obtained from the additional experiments on the development set, we heuristically determined the optimal values of ݀ ଵ and ݀ ଶ for Dice coefficient and hyper-geometric distribution are 0.9 and 0.8 respectively. The optimal values of ݀ ଵ and ݀ ଶ for loglikelihood-ratio are 0.1 for both, on both data conditions. The results with count smoothing are reported in Table 4. The first three features still obtained improvement compared to the baseline. If one compares the results to those without count smoothing, on large data condition, the improvements are very small for log-likelihoodratio and Dice coefficient; the improvements are more noticeable for hyper-geometric distribution, such as the 0.32 improvement in BLEU-score (from 31.06 to 31.38) that has been observed on NIST06 test set. On the small data track, the improvements produced by the counts smoothing are more similar; for Dice coefficient and hypergeometric distribution, the further improvements are about 0.2 BLEU-score, and 0.1 BLEU-score for log-likelihood-ratio. In total, the improvements over the baseline for the first three features are quite similar. The range is from 0.38 (dc) to 0.47 (llr) BLEU-score for large data, and from 0.56 (hg) to 0.64 (llr) for small data.
We then study the combination of association features. Because the link probability has not shown good improvement, we only experimented with combinations of the first three features. Table 5 illuminates the results for combinations of three features. Combinations of two features further improved the performance. "+dc+hg", and "+llr+hg",and "+dc+llr" are all improvements over using a single association feature. However, a combination of three features failed to obtain further improvement.  Table 5: Results (BLEU% score) of systems for the combinations of association features with counts smoothing.
If we take "+dc+hg" as the optimal system, the improvements over the baseline for large data condition are 0.58 BLEU-score for NIST04, 0.64 BLEU-score for NIST06, and 0.68 BLEU-score for NIST08; for small data condition, if we take "+llr+hg" as the optimal system, the improvements over the baseline are 0.68 BLEU-score for NIST04, 0.72 BLEU-score for NIST06, and 0.76 BLEU-score for NIST08.

Discussion
Association features can improve the performance because they provide additional information about how well phrase pairs translate to each other, beyond that yielded by phrase-based conditional probabilities and lexical weights. Figure  1 shows how this additional information may be applied, using some typical examples which are selected from the test sets.
Consider the first example. The only difference between the baseline output and the improved output is rains and heavy rains, they are translations of the Chinese phrase 豪 雨 (pronounces "háo yŭ" and meaning "torrential rains"). According to the reference, heavy rains should be the better translation. However, the phrase translation conditional probabilities and lexical weights of these two phrase pairs are am-1 reference baseline +dc+hg torrential rains hit western india , 43 people dead western india rains flooded 43 people were killed western india 's heavy rains flooded 43 people were killed 2 reference baseline +dc+hg the emergency government 's authority expires today . an emergency government 's mandate expires today . an emergency government 's authority expires today . 3 reference baseline +dc+hg " who are you looking for ? " " who you find ? " who are you looking for ? 4 reference baseline +dc+hg this system will also be gradually rolled out nationwide . this system will gradually open the country . this system will also gradually spread throughout the country . 5 reference baseline +dc+hg so , any mistake will lead to serious consequences . well , any error that will lead to serious consequences . well , any mistake will lead to serious consequences .  Table 7: Correlation coefficients between features for large data condition, where p i (i=1,2,3,4) are traditional phrase-table features: direct phrase-based translation probability, direct lexical weights, inverse phrase-based translation probability, inverse lexical weights. "lp" is link probability.  biguous. Table 6 gives values of the two association features; the target phrase heavy rains has higher value than the phrase rains on both Dice coefficient and hyper-geometric distribution features. This could help the decoder choose heavy rains as the final translation for the source phrase.
豪 雨 Dice ‫ܪ‬ rains 0.1666 11.67 heavy rains 0.1995 22.93 Table 6: Dice coefficient and hyper-geometric distribution scores of target phrases rains and heavy rains given source phrase 豪 雨.
Nonetheless, more association features cannot guarantee further improvement. The reason is that some association features are highly correlated with each other, since all of them are based on the co-occurrence and marginal counts. We study this issue via Pearson product-moment correlation coefficient (Rodgers and Nicewander, 1988) which measures the strength of a linear relationship between two features. We use the Equation (15) to compute the correlation coefficient.
Suppose we have a series of n values of two features H 1 and H 2 written as ‫ݔ‬ and ‫ݕ‬ (where i = 1, 2, …, n), then the correlation coefficient can be used to estimate the correlation of H 1 and H 2 . The correlation coefficient is written as: where ‫ݔ‬ҧ and ‫ݕ‬ ത are the sample means of H 1 and H 2 , ‫ݏ‬ ௫ and ‫ݏ‬ ௬ are the sample standard deviations of H 1 and H 2 . The correlation coefficient ranges from -1 to 1. The closer the coefficient is to either -1 or 1, the stronger the correlation between the features.
We randomly select 1,000 phrase pairs from the phrase table, and then compute the pair-wise correlation coefficients for the following features: a. Four traditional phrase-table futures; b. Four association features introduced in section 3.2. Table 7 and 8 list the correlation coefficients between all the features, where ‫‬ ሺ݅ ൌ 1,2,3,4ሻ are traditional phrase-table features: direct phrase-based translation probability, direct lexical weights, inverse phrase-based translation probability, and inverse lexical weights. Please note that a logarithm operation is applied to all the variables before they are input to compute the correlation coefficients.
Not surprisingly, link probability is highly correlated with the phrase-based direct and inverse probability, since all these three scores are conditioned on the link number ݈݅݊݇ሺ݂ ሚ , ݁ሻ. Moreover, we can also notice that link probability cannot guarantee an improvement for translation performance. The reason for this is that the information contained in the link probability has already been provided by either direct phrase-based translation probability or inverse phrase-based translation probability.
On the other hand, we can see that no original phrase-table features are highly correlated with Dice coefficient, log-likelihood-ratio, or hypergeometric distribution (the correlation coefficient of greatest magnitude is 0.3729 which is the correlation between log-likelihood-ratio and inverse lexical weights p 4 ). Thus, these three association features can provide additional information about the phrase-pair to the decoder, improving the translation performance. However, these three features are correlated with each other; in particular, the log-likelihood-ratio is highly correlated with the hyper-geometric distribution. From the experiments involving feature link probability, we see that it is difficult to obtain further improvement if the new feature is highly correlated with existing features. This could explain why exploiting three features together does not yield an improvement over using only two of them.

Conclusions
In this paper, we proposed a way of using phrase pair association measures as new feature functions to enhance the translation model. These features are estimated from counts of phrase pair co-occurrences and the counts of the phrases themselves. We exploited and compared four well known and widely used feature functions: the Dice coefficient, log-likelihood-ratio, hyper-geometric distribution, and link probability. Experiments were carried out on NIST tasks on large and small data conditions. The results on the two tasks both demonstrated that the translation model will perform better if the association features are integrated into a standard phrase table. Moreover, we studied the correlation between the features to predict the usefulness of a new feature given the existing features. We learned that it is difficult to obtain further improvement when applying a new feature which is highly correlated with existing features, thus for the technique of using correlation as a possible cheap way to search for useful new features.