{'fraction_non_alphanumeric': 0.029905975128905065, 'fraction_numerical': 0.018926296633303002, 'mean_word_length': 4.545240497813656, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 2, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Clustering Comparable Corpora For Bilingual Lexicon Extraction

We study in this paper the problem of enhancing the comparability of bilingual corpora in order to improve the quality of bilingual lexicons extracted from comparable corpora. We introduce a clustering-based approach for enhancing corpus comparability which exploits the homogeneity feature of the corpus, and finally preserves most of the vocabulary of the original corpus. Our experiments illustrate the well-foundedness of this method and show that the bilingual lexicons obtained from the homogeneous corpus are of better quality than the lexicons obtained with previous approaches.


Introduction
Bilingual lexicons are an important resource in multilingual natural language processing tasks such as statistical machine translation (Och and Ney, 2003) and cross-language information retrieval (Ballesteros and Croft, 1997). Because it is expensive to manually build bilingual lexicons adapted to different domains, researchers have tried to automatically extract bilingual lexicons from various corpora. Compared with parallel corpora, it is much easier to build high-volume comparable corpora, i.e. corpora consisting of documents in different languages covering overlapping information. Several studies have focused on the extraction of bilingual lexicons from comparable corpora (Fung and McKeown, 1997;Fung and Yee, 1998;Rapp, 1999;Déjean et al., 2002;Gaussier et al., 2004;Robitaille et al., 2006;Morin et al., 2007;Garera et al., 2009;Yu and Tsujii, 2009;Shezaf and Rappoport, 2010). The basic assumption behind most studies on lexicon extraction from comparable corpora is a distributional hypothesis, stating that words which are translation of each other are likely to appear in similar context across languages. On top of this hypothesis, researchers have investigated the use of better representations for word contexts, as well as the use of different methods for matching words across languages. These approaches seem to have reached a plateau in terms of performance. More recently, and departing from such traditional approaches, we have proposed in (Li and Gaussier, 2010) an approach based on improving the comparability of the corpus under consideration, prior to extracting bilingual lexicons. This approach is interesting since there is no point in trying to extract lexicons from a corpus with a low degree of comparability, as the probability of finding translations of any given word is low in such cases. We follow here the same general idea and aim, in a first step, at improving the comparability of a given corpus while preserving most of its vocabulary. However, unlike the previous work, we show here that it is possible to guarantee a certain degree of homogeneity for the improved corpus, and that this homogeneity translates into a significant improvement of both the quality of the resulting corpora and the bilingual lexicons extracted.

Enhancing Comparable Corpora: A Clustering Approach
We first introduce in this section the comparability measure proposed in former work, prior to describing the clustering-based algorithm to improve the quality of a given comparable corpus. For convenience, the following discussion will be made in the context of the English-French comparable corpus.

The Comparability Measure
In order to measure the degree of comparability of bilingual corpora, we make use of the measure M developed in (Li and Gaussier, 2010): Given a comparable corpus P consisting of an English part P e and a French part P f , the degree of comparability of P is defined as the expectation of finding the translation of any given source/target word in the target/source corpus vocabulary. Let σ be a function indicating whether a translation from the translation set T w of the word w is found in the vocabulary P v of a corpus P, i.e.: and let D be a bilingual dictionary with D v e denoting its English vocabulary and D v f its French vocabulary. The comparability measure M can be written as: where # w (P) denotes the number of different words present in P. One can find from equation 1 that M directly measures the proportion of source/target words translated in the target/source vocabulary of P.

Clustering Documents for High Quality Comparable Corpora
If a corpus covers a limited set of topics, it is more likely to contain consistent information on the words used (Morin et al., 2007), leading to improved bilingual lexicons extracted with existing algorithms relying on the distributional hypothesis. The term homogeneity directly refers to this fact, and we will say, in an informal manner, that a corpus is homogeneous if it covers a limited set of topics. The rationale for the algorithm we introduce here to enhance corpus comparability is precisely based on the concept of homogeneity. In order to find document sets which are similar with each other (i.e. homogeneous), it is natural to resort to clustering techniques. Furthermore, since we need homogeneous corpora for bilingual lexicon extraction, it will be convenient to rely on techniques which allows one to easily prune less relevant clusters. To perform all this, we use in this work a standard hierarchical agglomerative clustering method.

Bilingual Clustering Algorithm
The overall process retained to build high quality, homogeneous comparable corpora relies on the following steps: 1. Using the bilingual similarity measure defined in Section 2.2.2, cluster English and French documents so as to get bilingual dendrograms from the original corpus P by grouping documents with related content; 2. Pick high quality sub-clusters by thresholding the obtained dendrograms according to the node depth, which retains nodes far from the roots of the clustering trees; 3. Combine all these sub-clusters to form a new comparable corpus P H , which thus contains homogeneous, high-quality subparts; 4. Use again steps (1), (2) and (3) to enrich the remaining subpart of P (denoted as P L , P L = P \ P H ) with external resources.
The first three steps aim at extracting the most comparable and homogeneous subpart of P. Once this has been done, one needs to resort to new corpora if one wants to build an homogeneous corpus with a high degree of comparability from P L . To do so, we simply perform, in step (4), the clustering and thresholding process defined in (1), (2) and (3) on two comparable corpora: The first one consists of the English part of P L and the French part of an external corpus P T ; The second one consists of the French part of P L and the English part of P T . The two high quality subparts obtained from these two new comparable corpora in step (4) are then combined with P H to constitute the final comparable corpus of higher quality.

Similarity Measure
Let us assume that we have two document sets (i.e. clusters) C 1 and C 2 . In the task of bilingual lexicon extraction, two document sets are similar to each other and should be clustered if the combination of the two can complement the content of each single set, which relates to the notion of homogeneity. In other words, both the English part C e 1 of C 1 and the French part C f 1 of C 1 should be comparable to their counterparts (respectively the same for the French part C f 2 of C 2 and the English part C e 2 of C 2 ). This leads to the following similarity measure for C 1 and C 2 : where β (0 ≤ β ≤ 1) is a weight controlling the importance of the two subparts (C e 1 , C f 2 ) and (C e 2 , C f 1 ). Intuitively, the larger one, containing more information, of the two comparable corpora (C e 1 , C f 2 ) and (C e 2 , C f 1 ) should dominate the overall similarity sim(C 1 , C 2 ). Since the content relatedness in the comparable corpus is basically reflected by the relations between all the possible bilingual document pairs, we use here the number of document pairs to represent the scale of the comparable corpus. The weight β can thus be defined as the proportion of possible document pairs in the current comparable corpus (C e 1 , C f 2 ) to all the possible document pairs, which is: where # d (C) stands for the number of documents in C. However, this measure does not integrate the relative length of the French and English parts, which actually impacts the performance of bilingual lexicon extraction. If a 1-to-1 constraint is too strong (i.e. assuming that all clusters should contain the same number of English and French documents), having completely unbalanced corpora is also not desirable. We thus introduce a penalty function φ aiming at penalizing unbalanced corpora: The above penalty function leads us to a new similarity measure sim l which is the one finally used in the above algorithm:

Experiments and Results
The experiments we have designed in this paper aim at assessing (a) whether the clustering-based algorithm we have introduced yields corpora of higher quality in terms of comparability scores, and (b) whether the bilingual lexicons extracted from such corpora are of higher quality. Several corpora were used in our experiments: the TREC 1 Associated Press corpus (AP, English) and the corpora used in the CLEF 2 campaign including the Los Angeles Times (LAT94, English), the Glasgow Herald (GH95, English), Le Monde (MON94, French), SDA French 94 (SDA94, French) and SDA French 95 (SDA95, French). In addition, two monolingual corpora Wiki-En and Wiki-Fr were built by respectively retrieving all the articles below the category Society and Société from the Wikipedia dump files 3 . The bilingual dictionary used in the experiments is constructed from an online dictionary. It consists of 33k distinct English words and 28k distinct French words, constituting 76k translation pairs. In our experiments, we use the method described in this paper, as well as the one in (Li and Gaussier, 2010) which is the only alternative method to enhance corpus comparability.

Improving Corpus Quality
In this subsection, the clustering algorithm described in Section 2.2.1 is employed to improve the quality of the comparable corpus. The corpora GH95 and SDA95 are used as the original corpus P 0 (56k English documents and 42k French documents). We consider two external corpora: P 1 T (109k English documents and 87k French documents) consisting of the corpora LAT94, MON94 and SDA94; P 2 T (368k English documents and 378k French documents) consisting of Wiki-En and Wiki-Fr. After the clustering process, we obtain the resulting corpora P 1 (with the external corpus P 1 T ) and P 2 (with P 2 T ). As mentioned before, we also used the method described in (Li and Gaussier, 2010) on the same data, producing resulting corpora P 1 (with P 1 T ) and P 2 (with P 2 T ) from P 0 . In terms of lexical coverage, P 1 (resp. P 2 ) covers 97.9% (resp. 99.0%) of the vocabulary of P 0 . Hence, most of the vocabulary of the original corpus has been preserved. The comparability score of P 1 reaches 0.924 and that of P 2 is 0.939. Both corpora are more comparable than P 0 of which the comparability is 0.881. Furthermore, both P 1 and P 2 are more comparable than P 1 (comparability 0.912) and P 2 (comparability 0.915), which shows homogeneity is crucial for comparability. The intrinsic evaluation shows the efficiency of our approach which can improve the quality of the given corpus while preserving most of its vocabulary.

Bilingual Lexicon Extraction Experiments
To extract bilingual lexicons from comparable corpora, we directly use here the method proposed by Fung and Yee (1998) which has been referred to as the standard approach in more recent studies (Déjean et al., 2002;Gaussier et al., 2004;Yu and Tsujii, 2009). In this approach, each word w is represented as a context vector consisting of the words co-occurring with w in a certain window in the corpus. The context vectors in different languages are then bridged with an existing bilingual dictionary. Finally, a similarity score is given to any word pair based on the cosine of their respective context vectors.

Experiment Settings
In order to measure the performance of the lexicons extracted, we follow the common practice by dividing the bilingual dictionary into 2 parts: 10% of the English words (3,338 words) together with their translations are randomly chosen and used as the evaluation set, the remaining words being used to compute the similarity of context vectors. English words not present in P e or with no translation in P f are excluded from the evaluation set. For each English word in the evaluation set, all the French words in P f are then ranked according to their similarity with the English word. Precision and recall are then computed on the first N translation candidate lists. The precision amounts in this case to the proportion of lists containing the correct translation (in case of multiple translations, a list is deemed to contain the correct translation as soon as one of the possible translations is present). The recall is the proportion of correct translations found in the lists to all the translations in the corpus. This evaluation procedure has been used in previous studies and is now standard.

Results and Analysis
In a first series of experiments, bilingual lexicons were extracted from the corpora obtained by our approach (P 1 and P 2 ), the corpora obtained by the approach described in (Li and Gaussier, 2010) (P 1 and P 2 ) and the original corpus P 0 , with the fixed N value set to 20. Table 1 displays the results obtained. Each of the last two columns "P 1 > P 0 " and "P 2 > P 0 " contains the absolute and the relative difference (in %) w.r.t. P 0 . As one can note, the best results (in bold) are obtained from the corpora P 2 built with the method we have described in this paper. The lexicons extracted from the enhanced corpora are of much higher quality than the ones obtained from the original corpus . For instance, the increase of the precision is 6.9% (30.5% relatively) in P 1 and 23.5% (104.0% relatively) in P 2 , compared with P 0 . The difference is more remarkable with P 2 , which is obtained from a large external corpus P 2 T . Intuitively, one can expect to find, in larger corpora, more documents related to a given corpus, an intuition which seems to be confirmed by our results. One can also notice, by comparing P 2 and P 2 as well as P 1 and P 1 , a remarkable improvement when considering our approach and the early methodology.
Intuitively, the value N plays an important role in the above experiments. In a second series of experiments, we let N vary from 1 to 300 and plot the results obtained with different evaluation measure in Figure 1. In Figure 1(a) (resp. Figure 1(b)), the xaxis corresponds to the values taken by N, and the yaxis to the precision (resp. recall) scores for the lexicons extracted on each of the 5 corpora P 0 , P 1 , P 2 , P 1 and P 2 . A clear fact from the figure is that both the precision and the recall scores increase according to the increase of the N values, which coincides with our intuition. As one can note, our method consistently outperforms the previous work and also the original corpus on all the values considered for N .  Figure 1: Performance of bilingual lexicon extraction from different corpora with varied N values from 1 to 300. The five lines from the top down in each subfigure are corresponding to the results for P 2 , P 2 , P 1 , P 1 and P 0 respectively.

Discussion
As previous studies on bilingual lexicon extraction from comparable corpora radically differ on resources used and technical choices, it is very difficult to compare them in a unified framework (Laroche and Langlais, 2010). We compare in this section our method with some ones in the same vein (i.e. enhancing bilingual corpora prior to extracting bilingual lexicons from them). Some works like (Munteanu et al., 2004) and (Munteanu and Marcu, 2006) propose methods to extract parallel fragments from comparable corpora. However, their approach only focuses on a very small part of the original corpus, whereas our work aims at preserving most of the vocabulary of the original corpus. We have followed here the general approach in (Li and Gaussier, 2010) which consists in enhancing the quality of a comparable corpus prior to extracting information from it. However, despite this latter work, we have shown here a method which ensures homogeneity of the obtained corpus, and which finally leads to comparable corpora of higher quality. In turn such corpora yield better bilingual lexicons extracted.