{'fraction_non_alphanumeric': 0.02614864864864865, 'fraction_numerical': 0.007702702702702703, 'mean_word_length': 4.843268851164627, 'pattern_counts': {'":': 1, '<': 1, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Multilingual Lexical Database Generation from Parallel Texts in 20 European Languages with Endogenous Resources

This paper deals with multilingual database generation from parallel corpora. The idea is to contribute to the enrichment of lexical databases for languages with few linguistic resources. Our approach is endogenous: it relies on the raw texts only, it does not require external linguistic resources such as stemmers or taggers. The system produces alignments for the 20 European languages of the 'Acquis Communautaire' Corpus.


Automatic processing of bilingual and multilingual corpora
Processing bilingual and multilingual corpora constitutes a major area of investigation in natural language processing. The linguistic and translational information that is available make them a valuable resource for translators, lexicographers as well as terminologists. They constitute the nucleus of example-based machine translation and translation memory systems. Another field of interest is the constitution of multilingual lexical databases such as the project planned by the European Commission's Joint Research Centre (JRC) or the more established Papillon project. Multilingual lexical databases are databases for structured lexical data which can be used either by humans (e.g. to define their own dictionaries) or by natural language processing (NLP) applications.
Parallel corpora are freely available for research purposes and their increasing size demands the exploration of automatic methods.
The 'Acquis Communautaire' (AC) Corpus is such a corpus. Many research teams are involved in the JRC project for the enrichment of a multilingual lexical database. The aim of the project is to reach an automatic extraction of lexical tuples from the AC Corpus.
The AC document collection was constituted when ten new countries joined the European Union in 2004. They had to translate an existing collection of about ten thousand legal documents covering a large variety of subject areas. The 'Acquis Communautaire' Corpus exists as a parallel text in 20 languages. The JRC has collected large parts of this document collection, has converted it to XML, and provide sentence alignments for most language pairs (Steinberger et al., 2006).

Alignment approaches
Alignment becomes an important issue for research on bilingual and multilingual corpora. Existing alignment methods define a continuum going from purely statistical methods to linguistic ones. A major point of divergence is the granularity of the proposed alignments (entire texts, paragraphs, sentences, clauses, words) which often depends on the application.
A common problem is the delimitation and spotting of the units to be matched. This is not a real problem for methods aiming at alignments at a high level of granularity (paragraphs, sentences) where unit delimiters are clear. It becomes more difficult for lower levels of granularity (Simard, 2003), where correspondences between graphically delimited words are not always satisfactory.

The multi-grained endogenous alignment approach
The approach proposed here deals with the spotting of multi-grained translation equivalents. We do not adopt very rigid constraints concerning the size of linguistic units involved, in order to account for the flexibility of language and translation divergences. Alignment links can then be established at various levels, from sentences to words and obeying no other constraints than the maximum size of candidate alignment sequences and their minimum frequency of occurrence. The approach is endogenous since the input is used as the only used linguistic resource. It is the multilingual parallel AC corpus itself. It does not contain any syntactical annotation, and the texts have not been lemmatised. In this approach, no classical linguistic resources are required. The input texts have been segmented and aligned at sentence level by the JRC. Inflectional divergencies of isolated words are taken into account without external linguistic information (lexicon) and without linguistic parsers (stemmer or tagger). The morphology is learnt automatically using an endogenous parsing module integrated in the alignment tool based on (Déjean, 1998).
We adopt a minimalist approach, in the line of GREYC. In the JRC project, many languages do not have available linguistic resources for automatic processing, neither inflectional or syntactical annotation, nor surface syntactic analysis or lexical resources (machine-readable dictionaries etc.). Therefore we can not use a large amount of a priori knowledge on these languages.
The current version of our alignment system deals with one language pair at a time, whatever the languages are. The algorithm takes as input a corpus of bitexts aligned at sentence level. Usually, the alignment at this level outputs aligned windows containing from 0 to 2 segments. Oneto-one mapping corresponds to a standard output (see link types "1-1" above). An empty window corresponds to a case of addition in the source language or to a case of omission in the target language. One-to-two mapping corresponds to split sentences (see link types "1-2" and "2-1" above).
Formally, each bitext is a quadruple < T1, T2, Fs, C> where T1 and T2 are the two texts, Fs is the function that reduces T1 to an element set Fs(T1) and also reduces T2 to an element set Fs(T2), and C is a subset of the Cartesian product of Fs(T1) x Fs(T2) (Harris, 1988).
Different standards define the encoding of parallel text alignments. Our system natively handles TMX and XCES format, with UTF-8 or UTF-16 encoding.

The Resolution Method
The resolution method is composed of two stages, based on two underlying hypotheses. The first stage handles the document grain. The second stage handles the corpus grain.

Hypotheses
hypothesis 1 : let's consider a bitext composed of the texts T 1 and T 2 . If a sequence S 1 is repeated several times in T 1 and in well-defined sentences 1 , there are many chances that a repeated sequence S 2 corresponding to the translation of S 1 occurs in the corresponding aligned sentences in T 2 . hypothesis 2 : let's consider a corpus of bitexts, composed of two languages L 1 and L 2 . There is no guarantee for a sequence S 1 which is repeated in many texts of language L 1 to have a unique translation in the corresponding texts of language L 2 .

Stage 1 : Bitext analysis
The first stage handles the document scale. Thus it is applied on each document, individually. There is no interaction at the corpus level. Determining the multi-grained sequences to be aligned First, we consider the two languages of the document independently, the source language L 1 and the target language L 2 . For each language, we compute the repeated sequences as well as their frequency.
The algorithm based on suffix arrays does not retain the sub-sequences of a repeated sequence if they are as frequent as the sequence itself. For instance, if "subjects" appears with the same frequency than "healthy subjects" we retain only the second sequence. On the contrary, if "disease" occurs more frequently than "thyroid disease" we retain both.
When computing the frequency of a repeated sequence, the offset of each occurrence is memorized. So the output of this processing stage is a list of sequences with their frequency and the offset list in the document.

Handling inflections
Inflectional divergencies of isolated words are taken into account without external linguistic information (lexicon) and without linguistic parsers (stemmer or tagger). The morphology is learnt automatically using an endogenous approach derived from (Déjean, 1998). The algorithm is reversible: it allows to compute prefixes the same way, with reversed word list as input.
The basic idea is to approximate the border between the nucleus and the suffixes. The border matches the position where the number of distinct letters preceding a suffix of length n is greater than the number of distinct letters preceding a suffix of length n-1.
For instance, in the first English document of our corpus, "g" is preceded by 4 distinct letters, "ng" by 2 and "ing" by 10: "ing" is probably a suffix. In the first Greek document, "ά" is preceded by 5 letters, "κά" by 1 and "ικά" by 10. "ικά" is probably a suffix.
The algorithm can generate some wrong morphemes, from a strictly linguistic point of view. But at this stage, no filtering is done in order to check their validity. We let the alignment algorithm do the job with the help of contextual information. Vectorial representation of the sequences An orthonormal space is then considered in order to explore the existence of possible translation relations between the sequences, and in order to define translation couples. The existence of translation relations between sequences is approximated by the cosine of vectors associated to them, in this space.
The links in the alignment file allow the construction of this orthonormal space. This space has n o dimensions, where n o is the number of non-empty links. Alignment links with empty sets (type="0-?" or type="?-0") corresponds to cases of omission or addition in one language.
Every repeated sequence is seen as a vector in this space. For the construction of this vector, we first pick up the segment offset in the document for each repeated sequence.
"thyroid cancer": list of segments where the sequence appears 45,46,46,48,51,51 Then we convert this list in a n L -dimension vector v L , where n L is the number of textual segments of the document of language L. Each dimension contains the number of occurrences present in the segment.

Sequence alignment
For each sequence of L 1 to be aligned, we look for the existence of a translation relation between it and every L 2 sequence to be aligned. The existence of a translation relation between two sequences is approximated by the cosine of the vectors associated to them. The cosine is a mathematical tool used in in Natural Language Processing for various purposes, e.g. (Roy & Beust, 2004) uses the cosine for thematic categorisation of texts. The cosine is obtained by dividing the scalar product of two vectors with the product of their norms.
We note that the cosine is never negative as vectors coordinates are always positive. The sequences proposed for the alignment are those that obtain the largest cosine. We do not propose an alignment if the best cosine is inferior to a certain threshold.

Stage 2 : Corpus management
The second stage handles the corpus grain and merges the information found at document grain, in the first stage.

Handling the Corpus Dimension
The bitext corpus is not a bag of aligned sentences and is not considered as if it were. It is a bag of bitexts, each bitext containing a bag of aligned sentences.
Considering the bitext level (or document grain) is useful for several reasons. First, for operational sake. The greedy algorithm for repeated sequence extraction has a cubic complexity. It is better to apply it on the document unit rather than on the corpus unit. But this is not the main reason.
Second, the alignment algorithm between sequences relies on the principle of translation coherence: a repeated sequence in L1 has many chances to be translated by the same sequence in L2 in the same text. This hypothesis holds inside the document but not in the corpus: a polysemic term can be translated in different ways according to the document genre or domain.
Third, the confidence in the generated alignments is improved if the results obtained by the execution of the process on several documents share compatible alignments.

Alignment Filtering and Ranking
The filtering process accepts terms which have been produced (1) by the execution on at least two documents, (2) by the execution on solely one document if the aligned terms correspond to the same character string or if the frequency of the terms is greater than an empirical threshold function. This threshold is proportional to the inverse term length since there are fewer complex repeated terms than simple terms.
The ranking process sorts candidates using the product of the term frequency by the number of output agreements.

Results
The results concern an alignment task between English and the 19 other languages of the AC-Corpus. For each language pair, we considered 500 bitexts of the AC Corpus. We join in annexes A, B, and C some sample of this results. Annex A deals with English-French parallel texts, Annex B deals with English-Spanish parallel texts and finally Annex C deals with English-German ones. We discuss in the following lines of the English-French alignment.
Among the correct alignments, we find domain dependant lexical terms:

Discussion
First, the results are similar to those obtained on the Greek/English scientific corpus.
Second, it is sometimes difficult to choose between distinct proposals for a same term when the grain vary: Member/membre~ Member State~/membre~ Member States/États membres State/membre State~/membre~. There is a problem both in the definition of terms and in the ability of an automatic process to choose between the components of the terms.
Third, thematic terms of the corpus are not always aligned, since they are not repeated. Corefence is used instead, thanks to nominal anaphora, acronyms, and also lexical reductions. Accuracy depends on the document domain. In the medical domain, acronyms are aligned but not their expansion. However, we consider that this problem has to be solved by an anaphora resolution system, not by this alignment algorithm.

Conclusion
We showed that it is possible to contribute to the processing of languages for which few linguistic resources are available. We propose a solution to the spotting of multi-grained translation from parallel corpora. The results are surprisingly good and encourage us to improve the method, in order to reach a semi-automatic construction of a multilingual lexical database.
The endogenous approach allows to handle inflectional variations. We also show the importance of using the proper knowledge at the proper level (sentence grain, document grain and corpus grain). An improvement would be to calculate inflectional variations at corpus grain rather than at document grain. Therefore, it is possible to plug any external and exogenous component in our architecture to improve the overall quality.
The size of this "massive compilation" (we work with a 20 languages corpora) implies the design of specific strategies in order to handle it properly and quite efficiently. Special efforts have been done in order to manage the AC Corpus from our document management platform, WIMS.
The next improvement is to precisely evaluate the system. Another perspective is to integrate an endogenous coreference solver (Giguet & Lucas, 2004).