{'fraction_non_alphanumeric': 0.030305240509098864, 'fraction_numerical': 0.007804237628095256, 'mean_word_length': 4.9432162982228, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 2, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Portuguese-English Word Alignment: some Experiments

In this paper we describe some studies of Portuguese-English word alignment, focusing on (i) measuring the importance of the coupling between dictionaries and corpus; (ii) assessing the relevance of using syntactic information (POS and lemma) or just word forms, and (iii) taking into account the direction of translation. We first provide some motivation for the studies, as well as insist in separating type from token anlignment. We then briefly describe the resources employed: the EuroParl and COMPARA corpora, and the alignment tools, NATools, introducing some measures to evaluate the two kinds of dictionaries obtained. We then present the results of several experiments, comparing sizes, overlap, translation fertility and alignment density of the several bilingual resources built. We also describe preliminary data as far as quality of the resulting dictionaries or alignment results is concerned.


Word alignment and its evaluation
Word alignment is a relatively well-known technology for both statistical and example based machine translation (SMT and EBMT), but there are still several open research questions involved. As already pointed by others, the term "word alignment" has been used to depict two different tasks, which we will call in the present paper "(word) type alignment" and "(word) token alignment". The former has also been called "bilingual lexicon acquisition" (from parallel aligned corpora) by (Karlgren and Sahlgren, 2005). The latter, the process of establishing translation relationships between words belonging to two chunks of text has been termed "bilingual word alignment" by (Dagan et al., 1993); or just "bilingual alignment" by (Mihalcea and Pedersen, 2003). Instead of alignment -which conveys, at least originally, the idea of order preservation -, correspondence would be a more suitable designation, even if one did not separate beween type and token. For example, (Moore, 2001) uses the more adequate description of "translation relationships among words", and (Melamed, 2000) has moved to "translation equivalence among words", but unfortunately it was the phrase "word alignment" that won. In any case, we are here interested in looking in more detail into these two tasks and their evaluation: we are not arguing here that they are unrelated, but that they should be conceptually separated to be adequately evaluated. Word-type models are usually obtained from processing word-tokens, so there is often, though not always, a relevant connection between word-token co-occurrences in a bi-text and the word-type bilingual dictionary that the method gives origin to. In this paper, we want to test empirically some of the assumptions involved in (type and token) word alignment, as well as study the interdependence of the two in the case of our tools. To fix terminology, we use the term probabilistic translation dictionary (PTD) to denote the result of word type alignment, and the term word correspondences to re-fer to the specific relationships between specific words in bi-texts. The set of all word correspondences will be called a token dictionary. We will provide a brief literature review of the evaluation methods for both kinds of tasks: For type-alignment, the standard methods employ coverage (how many of the words in the corpus occur in the dictionaries) and accuracy (manual investigation of a sample of entries, compared to published dictionaries or checked in the corpus). For token alignment, (Melamed, 1998) created a golden standard using the Bible for the French English pair. However, consideration of his detailed instructions uncovered deep disagreement with our own views of word alignment. For example: to the land of honey and milk, although phrasally related to the French pronoun y in a particular passage, does not seem to us to be a good enough reason for making land-y, honey-y, and milk-y links, as Melamed does (for the sake of maximal annotation). We doubt whether so much disagreement about what a correct/usable/interesting word token aligner should do does not preclude indeed a comparison with this work. In fact, if there is such a wide disagreement as to what a token link should be, maybe it is completely vacuous to compare word alignment systems. A detailed comparison of two kinds of word aligners and the different assumptions used by each, namely the works by Moore and Melamed can be appreciated more fully in (Santos, 2007). Tasks like word spotting (Véronis and Langlais, 2000), which can be agreed upon, are probably the only ones liable to comparative evaluation. Other tasks such as support for bilingual lexicographers, and more informed translation browsers, could be also produce some indirect evaluation results, provided it were possible to quantify user satisfaction in using those systems.

Research questions
Some of the issues we are interested in are: • To assess the quality of word type alignment, what is the importance of the underlying corpus in order to build a good Probabilistic Translation Dictionary (PTD)?
• How relevant are the issues of balance, textual genre and translation direction for building a PTD?
• Likewise, if one uses a PTD based on a particular corpus to do word token alignment of a different corpus, how much degradation is it to be expected as compared to using the "proper" PTD?
• Generalizing, can we extrapolate this last question to provide a measure for comparing bilingual corpora? Most work so far has concentrated on monolingual corpus comparison (Kilgarriff, 2001).
Another relevant subject for machine translation is a measure of the word vs. phrase correspondence that a particular word alignment can produce. This can be considered an approximation to the translation difficulty of the particular language pair, and also provide a ceiling for what can be expected of aligment, or correspondence.
In this paper, we will however only deal with the following questions:

How important is text type?
It is common to read that there are huge differences in translation practice according to text type, but there is hardly any measure or even quantitative support for this claim. One might partially pin down creativity in translation using the measures "number of different translations per word" and "number of no-one-to-one-translations used". Apriori we would predict that creative text types (and creative translation practices) would rank higher in both of those measures. We will therefore propose a quantitative analogue of "translation fertility" based on PTDs and investigate how sensitive it is to translation direction, and especially text type. It is possible that we can also pinpoint which lexical or grammatical areas are more sensitive to these distinctions. From an MT developer point of view, it is highly relevant to measure which part -if any -of the translation between two languages is common to different text types, and which one is more variable.

Is translation direction relevant?
Interestingly, significant differences in machine translation depending on the translation direction have been reported, but they often remain unexplained, and in any case not explicitly quantified. For example, (Way and Gough, 2005) mention that the English-French and the French-English direction behave differently as far as their comparison of EBMT and SMT systems is concerned, while (Talbot, 2005) states that "models trained with German as the source language tend to have significantly lower AER than those with English". Also (Koehn, 2005) notes this in connection with the Eu-roParl languages: "some languages are more difficult to translate into than from", and mentions that translation into morphologically rich languages, as opposed to into English, has been comparatively neglected.

How important is syntactic analysis for word alignment?
What -if any -is the impact on word alignment of proper name recognition, multiword detection, lemmatization and POS labelling is something that we want to assess, by creating different alignment dictionaries according to these different possibilities. (Choueka et al., 2000) used lemmatized versions of their English-Hebrew parallel texts, but did not compare with non-lemmatized versions.

COMPARA
COMPARA (Frankenberg-Garcia and Santos, 2003) is a large human-edited parallel corpus, whose sentence alignment, sentence separation, lemmatization and POS tagging have been human revised (the two last so far for Portuguese only) (Santos and Inácio, 2006 (Santos and Frankenberg-Garcia, 2007).

EuroParl
We have also used EuroParl (Koehn, 2002;Koehn, 2005) aligned for the same language pair. Europarl is publically available 2 and V2 contains more than one million translation units, with around 30 million words in each language. In contradistinction to COMPARA, Europarl's sentence alignment has not been manually revised, nor is translation direction known (in fact many of the translation units may not even be directed translated between English and Portuguese, coming from other source languages). It belongs to another genre (transcription of parliamentary debates); and language varieties of both Portuguese and English are restricted to European (and even Europese) parlance.

NATools
NATools 3 is an open source workbench for parallel corpora processing developed upon the Twente-Aligner (Hiemstra, 1996), which is based on an expectation maximization (EM) algorithm. Its tool set includes a sentence aligner, a probabilistic translation dictionary (PTD) extractor (Simões and Almeida, 2003) (= a word type aligner), and a word (token) aligner.

Probabilistic translation dictionary creation (nat-create)
The algorithm employed to create the PTDs has been described in detail elsewhere (Simões, 2004). Basically, from a set of aligned sentences nat-create creates a list of entries, to each it is associated its frequency and the probability of the several translations, see example below: Let us just present the assumptions (or limitations) of the work reported in this paper: (i) sentences longer than 500 words were ignored by the algorithm; (ii) a fixed maximum of eight translations per entry was postulated; and (iii) capitalization was removed. We then used two different strategies to build PTDs for this paper: a) no filtering at all; b) only keeping as dictionary entries words occurring more than 5 times in the source language (SO>5), and maintaining only those translations that corresponded to a probability higher than 5% (TP>5%). So far, this program only creates 1-1 alignments, but uses the tokenization provided at the input. This means one can cheat the system by sending already glued multiword chunks (which cannot be discontinuous), as described below.

Token alignment (nat-chunker)
In token alignment, we use the extracted dictionaries to align terms across sentences. First, we create a matrix of translation probabilities (from the PTD) between all words in the two sentence-aligned sentences, then we apply a smoothing algorithm, followed by application of some patttern matching rules, to deal with usual word order changes across the two languages. The result that maximises the number of cells put into correspondence wins. Figure 1 shows an example of a (rather good) alignment matrix from EuroParl. discussão discussion sobre about fontes de financ. alternativas alternative sources of financing para for a the aliança radical europeia european radical alliance Table 1: Extracted aligned pairs.
In the previous example, a very clean and easy piece chosen for illustrative purposes, there is one 4:4 and one 3:3 chunk alignment. In general, there are often 2:2 or 3:1 or 10:5 alignments where one cannot go further down pinpointing which corresponds to which. This is a consequence of the linguistic differences between the languages, a ceiling to what word alignment can meaningfully accomplish. Measuring this ceiling is unfortunately outside the scope of the present paper. Now, what we really want for our purposes here is the individual alignments among words, so we built nat-mkTokenDic, which, from the output of nat-chunker, identifies the possible translation relationships between words. All instances of one-to-one word pairs will count as token-word correspondences. So, for example discussão : discussion and para : for would be gathered by this procedure, but not europeia : european. The result of this process for the whole corpus we call a token dictionary; i.e., simply the accumulation of all translations a given word had in the particular parallel text which was aligned. For practical purposes, these token dictionaries are stored in the same format of PTDs, the difference being that the numbers associated with the target language words reflect relative frequencies in a particular parallel corpus and not probability estimations of any sort. nat-chunker also produces a composite measure of its performance in a particular corpus, by computing the average size of chunks obtained (ASC), as well as providing the distribution of chunk size (i.e., how many 1-1, 1-2, 2-5 etc.). Let us in any case note that token dictionaries are currently a proper subset of PTDs, i.e., there is no mechanism to infer new word alignments, such as from torneira esverdeada partida -broken greenish tap and the entries torneiratap and partida -broken to obtain the new alignment esverdeada -greenish. This would considerably improve the (token) alignment process and would allow us to get more interesting results, but has to be left for future work.

Dictionary comparison (nat-compareDicts)
NATools also includes a program that compares dictionaries: globally, by providing the number of identical, overlapping and intersective entries in the two dictionaries, and per word, in an interactive mode, allowing human inspection of the two full entries. Due to the similar format of the two kinds of dictionaries, this program can also be applied to the comparison of a PTD and a token dictionary, or to two token dictionaries.

Describing dictionaries (nat-descDict)
We have developed a new module for NATools which computes the following characateristics for a PTD or a token dictionary.
• translation fertility (TF): the average number of translations found in a dictionary (PTD or token dictionary); • type or token alignment density (tyAD or toAD): this is the ratio of aligned tokens in a parallel text after token alignment (applies therefore only to token dictionaries), either counting different forms that got aligned (over all different forms in the target side of the corpus): tyAD; or counting the number of all forms that got aligned over all forms of the target side of the corpus

PTD overlap
We have built several PTDs from EuroParl and COM-PARA, which we proceed to describe now. Using the full corpora, the resulting (unfiltered) PTDs include about 137,000 and 71,000 entries respectively for the Portuguese language (and 87,000 and 45,000 for English).
If we consider the filtered ones, which should be more reliable estimators of the translation relationship, we get sizes of 47,000 and 16,000 entries for EuroParl and COMPARA. Numbers concerning the size of these and other PTDs are shown together in Table 7

The influence of genre
While this clearly illustrates the difference between the two domains, we want to know how much word alignment becomes degraded by using the same or different genre PTDs. We have therefore created two new corpora from each corpus: COMPARA reduced (CMPred) has 90% of COM-PARA contents (excluding void 1-0 alignment units); CMPtst includes the remaing 10% for testing purposes. Then we have also created a PTD based on an Europarl subcorpus with roughly the same size as COMPARA, named EuroParlred2. 4 Because we wanted to make the two corpora as comparable as possible, the smaller one had to be the determining one in terms of size. We got also a corresponding EuroParltst with the same size in alignment units of CMPtst. Comparing the sizes of the resulting unfiltered PTDs, it is clear that the subject diversity is higher in COMPARA than in EuroParl, as should be expected. But this also means that when it comes to the filtered ones Eu-roParl is able to preserve more entries. Then, we word-aligned the two test corpora, using the corresponding PTD and the "other corpus" PTD. Results in term of token dictionary size are in  Results show what we expected, namely that changing genre degrades alignment, and they allow us to measure how much: the size of the PTDs is reduced to 71% and 76% in COMPARA in Portuguese and in English, while for EuroParl the corresponding numbers are as low as 38% and 44%. This shows that a fiction dictionary seems to be far more unsuitable to align political discourse than the opposite (all things being equal: size of training and test materials). The differences between languages are not very significant here, although English degrades slightly less in both corpora. This is most probably due to a higher number of different forms in Portuguese (e.g. adjectives have 4 different forms vs. one in English, verbs have more than 70 forms).

Comparing content
Size may not say much about the quality or accuracy of a given PTD, so we performed a comparison of the (full) PTDs in the colour domain, reusing the extensive lists of colour words appearing in COMPARA that have been manually compiled by the COMPARA team and which are publicly available . The Portuguese list includes 420 forms denoting colour (218 lemmas), while the English list includes 491 forms (see  for more details). We have then computed, for all elements of the lists which stood as entries in the PTDs, how often the best translation was also a colour, and how often at least one of the translations was also a colour. Results are shown in Table 5, where we have used the two versions of COMPARA (forms and lemmas) to see if using lemmas would fare better for Portuguese.
Before we proceed to analyse in some detail these results, it is important to note that colour in politics or in daily life (fiction) has quite a different scope, so it would be foreseeable that the two corpora would provide a different picture altogether of the colour domains in the two languages. In fact, colour as denoting political affiliation seems to be by far the most frequent meaning in EuroParl, with green (3630 occurrences) and greens (1173)   hair are some of the top collocations for colours in COM-PARA and probably in general for fiction texts in the two languages.
In any case, what we got was unexpected, in that English results lie significantly lower than those for Portuguese. The only explanation is that the threshold of a maximum of 8 translations is much more damaging in the English to Portuguese direction (because Portuguese has more forms) than in the Portuguese to English direction, but while using lemmas should in principle solve or aliviate this problem, it does not (results for CMPlemma are even worse). Further investigation of this issue is thus in order, using the manual annotation of COMPARA (in both languages) to distinguish between colour meaning of a word or another meaning of a homograph.
In any case we proceeded to analyse the corresponding token dictionaries, whose numbers are displayed in Table 6, since the results from the PTDs, as they stand, are not encouraging regarding quality of the overall PTDs created.  The results are definitely better, which brings some hope that the word token alignment may be helpful. Note that not necessarily a colour term should be translated by a colour term. (Some colour terms have other, sometimes even more proeminent, meanings, e.g. silver, orange or rose). Still, the translation/alignment seems to work much better from Portuguese to English, no matter the genre. Let us also report a previous evaluation of EuroParl PTDs in the Portuguese to English direction, presented in (Simões, 2008). Filtering was done such as keeping only words with frequency higher than 50, and translation probability above 20%. Then 1000 pairs of (entry-word,translation) were randomly sampled and manually evaluated, yielding 85% correctness.

PTD
It is difficult to compare to the present results, given that most colour terms would not survive the filtering used.  Table 7: Sizes and translation fertilities of the several PTDs. For each, we present first the unfiltered and then the filtered one.

Translation direction
To study the import of translation direction -to be more precise, the influence of building PTDs based on texts of only one translation direction -, we created two dictionaries EtoP and PtoE, obtained by partitioning COMPARA in two parts (original English, and original Portuguese).
We have then measured, as in the EuroParl-COMPARA comparison, the partial overlap and the degradation after alignment with the corresponding part or the other, displayed in Table 8. For comparison purposes, we also used the full PTD created with COMPARA and the one with Eu-roParl to token word align these corpora.  Note that it is probable that a considerable fraction of Eu-roParl originated in the English to Portuguese direction, which would predict that using EuroParl would be more successful to align the EtoP subcorpus. But the difference was not high. On the contrary, what was remarkable in the results was that we observed a general marked degradation for the PtoE part. In fact, dictionary size halves for that corpus if we use the other direction PTD (both in Portuguese and in English), compared to a milder degradation for EtoP. This is hard to explain. Given our data so far, we have to accept, or at least not reject, that it comes from harder texts and not necessarily from any language difference. Given that there is no overlap between the English-speaking authors of COMPARA and the Portuguese-speaking ones, for all purposes the two sub-corpora are different and PtoE may be more difficult to align. We can also see from the table that more data (on which the PTD is built) increases performance, but that (in this case) the other genre cannot in most cases supplant the same. (So 60 million words of EuroParl are hardly better than 1.5 million words in COMPARA.)

Grammatical analysis
Finally, we tried to check whether linguistic processing would improve any of the two kinds of word alignment, by adding several different additional pieces of information (in the Portuguese side only) and comparing the performances. The first change (changed corpus) was recognizing multiword proper names and joining them as one token (CMPproper), the second was recognizing multiword expressions from a Portuguese point of view and joining them in one token (CMPmwe), and doing both (CMPmweprop). It is not obvious that either of these strategies would help, since no corresponding processing is so far done on the English side. (This is work that will have to be done.) On the other hand, the question of using lemmata instead of word forms may be more promising, given that Portuguese is morphologically richer than English, especially in what verbs are concerned. CMPlemma does just that, while CMPpos enriches the lemma with the additional POS.
Although it can be claimed that a parallel processing should be done to both languages for ideal results, we think it is interesting to investigate adding this information to just one side as well. 5 However, apart from presenting their sizes in Table 7, further automatic comparison between these different dicionaries is hampered by the fact that they have different units and therefore it is hard to come up with a meaningful measure. We have nevertheless measured the size of the corresponding token dictionaries, under the assumption that a larger token dictionary size means better performance, displayed in  Table 9: Size, translation fertility and alignment density of the several token dictionaries.
rally) the number of the entries (the size of the dictionary, both PTD and token dictionary), it keeps the translation fertility to the same level. In English as source, the fact that the Portuguese translation is lemmatized does not give so good results, and it even decreases TF. We note that there is a marked asymmetry in the need or interest of lemmatization or other linguistic processing depending on the source language, that should be investigated in more detail. Possibly depending on the language pair, some features should be merged and others not. Except for the linguistically analysed corpora (lemma and pos) both translation fertility and alignment density are higher for the English-Portuguese dictionaries (which also have fewer entries). However, token alignment density seems to correlate negatively with type, and we have to investigate the matter more thoroughly.

Concluding remarks
In addition to develop a set of particular tools to compare and assess bilingual dictionaries, available to the research community, we have also formulated a set of questions that we attempted to answer, as well as suggested a number of evaluation measures to be employed to characterize these objects. Starting by these latter, we defined translation fertility to characterize a kind of parallel text, or genre, or corpus, by the average number of translation candidates in the PTD.
(Strictly speaking, it is the parallel corpus that is being given that measure; but if it is big and representative enough we may talk about the kind of text or genre instead.) We used alignment density to characterize the ratio of aligned tokens in a parallel text after token alignment. While this may be in the first place a measure of the quality of the aligner, note that, assuming an ideal aligner, this measure would be higher for more creative text types than for very predictable ones, in that new translations (or uses) of terms would pop up with higher frequency. We have suggested yet another creativity measure related to the average size of word correspondences after word-token alignment (ASC), but have not computed it here because it reduces trivially to alignment density in our case, given that, currently, there is no refinement done by our token word aligner, nor any improvement of token dictionaries based on 1-to-N correspondences. It will turn out to be genuinely different and more relevant as soon as PTDs, and token dictionaries, include not just 1-to-1 translations, but N-to-M as well, as linguistically appropriate. Summing up the partial answers gathered to our research questions, very briefly: (a) we did measure the influence of training in other corpora and training with other genre, (b) we noted -once again as former researchers before us -that the translation direction is relevant; and (c) found out that use of other units of analysis (such as lemmas or lemma-pos sets) brings mixed results. In fact, it may well that be the most obvious conclusion of this study was that there was a fundamental flaw in the PTD creation design (the absolute limit of eight translations instead of a relative limit based on probability mass), and that to produce linguistically motivated tools and results this has to be changed. In any case, the result of our work, the tools and the dictionaries, are publicly available for inspection in http: //natools.sf.net/ and therefore readers are welcome to make their own investigations or measures, for example with other term lists or semantic domains.