{'fraction_non_alphanumeric': 0.030686038178382957, 'fraction_numerical': 0.007166590657371815, 'mean_word_length': 4.558392178164041, 'pattern_counts': {'":': 1, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Data selection for compact adapted SMT models

Data selection is a common technique for adapting statistical translation models for a specific domain, which has been shown to both improve translation quality and to reduce model size. Selection relies on some in-domain data, of the same domain of the texts expected to be translated. Selecting the sentence-pairs that are most similar to the in-domain data from a pool of parallel texts has been shown to be effective; yet, this approach holds the risk of resulting in a limited coverage, when necessary n-grams that do appear in the pool are less similar to in-domain data that is available in advance. Some methods select additional data based on the actual text that needs to be translated. While useful, this is not always a practical scenario. In this work we describe an extensive exploration of data selection techniques over Arabic to French datasets, and propose methods to address both similarity and coverage considerations while maintaining a limited model size.


Introduction
Data selection (DS) is a key method for domain adaptation. It is commonly used in statistical machine translation (SMT) for selecting from a pool of parallel sentences a subset that is more similar to the target domain, and using it to train the translation model, the language model, or both. Data selection can also be used when more compact models are needed due to memory limitations, for instance. A wide variety of selection methods have been used over the years, where the main principle is to measure the similarity of sentences from the pool to some indomain data, either the development or the (source side of the) test set. Such similarity is often based on information theory metrics, like perplexity, applied to either side of the training data (source or target) or -as often turned out to be more effective -to both. One problem these methods suffer from is that they "overfit" the in-domain data, since sentences with unseen ngrams are less likely to be selected. Some methods address this coverage issue by expanding the selected training data to include more instances of previously unseen or infrequent n-grams. Techniques based on information retrieval (IR) have also been widely used for data selection, especially for choosing training or tuning data based on the test set, either from parallel corpora or by mining comparable corpora to find relevant parallel phrases or sentences. This produces highly adapted models for the test set, and generally reduces the number of out-of-vocabulary (OOV) words. The limitation of such methods is that they typically rely on the availability of the text for translation before the final model is produced. While sometimes possible, this is not always the case.
This work proposes an in-depth investigation of several data selection techniques for SMT adaptation. We experiment with less conventional and challenging domains for Arabic to French translation (translation of Web blogs and of dialectal conversation transcripts), and with very little available in-domain data. Our work provides insights into the usefulness of the different selection methods to each dataset.
In addition, we propose two data selection methods that aim to ease the tension between the similarity and coverage objectives, while the need for compact models that require rather aggressive data filtering, is taken into account. In the first proposed method, an IR model is added to a perplexity-based one, resulting in a model that is ready to use with or without relying on the test set. The second, denoted AVSF, is a domain-adaptation-fitted version of VSF (Lewis and Eetemadi, 2013), an algorithm which was designed to reduce model size without jeopardizing n-gram coverage. Both proposed methods are able to show competitive results in comparison to prior DS techniques. Over one dataset, their performance is better than other assessed methods; over the other -where data selection with size constraints proved more difficult -they outperform methods with comparable training data size, demonstrating a good tradeoff between translation performance and model size.
The rest of the paper is organized as follows. Section 2 presents common approaches for data-selection in SMT. Section 3 details the setting of our experiments. Experiments based on information theory metrics are presented in Section 4. Our proposed methods are described in Sections 5 (IR-based adaptation) and 6 (adapted VSF). Section 7 shows a summary of the results and outlines our main conclusions.

Data selection for domain adaptation
Data selection is a way to adapt the about-to-be-trained model by using only the part of the training data that is more similar to the target domain. DS is a general approach, that has been applied to other tasks other than SMT, including Chinese word segmentation and Part-of-Speech tagging (Song et al., 2012). In SMT, DS is common practice for domain adaptation, where a subset of the bilingual parallel corpus is used for training some or all the models comprising typical phrase-based SMT models (translation, reordering and language models). Apart from better adaptation, data selection has the advantage of making the training set, and in turnthe generated models, smaller. This is a factor we consider in this work, avoiding methods or parameters with which the training data is not significantly smaller than the entire set.
In this section we review the most common techniques used for selecting training data in SMT. The assumption is that we have a small in-domain corpus, denoted I, and a large "general" bilingual corpus, G, a pool from which we wish to select bi-sentences that will help better translate texts of the same domain as I.

Information theory metrics
One prominent line of research is using information theory metrics to assess each one of the sentences in the pool, G, and choose the ones that are most similar to the provided in-domain data, I. Perplexity (PP) and cross-entropy (CE) Perplexity (PP) is perhaps the best known metric for DS for SMT. The idea is to compute the perplexity of a language model (LM) built on I, measure the perplexity of this LM over each sentence in G, and select the m sentences with the lowest PP scores. Gao et al. (2002) and Moore and Lewis (2010) employed this metric for language model adaptation, and in (Foster et al., 2010;Yasuda et al., 2008) it was used for translation model adaptation. The same technique is sometimes (e.g. Moore and Lewis (2010)) referred to as cross entropy (CE). Since, by definition, the cross-entropy is simply the exponent in the perplexity score, the base of the exponent is 2 and all scores are positive, a smaller PP means a smaller CE and vice versa. In other words, selecting m sentences with the lowest PP scores or with the lowest CE scores is equivalent.
Cross-entropy difference (CED) While PP (or its equivalent CE) were used extensively before, Moore and Lewis (2010) proposed using the difference between the cross entropies scores with respect to the in-domain corpus and with respect to the general corpus. The idea is to prefer sentences that are typical to the in-domain (i.e. low CE) and untypical to the general domain (i.e. high CE). As above, the sentences with the lowest cross entropy difference are selected. Axelrod et al. (2011) have extended over the CED method by computing it bilingually, taking the average of the CED over the source and over the target. They showed that bilingual CED outperforms the monolingual version, as well as other related metrics. In their experiments, best results were achieved when the in-domain and the reduced-general corpora were placed in two different translation tables. This method is still considered the state of the art in this line of research.
In our experiments we assess all the above metrics, both monolingually and bilingually, over two different datasets. We validate that bilingual data selection works best for translation models, but show that target-based selection sometimes outperforms it for language model training. Further, quite intuitively, we learn that when the general corpus is rather similar to the in-domain one, perplexity outperforms cross entropy difference. In such cases, data selection serves mainly for the purpose of reducing model size rather than performance improvement. See Section 4 for further details about these results.

Retrieval-based selection
A different approach for data selection is based on Information Retrieval (IR). Typically, the data is selected based on the source side of the test set. This makes this approach appropriate only for certain scenarios, when the text to translate is known in advance, and when the translation can be delayed until a model is constructed for it. Eck et al. (2004) used this approach to adapt the LM of an SMT model; Hildebrand et al. (2005) adapted both the translation model (TM) and the LM by using the Lemur IR system 1 for searching the training set for sentences that are similar to each of the test set sentences, and training the model over the retrieved dataset. Perplexity was used in their work to determine the size of the selection. They evaluated the effect of removing from the retrieved set duplicate sentences, caused by sentence being retrieved by multiple queries, and found it was not leading to significant performance changes. Lu et al. (2007) also used Lemur to select sentences from the bilingual corpus based on the test set. Giving more weight to duplicate sentences resulted with somewhat better results in their experiments. In our experiments, we remove duplicate sentences in order to keep the model smaller. Their results, with the top-1000 sentences for each query, improved somewhat over the baseline that used the entire bilingual corpus, while reducing the phrase-table size by almost 30%. IR was also used for related domain adaptation tasks. For instance, Chen et al. (2012) used the test set to generate an adapted tuning set, and in (Afli et al., 2012(Afli et al., , 2013 retrieval is used to expand the training data from comparable, rather than parallel, corpora. While methods based on information-theory metrics aim to find sentences that are similar to the development set (with no real guarantee that it will be similar to the test set), IR-based approaches can achieve tighter adaptation by specifically considering the test set. Yet, as mentioned, it is not always plausible to assume that the text to translate is available in advance. When adaptation for the test set is not possible, we risk encountering at translation time unseen or infrequent n-grams that do occur in the parallel corpus but were not selected. Gascó et al. (2012) addressed the issue of lacking coverage by expanding the training data with more sentences that contain its infrequent source n-grams, and showed how to reduce the selec-tion to the test set n-grams. Lewis and Eetemadi (2013) suggested a related approach, termed Vocabulary Saturation Filter (VSF), that filters the training data to contain only a certain, limited number of each source and target n-gram. Keeping track of the n-gram occurrences, they pass over the training data and add each sentence pair to the selected set only if the pair contains n-grams that occur less than the predefined threshold. This method ensures that every possible n-gram in the pool will be represented in the selected training data. It does not, however, make use of any in-domain data as its motivation is to reduce the model size, rather than domain adaptation. In Section 6 we show a simple way to adapt this method for this task.

Experimental Setting
Our experiments were conducted within the setting of the TRAD project, 2 on the Arabic-French language pair. Below we describe the datasets, corpora and the setting of our experiments.
Evaluation datasets Two different datasets were evaluated, and for each a development set was provided, consisting of approximately 10, 000 source tokens (of untokenized text). This is the only in-domain data we had at our disposal. The datasets are described below, and their size details are given in Table 1, in terms of source and target tokens, as well as in terms of the number of (bi-)sentences. These datasets are different by nature; this is reflected, for example, by the length of the sentences, evident from the table.
• EGYP 3 : Transcription of conversations in Egyptian dialectal Arabic.   Evaluation metric We evaluate our translations with BLEU (Papineni et al., 2002) using an official script of the evaluation campaign that removes punctuations and lowercases the detokenized output.
Corpora The list of bilingual corpora we used is given below. Table 3 shows the size of each corpora. As our pool of bi-sentences we use a concatenation of these corpora according to the listed order. This corpus, of 14.5M bi-sentences, was also used to produce baseline models in our experiments.  Table 3: Bilingual corpora for training.
SMT system and preprocessing We used Moses (Koehn et al., 2007) as our phrase-based SMT system. IRSTLM (Federico et al., 2008) was used to train 5-gram language models over the target side of the (selected) bilingual corpora. Arabic tokenization was done similarly to MADA-TOKAN (Habash et al., 2009), but due to project constraints we used a reimplementation of this tokenizer. More precisely, 300M Arabic words of the MULTI-UN corpus, segmented using MADA, were used to train a tokenizer using OpenNLP. 7 To improve tokenization of punctuations, we applied the Moses tokenizer after the Arabic segmentation was applied. The translation models are trained on lowercased tokenized text and we apply a standard detokenizer prior to evaluation.
IR system For indexing and retrieval, we used Lucene, 8 with its default settings. We indexed all unigrams and bigrams of the preprocessed Arabic side of the bilingual concatenated corpus, and used the inverted index for retrieval for both datasets. The preprocessing of the indexed corpus is identical to that applied to any other source data, such as the development or test sets.

Perplexity-based adaptation
First, we assess domain adaptation techniques based on information theory metrics. The concatenated corpora is our pool, G, and each small development set, D, is used both for tuning and for representing the in-domain data (I). Ideally, these tasks would use different sets, but we had no additional in-domain data available, and the data we did have was too small to split.

Perplexity (PP)
We train a LM using the development set, D, and compute its perplexity for each sentence in G.
We refrain from building an incremental language model over G and measuring its perplexity over D, but rather use the more intuitive parameter m to determine in advance how many bisentences we wish to use for generating the model. We experimented with selection using perplexity over the source (denoted pp-src), over the target (pp-tgt), and over both (pp-bi), where the sum of perplexities of a bi-sentence is used. Once scored, the m bi-sentences with the lowest perplexity scores are selected. Table 4 shows an example of these experiments, when using the BLOGS dataset and selecting from the MULTI-UN corpus to generate the SMT model. As seen in the table, DS based on both source and target proves most useful, and selection based on the source alone yields particularly inferior results. This outcome was consistent over additional values of m, over different pools of bilingual data and over the EGYP dataset as well.

Cross-entropy difference (CED)
We assessed the methods suggested by Moore and Lewis (2010) and Axelrod et al. (2011) for monolingual or bilingual CED. As above, we used the development set, D, as our in-domain data; for the general data, we used a random sample of G of the same size as D. Following the PP results, we assessed selection using CED based on target alone and based on both source and target, but not over the source alone. We further assessed the option to select data for the TM bilingually but for the LM monolingually, using only the target side of the corpus. A subset of our results over the EGYP dataset, using perplexity and CED, is shown in  From Table 5 we learn that for the EGYP dataset: (i) adapting the LM based on the target only is better than bilingual adaptation, and (ii) CED outperforms PP over any selection size. In contrast, as shown in Table 6, for BLOGS, PP is the preferred choice. Another differ-ence between the datasets is that BLOGS requires a larger amount of selected data to reach the performance of the simple baseline. This property of the datasets surfaced all along our experiments, and indicate that the BLOGS dataset is closer to the general corpus than EGYP. Different selection strategies had to be followed.  For subsequent experiments, we chose the 2M selection size which obtained good results in our experiments while remaining reasonable in terms of model size. 9 With respect to the selection metric, we use the best-performing one for each dataset: ced (ced-bi for TM and ced-tgt for LM) for EGYP, and pp-bi for BLOGS.

IR over an adapted model
Data-selection methods based on information retrieval were described in Section 2. The principle is to generate a tightly adapted model for the (source side of the) test set by obtaining parallel corpora that covers its n-grams. We implement this approach, following prior work, with some modifications. Yet, realizing that in real-world scenarios, using the test set is not always feasible, our goal is to provide a model that can also support real-time response. In this section we describe a method that achieves this goal and the experiments using it over the two datasets. The idea is to enable working in both immediate and delayed modes. To that end, we create a model that is IR-ready, but still IR-independent. That is, a model that would be relatively easy and fast to update when we receive a text to translate and can afford a short delay, but that can also perform well when an immediate translation is required.
To support IR-based adaptation we create an inverted index of the entire preprocessed source side of the bilingual corpus. All n-grams are indexed, up to a maximal predefined length (2, in our experiments). Then, we train an adapted model based on CED or PP, using some in-domain data as described in Section 4. Since this model will not be used as our final one, and we are only after its tables (phrase table, language model and reordering table), it does not need to be tuned. Next, we let another part of the in-domain data play the role of the test set. We extract n-grams from its source side, optionally ordering them (see below). We keep track of the number of occurrences of each n-gram in the retrieved set that is being collected, R. Searching up to k instances of each n-gram, we add to R the source sentences retrieved by searching exact matches of the n-gram, and their corresponding target sentences, and update the counts of each n-gram that appears in them. When the next n-gram is up for search, we deduct its current count in R from the maximum number of requested hits. The motivation is to obtain enough instances of each n-gram while keeping the retrieved dataset small for space and speed considerations. Note that this does not constrain the absolute number of occurrences of each n-gram, and in general, more frequent n-grams will tend be more frequent in R. With the motivation of keeping the model compact, and since the results of retrieval may overlap when separate queries lead to identical retrieved sentences, we perform a bilingual de-duplication. That is, we remove all identical bi-sentences from the selected parallel corpus, allowing duplicates to remain in either source or target, but not both. The de-duplicated set, R , is then used to construct additional translation and language models. The two types of adaptation are then combined in a single log linear setting, where each makes up one TM or LM. That is, we add an additional TM or LM model based on the IR data rather than train models with the entire selected data. Such separation was shown to be helpful by, e.g., Axelrod et al. (2011), and enables quickly generating the models (Mirkin and Cancedda, 2013). The updated configuration, with 2 TMs and 2 LMs, is then tuned using the development set D, producing a ready-to-use model. If test-set adaptation is possible, we apply the same IR selection over the text for translation, and use the TM and the LM generated with it to substitute those created with the previouslyavailable in-domain data. If we can afford, time-wise, to re-tune the model, that would be the preferred choice. Yet, tuning is a lengthy process, and if the models are of similar properties, tuning may be skipped, as shown in (Mirkin and Cancedda, 2013).
The above steps are summarized in Algorithm 1. As mentioned, only a small in-domain dataset was provided for us in these experiments. Due to this constraint, the same dataset was used for multiple roles: as a development (tuning) set, and as the seed dataset, guiding the PP and IR adaptations. If more in-domain data is available, these tasks should be performed using different sets, in order to avoid overfitting. In Algorithm 1, we refer to the in-domain data as D, regardless of its role, as was actually done in our experiments.

IR experiments
To run this method, like most methods using domain adaptation with IR, one must determine the values of a set of parameters. Below we list these parameters and describe the outcomes of exploring their usage over the development sets. Here, only the retrieved set is used to construct the SMT model, without the PP-adapted one, in order to let the different parameter values be better reflected in the results.
• k, the number of requested search hits: A higher value results with more training data, at the cost of increasing model size. We experimented with k values of 100, 500 and 1000, where k = 1000 yielded the best results.
• n, the maximal number of tokens in the queried n-gram: A search for a unigram matches all longer n-grams that contain it. However, since we limit the number of hits per query, it is not always the case in practice. Longer n-grams potentially constitute a better match to the query, and may therefore be valuable. Still, our experiments showed that using n = 1 is usually sufficient, possibly since the k value we used was able to obtain enough matches.
• The IR similarity metric: we used the default metric used by Lucene: a tf-idf weighted cosine similarity between the query and the document (a sentence, in our case).
• The order of searched n-grams: Since the retrieved set is sensitive to the order of the search, we assessed several ordering techniques: (i) "As-is": the original order of n-grams in the searched set.
(ii) Decreasing frequency: searching for the more frequent n-grams first.
(iii) Ratio of frequencies in the search set (e.g. mathcalD, representing in-domain data) and the entire pool. The idea is to give priority to words that are more prominent in the in-domain set in comparison to their "regular" prominence, much like CED. Since the number of occurrences in the entire corpus may be very high, we take its squared root, and add 1 to it, to avoid division by 0: , where w is the n-gram under consideration.
In this set of experiments, ordering based on the last option showed but a slight improvement over the default, as-is, order. Hence, we did not further reorder the n-grams prior to the search.
Another outcome of exploring the IR parameters over the development sets was that the baseline, using the entire pool, beats the IR-based selection for BLOGS (26.04 vs. 24.80 with k = 1000), while for EGYP, a very small retrieved set obtains as good results as the entire pool (10.65 vs. a baseline of 10.63).
So far we have selected the non-IR adaptation method and the IR parameters. We now turn to assess the proposed combined model. We compare using different sets in the retrieval table -the development set, the test set, or both concatenated. In the latter case, we add the alreadyobtained R D to R T since it provides more statistics for generating the TM and LM, without requiring additional on-the-fly retrieval. Processing the larger dataset indeed takes longer, but is still relatively limited in comparison to the entire size of the data used in the model. Each experiment using the test set used the tuning of the development set configuration with the same parameters, only replacing the phrase table and LMs.
The results are shown in Tables 7 and 8. First, the tables show that for both datasets, even when using only the development set (the R D parts of the tables), the proposed IR model improves over PP-or CED-adapted models of a similar size. In the case of EGYP, it improves over any other model we have managed to generate. 10 Second, unsurprisingly, using the test set improves translation performance. The improvement is more significant for BLOGS than EGYP. Checking the reduction of OOVs when using the test sets, we learned that the test set significantly reduced the number of output sentences with OOVs for BLOGS (by 36%, for   k = 1000); for EGYP the reduction was more modest (17%). As seen in our additional results, this dataset is very different from the available bilingual corpus and much of its vocabulary does not occur in the pool at all. Adding the development set to the test set was not found helpful for EGYP, but was so for BLOGS, where, generally speaking, improvement was observed with any addition of data.

Adapted Vocabulary Saturation Filter (AVSF)
The proposed IR method overcomes to some extent the "overfitting" problem of PP-based techniques. Still -when test-set adaptation is not possible -it considers the development set as its core source for adaptation, and is therefore prone to have limited coverage of the actually necessary n-grams. VSF, an algorithm suggested by Lewis and Eetemadi (2013), aims to reduce the training data by including each n-grams only a certain number of times. Thus, any n-gram, up to the determined length, that appears in the training data, also appears in its compact version. We find this algorithm suitable to compensate for drawbacks of adaptation with PP and of IR without using the test set. VSF was not designed for domain adaptation and indeed, does not make use  of any domain-specific data, such as the development set. We propose an adapted version of VSF, denoted AVSF, to be used for domain adaptation. The order of the data provided to VSF has a direct effect on the selected training data. Lewis and Eetemadi (2013) discussed this issue, and ordered the training data by alignment score. They were not addressing domain adaptation in that work, and the adaptation for our task is simple: first we sort the training data according to a perplexity-based metric and then apply VSF over the reordered corpus. The idea is that sentences more relevant for the domain will be selected first and will be less likely to be skipped due to n-gram saturation. Table 9 shows experiments we conducted using this algorithm, in comparison to its nonadapted version. VSF has 2 parameters: n, the maximal length of the n-grams we are trying to cover, and t, the minimum required frequency of each n-gram. Experiments over the development sets, with smaller corpora, showed that while increasing n is useful, t does not make much difference; we therefore set t to 1 in all our experiments. Applying VSF to our (arbitrarily ordered) bilingual corpus, with n = 2 and t = 1 results with a large amount of selected data, 6.9M bi-sentences. We consider this size as contradicting one of our goals in this work -to keep the model small -and we therefore do not train a model using this selection. Yet, for AVSF, where the order is based on domain adaptation techniques, we can use n = 2 by applying the algorithm only over the top-m sentences of the ordered training data. This results in a smaller selected set, while guaranteeing a coverage of the more relevant part of the corpus n-grams.  The results show that increasing the size of the training data, within the size limitations we imposed, is helpful. Selection based on top-m is also beneficial, and provides more flexibility with respect to the VSF parameters. AVSF therefore proves very useful, and is also efficient in terms of run time, since once the corpus has been ordered according to PP   any case for DS using such metrics, each of the top-m bi-sentences needs to be traversed once, and go only through n-gram extraction.

Summary of results and conclusions
In Table 11 we summarize the prominent results obtained for each assessed method. The results show that AVSF and the proposed IR configuration outperform the state of the art selection with CED, even with smaller model sizes. AVSF seems to be doing better in that respect, while the IR model has the advantage of being able to tightly adapt to the test set, when some delay is permitted.
In conclusion, in this work we have investigated multiple data selection methods for domain adaptation in SMT. The generated model size played an important role in our research as we tried to achieve the best possible results with models that are trained on no more than 30% of the initial bilingual corpus. Considering the different characteristics of selection techniques, we proposed two separate methods that combine existing methods in order to benefit from the advantages of each one of them. Our extension to well-known IR-based adaptation proved competitive and enables supporting two modes of operation: instant and delayed translations. Our proposed adaptation of VSF to the task at hand was demonstrated to be useful in obtaining good performance through small models. An immediate extension of these results would be to apply the IR method over the AVSF adapted model.
Assessing two distinct datasets, we learned that DS methods are not always consistent in their success over different datasets. This is naturally reflected in the parameter values, but also in the method that needs to be used. In that respect, our experiments consistently showed that when the dataset is more "similar" to the pool (as reflected, e.g. in the BLEU scores), using more data is useful and PP is preferable over CED. Our experiments revealed some hints of how to anticipate that, but further research is required to be able to predict the most effective method for the domain and the range of parameters to assess, in order to reduce the search space. Achieving that can potentially cut the adaptation effort considerably.