{'fraction_non_alphanumeric': 0.02363477206922093, 'fraction_numerical': 0.011649047202208605, 'mean_word_length': 4.924212205823694, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Examining the Tip of the Iceberg: A Data Set for Idiom Translation

Neural Machine Translation (NMT) has been widely used in recent years with significant improvements for many language pairs. Although state-of-the-art NMT systems are generating progressively better translations, idiom translation remains one of the open challenges in this field. Idioms, a category of multiword expressions, are an interesting language phenomenon where the overall meaning of the expression cannot be composed from the meanings of its parts. A first important challenge is the lack of dedicated data sets for learning and evaluating idiom translation. In this paper we address this problem by creating the first large-scale data set for idiom translation. Our data set is automatically extracted from a widely used German-English translation corpus and includes, for each language direction, a targeted evaluation set where all sentences contain idioms and a regular training corpus where sentences including idioms are marked. We release this data set and use it to perform preliminary NMT experiments as the first step towards better idiom translation.


Introduction
Neural Machine Translation (NMT) (Bahdanau et al., 2015;Sutskever et al., 2014;Cho et al., 2014) has achieved substantial improvements in translation quality over traditional Rule-based and Phrase-based Translation (PBMT) in recent years. For instance, subject-verb agreement, doubleobject verbs, and overlapping subcategorization are various areas where NMT successfully overcomes the limitations of PBMT (Isabelle et al., 2017;Bentivogli et al., 2016). However, one of the remaining challenges of NMT is translating infrequent words and phrases (Koehn and Knowles, 2017;Fadaee et al., 2017) and idioms are a particular instance of this problem (Isabelle et al., 2017). Idioms are semantic lexical units whose meaning is often not simply a function of the meaning of its constituent parts (Nunberg et al., 1994;Kövecses and Szabo, 1996). The non-compositionality characteristic of idiom expressions exists in different degrees in a language (Nunberg et al., 1994). In English for example, for the idiom "spill the beans", the word 'spill' symbolizes 'reveal' and 'beans' symbolizes the 'secrets'. With the idiomatic expression "kick the bucket", on the other hand, no such analysis is possible. Isabelle et al. (2017) builds a challenge set of 108 short sentences that each focus on one particular difficult phenomenon of the language. Their manual assessment of the eight sentences consisting of an idiomatic phrase show that NMT systems struggle with the translation of these phrases. The challenge of translating idiom phrases in NMT is partly due to the underlying complexity of identifying a phrase as idiomatic and generating its correct non-literal translation, and partly to the fact that idioms are rarely encountered in the standard data sets used for training NMT systems. As an example, in Table 1 we provide an idiom expression in German and the literal and idiomatic translations in English. We observe that the literal translation of an idiom is  To illustrate the problem of idiom translation we also provide the output of three NMT systems for this sentence: GoogleNMT (Wu et al., 2016), DeepL 1 , and the OpenNMT implementation (Klein et al., 2017) based on Bahdanau et al. (2015) and Luong et al. (2015). All systems fail to generate the proper translation of the idiom expression. This problem is particularly pronounced when the source idiom is very different from its equivalent in the target language, as the case here. Although there are a number of monolingual data sets available for identifying idiom expressions (Muzny and Zettlemoyer, 2013;Markantonatou et al., 2017), there is limited work on building a parallel corpus annotated with idioms, which is necessary to investigate this problem more systematically. Salton et al. (2014) selected a small subset of 17 English idioms, collected 10 sentence examples for each idiom from the internet, and manually translated them into Brazilian-Portuguese to use for the translation task.
Building a hand-crafted data set for idiom translation is costly and time-consuming. In this paper we automatically build a new bilingual data set for idiom translation extracted from an existing general-purpose German↔English parallel corpus.
The first part of our data set consists of 1,500 parallel sentences whose German side contains an idiom, while the second consists of 1,500 parallel sentences whose English side contains an idiom. Additionally, we provide the corresponding training data sets for German→English and English→German translation where source sentences including an idiom phrase are marked. We believe that having a sizable data set for training and evaluation is the first step to improve idiom translation.

Data Collection
In this work we focus on German↔English translation of idioms. This is an established language pair and is commonly used in the machine translation community. Automatically identifying idiom phrases in a parallel corpus requires a gold standard data set annotated manually by linguists. We use the dict.cc online dictionary 2 containing idiomatic and colloquial phrases, which is built manually, as our gold standard for extracting idiom phrase pairs. Examining the WMT German↔English test sets from 2008 to 2016 (Bojar et al., 2017), we observe very few sentence pairs containing an idiomatic expression. The standard parallel corpora available for training however contain several such sentence pairs. Therefore we automatically select sentence pairs from the training corpora where the source sentence contains an idiom phrase to build the new test set. Note that we only focus on idioms on the source side and we have two separate list of idioms for German and English, hence, we independently build two test sets (for German idiom translation and English idiom translation) with different sentence pairs selected from the parallel corpora.
German idiom allesüber einen kamm scheren English equivalent to measure everything by the same yardstick Matching German sentence Aber man kann eben nicht alle Inselnüber einen Kamm scheren.

English translation
But we cannot measure everyone by the same standards.
German idiom in den kinderschuhen stecken English equivalent to be in the fledgling stage Matching German sentence Es steckt immer noch in den Kinderschuhen.

English translation
It is still in its infancy.  For instance, in German, the subject can appear between the verb and the prepositional phrase making up the idiom. German also allows for several re-orderings of the phrase. In order to generalize the process of identifying idiom occurrences, we lemmatize the phrases and consider different re-ordering of the words in the phrase as an acceptable match. We also allow for a fixed number of words to occur in between the words of an idiomatic phrase. Table 3 shows two examples of idiom occurrences that match these criteria. Following this set of rules, we extract sentence pairs containing idiomatic phrases, and create a set of sentence pairs for each unique idiom phrase. In the next step we sample without replacement from these sets and select individual sentence pairs to build the test set. In order to build the new training data, we use the remaining sentence pairs in each idiom set as well as the sentence pairs from the original parallel corpora that did not include any idiom phrases. In this process, we ensure that for each idiomatic expression there is at least one occurrence in both training and test data, and that no sentence is included in both training and test data. Figure 1 visualizes the process of constructing the new training and test sets. As a result, for each language direction, we obtain a targeted test set for idiom translation and the corresponding training corpus representing a natural distribution of sentences with and without idioms. We annotate each sentence pair with the canonical form of its source-side idiom phrase and its equivalent in the target language. Table 2 provides some statistics of the two data sets. For each unique idiom in the test set, we also provide the frequency of the respective idiom in the training data. Note that this is based on the lemmatized idiom phrase under the constraints mentioned in Section 2. and is not necessarily an exact match of the phrase. Table 4 shows several examples from the data set for Ger-man idiom translation. We observe that for some idioms the literal translation in the target language is close to the actual meaning, while for others it is not the case. One side effect of automatically identifying idiom expressions in sentences is that it is not always accurate. Sentence pairs where an idiom expression was used as a literal phrase (e.g., "spill the beans" to literally describe the act of spilling the beans) will be identified as idiomatic sentences.

Translation Experiments
While the main focus of this work is to generate data sets for training and evaluating idiom translation, we also perform a number of preliminary NMT experiments using our data set to measure the problem of idiom translation on large scale data. In the first experiment following the conventional settings, we do not use any labels in the data to train the translation model. In the second experiment we use the labels in the training data as an additional feature to investigate the effect of informing the model of the existence of an idiomatic phrase in a sentence during training. We perform a German→English experiment by providing the model with additional input features. The additional features indicate whether a source sentence contains an idiom and are implemented as a special extra token <idm> that is prepended to each source sentence containing an idiom. This a simple approach that can be applied to any sequence-to-sequence architecture. Most NMT systems have a sequence-to-sequence architecture where an encoder builds up a representation of the source sentence and a decoder, using the previous LSTM hidden states and an attention mechanism, generates the target translation (Bahdanau et al., 2015;Sutskever et al., 2014;Cho et al., 2014). We use a 4-layer attention-based encoder-decoder model as described in (Luong et al., 2015) trained with hidden dimension size of 1,000, and batch size of 80 for 20 epochs.  In all experiments the NMT vocabulary is limited to the most common 30K words in both languages and we preprocess source and target language data with Byte-pair encoding (BPE) (Sennrich et al., 2016) using 30K merge operations.
We also use a Phrase-based translation system similar to Moses (Koehn et al., 2007) as baseline to explore PBMT performance for idiom translation.

Idiom Translation Evaluation
Ideally idiom translation should be evaluated manually, but this is a very costly process. Automatic metrics, on the other hand, can be used on large data sets at no cost and have the advantage of replicability. We use the following metrics to evaluate the translation quality with a specific focus on idiom translation accuracy: BLEU The traditional BLEU score (Papineni et al., 2002) is a good measure to determine the overall quality of the translations. However this measure considers the precision of all n-grams in a sentence and by itself does not focus on the translation quality of the idiomatic expressions.

Modified Unigram Precision
To specifically concentrate on the quality of the translation of idiom expressions, we also look at the localized precision. In this approach we translate the idiomatic expression in the context of a sentence, and only evaluate the translation quality of the idiom phrase.
To isolate the idiom translation in the sentence, we look at the word-level alignments between the idiom expression in the source sentence and the generated translation in the target sentence. We use fast-align (Dyer et al., 2013) to extract word alignments. Since idiomatic phrases and the respective translations are not contiguous in many cases we only compare the unigrams of the two phrases. Note that for this metric we have two references: The idiom translation as an independent expression, and the human generated idiom translation in the target sentence. Word-level Idiom Accuracy We also use another metric to evaluate the word-level translation accuracy of the idiom phrase. We use word alignments between source and target sentences to determine the number of correctly translated words. We use the following equation to compute the accuracy: H is the number of correctly translated words, I is the number of extra words in the idiom translation, and N is the number of words in the gold idiom expression. Table 5 presents the results for the translation task using different metrics. Looking at the overall BLEU scores, we observe that baseline performance on the idiom-specific test set is lower than on the union of the standard test sets (WMT 2008(WMT -2016. While the scores on these two data sets are not directly comparable, this result is in line with previous findings that sentences containing idiomatic expressions are harder to translate (Isabelle et al., 2017). We can also see that the performance gap is not as pronounced for PBMT systems, suggesting that phrase-based models are capable of memorizing the idiom phrases to some extent. The NMT experiment using a special input token to indicate the presence of an idiom in the sentence performs still better than PBMT but slightly worse than the NMT baseline in terms of BLEU. Despite this drop in BLEU performance, by examining the unigram precision and word-level idiom accuracy scores, we observe that this model generates more accurate idiom translations. These preliminary experiments reiterate the problem of idiom translation with neural models, and in addition show that with a labeled data set, we can devise simple models to address this problem to some extent.

Conclusion
Idiom translation is one of the more difficult challenges of machine translation. Neural MT in particular has been shown to perform poorly on idiom translation despite its overall strong advantage over previous MT paradigms (Isabelle et al., 2017). As a first step towards a better understanding of this problem, we have presented a parallel data set for training and testing idiom translation for German→English and English→German. The test sets include sentences with at least one idiom on the source side while the training data is a mixture of idiomatic and non-idiomatic sentences with labels to distinguish between the two. We also performed preliminary translation experiments and proposed different metrics to evaluate idiom translation. We release new data sets which can be used to further investigate and improve NMT performance in idiom translation.