{'fraction_non_alphanumeric': 0.033109619686800894, 'fraction_numerical': 0.013998082454458294, 'mean_word_length': 4.81635687732342, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Crossing the border twice: Reimporting prepositions to alleviate L1-specific transfer errors

We present a data-driven approach which exploits word alignment in a large parallel corpus with the objective of identifying those verb- and adjective-preposition combinations which are difficult for L2 language learners. This allows us, on the one hand, to provide language-specific ranked lists in order to help learners to focus on particularly challenging combinations given their native language (L1). On the other hand, we provide extensive statistics on such combinations with the objective of facilitating automatic error correction for preposition use in learner texts. We evaluate these lists, first manually, and secondly automatically by applying our statistics to an error-correction task.


Introduction
Computational Linguistics and Learner Error research have made impressive progress recently, but they have not reached their collaborative potential yet (Granger and Lefer 2016, p. 281). For example, while language teaching materials contain lists of idioms and phrasal verbs, the decision for which items to include often does not take actual frequency of use or particular difficulties for learners with specific backgrounds into account.
The current paper addresses this shortcoming, by exploiting large parallel and error-annotated learner corpora. We focus on verb-preposition combinations (VPC), including phrasal verbs and adjective-preposition combinations (APC) obtained from a large parallel corpus (Europarl). For brevity's sake we only describe VPC here.
Our aim is to provide practical and customized help to the learner of a language, here English, by This work is licenced under a Creative Commons Attribution 4.0 International Licence.
Licence details: http://creativecommons.org/licenses/by/4.0/ pointing out errors that are likely to be made and to correct them where they occur. In particular, we provide a) a list of VPC/APC that vary considerably between languages, b) a list of specific VPC/APC errors that are to be expected from a native speaker of a particular language, and c) a resource which detects probably incorrect VPC/APC uses and suggests a correction. Concerning c), advances have been made recently due to the CoNLL shared tasks on grammatical error correction (Ng, M. S. Wu, Y. Wu, et al. 2013;Ng, M. S. Wu, Briscoe, et al. 2014), and due to systems targeting preposition errors (Tetreault and Chodorow 2008;Boyd et al. 2012). We evaluate our results on ICLE (Granger, Dagneaux, et al. 2002), the FCE dataset (Yannakoudakis et al. 2011), and the NICT Japanese Learner English Corpus 1 . Furthermore, we exploit ICLE in combination with the British National Corpus (BNC) (Aston and Burnard 1998) to attain collocation statistics which allow us to evaluate the proposed suggestions for corrections.
Non-standard uses by language learners, which we refer to as errors here, can be found at any linguistic level. Some errors can be detected easily by current word-processing tools (e.g. spelling errors) or by re-reading, or consulting dictionaries. But particularly in areas where grammar and lexis interact, there is typically a lack of tools.
One frequent source of lexico-grammatical errors are VPC. While semantically transparent prepositions (e.g. stand on) are relatively stable cross-linguistically, the frequent nonsemantic prepositions (e.g. wait for) and phrasal verbs (e.g. depend on) show enormous cross-linguistic variation. VPC are difficult to acquire for language learners (Gilquin, Granger, et al. 2011, pp. 59-60). Phrasal verbs represent "one of the most notoriously challenging aspects of English language instruction" (Gardner and Davies 2007, p. 339;see also Gilquin 2015). In the CoNLL shared tasks, prepositional errors were the third most frequent error type at 5 to 9 % of all errors (only determiner errors and noun number are more frequent). We include APC as they are often similarly difficult to acquire for learners of English. Benson et al. (2009) recognize APC as an independent category in addition to VPC.

Corpus Preparation
We extracted parallel text units in English We identified approximately 40 million tokens in five languages: English, French, German, Italian and Spanish. Finnish and Polish have considerable fewer tokens than the other languages (30 million and 10 million, respectively). 2

Tagging and Lemmatization
For tagging and lemmatization, we used TreeTagger (Schmid 1994). To increase tagging accuracy for words unknown to the language model, we had to extend the tagging lexica, especially the German one, with lemmas and part-of-speech tags for frequent words. Moreover, we used the word alignment information between the languages (see below) to disambiguate lemmas for those tokens where the TreeTagger provided multiple lemmatization options. 3

Alignment
On the sentence segments identified (about 1.7 million per language), we performed pairwise sentence alignment with hunalign (Varga et al. 2005) and based on that word alignment with GIZA++ (Och and Ney 2003;Gao and Vogel 2008) and the Berkeley Aligner (Liang et al. 2006). While the Berkeley Aligner computes bidirectional word alignments, the alignments of GIZA++ are unidirectional and thus need to be symmetrized if bidirectional alignments are required. We chose the 2 For Polish this is due to the fact that Poland joined the European Union in 2004 and translations for the debates are only available from 2006 onwards. In case of Finnish, the gap can be explained by the language itself which features a rich morphology, thus resulting in less tokens at the expense of more word forms. 3 The disambiguation approach is similar to the one Volk et al. (2016) describe, except that we combine alignment information for all languages simultaneously. union symmetrization method since it increases recall. Word alignment was performed on the types of all tokens and on lemmas of content words. 4 For the latter, we mapped the individual tag sets to the universal tagset defined by Petrov et al. (2012) and defined content words to be those tokens being tagged as nouns, verbs, adjectives or adverbs.

Parsing
We used MaltParser (Nivre et al. 2006) to derive syntactic dependency relations in English. For parsing our tagged texts, we had to map several part-of-speech tags beforehand as the standard English parameter file distributed with TreeTagger slightly differs. 5

Methods
In the following, we first present our concept of backtranslating prepositions based on automatic annotation and alignment frequencies. We then apply it to VPC and introduce our method for error correction.

Distributions
In a first step, we calculate a lemma distribution matrix by aggregating lemma counts on token alignments. This matrix tells us the translation ratio of each lemma. Each cell contains the probability of a lemma in the source language to be translated into a lemma in the foreign language. For example, the English verb suffer is translated to German leiden in 42 % of the cases. We then retrieve the set of all English VPC (consisting of verb λ v and preposition λ p with the verb showing a syntactic 'preposition' relation to the preposition as depicted in Fig. 1) and calculate the distribution of observed prepositions. For example, the English verb suffer occurs with the preposition from in 26 % of all cases, but also, more rarely, with other prepositions. 6 We do not attempt to make a distinction between phrasal verbs, PP complements or PP adjuncts in our data-driven approach.
For each VPC, we count the foreign prepositions λ p ′ as they are aligned with the source VPC's prepositions λ p . 7 We do this step for each language separately.

Backtranslation Score (BTS)
By multiplying these foreign prepositions with the lemma distribution matrix, we obtain a list of English prepositions and values that we call backtranslation score (BTS). BTS tells us how preferred a certain source language preposition is 8 for a foreign language, given a particular VPC.

Backtranslation Ratio (BTR)
We then normalize BTS to what we refer to as backtranslation ratio (BTR), such that the BTR of the correct English preposition for a particular verb and language is 1.0, i.e. each preposition's BTS divided by the BTS of the correct original preposition, which is shown in Table 1. A BTR above 1.0 indicates that it is more likely to choose a wrong preposition than the correct one, according to our language model, which is based on alignment (see Appendix for the most likely incorrect preposition per language, with their BTR).
The BTR calculated for English VPC give us an impression of how difficult the preposition of a particular expression would be for a speaker of the respective language. For instance, the highest BTR for the verb aim is 2.74 for German (preposition on, presumably due to German zielen auf ) and 2.81 for French (preposition in, indirectly due to French viser + object, see next subsection) while at is 1.0 by definition.
We also include the raw frequency of VPC and derive the final ranking for each VPC and language based on both normalized scores. 9 For space rea- sons, we only present the intersection of all language specific VPC and APC lists in Table 2.

Suggestions for Corrections
In addition to lists of difficult VPC and APC, we also suggest a correction for incorrect combinations based on the distribution of prepositions retrieved. Errors can be simple misproductions such as typos or copy-paste errors, which are typically spotted when carefully re-reading a text. But when speakers of certain linguistic backgrounds keep producing the same non-standard form repeatedly, often due to native language influence such as transfer, they make errors which are more difficult to detect for them, and thus a resource which spots these is particularly helpful. These errors follow a repeated pattern, often reaching collocational status. Schneider and Gilquin (2016) use collocation-based statistics to detect such non-standard VPC by measuring the expected (E) collocational strength in Learner English (based on the International Corpus of Learner English (ICLE)), compared to the observed (O) collocational strength in native English (based on the BNC).
We detect VPC errors following the same method, then address the question if we can provide the appropriate correction. Given an incorrect VPC, we suggest the most likely preposition, given the verb. As some errors involve a preposition instead of a direct object, our algorithm suggests to  use a direct object if the raw frequency of a verb is at least twice as high as the number of VPC involving that verb.

Results
In the following, we present results for all three aims identified above. As we cannot give the full lists of recommended language-specific lists here, 10 we will focus instead on three verb-preposition combinations that are particularly useful to concentrate on and to learn for native speakers of German: • suffer from: corresponds to German leiden unter, the preposition 'unter' directly translates as 'under'. • wait for: corresponds to German warten auf, 'auf' directly translates as 'on'. • consist of: corresponds to German bestehen aus, 'aus' directly translates as 'from'.
The recommended lists overlap, yet also differ considerably between languages. The amount of overlapping VPC of the whole lists ranges from 58 % for German-Polish to 97 % for French-Italian, reflecting the typological similarity of the languages. We consider those items that occur in each of the 5 language-specific lists as generally hard to learn. This language-independent list is given in Table 2.
The list of the top true positives, i.e. the correct suggestion for erroneous or non-standard uses of VPC/APC structures from Schneider and Gilquin (2016) is given in Table 3. The first column shows the verb or adjective, the second column the incorrect preposition, the third column the manually corrected preposition. obj means that the manual annotation suggests to use a direct object instead of a PP (e.g. attack against someone has manually been corrected to attack someone), and n/a means that the manually suggested correction is more complex, e.g. diverse by has manually been corrected to different according to. The ultimate column shows whether the automatic correction matches the manual correction.

Evaluation
We have evaluated our approach in two ways, which we describe in the following.   (Schneider and Gilquin 2016).
First, we have evaluated the list of languageindependent suggestions. In column 3 of table 2, we consider an item a true positive if it contains a non-semantic, non-compositional preposition, or if the preposition is language-specific. Precision is at 72 %. Our method does not seem to work reliably on the preposition 'into', which does not exist as a preposition in most languages, but which is semantically transparent. We thus decided to exclude this preposition in the second evaluation, given in columns 4-6, in which we check if errors corresponding to this type occur in learner corpora. 74 % of the remaining combinations are found in at least one of the learner corpora.
Second, we have tested the ability of our method to correct frequent non-standard or erroneous verband adjective-preposition combinations. The results are given in Table 3. PREP is the erroneous preposition, CORR the suggested correction by our algorithm, and MATCH? indicates if the suggested correction is correct. The results indicate a precision of 79.2 %, and the upper bound (n/a cannot be predicted correctly) is 95.8 %. Some of the errors may stem from the fact that the European parliament uses some fixed phrases that are rare in other registers. Tetreault and Chodorow (2008) report 80 % precision at 19 % recall on the task of recognizing preposition errors in essays written by non-native students. Boyd et al. (2012) report 40 % F-score on recognizing preposition errors, and 30 % Fscore on correcting them. The best performing system on the prep type of error in Ng, M. S. Wu, Briscoe, et al. (2014) is Felice et al. (2014), who report about 40 % precision and recall using a combination of a rule-based and an SMT approach. As their task includes both recognizing and correcting preposition errors, and all the above approaches use token-based evaluation while ours is type-based, a comparison is difficult to make, but our results appear to be competitive.

Conclusions
We have employed word alignment in a large parallel corpus to identify potentially difficult VPC/APC. We have compiled language-specific ranked lists in order to help learners to focus on particularly challenging combinations given their native language (L1). We have also combined the language-specific findings into a list of generally difficult combinations. As expected, Ro-Proceedings of the Joint 6th Workshop on NLP for Computer Assisted Language Learning and 2nd Workshop on NLP for Research on Language Acquisition at NoDaLiDa 2017 mance languages exhibit a larger overlap of combinations than German, and Polish is particularly different.
We evaluate our procedure in two ways. First we have manually assessed the precision of the language-independent list, which obtains 72 % precision. Secondly, we apply our method to an error correction task to predict the intended preposition given frequent erroneous VPC or APC. We achieved a precision of 79.2 %.
For future work, we plan to conduct the same calculations for other languages so that we will be able, for instance, to predict potentially erroneous use of German prepositions by native speakers of other languages.