{'fraction_non_alphanumeric': 0.03704309428407883, 'fraction_numerical': 0.01954370758034181, 'mean_word_length': 4.829361296472832, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Recognizing English Learners' Native Language from Their Writings

Native Language Identification (NLI), which tries to identify the native language (L1) of a second language learner based on their writings, is helpful for advancing second language learning and authorship profiling in forensic linguistics. With the availability of relevant data resources, much work has been done to explore the native language of a foreign language learner. In this report, we present our system for the first shared task in Native Language Identification (NLI). We use a linear SVM classifier and explore features of words, word and character n-grams, style, and metadata. Our official system achieves accuracy of 0.773, which ranks it 18 th among the 29 teams in the closed track.


Introduction
Native Language Identification (NLI) (Ahn, 2011;Kochmar, 2011), which tries to identify the native language (L1) of a second language learner based on their writings, is expected to be helpful for advancing second language learning and authorship profiling in forensic linguistics. With the availability of relevant data resources, much work has been done to explore the effective way to identify the native language of a foreign language learner (Koppel et al., 2005;Wong et al., 2011;Hirst, 2012a, 2012b;Bykh and Meurers, 2012;Crossley and McNamara, 2012;Jarvis and Paquot, 2012;Tofighi et al., 2012;Torney et al. 2012).
To evaluate different techniques and approaches to Native Language Identification with the same setting, the first shared task in Native Language Identification (NLI) was organized by researchers from Nuance Communications and Educational Testing Service . A larger and more reliable data set, TOEFL11 , was used in this open evaluation. This paper reports our NLI2013 shared task system that we built at the Department of Computer Science, Henan University of Technology, China. To be involved in this evaluation, we would like to obtain a more thorough knowledge of the research on native language identification and its state-ofthe-art, as we may focus on authorship attribution (Koppel et al., 2008) problems in the near future.
The NLI2013 shared task is framed as a supervised text classification problem where the set of native languages (L1s), i.e. categories, is known, which includes Arabic, Chinese, French, German, Hindi, Italian, Japanese, Korean, Spanish, Telugu, and Turkish. A system is given a large part of the TOEFL11 dataset for training a detection model, and then makes predictions on the test writing samples.
Inspired by our experience of dealing with different text classification problems, we decide to employ a linear support vector machine (SVM) in our NLI2013 system. We plan to take this system as a starting point, and may explore other complex classifiers in the future. Although in-depth syntac-tic features may be helpful for this kind of tasks (Bergsma et al., 2012;Wong and Dras, 2011;Swanson and Charniak, 2012;Wong et al., 2012), we decide to explore the effectiveness of the traditional word and character features, as well as style features, in our system. We would like to verify on the first open available large dataset whether these traditional features work and how good they are. We submitted four runs with different feature sets. The run with all the features achieved the best accuracy of 0.773, which ranks our system 18th among the 29 systems in the closed track.
In the rest of this paper we describe the detail of our system and analyze the results. Section 2 gives the overview of our system, while Section 3 discusses the various features in-depth. We present our experiments and discussions in Section 4, and conclude in Section 5. Figure 1 gives the architecture of our NLI2013 system, which takes machine learning framework. At the training stage, annotated data is first processed through preprocessing and feature extraction, then fed to the classifier learning module, and we can finally obtain a NLI model. At the testing stage, each test sample goes through the same pre-processing and feature extraction modules, and is assigned a category with the learned NLI model. Data Preprocessing: this module aims at transforming the original data into a suitable format for the system, e.g. inserting the category information into the individual writing sample and attaching metadata to essays.

System Description
Feature Extraction: this module tries to obtain all the useful features from the original data. We considered features like: word, word n-gram, character n-gram, style, and available metadata.
Linear SVM training and testing: these two modules are the key components. The training module takes the transformed digitalized vectors as input, and train an effective NLI model, where the testing module just applies the learned model on the testing data. As linear support vector machines (SVM) achieves quite good performance on a lot of text classification problems, we use this general machine learning algorithm in our NLI2013 system. The excellent SVM implementation, Libsvm (Chang and Lin, 2011), was incorporated in our system and TFIDF is used to derive the feature values in vectors. Then, we turn to focus on what features are effective for native language identification. We explore words, word n-grams, character n-grams, style, and metadata features in the system.

Features
In this section, we explain what kind of features we used in our NLI2013 system.

Word and Word n-gram
The initial feature set is words or tokens in the dataset. As the dataset is tokenized and sentence/paragraph split, we simply use space to delimit the text and get individual tokens. We remove rare features that appear only once in the training dataset. Words or tokens are transformed to lowercase.
Word n-grams are combined by consecutive words or tokens. They are expecting to capture some syntactic characteristics of writing samples. Two special tokens, "BOS" and "EOS", which indicate "Beginning" and "Ending", are attached at the two ends of a sentence. We considered word 2grams and word 3-grams in our system.

Character n-gram
We assume sub-word features like prefix and suffix are useful for detecting the learners' native languages. To simplify the process rather than employing a complex morphological analyzer, we consider character n-grams as another important feature set. The n-grams are extracted from each sentence by regarding the whole sentence as a large word / string and replacing the delimited symbol (i.e. white space) with a special uppercase character 'S'. As what we did in getting word ngrams, we attached two special character "B" and "E" at the two ends of a sentence. Character 2grams, 3-grams, 4-grams, and 5-grams are used in our system.

Style
We would like to explore whether the traditional style features are helpful for this task as those features are widely used in authorship attribution. We include the following style features: • __PARA__: a paragraph in an essay; • __SENT__: a sentence in an essay; • PARASENTLEN=NN: a paragraph of NN sentences long; • SENTWDLEN=NN: a sentence of 4*NN words long; • WDCL=NN: a word of NN characters long;

Other
As the TOEFL11 dataset includes two metadata for each essay, English language proficiency level (high, medium, or low) and Prompt ID, we include them as additional features in our system.

Dataset
The dataset of the NLI2013 shared task contains 12,100 English essays from the Test of English as a Foreign Language (TOEFL). Educational Testing Service (ETS) published the dataset through the LDC with the motivation to create a larger and more reliable data set for researchers to conduct Native Language Identification experiments on. This dataset, henceforth TOEFL11, comprises 11 native languages (L1s) with 1,000 essays per language. The 11 covered native languages are: Arabic, Chinese, French, German, Hindi, Italian, Japanese, Korean, Spanish, Telugu, and Turkish. In addition, each essay in the TOEFL11 is marked with an English language proficiency level (high, medium, or low) based on the judgments of human assessment specialists. The essays are usually 300 to 400 words long. 9,900 essays of this set are chosen as the training data, 1,100 are for development and the rest 1,100 as test data.

Official Results
Accuracy, which measures the percentage of how many essays are correctly detected, is used as the main evaluation metric in the NLI2013 shared task. Table 1 gives the official results of our system on the evaluation data. We submitted four runs with different feature sets: HAUTCS-1: all the features, which include words, word 2-grams, word 3-grams, character 2grams, character 3-grams, character 4-grams, character 5-grams, style, and other metadata features; HAUTCS-2: uses words, word 2-grams, word 3-grams, style, and other metadata features; HAUTCS-3: uses words, word 2-grams, word 3-grams, and other metadata features; HAUTCS-4: uses words or tokens and other metadata features.
For the runs HAUTCS-2, HAUTCS-3, and HAUTCS-4, we combined the development and training data for learning the identification model, where for the HAUTCS-1, it's a pity that we forgot to include the development data for training the model.
Our best run (HAUTCS-1) achieved the overall accuracy (0.773). The system performs best on the German category, but poorest on the Hindi category, as can be easily seen on figure 2.
Analyzing the four runs' performance showing on figure 2, we observe: word features are quite effective for Telugu and Hindi categories, but not powerful enough for others; word n-grams are helpful for languages Chinese, French, German, Korean, and Turkish, but useless for others; Style features only boost a little for French; Character ngrams work for Arabic, Chinese, French, Japanese, Spanish, and Turkish; Spanish category prefers character n-grams, where Telugu category likes word features. As different features have different effects on different languages, a better NLI system is expected to use different features for different languages.
After the evaluation, we experimented with the same setting as the HAUTCS-1 run, but included both training and development data for learning the NLI model. We got accuracy 0.781 on the new released test data, which has the same format with paragraph split as the training and development data.
As we include style features like how many paragraphs in an essay, the old test data, which removed the paragraph delimiters (i.e. single blank lines), may be not good for our trained model. Therefore, we did experiments with the new test data. Unfortunately, the accuracy 0.772 is a little poorer than that we obtained with the old test data. It seems that the simple style features are not effective in this task. As shown in table 1, HAUTCS-2 performs poorer than HAUTCS-3, which helps us derive the same conclusion.

Additional Experiments
We did 10-fold cross validation on the training and development data with the same setting as the HAUTCS-1 run. The data splitting is given by the organizers. Accuracies of the 10 runs are show in table 2. The overall accuracy 0.799 is better than that on the test data.  Table 2. Results of 10-fold cross validation on the training and development data.
To check how metadata features work, we did another run HAUTCS-5, which uses only words as features. This run got the same overall accuracy 0.756 on the old test data as HAUTCS-4 did, which demonstrates that those metadata features may not provide much useful information for native language identification.

Conclusion and Future Work
In this paper, we report our system for the NLI2013 shared task, which automatically detecting the native language of a foreign English learner from her/his writing sample. The system was built on a machine learning framework with traditional features including words, word n-grams, character n-grams, and writing styles. Character n-grams are simple but quite effective.
We plan to explore syntactic features in the future, and other machine learning algorithms, e.g. ECOC (Li and Vogel, 2010), also deserve further experiments. As we discussed in section 4, we are also interested in designing a framework to use different features for different categories.