{'fraction_non_alphanumeric': 0.025365035952181394, 'fraction_numerical': 0.01574837884335436, 'mean_word_length': 4.975224037954665, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Developing NLP Tools with a New Corpus of Learner Spanish

The development of effective NLP tools for the L2 classroom depends largely on the availability of large annotated corpora of language learner text. While annotated learner corpora of English are widely available, large learner corpora of Spanish are less common. Those Spanish corpora that are available do not contain the annotations needed to facilitate the development of tools beneficial to language learners, such as grammatical error correction. As a result, the field has seen little research in NLP tools designed to benefit Spanish language learners and teachers. We introduce COWS-L2H, a freely available corpus of Spanish learner data which includes error annotations and parallel corrected text to help researchers better understand L2 development, to examine teaching practices empirically, and to develop NLP tools to better serve the Spanish teaching community. We demonstrate the utility of this corpus by developing a neural-network based grammatical error correction system for Spanish learner writing.


Introduction
Studies in second language acquisition benefit from large quantities of corpus data from L2 learners, which facilitate the analysis of developmental patterns with respect to their emerging grammar (Hawkins and Buttery, 2009). In addition, the development of effective NLP systems for second language instruction, such as automated grammatical error correction and automated student assessment, depends on the availability of large annotated learner corpora (Kasewa et al., 2018). While annotated learner corpora of English are widely available, large learner corpora of Spanish are less common, and as a result, the field has seen little datadriven research on the developmental processes that underlie Spanish language learning, or on the development of NLP tools to assist teachers and students of Spanish. This may come as unexpected, considering the fact that there exists a relatively high demand for learning Spanish; in 2013, fifty-one percent of students enrolled in university language courses in the United States studied Spanish (AAAS, 2016) and there are over 21 million learners of L2 Spanish across the globe (Instituto Cervantes, 2019). Given the large demand for Spanish courses, a need exists for NLP tools to help instructors better assess incoming students' abilities and to assist with grading and correction of student writing. However, without adequate datasets, the development of such tools is challenging. We describe a project that seeks to address the limited availability of annotated Spanish learner data through the development of COWS-L2H (the Corpus Of Written Spanish -L2 and Heritage speakers). This growing corpus aims to be the largest and most comprehensive corpus of learner Spanish freely available to the research community. In addition to raw texts, COWS-L2H has been annotated with corrections for specific common errors made by learners of Spanish. Finally, a large portion of the corpus has been manually corrected by Spanish instructors, providing valuable parallel training data for NLP tasks such as open-ended grammat-ical error identification and correction. The development of the present corpus is necessary given the notable gaps that exist in the current array of available learner corpora of Spanish, as well as the limited work that has been done in NLP tools for Spanish instruction. To demonstrate the utility of the corpus, we train a grammatical error correction (GEC) system based on a neural encoder-decoder architecture using the COWS-L2H data. Our system is the first to attempt grammatical error correction for Spanish without a focus on specific errors, and demonstrates how this new data resource opens possibilities for NLP researchers to work with learner Spanish.

Current Spanish Learner Corpora
Text written by fluent Spanish speakers is widely available (e.g. Wikipedia and other corpora, such as the Corpus del Español (Davies, 2002) and the various corpora organized by the Real Academia Española). Additionally, several corpora of transcribed spoken Spanish produced by L2 learners, such as CORELE (Campillos Llanos, 2014), the Corpus Oral de Español como Lengua Extranjera, and SPLLOC (Mitchell et al., 2008), the Spanish Learner Language Oral Corpus, are available to the research community. However, few corpora of written learner Spanish are available for use by NLP researchers, and those corpora of written learner Spanish that have been compiled do not include annotations to facilitate training NLP models. For example, CAES, the Corpus de Aprendices de Español (Rojo and Palacios, 2016), is searchable via a concordancer, but it is not error-annotated and raw text is not easily downloadable. Other potentially promising corpora of L2 Spanish for NLP researchers include Aprescrilov (Buyse and González Melón, 2012) and the Corpus Escrito del Español como L2, or CEDEL2 (Lozano and others, 2009), which contain approximately 1 million and 750,000 words, respectively. While the these corpora are well suited to language modeling and other unsupervised NLP tasks, they do not contain error annotations needed to develop error correction tools for language learners. We hope to begin to address the lack of annotated data in Spanish written learner corpora with the present project.

The COWS-L2H Corpus
The COWS-L2H project aims to provide a corpus of learner Spanish to the research community which is freely and publicly available, which contains metadata that is relevant to researchers, and which is error-annotated and corrected for use in developing NLP applications. To encourage use of the corpus by researchers, all data is available on the project Github repository 1 .

Data collection
To date, we have collected 3,516 essays from 1,370 individual students enrolled in university-level Spanish courses in the United States. Of these, 1,695 essays, consisting of 33,950 sentences, from 737 students are currently available in the project repository. We currently have an ongoing data collection, anonymization and annotation process, and additional essays will be added to the public release of the corpus as soon as possible. Based on the current growth rate of the corpus, we expect the corpus to contain over 4,000 essays and 1.2 million tokens by the summer of 2020. We have recently entered into a collaboration with a Spanish university to assist in our annotation and anonymization efforts, which we anticipate will greatly increase the rate at which we are able to make additional data public. The students who submit essays to the corpus are enrolled in one of ten undergraduate quarter-long Spanish courses, which we group into four categories, as shown in Table 1. The distribution of the essays across the levels is uneven due to the distribution of students enrolled in our Spanish courses. Because more students enroll in beginning Spanish courses than in advanced levels, a larger number of essays submitted to the corpus come from these beginner-level courses. During each academic quarter (ten weeks of instruction), participants are asked to write two essays in Spanish that adhere to a minimum of 250 and a maximum of 500 words, though students enrolled in Spanish 1 are allowed to write essays with a lower minimum word count, due to the fact that many of these students are true beginners in L2 Spanish who would possess relatively little vocabulary and grammatical resources of their own. Participants are asked to write each essay they submit in response to one of two short prompts. Both prompts are presented with a distinct brevity, to allow for a broad degree of creative liberty and open-ended interpretation on the part of the writer. To test the effect of prompt on student writing and promote diversity in our corpus, we periodically change the prompts presented to students. To date we have presented four essay prompts. For the first set of compositions, collected from 2017 to 2018, participants were asked to write about "a famous person" and "the perfect vacation." For more recent compositions, collected from 2018 to the present, the prompts were "A special person in your life" and "A terrible story". We have collected an average of 900 essays in response to each of the four prompts we have used to date. Given the diverse backgrounds of our students, especially those who enroll in courses for Heritage speakers, identifying the specific variety of Spanish in the essays is challenging; however, our courses are generally taught using a standard variety of academic Spanish, so we expect this to be the predominant variety in the corpus. Students provide information about their linguistic background which we include as metadata in the corpus; this metadata may elucidate variability in usage resulting from students' past experience with Spanish. The metadata also allows us to test the effects variables such as L1 on student writing. Finally, the linguistic metadata may facilitate the use of filtered subcorpora for targeted training of NLP systems; for example, Nadejde and Tetreault (2019) demonstrate that grammatical error correction systems benefit from adaptation to L1 and proficiency level.

Error annotation
One of the primary goals of this project is to annotate grammatical errors in the corpus in a way that writing patterns typical of Spanish as a foreign language produced by student participants can be identified, catalogued, and easily utilized by researchers who use the corpus. To this end, we have begun the process of error-tagging the corpus based on specific error types; the first two error types for which we have completed annotation are gender and number agreement, and usage of the Spanish a personal. We chose to annotate these specific error types based on research questions we wished to explore, but we intend to expand our error annotations in the future, as our annotation scheme can be readily adapted to additional error types we chose to annotate. Further, we encourage other researchers to adapt the annotation scheme to the annotation of other error types and contribute their work to the COWS-L2H project. Our current team of annotators consists of four graduatelevel Spanish instructors who have native or near-native fluency in Spanish. As previously mentioned, we are expanding our error annotation project through a collaboration with a Spanish university, which will allow us to significantly expand both the number of annotators and the scope of our error annotation project in the near future. Our in-text error-tagging scheme is as follows: [error]edit<annotation>.
Consider the example error in (1), and its annotation in (2): (1) Yo vivo en el ciudad. "I live in the city." Error type α F 0.5 Gender-Number 0.780 0.784 "a personal" 0.741 0.730 Average 0.761 0.757 In (2), the first set of brackets encloses the words in the error in question, the curly brackets that follow give the corrected edit, and the angle brackets house the error tags. In this case, the tags indicate that the error was a gender agreement error (ga), that masculine gender was erroneously produced in place of the correct feminine gender (fm), and that the error occurred on the article (art). A full description of the error annotation scheme is provided with the dataset in the corpus GitHub repository. Each essay is annotated by at least two of our four annotators to ensure the accuracy of our annotations and the suitability of our annotation scheme.
Due to the open-ended nature of the annotation task (any token can be considered a possible position of annotation), determining the best measurement for inter-annotator agreement is challenging. In Table 2, we report Krippendorf's α (Krippendorff, 2011) considering every token as an annotation position. Thus, if both annotators choose to not annotate a token, indicating that the token is correct, we treat this lack of annotation as agreement. This choice makes sense because, by not making an explicit annotation on a given token, the annotators are implicitly labeling the token as correct. An alternative method of calculating agreement would be to consider only positions where at least one annotator indicated an error; however, this choice would ignore all positions at which both annotators agreed that no error exists, which is itself a form of agreement. To put our agreement values in more familiar context, we also report the F 0.5 score, commonly used in GEC, using one annotator as ground-truth. In terms of both Krippendorf's α and F 0.5 , our annotators show strong agreement.

Parallel corrected text
In addition to annotation of selected errors, our goal is to include corrected versions of the essays in this corpus. Currently, the compositions collected in this project are corrected by two doctoral student associate instructors of Spanish. Both have native or near-native command of Spanish, have previously taught the Spanish courses from which the students have been recruited to participate in this project, and thus are accustomed to recognizing, interpreting, and correcting errors made by students of L2 Spanish.
To date, we have corrected approximately one-fifth of the essays in the corpus, for 12,678 sentences (168,937 tokens) of corrected text. The distribution of corrected essays is shown in Table 3. Unlike the error annotations, which target specific errors, the corrections made to this set of essays are more holistic in the manner of an instructor correcting a student's work. The result of the correction process is a corrected version of the text, from which corrections can   (Bryant et al., 2017). Additionally, we align the original and corrected sentences to create parallel data that can be used for training NLP systems such as grammatical error correction. To our knowledge, our corpus represents the first parallel dataset of corrected Spanish text available to researchers. As with our error annotations, we are in the process of completing additional corrections and anonymization, and will make more data publicly available as soon as practical. As can be seen in Table 3, the largest portion our currently annotated corpus come from beginning students; completing additional corrections will allow us to present a larger number of errors from students at more advanced levels.

Course Level Essays Tokens Errors
Given the wide variety of ways a sentence can be corrected, our goal is to have each essay corrected by three individuals. Multiple corrections will increase error coverage in our training data and will provide additional test references for NLP researchers who are trying to build automated error identification and correction models.
Original Las chicas encanta lo mucisca y su cabello . Corrected A las chicas les encantan su música y su cabello .

Grammatical error correction
We hope that researchers will take advantage of the COWS-L2H data to build new NLP tools for Spanish learners. To demonstrate the utility of the COWS-L2H corpus to NLP researchers, we implement a grammatical error correction system using the parallel error-corrected sentences from the corpus.

Previous work in GEC
The machine translation approach to GEC frames error correction as a monolingual translation task in which the source and target languages are "with errors" and "without errors," respectively (Ng et al., 2014). Approaches developed originally for translation between different languages have been adapted and applied successfully to grammatical error correction (Napoles and Callison-Burch, 2017;Leacock et al., 2010, p. 95). Similar monolingual translation approaches have been used for paraphrase generation (Quirk et al., 2004) and text simplification (Coster and Kauchak, 2011). Recent work has shown neural machine translation (NMT) (Bahdanau et al., 2014) to be an effective Original En 1990 se fundó la radio pública Ràdio Nacional d'Andorra . Noised 1990 se fundó la radio pública Ràdio nnliocaa años d'Andorra . Table 5: Artificial noising of data approach to the GEC task (Zhao et al., 2019;Chollampatt and Ng, 2018;Junczys-Dowmunt et al., 2018). While framing the problem of error correction as a monolingual translation task is promising, the approach requires parallel training data (Rei et al., 2017), which if not publicly available, must be created by manually correcting text containing errors or by artificially generating errors in grammatical text. Kasewa et al. (2018) demonstrate the use of artificial errors to train a GEC system; however, their method requires real-world parallel text to train their noise model used to generate artificial errors in grammatical text. Similarly, Xie et al. (2018) use a noising model trained on a "seed corpus" of parallel sentences to build an effective GEC system trained on artificially generated parallel noised data. Junczys-Dowmunt et al. (2018) show that effective neural GEC can be achieved with a relatively small amount of parallel training data when techniques such as transfer learning are employed. Each of these approaches were applied to error correction in English only. Grundkiewicz and Junczys-Dowmunt (2019) expands this work, demonstrating a GEC system for German and Russian which use small corpora of corrected text to fine-tune his baseline system trained on artificial data.

Data
We use two distinct datasets in training our GEC system for Spanish learners. First, we generate artificial parallel training data by randomly adding noise to a dataset of 1.3 million grammatical sentences from the Polyglot dump of Spanish Wikipedia (Al-Rfou et al., 2013), with 150,000 sentences set aside for validation. Similar to the method of Zhao et al. (2019), the addition of random noise consists of three operations -token deletion, token insertion (from the fifty most common tokens in the corpus), and token scrambling, with each operation applied to 10% of tokens. The resulting dataset thus has approximately 30% of tokens mutated in some way, as shown in Table 5. Adding noise to the data in this manner follows the idea behind a denoising auto-encoder (Vincent et al., 2008) which learns underlying features in the process of denoising data. By training on our artificial noisy data, the system builds a language model of Spanish which enables it to construct grammatical Spanish sentences from noisy input. While Junczys-Dowmunt et al. (2018) show that selective augmentation using suggestions from a spell-checker results in better performing GEC systems, we chose to use this simpler method of data noising for the purpose of demonstrating the value of the COWS-L2H data. Our learner data, which we reserve for fine-tuning and testing, consists of 10,000 parallel uncorrected and corrected sentences drawn from COWS-L2H, with 1,400 sentences set aside for validation and 1,400 for testing.

Model and training procedure
We train a neural machine translation (NMT) model with a 4-layer bi-directional LSTM encoder and an LSTM decoder, both with 500 hidden units in each layer. We implement the model using OpenNMT-py (Klein et al., 2017). We use the Adam optimizer with a learning rate of 0.001, a dropout of 0.3, and a batch size of 64. Our decoder uses beam search with a beam size of 5. We train for 40,000 steps on 1.04 million sentences of the artificially noised Wikipedia data. After this initial training is complete, we fine tune the model for an additional 5,000 training steps on 10,000 sentences of parallel COWS-L2H data to achieve our final model.

Results
We evaluate model performance using the ERRANT scorer (Bryant et al., 2017), which although designed to annotate errors in English text is capable of aligning edits in Spanish text; we reviewed the ERRANT output to ensure that alignments were accurate. With a model trained as described above, we achieve an F 0.5 score of .224. While this figure is lower than state-of-the-art error correction systems for English evaluated on the CoNLL-2014 dataset, we must conduct further research to determine the overall performance of this system relative to other possible model configurations using the COWS-L2H dataset. As this system is the first NMT-based grammatical error correction for Spanish learners, we have no specific baseline with which to compare our model. One important consideration is that the manual corrections made to the COWS-L2H essays were done in the manner of a teacher correcting a student's writing, and thus include many corrections which are more stylistic than grammatical in nature. We can, however, confirm the effectiveness of our training procedure that combines artificially noised data with Spanish learner text. In Table 6 we show results for training on the artificially noised data alone, the parallel data alone, and the final model pretrained on the artificial data and fine-tuned on the parallel COWS-L2H data. We believe that the model performance could be improved by increasing the amount of artificial training data and by developing a better method of inserting errors. However, as the purpose of this system is to highlight the utility of the COWS-L2H data, we chose to use a simple method of pretraining our model.

Conclusion
Large corpora of language learner text are critical to the development of effective NLP tools for the L2 classroom. Such tools could assist teachers with time-consuming grading and assessment, and give students rapid feedback. While annotated learner corpora of English are widely available, large learner corpora of Spanish are less common, and those that exist do not include annotations needed for development of error correction tools for language learners. As a result, the field has seen little research in the development of NLP tools designed to benefit Spanish language learners and teachers. COWS-L2H is a freely available corpus of Spanish learner data which includes raw text, student demographic and linguistic information, error annotations and parallel corrected text. We demonstrate the utility of this data to NLP researchers by training a GEC system on COWS-L2H parallel data. By pretaining a GEC system on artificially noised Spanish Wikipedia text then fine-tuning on parallel data from COWS-L2H, we create the first NMT-based GEC system for Spanish. In future work, we hope to improve GEC performance for Spanish by experimenting with alternative system architectures and developing more linguistically motivated methods of error insertion rather than adding noise in a completely random manner. Finally, we hope that the availability of this new resource will spark additional research into the development of NLP tools for Spanish education.