{'fraction_non_alphanumeric': 0.04233611825192802, 'fraction_numerical': 0.01622750642673522, 'mean_word_length': 4.787540678754068, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Word Order Sensitive Embedding Features/Conditional Random Field-based Chinese Grammatical Error Detection

This paper discusses how to adapt two new word embedding features to build a more efficient Chinese Grammatical Error Diagnosis (CGED) systems to assist Chinese foreign learners (CFLs) in improving their written essays. The major idea is to apply word order sensitive Word2Vec approaches including (1) structured skip-gram and (2) continuous window (CWindow) models, because they are more suitable for solving syntax-based problems. The proposed new features were evaluated on the Test of Chinese as a Foreign Language (TOCFL) learner database provided by NLP-TEA-3&CGED shared task. Experimental results showed that the new features did work better than the traditional word order insensitive Word2Vec approaches. Moreover, according to the official evaluation results, our system achieved the lowest (0.1362) false positive (FA) and the highest precision rates in all three measurements.


Introduction
In recent years, the rise of Asian economies and nearly 20 years of rapid development of China has led to a corresponding interest in the study of Standard Chinese ("Mandarin") as a foreign language, the official language of mainland China and Taiwan.
However, it might be a great challenge for those CFLs to learn how to write an essay or report in Chinese. Because approximately 3,000 Chinese characters and 5,000 words are required for receiving Test of Chinese as a Foreign Language (TOCFL) certificate in advanced level 1 . Beside, Chinese is an analytic language, in that they depend on syntax (word order and sentence structure) rather than morphology, i.e., changes in form of a word, to indicate the word's function in a sentence. And Chinese also makes heavy use of grammatical particles to indicate aspect and mood, such as like 了 (le , perfective), 還 (hái, still), 已經 (yǐjīng, already), and so on.
For example, Lee et al. (2013) applied a set of handcrafted linguistic rules with syntactic information to detect errors occurred in Chinese sentences written by CFLs. Lee et al. (2014) then further implemented a sentence judgement system that integrated both rule-based an and n-gram statistical methods to detect grammatical errors in Chinese sentences. Lin and Chen (2015) proposed a system which measured the likelihood of sentences generated by deleting, inserting, or exchanging characters or words in which two sentence likelihood functions were proposed based on frequencies of space removed version of Google n-grams.
On the other hand, Xiang (2015) utilized an ensemble classifier random feature subspace method for CGED task. Cheng et al. (2014) proposed a CRF-based method to detect word ordering errors and a ranking SVM-based model to suggest the proper corrections. Finally, Chen et al. (2015) and Yeh et al. (2015) also adopt CRFs and collected a set of common grammatical error rules for building CGED systems.
Among these two methods, the classification-based approach, especially the CRF-based one is quite promising. Because, CRFs treat the CGED problem as a sequence-to-sequence mapping task, it could then model well the word ordering and sentence structure. However, traditional CRF-based approaches often only take current and few neighbouring words and their POS tags as the input features. This may limit CRFs' horizon vision. Besides, word-based features will result in the sparse training data problem, since the total number of Chinese words is more than 160,000 2 .
To alleviate these two difficulties, this paper would like to discuss how to adapt word embedding features to alleviate the sparseness issue and especially how to extract two new word order sensitive embedding features proposed by  to capture ordering information. The major idea is to apply word order sensitive Word2Vec approaches including (1) Structured Skip-gram and (2) CWindow models. Because they seriously take word ordering information into account and are therefore more suitable for solving syntax-based problems. By this way, we hope we could build a more efficient CGED system.
2 System Implementation for NLP-TEA-3&CGED shared task The block diagram of our proposed system is shown in Fig. 1. It has a CRF-based traditional Chinese parser for word segmentation and POS tagging frontend and a CRF-based CGED backend. But the major enhancement comparing with other CRF-based approaches is that it applies the word order sensitive Word2Vec module to extract word embedding vectors and then does word clustering to generate input features for CRF-based CGED module. In the following subsections, several system components will be discussed in more detail, including (1) traditional Chinese parser, (2) word order sensitive Word2Vec, (3) grammatical rule and (4) CRFbased CGED models.

Traditional Chinese Parser
The parser used in this system (as shown in Fig. 2) is a CRF-based system for traditional Chinese. It has three main modules including (1) text normalization, (2) word segmentation and (3) POS tagging. This parser was trained using Sinica Balanced Corpus ver. 4.0 3 . Its performance is as follows: The Fmeasure of the word segmentation is 96.72% for the original database and 97.50% for the manually corrected corpus. The difference between precision and recall rate is less than 0.06%. The accuracy of the POS (47-type) tagging is 94.97%.
It is worth noting that this CRF-based Chinese word segmentation and parser is originally built for automatic speech recognition (ASR). So another purpose of this study is to examine how generalization and sophistication our parser is. Since Chinese words are not well defined (without word boundaries), a high quality Chinese word segmentation and parser is essential for building an effective word embedding representation and a good CGED system.

Word Order Sensitive Embedding Feature Extraction
One way to alleviate the sparse training data problem is to use word classes instead of words themselves as the input features for CRF-based CGED system. The most widely used tools for building word clustering are the models described in (Mikolov 2013a, b, c), including the "Skip-gram" and the "Continuous Bag-of-Words" (CBOW) models. However, since these models only give a word a single embedding feature vector, they are insensitive to word order and may not be suitable for CGED tasks. Therefore, in this paper, we will adopt two new word order sensitive embedding approaches including (1) CWindow and (2) Structured Skip-gram (see Fig. 3) models proposed by  to take word ordering information into account.
Basically, CWindow defines a output predictor O ∈ ℝ (|V |×(2c×d)) that takes a (2c × d)-dimensional vector [e(w −c ), …, e(w −1 ), e(w 1 ), …, e(w c )] (the embeddings of the context words) as input. Words in different position hence have different weights. Structured Skip-gram, on the other hand, defines a set of c × 2 output predictors (O −c , ..., O −1 , O 1 , O c ), with size O ∈ ℝ <(|V |)×d, to predict the outputs according to their positions. These two models then will generate word order sensitive embeddings features. By this way, it should be easier for CGED system to detect abnormal word ordering or sentence structures.

CRF-based Chinese Grammatical Error Detection
The performance of CRF-based model is mainly decided by the quality of feature engineering. In this work, five different features and four sets of feature templates are designed for building our CRF-based CGED system. First, the five features are (1) word length, (2) POS, (3) reduced POS 4 , (4) word class index, (5) errorprone word indicator. Fig. 4 shows an example of how these features are putting together with the grammatical error-type ground-truth to form a training data file. Second, the four sets of feature templates are specified in Table 1. They consider some combinations of the five features (and their n-grams). It is worth noting that the fourth template "(POS n-1 EPWI n POS n+1 )" could be treated as a generalization of the "Error-prone words (word + POS)" pattern proposed in Yeh (2015) (mentioned in previous subsection).

NLP-TEA & CGED Shared Task
The goal of the NLP-TEA3&CGED shared task is to develop systems to automatically diagnose Chinese sentences written by CFLs. The systems should indicate where and what type of errors are embedded in CFLs' sentence. In the following experiments, the effectiveness of the error-prone word templates was first checked. Then the performance of the new "CWindow" and "Structured Skip-gram" were compared with the original "Skip-gram" and the "CBOW" models. Finally, the official evaluation results of our three CWindow-based submissions were discussed.

TOCFL learner database
The TOCFL learner database (NLP-TEA3) provided by the organizers was used to develop our CGED system. In order to enlarge the pool of training samples, the data sets of the two previous editions of this shared task, i.e., NLP-TEA1 ) and NLP-TEA2 (Lee et al. (2015)) are also added together. In the end, there are in total 63,462 sentences for system development.  The development data was further divided into a training and a testing subsets by a ratio of 9:1. Therefore, there are 57,116 and 6,346 sentences in the training and testing subsets, respectively.
In all methods, the vector size was set to 300 and using a context window of 13 (6+1+6) words. To speed up the computation, the probability of a target word was estimated with the hierarchy Softmax method. After the vector space is established, k-mean algorithm was utilized to cluster all words into 1,024 classes.
Finally, the CRF++ toolkit developed by Kudo 11 was utilized to build our CRF-based CGED system. It is worth noting although there are four different Word2Vec frontends but the CRF backend is the same (except the input word class features) for all following experiments.

Preliminary Results on Development Dataset
First of all, Table 3 shows the impact of (with and without) error-prone words evaluated using a CWindow/CRF-based system. According the results, it indicates that those special words did help to improve the performance of our CRF-based CGED system. Therefore, those error-prone words will be considered in all systems reported below.  Table 3: Performance comparison on the effectiveness of adding the error-prone words feature templates on a CWindow/CRF-based CGED system.
Second, Table 4 showed the performance of the Structured Skip-gram-, CWindow-, Skip-gram-and CBOW-based CGED systems (all take error-prone words into account). It is found that CWindow achieved the best F1-score. Because F1-score is the most balanced performance measurement, all our submissions will use the CWindow-based approach.

Approach
Accuracy  Table 4:Performance of the Structured Skip-gram-, CWindow-, Skip-gram-and CBOW-based CGED systems (all take error-prone words into account).

Official Evaluation Results
Three runs (NCTU+NTUT-Run1, Run2 and Run3) were submitted to NLP-TEA 2016 CGED shared task for official evaluation. All submissions are CWindow-based systems, since CWindow achieved the best performance in preliminary experiments. The only difference between these three runs is that they have different FA performance (i.e., different operating points). Table 5 shows the official evaluation results of our three submissions. Among three submissions, Run1 has the lowest FA and highest precision rate in all three measurements comparing with other participants. Especially, Run1 achieved 0.1362 FA, 0.4603 accuracy, 0.2542 precision and 0.0483 recall rate in position-level. Since FA is the most important factor that influences users' experiences on CGED applications, the proposed approach is quite promising.

Conclusion
In this paper, a word order sensitive embedding features/CRF-based CGED system was proposed and implemented for participating the NLP-TEA-3&CGED shared task. The experimental results showed that the proposed new features did work better than the traditional word order insensitive Word2Vec approaches. Moreover, according to the official evaluation results, our system achieved the lowest FA (0.1342) and the highest precision rates in all three measurements among all participants. Therefore, the proposed approach is a promising one and will be further explored in the near future. Finally, the latest version of our traditional Chinese parser is available on-line at http://parser.speech.cm.nctu.edu.tw.