{'fraction_non_alphanumeric': 0.028912674585563243, 'fraction_numerical': 0.006852891267458556, 'mean_word_length': 4.971551052221356, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Domain Adaptive Chinese Word Segmentation Method for power data security classification

Chinese word segmentation (CWS) is an important task for Chinese NLP, and also an essential pre-processing step to establish a word-root database for security classification of power data, covering different domains such as laws & regulations, power. It is impracticable to label a large number of training corpus for each domain, which brings great challenge to the supervised statistical learning method to carry out effective CWS. Therefore, a Chinese word segmentation approach based on dictionary and semi-supervised conditional random field (SS-CRF ) is presented. At first, a CRF model for CWS is trained with self-training and active learning algorithms and used to conduct CWS task. Then the dictionary features are introduced to correct the result of CRF based segmentation by adopting RMM algorithm. Experiments on a cross domain segmentation task show that the proposed method can effectively improve the domain-adaptive performance of CWS.


Introduction
With increasing development of the construction of the power Internet of things, how to ensure data security is particularly important. At present, research on encryption, desensitization and leakage prevention of power data security has been carried out at home and abroad, but the data security classification technology has not been developed in depth, and the research on the term root of power data security by using NLP technology is also relatively insufficient. However, CWS is the basis for the establishment of term-root database, involving several different domains, such as laws and regulations, power business.
Word segmentation is an essential fundamental task in Natural Language Processing (NLP) for those languages without word delimiters, e.g., Chinese [1]. In recent years, there has been a lot of work in the research field of CWS. Existing approaches for CWS roughly fall into three categories: dictionary-based methods, sequence modeling methods with statistical techniques and neural-network-based methods. It is difficult to deal with the changes in language usages for the dictionary-based method due to the bottleneck brought by the expansion of dictionaries. Statistical methods such as Hidden Markov Model (HMM) [2]and conditional random field (CRF) [3], are reported to achieve high accuracy with largescale annotated dataset for they can easily learn new words from corpora. In these CWS methods, a key problem is to build the feature vector for each character in sentences, which is constructed via a manual engineering and so is very time-consuming and hard-work. Besides, human annotated corpora of special domains are of resource-intensive. Therefore, the performance of this kind of method is practically not satisfying in domain-specific system. In order to improve the performance of domain adaptation for [11]. Some research reports that these neural networks show better performance than popular statistical algorithms. However, these methods usually depend on numerous labeled sentences. It is very difficult to correctly segment the sentences containing these words which are scarce or absent in training data by using these methods.
To deal with the practical case where labeled data is limited but considerable unlabeled data is readily available, this paper combines the semi-supervised statistical method with the dictionary method to improves the adaptation performance with the help of a small amount of labelled data. Firstly, a semisupervised CRF word segmentation model incorporated with active learning algorithm is trained with the labeled and unlabeled data, and then this CWS model is applied to test the target text to yield a segmentation result. At last, the dictionary-based maximum inverse matching method is applied to perform a second word segmentation with the help of domain dictionaries to raise the recognition accuracy of professional terms. Empirical studies show that the method proposed in this paper is able to effectively improve the adaptation performance of CWS, especially when the labeled training data is insufficient. Our contributions can be summarized as follows: (1) We combine the semi-supervised CRF and the dictionary-based method for improving the domain-adaption performance of CWS in case of insufficient annotated corpora in specific domain.
(2) We integrate active learning algorithm into self-training algorithm of the semi-supervised CRF for better performance.

Conditional random fields (CRF)
CRF is a probabilistic framework and also a type of discriminative model, where the parameters are estimated by maximizing the joint probability over observation and label sequences. CRF has been studied in depth and performed well for sequence labeling task since it was first proposed by (John Lafferty,2001). Linear-chain CRF is the typical type of the CRF models and most widely used in machine learning tasks, in which a conditional probability for a labeling sequence Y=(y1, y2,…, yn) given in an input sequence X=(x1, x2,…, xn) is defined as : is a feature function, k  is a learned weight associated with the feature function and ZX is the normalization factor over all state sequence, which is calculated according to the following formula: For an input X, the most probable label sequence can be efficiently determined by adopting the Viterbi algorithm.
Where U(θ) is the regularization term, being used to prevent over-fitting; α is a hyper-parameter which is used to balance the proportion between the logarithmic likelihood term and the regularization term.
The parameter optimization in (3) is usually conducted by using L-BFGS algorithm, which simplifies the computing complexity and alleviates memory burden through the approximation of BFGS algorithm.

Self-training algorithm and active learning algorithm
Semi-supervised learning is a statistical machine learning method which combines supervised and unsupervised learning. It can train and classify a large number of unlabeled corpus and a small number of labeled corpus without manual intervention. Therefore, in case of small amount of labeled corpus available, semi-supervised learning is a good approach for obtaining a model with strong generalization ability. As the simplest but effective semi-supervised learning algorithm, self-training algorithm is adopted in this paper. However, these pseudo labels may contain quite some wrong labels, which can impair the performance of subsequent training procedures due to accumulative errors.
On the other hand, active learning algorithm is to actively select unlabeled sample labels which are beneficial to model training so as to reduce the labeling cost and the computational scale as much as possible. It has been successfully applied for NLP task. Therefore, the active learning algorithm is incorporated into semi-supervised learning with self-training technique to select high-quality pseudo labels in this paper.

Formatting author affiliations Semi-supervised CRF learning
In this paper, the samples that cannot be labeled correctly by the pre-training model are called positive data and stored in the unlabeled data set. Here, the positive data will be selected according to confidence value. In CRF model, the conditional probability P(Y1X) is smaller, the probability of achieving the optimal solution Y given observation sequence X is smaller. Therefore, P(Y1X) can represent the confidence score of labeling results. Here multiple groups of data are selected as positive data, whose P(Y1X) is smaller than the specified threshold.
The flowchart is shown in Figure 1 for the semi-supervised CRF learning algorithm proposed in this paper. The procedure for learning the CRF model is as follows: Input: labeled training sample set L and unlabeled training sample set U.
Step1: Obtain a small number of labeled corpus as L; Step2: Train CRF by using L to achieve the model Ck; Step3: Tag U with Model Ck and estimate the confidence score P (k) (YU1XU); Step4: Select multiple groups of data with confidence score lower than the set threshold in U as positive data and mark as UP; Step5: Label UP manually and then mark them as u; Step6: Add u to L and delete u from U; Step7: Iterate the above process for K times until the convergence condition is satisfied. Here the convergence condition is set as:

Domain adaption CWS method
The matching methods based on dictionary have achieved great success in CWS. Compared with the word segmentation corpus used in the statistical model, dictionaries are easier to obtain. But the previous methods based on dictionary perform worse than statistical model in solving the ambiguity problem of word segmentation. Therefore, the statistical CWS model and the dictionary are combined to improve the accuracy and domain adaptability of CWS in this paper.

Tag Set and Feature Template
CRF is applied for CWS by conducting a sequence labeling task, that is, labeling the position of each Chinese character in word which the character belongs to. This paper used the 4-tag set [3]. A context of three characters for feature generation and feature templates represented in Table 1 are adopted in this paper, which are called as basic features.

Feature representation of dictionary information
In order to represent features of dictionary information in CRF model, for a given sentence X= (x1, x2,…, xn) and the dictionary D, two functions about the jth character cj (1≤ j≤ n) are defined as follows: Where W denotes a word. fE(X, j, D) represents the maximum length of a word which is obtained at the position j in a sentence X by using reverse maximum matching (RMM) according to dictionary D. fM(X, j, D) represents the maximum length of the word including the character Cj but not ending with Cj and ending with Cs (s > j), which is obtained in a sentence X by using RMM according to D.
The extended features related to dictionary D introduced in this paper are shown in Table 2. Assuming the current position is j, [fE]i = fE(X, j+i, D), [fM]i = fM(X, j+i, D). For example, a sentence is "供电服务质量好", if the words related to this sentence in the dictionary include "供电","服务","供电 服务","质量","服务质量", "好", and the current character is "务", then

Domain adaption CWS
The following Figure 2 demonstrates the framework of domain-adaption-CWS method which introduces the domain dictionary into the proposed semi-supervised CRF model. Firstly, the training and test corpus are preprocessed and then feature selection and extraction are carried out. Secondly, the semi-supervised learning method described above is employed to learn CRF model with training corpus so as to generate an optimal CRF model, in which the L-BFGS algorithm is employed for parameter estimation of the CRF. Thirdly, the CRF model is used to segment words of the test corpus with the Viterbi algorithm.

Experimental Settings
The labeled training corpus used in this paper is PKU corpus provided by SIGHAN CWS bake-off 2005, which is the news corpus of People's Daily. The size of the corpus is 7548kb. Since there is no authoritative and unified corpus of laws and regulations on network security and electric power business, 200 relevant documents are collected manually, such as "Network security law of the people's Republic of China", "Provisions on the confidential scope of State Grid Corporation of China", "Provisions on classified scope of State Grid Corporation of China", with a total of 63058 words. Then, Antconc 3.5.7 is used to make the unlabeled corpus based on the collected texts. The domain dictionary includes Chinese law dictionary and power term one. By implementing the new-word-recognition algorithm proposed in reference [12], the new words are mined in the test corpus, and then the top 300 words are picked up manually to update the domain dictionary. CRF++-0.58, a famous CRF framework written in C++, is implemented to carry out training and inference work in this paper. Precision (P), recall (R) and F1-score (F1) are used as the evaluation metric of our model. Three groups of experiments were conducted to verify the effectiveness of the proposed CWS method.
Test 1: The CRF model for CWS was trained by using this method presented in this paper, where the threshold value of confidence is 25%, the data with confidence lower than 25% are manually annotated and then added to the labeled training set for the next iteration. Then the CRF model is used to segment the test text.
Test 2: The domain dictionary was introduced to correct the results achieved by Test1. Test 3: The semi-supervised CRF with self-training method was used to learn the training corpus iteratively. The samples with confidence higher than 85% were directly added to the training sample set for the next iteration until convergence. After the first CWS is completed by the CRF model, the domain dictionary is introduced for correction.

Experimental Results and Analyses
When the hyper-parameter α varies between 0.001 and 0.1, F1 in the network security laws and regulations corpus increases first and then decreases, and is maximal at α = 0.01; when α changes from 0.0001 to 0.01, F1 in the power business corpus shows the same changing trend, and reaches a maximum at α = 0.001. When α is the optimal value, the inference results in three experiments are shown in table 3. The following conclusions can be drawn from table 3: (1) Adding dictionary and integrating active learning algorithm into self-training semi-supervised learning are both able to improve the performance of CWS.
(2) In test 3, the CWS result achieved by using the semi-supervised method with self-learning algorithm is not good. Because network security and power engineering both have their unique text characteristics while the labeled corpus used by pre-training CRF model belong to the common domain, the accuracy of word segmentation in specific domain will be reduced. Besides, errors will be accumulated in the process of iterative learning, which will lead to poor performance of the classifier.
(3) In test 2, the method proposed in this paper is used to segment text, in which the data with low confidence are manually labeled to reduce the accumulation of errors in the process of iterative learning. The test result shows that the semi-supervised learning method proposed in this paper can work better on CWS in specific domain.

Conclusions
This paper proposes a semi-supervised CRF model combining self-training learning with active learning, which can improve the performance of cross-domain CWS. In process of training CRF model with the self-training algorithm, the data with low confidence are manually annotated and added to the labeled samples to reduce the accumulation of errors of iterative learning. Besides, in order to further improve the domain adaptability of CWS, the dictionary features are integrated into CRF model to correct the CRF segmentation results by adopting RMM algorithm. Empirical studies show that the proposed method can effectively improve the domain-adaptive performance of CWS, especially when the labeled training data is insufficient. In our future work, we would like to automatically mine various domain related words to expand domain dictionary so that our word segmentation approach can adapt to various domains.