{'fraction_non_alphanumeric': 0.038074000660720184, 'fraction_numerical': 0.015279154278163199, 'mean_word_length': 4.564797794117647, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Unified Multi-Criteria Chinese Word Segmentation with BERT

Multi-Criteria Chinese Word Segmentation (MCCWS) aims at finding word boundaries in a Chinese sentence composed of continuous characters while multiple segmentation criteria exist. The unified framework has been widely used in MCCWS and shows its effectiveness. Besides, the pre-trained BERT language model has been also introduced into the MCCWS task in a multi-task learning framework. In this paper, we combine the superiority of the unified framework and pretrained language model, and propose a unified MCCWS model based on BERT. Moreover, we augment the unified BERT-based MCCWS model with the bigram features and an auxiliary criterion classification task. Experiments on eight datasets with diverse criteria demonstrate that our methods could achieve new state-of-the-art results for MCCWS.


Introduction
Chinese Word Segmentation (CWS) is a fundamental task in Chinese Natural Language Processing (NLP), which aims to identify word boundaries in a Chinese sentence consisting of continuous characters. Most approaches transform the CWS problem into a character-based sequence labeling problem, in which each character is assigned a label to denote its position in the target word (Xue, 2003;Zheng et al., 2013;Chen et al., 2015a,b;Ma et al., 2018;. Recently, the CWS task of multiple segmentation criteria is studied, which can be formulated as Multi-Criteria Chinese Word Segmentation (MCCWS) task (Chen et al., 2017;He et al., 2019;Gong et al., 2019;. In MCCWS, a sentence can be segmented into different word units in different criteria. An example of MCCWS is showed in Table 1.
Existing MCCWS methods can be divided into  two frameworks: the multi-task learning framework and the unified framework. The multi-task learning framework treats MCCWS as a multi-task learning problem (Caruana, 1997), with each criterion as a single task (Chen et al., 2017;. A shared layer is used to obtain the common features of all criteria, and a private layer of each criterion obtains the criterion-specific features. The multi-task learning framework can learn shared knowledge but not efficiently, which suffers from high model complexity and complex training algorithm. The unified framework employs a fully shared model for all criteria in MCCWS, and the criterion is fed as input (He et al., 2019;Gong et al., 2019;. The unified framework is more concise, in which the shared knowledge are fully used and training process is simplified, compared with the multi-task learning framework. Recently, the pre-trained BERT language model (Devlin et al., 2019) have showed its powerful capability to learn prior linguistic knowledge, which have been proved beneficial for many NLP tasks.  also introduced BERT to MC-CWS in the multi-task learning framework and achieved excellent performance. However, due to the powerful ability of BERT, it is unnecessary to use multiple criterion-specific projection layers, which motivates us to explore BERT for MCCWS in the more concise unified framework.
In this paper, we propose a BERT-based unified MCCWS model, to incorporate the superiority of unified framework and pre-trained language model for the first time. Besides, the bigram features have been proved to be effective for CWS (Chen et al., 2015b;. Thus, a fusion layer and an attention layer are further employed to integrate the bigram features and the output of BERT. To ensure the unified model retrains the criterion information, an auxiliary criterion classification task is introduced to further boosts the performance. Different to the previous work, our model does not use the CRF (Lafferty et al., 2001) inference layer, thereby being more efficient in inference phase by fully utilizing the ability of parallel computation. Experiments show that our model can achieve new state-of-the-art results on eight standard datasets.

Methodology
As previous work (Chen et al., 2017;, the MCCWS task is viewed as a character-based sequence labeling problem. Specifically, given a Chinese sentence X = {x 1 , x 2 , ..., x T } composed of continuous characters and a criteria c, the model should output a CWS label sequence Y = {y 1 , y 2 , ..., y T } with y t ∈ {B, M, E, S}, denoting the beginning, middle, end of a word, or a word as single character. In our model, we first augment the input sentence by inserting a specific criterion token before it, and encode the augmented sentence using the BERT model (Devlin et al., 2019). Besides, we convert character-based bigram features into bigram embedding. Then, we use a fusion gate mechanism to integrate the hidden representation from BERT with the bigram embedding. A multi-head attention is followed to interchange context information among the fused representations. Finally, a multi-layer Perceptron (MLP) decoder is used to decode the segmentation labels. Besides, an auxiliary criterion classifier is employed to reconstruct the criterion class from the input. The overall architecture of our model is displayed in Figure 1.

Augmented Input Sentence
We add a specific criterion token before the original input sentence X to obtain the augmented sentence X = {x 0 , x 1 , .., x T }. For example, we add the token <pku> before the original sentence indicating that it adheres to the PKU criterion.

Encoder
BERT BERT (Devlin et al.,   2017) based bidirectional language model pretrained on a large-scale unsupervised corpus. We employ BERT as our basic encoder for MCCWS to introduce prior linguistic knowledge, converting the augmented sentence to hidden character representations: where H ∈ R (T +1)×d h .
Bigram Embedding The bigram features have proved beneficial for MCCWS (Chen et al., 2017;He et al., 2019;. Therefore, we construct the bigram feature for every character by concatenating it with the previous character, Then, we convert the bigram features to bigram embedding vectors by looking them up in an embedding table where E ∈ R T ×de .

Fusion Layer
We use a fusion gate mechanism to integrate the hidden character representations and the bigram embedding representations. We refer to the t-th hidden character vector as h t ∈ R d h , the t-th bigram embedding as e t ∈ R de . The fusion gate mechanism can be formulated as follows, where h t , e t , g t , f t ∈ R d h , g t is the fusion gate vector, f t is the fused vector, and σ is the sigmoid function.
Multi-Head Attention Next, we employ a multihead attention (Vaswani et al., 2017) to obtain the contextual representations. It is necessary to contextualize the fused representations because the fusion layer can only integrate the unigram and bigram representations character-wise, lacking the knowledge in the context. The multi-head attention with residual connection (He et al., 2016) and layer normalization (Ba et al., 2016) can be formulated as where F ∈ R T ×d h is the fused representations and O ∈ R T ×d h is the final output representations from the encoder.

CWS Label Decoder
The output representations are converted into the probabilities over the CWS labels by an MLP layer, where W o ∈ R 4×d h . P (y t |X, c) means the probability that the t-th CWS label is y t , and P (Y |X, c) means the probability of the label sequence Y , given the input sentence X and the criterion c.

Auxiliary Criterion Classifier
To avoid the criterion information lost in forward propagation, we add an auxiliary criterion classifier to reconstruct the criterion from the hidden representations. The probability of reconstructing criterion c is where W ∈ R C×d h and C is the number of criteria.

Loss
We use the negative log likelihood objective as our loss function, and add the losses for CWS labeling and criterion classification directly log P (Yi|Xi, ci) + log P (ci|Xi, ci), where N is the number of training samples.

Datasets
We experiment on eight CWS datasets from SIGHAN2005 (Emerson, 2005) and SIGHAN2008 (Jin and Chen, 2008), among which MSRA, PKU, CTB, NCC and SXU are simplified Chinese datasets, while AS, CKIP and CITYU are traditional Chinese datasets. As previous work (He et al., 2019;, we convert all traditional Chinese datasets into simplified Chinese using the OpenCC library 1 . All datasets are preprocessed by converting all full-width digits, punctuation and Latin letters to half-width, and replacing the continuous English characters and digits with a specific token respectively.

Settings
We use the Chinese BERT model pre-trained with whole word masking (BERT-wwm) (Cui et al., 2019), whose number of layers is 12, number of heads if 12, hidden size is d h = 768. We use the bottom 6 layers as our BERT model for the balance of performance and speed. The size of bigram embeddings is set to be d e = 100 and the bigram embeddings are pre-trained on the Chinese Wikipedia corpus as . The dropout probability of all hidden layers is set to be 0.1. The AdamW (Loshchilov and Hutter, 2019) optimizer is used in our fine-tuning process, with β 1 = 0.9, β 2 = 0.999 and weight decay of 0.01. The initial learning rate is 2e-5, and a linear warmup of ratio 0.1 is used to adjust the learning rate dynamically. We set the number of epochs to 10 and the batch size to 64. Our model is implemented using the fastNLP library 2 . Table 2 shows results of on the test sets of eight CWS dataset, with F1 values as metrics. The results are displayed in four blocks.

Overall Results
The first block shows the single criterion methods for CWS. Bi-LSTM (Chen et al., 2017) is the baseline method trained on each dataset separately.
The second block shows the multi-task learning framework methods. Adversarial (Chen et al., 2017) constructs a multi-task model which adopts an adversarial strategy to learn more criteriainvariant representations.
Multi-task BERT  uses pre-trained BERT as the sentence encoder.
The third block shows the unified framework methods. Unified Bi-LSTM (He et al., 2016) introduces two artificial tokens at the beginning and    (Vaswani et al., 2017) as the basic encoder. The fourth block shows our methods for MC-CWS. Unified BERT is the complete model discussed in Section 2, which incorporates the unified framework and the pre-trained BERT model. It is also augmented by the fused bigram features and the auxiliary criterion classification task. Besides, -Bigram is the Unified BERT model without bigram features, while the -CLS model is the model without criterion classification and -CLS -Bigram is the model without bigram features and criterion classification.
From Table 2, we could see that our proposed methods outperform previous methods obviously on nearly all datasets, especially two previous best methods: BERT based multi-task method Multitask BERT and unified method Transformer. These promotions should be attributed to the prior knowledge introduced by BERT and the shared knowledge captured by the unified framework. Besides, we could see that the complete Unified BERT model outperforms the model without bigram features (-Bigram) and the model without auxiliary criterion classification (-CLS), which shows the benefit of bigram features and criterion classification. The complete model obtains the best average performance and achieve new state-of-theart results for MCCWS, showing the effectiveness of our method.

OOV Words
OOV words means Out-of-Vocabulary words. According to previous work (Ma et al., 2018;, OOV error is a critical contribution to the total error of CWS. We use OOV recall as metrics to evaluate the performance on OOV words. The results are showed in Table 3.
We can see that our proposed method can achieve best OOV performance on five datasets, and comparable performance on the other three datasets. The results demonstrate the effectiveness of our method on OOV words.

Conclusion
In this paper, we mainly focus on the Multi-Criteria Chinese Word Segmentation (MCCWS) task. We propose a new method which incorporates the superiority of the unified framework and the pre-trained BERT model. Augmented with bigram features and an auxiliary criterion classification task, we achieve the new state-of-the-art results for MCCWS.