{'fraction_non_alphanumeric': 0.027188998233661367, 'fraction_numerical': 0.007317688619732526, 'mean_word_length': 5.03004944845949, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Unsupervised Chinese Word Segmentation with BERT Oriented Probing and Transformation

Word Segmentation is a fundamental step for understanding Chinese language. Previous neural approaches for unsupervised Chinese Word Segmentation (CWS) only exploits shallow semantic information, which can miss important context. Large scale Pre-trained language models (PLM) have achieved great success in many areas because of its ability to capture the deep contextual semantic relation. In this paper, we propose to take advantage of the deep semantic information embedded in PLM (e.g., BERT) with a self-training manner, which iteratively probes and transforms the semantic information in PLM into explicit word segmentation ability. Extensive experiment results show that our proposed approach achieves state-of-the-art F1 score on two CWS benchmark datasets.


Introduction
There exist many low resource fields and languages where labeled word segmentation is inaccessible, which makes unsupervised word segmentation desirable. Previous unsupervised word segmentation methods mainly apply statistical models to either evaluate the quality of possible segmented sequence with discriminative models (e.g., Mutual Information (Chang and Lin, 2003)) or estimate the generative probabilities with generative models (e.g., Hidden Markov Model (Chen et al., 2014)). However, these statistical methods can only make use of the limited contextual information, thus yielding less competitive performance.
With the thrive of neural networks, researchers have applied neural models for unsupervised word segmentation. Sun and Deng (2018) propose a segmental language model (SLM) to estimate the generative probability with recurrent networks. Although SLM can exploit more contextual information compared with statistical models, it is still weak in modeling deep semantic information, limited by its model capacity and training data scale.
Pre-trained language models trained on large scale data have shown superior ability to model contextual information, and achieve great success in various tasks (Peters et al., 2018;Devlin et al., 2019;Radford et al., 2019). Inspired by the attempt for interpreting BERT (Wu et al., 2020), we propose to take advantage of the semantic representation ability of BERT to evaluate the closeness between characters in a probing manner. To be more specific, we assume that the difference between masking one character and masking several adjacent characters as a whole reveals the closeness between that character and the adjacent ones.
Although this probing-based method can take advantage of the large amount of knowledge embedded in BERT, it only implicitly exploits the representation ability of BERT. To transfer the implicit knowledge into explicit segmentation boundary, we propose to apply a self-training method that transforms the segmentation decision from generative methods with high confidence into traditional "BI" sequence labeling system, which is then treated as the supervision signals for a discriminative model.
To combine the advantage of both generative and discriminative models, we propose to iteratively train the discriminative model and generative model under the supervision signal from their counterparts. To select the model with the best performance in the unsupervised setting, we propose an evaluation module that evaluates the quality of the word boundaries with masked prediction accuracy based on the assumption that the closer two characters are, the bigger loss masking one adjacent character would bring.
We conduct experiments on two Chinese Word Segmentation benchmark datasets in an unsuper-vised manner. Experiment results show that our method can outperform the strong baseline models and achieve state-of-the-art results in unsupervised CWS. Extensive analysis shows the effectiveness of the proposed modules.
We conclude our contributions as follows: • We propose an unsupervised word segmentation method that segments tokens by probing and transforming PLM with generative and discriminative modules, which are trained in a mutual promotion manner and selected for inference with an evaluation module.
• Experiment results show that our proposed method achieves the state-of-the-art result in unsupervised CWS. Extensive analysis testifies the effectiveness of the proposed modules.

Related Work
Previous unsupervised word segmentation methods can be roughly classified as generative and discriminative two ways. Generative models focus on finding the segmented sequence with the highest posterior probability. Hierarchical Dirichlet process (HDP) model (Goldwater et al., 2009), Nested Pit-manYor process (NPY) (Mochihashi et al., 2009), Hidden Markov Model (HMM) (Chen et al., 2014) and SLM (Sun and Deng, 2018) are all different ways to estimate the generative probabilities for segmented sequences. On the other hand, discriminative models focus on designing a measure to evaluate the segmented sequences. Mutual Information (MI) (Chang and Lin, 2003), normalized Variation of Branching Entropy (nVBE) (Magistry and Sagot, 2012) and ESA (Wang et al., 2011) apply co-occurrence based measurement to evaluate the segmented sequences.

Approach
In this section, we describe our BERT oriented probing and transformation based unsupervised word segmentation approach. Our model mainly consists of three parts, a generative module that suggests the plausible word boundaries by probing BERT, a discriminative module that transforms the implicit boundary information into explicit sequence labels, and an evaluation module that estimates the performance of the model in an unsupervised manner.

Algorithm 1 Unsupervised Word Segmentation Procedure
Require: Generative Module G, Discriminative Module D, Evaluation Module E, sequences to be segmented X. i = 0 while True do Segment the sequences X with G into X g Transform the segmented X g into "BI" labels Train D with high confident segmentations in

Overview
Because our method works in an unsupervised manner, we propose to get the original word boundary information by probing BERT, which reveals the word boundaries by measuring the distance between masking a span and masking a token using the generative module. This distance reflects the closeness between the masked token and the masked span (separately). Then the discriminative module transforms the word boundaries suggested by the generative module into explicit segmentation labels to enable the self-training process. To combine the advantages of both generative and discriminative modules, two modules are iteratively trained with the word boundaries suggested by the updated counterpart with high confidence. To decide when to stop this iterative self-training procedure, an evaluation module is proposed to evaluate the segmented sequence, which early stops the iterative process with the model parameters that yields the best performance.

Generative Module
The proposed generative module works by probing a pre-trained language model (e.g., BERT) with masks on tokens. Assume the input sequence to be [x 1 , x 2 , · · · , x n ]. We first mask one token at a time in order. The representation at i-th position given by BERT after masking x i is H i . Then we mask two successive tokens at a time in order. H i,j is the representation given by BERT at i-th position after masking both x i and x j . Note that it is different for the representation at j-th position after masking both x i and x j , which we denote as H j,i .
The intuition behind the generative model is that we assume if two tokens x i , x j are inherently close and should be combined as a word, the difference between masking i-th and j-th token together and solely masking i-th token should be large, which is reflected by the probing distance d, On the contrary, if two tokens are loosely connected, d should be small. This assumption follows the intuition that if x i is largely dependent on x j , masking x j should bring a relatively big influence on the representation. This indicator is applied to segment token sequence with a threshold, that is to say, if d ≥ threshold, we combine the two tokens x i and x j , if d ≤ threshold, we segment x i and x j .

Discriminative Module
The generative module can only exploit the implicit segmentation revealed by BERT. Furthermore, it is not very friendly when the word length gets longer. To overcome these drawbacks, we propose to transform the segmentation information provided by the generative module with high confidence into traditional supervised sequence labeling scheme with "BI" labels, which indicates the role (position) of the token to be "beginning" ("B") or "inside" ("I") of a word.
We train the discriminative module by finetuning BERT on the transformed sequence labels with an additional output layer projecting the representation into "BI" labels. Since the results given by the generative module can be noisy, we only adopt the combined words with relatively high confidence, which is realized by strict thresholds for the generative module. If d ≥ threshold l , we combine the two tokens x i and x j , if d ≤ threshold h , we segment x i and x j . threshold l indicates lower bound, threshold h indicates higher bound.

Iterative Training and Evaluation Module
We assume that the generative module and the discriminative module can capture segmentation information from different aspects. Therefore, we propose a self-training procedure, which promotes both the generative module and the discriminative module by making them learn from the high confident predictions of the counterpart.
To make the generative module learn from the discriminative module, we design a Euclidean distance based MSE loss function loss generative = ∥d − threshold∥ 2 to push the distance between two tokens predicted to be in the same word to be larger than a threshold and vice versa. The loss is activated only when the generative module makes different predictions from the discriminative module.
To prevent the self-training procedure from being over-fitting, we propose to keep the MLM objective while fine-tuning on the word segmentation objectives, and early stop the training with an evaluation module. The intuition behind the evaluation module is that predicting a masked token with the token inside the same word is much easier than predicting this masked token with the token outside that word. Formally, let the cross-entropy of predicting the i-th token x i with the masked language modeling ability of BERT when masking two adjacent tokens x i,j be CE i,j , we assume that if x i,i+1 rather than x i−1,i belongs to the same word, because x i+1 provides more information for prediction when masking x i−1,i .
We apply this principle to inspect the segmentation results from either the discriminative module or the generative module. When the evaluation module detects performance decline, the training procedure stops, and the discriminative module with the best performance is used as the final word segmentation model.

Experiment
In this section, we show the results and analysis on two CWS benchmark datasets, PKU and MSR for a fair comparison, which are provided by the Second Segmentation Bake-off (SIGHAN 2005) (Emerson, 2005. There are 104K and 107K words in the test set of PKU and MSR datasets respectively.

F1-score
Evaluation score F1-score Figure 1: The relation between evaluation score and F1 score on the development set. The evaluation score shows good coherence with F1 score. We select the model with best evaluation score, which also achieves the best F1 score on the development set.
released by Huggingface. 2 The tokenizer tokenizes the sentence into Chinese characters, which involves with no word (segmentation) information. We randomly initialize the discriminative module, which is trained for 2 epochs using sequence labels transformed from the generative module with high confidence. threshold l is 8 and threshold h is 12. We use AdamW (Loshchilov and Hutter, 2019) optimizer with the learning rate of 1e-4.

Results
In Table 1 we show the F1 score on PKU and MSR. From the results, we can see that our model yields much better results than the previous models and achieves state-of-the-art results. We assume the reason behind is that our model can take advantage of the large pre-trained language model, which encodes abundant language matching knowledge and can better model the context with big model capac-  ity. Moreover, we can observe that the neural-based model SLM does not outperform the traditional statistical Joint method, but gives better results than other traditional generative models. This indicates that combining generative and discriminative methods can benefit the results. Moreover, our model does not need to constrain the longest word length compared with SLM-2, SLM-3, etc., which provides more flexibility. This is achieved by introducing the discriminative module, which segments the words under the sequence labeling scheme.

Ablation Study
In Table 2 we show the results for removing the designed modules. "Generative only" means we only use the generative module described in section 3.2, where a hard threshold of 10 is used to decide the word boundary. "+Discriminative" means we use the discriminative module after learning from the generative module described in section 3.3 without iterative training and mlm loss. From the results, we can see that revealing the implicit word boundary information by probing BERT can only provide performance comparable to traditional statistical models. Transforming the implicit knowledge into explicit segmentation labels (+Discriminative) can give big promotion, which makes better use of the big amount of semantic knowledge encoded in PLM. Moreover, the proposed iterative training process and mlm loss further help improve the overall performance by combining the advantages of both generative and discriminative modules. Effect of Evaluation Module In Figure 1, we show the relation between the evaluation score described in section 3.4 and the development F1 score.
We can see that the model with the best evaluation score achieves the best F1 score in the development set, and it generally coordinates with the variation trend of the F1 score, which makes the evaluation score a reasonable indicator to select the best model in the unsupervised setting.

Case Study
In Table 3 we show a concrete example of the segmentation results of SLM and our proposed method. Both two methods basically give correct word segments. The disagreement mainly lies in "送交市 政府" (give to the city government). Compared with other words, " 送交" can be relatively rare and bears very similar meaning with the single character "送", which makes SLM wrongly segment "送交" apart. On the contrary, our method is built based on BERT trained on a large corpus, which makes our model able to recognize these relatively rare words. For the part "市政府", where our model chooses to split, we assume that this is because similar contexts are often seen such as " 北京市" (Beijing City), where "市" should be separated from "政府" (government). Furthermore, separating "市政府" into two words does not affect the understanding of the original text, and is more dependent on the segmentation fineness.

Conclusion
In this paper, we propose a BERT oriented Probing and Transformation method for unsupervised Word Segmentation. Our proposed model reveals the semantic information encoded in PLM into word boundary information by probing and transforming the token representations into explicit sequence labels. Experiment results on two benchmark CWS datasets show that our method achieves state-ofthe-art F1 score. The proposed method works in an unsupervised manner, which can help understand low resource and endangered languages and thus protecting language diversity.