{'fraction_non_alphanumeric': 0.034514042284632375, 'fraction_numerical': 0.008756705585358158, 'mean_word_length': 4.875550405561993, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Two Languages Are Better than One: Bilingual Enhancement for Chinese Named Entity Recognition

Chinese Named Entity Recognition (NER) has continued to attract research attention. However, most existing studies only explore the internal features of the Chinese language but neglect other lingual modal features. Actually, as another modal knowledge of the Chinese language, English contains rich prompts about entities that can potentially be applied to improve the performance of Chinese NER. Therefore, in this study, we explore the bilingual enhancement for Chinese NER and propose a unified bilingual interaction module called the Adapted Cross-Transformers with Global Sparse Attention (ACT-S) to capture the interaction of bilingual information. We utilize a model built upon several different ACT-Ss to integrate the rich English information into the Chinese representation. Moreover, our model can learn the interaction of information between bilinguals (inter-features) and the dependency information within Chinese (intra-features). Compared with existing Chinese NER methods, our proposed model can better handle entities with complex structures. The English text that enhances the model is automatically generated by machine translation, avoiding high labour costs. Experimental results on four well-known benchmark datasets demonstrate the effectiveness and robustness of our proposed model.


Introduction
"One language sets you in a corridor for life. Two languages open every door along the way." -Frank Smith, Psycholinguist Named entity recognition (NER) is the task of determining spans and semantic categories of named entities such as organization (ORG), person (PER) and location (LOC) in given free text. As the cornerstone of a wide range of natural language processing tasks, NER plays an essential role in many downstream tasks, such as relation extraction (Ze- lenko et al., 2003) and question answer (Diefenbach et al., 2018).
Compared with English NER, Chinese NER meets a series of challenges caused by the characteristics of Chinese. Aside from the lack of natural word boundary information, Chinese named entities usually vary significantly in length and have complex compositional structures (Dong et al., 2016). At the same time, the annotated data of Chinese NER is relatively scarce, and it is difficult and costly to annotate the data manually (Liu et al., 2022). Hence, without more annotated data, it is a promising approach to improve Chinese NER by leveraging external information resources, which has attracted more and more research attention.
One way to utilize external resources is to perform Chinese NER with bilingual constraints. Previous works  have demonstrated that the joint use of bilingual information in Chinese and English can significantly improve performance in the Chinese NER task. In addition, incorporating vocabulary knowledge has also become a promising solution (Zhang and Yang, 2018;Mengge et al., 2020). What's more, several studies were proposed to exploit information from other modalities to supplement the representation of Chinese text Sun et al., 2021b;Sui et al., 2021). However, despite the success of the abovementioned methods in Chinese NER by introducing external information, these methods still have the following limitations. First, the external resources utilized by these methods are mainly obtained manually, which increases the cost significantly. Second, existing methods for boosting NER with bilingual constraints  rely on bilingual word alignment information and both Chinese and English sentences need to be manually annotated, which limits its usage. Third, as the state-of-the-art approaches based on deep neural networks for Chinese NER, lexical enhancement methods and multimodal methods still fail to effectively handle entities with complex composition structures, which, however, are frequently observed in Chinese NER tasks. The example in Figure 1 illustrates one of such dilemmas in Chinese NER. In this example, due to the complex component structure, the ORG entity "美 国'亚特兰蒂斯'号航天飞机(The US space shuttle Atlantis)" tends to be incorrectly labeled by the NER model as a LOC entity "美国(USA)" and an ORG entity "'亚特兰蒂斯'号航天飞机(Space shuttle Atlantis)". However, it is encouraging that the clues of the English expression, such as the preposition "on", will potentially alleviate this type of incorrect labeling that often occurs in Chinese NER tasks.
To address the above issues, in this work, we propose to boost the performance of Chinese NER with the unlabelled English text translated from the corresponding Chinese text. English texts are automatically generated through the publicly available neural machine translation API without any extra human labor all the way through. Besides, considering the Chinese texts and corresponding English translations as two different modalities, we perform bilingual enhancement of Chinese NER with multimodal NER approach. Furthermore, based on the fact that a word in the text will only be strongly correlated with a small fraction of words in the translated text, we propose a bilingual interaction enhancement model based on Adapted Cross-Transformers with Global Sparse Attention (short for ACT-S). The interaction of bilingual information as well as intra-linguistic interaction are taken into account in the our model.
The primary contributions of this work can be summarized as follows: (1) We improve the perfor-mance of Chinese NER by bilingual enhancement, based on the unlabeled translated English text automatically generated using the Neural Machine Translation API. To the best of our knowledge, this is the first end-to-end NER method that effectively exploits bilingual information.
(2) We further propose the neural module called Adapted Cross-Transformers with Global Sparse Attention to simultaneously model bilingual interactions and inter-lingual interactions. So far as we know, it is the first attempt to use global sparse attention mechanisms for multimodal information interaction. (3) Experimental results on four Chinese NER datasets show that our proposed model achieves superior performance to other strong baseline models.

Chinese NER with lexicon enhancement
In Chinese NER, a series of recent works focus on introducing lexical boundaries and semantic information by word matching. Zhang et al. (2018) proposed to introduce semantic and boundary information of the lexicon through the lattice structure in LSTM. Afterwards, some CNN-based NER lexical enhancement methods, such as LR-CNN (Gui et al., 2019a), were proposed. Graph neural networks have also been applied to Chinese NER word enhancement tasks, a typical one of which is LGN (Gui et al., 2019b). And Transformer-like encoders fusing lexical information are also used for Chinese NER tasks, including PLTE (Mengge et al., 2020) and FLAT . In addition, some work (Ma et al., 2020; has proposed to fuse lexical information into the word embedding representation instead of integrating word information into the model encoder. However, the lexicon enhanced Chinese NER approach still cannot effectively deal with entities with complex composition structures. And our approach utilizes bilingual clues that can alleviate this problem in Chinese NER.

Multimodal NER
In recent years, with the development of multimodal information processing technology, the multimodal NER has emerged. In the field of English information extraction, existing multimodal NER works Sun et al., 2021a; have focused on using image clues to improve NER on Twitter. As for Chinese NER, introducing information from other modalities has CN:美国"亚特兰蒂斯"号航 天飞机上的宇航员...

BERT-Chinese
EN:Astronauts on the US space shuttle Atlantis... also become a promising solution. On the one hand, the multimodal information of Chinese characters is used to mining the semantics in the structure of Chinese characters (Sun et al., 2021b;. On the other hand, multimodal information of the whole Chinese sentence, such as the audio content of the text (Sui et al., 2021), is also used to improve the word representation for Chinese NER. Compared with existing methods, the method we propose makes use of relative distance information while focusing on strongly correlated units during modal interactions.

Sequence labelling with bilingual clues or translation
Some previous works  have demonstrated that constraints in bilingual parallel annotated corpora can be used to improve NER performance in two languages. But different from them, our proposed approach can take advantage of the hints in the unannotated English texts which are automatically translated by Neural machine translation tools. In addition, in crosslingual sequence labelling tasks, translation methods (Mayhew et al., 2017;Fei et al., 2020;Zhen et al., 2021) are used to migrate annotation information from rich languages to low-resource languages. Unlike these previous works, the only resources used by our model in the additional language are texts with no annotation information. Furthermore, the model automatically learns all bilingual lexical alignment information with no assistance from any bilingual alignment tool.

Overall Architecture
Task Formulation: Given a Chinese text S c = (cw 1 , ..., cw i , ..., cw n ) and its corresponding English translated text E c = (ew 1 , ..., ew l , ..., ew m ) , where cw i represents the i-th Chinese character and ew l represents the l-th character of English translated text, the goal of the task is to utilize the information in the bilingual text to determine the spans and types of all named entities in the Chinese text. In this work, we formulate the task as a sequence labeling task. And the BMES (Beginning, Middle, End, Singleton) (Xue, 2003) tagging scheme is adopted. The architecture of our proposed model is shown in Figure 2. In our model, we absorb the inspiration from the unified multimodal Transformer encoder widely used in vision-language tasks (Tsai et al., 2019;. And similar to MECT , we introduce distance-aware and direction-aware components in Transformer attention. However, unlike previous works, we propose to use global sparse attention in the Cross-Transformer to reduce the noise in the information fusion process of the two modalities. Each part of the model is introduced in detail in the following sections.

Word Representations
Since many previous works have proven the effectiveness of the pre-trained language model in Named Entity Recognition task , we use BERT (Devlin et al., 2019) as our contextualized representation encoder for both Chinese text and English translated text. To fit the BERT encoding procedure, we add two special symbols [CLS] and [SEP] at the beginning and end of the input sentence respectively, and we discard the representation vectors of [CLS] and [SEP] at the end of the BERT encoding computation. If a word is tokenized into several subwords by the Byte Pair Encoding (BPE) algorithm used in BERT, we found empirically that the model performs better when the average pooling method is used to merge the representations of the subwords that belong to the same word into a single representation vector. Thus, we can obtain the word representation generated using BERT:

Adapted Cross-Transformer with Global Sparse Attention
This section presents our first proposed Adapted Cross-Transformer with Global Sparse Attention (ACT-S) for bilingual information interaction in detail. As illustrated in Figure 2(a), several parameter independent ACT-Ss are used in our model for both inter-and intra-language interactions between bilinguals. The implementation details of ACT-S are shown in Figure 2(b). Motivation: In multimodal tasks, the widely used cross-modal Transformers Sui et al., 2021; typically use Softmax to normalize the cross-modal attention distribution of each head. As a result, in multi-head cross-modal attention, each unit is represented by all the units in the other modality by multiple different weighted averages. However, it is essential that a word is only associated with a small number of words in the other language in bilingual enhanced Chinese NER. Based on the above observations, we propose for the first time to incorporate the global sparse attention mechanism in a cross-modal Transformer to learn bilingual interactive word representations, which exclude the interference of irrelevant words in the other language. To our knowledge, it is the first time that global sparse attention is used for a multimodal task.
For inputs X = (x 1 , ..., x u , ..., xn) and Y = (y 1 , ..., y v , ..., ym) , we treat X ∈ Rn ×d as queries, and Y ∈ Rm ×d as keys and values. The input Q, K, and V are obtained by linear transformation of X and Y : To provide the attention mechanism in ACT-S with the ability of both distance perception and direction perception, we adopt the component of sensing relative distances similar to that in MECT  in the attention matrix computation process: where u is the index of the word in the target language and v is the token in the other language, Q u , K v is the query vector and key vector of word x u , y v respectively , R u−v ∈ R d k is the relative position encoding, p in Eq. 5 is in the range 0, d k 2 , α ∈ R d k and β ∈ R d k are learnable parameters.
Different from MECT, we add a key bias term K (h) v R T u−v to attention matrix to represent the bias of v-th token in the key sequence on certain relative distance and we empirically found that models perform better with it. And in this work, we consider that words in the target language are only relevant to a small number of words in another language. To learn a better representation of the target language words guided by the relevant words in another language. For the first time, we introduce sparse prior information to the global attention distribution via the top-k mask operation T km(·), which is formulated as follows: Where k is a hyperparameter, masking constant C ≪ 0, A u,: = Q u,: . The Adapted Global Sparse Attention matrix for the h-th attention head is then calculated as: And the attention score is calculated as follows: where W o ∈ R d×d is a learnable parameter. Then, we stack the following sub-layers on top to obtain the text representation based on bilingual interaction:Ô = LN (X + Atten) where F F N is the feed-forward network, LN is the layer normalization. Both of these operations are consistent with those in vanilla Transformer (Vaswani et al., 2017).

Chinese representation with bilingual interaction
Given a Chinese text sequence C = (c 1 , c 2 , ..., c n ) and an English text sequence E = (e 1 , e 2 , ..., e m ), obtaining the Chinese representation of fused bilingual information requires two steps: inter-modal fusion and intra-modal fusion.
Inter-modal interaction: In order to learn a better representation of Chinese with the aid of information in the English text, we first employ ACT-Ss to get the English-Aware Chinese Representation. Similarly, to align each English word with its closely related Chinese characters, i.e., assigning high/low attention weights to its related/unrelated Chinese characters, we also use parameter independent ACT-S to gain the Chinese-Aware English Representation: where t ∈ {1, · · · , N } , N denotes the number of interactions between bilingual modalities,

Intra-modal interaction:
In order to learn the dependencies between Chinese words, we use another ACT-S to obtain the Intra-Chinese Interaction Representation: Then, we concatenate A = (a 1 , a 2 , ..., a n ) and J (N ) and input them into a fully connected layer to obtain the final hidden representations H = (h 1 , h 2 , ..., h n ): where ⊕ denotes the concatenation operation, W F and b are learnable parameters.
At last, we pass the final hidden representations H to a Conditional Random Field (CRF) (LAF-FERTY, 2001) module.

Datasets
We used four publicly available Chinese NER datasets, including Weibo NER (Peng and Dredze, 2015), Resume NER (Zhang and Yang, 2018), MSRA (Levow, 2006) and Ontonotes 4.0 (Weischedel et al., 2010). The corpus of MSRA and Ontonotes 4.0 comes from news, the corpus of Weibo comes from social media, and the corpus of Resume comes from the resume data in Sina Finance. The splitting methods and other pre-processing methods of datasets follow those in (Zhang and Yang, 2018;.

Text Translation
Neural machine translation (NMT) methods have achieved state-of-the-art performance for the translation of a wide range of language pairs (Vaswani et al., 2017). Therefore, automatic text translation with NMT is applicable. In this work, we employ Baidu Translation API 1 and Tencent Translation API 2 to automatically translate Chinese text into English, respectively, and both machine translation systems have achieved the highest BLUE score on the WMT Chinese-English task Wang et al., 2021b) in recent years.

Models
Resources Weibo Resume MSRA Ontonotes 4.0  ♦ marks results produced with BERT-wwm. ▲ marks results produced with BERT-base-Chinese. * marks the results implemented in the fastNLP 3 framework. 'L' denotes using the lexicon resources. 'RC' denotes the radical information of Chinese. 'T-T' denotes using the bilingual information from the Tencent Translation API and 'T-B' denotes using the bilingual information from the Baidu Translation API.

Baseline Methods
To demonstrate the effectiveness of our proposed model, we compare it with several strong baseline models for Chinese NER: (1)

Implement Details
The English word representations E are initialized with the cased BERT-base-English model pretrained by Devlin et al. (2019). In addition, to make the comparison with the results of the baseline models convincing, we use BERT-base-Chinese (Devlin et al., 2019) and BERT-wwm (Cui et al., 2021) to initialize the Chinese word representations C separately to get different experimental results for comparison and fine-tuned during training. All the neural models are implemented with PyTorch and fastNLP 3 . More implementation details are described in Appendix A.

Main Results
We compared our proposed method with the state of the art methods. The experimental results are reported in Table 1, which is divided into two blocks. The methods in the first block use the Chinese word representation from BERT-wwm. And the methods in the second block use the Chinese word representation from BERT-base-Chinese. Our model achieves a significant and consistent performance boost over current SOTA models on four Chinese NER datasets. From the results, we can observe that: (1) In comparison with the methods without external resources (BERT-Tagger, LSTM-CRF[BERT] and TENER[BERT]), our model achieves a significant performance boost. Because our model makes use of the rich prompt information in the English text that help to determine the boundaries and types of entities. It demonstrates the significant effect of introducing bilingual information compared to just using the internal features of Chinese.
(2) Compared with the lexicon enhancement method using pre-trained BERT-wwm Chinese representation, our model has superior performance, i.e., +2.25 , +0.88, +0.25 ,+1.34 on Weibo, Resume, MSRA, Ontonotes4.0, respectively. When compared with baselines with BERT-base-Chinese, the performance of our model is still be competitive, i.e., +1.8, +0.14, +0.44, +0.75 on Weibo, Resume, MSRA, Ontonotes4.0, respectively. This verifies our claim that, compared with the exter-  nal resources used by most Chinese NER models, incorporating the rich information in English is a promising way to improve the performance of Chinese NER tasks.
(3) All translated English texts used in this work are automatically generated from the publicly available machine translation API. Compared with the baseline models with external resources, our proposed bilingual enhanced approach improves performance and requires no artificially generated external knowledge. From another perspective, we believe that our proposed model transfers the knowledge in the neural machine translation model to enhance the Chinese NER model.
(4) Our proposed model could achieve state-ofthe-art performance in both cases with English texts automatically translated by two different machine translation systems. This suggests that the performance improvement of our model is not dependent on a specific machine translation system.
(5) Even when the size of training set is small, such as Weibo NER, the performance improvement of our model over other baselines is still significant. This demonstrates that our proposed model is not data-hungry and has promising potential in lowresource NER scenarios.

Ablation Study
To study the contribution of the main components in our model, we conducted an ablation study on all four datasets. The results are reported in Table  2. And we can observe the following facts: (1) To demonstrate the advantage of global sparse operation, we remove it from the model. The results show that, without global sparse operation, the performance of the model degrades severely, which indicates that there exist severe interference of irrelevant words during the process of bilingual interactions. The global sparse operation in ACT-S substantially improves the performance of our model, demonstrating its effectiveness while achieving higher interpretability for our method.
(2) The component of sensing relative distance in ACT-S has a significant positive impact on the performance of the proposed model, and the model without it shows a certain degree of performance degradation. This illustrates the validity of the relative distance-aware component in our model.
(3) We empirically found that the model performs better at word-level bilingual interactions than at subword-level.
(4) There are positive effects of introducing a key bias term into the cross-attention matrix.   Figure 5: Influence of attention sparsity k. Γ is the length of the text sequence.

Impact of machine translation quality & missing translated texts
To illustrate the robustness of our model, we set up separate experiments to investigate the effect of machine translation quality and missing translated text on our model. We randomly replace words in the automatically translated English text with the mask token [MASK] in a fixed proportion to simulate a reduction in the machine translation quality. Similarly, we randomly replace the entire automatically translated English text with a fixed-length(average length of samples in the training set) sequence of the mask token [MASK] at a certain rate for all samples. As shown in Figure 3 and Figure 4, our model can achieve excellent performance even in cases where the translated text has little noise or is missing. And when the translated text is almost full of noise or even 100% missing, our model still outperforms TENER [BERT], a strong baseline that does not use any external resources, which demonstrates the robustness of our model. In addition, it demonstrates that the performance improvement of our model originates not only from bilingual resources but is also related to the model itself.

Impact of attention sparsity k.
We also conducted experiments to verify the impact of the attention sparsity control factor k in ACT-Ss. The results reported on the four datasets are shown in Figure 5. From the results, we can see that the inappropriate sparsity of global attention significantly degrades the performance of the model. If the global cross-attention matrix is too sparse, such as in the case of k = 1, the ACT-S module cannot learn sufficiently about the dependencies between  bilingual texts. At the other extreme, if the attention matrix takes too many interactions between bilingual words into account, the model will not achieve the best performance either. This indicates the sparse nature of lexical dependencies between bilingual texts. Furthermore, the experimental results suggest that it is applicable and interpretable to introduce sparse attention rather than full attention in bilingual interactions. Table 3 illustrates one typical example where our proposed bilingual enhancement model successfully tackles the dilemma of the complex structure of entity composition in the Chinese NER task. Most of the existing Chinese NER methods utilize only the internal features of the Chinese language, which makes it difficult to tag entities with complex composition structures correctly. When rich cues in English are leveraged, this problem can be alleviated. In addition, it can be seen from these two cases that the clues in the different English translations are all beneficial for the correct labeling of complex entities.

Conclusion
In this paper, we propose an Adapted Cross-Transformers with Global Sparse Attention (ACT-S) module to explore bilingual interaction information to improve the performance of the Chinese NER task. Several parameter independent ACT-Ss are employed in our work to capture the rich information in both English and Chinese. We evaluate the proposed model on four Chinese NER datasets and the experimental results illustrate that our method achieves significant and consistent improvement compared to other baselines. In the fu-ture, we will explore how to improve Chinese NER with features from languages other than English and extend our model to other sequence labelling tasks.