{'fraction_non_alphanumeric': 0.03506345309312217, 'fraction_numerical': 0.013070991389743767, 'mean_word_length': 4.705267311106727, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Zero Pronoun Resolution with Attention-based Neural Network

Recent neural network methods for zero pronoun resolution explore multiple models for generating representation vectors for zero pronouns and their candidate antecedents. Typically, contextual information is utilized to encode the zero pronouns since they are simply gaps that contain no actual content. To better utilize contexts of the zero pronouns, we here introduce the self-attention mechanism for encoding zero pronouns. With the help of the multiple hops of attention, our model is able to focus on some informative parts of the associated texts and therefore produces an efficient way of encoding the zero pronouns. In addition, an attention-based recurrent neural network is proposed for encoding candidate antecedents by their contents. Experiment results are encouraging: our proposed attention-based model gains the best performance on the Chinese portion of the OntoNotes corpus, substantially surpasses existing Chinese zero pronoun resolution baseline systems.


Introduction
In natural languages, expressions that can be deduced contextually by people are frequently omitted in texts. This is special the case in pro-dropped languages, such as Chinese, where a kind of anaphoric expression is frequently eliminated. A zero pronoun is a gap in the sentence that is found when a phonetically null form is used to refer to a real-world entity (Chen and Ng, 2016). We here show a case of zero pronouns from the OntoNotes-5.0 dataset.
In this earthquck φ 1 some rooms collapsed, if there exsit some room quality issues, φ 2 will need to call to account.
We use φ to represent the zero pronouns in this example. Among these zero anaphoras, we can assign the mention "政府/the government" that appears in leading text, to be the antecedent of φ 2 while there are no such mentions for φ 1 . Hence, φ 2 is an anaphoric zero pronoun, and φ 1 is the un-anaphoric case.
With the fact that zero pronouns are gaps that have no text, it is almost impracticable to representant the zero pronouns by themselves. This issue has received increasingly attention. In recent time, deep neural network methods for Chinese zero pronoun resolution (Chen and Ng, 2016;Yin et al., 2017a;Yin et al., 2017b) have been proposed and are intended to encode zero pronouns into the vector-space semantic by additional elements. Chen and Ng (2016) propose a neural network model that learns to encode the anaphoric zero pronoun by using the leading word and governing verb, which leads to the insufficient text issue. To better use associated text information for expressing zero pronouns, Yin et al. (2017a) present the ZP-centered-LSTM architecture that learns to encode zero pronouns by their text words. However, it could bring with some defects: their model regards all the words in the sentence equally, thus fails in capturing informative parts of the sentence. As described in (Chen and Ng, 2016), the clause information of a zero anaphora is very important in explaining these gaps, it is a natural way of modeling zero pronoun by focusing on some important texts. For instance, in sentence "two trains in ten weeks ago φ crashed in their way to the southern German countryside", "crashed in their way" is an important clue for the zero pronoun but "several weeks ago" is not. Under this consideration, a systematic solution that can encode zero pronouns by focusing on informative parts of associate texts is the better choice. Besides, another important issue for the task of zero pronoun resolution is modeling candidates. Recent researches use context and content words to encode candidates (Yin et al., 2017a;Yin et al., 2017b). Typically, these words are modeled in a sequential way by recurrent neural networks. We argue that some of the words in noun phrases contains more important information than others, a need that leads to the usage of attention mechanism.
To alleviate the above-mentioned issues, in this paper, we propose a novel attention-based neural network model to deal with the task. Following existing neural network work for Chinese zero pronoun resolution (Chen and Ng, 2016;Yin et al., 2017a;Yin et al., 2017b), we focus on anaphoric zero pronoun resolution task, introducing a pair-wise model to resolve anaphoric zero pronouns. For some natural language processing tasks (Mnih et al., 2014;Tang et al., 2016), people investigate to apply attention mechanism on top of the convolutional neural network or recurrent neural network to introduce an extra source of information to guide the modeling of useful information. However, since zero pronouns are simple gaps that have no such kind of extra information, the above-mentioned attention mechanism can rarely be directly practiced for modeling these gaps. Inspired by (Lin et al., 2017), we here investigate the usage of a self-attentive mechanism for encoding the zero anaphoras. With the help of self-attentive mechanism, our model is able to effectively focus on informative texts of the zero pronouns and therefore captures essential information on encoding the zero pronouns. In addition, on purpose of modeling informative texts of mentions (noun phrases), we propose an attention-based recurrent neural network to build the mention encoder. With the help of representative vector of zero anaphoras, our model is able to effectively focus on important parts of mentions and therefore brings an efficient way of expressing candidates at the semantic level. Empirically, we show that our method has brought performance gains in baselines, achieving great performance on the widely used OntoNotes-5.0 dataset. Our contributions are three-fold: • By utilizing the self-attention mechanism, our model is able to focus on informative parts of associate texts when modeling zero pronouns, leading to an effective way of capturing useful information for interpreting the zero pronouns; • Our model is capable of modeling candidate antecedents by their informative words with the help of zero pronouns, which in return brings a better way of explaining candidate antecedents; • We show that our model substantially surpasses all baseline systems, gains state-of-the-art performance on the benchmark dataset.
In Section 2, we will discuss related work on zero pronoun resolution. Next, we will intorduce our attention-based neural network model in Section 3. In Section 4, empirical evaluation results are shown. And finally, we conclude in Section 5.

Related Work
In this section, we give a brief summary of early efforts for zero pronoun resolution both for Chinese and other languages.

Zero Pronoun Resolution for Chinese
Converse (2006) is the first rule-based study that integrate Hobbs-algorithm into the resolution of zero pronoun in the Chinese Treebank (Xue et al., 2005). After that, a variety of learning-based methods have been investigated. Zhao and Ng (2007) use the learning-based model to locate and resolve zero anaphoras. They investigate a serious of features and apply the decision-tree algorithm to train the classifier. To better capture the syntactic-level information, Kong and Zhou (2010) introduce the context sensitive tree-kernel unified framework for zero anaphor resolution. On the base of Zhao and Ng (2007), Chen and Ng (2013) further investigate their model, introducing two extensions to the resolver, namely, novel features and zero pronoun links. However, these work deeply rely on annotation dataset. To alleviate this issue, Chen and Ng (2014) present the first unsupervised model that first convert zero anaphoras into ten pre-defined pronouns and then apply a ranking-based pronoun resolution model to select antecedent mentions. Chen and Ng (2015) build a discourse-aware model that can jointly locate and resolve zero anaphoras.
More recently, with the advance of neural network techniques, deep-learning-based methods are introduced and have been demonstrated to be effective for this task. Chen and Ng (2016) first introduce a feed-forward neural network framework, where zero anaphoras are encoded by its previous word and headword. However, their model overlooks context of a zero anaphora, which inevitably misses some valuable information. Naturally, some works try to alleviate this issue by investigating information from associate texts. Yin et al. (2017a) introduce a novel memory-based network neural network model that learns to encode zero anaphoras by its texts and antecedent mentions. They take advantage of multihops architecture, producing abstract information from external-memories as hints for explaining zero anaphoras. Yin et al. (2017b) focus on encoding global-information for candidates, where a hierarchical candidate encoder is introduced that learns to model the candidates. Liu et al. (2017) investigate the issue of generating pseudo training-data for the task of zero anaphora resolution. They use a novelty two-step training strategy that helps to overcome the diversity between the generated pseudo training-data and the real one. Even though these above-mentioned methods can reveal the semantic of zero anaphoras by its context, they regard all the words equally, overlooking the diversity of different words. In this paper, we focus on exploring an effective way of modeling zero pronoun by using the associated texts. More specifically, we integrate a novel self-attentive mechanism, which provides our model an ability to focus on multi-aspects text, benefiting the encodings of zero anaphoras. In addition, by employing an attention-based technique for modeling candidates, our model learns to encode more informative parts of the mentions. All these bring advantages to the resolution of zero pronouns.

Zero Pronoun Resolution for other Languages
There has been a variety of work on zero pronoun resolution for other languages besides Chinese, such as Korean and Japanese. These methods could be categorized as rule-based and learning-based. Ferrández and Peral (2000) investigate a rule-based method that can encode preferences for candidates for resolving zero anaphoras in Spanish. In recent time, learning-based methods (Han, 2006;Iida and Poesio, 2011;Isozaki and Hirao, 2003;Iida et al., 2006;Iida et al., 2007;Sasano and Kurohashi, 2011;Iida and Poesio, 2011;Iida et al., 2015;Iida et al., 2016) have been well studied. Iida et al. (2016) present a novel CNNbased deep neural network model for intrasentential subjective zero anaphora resolution in Japanese. As clues, they use both the surface-word and the dependency tree-structure of a sentence. Their model gains higher precision, which is needed for real-world natural language processing (NLP) applications.

Methodology
We introduce an attention-based neural network model for anaphoric zero pronoun resolution. Compared to the prior studies that have the underutilized context of zero pronouns, we investigate an attention mechanism that helps to effectively capture useful information from associate texts. We here present the methodology in details, which include the preliminary, the architecture of proposed attention-based neural network and training objective of the model.  Figure 1: Framework of the proposed attention-based neural network for zero pronoun resolution. The w i for zero pronoun part means the i-th word in the associated sentence, w zp−i is the i-th word before the zero pronoun and w zp+i is the i-th word behind the zero pronoun. w np i is the i-th word of the noun phrase. After generating the representative vector of zero pronoun and candidate mention, we generate its resolution score by going through two tanh layers.

Preliminary
In the first place, we select its candidate mentions from the associated texts. More specifically, we choose the noun phrases that appear within two sentences from the zero anaphora to be the candidates.
In addition, we choose the strategy used in prior approaches (Chen and Ng, 2016;Yin et al., 2017a) for Chinese zero pronoun resolution, reserving those mentions that are max noun phrases or modifier noun phrases as candidates. By doing so, we can recall most (about 98%) of the antecedents with a small loss. Then, what we desire the resolver to do is to accurately recognize antecedents for zp from its candidates list N P = {np 1 , np 2 , ..., np n }.
Our model is basically a pair-wise model (Zhao and Ng, 2007;Chen and Ng, 2016;Yin et al., 2017b). For each candidate mention of zp, we classify it into two classifications, namely, "corefer" that means the candidate is the antecedent of the zero pronoun; or otherwise, "un-corefer". We build the classifier by applying the attention-based techniques. More specifically, an attention-based neural network model is utilized that generates the coreference score of each zero pronoun-candidate antecedent pair.

Attention-based Neural Network Model
In this part, we introduce our attention-based neural network for anaphoric zero pronoun resolution in detail. The architecture of our model is shown in Figure 1. For an anaphoric zero pronoun zp, we convert the input instances into real-valued vectors by using its context words. With the help of the self-attentive mechanism, our model learns to represent zero pronouns by focusing on different parts of contexts. In this way, we generate the representative vector of zp as v zp . In addition, when dealing with candidate mentions, our model learns to encode the candidate mentions by using its informative content words. We here manipulate the v zp as an external vector to attend to the informative parts of its candidate mentions.
By doing so, our model captures the important information of each candidate according to the zp. After that, we generate the representative vectors of candidates as {v np 1 , v np 2 , ..., v npn }. Lastly, we feed the representative vectors of the candidate and zero pronoun to a two-layer neural network, generating the resolution score for each zero pronoun-candidate antecedent pair. After that, we obtain the resolution probability for each candidate. The candidate mention with the biggest probability is regarded as the final result. Basically, our model involves three modules, namely, a zero pronoun encoder; a candidate antecedent encoder; and a feed-forward neural network that learns to score each candidate antecedent.

Modeling Zero Pronoun
Inspired by Lin et al. (2017), we investigate the self-attention mechanism when modeling zero anaphoras. In practice, we use two recurrent neural networks (RNNs) to encode the preceding and following texts of the zero anaphora (Yin et al., 2017a). The last hidden vectors of these two RNNs are concatenated as the vector-space semantic of the zero pronoun. On top of the employed recurrent neural network architecture, we apply the proposed attention technique that helps the model to capture the informative parts of associated texts.
Especially, our self-attention mechanism provides the attention-weight vectors for hidden states of the employed RNN architectures. We then dot these attention-weight vectors with counterpart hidden states and use the weighted summation vector as the representative vector for the zero pronoun. For a zero pronoun, we map its associated text words as sequential embeddings.
Context precedding = (w 1 , w 2 , ..., w zp−1 ) (1) where w i is a d dimensional embedding for the ith word in the sentence. After that, we generate the representative vector of the preceding and the following text by using two separate RNNs: where RN N pre and RN N f ol are two employed RNNs that model the precedding and following context of the zero pronoun independently. After that, we get the hiddent vector for each word, which has the dimension of u. We represent all the hidden states of RN N pre and RN N f ol as H pre ∈ R npre×u and H f ol ∈ R n f lo ×u , seperately: We then apply the self-attention mechanism, which computes linear-combinations of the hidden vectors in H pre and H f ol . The attention mechanism takes H pre (or H f ol ) as the inputs and produces a matrix of attention-weight A pre (A f ol ): where W 1 is a weight matrix with a shape of d a -by-u and W 2 is in shape of r-by-d a ; r represents the number of hops of attention we choose. The sof tmax() is performed along the second dimension of its input. In this way, the attention matrix A could be seen as a multi-hope attention matrix. Comparing with the single-attention matrix, such a mechanism enables our model to focus on different parts of the contexts, bringing a more efficient way of modeling sentence-level information for the zero pronoun at the semantic level. We get the r weighted sums by multiplying the attention matrix A and hidden states H, regarding the resulting matrix as the representative vector of the zero pronoun's preceding and following texts: Subsequently, we obtain the representative vectors of the associated text of the zero pronoun by averaging the row vectors in each representative matrix (namely, M pre and M f ol ). After that, these two vectors are concatenated as the representative vector of the zero pronoun. Our experiments show that the attentionbased model leads to a considerable improvement from the non-attentive model, indicating that the selfattention mechanism can help to better encode the zero pronoun, focusing on informative parts of the associated texts.

Modeling Candidate Antecedent
We here build the candidate antecedent encoder by using an RNN architecture, whose input is comprised by the words in the candidate antecedent. In an effort to better align the more informative parts of phrases to the anaphoric zero pronoun, we here integrate an attention technique into our model. In this work, we use a gating-function as our attention mechanism. Especially, given the representative vector of the anaphoric zero pronoun v (zp) , the output vector and input embedding vector of RNN in the candidate mention part at time t, h (np) t and e t , the attention mechanism computes a gate as: , v (zp) ), where att is defined as: where W (att) and b (att) are parameters to be learned, m is the number of words in the mention. After that, we regard the averaged attention-hidden vector as the vector-space semantic of the candidate antecedent, which takes considerations of a hierarchy of historical semantic:

Calculating Resolution Scores
After generating the representive vector of zero pronoun, v zp and vectors of its candidates {ṽ np 1 ,ṽ np 2 , ...,ṽ npn }, we calcualte the resolution score for each zero pronoun-candidate antecedent by using a two-layers feed-forwd nerual network. Taking v zp and its i-th candidate mention v np i as inputs, our model calcuate the resolution score by going through two tanh layers: where ], W (s) and b (s) are the parameters of this feed-forward neural network. In addition, to better capture the syntactics, position and other relations between an anaphoric zero pronoun and its candidates, we encode hand crafted features (v (f e) ) as inputs to our neural network model. We utilize the features from exsiting work on zero anaphora resolution (Chen and Ng, 2013;Chen and Ng, 2016), map them into vectors to estimate the resolution score for the zero pronoun-candidate mention pair as: where score i denotes the probability of the i-th candidate mention (np i ) being predicted to be the antecedent, and s −1 is the output vector of the second hidden layer. After that, we obtain the resolution scores for all the candidates {score 1 , score 2 , ..., score n }. The candidate mention with the biggest score is eventually selected to be the antecedent of the anaphoric zero pronoun.

Training Objective
Same as Yin et al. (2017b), we train our model by minimizing the cross entropy error of coreference classification. The training objective is defined as: where T represents all training instances, N P is the candidate-set of the anaphoric zero pronoun zp; δ(zp, np) represents the coreference of zp and its candidate mention np: if they are coreference, δ(zp, np) = 1 or otherwise, δ(zp, np) = 0.

Evaluation Metrics
Same as early work on Chinese zero pronoun resolution (Zhao and Ng, 2007;Chen and Ng, 2016;Yin et al., 2017a;Yin et al., 2017b), we manipulate to evaluate the quality of our model by Recall, Precision and F-score (denoted as F). More specifically, recall and precision are defined as: where a "Res Hit" means that the anaphoric zero pronoun is successfully identified and successfully resolved to a candidate mention that is in the same coreference chain as in the golden answer key annotated in the dataset.

Experiment Settings
Same to existing work on Chinese zero pronoun resolution (Chen and Ng, 2016;Yin et al., 2017a;Yin et al., 2017b), we run experiments on the Chinese part of the OntoNotes-5.0 dataset 1 used in the Conll-2012 task. Because zero pronoun coreferences are only annotated in the training and development set, we thus train our model on the training dataset and evaluate the model on the development dataset. We use the recent zero pronoun resolution systems for Chinese as our baselines, namely, a learningbased model (Zhao and Ng, 2007); an unsupervised method (Chen and Ng, 2015); and others are deeplearning-based methods (Chen and Ng, 2016;Liu et al., 2017;Yin et al., 2017a;Yin et al., 2017b). As we are focusing on the anaphoric zero pronoun resolution, we run experiments by directly employing golden parse tree and golden anaphoric zero pronouns that are annotated in the dataset. In addition, documents in the datasets are from 6 sources: BN (Broadcast News), NW (Newswire), BC (Broadcast Conversation), WB (Web Blog), TC (Telephone Conversation) and MZ (Magazine). We report the overall results on the complete test dataset and also the result from the different source of the dataset.  Table 2: Experiment results on the test dataset, including the results on the overall dataset and different sources of the dataset. The first six columns show the results on the different source of documents and the last is the overall result. The strongest F-score in each row is in bold. The parenthesized number beside a source's name is the number of anaphoric zero pronouns in that source.

Hyperparameter
To tune the hyperparameters of our model, 20% of the training dataset are reserved as a held out development set. Such a strategy is also utilized in the baseline systems (Chen and Ng, 2016;Yin et al., 2017a). We randomly initialize the parameters and minimize the loss-function by Adagrad (Duchi et al., 2011) with learning-rate 0.003. The input embedding vector dimension is 100, the dimension of hidden layer of RNNs (namely, the u) is 256 and d a is 128. Besides, we fix the dimensions of two hidden layer of the feed-forward neural network to 256 and 512. We add the dropout (Hinton et al., 2012) with a probability of 50% on the output of each layer. The code for this work is released in https://github.com/qyyin/AttentionZP.git.

Experiment Results
We report the experiment results (F-score) of our model and the baselines in Table 2. The number of hops of attention is fixed to be 2 (r = 2), where we gain the best result. We report the overall results on the complete test dataset and also the results for each source of documents. As we can observe that our model gains 57.3% in overall F-score, which significantly beats the best baseline system (Liu et al., 2017) by 2.0%. In addition, we run experiments on different sources of test corpus, as shown in the first six columns Table 2. The parenthesized number beside a source's name represents the number of anaphoric zero pronouns in that source. We can observe that our model improves performance significantly in 5 of 6 sources of the dataset. More specifically, our model beats the best baseline (Liu et al., 2017) on all documents in F-score: by 5.1% (source NW), 1.2% (source MZ), 1.5% (source WB), 4.6% (source BN), 2.1% (source BC) and 0.3% (source TC). The main reason why our model gains worse performance on source "TC" lie in the short length of text in this source, which makes our model hard to learn to capture useful information for expressing the zero pronouns. Besides, there are full of numerous verbose words such as "呃/Er", "哟/Yo", which brings difficulties for our model to accurately encode a zero pronoun by focusing on its informative contexts. More efforts could be performed in order to encode the associated text of a zero pronoun in a more efficient way, modeling word-sequences beyond the sentence boundary, for instance. We show in Figure 2 the learning curve of our model on the development dataset. As we can observe, after the first epoch, the F-score on the test dataset is about 46%, and it gradually grows to 52% after about 30 iterations when performance starts to plateau. It is well accepted that modeling useful parts of associated text play an important role in encoding the zero pronouns. By applying the self-attentive mechanism, our model learns to focus on important parts of the contexts, revealing multi-aspect sentencelevel information in a more efficient way than those without attention. On the other side, by assigning different attention to the words in a candidate mention with respect to the information of the counterpart zero pronoun, our model learns to encode candidate mentions in a more natural way. Hence, both of them bring benefit to selecting accurate antecedents, leading to better performance.
In addition, having multi-aspect sentence-level information is expected to afford more abundant in- formation about the encoded zero pronoun, we thus evaluate how the improvement can be brought by tuning r in the self-attentive mechanism. We vary r from 2 to 6, as is shown in Figure3. We can observe that the best performance is reached when r = 2. The results are not confusing because we are tried to focus on the informative part for zero pronouns, and we cut the sentence into two separate parts that are zero pronoun-centric, when r = 2 means to attend on totally 4 parts of the sentence, thus our model performances well in this situation. For illustrating the effect of the proposed attention mechanism for zero pronoun and candidate mentions, we also run an experiment without applying the attention mechanism for both zero pronouns and candidates, which is r = 0 in Figure 3. As we can observe that the performance drops by removing the attention mechanism. For the model without the attentive mechanism, the performance drops by 0.9% in F-score compared with that of the full model. Because that the proposed self-attentive mechanism for zero pronoun brings our model an ability to access multi-aspect sentence-level information, removing such an architecture unsurprising influences the results significantly. With an inspiration that not all the words are equally important for explaining the mention, the performance of removing the attention for candidate mentions is reasonable weak. All these show the benefit of our attention-based model.
Lastly, we give a case study to illustrate the power of our self-attentive mechanism, as is shown in Figure 4. From the figure, we can tell that the model successfully focuses on informative parts of the texts. Though there are redundancies between different hops of attention, our model can capture useful information for explaining the zero pronoun (denoted as "*pro*" in the picture). Our model learns to focus on the informative words such as "数字证书/digital certificate", which is essential for expressing the zero pronoun. a然 数字 c书 T 网上 dC 的 c明 ， 但 *pro* 并不 能 作为 文件 c书 存P 在 ＰＣ机 的 硬盘 中 ， 由此 e免 b g客 用 木f 程序 hH ， 即用即插i Thou*h d,*,tal cert,f,cate ,s the ,dent,f,cat,on on the ,nternet but *pro* cannot be re*arded as a cert,f,cate f,le stored ,n the d,s. of a PC thus to avo,d be,n* stolen by Hac.ers w,th Tro-an l,.e plu*-and-play. Figure 4: Heat maps of our attention-based model. In this case, we show the detailed attention weight taken by the attention matrix taken by the attention matrix (r = 2). Darker color means higher weight.

Conclusion
We proposed a novel attention-based neural network model for Chinese zero pronoun resolution. Using recent advances in attention mechanism, we developed a self-attentive architecture for modeling zero pronouns, which enables our model to focus on parts of the associated texts. In addition, we also investigated an attention-based candidate antecedent encoder that learns to model important parts of the noun phrases with respect to the representative vector of zero anaphoras. Our experiments demonstrated that our method significantly surpasses the state-of-the-art on a benchmark dataset for anaphoric zero pronoun resolution. Future work will evaluate our model on other natural language processing problems, such as anaphora resolution for Chinese and English. We also plan to investigate training the anaphora-specific embedding that could better reveal the descriptive attribute for the zero anaphoras.