{'fraction_non_alphanumeric': 0.03537697443413125, 'fraction_numerical': 0.01249796450089562, 'mean_word_length': 4.622568093385214, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Learning to Encode Evolutionary Knowledge for Automatic Commenting Long Novels

Static knowledge graph has been incorporated extensively into sequence-to-sequence framework for text generation. While effectively representing structured context, static knowledge graph failed to represent knowledge evolution, which is required in modeling dynamic events. In this paper, an automatic commenting task is proposed for long novels, which involves understanding context of more than tens of thousands of words. To model the dynamic storyline, especially the transitions of the characters and their relations, Evolutionary Knowledge Graph(EKG) is proposed and learned within a multi-task framework. Given a specific passage to comment, sequential modeling is used to incorporate historical and future embedding for context representation. Further, a graph-to-sequence model is designed to utilize the EKG for comment generation. Extensive experimental results show that our EKG-based method is superior to several strong baselines on both automatic and human evaluations.


Introduction
In the past few years, the field of text generation witnesses many significant advances, including but not limited to neural machine translation (Vaswani et al., 2017;Gehring et al., 2017), dialogue systems  and text generation (Clark et al., 2018;Guo et al., 2018). By utilizing the power of the sequence-tosequence (S2S) framework (Sutskever et al., 2014), generation models can predict the next token based on the previous generated outputs and contexts. However, S2S models are not perfect. One of the obvious drawbacks is that S2S models tend to be short-sighted on long context and are unaware of global knowledge. Therefore, how to incorporate global or local knowledge into S2S models has been a long-standing research problem. The relations between Yanxiao, Xuner and Nalanyanran are evolutionary. And the characteristic of Yanxiao changes over time.
There lie two different directions to include knowledge into S2S models. On the one hand, many efforts Guan et al., 2019;Li et al., 2019b) have been taken to address the short-sighted problem in text generation by explicitly modeling unstructured context. Nevertheless, these approaches rely heavily on the quality and scale of unstructured context, and become intractable when applying to scenarios where the context length increases drastically (e.g., commenting a full-length novel). On the other hand, researchers (Beck et al., 2018;Marcheggiani and Perez-Beltrachini, 2018;Li et al., 2019a) try to combine knowledge with S2S by employing preprocessed structured data (e.g., knowledge graph) which naturally avoid the difficulty for context length. However, those models are oriented to static knowledge, and hence hardly model events where temporal knowledge evolution occurs.
The dynamic knowledge evolution is very common in full-length novels. In a novel, a knowledge graph can be constructed by using entities (characters, organizations, locations etc.) as vertices together with the entity relations as edges. Obviously, a single static knowledge graph is hardly to represent the dynamic story line full of dramatic changes. For example, a naughty boy can grow up into a hero, friends may become lovers, etc. In this paper, we proposes Evolutionary Knowledge Graph (EKG) which contains a series of sub-graph for each time step. Figure 1 illustrates EKG for the novel "Fights Break Sphere". At three different scenes, "Yanxiao", the leading role of the novel, is characterized as "proud boy", "weak fighter", and "magic master" separately. At the same time, the relation between "Yanxiao" and "Xuner" is evolved from friend into lovers, and finally get married, and the relation between "Yanxiao" and "Nalanyanran" is changed over time as "engagement→divorce→friend".
EKG is important for commenting passage of novels since it is the dramatic evolution and comparison in the storyline but not the static facts resonate the readers most. As illustrated in Figure 1. When commenting passage sampled from the T -th chapter of a novel, the user-A refers to a historical fact that "Nalanyanran" has abandoned "Yanxiao", while the user-B refers to the future relation between "Yanxian" and a related entity "Xuner" that they will go through difficulties together. In this paper, EKG is trained within a multi-task framework to represent the latent dynamic context, and then a novel graph-tosequence model is designed to select the relevant context from EKG for comment generation.

Related Work
graph-to-sequence model has been proposed for text generation. Song et al. (2018), Beck et al. (2018), and Guo et al. (2019) used the graph neural networks to solve the AMR-to-text problem. Bastings et al. (2017) and  utilized graph convolutional networks to incorporate syntactic structure into neural attention-based encoder-decoder models for machine translation. In comment generation, Graph2Seq (Li et al., 2019a) is proposed to generate comments by modeling the input news as a topic graph. These methods are using static graph, and did not involve the dynamic knowledge evolution.
Recently, more research attention has been focused on dynamic knowledge modeling. Taheri et al. (2019) used gated graph neural networks to learn the temporal dynamics of an evolving graph for dynamic graph classification. Trivedi et al. (2017Trivedi et al. ( , 2019; Kumar et al. (2019) learned evolving entity representations over time for dynamic link prediction. Unlike the EKG in this paper, they did not model the embeddings of the relations between dynamic entities. Iyyer et al. (2016) proposed an unsupervised deep learning algorithm to model the dynamic relationship between characters in a novel without considering the entity embedding. Unlike these methods, our EKG-based model represents the temporal evolution of entities and relations simultaneously by learning their temporal embeddings, and hence has an advantage in supporting text generation tasks.
To our knowledge, few studies make use of evolutionary knowledge graph for text generation. This may due to the lack of datasets involving dynamic temporal evolution. We observed that novel commenting need to understand long context full of dramatic changes, and hence build such a dataset by collecting full-length novels and real user comments. The dataset with its EKG will be made publicly available, and more details can be found in Section 2.
The main contributions of our work are threefold: • We build a new dataset to facilitate the research of evolutionary knowledge based text generation.
• We propose a multi-task framework for the learning of evolutionary knowledge graph to model the long and dynamic context.
• We propose a novel graph-to-sequence model to incorporate evolutionary knowledge graph for text generation.

Dataset Development and Evolutionary Knowledge Graph Building
To facilitate the research of modeling knowledge evolution for text generation, we build a dataset called GraphNovel by collecting full-length novels and real user comments. Together with the corresponding EKG embeddings, we will make the dataset public available soon. We detail the collection of the dataset below.

Data collection
The data is collected from well-known Chinese novel websites. To increase the diversity of data, top-1000 hottest novels are crawled with different types including science fiction, fantasy, action, romance, historical, and so on. Then we filter out novels due to the following 3 considerations: 1) the number of chapters is less than 10, 2) few entities are mentioned and 3) lack of user comments. Each remained novel includes chapters in chronological order, a set of user-underlined passages, and user comments for the passages. Then, we use the lexical analysis tool (Jiao et al., 2018) to recognize three types of entities (persons, organizations, locations) from each novel. Due to many of the nickname in novels, the identified entities from the tool contains much noise. To improve the knowledge quality, human annotators are asked to verify the entities, and add missing ones. Then all the paragraphs containing mentions of two entities are identified, and will later serve as a representation of the entity relations at that specific time step.
As for the highlighted novel passage and user comments, three criteria are used to select high quality and informative data: 1) The selected passage must contain at least one entity; 2) the selected passage must be commented by at least three users; 3) comments related to the same passage are ranked according to the upvotes, and the bottom 20% are dropped. Notably, those highlighted passages have a degree of redundancy because users tend to highlight and comment passages at very similar positions. Thus we merge passages which have more than 50% overlapping rate. This operation can effectively reduce the quantity of passages by 30%.

Core Statistics
The dataset contains 203 novels, 349,695 highlighted passages and 3,136,210 comments totally. Due to diverse genre of novels in our dataset, the number of entities and relations per novel varies widely with range [10 2 , 10 5 ]. And the number of comments per passage changes a lot with a range of [3, 2 × 10 3 ], because it depends on how interesting the corresponding passage is.
We partitioned the dataset into non-overlapping training, validation, and test portions, along novels (See Table 1

Build up Evolutionary Knowledge Graph
Then for each novel, we build up a knowledge graph which consists of a sequence of sub-graphs. Obviously, it is sensible to build up a sub-graph for each import scene of the novel, and build up more sub-graphs around the critical transitions in the stroryline. In this paper, we assume each chapter usually represents an integral scene, and hence build a sub-graph for each chapter of the novel. Then for each chapter, the entities being mentioned in the chapter are the vertices of the corresponding sub-graph. And if a paragraph consists two of the entities, an edge is created between the two entities. In such a way, a sequence of sub-graphs are constructed, and form our EKG. In the next section, we will formulate the embedding computation of the EKG, and its application for comment generation.

Model Formalism
In this section, we formulate our approach in details. First, the training of EKG embedding is presented. Then a graph-to-sequence model is shown to utilize the EKG for comment generation. The architecture of the model is shown in Figure 2.

Definition
For a novel with n e entities and n r relations, define a global evolutionary knowledge graph: , where T is the number of chapters 1 (or time periods); G(t) = V (t), E(t) , Figure 2: Architecture of our model. First, the EKG is trained under a multi-task learning framework. Then a graph-to-sequence model is traind to utilize the EKG for text generation.
is a temporal knowledge graph of the chapter t; ., e nr (t)} is the set of edges between two vertices.
Given a passage C from the chapter t with c e entities and c r relations, a local EKG related to it can is a sub-graph of the global EKG: Then the local EKG of the passage is a sequence of local temporal knowledge sub-graphs with T × c e vertex embeddings and T × c r edge embeddings.

EKG Embedding Training
Inspired by the consistent state-of-the-art performance in language understanding tasks, we use offthe-shelf Chinese BERT model (Devlin et al., 2019) to calculate the initial semantic representation of sentences. Considering the fact that entities are either out-of-vocabulary or associated with special semantics within the novel context, we propose the following algorithm to jointly learn entity and relation embeddings in EKG: Vertex embedding learning. The passages containing mentions of entity v will contribute to learn the embedding of v. Specifically, the i-th passage is tokenized while the entity mention v i is masked with token "[MASK]". Then resulted tokens are fed into the pre-trained Chinese BERT model, and f v i is obtained as the output feature corresponding to the mask token. Within the chapter t, there exists N t v sentences containing vertices v. The embedding of v is learned by optimizing the following softmax loss summation which models the probabilities to predict the masked entities as v.
where W t v is learnable parameter and denotes the embedding of v from the chapter t. Usually the semantic representations of entities change smoothly over time, so we propose a temporally smoothed softmax loss to retain the similarity of entity embeddings from successive time periods: (3) where λ 0 , λ 1 and λ 2 are smooth factors; and only valid probabilities are included when t = 1 or T . Then the overall loss for all time periods and all vertices is: Edge embedding learning. Since the number of relations equal the number of co-occurrence for any two entities, it is infeasible to employ an embedding matrix to model the relation. Therefore, a Relation Network (RN) is proposed to learn the edge embeddings in the TKGs as shown in Figure 3. Specifically, the RN takes two vertex embeddings as input, and feed them into the first hidden layer to obtain the embedding of the edge r. Then the embeddings of two vertices and one edge are concatenated and fed into second hidden layer to reconstruct the sentence. We also use the pre-trained BERT (Devlin et al., 2019) to obtain the representation f c for the whole sentence. The f c is taken from the final hidden state corresponding to "[CLS]" token because it aggregates sequence representation.
A reconstruction loss applied to the network is optimized to jointly learn RN and edge embeddings: where p stands for a pair of vertices; p + represents a positive pair with two vertex related to the edge, and p − represents a negative pair with one vertex related to the edge and the other unrelated. The overall loss for learning edge embedding is: Combining the vertex and edge loss above, our final multi-task loss is: where λ r is a hyperparameter to be tuned.

Graph-to-sequence modeling
After EKG embedding learning, we propose a graph encoder to utilize the embeddings of the EKG for comment generation. From the learned G local ekg , we can obtain vector sequences V i = {v i (1), .., , v i (2), v i (T )} for each vertex, and E i = {e i (1), e i (2), ...e i (T )} for each edge. All these sequences are fed into Bi-LSTM to integrate information from all time periods. Then final representation of the vertices and edges are taken from the final hidden state of Bi-LSTM corresponding to the time step t.
Further, our graph encoder employs graph convolutional networks (Zhou et al., 2018) to aggregate the structured knowledge from the EKG and then is combined into a widely used encoderdecoder framework (Vaswani et al., 2017) for generation.
our graph encoder is based on the implementation of GAT (Veličković et al., 2018). The input to a single GAT layer is a set of vertices and edge features, denoted as F v = { v 1 , v 2 , ..., v ce }, and F r = { r 1 , r 2 , ..., r cr }, where c e is the number of vertices, and c r is the number of edges from the passage. The layer produces a new set of vertex features, F v = { v 1 , v 2 , ..., v ce } as its output.
In order to aggregate the structured knowledge from input features and transform them into higherlevel features, we then perform self-attention on both the vertices and the edges to compute attention coefficients where W is learnable parameter; g and h are mapping functions; α e ij and α r ij indicate the importance of neighbor features to vertex i; they are normalized using softmax.
Once obtained, the normalized attention coefficients are used to compute a linear combination of the features corresponding to them, to serve as the final output features for every vertex: Then the graph encoder is combined into the encoder-decoder framework (Vaswani et al., 2017), in which a self-attention based encoder is used to encode the passage. To aggregate structured knowledge, the encoding of all vertices from graph encoder are concatenated with the output of the passage encoder, and fed into a Transformer decoder for text generation. The whole graph-tosequence model is trained to minimize the negative log-likelihood of the related comment.

Experiment
In this section, we first introduce the experimental details, and then present results from automatic and human evaluations.

Details
We train all the models using training set and tune hyperparamters based on validation set. The automatic and human evaluations are carried out based on test set. During the training of the EKG, we first learn the vertex embeddings and fix them during the subsequent training of edge embeddings. Our GAT-based graph encoder is based on the entities from the passage. we keep K entities for each passage: if the number of entities 2 is smaller than K, we use breadth-first searching on the global graph to fill the gap, otherwise we filter the loworder entities according to entity frequency. We set K to 5 which is selected by validation. Label smoothing is used in the smoothed softmax loss. We denote our full model as "EKG+GAT(V+E)", its variant which only use the first term in Equ. 10 as "EKG+GAT(V)" and name the other variant "EKG", which does not use GAT-based graph encoder and feeds the encoding of vertices into the Transformer decoder directly.

Hyperparameters
In our model, we set λ 1 = 0.5, λ 2 = 1.0, λ 3 = 0.3 for the smoothed softmax loss; set α to 0.0 for the reconstruction loss and λ r to 1.0 for the multi-task loss; the number of self-attention layers in our passage encoder is 6; the number of Bi-LSTMs is 2 and the length of its hidden state is 768; and the GAT-based graph encoder has two layers. To stabilize the training, we use Adam optimizer (Kingma and Ba, 2014) and follow the learning rate strategy in Klein et al. (2017) by increasing the learning rate linearly during the first 5000 steps for warming-up and then decaying it exponentially. For inference, the maximum length of decoding is 50; beam searching is used with beam size 4 for all the models.

Evaluation metrics
we use both automatic metrics and human evaluations to evaluate the quality of generated novel comments. Automatic metrics: 1) BLEU is commonly employed in evaluating translation systems. It is also introduced into comment generation task (Qin et al., 2018;Yang et al., 2019).
we use multi−bleu.perl 3 to calculate the BLEU score.
2) ROUGE-L((Lin, 2004)) uses longest common subsequence to calculate the similar score between candidates and references. For calculation, we use a python package called pyrouge 4 . These metrics also support the multi-reference evaluations on our dataset. Human evaluations: 1) Relevance: This metric evaluates how relevant is the comment to the passage. It measures the degree that the comment is about the main storyline of the novel.2) Fluency: This metric evaluates whether the sentence is fluent and judges whether the sentence follows the grammar and whether the sentence has clear logic. 3) Informativeness: This metric evaluates how much structured knowledge the comment contains. It measures whether the comment reflects the evolution of entities and relations, or is just a general description that can be used for many passages. All these metrics have three gears, the final scores are projected to 0∼3.

Baseline Models
We describe three kinds of models used as baselines. All the baselines are implemented according to the related works and tuned on the validation set.
• Seq2Seq models (Qin et al., 2018): those models generate comments for news either from the title or the entire article. Considering there are no titles in our dataset, We compare two kinds of models from their work. 1) Seq2Seq: it is a basic sequence-to-sequence model (Sutskever et al., 2014) that generate comments from the passage; 2) Attn: sequence-to-sequence model with an attention mechanism (Bahdanau et al., 2014). For the input of the attention model, we append the related entities to the back of the passage.
• Graph2Seq (Li et al., 2019a): this is a graphto-sequence model that builds a static topic graph for the input and generates comments based on representations of entities only. A two-layer transformer encoder is used in their work. For fair comparison, we use 6-layer transformer encoder to replace the original and denote the new model as Graph2Seq++. Table 2 shows the results of both automatic metrics and human evaluations. In automatic metrics, our proposed model has best BLEU and ROUGE-L scores. For BLEU, our full model EKG+GAT(V+E) achieves 7.01 score, which is 0.59 higher than that of the best baseline Trans.+CTX. The Graph2Seq++ has a BLEU score 5.56 which is obviously lower than the EKG+GAT(V+E). The main reason is that the Graph2Seq++ is based on static graph and cannot make use of the dynamic knowledge. For Rouge-L, our models all have ROUGE-L scores higher than 20%, and is 0.79% slightly better than that of Trans., which is the best among all baselines; the ROUGE-L score of Trans. is higher than of the Trans.+CTX, which is opposite of that in BLEU.

Evaluation results
In human evaluations, we randomly select 100 passages from the test set and run all the models in Table 2 to generate respective comments. We also provide one user comment for each passage to get evaluations of human performance. All these passage-comment pairs are labeled by human annotators. In relevance metric, our full model EKG+GAT(V+E) has better relevance score than all the baselines. It means our model can generate more relevant comments and better reflect the main storyline of the novel. For all this, there still exists significant gaps when compared to the human performance. In fluency and informativeness metrics, our EKG+GAT(V+E) model has achieves higher score compared to all baselines. It illustrates that the generated comments by our proposed model are more fluent and contains more attractive information.
utilize the evolutionary knowledge from vertices and edges, and make the generated comments more relevant and informative.
Analysis of the number of entities: the corresponding local EKG is constructed based on the entities from the passage. To explore the influence of the number (N ) of entities, we report BLEU scores of our full model based on different number 5 in Figure 4(a). The best BLEU score is achieved at N = 5. The BLEU score at N = 0 belongs to the Transformer(Trans.). And our full model is robust to the number of entities because the BLEU scores are stable when N is in the range [2,7].
Analysis of the number of time periods: we also report the BLEU scores under the different number of time periods in Figure 4(b). Our full model achieves the best BLEU score of 7.01 at N = 4, which is 0.41 higher than that of the static graph at N = 1. It illustrates that the dynamic knowledge is useful for improving the performance.
Case study: we provide a case study here. Four passages that need to be commented are extracted from a novel chronologically and shown in Table 3. For comparison, we use Trans.+CTX and Graph2Seq++, which have the best relevance 5 We do not report the BLEU score of the full model when N = 1 because there are no edges included. and informativeness scores among baselines respectively. To start with, within each case, we find that the generated comments from our model are more informative, while the generated outputs from other models tend to be general or common replies, which proves the effectiveness of our knowledge usage.
From another perspective, we observe that our model can make use of the dynamics of knowledge. Let us take a look at P3, our generated comment describes that Zeng Guofan was born in the south of the country, which is in accordance with the passage described in P1. Similar interactions can be found in all four cases, which support our claims above.

Conclusion
In this paper, we propose to encode evolutionary knowledge for automatic commenting long novels. We learn an Evolutionary Knowledge Graph under a multi-task framework and then design a graph-to-sequence model to utilize the EKG for generating comments. In addition, we collect a new generation dataset called GraphNovel to advance the corresponding research. Experimental results show that our EKG-based model is superior to several strong baselines on both automatic metrics and human evaluations. In the future, we plan to develop new graph-based encoders to generate personalized comments with this dataset.