{'fraction_non_alphanumeric': 0.032984191009043005, 'fraction_numerical': 0.011710363671849586, 'mean_word_length': 4.8117202268431, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Document Summarization using Word and Part-of-speech based on Attention Mechanism

Automatic text summarization is one of the research hotspots in natural language processing (NLP). Recently, there has been a considerable research interest in summarization. While most of the current researches about automatic text summarization are only based on word embedding features, which has the problem of too few text features and can’t effectively utilize some other features of the text. In this paper, we propose a novel deep learning model called Document Summarization using Word and Part-of-speech based on Attention Mechanism (WPABS). We employ the word embedding and part-of-speech embedding to make full use of the features of the text and evaluate our model on English datasets Gigaword and DUC-2004. The experimental results show that our model is better than most methods. Also, compared with WABS(Word Attention-Based Summarization), our proposed model performs better. It is proved that it is useful to combine the word and part-of-speech.


1.Introduction
With the rapid development of the Internet, network information has shown an explosive growth, including a variety of videos, pictures, and text, of which text information occupies the most important part. However, because of the huge scale of network data and the complexity of data, people need to spend a lot of time finding the information they care about from the text. Therefore, it is urgent to find a method that can compress the text information and filter out the useless information, which can quickly and accurately obtain useful information from the Internet.
As one of the hot topics in Natural Language Processing, automatic text summarization aims to capture the central idea of the original text and describe the full text with the shortest information as far as possible. It can help users browse and digest the text information on the Internet better. In 1958, Luhn [1] published the first paper about the summarization. He proposed a method based on word frequency to generate text abstracts by calculating the frequency of keywords in the text to find the central sentence of the text. Since then, after many scholars' research, automatic text summarization has made great progress.
At present, automatic text summarization can be divided into two categories, extractive and abstractive. The former extracts the existing sentences from the documents and combines them together to ensure the readability of the sentences. For example, the LexRank algorithm proposed by Erkan et al. [2] which introduced a stochastic graph-based method for computing the relative importance of textual units for Natural Language Processing, which classified text and vocabulary mainly by calculating the similarity between sentences. Beaux et al. [3] proposed the phrase 2 reinforcement algorithm, which constructed a word graph with the topic as the root node, each sentence was converted into a word chain and added to the graph. Then, it updated the words which appeared many times to generate a weighted directed acyclic graph. At last, it selected the path with the maximum weight as the summary. Generally speaking, the extractive summary is relatively simple and mature, while the quality of extraction is not so good. Abstract text summarization is mainly based on the comprehension of the document and reorganization of the text content, which is much more like human to summarize the document. In recent years, deep learning has made a breakthrough in many fields of Natural Language Processing, such as part-of-speech tagging [4], text segmentation [5], intelligent question and answering [6] and named entity recognition [7]. More and more researchers also use the deep neural network for automatic text summarization. In 2014, Google proposed the Sequence-to-Sequence model [8,9], which opened the upsurge of end-to-end network research in NLP.
In this paper, we propose an automatic text summarization method which integrates word and partof-speech. Because most of the current researches about automatic text summarization are only based on word embedding features, which has the problem of too few text features and can't effectively utilize some other features of the text. For this reason, we add the part-of-speech features which can effectively use the features of each word to learn the part-of-speech combination of words by deep semantic features and better generate text summarization.

The methods of document summarization
Recently, deep learning has made breakthroughs. In the field of automatic text summarization, Rush et al. [10] proposed a data-driven understanding summarization model, which used seq2seq structure with a convolutional neural network as encoder, feed-forward neural network as decoder. Hu et al. [11] constructed a large-scale and high-quality Chinese text summary dataset LCSTS and experimented on the dataset using seq2seq model, which provided a basic learning method for the study of Chinese automatic text summarization. Gehring et al. [12] proposed an architecture based entirely on convolutional neural networks(ConvS2S). The model was mainly used for machine translation which achieved the state-of-the-art performance in some translation tasks. The authors also tried to use the model for automatic text summarization, the experimental results showed the ConvS2S could achieve the performance of the state-of-the-art. Based on ConvS2S, Wang et al. [13] proposed a topic-aware topic ConvS2S model with reinforcement learning for automatic text summarization which can get some high-level contextual information for the summarization. It advanced state-of-the-art methods on various benchmark dataset. Also, it combined the reinforcement learning and deep learning, which was a good idea to combine them in the automatic text summarization task.

Attention mechanism
Attention mechanism can concentrate on its input subsets, which can find more critical information about the current target from many inputs. It borrows from human visual attention when human scans an object. People usually focus on a specific area of the object while ignoring some secondary information. In recent years, neural network based on attention mechanism has gradually become a hot topic in deep learning. In 2014, Mnih et al. [14] used attention mechanism to classify images on RNN models, focusing on important parts of an image and reducing the complexity of tasks. After that, the research upsurge based on attention mechanism neural network is opened up. Subsequently, Bahdanau et al. [15] applied the attention mechanism to the field of NLP for the first time. They applied the attention mechanism to the task of machine translation, enabling translation and alignment simultaneously. Since then, various deep learning networks incorporating attention mechanisms have been gradually extended to various NLP tasks.  [16] is widely used in the field of NLP, such as machine translation, text summarization and machine Q&A. The prototype of the architecture was proposed by Cho et al. in 2014. The model encodes the sentence or article into a certain semantic and then decodes the semantic into another expressed sentences or articles. Its form is as follows, <Source, Target>. our goal is to give the input sentence, expecting to generate the target sentence through the Encoder-Decoder framework. The encoder encodes the input sentences and converts the input sentences into an intermediate semantic representation

3.Our proposed model
In this section, we describe the network structure of the proposed model. The architecture of the proposed model is shown in Figure 1, the model mainly includes three layers: Embedding layer, Encoder layer and Decoder layer.

Word Embedding
Word embedding makes words into dense vectors that the computer can understand. The traditional representation of word embedding, one-hot representation [17] makes each word as a vector, where only one dimension represents number one while the others represent number zero. This representation can't express the relationship between different words. Also, there is a problem of large storage space. Luckily, the distribution representation [18] can effectively solve these problems. It can express the similarity between different words and contain more information with a little memory.
In this paper, we use NLTK 1 which is a set of Natural Language Processing tools based on python to get each English word. For example,

Part-of-speech Embedding
In linguistics, part-of-speech is the basic grammatical attribute of vocabulary. It can classify the categories of the word and also contain semantic information. But most Natural Language Processing tasks, they just focus on the surface semantic of words. Sometimes it is difficult to make full use of the information in the text. Therefore, we can use other features. For example, part-of-speech, for a summarization task, it is useful to the result if we also know the part-of-speech of a word. As we train the model, the network can gradually learn the characteristic of the combination of part-of-speech.
In this paper, we also use NLTK to get part-of-speech of each English word. For example,

Bi-LSTM
Long Short-Term Memory (LSTM) [19] is a special Recurrent Neural Network(RNN) [20], first proposed by Sepp Hochreiter et al. in 1997, which solved the problem of the disappearance of RNN gradient or gradient explosion [21] and can learn long-term dependencies. Figure 3. The structure chart of Cell unit. Compared with the traditional RNN, the LSTM has a controller cell that can determine whether the information is useful. As shown in Figure 3, three gate structures are set in the Cell: an input gate, a forgotten gate and an output gate. They can recode and update information from the memory unit. The state updates status of each door at time t as follows: Bidirectional Long Short-Term Memory (Bi-LSTM) [22] is an improvement of LSTM. Due to the serialization of information in LSTM, there is a sequential order in information processing, and it is impossible to access the contextual information. Bi-LSTM uses two LSTM networks to train together, one training sequence starts from the beginning and another starts from the back. Finally, both training sequences are connected to the same output layer. It will synthesize the past and future information of each point. Figure 4 is the network structure of the Bi-LSTM.  In the formula (6), we multiply the deep semantic expression of words and part-of-speech to compromise two features.

Attention mechanism
In an automatic text summarization task, there must be some words which are more important to the text summarization. The attention mechanism can distinguish the importance of the target unit and other units. In this model, we use the attention mechanism to focus on the valuable parts of the outputs of the Bi-LSTM layer to allocate more weights.
Where ij α represents the weight of allocation, it assigns different weights when we output the result of number i according to the input j h . We calculate ij α as follows : Where -1 i s is the output of hidden layer of decoder layer when we output the result of number i-1.Then we adopt max soft act on ij Y to get the probability ij α called attention matrix, which represents the importance of the different hidden state. Attention mechanism, according to the ij α , which can treat the outputs of the hidden layer differently and pay more attention to the larger weight of the hidden layer.

Decoder layer
In this layer, we use beam search [23] of size 10 to generate abstracts. Also, according to the length of the text, we use three buckets for sampling. In our model, we take buckets = [(30, 10), (100, 20), (200,  10) in the array buckets, '30' represents the max size of the text, while '10' represents the max size of the summary.

Corpus
In the paper, we use the Gigaword 2 dataset as training set and verification set. We use about 3.8 million samples for training and 189 thousand samples for verifying.
For the test set, we use Gigaword and DUC-2004 3 dataset for evaluation. We follow by [10] to use the same randomly held-out test set of 2000 sentence-summary pairs. The DUC-2004 dataset has 50 topics, each topic contains 10 documents.

Evaluation Metrics
In the paper, we adopt the current mainstream automatic text summarization evaluation method called ROUGE [24] to compare the model results with the reference summaries, including ROUGE-N (N=1, 2), ROUGE-L.
Rouge-N is defined as follows: Rouge-L is defined as follows: Where X is the reference summary, Y is the candidate summary and m is the length of reference summary. LCS (longest common subsequence): Given two sequences X and Y , the sequence that maximizes the length of the common subsequence is the longest common subsequence of X and Y .

Experimental configuration
We use 300 hidden units for both encoders and decoders. All embedding, including word embedding and part-of-speech embedding have dimensionality 200. We use the learning rate of 0.0001 with Adam [25]. In the experiments, we set 200 samples per batch with a total of 300,000 training steps.

Baselines
We compare WPABS model with the following baselines: ABS(Attention-Based Summarization) [10]: A data-driven understanding abstract model, which uses seq2seq structure. It takes convolutional neural network as the encoder and feedforward neural network as the decoder. Also, it integrates the attention mechanism into the encoder. ABS+(Attention-Based Summarization+) [10]: Based on the ABS, which adds the feature function. RAS(Recurrent Attention Summarizer) [26]: A conditional recurrent neural network which generates a summary of an input sentence.
SEASS(Selective Encoding for Abstractive Sentence Summarization) [27]: A selected encoding model to extend the sequence-to-sequence framework for abstractive sentence summarization.
ConvS2S(Convolutional Sequence to Sequence Learning) [12]: Its encoder and decoder are based on convolutional neural work which achieves the performance of state-of-the-art. Also, compared to the best model Reinforced-Topic-ConvS2S, our result is similar to its on ROUGE-1(R). But our model outperforms better on ROUGE-2(R) and ROUGE-L(R).
What's more, as we can see from the results of the experiments on the Gigaword and DUC-2004, the results of the Gigaword is better because the model is training on the Gigaword dataset.   table 3 and table 4, we contrast the results between the WABS and WPABS on Gigaword and DUC-2004. Compared with WABS which doesn't merge the part-of-speech, our model WPABS has been improved a little. It is proved that the proposed idea to merge the part-of-speech is useful. Especially, we notice that ROUGE-2 of WPABS improves much than others.

5.Conclusions
In this work, we explore the advantage of combining the word and part-of-speech to improve the performance of automatic text summarization. In linguistics, part-of-speech is the basic grammatical attribute of vocabulary, which also contains some useful features. By taking advantage of this, we present a novel WPABS model that combines the word and part-of-speech. Our experiments conducted on the Gigaword and DUC-2004 datasets show that our model performs better than most baselines, so it is useful to combine word and part-of-speech. `