{'fraction_non_alphanumeric': 0.029819089207735495, 'fraction_numerical': 0.01846537741734248, 'mean_word_length': 4.798408488063661, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 2, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
AGRank: Augmented Graph-based Unsupervised Keyphrase Extraction

Keywords or keyphrases are often used to highlight a document’s domains or main topics. Unsupervised keyphrase extraction (UKE) has always been highly anticipated because no labeled data is needed to train a model. This paper proposes an augmented graph-based unsupervised model to identify keyphrases from a document by integrating graph and deep learning methods. The proposed model utilizes mutual attention extracted from the pre-trained BERT model to build the candidate graph and augments the graph with global and local context nodes to improve the performance. The proposed model is evaluated on four publicly available datasets against thirteen UKE baselines. The results show that the proposed model is an effective and robust UKE model for long and short documents. Our source code is available on GitHub.


Introduction
The mainstream unsupervised keyphrase extraction (UKE) approaches fall into one of three types: statistical, graph-based, and deep learning approaches. The statistical methods include the TF-IDF-based approach and other recent works (Campos et al., 2020;Beliga et al., 2016), which utilize term frequency, document frequency, word offsets and the number of n-grams to calculate the importance of the candidates. The graph-based methods treat the candidates as the nodes in a graph (Gollapalli and Caragea, 2014;Wan and Xiao, 2008). The edges are calculated based on candidates' co-occurrences, semantic similarity, or other relations. Graph-based algorithms then determine the importance of candidates. Several recent studies have shown that embedding-based methods can achieve excellent performance on unsupervised keyphrase extraction, such as JointModeling (Liang et al., 2021), Atten-tionRank (Ding and Luo, 2021), SIFRank (Sun et al., 2020), KeyGames (Saxena et al., 2020) and EmbedRank (Bennani-Smires et al., 2018). These approaches base candidates' importance on the distance or similarity of candidate embeddings, and some consider the global or local context.
We propose an augmented graph-based unsupervised model to identify keyphrases from documents. The model extracts attention from the pre-trained BERT model to generate a candidate keyphrase graph, then augments the attention graph with nodes that present the global and local context. Similar to the baseline approaches, noun phrases are extracted as candidates representing the nodes on the graph. The co-occurrence of candidates determines graph edges within the sentential context. Edge weights between the candidates are calculated based on the mutual attention extracted from the pre-trained BERT model and the indexes of the sentences where the candidates are located. The candidate graph adds the global and local contexts as document and sentence nodes. The edge weights between the document node and candidates and the edge weights between sentence nodes and candidates are calculated based on the cosine similarity between their embeddings. The graph is then adjusted by removing nodes and edges based on the document frequency and edge weights. Finally, the ranking of each candidate is calculated using the weighted PageRank algorithm.
We summarize our contributions as follows: • A novel augmented graph-based unsupervised keyphrase extraction (UKE) model considering global and local context is proposed and evaluated using four benchmark datasets.
• The mutual attention extracted from the pretrained language model is utilized to build a weighted graph.
• The proposed model works better than or is competitive with the state-of-the-art UKE baselines.

Methodology
Our model has three main parts: (1) Candidate Graph Generation, in which we convert each document into a weighted graph with candidates as nodes and attention between candidates in a sentential context as weighted edges; (2) Graph Augmenting, in which we add a document node and sentence nodes to emphasize the global and local context and their relations to the candidates; (3) PageRank Scoring, in which we apply the weighted PageRank algorithm on the graph to rank candidates to identify keyphrases.

Candidate Graph Generation
To build the candidate graph, we first extract candidates from a document, then add weighted edges between each pair based on the sentence level self-attention mechanism. Furthermore, the edge weights also are influenced by the importance of the sentences containing the candidates' pairs. Candidates Generation. The candidates are extracted using the module implemented in the previous approach (Bennani-Smires et al., 2018). The module first uses part of speech (PoS) to tag the nouns, verbs, pronouns, and adjectives. Then, the noun phrases are extracted using the NLTK 2 package as candidates. In our research, the punctuation are removed from the candidates, except '-'. The stemming is applied to candidates and ground truth keyphrases for model building and performance evaluation. The effectiveness of stemming is investigated in the ablation study section.
Edge Weight Generation. The generation of edge weight is based on the mutual attention between candidates extracted from the pre-trained BERT model (Devlin et al., 2018). Clark et al. (2019) have shown that important syntactic and semantic information is captured in attention maps of the pre-trained BERT model. To compute the mutual attention between candidates, we utilize the methods introduced by Ding and Luo (2021) and Clark et al. (2019) to extract attention between words. The attention between words is then aggregated to attention between phrases.
For a sentence with n words, the mutual attention mapping between words can be presented as a matrix (A).
a ij is the attention value that word w i projects to word w j within the same sentence s. If a candidate is a phrase with multiple words, we sum the word attention into phrase attention. Given candidate c 1 = {w : w i ∈ c 1 } with n words and candidate c 2 = {w : w j ∈ c 2 } with m words, the attention between c 1 and c 2 is the sum of the attention that the words in c 1 project to the words in c 2 , shown as Equation 1.
(1) Fig. 1 shows a visual example of the mutual attention values between phrases. Given a document's title -"Standards for service discovery and delivery", the colored rows in the heatmap represent the attention project from words/phrases labeled on the y-axis to the words/phrases labeled on the x-axis.
To generate undirected weighted edges, we sum edge weights from c 1 to c 2 and from c 2 to c 1 in all sentences containing c 1 and c 2 , shown as Eq. 3.
v(c 1 , c 2 ) = s∈doc (v s < c 1 , c 2 > +v s < c 2 , c 1 >) (3) Edge Weight Adjustment. Campos et al. (2020) has shown that the first few sentences of an article often summarize the main topic and emphasize the domain of the work. Therefore, we adjust the edge weights (v) according to the positions of the sentences (i s ) containing the edges (Eq. 4).
(4) The weight of edge (c 1 , c 2 ) increases proportionally according to the index (i s ) of the first sentence containing candidates c 1 and c 2 . k is the threshold for sentence position. When the sentence index exceeds k, the edges contained in the sentence have no weight adjustment. k can be fine-tuned in terms of the number of sentences based on the length of the document. For a long article, the threshold k can be set to the number of sentences in the abstract or introduction. Articles in different fields will have different k. In the following ablation study, we explored the effect of different k. k is designed to be a multiple of 10. For short documents containing less than ten sentences, k is set to 10.

Graph Augmenting
The candidate graph does not consider the relations between each candidate and the document's global context and the relations between the candidates and each sentence's local context. Hence, we add document and sentence nodes to augment the candidate graph with the global and local context. Document Node. The candidates ({c 1 , ..., c r }) extracted from the document are concatenated as the document node representation. The edge weight between the document node d and a candidate node c is their embeddings' cosine similarity, shown in Equation 5.
The document node embedding (e d ) and the candidate node embedding (e c ) are generated by feeding the text representations of the document or candidate into a pre-trained BERT model. The selfattention mechanism of BERT generates a contextbased embedding for each member word of a text.
A document or candidate node's embedding is generated by summing up the member words' embeddings of the node. We use the bert-embedding 3 package to generate word-level embeddings.
The α d is a coefficient value to adjust edge weights between the document and candidates. It can be set to the average number of sentences in a corpus.
Sentence Nodes. A sentence node is represented using its original sentence content. The sentence node embedding (e s ) is generated using the same way as the document node embedding generation. The edge weight between candidate c and sentence s equals the cosine similarity of their embeddings, shown in Equation 6.
v(s, c) = e c · e s ||e c || · ||e s || (6) Figure 2 shows a visualization example of an augmented graph of a document randomly selected from the dataset Inspec. The blue-colored nodes represent the stemmed candidates. The document node and the sentence nodes are pink-colored and green-colored, respectively. The edge weights between pairs of candidates and between candidates to document or sentence nodes are shown. For demonstration purposes, the edge weights are multiplied by ten and rounded. The original document content is shown in Fig. 3, and the ground truth keyphrases are highlighted. In this example, 'Service Location Protocol' is a labeled keyphrase. In the augmented graph, the edge weight between nodes 'servic locat protocol' and 'race' is high as calculated using BERT mutual attention. 'Service discovery' is another labeled keyphrase and occurs in four different sentences. Hence, in our augmented graph, the node 'servic discoveri' has connections with many candidates. This example reveals that our augmented graph has the mechanism to emphasize the importance of the edges and the nodes based on the document content.
Graph Pruning. To reduce the computational cost and improve the performance, we prune the graph by removing some nodes based on their NLP features and some edges based on the edge weights distribution (Faralli et al., 2018). The following steps are applied: (1) Remove the candidate node when its document frequency exceeds some threshold. High document frequency often indicates that the term is a generic one in a corpus. For each corpus, we calculate the document frequency of all candidates and determine the threshold by the Elbow law 4 .
(2) Remove the edge between a pair of candidates when the edge weight is lower than a threshold, such as the 25 th percentile of the candidatecandidate edge weights distribution.
(3) Remove the edge between a sentence and a candidate when the edge weight is lower than a threshold (p s ) determined by the sentencecandidate edge weights distribution. 4 https://pypi.org/project/kneed/

PageRank Scoring
The pruned graph is fed into the weighted PageRank algorithm (Xing and Ghorbani, 2004) to calculate the importance score of each candidate. The score (P R(c)) of a candidate (c) is calculated as Equation 7.
Where δ is the dampening factor, c n is a neighbor node of c, and B c is the set of all candidate c neighbors. v(c, c n ) is the weight of the edge (c, c n ). The weighted PageRank algorithm considers in-edge and out-edge weights. Since we have an undirected graph, in-edge and out-edge weights are treated the same.
During the final ranking, the document and sentence nodes are excluded, and the candidates with a high document frequency, e.g., higher than a threshold df θ , are also excluded.

Datasets and Evaluation Metrics
The performance of our model is evaluated on four benchmark datasets 5 . Datasets Inspec (Hulth, 2003) and SemEval2017 (Augenstein et al., 2017) contain short documents, and datasets Se-mEval2010 (Kim et al., 2010) and Nguyen2007 (Nguyen and Kan, 2007) contain long documents. Table 1 summarizes the basic statistics of the datasets. The performance of keyphrase extraction is evaluated using F1 scores at the top 5, 10, and 15 ranked keyphrases.
To make an appropriate comparison with the baselines, we follow the common practice of using the uncontrolled annotated keyphrases of dataset Inspec and using the test set of SemEval2010 with 100 documents in our experiment. All extracted and labeled keyphrases are stemmed for evaluation.

UKE Baselines
We compared our model against 13 baseline unsupervised keyphrase extraction models categorized into three categories: (1)

Hyperparameter Setting
The BERT-Base is used for attention extraction (Clark et al., 2019) and node embedding generation 14 . The Hyperparameters for each dataset are fine-tuned and set as follows: For all datasets, δ is set to 0.85, and α d is set to the average sentence number of the corpus. For Inspec and SemEval2017, k is set to 10, df θ is set to 5, and p s is set to the 60 th and the 75 th percentile, respectively. For SemEval2010, k is set to 20, df θ is set to 25. For Nguyen2007, k is set to 90, df θ is set to 45. Sentence nodes are not added to the augmented graphs for SemEval2010 and Nguyen2007 due to the computational cost and the need.
On a computer with an Intel i7 9700k, 48G RAM and RTX 2060 graphics card, generating an augmented graph costs less than 10 seconds for a short document and about one minute for a long document. Table 2 compares AGRank and the baseline UKE models using F1@5, 10, and 15. The values for baseline models are those presented in the original papers or better results published in other papers recently. Since not all datasets are used in the original papers, we applied the baselines to the datasets using the published code. Those produced results are tagged with *.

Results
In most cases, the deep learning-based or mixed models outperform the statistical and graph-based models on short document datasets (Inspec and Se-mEval2017). AGRank outperforms all UKE baselines on Inspec and performs better than most baselines except AttentionRank on SemEval2017.
Our proposed model has more apparent advantages on long document datasets. For the dataset SemEval2010, the F1@5 score is more than 3% higher than the best UKE baseline, and F1@10 and @15 are also about 2% higher than the best UKE baseline.
It is worth noting that the AGRank can often rank the keyphrases in the top 5. The results show that the F1@5 values gained by AGRank on all datasets are 1.5% -3% higher than the bestperformed UKE baseline model on Inspec, Se-mEval2010, and Nguyen2007. The F1@5 value gained by AGRank is also competitive with the best UKE baseline model -AttentionRank, on the SemEval2017 dataset.  Candidate stemming causes nodes to merge and change the graph's structure. Table 3 compares the performance of AGRank with and without stemming. The results show that stemming improves the model performance on SemEval2017, SemEval2010 and Nguyen2007. However, the improvements on the Inspec are not significant.

Analysis of Graph Augmenting and Edge Weight Adjustment
The proposed model augments the graph by adding document and sentence nodes to provide global and local context. We present the impact of the context nodes in Table 3. The model takes better advantage of document node addition on Inspec. In contrast, the sentence node addition contributes more to the model performances on SemEval2017. Interestingly, the model performance on Se-mEval2010 and Nguyen2007 are marginally better without document node addition. We think the document node generated for a long document cannot sufficiently capture the overall context by generating one single embedding.
In our model, the weights of edges between candidates are also adjusted according to the sentence position. From Table 3, the edge weight adjustment based on sentence position has a higher impact on SemEval2010 and Nguyen2007. Without using it, the performance could drop up to 2%.

Analysis of Hyperparameters
We evaluated the impact of the hyperparameters of our model. Fig. 4 shows the hyperparameter tuning of k -the parameter to adjust edge weights by sentence position, p s -the parameter to remove the edges between the sentences and candidates based on weight distribution, and df θ -the parameter to exclude the candidates based on document frequency. Note that the tuning study of parameter k only applies to long documents. For short documents with less than ten sentences, k is set to 10. The parameter p s is only applicable to short documents since sentence nodes are not added to the augmented graphs for long documents due to the computational cost.
We investigated the impact of threshold k from 10 to 130 with a step size of 10. Fig. 4 shows that for the long document dataset SemEval2010, the best F1@15 is gained when the first 20 sentences are considered. Whereas for the long document dataset Nguyen2007, the highest F1@15 is achieved when the first 90 sentences are considered. These results show that adjusting candidates' mu- We showed that adding sentence nodes can slightly improve the performance on short document sets, but the sentence nodes might not have strong relationships with all the candidates. Tuning the number of edges between sentence nodes to candidates can reduce the computational cost and optimize the model performance. We adjusted the p s from 0 to the 90 th percentile based on the weight distribution. Fig. 4 shows that for datasets Inspec and SemEval2017, optimal p s are between the 60 th and the 80 th percentile.
We also tuned df θ to see its impact on the performance. Fig. 4 shows that df θ has less impact on short document sets -Inspec and SemEval2017. Performance of F1@15 can improve about 2% after tuning the df θ on long document sets -Se-mEval2010 and Nguyen2007.

Case Study
AGRank performs closely with the AttentionRank on short documents. To observe the difference between AGRank and AttentionRank, we randomly select a document in SemEval2017. The heatmap in Fig. 5 presents the importance scores of the candidates calculated by the two models. We normalized the original scores to highlight the candidates with a heatmap. The labeled keyphrases are bold, italic, and underlined. AGRank scores higher for keyphrases 'construct model' and 'low emotional involvement', whereas the AttentionRank ranks 'online teaching reformation' higher. Since Atten-tionRank uses accumulated self-attention, long candidates with multiple words obtain higher scores. JointModeling performs well on the long document set SemEval2010. Fig. 6 shows the performances of JointModeling and AGRank on a selected paragraph taken from an article in Se-mEval2010. The heatmap shows the difference in the strategies of the two models. AGRank has fewer candidates than JointModeling, which attribute to our graph pruning step. The candidates with high document frequency and small neighbor edge weights are removed. Since the edge weights of the augmented graph are generated based on the extracted attention of the pre-trained BERT model, AGRank assigns high scores to 'commitment' and 'Bayesian games'.

Related Works
The unsupervised keyphrase extraction approaches can be categorized into statistical, graph-based, and deep learning-based or mixed methods. The mod- Figure 6: Comparison on Long Document els based on statistical techniques convert contextual information into statistical features of candidates and then calculate candidate scores for ranking. Rose et al. (2010) utilized the ratio of word frequency and the number of co-occurring neighbors to evaluate the importance of the candidates. Besides term frequency and neighbor co-occurrence, Campos et al. (2020) also considered more contextual features to identify keyphrases, including the offsets of the candidates, the sentence positions of the candidates first shown, etc. Models based on graph methods treat candidates as nodes of the graph, convert certain relations between candidates into edges of the graph, then use a graph algorithm to calculate the candidates' scores (Mihalcea and Tarau, 2004). Wan and Xiao (2008) utilized a clustering method to select k-Nearest-Neighbor documents to create a graph for a single document and used a graph sorting algorithm to generate keyphrases. Bougouin et al. (2013) employed a clustering method to generate several topics of a document and assign the topics to candidates, then utilized the TextRank model to rank topics; the most representative candidates of the top-ranked topics are extracted as keyphrases. Wang et al. (2014) utilized the word embedding and word frequency to generate weighted edges between words, then used the weighted PageRank algorithm to compute candidate scores and rankings. Florescu and Caragea (2017a) proposed the Position-Biased PageRank algorithm, which incorporates the candidate positions in the document into the ranking calculation. Boudin (2018b) proposed the Multipartite graph model, which encodes the topic information within a multipartite graph to utilize candidate mutual relations. yeon Sung and Kim (2020) extracted hierarchical relationships to determine which edges and phrases should be used and evaluated the nodes according to their inflowing edges. Bennani-Smires et al. (2018) proposed the EmbedRank, which uses a pre-trained language model to generate the document and candidate embeddings and calculate the similarity between them to select more representative keyphrases. Sun et al. (2020) proposed SIFRank, which invokes both the similarity between candidate and document embeddings and the candidate position and frequency to calculate the correlation between candidates and the document. Saxena et al. (2020) investigated an evolutionary game theory model that uses candidate embeddings and statistics to calculate confidence scores to determine whether a candidate is a keyphrase. Ding and Luo (2021) extracted attention mapping weights and then integrated accumulated attention weights with the cross-attention similarity to rank the candidates. Liang et al. (2021) integrated bounded sentences and candidate local relations based on document-to-candidate global relations, then used both jointly to determine the importance of candidates. Zhang et al. (2021) proposed MDERank, which ranked candidates using the similarity between the BERT embeddings of the source document and the masked document.

Limitations, Conclusions, and Future Work
Although our augmented graph-based model performs better than the compared baselines, the graph augmentation process is designed with quite a few hyperparameters that need to be tuned for datasets of different domains to obtain optimal performance. We believe this can be further improved by automating the hyperparameter tuning process. Our research investigated the integration of graph-based and deep learning-based models for unsupervised keyphrase extraction. The pre-trained BERT model is utilized to extract candidates' mu-tual attention to build the initial graph. Global and local context information are added through graph augmenting. PageRank algorithm is used to calculate the ranking scores. We compared the proposed model against 13 baseline unsupervised keyphrase extraction models on four benchmark datasets. The ablation study shows that the edge weight adjustment based on sentence position has a higher impact on the long document sets. Adding the document and sentence nodes improves the performance for short document sets.
Future work includes investigating possible solutions to reduce the number of parameters and improve efficiency. We also plan to compare our unsupervised model against supervised keyphrase extraction models to demonstrate the advantages and performances.