{'fraction_non_alphanumeric': 0.033639314899809696, 'fraction_numerical': 0.01882906078585022, 'mean_word_length': 5.2556722689075634, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A Decade of In-text Citation Analysis based on Natural Language Processing and Machine Learning Techniques: An overview of empirical studies

Citation analysis is one of the most frequently used methods in research evaluation. We are seeing significant growth in citation analysis through bibliometric metadata, primarily due to the availability of citation databases such as the Web of Science, Scopus, Google Scholar, Microsoft Academic, and Dimensions. Due to better access to full-text publication corpora in recent years, information scientists have gone far beyond traditional bibliometrics by tapping into advancements in full-text data processing techniques to measure the impact of scientific publications in contextual terms. This has led to technical developments in citation context and content analysis, citation classifications, citation sentiment analysis, citation summarisation, and citation-based recommendation. This article aims to narratively review the studies on these developments. Its primary focus is on publications that have used natural language processing and machine learning techniques to analyse citations.


Introduction
While writing a publication, authors usually cite the publications that have influenced the research that they describe in order to explain the framework in which the research took place (Shadish et al., 1995;Turney, 2002). In the past decades, the most significant indicator of the scientific impact of a publication has been considered to be its citation count, and this has been frequently used to evaluate the performance of faculty members, research institutions, and universities (Safer and Tang, 2009). According to Anderson (2006), citation counts are not only an impact metric but also serve as an indicator to capture the overall quality of research. Citations also play a significant role by investigating both the historical roots and the novelty of new research. Through citations, measurable links can be established between the citing and cited documents to establish the 'web' of knowledge (Judge et al., 2007). Mercer, Di Marco, and Kroon (2004) proposed that the relationship between the mutually relevant publications is based on citations, while Small (2004) argues that it represents the metaphorical payment of the scholarly debt.
According to Aljaber et al (2010), the text that is used to describe the cited publication in the citing publication is termed as citation context; the citation content refers to the cited publication. In general, the citation context and citation content analysis is an extended form of the citation analysis used by the scholars to comprehend the impact of a citation. Zhang, Ding, and Milojević (2013) suggest defining citation context/content analysis as a technique to complement classic citation analyses. These context/content analyses are motivated by the need for more accurate bibliometric measures that evaluate the impact of research both qualitatively and quantitatively (Chang, 2013;Galgani et al., 2015). Applying these analyses, various schemes have been proposed to classify the functions of and the reasons for a citation -the analysis of the specific relationship between the cited and citing publications has also been conducted (Hooten, 1991;Teufel et al., 2006).
Conventional citation analysis is quantitative in nature and takes no account of contextual information, whereas citation content and context analyses consider both qualitative and quantitative factors (Cronin, 1984). In recent times, open access to full-text research publications and technical advancements have brought about extensive changes in the methods and techniques to analyse the context/content of citations (McCain and Turner, 1989). Therefore, identifying and describing these changes are an important motivation to write this review. Citation context and citation content analysis analyse the text around the citation anchors. According to Zhu, Turney, Lemire, and Vellino (2015: 9) "when a reference is mentioned in the body of the citing paper, the text that appears near the mention is called the citation context". The context is established through the citation's location in the citing text, the words around it and its semantic context, each with implications (Bornmann et al., 2018). Tahamtan and Bornmann (2019) explain the difference between context and content analyses as follows: "In citation content analysis, the semantic content of the text surrounding a given citation (cited document) within the citing document(s) is read to characterize the cited document.
However, in citation context studies, the citing text around the reference anchor is analysed.
In other words, the text around citations (cited documents) in the citing document is used to characterize citations in the citing document. It is not the objective of citation context studies to yield information about the content of a certain cited document, but to characterize the citation process of the citing authors." (p. 1652) Over the past half-century, several reviews have been published on the topic of citation analysis. Small (1982) and Cronin (1984) presented the earliest reviews on the purpose of citation and its possible classification. A survey by MacRoberts and MacRoberts (1989) reviewed the problems in citation analysis pertaining to cursory attention, i.e., biased citations, self-citations, etc. Liu (1993) presented a review focusing on citation motivation, function, concept, and quality to explore the norms relevant in the citation process and the complexities involved in following the norms. Ding et al. (2014) conducted a survey on citation content analysis that examined the foundation, methodologies, and application. Hernández and Gómez (2015) Table 1 compares some seminal reviews of in-text citation analysis published 2004 or later, emphasising their approach and review focus. These reviews have usually focused on conventional challenges, such as citation behaviour, the role of citations, and citation classification. None have focused, however, on approaches using Machine Learning (ML) and Natural Language Processing Pre-print submitted to Scientometrics *Corresponding author's email: saeed-ul-hassan@itu.edu.pk 4 (NLP) in citation context analyses (Yin et al., 2011), citation content analyses Jeong et al., 2014), classification of citations (Cohen et al., 2006), citation sentiment analysis (Hernández and Gómez, 2015;Yousif et al., 2019), and citation summarisation (Gambhir and Gupta, 2017;Karimi et al., 2018).

Table 1. Previous surveys on citation analysis
Article Approach Review focus White (2004) nonsystematic Reviewed the contributions in the field of information science, related to citation classification, citation content, and context analysis that were published after 1970. Bornmann and Daniel (2008) nonsystematic The authors presented a review of publications on citing behaviour by studying the research articles published from 1960 to 2005. Ding et al. (2014) nonsystematic A survey on citation content analysis has been conducted that examined the foundations, methodologies, and applications of citation content analysis. Hernández-Alvarez andGómez (2015, 2016) nonsystematic The authors provide a survey on articles that worked on the problem of citation context identification and classification. They analysed the latest techniques and data repositories used for citation context analysis. Mäntylä et al. (2018) nonsystematic The review focuses on the top-cited papers from Scopus and Google Scholar on citation sentiment analysis that were published between 2005 and 2016. Tahamtan and Bornmann (2019) systematic The review focuses on studies investigating the relationship between cited and citing documents. The authors review studies published between 2006 and 2018. Ma et al. (2020) nonsystematic The purpose of this review is to identify the methods and information used for citation-based recommendation systems.
In contrast to the existing studies on citation analysis, the current review adopts a systematic approach for the collection and analysis of the studies on citation context and content analysis from Jan 2005 to Dec 2019. The review focusses on the latest technological developments from the past decade, which leverages state-of-the-art NLP and ML approaches to convert unstructured citation contexts into a useable format to obtain insights from the citation anchor data.
In recent years, due to the availability of full-text publications and improved ability to process large textual datasets, significant advances have been made in the analysis of scholarly documents (Safder and Hassan, 2019). The advances have allowed scholars to use a variety of features and ML techniques to determine the citation function, analyse citation polarities, generate citationbased summaries, and build citation-based retrieval systems. Our review comprehensively demonstrates the progress in NLP and ML techniques in this domain and investigates the methodological strengths and weaknesses of the various studies.
The prominent techniques in NLP include n-grams, bag-of-words, and word2vec, while the bestknown ML classifiers are: -Support Vector Machine (SVM: a discriminative classifier that is trained on a labelled dataset, outputting an optimal hyperplane to classify unlabelled data), -Naïve Bayes (NB: a probabilistic classifier that refers to the conditional independence of each feature and is particularly used when the dimensionality of the inputs is high), -Multinomial Naïve Bayes (MNB: an instance of a NB that uses a multinomial distribution for each feature), -Hidden Naïve Bayes (HNB: an extended form of NB that retains its efficiency and simplicity while relaxing its independence assumption), -Maximum Entropy (MaxEnt: a probabilistic classifier that finds weights for the features that maximise the likelihood of the training data), -Decision Tree (DT: a predictive modelling approach in which trees are generated using an algorithmic technique that identifies ways to categorise a dataset, based on several conditions), -Random Forest (RF: a non-parametric algorithm that uses multiple decision trees), -K Nearest Neighbour (KNN: a non-parametric algorithm that classifies a data point based on what group is nearest to the particular data point), and -Logistic Regression (LR: a type of statistical analysis that predicts the outcome of a dependent variable based on independent variables).
Some studies have used deep learning classifiers such as Artificial Neural Network (ANN: inspired by biological neural networks, i.e., the human brain, and built to simulate humans' interconnected processes), Convolutional Neural Network (CNN: a special type of neural networks designed for cognitive tasks like image processing and NLP), Recurrent Neural Network (RNN: an improved variation of neural networks with a short-term memory to retain the contextual information from earlier results), and Long Short-Term Memory (LSTM: a variant of an RNN that uses the shortterm memory of RNN neurons and makes them last longer). Table 2 shows an overview of the studies included in this review and outlining the pros and cons of the various NLP and ML techniques.   • Easy to observe similarities in the data due to dimensionality reduction • Capable of handling several types of classification problems The rest of this review is organised as follows. The subsequent section provides the background to citation indexing. Section 3 presents the systematic approach taken to obtaining the publications for this review. Section 4 presents a detailed review of in-text citation analysis and its applications.
Section 5 presents the concluding remarks and future research directions.

Background to Classic Citation Indexing
The bibliometrics field was pioneered by Eugene Garfield. With citation indexes, he introduced new tools for bibliographical research (Garfield, 1955(Garfield, , 1956). An early assessment of the role of citations in science communication was undertaken by Salton (1963). He discussed their role as a pointer to another publication and confirmed that they are a useful supplement to keywords in identifying relevant documents. Salton (1963) showed that integrating the textual and citation information of a publication significantly improves the performance of a retrieval system. Garfield (1965) noted the importance of classification schemes for citations, arguing that authors may have various motivations, as shown in Table 3. According to Voos and Dagaev (1976), the number of times that a particular publication is cited in an another publication is an indication of its subject relevance to that publication. An analysis by Bonzi (1982) of a cited work across many subjects and types of publication attempted to predict the similarities between the cited and the citing publication from the citation's location in the text, and found that citing a document might serve various functions. Prabha (1983) carried out an empirical study on citation behaviour and found that only one-third of all cited sources are considered important by the authors citing them, and that the way in which authors use citations varies by both discipline and specialty (Hurt, 1987).
Many challenges are associated with manual citation context and content analysis as well as citation categorisation. First, in many studies, individuals who are not experts in the area of reported research provided subjective judgements in performing the manual analysis. Thus, unreliable results may be obtained. Second, the categorisation of a citation by using the anchor text around the citation in full-text involves considerable time and human effort (Pride and Knoth, 2017). Third, a citing sentence can contain multiple references and, even though it may include explicit reference to the target work, another part of the same sentence may not refer to that work at all (Jha et al., 2017). Fourth, a citation may appear several times in a publication; therefore, the 8 citation of a specific publication can have more than one function and can be categorised differently (Erikson and Erlandson, 2014). Table 3. Reasons for citing an article (Garfield, 1965: 85)  Since manual analysis is a time-consuming and tedious task (Bertin, Atanassova, Gingras, et al., 2016), citation content and context studies have tended to be performed on small datasets (Bornmann et al., 2018). Moravcsik and Murugesan (1975) proposed a manual citation classification scheme for citations in the field of physics and concluded that 40% are merely acknowledgements. This scheme was also used by Chubin and Moitra (1975) who broke it down into fewer classes; their results indicated that the number of negational citations were high for a short period right after publication, but then decayed quickly. The study concluded that 80% of citations in the field of science studies were to confirm a statement or to point to further relevant information. Oppenheim and Renn (1978) used the same scheme to analyse why some old publications continued to be cited, finding that a high number of citations is associated with both their authors' public profile and writing skills. While the systematic work to classify citations on the basis of personal judgement has begun with Frost (1979), Finney (1979) was among the originators of automated systems. Finney (1979) classified medical literature using seven categories and concluded that classification should be based on a citation's location and the cue words (terms) around the citation in the full-text. Small (1982) examined how citations are used in the citing publications and argue that not all citations are of equal importance.
In recent years, access to full-text scholarly publications allows the scientific community to extract various features of a citation, particularly those relating to its function and purpose (Abu-Jbara et al., 2013;Siddharthan and Teufel, 2007), location (Boyack et al., 2018), polarity (Hatzivassiloglou and McKeown, 1997), and linguistic pattern (Ikram and Afzal, 2019). Some studies have used publications in XML format to develop classifiers to identify a citation's function, purpose, and polarity (Jha et al., 2017), thus, demonstrating that the analysis of large-scale datasets and feature extraction (citation context, citation location, sentiment, etc.) have become considerably easier and faster (Hu et al., 2013). These developments have led to new areas of research, such as the use of citation contexts for creating summaries of scientific publications (Cohan and Goharian, 2018;Hoffmann and Pham, 2003) and improving scholarly recommendation and retrieval systems (Doslu and Bingol, 2016;Fang, 2017;Zarrinkalam and Kahani, 2013).

Data Collection
In order to conduct our literature overview, we adopted a systematic approach. We started with seed articles (publications from our previous research on the topic) and, by reading them, compiled a list of the candidate keywords. On the basis of these keywords, we constructed a complex query to search for the relevant literature in Scopus: title-abs-key("automatic citation classification") or title-abs-key("automated citation classifier") or title-abs-key("automatic indexing" and "citation analysis") or title-abs-key("automatic feature selection" and "citation analysis") or title-abskey("bibliographic reference classification") or title-abs-key("citation classification") or title-abskey("citation context") or title-abs-key("citation polarity") or title-abs-key ("citation relation") or title-abs-key("citation-based summaries") or title-abs-key("content citation analysis") or title-abskey ("content-based citation analysis") or title-abs-key("contextual information" and "citation analysis") or title-abs-key ("scientific citation classification").

Figure 1:
Year-wise distribution of selected publications The above query returned 368 publications that were indexed in Scopus from 2005 to 2019. We combined all refereed documents in these publications with our seed publications. The extended dataset of over 400 publications was then screened to exclude any irrelevant publications, by reading through their titles and abstracts. For instance, the paper 'Scholarly networks on resilience, vulnerability, and adaptation within the human dimensions of global environmental change' were among the most-cited publications (cited 274 times) in our dataset. We discarded it, yet, as it does not focus on the process of citation. Of the remaining publications, as we sought to review only those publications that employ NLP or ML (the focus of this review) and were published after 2005, we removed all irrelevant publications by reading through the abstract of each publication.
We finally left with a set of 65 publications, of which 26 were articles published in journals, and the remaining 39 were conference papers. This reflects that conference venues mostly drive the research in the area of in-text citation analysis using NLP and ML methods. We also observed that of the 65 selected publications, journal articles had been slightly increased in recent years (see Figure 1).
Furthermore, to perform keyword-and author-level analysis on our publication set, we needed information pertaining to the authors, citation counts, venues, etc. To retrieve this information, we searched for the Scopus ID of each publication and constructed a Scopus query to yield the required information. Figure 2 shows the overall systematic approach taken by this review. To obtain an overview of the active authors in the field of citation context and content analysis over the years, we analysed the citation patterns by using the CitNet Explorer software (www.citnetexplorer.nl;van Eck and Waltman, 2017). In order to locate the major players, we imposed a threshold whereby each publication had to have a minimum of two citations. Figure 3 shows the authors' network of citing publications based on their citation score. Each circle in the network represents a publication's author (since there may be multiple authors of a publication, however, we used only the first author's last name) and is connected to other authors via a direct citation link.
As Figure 3 shows these authors' works form three major clusters that are constructed using internal citation links (based on the publications in our dataset). Eugene Garfield, the founder of the field of bibliometrics, is located at the top of the network. The studies (in light-blue and purple) are the initial studies on citation analysis, such as those by: (1) Garfield (1955), who presents the reasons for citing publications and a classification scheme for citing behaviour; (2) Merton (1957), the founder of modern sociology of science, who basically states that, by citing their work, scholars pay tribute to colleagues; (3) Lipetz (1965), who examines the relationship between citing and cited documents and identifies several citing reasons; and (4) Spiegel-Rosing (1977), who 12 conducts a citation context study using articles from Science Studies. The cluster on the right side of the network (in blue) pertains to human-assisted qualitative studies. These include: Small and Greenlee's (1980) explanation of the significance of citation context, paving the way for cocitation analysis and semantic studies of shared knowledge in the scientific domain; Peritz (1983) who stated that the role of citations varies across the disciplines, proposing an eight-category assessment of citation quality; and Conrad and Dabney (2001) who suggested the automatic classification of citation styles (i.e., IEEE, APA, etc.), which professionals had previously identified manually.

Figure 3. A network of authors associated with citation content and context analysis
The circles on the right (in green) represent authors who focused on the automated analysis of a citation's context. In order to understand the relationship between cited and citing publications, Nanba and Okumura (1999) propose using cue phrases (phrases around a citation, including adverbs, pronouns, negative expressions, etc.) for the extraction and classification of citation contexts. Garzone and Mercer (2000) propose an automated classification scheme that consists of 35 classes, based on a citation's cue words and section in the article in which it appears. Nakov, Schwartz, and Hearst (2004) found that the citation context aligns very well with rich information and can be used for document summarisation, training, and testing data for semantic analysis and information retrieval. In this review, we are especially interested in the green nodes and links.

13
To examine the keywords associated with the publications in our set, a network based on authordefined keywords has been constructed using VOSviewer (www.vosviewer.com;van Eck and Waltman, 2017) to visualise frequent keywords. We have selected 79 terms, applying a minimum of two occurrences as a threshold so that we focus on the major research streams. The size of the nodes is determined by counting the total occurrences, while the co-occurrence of terms determine the width of the links: the higher the co-occurrence, the thicker the line. Figure 4 shows a detailed view of the research landscape of citation content and context analysis, presenting the terminologies in this field. The clusters of associated keywords appear in the network, indicating their relatedness in terms of relevant articles. functions, (4) implying schemes for citation context classification and the use of n-grams as a feature for citation classification, (5) building citation-based networks, (6) information retrieval, and (7) applying NLP techniques to design citation recommendation and ranking systems.
Relatively new topics in this research area are the measurement of knowledge flows between academic institutions and authors, the automatic classification of citations using NLP and ML techniques, the design of context-aware citation recommendation systems, and the discrimination between influential and non-influential citations.
In this literature review, we are especially interested in new advancements that leverage NLP and ML approaches in citation context, citation content and citation sentiment analysis, citation-based summarisation, and citation-based retrieval systems. These topics were not in the focus of previous reviews on citation content and context analysis.

Review of In-text Citation Analysis
Due to the various citation formats and styles, the identification of a citation anchor in a publication is a difficult and challenging task (Cronin, 1982;Shahid et al., 2015). In the past, many methods have been proposed to detect in-text citations, such as CiteSeer (Giles et al., 1998) and Cora (McCallum et al., 2000). Councill et al. (2008) pioneered an open-source package (ParsCit) to locate reference strings. ParsCit employes ML methods coupled with a heuristic framework to find and delimit reference strings, locate the context of a citation, and parse the document structure accurately. Lopez (2009) launched the online tool GROBID (GeneRation Of BIbliographic Data) to identify and parse the citation anchors in citing documents. Tkaczyk and Bolikowski (2015) invented the tool CERMINE (Content ExtRactor and MINEr) to extract the context of citations.
The automatic detection of in-text citation anchors has led to many new research areas. Based on the various research problems, methods, and techniques found in the studies of our publication set, we have divided this review into five distinct parts: citation content and context analysis, citation classification, citation-based sentiment analysis, citation-based summarisation, and citation recommendation systems. We discuss the corresponding studies in the following sections. A summary of the reviewed articles can be found in Table 4. In full-text articles, citations are distributed unevenly, more than 50% of the citations belong to the introduction section. Ding et al. (2013) Journal of the American Society for Information Science and Technology (JASIST) 866 papers Highly re-cited references (the same publication is cited multiple times in the citing paper) appear mostly in the introduction and literature review sections.
Abu-Jbara, Ezra, and  ACL Anthology 30 citing papers (3,500 citation contexts) In the interpretation of citation contexts, lexical features (determiners and conjunction adverbs) are more significant than structural features (position and reference). Bertin and Atanassova (2014) Public Library of Science (PloS) Journals 9,446 papers (459,834 citation sentences) The frequency of verbs in citation contexts depends on the paper sections in which they appear and 50% of the verbs were present in the introduction section. Fujiwara and Yamamoto (2015) PMC-OAS 545,147 papers Papers cited in less than five citation contexts account for around 76% of all the cited papers in the database. Hu, Chen, and Liu (2015) Journal of Informetrics 350 papers (11,327 citations) In research articles, succeeding citations are more intentional and purposeful than first-time citations. Bertin, Atanassova, Sugimoto, and Lariviere (2016b) PLoS Journals 75,000 citing papers (3 million citation sentences) The word 'show' is the most frequently occurring verb in citation contexts among all paper sections.

Bertin and
Atanassova (2017) PLoS Journals 80,000 papers (3,528,514 citing sentences) 41% of the citation sentences contain MIR, most of them appear in the introduction section.

Small, Tseng, and
Patek (2017) PubMed Central Open Access Subset (PMC-OAS) 1.1 million papers Only 46% of the articles that had 'discovery' words in their citing sentences (citances) were actually scientific discoveries.

Hu et al. (2017)
Journal of Informetrics 350 papers (16,917 citations) 25% of the references were mentioned multiple times and located in close proximity. Boyack et al. (2018) Elsevier and PubMed Central 5 million papers The references that are mentioned just once are typically more highly cited than references that are mentioned multiple times. Citation classification Teufel, Siddharthan, and Tidhar (2006) ACL Anthology

papers
The proposed annotation scheme achieved 77% accuracy in determining citation functions. Siddharthan and Teufel, (2007)  The highest frequency of citation contexts belongs to the extend class (50%). The next classes were to criticise (30.14%), compare (13.88%), and improve (3.83%). Cue phrases identify the relationship between cited and citing articles accurately. Abu-Jbara, Ezra, and Radev (2013) ACL Anthology 30 citing papers (3,500 citation contexts) The occurrence of the used class is highest (14.7%) followed by criticism, substantiation, and basis classes. Zhu, Turney, Lemire, and Vellino (2015) ACL Anthology 100 citing papers (3,143 papers cited) The total number of times a paper was cited in the citing paper was the most predictive feature for impact.

Valenzuela, Ha, and
Etzioni (2015) ACL Anthology 465 cited-citing paper pairs Most citations in papers are non-influential and only a small proportion (14.6%) is influential. Hassan, Akram, and Haddawy (2017) ACL Anthology 465 cited-citing paper pairs The most significant feature to identify the important cited papers for the citing papers is the 'similarity between the cited paper's abstract and the text around the citation in the body of the citing paper'. Jha et al. (2017) ACL Anthology 30 citing papers (3,500 citations) The SVM classifier outperformed the naïve bayes and logistic regression classifiers in determining citation functions with an accuracy of 70.5%. Pride and Knoth (2017) ACL Anthology 415 cited-citing paper pairs The combination of three features (total number of direct citations, author overlap, and abstract similarity) led to good classification results. Four-sentence context is better than singlesentence in sentiment analysis; the reduction leads to a loss of much of the sentiment. Li, He, Meyers, and Grishman (2013) PubMed Central -A class imbalance problem in the data corpus leads to a poor performance of polarity classifiers. Abu-Jbara, Ezra, and Radev (2013) ACL Anthology 30 citing papers (3,500 citation contexts) The use of contexts improved the polarity systems' accuracy significantly by 10 percentage points. Features associated with subjectivity, such as negation, subjectivity cues, and speculation, are important for the classification of polarity. Sula  A high value of n-grams (about n = 5) achieves the best results for sentiment classification.

Clinical Trial Papers
Citation-based summarisation Elkiss et al. (2008) PubMed Central 2,497 papers Citing sentences contain additional information for automatic summarisation which can be used as a substitute in the absence of abstracts. Mei and Zhai (2008) Medical Literature Analysis and Retrieval System Online (MEDLINE)

11,000 papers
The C-lexrank model attained 6 and 4 percentage points higher nugget based-pyramid scores than the C-RR and LexRank models. Mohammad et al. (2009) ACL Anthology 11,000 papers Both abstract and citation contexts contain valuable information, effective for the summarisation task. Kaplan, Iida, and Tokunaga (2009) Computational Linguistics

papers citing 4 papers
The identification of citation contexts using the co-reference analysis approach achieved F1scores of about 85% for research paper summarisation. Qazvinian and Radev (2010) Computational Linguistics 10 papers Summaries based on contextual information and citing sentences have higher quality (Pyramid score 0.63) than those generated by using citing sentences alone (Pyramid score 0.41). Abu-Jbara and Radev (2011) ACL Anthology 55 papers (4,335 citation sentences) Sentence filtering has a significant impact on the summarisation results and a proposed technique achieved a 0.53 ROUGE score. Tandon and Jain (2012) Microsoft Academic (MA) 30 papers (500 citation contexts) Adjective, verb, and bi-gram models beat the accuracy of the LM model. The LM accuracy could be improved by cleaning and increasing the data. Barrera and Verma (2012) DUC 2002 533 papers The position model prioritising sentences closer to headings achieved better results than TextRank and MEAD, because sentences close to the headings are likely to be more content representative for summarisation. Qazvinian et al. (2013) ACL Anthology

papers
The generation of technical summaries benefits considerably from the use of citing sentences. Conroy and Davis (2015) Biomedical Summarisation data 20 documents (referenced papers) with 10 documents (citation papers) The term frequency vector space model is a good basic model, but the nonnegative matrix factorisation exceeds the ROUGE scores of computed summaries. Klampfl, Rexha, and Kern (2016) Computational Linguistics 40 annotated sets of citing and reference papers The TextRank algorithm performed best in extracting the 'most relevant' key terms and sentences for summarisation. Nomoto (2016) Computational Linguistics

4,608 training instances
The performance of summarisation models may lack due to the inability to clearly define false and true targets. Lu, Mao, Li, and Xu (2016) Computational Linguistics 40 annotated sets of citing and reference papers For facet identification, SVM achieved the best average performance (F1=0.65). Naïve Bayes tackled the class imbalance problem efficiently and showed balanced performance on each facet. Cao, Li, and Wu (2016) Computational Linguistics

annotated sets of citing and reference papers
The distribution of facet among the data is highly unbalanced, 60% belong to the method facet while only 9% belong to hypothesis. Verma  The probabilistic neural network model improves the overall recommendation with a 9 percentage points gain on recall compared to the LDA. Gupta and Varma (2017) Arnetminer Training (

Citation Context and Content
The extraction of citation context is a vital task for studying the various facets of the relationship between citing and cited publications. Good access to the data is essential. Fujiwara and Yamamoto (2015)

Citation Context Window Size
Studies have presented varying views about the extent of the context relevant to analysing the connection between the cited and citing publication. Ritchie, Robertson, and Teufel (2008) compared various sizes that facilitate the application of ML techniques. They used the following nine categories to define citation context: none (contains no citation context), 1sent (contains only the citing sentence); 3sent (contains the citing sentence plus one sentence before and one after); 1sentupto (contains one sentence context, truncated at the next citation); 3sentupto (contains three sentences context, truncated at the next citation); win50 (contains 50 words on the left and right of a citation); win75 (contains 75 words on the left and right of a citation); win100 (contains 100 words on the left and right of a citation); and full (contains the full citing paper). Ritchie, Robertson, and Teufel (2008) assumed that words that describe the cited publication are located close to the citation in the full-text than words that are far away from the citation.
The results of the study showed that the 3sented context performed better than 1sent, that 1sentupto and 3sentupto performed worse than 1sent and 3sent, respectively. win50 performed worse than either win75 or win100. A comparison of the effectiveness of window-based and sentence-based contexts proved the greater usefulness of sentence-based contexts; expanding the context size did not ensure any better identification of the contexts. In a succeeding study, Athar and Teufel (2012a) suggested the use of four sentences (the citing sentence, one sentence before the citing sentence, and two sentences following the citing sentence) as an appropriate citation context window.
According to the authors, the longer citation contexts (four sentences) are more effective than shorter citation contexts (only the citing sentences), as the longer citation contexts contain more descriptive terms from the citing article. The four-sentence context window is now considered as a quasi norm for citation context analysis studies (Teufel, Siddharthan, and Tidhar, 2006).

Feature Extraction for Context Identification
In automatic context identification, feature extraction is a non-trivial and vital component in the training of ML models. Angrosh, Cranefield, and Stanger (2010) (2010). The authors showed that lexical features (determiners and conjunction adverbs) are more significant than structural features (position and reference). The F1 measure score achieved by Abu-Jbara, Ezra, and Radev (2013) is 15 percentage points higher than that of Angrosh, Cranefield, and Stanger (2010). The results point out, therefore, that their feature set (see Table 5) should be preferred in future studies for the task of identifying citation contexts.

In-text Citation Distribution
Several studies have focused on the distribution of in-text citations by considering the IMRaD structure (i.e., introduction, methods, results, and discussion) in which a citation may appear. Hu, Chen, and Liu (2013)  there is a high probability that it is a review. Another study focusing on section-specific citation distributions has been presented by Ding et al. (2013). They used a larger dataset than Hu, Chen, and Liu (2013) consisting of 866 articles from JASIST. The authors included only highly cited references and used structural and semantic features in the analyses. They reported that the use of citations in the documents is unequal: some cited references appear multiple times and some only once in the text. The study reported that 78% of citations were cited less than three times, and the most highly cited publications are found in the introduction and related work sections.
In a succeeding study, Hu, Chen, and Liu (2015) also investigated the phenomenon of recurring citations. Among the total of 11,327 citations from 350 articles, 74.3% (8,417 citations) were cited only once in the citing publication, 25.7% (2,910 citations) were cited twice or more. The study examined the frequency of citations in two perspectives: citation context and citation location. In citation location analysis, they determined the location distribution of recurring citations by using the IMRaD structure and found that the most cited reference was cited in a similar section. This result revealed that a reference seems to be cited within a single topic or context. For the citation context analysis, Hu, Chen, and Liu (2015)  The study also found that references that are cited only once in the publication are more frequently highly cited than references that are mentioned multiple times. The reason is that citations cited only once are usually older and thus have higher citation counts. Bertin and Atanassova (2018)   In the biomedical domain, Small, Tseng, and Patek (2017) recently explored the citing sentence, called 'citance,' to identify the word 'discoveries' for classifying the publications that present scientific discoveries. The authors used 1.1 million full-text publications from the PubMed Central

Citations' Role According to Position
Open Access Subset (PMC-OAS) database. They trained an ML classifier on the citance vocabulary and found that only 46% of the publications that had the term 'discovery' in their citance were scientific discoveries. The authors concluded that the term 'discovery' is not a reliable feature to find (citing) publications that present discoveries. They also presented the top 10 words associated with discoveries; the top three are 'discovered,' 'first,' and 'important.' Concerning the techniques used for classification, they concluded that the ridge regression classifier performed best, with an accuracy of 94%.

Citation Classification
Research on the automatic classification of citations has been undertaken by many authors using various classifiers, data corpora, in-text features, and numbers of classes. Among the pioneers, Teufel, Siddharthan, and Tidhar (2006) used a supervised IBK classifier (which is similar to KNN and uses a distance measure to locate k nearest instances in the training data for each test instance) with shallow and linguistically inspired features including cue phrases. The authors used a corpus of 116 documents from the ACL Anthology to present an automated annotation scheme of 12 classes. Table 6 lists these classes. The study shows that the class 'PMot' appears nearer to the beginning of the publications; the comparative result classes (CoCoR-, CoCoR0) mostly appear near the end of publications. The system identifies the class of each citation with an accuracy of 77%. After the publication of this pioneering study, several subsequent studies have revealed various essential features for the task of citation classification.

Feature Extraction for Citation Classification
Abu-Jbara, Ezra, and Radev (2013) used a citation taxonomy of six classes by merging the 12 classes proposed earlier by Teufel, Siddharthan, and Tidhar (2006). They experimented with multiple classifiers and employed a battery of features, and found that the SVM classifier outperformed the others, achieving an accuracy of 70.5% in predicting correct citation categories.
Although the scheme could not match the accuracy of the classifier by Teufel, Siddharthan, and Tidhar (2006); the study by Abu-Jbara, Ezra, and Radev (2013)  Assessing the usefulness of the studies by Siddharthan and Teufel, (2007), Dong and Schäfer (2011), and Jochim and Schütze (2012) for feature extractions is a challenging task, as each of them has used different datasets and annotation schemes. However, in terms of the micro-F1 score, the scheme by Jochim and Schütze (2012) performed comparably well. It is an important advantage of this study that the manually annotated data corpus of the study is publicly available for future research.

Role of Linguistic Features for Classification
Some studies investigated the role of linguistic features (n-grams and cue words) in citation contexts to determine the relationship between the citing and cited publications. One of these studies was undertaken by Agarwal, Choubey, and Yu (2010) and 'improve' (3.83%). Wang, Villavicencio, and Watanabe (2012) reported that a high number of cue phrases identifies the relationship between the cited and citing articles more accurately than a low number. The appearance of multiple cue phrases in the same sentence may result in low precision.
In a recent study, Small (2018) examined hedging words that best classify those citations located in either the method or the non-method sections, using a set of 1,000 biomedical articles. An LR model revealed that the frequency of hedging words such as 'may,' 'show,' and 'suggest' was higher in the citances of non-methods sections, whereas the hedging word 'using' were more frequent in the citances of methods sections. Small (2018) concluded that the predictive ability of the word 'using' in classifying citation contexts into the method and the non-method sections was higher than that of other hedging words, with a degree of accuracy of 89.5%.
The results of the studies in this section reveal that the SVM model with a specific set of features might outperform the other classifiers in determining citations accurately. Since the size of a dataset influences the features like unigram and n-gram, these features can perform better with large datasets.

Important versus Non-important Citations
In recent years, some research on classification has shifted from multiple categories to only two and 'self-citation' were the best predictors of an influential citation. Pride and Knoth (2017) combined the 40 features presented by Zhu, Turney, Lemire, and Vellino (2015) with the 12 features presented by Valenzuela, Ha, and Etzioni (2015) to find out the most influential features for citation function classifications. They found that a combination of just three (total number of direct citations, author overlap, and abstract similarity) led to better classification results. Similar to Valenzuela, Ha, and Etzioni (2015) and Zhu, Turney, Lemire, and Vellino (2015), they reported that the 'number of times a reference is cited in the citing paper' is the strongest predictor of a citation's influence.

F1
Total number of citations of a reference

F2
Number of citations in the current paper to the cited paper

F4
Citations in literature review section

F7
Citations in discussion section

Citations in conclusion section
Cue wordbased features

F9
Cue words for related work citations

F10
Cue words for comparative citations

F11
Cue words for using the existing work

F12
Cue words for extending the existing work Textual features

F13
Similarity between abstract of cited paper and text of citing paper

F14
The cited paper and citing paper share at least one author The annotation scheme of Valenzuela, Ha, and Etzioni (2015) provides useful guidelines for distinguishing important from non-important citations. Hassan, Akram, and Haddawy (2017) extended the feature set of Valenzuela, Ha, and Etzioni (2015) by six novel features in three categories. Table 7 depicts the complete feature set along with descriptions. The authors experimented with the five most common classifiers (NB, KNN, SVM, RF, and DT) and showed that the RF classifier with the proposed features improved the AUCPR by 4 percentage points compared to that presented by Valenzuela, Ha, and Etzioni (2015). They also evaluated each feature's importance by implementing the extra-tree classifier and realised that the newly proposed features were among the top eight features.
In a more recent study, Hassan, Imran, Iqbal, Aljohani, and Nawaz (2018) addressed the problem of citation classification by comparing two traditional ML models, SVM and RF, with the LSTM model. They applied four state-of-the-art models (Teufel, Siddharthan, and Tidhar, 2006;Abu-Jbara, Ezra, and Radev, 2013;Valenzuela, Ha, and Etzioni, 2015;Hassan, Akram, and Haddawy, 2017) and a new model for classifying citations as either important or incidental. They extracted 64 features using a dataset presented by Valenzuela, Ha, and Etzioni (2015).  . The unsupervised ML-technique SOM was deployed to obtain a qualitative understanding of the features and a good data visualisation. The SOM reduced the data to two dimensions and mapped each citation to a specific neuron (the smallest unit that performs a mathematical function). The non-important class formed an independent cluster with adjacent neurons; only 14% of the data belongs to the important class and formed independent clusters with many neurons (but not a large cluster). In terms of clustering, the results showed that it is easier to identify the non-important than the important class citations.
However, the results were affected by the class imbalance issue, as only 14% of the data belong to the important class.
In a similar way to the work by Hassan, Akram, and Haddawy (2017), Tuarob et al. (2019) investigated citation context data to capture the evolution of algorithms in the scientific literature.
The authors argued that in many cases, new algorithms are not developed entirely from the scratch; instead, they are built by extending the existing algorithms. Therefore, the authors classified citation contexts according to two schemes: 'utilize' (the algorithm is either used or extended in the citing document); or 'not utilize' (the algorithm is only mentioned in the citing document). A dataset of about 8,796 citation contexts was randomly selected, representing a variety of study domains, venues, and document types. To characterise the usage of the algorithms, the authors presented context-based and content-based features. The best average results (F1 = 0.905) in the binary class were achieved for the 'usage' class by the SVM classifier, which combined context and content features.
Among the studies that have addressed the problem of classifying citations into important and nonimportant classes using supervised classification models, the solution by  seems to achieve the most promising accuracy. The authors provide the extracted feature file and implementation code on GitHub for the reproducibility of their work -based on the dataset published by Valenzuela, Ha, and Etzioni (2015).

Citation-based Sentiment Analysis
Citation-based sentiment analysis means the classification of citations into positive, negative, and neutral classes. A citation is marked as positive if it emphasises the strength of a cited paper, and it is marked as negative if it points out the weakness of a cited paper. Neutral means that the citation is rather descriptive in nature.

Context Window Selection for Sentiment Classification
Several studies have investigated the influence of the context window size on the performance of automatic systems for classifying citation sentiments. Athar and Teufel (2012a) observed the importance of context while analysing the sentiments of citations, using a dataset of 1,741 citations from ACL Anthology. They suggested two methods for context utilisation. First, they considered the citing sentence plus a context window (one sentence before the citation, citing sentence, and the two sentences after the citation) to train a classifier on the merged texts after extracting features (1 to 3 n-grams). Second, each sentence in a context window of four sentences was treated as an individual sentence and was annotated using a four-class (positive, negative, neutral, and exclude) annotation scheme. The study of the authors revealed that sentiment analysis achieves better results when it does not consider the merged context. The context window of four sentences, which was treated as an individual sentence, provided useful information for sentiment detection.
In contrast to this first study, Athar and Teufel (2012b) automatically identified the sentiments of cited papers using n-gram and dependency features on an annotated benchmark corpus by Athar (2011). They compared their approach of a four-sentence context to that of a single sentence and proved that overlooking citation context would affect the accuracy of a sentiment system, especially with respect to citations including criticism. They found that a system based on a single sentence results in a loss of sentiment, due to lesser available information. The results generally point out that a system based on four-sentences contexts should be favoured in sentiment classifications.

Role of Linguistics features for Sentiment Classification
While some studies have focused on the influence of the context window size, others have investigated the usefulness of (linguistic) features for citation sentiment analysis. Athar (2011) examined the effectiveness of novel features, including n-grams, dependency relation, scientific lexicon, and sentence splitting (by utilising citing sentences only). Of the 8,736 annotated citations, 1,472 were used as a training corpus and the rest as a testing dataset consisting of 244 negative, 743 positive, and 6,277 neutral citations. The author noted that tri-grams and dependency features are best at automatically identifying a citation's sentiment, achieving a macro-F1 of 89%.
Abu-Jbara, Ezra, and  annotated the ACL dataset to train the SVM classifier with 10fold cross-validation techniques and a list of features (see Table 8). The authors undertook an analysis of each feature's importance using the chi-squared test and concluded that features associated with subjectivity, such as negation, subjectivity cues, and speculation, are important for the classification of polarity achieving an F1 value of 74%. While the polarity features proposed in the study by Abu-Jbara, Ezra, and Radev (2013) did not outperform the model proposed by Athar (2011), one can conclude from the results that the consideration of context improves the model's accuracy significantly. The closest subjectivity cue to the target reference or its representative or any anaphoric mention of it

Contrary expressions
Whether the citation context contains a contrary expression

Section
The heading of the section in which the citation appears

Dependency relations
All the dependency relations that appear in the citation context As the annotated dataset compiled by Athar (2011) achieved promising accuracy, this dataset has been used by many studies as a gold-standard to analyse the polarity of citations. One of these studies has been published by Ikram, Afzal, and Butt (2018). They proposed a classification model and explored the usefulness of n-grams in citation-based sentiment analysis. Their results showed that a higher value of n-grams (n = 5) yielded 2 percentage points better scores in determining the sentiment of citation contexts than lower values. The authors compared the efficiency of the proposed model with other available commercial tools for citation polarity, i.e., SEMANTRIA (a Microsoft Excel add-in) and THEYSAY (an online sentiment analysis tool). Their results revealed that the model followed the same trend line as depicted by SEMANTRIA and outperformed the results of THEYSAY by achieving an F1 score of 85.91%. Although SEMANTRIA produced the most precise predictions (F1 score of 96%) on the dataset by Athar (2011), it produced very different results across the datasets. Thus, it seems that SEMANTRIA has some weaknesses. (2019)  Various ML classifiers were trained in aspect-based sentiment classification with n-gram features.

Ikram and Afzal
The results showed that the SVM classifier outperformed the other classifiers, with a precision and recall of 85.6% and 85.2%, respectively. Similar to Ikram, Afzal, and Butt (2018), they suggested applying a high value of n-grams (about n = 5) to achieve the best results.

Influential Features for Sentiment Classification
Using a range of various features (e.g., closest verb or self-citation) and machine learning classifiers, some studies have employed sentiment classification models (Mäntylä et al., 2018;Pang and Lee, 2008). For example, Teufel, Siddharthan, and Tidhar (2006) designed an annotation scheme for the automatic classification of citation polarity, using cue words as a feature. They conducted two classification schemes: a four-way scheme (weak, contrast, positive, and neutral) and a three-way scheme (positive, negative, and neutral). The dataset was comprised of 116 conference papers from the Computation and Language e-print archive. The study revealed that 83% accuracy was achieved by the three-way classification, compared to 75% by the four-way classification. Hence, the three-way classification scheme produced more constructive results. Piao, Ananiadou, Tsuruoka, Sasaki, and McNaught (2007) suggested the polarity relation between cited and citing publications (i.e., attitudes of authors' approval or disapproval for the work they cite) is useful for information retrieval and text mining tasks. They designed a system for authors to search for publications in an extensive collection of articles. For this purpose, the authors collected and mapped citations (subjective words and sentiments) to form a network combined with an opinion polarity relation by employing semantic lexicon resources and NLP tools. The proposed system gathered cited articles and compiled a citation distribution list that shows the opinion polarity relations between articles and citations. Although the system appears promising, a drawback of this study is that the authors neither mentioned the accuracy of their model nor compared their model with other baselines.

Class Unbalancing in Sentiment Classification
The results by Li, He, Meyers, and Grishman (2013) highlight the class unbalancing issue in sentiment classification of citations and its effects on the performance of the classification models.
They suggested a unique scheme that includes a total of eight positive, one negative, and three neutral functions to study the sentiment of citations in PubMed. To classify a citation's function automatically, they used a ME-based approach with various syntactic and surface features. The model achieved a low accuracy of 67%. They concluded that its poor performance was due to the class imbalance in the data corpus. To solve the class imbalance problem, Sula and Miller (2014) eliminated the neutral class from the scheme (to have only meaningful classes) and To solve the class imbalance problem, Jha et al. (2017) also decided to eliminate the neutral class, since more than half of the citations were in this class. Using the ACL dataset, they set up two binary classification schemes with a battery of features inspired by the work of Abu-Jbara, Ezra, and . Citations were classified as subjective (polarized) or objective (neutral); subjective (polarized) citations were categorised as positive or negative. The results showed that the polarity classifier yielded more intuitive results than the subjectivity classifier. As the dataset was highly skewed (more than half of the citations were neutral), eliminating the neutral class increased the accuracy by up to 6 percentage points. The chi-squared analysis led to a similar conclusion as by Abu-Jbara, Ezra, and : subjectivity features, such as subjectivity cues, negation, and speculation, are more important than other features. Taşkın and Al (2018) created an automatic classification scheme for Turkish citations. To classify the citation context, NB and RF were tested using a 10-fold cross-validation, and various semantic and syntactic features. The results showed that the RF classifier accurately detected 96% of the positive and 70% of the negative citations, with an average accuracy of 89%. As the training dataset contains a lesser number of negative instances (only 0.8%) -similar to Sula and Miller (2014) -they believed that increasing the number in the negative class training dataset would increase the classifier's accuracy.
The results of the studies in this section point out that removing the neutral class would lead to a better accuracy of classification schemes. However, the classifier should have enough data to train.
Especially the training sets for the positive and negative classes should be large and balanced in number.

Citation-based Summarisation
Automatic text summarisation might be a way of producing a fluent and concise summary of a publication by capturing the overall meaning (and critical content). The summarisation approaches should be able to identify both the uniqueness and similarity between publications. A typical study in this area has been published by Verma and Lee (2017). The authors compared human-generated with automatically-generated summaries. Their results revealed that, on average, 9% of the words in human-generated summaries do not appear in the original article. Thus, it seems that humangenerated summaries have added values compared to automatically-generated summaries.
CL-SciSumm (the first medium-scale shared task on scientific document summarisation in the computational linguistics domain) divides the task of automatic summarisation into three categories: (1) search of relevant text-spans that identify the relationship between citing and cited document, (2) classification of discourse facets, and (3) generating of abstractive summaries (Chandrasekaran et al., 2019). We discuss each task in the following subsections.

Baseline Summarisation Models
Various summarisation techniques have been used hitherto as a baseline for summarisation tasks.
The MEAD summarisation system ) -the most elaborate publicly available platform for multi-lingual summarisation and evaluation -is a centroid-based summarisation system. The MEAD summariser has three components: first, feature extraction and its conversion into feature vector; second, conversion of feature vector into scalar value, and third, assignment of scores to sentences, and addition of similar sentences to the summary. Another technique named as LexRank summarisation (Erkan and Radev, 2004) builds a graph of all candidate sentences and then evaluates the meaningful sentences using eigenvector centrality. Although the edge between two candidate sentences might demonstrate the similarity between the sentences (and thus their meaningfulness), the authors proposed the LexRank system as sensitive to noisy data.
Two variants of the LexRank system have been proposed by Qazvinian and Radev (2008) for the summarisation of publications: Cluster Lexrank (C-lexrank) calculates LexRank within each cluster while Cluster Round-Robin (C-RR) picks sentences from each cluster in a Round-Robin way. The Round-Robin way starts with the largest cluster; then, the sentences are extracted in the order they are listed in each cluster. In the C-lexrank model, first, the reliable information (the 'nuggets') from each paper are prioritised manually, and second, a weight is assigned to each 'nugget' based on the evaluation. The results of the study show that the C-lexrank model attains an increased 'nuggets' based-pyramid score compared to the C-RR and LexRank model (6 and 4 percentage points, respectively).
Earlier this decade, Abu-Jbara and Radev (2011) presented an approach in which they divided the task of automatic document summarisation into three steps. The first step included the tasks of reference tagging, context identification, and sentence filtering. In the second step, the extracted representative sentences were classified, similar sentences were added into a cluster, and the LaxRank value of each sentence was computed. In the third step, the sentences were added into a summary based on the sentence ranking of cluster and LaxRank values. The authors used a dataset of 55 papers from the ACL Anthology Network and deployed a well-known evaluation metric called Recall-Oriented Understudy for Gisting Evaluation (ROUGE). They compared the proposed technique with the techniques used by Radev et al. (2004), Erkan and Radev, (2004) and Qazvinian and Radev (2008). The results demonstrated that the approach of the authors achieved a 12, 13, and 10 percentage points improved ROUGE score, respectively.

Importance of Citation Text for Summarisation
Citations are considered to be a significant source of information for generating automatic summaries in various text-mining areas (Li et al., 2017;Teufel and Moens, 2002). Teufel (2006) argues that citations include valuable subjective assessments of cited publications. These assessments can be exploited to generate a summary. Other authors agreed with Teufel (2006) and investigated the effectiveness of citing sentences in the task of summary generation such as Elkiss et al. (2008). These authors used the cosine similarity metric and examined the usefulness of citing sentences and abstracts in an automatic summary generation. Their study is based on the PubMed Central dataset of 2,497 articles. The authors concluded that -in the absence of an abstract -citing sentences may be a good substitute for automatic summarisation.
Using the ACL dataset, Mohammad et al. (2009) analysed the importance of citing sentences in summarisation by comparing it with abstract-based summaries and full-text summaries. They used two different approaches for evaluating automatic summarisation: the first approach is ROUGE, and the second approach is a nugget-based pyramid evaluation. In this evaluation, the elements of valuable information, the so-called 'nuggets,' from each article were prioritised manually. Then, a weight was assigned to each 'nugget' based on this evaluation. The ROUGE evaluation results showed that summaries generated from abstracts performed significantly better (p < 0.05) than summaries generated from citation contexts. Moreover, summaries generated using citation contexts obtained significantly better scores (p < 0.05) than those generated from full-text publications. The results from the nugget-based pyramid evaluation demonstrated thus that the summaries generated from citation contexts outperform those generated from abstracts and fulltext publications. Mohammad et al. (2009) concluded that both abstracts and citation contexts have unique information that can be used to improve summarisation results.
The study by Qazvinian et al. (2013) illustrated the importance of citing sentences for the creation of summaries for Question Answering (QA) and Dependency Parsing (DP) articles. Using stateof-the-art techniques, as applied by Mohammad et al. (2009), the study generated summaries based on three sets of information: abstract only, citation only, and full paper. For the evaluation, Qazvinian et al. (2013) used the pyramid score 'ratio of the sum of the weights of semantically relevant words to the sum of the weights of an optimal summary'. The study demonstrates that the generation of technical summaries benefits considerably from the use of citing sentences.
The studies reviewed in this section reveal that there is a small but quantifiable difference in the information content provided by citation contexts as compared to abstracts or full-texts. The results of the studies indicate therefore that the use of citing sentences might lead to improved extractive summaries of publications.

Identification of Text-Span (Task 1A)
We mentioned above the merits behind the extraction of citations in the running texts. Numerous studies addressed the task 1A to generate a summary of contributing publications using the dataset from CL-SciSum, which comprised 40 annotated sets of citing and cited papers from the CL domain. Kaplan, Iida, and Tokunaga (2009) presented a co-reference, chain-based approach with cosine similarity as a feature to extract the 'citation-site (c-site),' which is the block of text that contains both the citation and its context. The authors compared their approach with two baseline models (baseline 1 extracts only the citing sentence; baseline 2 extracts the sentences before and after the citing sentence) using a labelled dataset containing 38 articles from the ACL dataset citing four publications. The results demonstrated that the proposed approach achieved a high micro-average F1 score of 84%. However, it should be considered in the interpretation of this result that the study is based on a small and less representative dataset containing only 94 citing sentences. Qazvinian and Radev (2010) addressed a similar problem of contextual information extraction from publications as Kaplan, Iida, and Tokunaga (2009). The authors suggested a framework based on probabilistic inference. They modelled lexical similarities of sentences as a Markov random field to discover the patterns created by the context data. They employed a belief propagation mechanism to identify those sentences with the same context. For experimentation, they used ten papers from the ACL anthology. Their results demonstrated that the use of citation contexts (four sentences on each side of the reference anchor) with citing sentences improved the pyramid score considerably, from 0.41 to 0.63. Although the studies by Kaplan, Iida, and Tokunaga (2009) and Qazvinian and Radev (2010) used rather small datasets, they reveal that the generation of fluent scientific summaries is a non-trivial task in the absence of sufficient background information.
The study by Nomoto (2016) worked on the extraction of relevant citation texts using two hybrid models: a Term Frequency-Inverse Document Frequency-based (TF-IDF, a weighting scheme used in text mining to evaluate the importance of a word for a document in a corpus) similarity model; and a single-layer ANN that scores relevant citing text more highly than irrelevant text.
The results showed that the performance of the ANN-based model is better than that of the TF-IDF-based model. Nomoto (2016) suspected that the TF-IDF model's performance is lacking due to its inability to define false and true targets clearly. There may be some words that appear in both true and false targets, which could quickly derail the accuracy of classifiers. In a similar study to that by Nomoto (2016), Klampfl, Rexha, and Kern (2016) compared three different approaches for classification: (1) modified-tsr, where the TextSentenceRank algorithm is applied to the data. This algorithm is a modified version of the TextRank algorithm, which is a graph-based ranking algorithm to extract key sentences or key terms.
(2) tsr-sent-class, where a binary classifier is used to decide for each candidate sentence, whether it can be served as a reference text span.
(3) sectclass-tsr, where a binary classifier is used to decide for each section, whether it contains a reference text span. The authors found that modified-tsr achieved the best results by extracting both the 'most relevant' key terms and sentences.
In this section, we summarised some studies addressing the task 1A of automatic summarisation.
The comparison of the different approaches reveals that the TextRank algorithm approach presented by Klampfl, Rexha, and Kern (2016) seems to be a promising method for the extraction of text spans.

Facet Identification (Task 1B)
Addressing the task 1B of CL-SciSumm, some studies have considered the facet identification problem (i.e., the classification of referenced text spans into the following classes: 'implication,' 'method,' 'aim,' 'results,' and 'hypothesis') as a text-classification problem. The studies have used various modelling approaches to solve the facet identification problem based on the CL-SciSumm 2016 dataset.
The facet identification problem was targeted by Cao, Li, and Wu (2016), who stated that this problem is a multi-label classification task. They noted that the facet distribution of the dataset is exceptionally imbalanced, as 60% of the text spans are in the methods section and only 9% in the hypothesis section. They stated that DT could remember the patterns of all facets, whereas SVM and NB remember only those of the dominant class. Therefore, the DT classifier was employed (featuring TF-IDF vectors) and achieved a low micro-averaged accuracy of 59%. The study by Lu, Mao, Li, and Xu (2016) presented a similar issue with class unbalancing. The authors proposed a feature engineering approach to text-span classification and defined a set of features, including lexical features such as TF-IDF, topic similarity (cosine similarity between the citation and reference span), concept similarity, and sentence importance. After the extraction of the citation context, the context was classified based on the section of the article or the 'discourse facet' in which the text appears by applying the DT, SVM, and NB classifiers. In contrast to the study by Cao, Li, and Wu (2016), the authors revealed that the NB classifier achieved a balanced performance across all five facets while the SVM classifier achieved the highest micro-average accuracy of 65%. Cao, Li, and Wu (2016) and Ma, Xu, and Zhang (2018) proposed multi-stage actions for facet classification. First, they extracted similarity-based, position-based, and rule-based features for multi-label classifications. Second, they applied a sampling-based algorithm to pre-process the imbalanced data corpus. Third, each facet built a dictionary and assigned the reference span to the facet whose dictionary contained any span words. The results showed that the proposed system achieved an average precision of 0.7169 -a better result than that achieved by Cao, Li, and Wu (2016) and Lu, Mao, Li, and Xu (2016) -in identifying facets for summarisation using an imbalanced dataset. Overall, the study concluded that similarity-based features are more suitable than position-based features for summarisation. Moreover, in working with the class imbalance dataset, TF-IDF similarity and IDF similarity turned out to be two robust features.

Generating Summaries (Task 2)
Some studies in our publication set attempted to solve the task 2. Mei and Zhai (2008)  A similar LM approach for the classification of citation contexts as used by Mei and Zhai (2008) was proposed by Tandon and Jain (2012). Their LM model was trained on 30 articles from the MA search engine. The model used an opinion vocabulary with two types of terms (uni-grams and bi-grams) to indicate the context and the adjectives, verbs, and adverbs that describe the cited publication's opinion. In contrast to the findings of Mei and Zhai (2008), the study showed that a combination of adjectives, verbs, and bi-grams models beats the accuracy of the LM model, achieving a 68.54% average precision. The proposed LM model attained an average precision of 67%.
Based on the TextRank algorithm, further improvements for summarisation have been proposed by Barrera and Verma (2012). They suggested a combination of syntactic, semantic, and statistical methods using a dataset from the Document Understanding Conference (

Citation Recommendation System
For the task of recommending citations, citation recommendation systems match user queries with existing publications in a database and recommend publications that could be cited by the user.
According to the different user inputs, these systems can be divided into three main types: keywords-based citation recommendation, citation-list-based citation recommendation, and context-based citation recommendation. Keeping the scope of this review in mind, we focus only on context-based citation recommendation systems in this section. These systems take a few sentences as input and recommend a list of articles as possible citations based on the local context of input sentences. Context-aware approaches estimate the semantic similarity between an article and the citation contexts of articles citing that document. Articles to cite are recommended then, based on the similarity score.
Studies that have addressed the development of citation recommendation systems have used two techniques: topic modelling and neural network. Explanations of these two techniques are provided in the subsequent sections.

Topic Modelling
Early research on citation recommendation systems formalised the task of finding relevant publications to cite as topic discovery. Nallapati, Ahmed, Xing, and Cohen (2008)  Tang and Zhang (2009) also targeted the task of ranking the relevance of papers in a recommendation system for a given citation context. They proposed a two-layer restricted Boltzmann machine model known as RBM-CS, which discovers the relationship between articles based on the topics of the articles given as input. The results demonstrated that RBM-CS achieved 0.4237 MAP and performed better than LM-based techniques for citation recommendation. The desired accuracy of the model could not be attained, however, due to data sparsity issues (i.e., certain expected values were missing in the dataset).
The approach by He, Pei, Kifer, Mitra, and Giles (2010) generated a candidate citation set by taking co-authorship relations into account. The input of the system is a query document, and the system automatically discovers the positions at which citations should be located and recommends the documents for citing with high similarity scores. Using the CiteSeerX dataset, the system outperformed the baseline methods (uni-grams, bi-grams, dependency model, etc.) with an average precision of 47.2%. The authors claimed that other existing approaches do not perform well due to various noise (irrelevant words) in documents.
For developing ambitious citation recommendation systems based on topic modelling techniques, the data sparsity and noise issues ought to be resolved. Wang and Blei (2011) presented such a system: a collaborative topic regression model that combines the merits of probabilistic topic modelling and traditional collaborative filtering. They investigated the model using a relatively small dataset including 37 articles. The empirical results were promising: the proposed model was able to make relatively useful recommendations.

Neural Networks
In recent years, several studies have started to deploy neural networks for citation recommendations. In these systems, the neural networks have been trained on the contextual content of papers or by linkages in bibliographic networks. In one of the first studies, Huang et al. (2015) proposed the multilayer neural network model that learns document representations and words to compute the probability of citing a document for a given citation context. For implementing the model, the CiteSeer dataset containing 10,760,318 citation contexts was used.
The results based on this dataset revealed that the model could improve the overall recommendations with a 9 percentage points gain on recall compared with the LDA model. Gupta and Varma (2017) addressed the problem of article recommendation by introducing a novel approach that combines article content with a graph structure. They used a dataset from Arnetminer (a database that contains research articles published between 1958 and 2014) to test their approach.
The results demonstrated that the approach outperforms state-of-the-art techniques (i.e., TF-IDF and LDA) by 21 and 39 percentage points, respectively, in terms of MAP.
In the most recent study, Yang et al. (2018)

Discussion
Citation analysis is one of the most crucial methods in research evaluation. This is reflected, for instance, in the utilisation of citation data in several international university ranking systems. The use of citations in research evaluation is mainly based on the normative theory of citing. According to Merton (1973) The studies that we have included in this literature overview addressed several topics. In this section, we summarise the most important findings for these topics and discuss their (practical) implications. We will begin the summarisation with the challenges that are associated with citation in-text analyses. First, a citation context might contain multiple references to publications, and only a part of a specific context might be relevant for the focal publication. In these cases, it is often a difficult task even for human annotators (domain experts) to assign the context to a specific class. Second, most of the publications in this research area used data from the ACL Anthology.
There is a lack of variance in the employed datasets, therefore, raising questions concerning the generalisability of the findings. Third, while some databases are available with full-text access today, there is limited access to several full-text datasets due to copyright restrictions.
Several studies have investigated the optimal length of citation contexts for analysing the relationship between citing and cited articles. The results have shown that extended citation contexts appear to be more effective than shorter citation contexts (Aljaber et al., 2011). The extended citation contexts contain more descriptive terms from the citing article (Ritchie, Robertson, and Teufel, 2008). It seems that the optimum window size for the context analysis is four sentences. Alongside ascertaining the optimal length of citation contexts, many authors have also analysed the usefulness of various lexical and structural features in context extraction. In terms of the accuracy attained by the features, it seems that lexical features (determiners and conjunction adverbs) are more significant than structural features (position and reference) for context extraction (Abu-Jbara, Ezra, and . Numerous studies have used ML techniques for citation classifications with preferred features, such as linguistic features, cue words, contextual features (closest noun phrase and conjunctive adverb), and location features (position and section).
The outcomes point out that linguistic features and cue words are effective in achieving high evaluation scores.
In this literature overview, we have reviewed studies that applied NLP and ML techniques to analyse citation sentiments such as lexicon-based methods (e.g., SentiWordNet). These methods are used to search for terms that reveal the sentiments and classify citation contexts into different polarity levels. The studies have utilised various feature-sets: the uni-gram feature seems to be very popular whereby higher values of n-grams lead to better classification results (Ikram, Afzal, and Butt, 2018). In the studies reviewed here, SVM and NB classifiers are the most frequently used models for sentiment classifications (Abu-Jbara et al., 2013;Athar, 2011). Their classification results might be improved by eliminating the neutral class in the analyses. Most of the studies have reported that class imbalances' effects lead to a decrease in the accuracy of the employed classifiers. Another problem which also decreases the classifiers' accuracy is that the studies are based on small datasets (to reduce the annotation efforts). Against the backdrop of these problems, we believe that the use of large publication sets for citation sentiment tasks is an important future research direction by leveraging deep learning methods.
Citing sentences can contain additional information about the scientific document that has been cited. Therefore, the information can be used to characterise cited documents in addition to the information extracted from abstracts. In studies on citation-based summarisation which we included in this overview, a considerable weightage has been given to the citing sentence in summarisation as compared to solely abstract-based or full-text summaries. Since these studies have shown that the information content of citation contexts is lower than that of abstracts (or fulltexts), citation contexts should be preferred to be used in conjunction with other information (Qazvinian and Radev, 2010). In the absence of an abstract for summarisation, however, citation context may serve as a substitute (Elkiss et al., 2008).
The studies that have dealt with automatic summary generation in the past (as an alternative to a human-generated summary) classified this task into three steps: reference-span identification, reference-span classification, and summary generation. For reference-span identification, the TextRank method may be preferred since it outperformed other state-of-the-art approaches, as discussed by Klampfl, Rexha, and Kern (2016). For the task of facet identification, studies have revealed that the distribution of facet is highly imbalanced. The Naïve Bayes classifier achieved a low accuracy, but for a balanced performance among all facet, the SVM classifier achieved a high accuracy (Lu, Mao, Li, and Xu, 2016). The studies dealing with summary generation have shown that the language models are more beneficial than the baseline models such as MEAD and LexRank. The authors of these studies took into account linguistic patterns to generate summaries that are not just a combination of sentences (Ma, Xu, and Zhang, 2018).
There are several limitations associated with automatic summarisations: most of the studies have utilised relatively small datasets to tackle the summarisation problem. The use of large publication sets is vital for future research direction, which might lead to better generalisability and applicability of the results. We assume -based on the reviewed studies -that neural abstractive models need to be increasingly exploited to generate automatic summaries of publications by using deep-learning models (See et al., 2017). The increasing availability of full-text data and the rapid development of ML and deep-learning methods bring new challenges in citation sentiment analyses, such as multi-lingual summarisation (the processing of several languages and providing of summaries in a single language as output) and multi-document summarisation (the processing of several documents to compile single summary documents as an output). There has been a surge of novel methods for automatic summary generation as well as summary evaluation methods such as Rouge-1 (Lin and Hovy, 2003) or BLUE Score (Dreyer and Marcu, 2012). However, there is a need for evaluation metrics that can determine whether the generated summaries are coherent, sentence-wise, and grammatically correct.
Search engines and database providers utilise citation contexts to show how publications are related to each other. One goal of these analyses is to find relevant publications for citation recommendation systems. Our overview of the literature in this area shows that the studies mainly utilise two techniques to predict citations: topic modelling and neural networks. Topic modelling techniques such as LDA are the traditional ways of recommending citations. The problem with these techniques is yet that they do not perform well based on large and noisy datasets (He, Pei, Kifer, Mitra, and Giles, 2010). Therefore, to recommend citations based on larger datasets, neural Shardlow, Batista-Navarro, Thompson et al., 2018).

Concluding Remarks
This literature overview could reveal that the applications of NLP and ML techniques are significant and important developments in the area of citation context and content analyses.
However, the studies undertaking these analyses were constrained by several limitations. The first and foremost limitation is the tedious annotation task of the texts. While the manual tagging for training the classifier is a lengthy and challenging process, the automatic tagging produces poor results. Second, in many studies, scholars from various disciplines (such as social sciences, engineering, or health sciences) annotated field-specific datasets and employed NLP and ML models in their respective fields. Thus, cross-model comparisons of the various approaches are limited. Interdisciplinary research groups could be formed to annotate (comprehensive) standard datasets including papers from many fields. Third, only a few studies have published their code and dataset hitherto, making it virtually impossible to compare and replicate the empirical findings.
Thus, research in this area could be more openly done. Fourth, one of the most critical challenges in automatic citation content/context studies is the difficulty of extracting the citation context from the publications due to the variability in citation styles (which might differ between journals, publishers, and fields). The algorithms used for automatic approaches to obtaining citations may miss those citations that fail to adhere to the searched style. Thus, approaches should be developed in future research that tackles this problem.

Guidelines for the application of NLP and ML techniques
In the final section of this review on studies that have used NLP and ML techniques for citation context and content analyses, we provide a step-by-step guideline on how these techniques can be used in practice: The first task refers to the process of data collection: for citation context/content analyses, it is required to analyse full-text publications. Nowadays, some data repositories provide such full-text publications that can be used for these analyses. For example, the ACL Anthology includes more than 23,000 full-text publications with more than 350 venues. Another accessible dataset is provided by the open-access publisher PLoS. Currently, this dataset includes more than two million publications ranging across 200 disciplines.
The second task refers to the extraction of the citation context information, which is well manageable with standardised datasets such as those from ACL Anthology or PLoS. However, it is a difficult and challenging task with non-standardised datasets, primarily due to the possible use of multiple citation formats and styles. So far, no off-the-shelf solution can be used to sufficiently tackle this problem in the process of citation context extraction. An essential question in this process concerns the size of the context data around the citation anchor. We recommend (based on the results of the corresponding studies) to apply the four-sentences approach proposed by Athar and Teufel (2012): the citing sentence, one sentence before the citing sentence, and two sentences following the citing sentence should be extracted. The citation anchor can be identified by using regular expressions; the surrounding sentences can be extracted as desired then.
The third task concerns the annotation of the dataset for training the classifiers. We recommend using the annotated dataset published by Jochim and Schütze (2012) for citation classification, the classification of Valenzuela, Ha, and Etzioni (2015) for identifying important and non-important citations and the baseline by Yasunaga et al. (2019) for citation-based summarisation.
The fourth task refers to feature computation, which is a vital but time-consuming task. Many authors have not provided their datasets and codes which could be used as starting points for subsequent studies. Since  are one of the few authors who have published their dataset and code (on the GitHub repository), we recommend drawing on that material.
The fifth task concerns the necessity to make codes and datasets available. In order to stimulate research activities based on NLP and ML approaches, we would like to encourage all authors to make their annotated datasets and codes freely available.