{'fraction_non_alphanumeric': 0.02645029088253998, 'fraction_numerical': 0.008926678757330947, 'mean_word_length': 4.697530864197531, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 3, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Finding answers to COVID-19-specific questions: An information retrieval system based on latent keywords and adapted TF-IDF

The scientific community has reacted to the COVID-19 outbreak by producing a high number of literary works that are helping us to understand a variety of topics related to the pandemic from different perspectives. Dealing with this large amount of information can be challenging, especially when researchers need to find answers to complex questions about specific topics. We present an Information Retrieval System that uses latent information to select relevant works related to specific concepts. By applying Latent Dirichlet Allocation (LDA) models to documents, we can identify key concepts related to a specific query and a corpus. Our method is iterative in that, from an initial input query defined by the user, the original query is expanded for each subsequent iteration. In addition, our method is able to work with a limited amount of information per article. We have tested the performance of our proposal using human validation and two evaluation strategies, achieving good results in both of them. Concerning the first strategy, we performed two surveys to determine the performance of our model. For all the categories that were studied, precision was always greater than 0.6, while accuracy was always greater than 0.8. The second strategy also showed good results, achieving a precision of 1.0 for one category and scoring over 0.7 points overall.


Introduction
Coronavirus disease (also called COVID- 19) was first detected in Wuhan, in the Hubei province of China and was reported to the World Health Organization (WHO) by the Chinese government on 31 December 2019 [1]. The illness has been spreading since the beginning of 2020 and officially became a pandemic in March when the virus had already infected more than 150,000 people worldwide [2,3].
Since the beginning of 2020 and due to the social, economic and political repercussions generated by the evolution of the virus [4,5], the media and the scientific community have published an incredible amount of information on the aforementioned disease [6]. In just 3 months from the initial notification of the disease by the Chinese authorities, a total of 1596 publications had been generated about COVID-19, 66% of them from China, while in April, the number of publications rose to more than 6500 [7,8]. At a social level, the pandemic has also attracted the attention of social network users [5]. In these circumstances, the excess of information and the lack of knowledge about the pandemic have led to the spread of inaccurate or false information (also known as fake news).
In the current environment, it is necessary and relevant to establish strategies to analyse the large amount of information that is continuously being generated in the scientific world in order to make the task of responding to the different needs that arise from different fields easier. Such fields include health, social, economics, ethics, educational and political, to name a few.
In this article, we propose a method to extract scientific literature based on a variety of topics. Our method facilitates the work of identifying which works written about COVID-19 could help respond to certain questions. Furthermore, our method also makes it possible to differentiate between articles that do and do not address purely health-related issues. This article aims to make it easier for researchers to find and retrieve scientific literature related to complex or abstract topics, thus making it easier to find answers to complicated questions and to provide them with a complementary method for using information retrieval systems based on standalone keywords.
To do this, we worked with a dataset of articles on COVID-19 and other related areas. We extracted the different topics for each article, analysing its title and abstract. Then, we filtered the documents by performing a coincidence analysis on the terms of the topics with the terms of the query.
We propose a method to find answers to complex and abstract questions by exploring latent concepts hidden in the titles and abstracts of scientific works. Our main goal was to respond to the call to action from the White House and other various research groups. They had prepared a dataset of COVID-19-related articles so as to solve urgent and relevant problems related to the pandemic [9]. While plenty of proposals have been offered by different authors (some of them are mentioned in the following section), we wanted to design a simpler model that would be able to find answers on COVID-19 adequately. More importantly, humans have validated our method, while most authors use automatic systems to measure or benchmark their proposals.
While our method has been tested specifically with COVID-19 data, it can be generalised and used in different areas. Our work is structured as follows: • The 'State of the art' section reviews the latest published studies, mainly in the field of bibliometrics and natural language processing (NLP), as related to the COVID-19 disease.

•
The 'Methodology' section describes the dataset used and the proposed method.

•
The 'Results' section shows the results obtained by our method.
• Finally, the 'Conclusion' section states the main findings of our work.

State of the art
Since the outbreak of the pandemic, there have been many efforts to analyse the behaviour of the scientific community through Bibliometry. Lou et al. [10] and Nasab and Rahim [11] analyse the relationships between authors and highly cited articles about both the COVID-19 disease and the SARS-CoV-2 virus, as well as the number of papers produced by country, providing empirical data that proves the growing interest that the pandemic has had among not only authors, but also journals of very high impact. In both papers, scientific publications are also broken down by different subject areas, with epidemiology and virology being the fields that have received the greatest number of publications. Although [3] also performs a bibliometric analysis where they reached similar conclusions as [10,11], the authors also provided an interpretation for the significant differences in publications observed by country, based solely on the gross domestic product (GDP) and the number of inhabitants. A high level of saturation in the healthcare system can have a negative impact on the number of publications in a country. This is the case, for example, in Italy. This massive amount of information about the pandemic requires useful strategies that can help facilitate the scientific community in finding the desired information, while weeding out those works that may not be relevant for them. The large number of pre-existing works in the highly specific fields in the world of Medicine [3,10,11] also makes it more difficult to find information on certain less well-studied areas, such as education or ethics.
In addition, the evolution of the pandemic has unleashed plenty of social repercussions that should be studied and taken into consideration, for example, the vaccine opposition and COVID-19 denial movements which have had a considerable impact on social networks [12,13]. Work-from-home has also become a trend during the outbreak and is paving the way for an important transformation in terms of labour relations [14]. The pandemic has also had repercussions on cities, where marginalised populations are receiving a disproportional impact on their health and well-being. Urban planning is also learning and growing from this situation as well [15].
Lou et al. [10] highlight the limitations of their bibliometric analysis under the current conditions, where a large number of articles on the subject of COVID-19 are being published, in different languages. Being such a current event, it is impossible to draw firm conclusions in a field so dependent on current events.
In order to facilitate the task of obtaining relevant information for researchers, it is possible to apply techniques from the field of NLP. IBM, for example, offers a service to extract relevant content from a corpus of articles about COVID-19, allowing researchers to ask specific questions about COVID-19 and analyse the related existing information. 1 NLP is also being used to analyse the social interactions caused by the pandemic. Cinelli et al. [16] extract topics from a corpus of text from different social networks and perform a discourse analysis to infer key concepts that have been used by the users of social media and the patterns of information dissemination. Lopez et al. [17] analyze Twitter with text mining techniques in order to conduct a multi-language analysis of the speech. In Singh et al. [18] an analysis of comments on Twitter about the pandemic was carried out but, this time, by analysing the dissemination of truthful information and misinformation through the social network. In Schild et al. [19] the authors analysed the phenomenon of sinophobia on 4chan and Twitter in relation to the pandemic through word embeddings, linking different news about Donald Trump, the WHO and the Chinese Government.
With the aim of helping researchers apply NLP in the search for information concerning the pandemic, Riloff et al. [20] have designed a toolbox that includes a set of English dictionaries with relevant concepts related to COVID-19. Latif et al. [21] list the different areas of study where the use of artificial intelligence and machine learning could be relevant, as well as a compilation of datasets, resources and initiatives carried out to improve the current knowledge about the disease.
Keyword extraction models have been widely used to classify different domains of knowledge in scientific articles [22]. We can define keywords from two different perspectives: sociocultural and statistical. Traditionally, any word that includes culturally and socially relevant concepts has been intuitively considered as a keyword [23]. From the point of view of corpus linguistics, keywords are extracted by using statistical processes, commonly comparing their frequency in the text to be analysed with their frequency in a reference corpus. Using this type of technique, three types of keywords are usually obtained: proper names, concepts that explain the content of the text, and frequent words such as pronouns and prepositions that can be used as style and not content indicators [24].
Some information retrieval systems have been created to help researchers find relevant information. Named entity recognition (NER) can be useful in collecting COVID-19 information from statements, which can be considered an alternative to document retrieval systems. Wang et al. [25] developed a web-based system to find textual evidence from COVID-19 document corpus.
CO-Search is an information retrieval system that is able to extract search queries from natural language questions and to retrieve scientific literature about COVID-19 [26]. CO-Search uses a SBERT model to create a latent space with queries and documents. CO-Search results are evaluated using TREC-COVID, an evaluation system that helps researchers find searching algorithms and information discovering methods to manage the existing literature around COVID-19 [27].
For researchers, collecting scientific literature is crucial in order to be up to date with the newest and most relevant knowledge for their research areas and to provide solid background knowledge that will allow them to effectively contribute to their field. The explosive growth of new scientific literature makes it difficult to identify suitable papers and is becoming an increasingly complex task [28].
When doing a literature review, researchers need to maximise the 'relevance' of the collected literature, but 'relevance' is not directly measurable, and some level of uncertainty is inevitable [29]. While information retrieval systems are nowadays an cornerstone for researchers, question answering systems are becoming a powerful tool that may help to find more relevant knowledge [26,30].

Dataset description
In March 2020, due to the COVID-19 global pandemic, the Allen Institute for AI coordinated by the White House Office of Science and Technology Policy and in association with several initiatives published CORD-19, 2 an open dataset of over 50,000 papers on COVID-19, SARS-CoV-2, coronavirus and other related study areas.
The idea behind the publication of this dataset was that after the increase in the academic literature on COVID-19 [8], the computer science community could apply text mining and natural language processing methods in order to digest and retrieve significant information and provide it to the medical research community.
The CORD-19 dataset contains multilingual information, but we have only dealt with English papers in this study. Thus, a new stage removing non-English information was applied.
After deleting non-English or duplicated papers and discarding papers without abstracts, titles, or references, we got a reduced dataset of 25,004 instances. Each instance of the dataset has the following information: title, authors, abstract, body text, references and publication date.

Latent Dirichlet allocation
Latent Dirichlet Allocation (LDA) is a generative statistical model used to extract the latent topic structure of text documents. LDA is a machine-learning technique used in different areas such as retrieval field, document classification and topic modelling [31].
In this article, we have implemented the LDA provided by the Scikit-Learn project: a machine-learning library for the Python Programming Language. 3

Method
In this section, we will present the main contribution of this study: an information retrieval system that allows users to extract relevant papers when given certain search terms. Our approach is based on keyword extraction using the LDA topic modelling technique on pseudo-documents generated from the papers in the dataset.
The idea of this work is to relate both concepts of keywords. By adding latent keywords in the dataset papers, we can broaden the search spectrum and obtain more relevant results. As in other Information Retrieval Systems, the user introduces input information to perform a query and obtain an output. The input information is a set of keywords containing the most important ideas that the user wishes to examine. In our method and thanks to LDA, we extracted new latent keywords that complemented the input provided by the user.
From now on, we will refer to the input keywords provided by the query as input_u. Those latent keywords extracted by LDA will be referred to as topic_terms.
To be noted, topic_terms set provides information about the latent structure of the corpus and is independent of the query. input_u set gives information about the query so that their main goal is to facilitate which terms and concepts from the corpus are desired by the user.
In order to obtain the topic_terms set, we have processed each instance of the dataset in three phases: Pseudodocument generation, Text preprocessing and Topic modelling. Figure 1 shows a schematic representation of our model.

Pseudo-document generation.
In this initial step, for every paper in the dataset, we aggregate it into a pseudodocument using the title of the paper, the text of its abstract and the titles of its references. The generation of pseudodocuments is a commonly used strategy to combat data sparsity [32].
The use of the title and the abstract when extracting key information from a paper is quite common since it contains many keywords in a very limited space. The text of the paper is not usually taken into account since its content is usually quite heterogeneous, and the results vary greatly depending on the section being analysed [33]. That is why we have aggregated the title and the abstract and not the full text into the pseudo-document.
References are also a useful source of information in articles. Bibliographical coupling and co-citation were used in the very first approach to study relationships between articles [34]. In addition, references have been used to extract key concepts of a document in order to generate better keywords [35]. In the context of this work, where keywords are absent in the dataset, it is important to extract key concepts in order to help us to analyse the latent information of each article.
3.3.1.1. Text preprocessing. In this phase, we conducted a series of tasks that aim to transform the text into a more digestible form so that the irrelevant information is reduced and the LDA model can perform better.
To preprocess a pseudo-document, we performed the following tasks: • The text is converted to lowercase, and the punctuation is deleted.

•
We applied a stop words filter: Stop words are words that do not contribute any meaning to the document, such as articles, pronouns and prepositions. Removing stop words avoids generating style indicator keywords that are not useful in the desired context. We have used the list of clinical stop words provided by Ganesan et al. [36].
• Each word is lemmatised: The lemmatisation process is a linguistic process by which the root of a word is determined. We have used the spaCy lemmatiser algorithm. 4 • For reasons of efficiency, only unigrams are used in our model, but some n-grams have been kept for relevant reasons, those n-grams are as follows: WHO, Public Health, Social Media, Fake News and Social Sciences.
• Finally, papers written in languages other than English are eliminated in order to reduce the computing time in the keyword topic modelling step.
3.3.1.2. Topic modelling. LDA analysis was applied for each article of the dataset in order to extract the latent topics hidden in the documents. It is important to note that we have not applied the same model for the whole corpus. A new LDA model was generated for each article. Our goal here was to detect minority topics in the dataset that often can be hidden in a single article. Excess noise from popular topics could be introduced if the same LDA model was applied to the entire corpus.
After completing the above steps, we extracted the latent topics from the corpus. These latent topics do not depend on the user input but rather the documents inside the dataset. In any case, our goal here was not to use the topics extracted by LDA, but to work with the terms inside these topics. So, we combined the most relevant terms from each topic into a set of latent keywords. This set is called LDA terms.
The following steps combined the latent structure of the corpus with the concepts provided by the query.
3.3.1.3. Co-occurrence matrix. After modelling LDA terms for each article, we computed a co-occurrence matrix between all the input terms (term_u) and all the LDA terms (term_topic) for the whole corpus. So that we could count the number of times that a term_u co-occurs with a term_topic, in the corpus.

3.3.2.
Adapted TF-IDF co-occurrence matrix. TF-IDF combines two metrics used in text information processing techniques: term frequency (TF) and inverse document frequency (IDF). TF is a metric used to represent the number of occurrences of a term in a document, while IDF indicates the number of times a term appears in a corpus. TF-IDF represents the importance of a term in the document. Considering important a term which appears frequently in a document but is rare in the corpus. This term can be used to represent the key information of the document. Yellow squares represent steps required to obtain latent keywords in the dataset. The grey square represents the step that combines the latent information from the corpus with the user input terms.
In our study, we translated the idea behind TF-IDF and used it to analyse the relation between term_topic and term_u. TF-IDF's goal is to link the information provided by topic terms and input terms. The TF-IDF score obtained by each of the topic terms would depend on the query and, therefore, on the input terms.
First, we computed the topic frequency TF as the number of occurrences of an LDA term in all the LDA topics for each paper where S i (term) = 1 if the term is in LDA terms {term_topic} i for the document i, and p is the total number of documents in the corpus. Then, we computed the topic inverse document frequency (TIDF) as the inverse frequency of the number of times that a term occurs in the list of term topics for each article where c is a constant number. A greater value of c would indicate a lower TIDF score in relation to frequent terms, while smaller c values would indicate a better TIDF score in relation to frequent terms. For our dataset, we have used c = 2 as the value that allowed us to extract latent terms with similar popularity to the user inputs. Readers can refer to the Supplemental material section to see further analysis of the behaviour of the c value. A deeper justification about the use of the adapted version of TF-IDF that we have proposed in our article will be further explained in the section Energy and c parameter behaviour.
We defined LDA-Term Co-occurrence (LTC) as the number of times that a term_u co-occurs with a term_topic. This value is provided by the co-occurrence matrix defined above where C is the co-occurrence matrix. Our adapted TF-IDF metric is expressed as follows ATF:IDF term topic , term u À Á = LTC term topic , term u À Á · TIDF term topic À Á ð4Þ In our method, LTC fulfils the same role as term frequency in TF-IDF [37]. We worked with term topics instead of documents, so the aim of LTC is to weigh terms inside of term topics. The more frequent a term is present inside a term topic, the more important this term becomes for that term topic. LTC is a simple way to measure the importance of a term within a term topic. In the same way, if a term is very frequent in each term topic, then this term is not relevant or may give us irrelevant information. Therefore, the objective of TIDF is to present a high score to less frequent terms. Finally, we combined LTC with TIDF results to get a high score from those very frequent terms in only a few terms topics which are, at the same time, very infrequently seen throughout the rest of the term topics.
Finally, we built an adapted TF-IDF co-occurrence Matrix (ATF.IDF) where T is a set with all LDA term topics found in the corpus and U are the input terms.

Keywords expansion.
In this step, we expanded the input terms with new topics extracted from LDA terms. From the ATF.IDF co-occurrence matrix built in the previous step, we computed the sum of the ATF.IDF scores for all the LDA terms topic as Then, we applied an energy threshold to the score. This energy threshold is a classical mechanism to preserve information until reaching a specific threshold score [38] threshold ACO, energy ð Þ = total score ACO ð Þ· energy ð7Þ Term topics are ranked according to their ATF-IDF score so that the term topics with greater scores appear first The next step computed the cumulative sum of all term topics, according to their score, and selected only the LDA terms whose cumulative ATF-IDF score was under the threshold. Finally, these selected LDA terms were added to the input terms set selectedTerms ACO, energy ð Þ = t 1, t 2, , :::, t n f g f t j ∈ LDA^cumsum score t j À Á À Á < threshold ACO, energy ð Þ ð 9Þ We can thus repeat all the steps again using the new input terms in order to find new latent keywords. An example of keywords expansion can be seen in Table 3.

Document filtering.
For this final step, documents were filtered according to the expanded input terms set ( term u f g). So the filters were built using the initial user input terms, and the expanded input terms derived from the latent structure. Next, we extracted all the documents from the corpus that contained the keywords. The minimum number of keywords that had to be in a document can be set as an additional threshold. In our case, we have imposed a minimum of two keywords.

Time complexity
The time complexity was mainly affected by the execution of the LDA method and is O p · mnt + t 3 ð Þ ð Þwhere p is the number of LDA documents in the corpus, m is the number of topics, n is the number of terms, and t = min m, n ð Þ [39]. While this topic is not directly related to our work, there are some proposals to reduce the time complexity of LDA algorithms [39,40] that can be useful for readers.

Experimental set-up
In this article, we have executed the proposed method in three iterations, in order to extract three levels of keywords. After performing a hyperparameter optimisation search using the Grid Search method provided by the Scikit-learn library, 5 the best energy threshold for our dataset was 0.025.
The c parameter of TIDF was 2 (Please refer to Section 4 and check the Supplemental material of this article for more details about ATF.IDF).
In respect to LDA, the number of topics was set to 30, which gave us good results in terms of coherence. We used the UMASS-Coherence to determine the number of topics [41].

Results
In this section, we will proceed to show the results obtained using our proposed method. The goal of our Information Retrieval System was to find an answer to the question: 'What has been published about ethical and social science considerations?'. In addition, specifically, we wanted information from seven different thematic areas (See Table 1). 6 As explained in previous sections, the dataset provided by Kaggle does not contain information about the keywords of the articles, so we could only work with the title, abstract and references of each article. The absence of keywords is a challenge when identifying the category in which an article is framed, since the title and abstract do not always contain enough information on all the topics covered by a scientific work.
In this context, evaluating the performance of our method can be a challenging problem. Fortunately, the provided dataset contains enough information to identify the articles that compose the corpus; thus, we were able to perform a manual evaluation in two steps: 1. A priori evaluation: By checking the titles and abstracts of the selected article, it was verified whether the articles selected by our method could answer the question posed. This evaluation was performed exclusively using the data provided by the dataset.

2.
A posteriori evaluation: By manually obtaining each one of the selected articles, the articles were checked to see whether they would be able to answer the question posed. This evaluation was performed by using external data to the provided corpus by the dataset.
In addition, keywords of the selected articles were extracted and compared with the topics generated with our method in order to check the level of co-occurrence. Figure 2 shows an example of co-occurrence matrix generation.
For each thematic area, a list of keywords was chosen, as shown in Table 2. These keywords are the initial input terms that compound our queries and were chosen manually.

A priori evaluation
During 'a priori' evaluation, the top 10 most relevant papers for each thematic area were selected. We performed two surveys with the goal of checking whether the filtered papers were able to answer the question posed in the corresponding thematic area. To this end, in the first survey, the participants had to evaluate whether the title and the abstract of a set of articles were related to a thematic area. Thus, participants were only allowed to see the same information used by the method to filter and retrieve the selected papers. Participants were randomly assigned to two thematic areas and had to read the title and abstract of ten articles retrieved by our method for the assigned thematic areas.
There were three possible answers: 1. The article fits the thematic area. 2. The article does not fit the thematic area.
3. With the provided information, we cannot know whether the article fits the thematic area or not. Efforts to support sustained education, access, and capacity building in the area of ethics. 4 Efforts to establish a team at World Health Organization that will be integrated within multidisciplinary research and operational platforms and that will connect with existing and expanded global networks of social sciences. 5 Efforts to develop qualitative assessment frameworks to systematically collect information related to local barriers and enablers for the uptake and adherence to public health measures for prevention and control. This includes the rapid identification of the secondary impacts of these measures. (e.g. use of surgical masks, modification of health-seeking behaviours for SRH, school closures). 6 Efforts to identify how the burden of responding to the outbreak and implementing public health measures affects the physical and psychological health of those providing care for COVID-19 patients and identify the immediate needs that must be addressed. 7 Efforts to identify the underlying drivers of fear, anxiety and stigma that fuel misinformation and rumour, particularly through social media. Answers 1 and 2 helped to evaluate an article as a true positive and a false positive, respectively. Table 4 shows the results obtained in the survey per thematic area. The range of different responses given by the participants can be seen in Figure 3.
The survey was answered by 57 participants, all of them use Amazon Mechanical Turk, 7 a web platform where users get an economic reward for doing tasks that require human intelligence. All participants were native English speakers. The time to complete the survey and response patterns were analysed in order to prevent random responses. Table 4 shows the responses obtained in the survey. For all the thematic areas, the first answer (The article fits the thematic area) was the most common, being selected for the majority of articles. According to our results, the second thematic area ('Efforts to embed ethics across all thematic areas, engage with novel ethical issues that arise, and coordinate to minimize duplication of oversight'.) obtained the best results, with about 81% of users answering with the first answer while the last thematic area ('Efforts to identify the underlying drivers of fear, anxiety, and stigma that fuel misinformation and rumors, particularly through social media'.) achieved the worst results, with a score of only 67.144% for the first answer. Figure 3 shows the answer interval for every thematic. That means that, for example, in the first thematic area there was an article that was classified as a true positive by 50% of the participants. At the same time, there was an article that was classified as a true positive by 100% of the participants assigned to that thematic area. In addition, in the  Supplemental material section, we have included a Figure where readers can get additional information about the precision obtained in the survey results.
In the second survey, we randomly divided the same papers from the first survey into five groups, ensuring that each group had two papers per subject area (14 papers per group). Then, we arbitrarily assigned each participant to a single group.
Participants then had to read the abstract and the title of each paper in their assigned group and assign a thematic area. They also had the option of not assigning any thematic area if they felt unable to classify the paper.  Table 4. Responses per thematic area (See Table 1 for information about each thematic area).    For the second survey, we had 65 participants from Amazon Mechanical Turk and all participants were native English speakers. As in the previous survey, we analysed the time and response patterns to prevent random responses. Results from the second survey are presented in the confusion matrix shown in Table 5. From this confusion matrix, some metrics have been extracted and can be found in Table 6. As we can see, precision is, again, obtaining high scores. At this point, it is important to state that the second survey required participants to perform a harder task than in the first survey, so lower scores were expected. The recall also had high scores except for the third thematic area, where results were < 0.5. Also, accuracy and F-scores were over 0.5 points for each thematic area.
Precision, recall, accuracy, true/negative rates and F-scores were calculated by using definitions from Powers [42]. Despite the results obtained, 'a priori' evaluation has several limitations. First, it is still difficult, even for humans, to decide what the content of an article is by only checking its title and abstract. Second, while there are documents with very long abstracts, other articles have short or even non-existent abstracts. In addition, the dataset did not exclusively contain scientific papers, and it was possible to encounter editorial articles and other documents whose summaries were not homologated with abstracts.
In any case, 'a priori' analysis can be very useful in evaluating results obtained when working with the same conditions as used in our method.

A posteriori evaluation
To perform the 'a posteriori' evaluation, we downloaded and manually classified 150 papers by using the following guidelines: 1. Check whether the title, abstract and references contained terms or topics related to the thematic area. 2. Check whether keywords matched topics from the thematic area. 3. Check whether conclusions contained keywords or topics related to the thematic area. 4. Read the full article if the previous steps did not provide enough evidence to make a decision.
After this manual classification, from the subset of 150 previously classified papers, we randomly extracted 10 papers per thematic area and tested the performance of our model in terms of precision, accuracy, recall and F-scores. This type of evaluation allowed us to more precisely know whether an article was appropriately selected because we could access Table 7. Confusion matrix presenting results from 'a posteriori' results.  information from the whole document (title, authors, keywords, abstract, references and text). 'A posteriori' evaluation was then performed by the authors. 'A posteriori' results are described in Tables 7 and 8.
If we compare both evaluation strategies, 'a priori' (second survey) and 'a posteriori', as noted, the 'a posteriori' evaluation gets better scores for nearly all thematic areas. In T5, both evaluations scored a 100% in terms of precision.
The lack of information during the 'a priori' evaluation (first survey) was clearly reflected during the second step of the evaluation process. For example, for the first thematic area, one of the articles retrieved by the method was titled: 'Understanding perceptions of global healthcare experiences on provider values and practices in the USA: a qualitative study among global health physicians and program directors' [43].
The title and the abstract of the article did not contain words explicitly related to the thematic area like 'ethics', 'principles' and 'standards' so that it was more difficult to evaluate whether the article fit the topic or not. In addition, the article was not about COVID-19 disease or any other pandemic. Nevertheless, by carefully checking the article, it was clear that the article was a true positive. This article achieved the worst results in its thematic area, where only 50% of the participants believed that the article fit the topic, while 40% of participants believed that the article did not fit the topic. Meanwhile, our method retrieved this article as one of the top 10 for the thematic area.
From these results, we can say that it seems that using certain methods to extract the latent structure of a document can help us establish relationships between words that are not necessarily evident to the human mind. Though this is not the scope of our work, further analysis should be done.
On the contrary, there are two thematic areas where 'a priori' evaluation overcame 'a posteriori' results. For example, in T4, our method retrieved an article titled 'The Highest Cited Papers on Brucellosis: Identification Using Two Databases and Review of the Papers' Major Findings' [44]. This article is a bibliometric analysis but is not connected with key concepts to this thematic area, like 'Web Health Organization' or global network of social sciences, and yet most participants answered that the article fit the topic.

Energy and c parameter behaviour
As mentioned before, energy and c were parameters used in equations (7), (9) (energy), and equation 2 (c). We performed an 'a posteriori' evaluation strategy to study the behaviour of energy and the c parameter. According to our tests, the best performance was achieved when c was 2 and energy was 0.02. Nevertheless, it is important to note that every thematic area had a different score. For example, for thematic areas 2 and 3, c = 3 gave a better performance, while energy = 0.25 was the best option for thematic area 2. Figures 4 and 5 illustrate the behaviour of c and energy parameters for each thematic area, while Figure 6 shows an average F-score across all thematic areas.

The role of ATF.IDF
In Section 3, we introduced ATF.IDF, and we adapted the TF-IDF proposal. Figure 7 shows a comparison of the keyword expansion stage according to the value of c in terms of popularity. While Figure 7 groups terms by their stage in the method, Figure 8 shows the same information but clusters terms by popularity. The popularity of a term is the number of articles in the corpus where the term appears. For example, if term A appears in 2500 articles and term B appears in 12 documents, the popularity of term A (popularity = 2500) is greater than the popularity of term B (popularity = 12). At the same time, we can say that term B is a more specific term than term A.
In both figures, we analysed the input terms of thematic area number 5 (See Table 2) If we use c = 1, the input terms are expanded with the LDA topics described below: Finally, in Figure 8, elements represented with a grey mark are terms that had relations with the query but were not selected.
When using c = 2, expanded terms were represented by a diamond in Figure 7 for terms extracted during the first iteration and with a star for the second iteration.   . Surface plot representing F-score evolution according to c and energy parameters. The F-score is measured as the mean F-score for whole thematic areas.
In the corpus, 'health' and 'outbreak' were examples of very popular terms, which appeared throughout the corpus. For that reason, their frequency score was very high. We can observe how the second iteration takes very generalist terms into account. At the same time, expanded keywords are very far from the initial input terms.
For that reason, it is relevant to give a higher preponderance to specific terms by setting a proper value to c. This value will depend on the corpus structure and their word distributions in the corpus where a small number of concepts tend to be present in almost every document, greater values of c will be needed. Nevertheless, setting and a high value of c can lead to ATF.IDF values that are very close to zero, resulting in a random selection.
In our corpus, c = 2 was providing a good preponderance for specific terms, and during the first iteration, the expanded terms were very close to the initial query. In addition, the second iteration expanded the query to two terms that were even more specific than the user input terms. This behaviour was the opposite when c = 1, where the terms in the second generation were less specific than the first generation.
As we can observe in Figure 7, when c = 2, more terms were selected during the first iteration, and they were very similar in terms of popularity. When c = 1, fewer terms were selected during the first iteration, and popularity dispersion increased.
Initial query terms had very different popularity, but only the two most popular terms appeared in more than 10 3 documents. At the same time, all the terms selected for c = 1 were very popular, with all of them appearing in more than 10 4 documents. This situation contrasted with terms selected by ATF.IDF when c = 2, where the terms popularity was in the interval between 10 2 and 10 3 documents.
To see results for other thematic areas, please refer to the Supplemental material of this article.

Conclusion
In the current context of the COVID-19 pandemic, a massive influx of scientific knowledge is being produced. This amount of information makes it difficult for researchers to search for useful information.
Keywords are a very important source when identifying articles that fall within a specific category. In our method, we used LDA to generate keywords that could identify latent concepts existing in a scientific article. The co-occurrence analysis between input terms and LDA terms allowed us to retrieve articles that were close to a series of keywords related to the subject of study. ATF-IDF played a key role in the co-occurrence analysis, acting as a selection function and extracting query-related specific concepts while filtering generic ones.
Two different validation strategies were carried out: 'a priori' and 'a posteriori' evaluations. Both of them were performed by humans, and all in all, results were satisfactory. Therefore, our method can be a useful tool for identifying articles on very specific topics that might occupy less popular places within a corpus.
'A posteriori' evaluation, specifically, improved the performance scores obtained by 'a priori' evaluation by identifying relationships between concepts that were not as evident for humans as for latent models. Our method can be a valuable tool when used as an Information Retrieval System, with the capacity to focus on retrieving specific information about complex topics in a dataset where the thematic area has a secondary role. Nevertheless, we have performed our analysis working with the corpus provided by the White House in collaboration with the Allen Institute, where only abstracts and titles were presented in the corpus so that the full semantic information from papers has not been analysed. This limitation should be taken into consideration.
For future works, it is necessary to analyse the behaviour of this method in different thematic areas and to test our work with a full semantic information corpus.