{'fraction_non_alphanumeric': 0.031548099638168715, 'fraction_numerical': 0.020184803086032117, 'mean_word_length': 4.828163123039387, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
From Protocol to Screening: A Hybrid Learning Approach for Technology-Assisted Systematic Literature Reviews

In the medical domain, a Systematic Literature Review (SLR) attempts to collect all empirical evidence, that fit pre-specified eligibility criteria, in order to answer a specific research question. The process of preparing an SLR consists of multiple tasks that are labor-intensive and time-consuming, involving large monetary costs. Technology-assisted review (TAR) methods automate the different processes of creating an SLR and they are particularly focused on reducing the burden of screening for reviewers. We present a novel method for TAR that implements a full pipeline from the research protocol to the screening of the relevant papers. Our pipeline overcomes the need of a Boolean query constructed by specialists and consists of three different components: the primary retrieval engine, the inter-review ranker and the intra-review ranker, combining learning-to-rank techniques with a relevance feedback method. In addition, we contribute an updated version of the Task 2 of the CLEF 2019 eHealth Lab dataset, which we make publicly available. Empirical results on this dataset show that our approach can achieve state-of-the-art results.


Introduction
Systematic literature reviews (SLRs) in the medical domain seek to collect evidence from research publications that fit strict pre-specified eligibility criteria to answer a specific research question. They aim to minimize bias by using explicit, systematic methods documented in advance with a protocol (Higgins et al., 2019). Clinicians practice evidence-based medicine (EBM) by integrating their expertise with the best available external clinical evidence from SLRs (Sackett, 1997).
The process of preparing an SLR consists of multiple tasks that can be organized into four general stages: preparation, retrieval, appraisal, and synthesis (Tsafnat et al., 2014). The preparation stage includes the tasks of developing the research question, searching for relevant SLRs, writing the needed protocol, and defining a search strategy. The search strategy includes Boolean queries adapted for each medical database. These Boolean queries typically have very complicated syntax and are usually built by highly trained information specialists. The queries are submitted to medical databases during the retrieval stage, resulting in a vast set of possibly relevant studies . Subsequently, every study is screened in the appraisal stage, using the title and abstract (abstract-level assessment), and irrelevant studies are removed. Additional assessment is conducted based on the full-text (contentlevel assessment) of the remaining studies. In the last stage, data are extracted, converted, and synthesized from the relevant studies. The final scientific paper incorporates all this data with the addition of a meta-analysis of the included studies.
Current methods of creating an SLR are labor-intensive and time-consuming, involving large monetary costs. On average, an SLR costs approximately more than $140K and the time a scientist spends to complete it is 1.72 years (Michelson and Reuter, 2019). The most cost-intensive and time-consuming part when creating a systematic review is the screening process, i.e. the appraisal stage. On average, more than 5,500 documents are returned from the databases and less than 4.7% (3%) of them are relevant at the abstract (document) level (Borah et al., 2017;Kanoulas et al., 2018). Technology-assisted review (TAR) is a relatively new computer science field that employs information retrieval, machine learning and natural language processing (NLP) techniques, which are usually combined with domain specific knowledge, to reduce the workload on screening for SLRs (Miwa et al., 2014).
The primary contribution of this work is a novel method for TAR with state-of-the-art results. Our method implements a full pipeline from protocol to screening papers with state-of-the-art results, assisting the researcher in three parts of the process of preparing an SLR: the preparation, the retrieval, and the appraisal. The key novel characteristics of our method are: it relies solely on the protocol of a systematic review, overcoming the need for constructing a specialized Boolean query (preparation). it incorporates domain specialized features deriving from the latest advances of the NLP field (retrieval and appraisal).
As a secondary contribution, we publish an updated version of the Task 2 of the CLEF 2019 eHealth lab dataset . The updated dataset fixes previous format issues and provides an up-to-date version of the dataset that includes the latest revisions of the included SLRs.
This work is an extension of a previously published conference paper . Specifically, we extended our previous pipeline with a primary retrieval engine, fine-tuned the inter-review ranker features and adopted sentence embeddings for both inter-review and intra-review rankers. Our approach was one of the top approaches in Task 2 of the eHealth Lab of CLEF 2017 (Anagnostou et al., 2017) and CLEF 2018 .
The rest of this paper is organized as follows: After a discussion of the related work in Section 2, we introduce our approach in Section 3. In Section 4, we describe our case study and the corresponding dataset, and then discuss the results of our study, including a comparison of our method with the state of the art. Finally, in Section 5, we conclude this work and draw future directions.

Related Work
Several research papers have been published in the past on applications of text mining to assist in identifying relevant studies for a systematic review. Most of them are focused on reducing the number of studies needed to screen, increasing the speed of screening, and improving the workflow through screening prioritization. Early studies were focused on classifying documents as relevant or not to a review topic  and dealing with imbalanced datasets (Cohen, 2006). Later studies evaluated active learning techniques to deal with class imbalance (Wallace et al., 2010a,b;Miwa et al., 2014) and several studies exploited the advantages of using the Naive Bayes algorithm (Bekhuis and Demner-Fushman, 2012;Matwin et al., 2010;Frunza et al., 2010Frunza et al., , 2011. Furthermore, researchers experimented with algorithms such as EvoSVM (Bekhuis and Demner-Fushman, 2010) and k-nearest neighbors (Miwa et al., 2014) and different representations such as visual data mining (Felizardo et al., 2013) and Latent Dirichlet Allocation (LDA) (Miwa et al., 2014). Finally, Cohen et al. (2009) introduced an approach, combining topic-specific training data with data from other SLR topics and Karimi et al. (2010) were the first to compare ranked retrieval with Boolean querying.
The wide variety of datasets, algorithms, and evaluation methods explored in the above studies makes it difficult to draw any conclusions about the most effective approach. To address this issue, the CLEF eHealth Lab organized a task on Technology-Assisted Reviews in Empirical Medicine (Kanoulas et al., 2017 from 2017 to 2019. The task was aiming to bring together academic, commercial, and government researchers conducting experiments and sharing results on automatic methods to retrieve relevant studies. Lab participants were provided with a set of systematic review topics that were constructed by Cochrane 1 experts. Each topic contained the title and protocol of a systematic review and the corresponding Boolean query. The task was divided into two sub-tasks. In the "No Boolean Query" sub-task, participants had to complete the search effectively and efficiently bypassing the construction of the Boolean query. Therefore, participants had to first retrieve the documents from PubMed. In the second sub-task, called "Abstract and Title Screening", participants had to produce an efficient ordering of the documents, such that all of the relevant abstracts are retrieved as early as possible. The set of documents returned from the submitted Boolean query were also provided in this case. There was a great interest in this task, with many teams participating and presenting different and specialized approaches. Most of the participants proposed different learning-to-rank approaches (Hollmann and Eickhoff, 2017;Scells et al., 2017;Chen et al., 2017), while others also adopted active learning Grossman, 2017, 2018;Yu and Menzies, 2017) and sampling techniques (Di Nunzio et al., 2017Nunzio, 2019;Li and Kanoulas, 2019). Two teams worked with neural networks and deep learning (Singh et al., 2017;Lee, 2017). Furthermore, participants represented the textual data in a variety of ways, including topic models (Van Altena and Olabarriaga, 2017;Kalphov et al., 2017), TF-IDF (Alharbi and Stevenson, 2017;Alharbi et al., 2018;Alharbi and Stevenson, 2019), n-grams (Norman et al., 2017(Norman et al., , 2018Cohen and Smalheiser, 2018) and text embeddings (Hollmann and Eickhoff, 2017;Chen et al., 2017;Wu et al., 2018).
Recently, Zou et al. (2018); Zou and Kanoulas (2020) proposed an approach that looks for entities in the documents and asks questions to the users to retrieve the last relevant documents. Also, Scells et al. (2020b) proposed a computational approach to objectively derive search strategies for systematic reviews and also presented a novel approach that ranks documents for systematic review literature using rank fusion applied to coordination level matching by taking advantage of the boolean query (Scells et al., 2020a).

Our Hybrid Learning Approach
This section provides a detailed description of our hybrid learning approach for screening prioritization in systematic reviews. Our approach does not require the construction of a Boolean query by specialists, and consists of three consecutive components: initial retrieval, inter-review ranking, and intra-review ranking. Figure 1 illustrates our approach in detail.

Primary Retrieval Engine
In the first step of our approach, an initial retrieval of the relevant documents is performed using a traditional IR system based on the BM25 score. The title and the objectives of the systematic review, as defined in its protocol, are separately given as queries to the IR system. The final ranked list of documents is the combination of the retrieved documents from both queries. The normalized score for each query are summed to produce the final ranking score for each document. The aim of this component is to retrieve a large number of documents from a database in order to achieve very high recall, while at the same time reducing the number of documents processed in the following steps. In the case of a systematic review, the primary retrieval engine decreases the possibly relevant documents in the biomedical databases from millions to tens of thousands.

Inter-Review Ranker
The second component of our approach, the inter-review ranker, aims to bring the relevant documents at higher ranking position. This step includes a learning-to-rank (LTR) model that ranks the set of documents retrieved by the primary retrieval engine, according to their relevance and importance for the review topic. Each document is represented as a feature vector, where each feature indicates how relevant the document is with respect to the review topic. The LTR model is trained on previously produced SLRs. The main idea behind this model is that it can grasp the knowledge across different SLRs and then be able to produce an efficient document ranking for an unknown SLR. The list of features includes traditional scoring functions such as BM25 and TF-IDF, LETOR inspired features (Qin et al., 2010) and semantic fea-tures using Word2Vec (Mikolov et al., 2013) and a novel feature deriving from Sent2Vec (Pagliardini et al., 2018). The features are computed using the different fields of the systematic review's protocol, and the title and abstract of the documents. The full list of features is presented in Table 3.2. Details about the features are given below: 1. We consider two fields of a document d: the title and the abstract. Column "Document field(s)" indicates whether these fields are used separately (,) or concatenated into a single string (+). 2. We consider multiple protocol fields based on the type of the systematic review. We denote a protocol field as p. In our study, 4 different types of systematic reviews are considered; Diagnostic Test Accuracy (DTA), Intervention, Qualitative, and Prognosis. All types include a Title, Objectives, Types of Studies, and Types of Participants fields. DTA reviews further include the fields Index Tests, Target Conditions, and Reference Standards. Intervention reviews include the fields Types of Intervention and Types of Outcome Measures and Prognosis reviews include the Type of Outcome Measures field. 3. The number of occurrences of a protocol field's token p i in a document is denoted as c(p i , d). 4. The BM25 score is computed as in (Robertson, 2010). 5. Singular Value Decomposition (SVD) is performed upon the tf-idf. The cosine similarity is estimated from the reduced vectors of the two fields. 6. The Word Mover's Distance (WMD) of the word vectors is computed as in (Kusner et al., 2015). 7. Sentence embeddings are produced by a Sent2Vec pre-trained model (Chen et al., 2019).

Intra-Review Ranker
The intra-review ranker is the last component and last step of our approach and employs the screening process conducted by the researcher, during the appraisal stage. This step consists of a simple supervised learning model that is continuously (re)trained based on the reviewer's relevance feedback. For this classifier, sentence embeddings were extracted, from a pre-trained Sent2Vec model, to represent the documents. To the best of our knowledge we are the first consider such embeddings for TAR. Initially, the intra-review ranker is trained on the top-k documents as ranked by the inter-review ranker and assessed by the reviewer. If no relevant documents are found in the top-k documents, the review continues until the training set consists of both relevant and irrelevant documents. Then, iteratively re-ranks the rest of the documents, expanding the training set of the intra-review model with the top-ranked document, until the whole list has been added to the training set or a certain threshold is reached. The expansion of the training set is configured by 4 parameters. Two thresholds, t init and t final , are defined. After training with the  initial k-documents, the training set is expanded until t init is reached using a step s init . Then, s init is increased to s final and the expansion of the training set continues until t final . This iterative feedback and re-ranking mechanism is described in detail in Algorithm 1. The use of different steps and thresholds reduces the cost of feedback and the time needed to produce predictions since the classifier is considered sufficiently trained when the training set has reached a certain number of documents. Moreover, this procedure allows the researcher to set a specific number of documents to be assessed, while taking into consideration the human resources and cost required.

Empirical Study
This section describes the dataset we used for our study and details the updates we implemented to it. Furthermore, it presents our evaluation process for each of the individual components of our hybrid learning approach and specifies the parameters and tools used for our experiments. Finally, it discusses and compares our results with other approaches presented in the past.

Data
Our data come from Task 2, TAR in Empirical Medicine, of CLEF e-health lab series from 2019, which extends the data of the 2017 and 2018 versions Algorithm 1: Iterative relevance feedback algorithm of the intrareview ranker Input : The ranked documents R, of length n, as produced by the inter-review ranker, initial training step k, initial training step s init , secondary training step s final , step change threshold t init , final threshold t final (optional) Output: Final ranking of documents R -f inalRanking 1 f inalRanking ← ∅; of the lab. (Kanoulas et al., 2017. The training set consists of 90 systematic reviews: 70 Diagnostic Test Accuracy (DTA) studies, and 20 Intervention studies. The test set includes 7 DTA, 16 Intervention, 2 Qualitative, and 1 Prognosis studies. Each SLR in the dataset includes its protocol and two Qrel files (list of relevant documents) for the abstract-and content-level assessment respectively. All the SLRs can be found in the Cochrane Library 2 and the initial dataset that was provided by the organizers of the task is available on GitHub 3 .
After experimenting with the dataset, we noticed several issues both in the format of the dataset (i.e. misspelled tags in XML and folder names) and the integrity of the relative documents (i.e. SLRs are updated over time, have very few relevant documents, SLRs existing in both train and test set, deleted PMIDs and others). Therefore, we've updated the dataset by fixing all the format issues that we found and updated the content-level qrels by scraping the Cochrane library website. The abstract-level qrels could not be updated since documents returned by the boolean query are not available online and were initially provided by the organizers. We aimed to provide an up-to-date version of the dataset that will engage and motivate more researchers on technology assisted reviews.
For updating the qrels we scraped the included studies from the reviews' web page for each of the total 116 SLRs 4 . The majority of the referenced studies included the corresponding PMID. For those references missing the PMID we followed a similar procedure as the organizers of the e-health lab . The title of the reference with the missing PMID was submitted to the PubMed Search Engine 5 . If there was a match, the PMID of the returned document was included, if not, the returned result was examined further until a match was found. All other studies, without a match, were discarded under the assumption that these are not contained in PubMed.
The updated qrels contain 14.29% more documents than the previous qrels with 628 new PMIDs and 126 discarded PMIDs. The updated GitHub repository 6 contains the updated qrel files for the 2019 dataset, several format corrections, statistics on the new dataset and the references scraped from the Cochrane Library website along with their PubMed link and PMID.

Hybrid Learning Implementation
In this section, we describe all the different tools, libraries, and settings we used in order to implement and evaluate the approach presented in Section 3.
As our primary retrieval engine, we use the Elasticsearch search engine 7 along with the PubMed 2019 annual baseline 8 . Our index includes more than 29.100.000 articles. For each systematic review, we submit two queries to the Elasticsearch using the title or the objectives of an SLR's protocol. The queries are formulated using a dictionary created from the most common words in PubMed. Words with frequencies higher than 50% or with less than 10 counts were discarded. The term threshold was set to 100k. The returned documents from both queries are combined into a single list scored by the normalized rank of the individual lists. For each SLR, 100k documents were returned with the intention to achieve high total recall. The final query formulation (i.e. combination of two queries) was selected based on recall on the train set, after experimenting with other queries, such as single query, concatenation of queries, and different dictionaries.
For the second component of our approach, the inter-review ranker, we use the LambdaMart algorithm as implemented in the XGBoost (XGBRanker, default parameters, tree method="gpu hist") (Chen and Guestrin, 2016). For each document, retrieved by the primary retrieval engine, a vector with the features from Table 3.2 is created. The TF-IDF was constructed using the same vocabulary as in the initial ranking. The pre-trained Word2Vec model from the BioASQ challenge, trained on the PubMed abstract (Pavlopoulos et al., 2014), and the BioSentVec model (Chen et al., 2019), trained both on PubMed and MIMIC III clinical notes, were adopted for the Word2Vec and Sent2Vec model respectively.
Finally, we employ the intra-review ranker. The classifier used is a Linear SVM (scikit-learn library, regularization parameter C = 0.5) and the features for each document were extracted using the pre-trained BioSent2Vec model (Chen et al., 2019) applied on the concatenation of the title and the abstract of the article. As default parameters, we set k = 10, s init = 1, t init = 200, s final = 50 and t final = 1000. Thus, the top-10 ranked documents are obtained from the inter-review ranker and screened by the reviewer. The intra-review ranker is then trained and the document ranking is updated. The reviewer continues to screen the documents and for each document screened a new ranking is produced by continuously retraining the intra-review model. The process continues until 200 documents are reviewed. After that, the intra-review model retraining and update of the ranking of the documents occur every 50 documents reviewed. The process is repeated until the final threshold of 1000 documents is reached. We assume that reviewers always classify correctly a document as relevant or not.

Evaluation and Results
Our evaluation process is two-fold. We first evaluate the different components of our hybrid learning approach, presented in Section 3, and then, we compare our results with other state-of-the-art approaches. We aim to determine whether the retrieval of relevant documents for technology assisted reviews is pragmatic and efficient without the use of a boolean query. In both cases, we use the updated content-level relevant documents, final studies included in the studies after the full-text screening, and we evaluate our approach using both the train and the test set. Table 2 presents the results for each of the components of our approach using the train set. We used the leave-one-out method on the 90 SLRs included in the training set and evaluation measures are computed using the script provided by the organizers of the Task 2 of CLEF e-health lab (Kanoulas et al., 2017). The table includes the Recall@threshold where threshold= {10, 50, 100, 5000} documents, Mean Average Precision (MAP), Work Saved over Sampling at 100% recall (WSS@100)  and last rel. The metric last rel is the minimum number of documents returned to retrieve all relevant documents. The Recall@100k, which is the maximum recall our approach can achieve, is 0.9224. We consider recall@5000 documents as the maximum valid threshold. We believe a higher threshold than 5000 to be unrealistic in a real-world case scenario. In total, 4.645.817 articles were retrieved, by the primary retrieval engine (100k documents per SLR), after duplicates removal. Comparing the primary retrieval engine, with the inter-review ranker, we first notice that retrieving such an amount of documents helps us reach very high recall which is our main concern for this problem. However, the primary retrieval engine the Re-call@5000 remains low (0.7401). The gap between those thresholds is diminished by the next step of our approach, the inter-review ranker, which reaches a recall@5000 of 0.8339. Till this step, no relevance feedback from the reviewers is used. A similar increase occurs also at thresholds 10, 50, and 100 of recall. Likewise, the MAP, WSS@100, and last rel metrics are also improved. Notably, the last rel is decreased by more than 16000 documents. This is a clear indication of the significance of the inter-review ranking on bringing all the relevant documents higher in the ranking and, thus, more accessible by the reviewer on the final stage of our approach.

Components Validation
Regarding the intra-review ranking, we observe an additional improvement in all metrics. A great indication of this improvement is the last rel metric, where the minimum number of documents returned to retrieve all relevant documents is decreased to 2391 from 16693 documents, and the recall@5000 reaches a very high recall, 0.9100, which is suited for this task.
We also test our approach on the dataset's test set. Table 3 presents the results of each of the components of our hybrid learning approach. The interreview ranker was trained with the 90 SLRs of the training set and the parameters were set as described in Section 4.2. The initial retrieval component achieves a recall@100k documents of 0.7595, and a total of 2.172.204 were retrieved. We first notice that the recall@100k is much lower compared to the results of the training set. However, all the other metrics continue to improve as our hybrid methodology progresses.
We look further into this issue of the low recall by reporting the results separately for each type of SLR in Table 4. The Recall@100k for DTA, Intervention, Prognosis, and Qualitative is 0.9385, 0.8797, 0.2897, and 0.7667 respectively. We notice that Prognosis achieves much lower results on all the metrics compared to the other types, due to very low recall from the initial retrieval stage. Our dataset consists only of 1 prognosis SLR with 321 relevant articles out of a total of 1414 documents across all types of SLRs which justifies the low metric scores when averaged across all SLRs. Further research is needed to identify any possible oddities on the nature of prognosis SLRs. DTA SLRs achieve the highest results in all cases that can be attributed to the fact that the train set consists mostly (70/90) of DTA SLRs. Our assumption that the types of SLRs in the training set greatly affects the inter-review ranker's performance is also supported by the much lower scores of the Qualitative SLRs.

Feature importance & Parameter setting
In an attempt to better understand how our ranker components make predictions we look into the top-15 features of the inter-review ranker and how the predictions are affected by the different threshold parameters of the intrareview ranker. Table 5 presents the top-15 features sorted by the feature's importance scores from the XGBoost ranker used in the inter-review model. We first notice that traditional features, such as TF-IDF and BM25 based features, populate most of the positions in the list, with the highest score being held by the Z-score of BM25. The LETOR inspired features hold a lower position in our ranking while the newly introduced feature, the cosine similarity of Sent2Vec embeddings, is ranked second. This upholds to the trend of the significance of semantic word/sentence embeddings. Finally, the field used either in document or protocol level doesn't imply a notable influence on the final performance since all fields appear in different positions in the list. Table 6 shows the results of the intra-review ranking on the training set using different thresholds, t init , t final . We keep the rest of the parameters constant with k = 10, s init = 1, and s final = 50, since k is increased if both relevant and irrelevant documents are not present and small changes in s init , s final do not affect the results. Both step parameters are associated with the computational cost of this methodology and their effect is out of the scope of this paper. In an aim to keep the computational cost in coordination with a real case sce- nario, we adjust the parameters to the above values so as to have continuous and constant retraining, until a relatively low threshold is reached, and more "distant in time" re-trainings after our training is considered to be enough for effective ranking. After taking into consideration the above, we conclude that the best parameters are t init = 200, t final = 2000 since they achieve the second-best performance in all metrics and the relevance feedback needed is much lower than this of the best performing parameters (2000 documents vs 5000 documents). In a real-world case, all the parameters can be adjusted according to the resources and the outcome during the assessment.

Comparison with state-of-the-art method and a baseline
We compare our approach with three other approaches, the AUTO-TAR BMI method by Cormack and Grossman (2018), which is considered state-of-theart, a variation of our Hybrid methodology (Hybrid TFIDF) which we have previously submitted at Task 2 of CLEF e-health lab ) and a simple PubMed retrieval method. The AUTO-TAR BMI method uses a continuous active learning method where random documents are selected and evaluated by the reviewer. A constantly increasing number of documents are added to the ranking and a new training set is continuously created. The process ends when all the documents have been screened. For the implementation of the AUTO-TAR BMI, we followed Algorithm 1 as described in Cormack and Grossman (2018). The Hybrid variation method is similar to the one presented in Lagopoulos et al. (2018) which uses TF-IDF instead of the Sent2Vec for the representation of the documents in the intra-review model. Finally, for the PubMed method, we query the PubMed database 9 , using the Entrez Programming Utilities, with the title and the objective of each SLR. The normalized position of both queries is combined to a single ranking. For the current Hybrid approach (Hybrid Sent2Vec) we set the same parameters as in Section 4.2 but we choose the best thresholds for intra-review ranker as computed in Section 4.3.2 with t init = 200, t final = 2000.
Tables 7 and 8 present the results for each of the methods using the train set (leave-one-out) and the test set, respectively. Initially, we notice that our approach outperforms all the other approaches in almost all metrics, with the AUTO-TAR approach following. Specifically, in the training set, our hybrid approach greatly outperforms AUTO-TAR in all the recall thresholds, the MAP, and the last rel metrics, while AUTO-TAR marginally surpasses our method at WSS@100. Our approach also outperforms AUTO-TAR in the test set; nevertheless, the difference is marginal in almost all metrics with recall at thresholds 50, 100, and 5000 showing significant difference. However, hybrid learning uses much fewer resources in terms of relevance feedback compared to AUTO-TAR which asks for assessment for all 5000 documents. The addition of a stopping method or criterion could possibly distinguish the performance of the two approaches further and will be considered in future work.
We further examine how our approach compares to others by measuring recall@200 during the review process. Figures 2,3 show macro recall@200 for the AUTO-TAR and hybrid approach along with the 95% confidence interval for the train set and test set, respectively. We first spot large fluctuations in the hybrid approach compared to AUTO-TAR. This is due to the re-training process that the AUTO-TAR follows, which doesn't re-ranks the documents after each document has been reviewed, as the hybrid learning, but the retraining occurs after B documents have been reviewed, where B constantly increases with a rate of B 10 . Moreover, we notice that hybrid learning increases rapidly after the first few documents have been reviewed and it reaches high recall when 200 have been reviewed compared to AUTO-TAR which maintains a low increase rate. Both observations indicate that the intra-review ranking exploits better the relevance feedback compared to AUTO-TAR and is better suited to a live system, where relevant documents will appear sooner to the reviewer and the relevance feedback will have immediate effects. Fig. 2 Graph showing macro recall@200 for the Hybrid learning and AUTO-TAR approaches along with the 95% confidence interval at the train set.

Conclusion & Feature Work
In this work, we introduced a novel approach to screening prioritization for systematic reviews which consists of three consecutive components. Our approach doesn't make use of a boolean query and solely relies on the protocol of an SLR, assisting researchers in the preparation, retrieval, and appraisal stages of preparing an SLR. Furthermore, our inter-review ranking component learns from other reviews which enables it to adapt to other types of Fig. 3 Graph showing macro recall@200 for the Hybrid learning and AUTO-TAR approaches along with the 95% confidence interval at the test set.
reviews and possibly achieve better performance. Additionally, hybrid learning incorporates novel elements, such as sentence embeddings, which render our approach as a strong and present-day baseline.
We performed an empirical study on an updated version of the dataset provided by Task II of CLEF e-health lab which we also make publicly available. Our experiments show that our hybrid approach outperforms the state-ofthe-art and that is suitable for a real-world case scenario. Finally, our study uncovers a simple, transparent, and effective baseline for screening prioritization.
In the future, we plan to further investigate how we can improve our interreview component and minimize the relevance feedback needed to achieve total-recall and high precision by using active learning techniques and zeroshot learning. Moreover, we will look into stopping methods and how they affect performance. As a final step, we intend to build and test an application that employs our approach and benefits researchers starting a systematic review.