{'fraction_non_alphanumeric': 0.026984229576008274, 'fraction_numerical': 0.008014477766287487, 'mean_word_length': 5.05814408770556, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 1, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
RobotReviewer: evaluation of a system for automatically assessing bias in clinical trials

Abstract Objective To develop and evaluate RobotReviewer, a machine learning (ML) system that automatically assesses bias in clinical trials. From a (PDF-formatted) trial report, the system should determine risks of bias for the domains defined by the Cochrane Risk of Bias (RoB) tool, and extract supporting text for these judgments. Methods We algorithmically annotated 12,808 trial PDFs using data from the Cochrane Database of Systematic Reviews (CDSR). Trials were labeled as being at low or high/unclear risk of bias for each domain, and sentences were labeled as being informative or not. This dataset was used to train a multi-task ML model. We estimated the accuracy of ML judgments versus humans by comparing trials with two or more independent RoB assessments in the CDSR. Twenty blinded experienced reviewers rated the relevance of supporting text, comparing ML output with equivalent (human-extracted) text from the CDSR. Results By retrieving the top 3 candidate sentences per document (top3 recall), the best ML text was rated more relevant than text from the CDSR, but not significantly (60.4% ML text rated ‘highly relevant' v 56.5% of text from reviews; difference +3.9%, [−3.2% to +10.9%]). Model RoB judgments were less accurate than those from published reviews, though the difference was <10% (overall accuracy 71.0% with ML v 78.3% with CDSR). Conclusion Risk of bias assessment may be automated with reasonable accuracy. Automatically identified text supporting bias assessment is of equal quality to the manually identified text in the CDSR. This technology could substantially reduce reviewer workload and expedite evidence syntheses.


BACKGROUND AND SIGNIFICANCE
Assessing bias is a core part of systematic review methodology. Reviews typically use standardized checklists or tools to assess trial quality. 1 The Cochrane Risk of Bias (RoB) tool is one such tool. 2 It has been adopted across the Cochrane Library, and increasingly in systematic reviews published elsewhere. The tool comprises six core domains (Box 1), which reviewers score as being at high, low, or unclear risk of bias.
Bias assessment is time-consuming, taking experienced reviewers around 20 minutes for every study included in a systematic review. 3 The requirement to assess bias has been identified as an important factor preventing Cochrane reviews from being kept up to date. 4 Bias assessment is also subjective: individual reviewers have low rates of agreement, 3 though this improves somewhat when review specific guidance is provided. 5 Technology to assist reviewers in assessing bias has the potential to substantially reduce workload. An accurate system could make bias assessment quicker and more reliable, freeing up researcher time to concentrate on thoughtful evidence synthesis, and ultimately help keep systematic reviews up to date. 6 Our preliminary work demonstrated the feasibility of automated risk of bias assessment. 7 However, in this prior work we evaluated our method with respect to an imperfect reference standard, thus demonstrating the internal validity of our approach, but not whether the technology is mature enough to be used in practice. Specifically, since RoB assessment is subjective, we need to compare the quality of the ML approach with that of a human assessment.
In this paper, we introduce a novel machine-learning (ML) approach, which 1) models risks of bias simultaneously across all domains while 2) identifying text supporting these judgments. We substantially extend our previous model, namely by introducing a multi-task approach that exploits correlations between different bias types. We then evaluate this approach, benchmarking our predictive accuracy against an estimate of how well humans perform for this task. We obtain these human benchmarks by exploiting the fact that many trials are included in multiple systematic reviews (and therefore have multiple, independently conducted risk of bias assessments available). To evaluate the quality of supporting text, we use a blinded expert panel, who were asked to rate the quality of algorithm output versus the supporting text chosen by the original review authors.

OBJECTIVES
We describe the development and evaluation of RobotReviewer, (RobotReviewer refers to the user interface coupled with our machine learning technologies. Eventually, it will do more than automatic RoB assessment), a system to automate the assessment of bias of randomized controlled trials using the Cochrane RoB tool. For each domain in the RoB tool, the system should reliably perform two tasks: 1) determine whether a trial is at low risk of bias (document classification of low v high or unclear risk of bias), and 2) identify text from the trial report that supports these bias judgments (sentence classification of relevant v irrelevant). In the evaluation, we aim to compare model performance with the consistency of RoB assessments in published systematic reviews, to help judge to what extent bias assessments could be automated in real review production.

METHODS
From a machine-learning vantage point, 1) is a classification task and 2) is a data extraction problem. For both we can use supervised machine learning, in which model parameters are learned from manually annotated documents. 8 Unfortunately, collecting human annotations with which to train such systems is time-consuming and therefore expensive. This is especially true for biomedical text mining tasks, as these require costly expert annotators. Training corpora used in previous efforts for related tasks have been relatively small, comprising 100-350 documents. 8,9 Here we take a different approach: to obtain a large corpus of labeled data we use distant supervision, a machine learning methodology that exploits structured data in existing databases in place of direct human supervision. 10,11 Distant supervision involves automatically deriving labels for unlabeled data from existing resources, typically using heuristics that cover the majority of cases but that are imperfect (e.g., string matching). This produces noisy labels (having a higher rate of errors than manual annotation). However, because these are 'free' labels, we can build and exploit larger training datasets than would be otherwise feasible. Larger training datasets, in turn, have been shown to improve model performance. 12 We derive distant supervision from the Cochrane Database of Systematic Reviews (CDSR). The CDSR comprises more than 5,400 systematic reviews on health produced by members of the Cochrane Collaboration, an international non-profit organization. This dataset includes expert risk of bias assessments for clinical trials (see Box 2). Crucially, in many of these assessments, the review authors include direct quotes from the original trial report to justify their judgments. Therefore, in addition to the article-level risk of bias labels, we can also derive sentence-level annotations within full texts that indicate whether sentences were used in assessing the risk of bias for a particular domain. This derivation is accomplished by string matching. Figure 1 illustrates this schematically.
Automating the labeling of the clinical trials corpus Our method for corpus construction is outlined in Figure 2.

Trial Linkage
First, we sought full-text PDFs of the primary citation for clinical trials that were included in systematic reviews in the CDSR. The CDSR contains semi-structured reference data, but not unique identifiers. We therefore used the following high-precision strategy. For each trial included in a systematic review in the CDSR we conducted multiple searches of PubMed. Each search used non-overlapping subsets of citation information, any of which might be expected to uniquely retrieve the trial (e.g., search 1: articles matching full title; search 2: articles with exact author combination with matching publication year; search 3: articles with matching journal name, volume, issue, and page number). We considered a positive match where two or more searches retrieved the same article. We linked 52,454 of 67,894 studies included in reviews in the CDSR to a unique publication using this method, and obtained 12,808 of these publications in PDF format.

Pre-processing of CDSR Data
Although the Risk of Bias tool assesses 6 core biases, Cochrane review authors are free to assess other (often idiosyncratic) biases if they feel they are relevant. The CDSR contains >1400 unique strings identifying bias domains. Most of these referred to one of the core domains listed in Box 1. We manually mapped alternative descriptions to the domain labels, and excluded domains unique to individual reviews. For each linked study, we extracted the types of bias assessed, the bias judgments, and the justifications for the judgments.

Labeling PDFs Using Distant Supervision
Plain text was extracted from the PDFs using the pdftotext utility from xPDF. 13 The extracted text was tokenized into sentences and words.
For task 1 (document annotation), we algorithmically labeled each document as being at 'low' or 'high/unclear' risk of bias, using the judgment from the linked Cochrane review. We dichotomized this outcome consistent with typical practice in systematic reviews: reviewers often conservatively assume that 'unclear' studies have a 'high' risk of bias and conduct sensitivity analyses including only studies at low risk of bias.
For task 2 (sentence annotation) where quote data was available in the linked Cochrane review, sentences containing exactly matching text were labeled as relevant to the risk of bias domain. All other sentences were labeled as irrelevant. This strategy produces incomplete labels and specifically would be expected to have high precision and low recall. Cochrane review authors are likely to quote one or two sentences that they feel best justify a risk of bias judgment. Ideally, all text relevant to the bias decision would be labeled.

Box 1: Items from the Cochrane Risk of Bias
• Random sequence generation • Allocation concealment • Blinding of participants and personnel • Blinding of outcome assessment • Incomplete outcome data • Selective outcome reporting Box 2: Example of the risk of bias data stored in a Cochrane review for the domain allocation concealment, from Higgins et al. 16 Domain: Allocation concealment risk of bias: High justification: Quote: " . . . using a table of random numbers." Comment: Probably not done.

Machine-Learning Approach
We use a novel multi-task variant of the soft-margin Support Vector Machine (SVM) 14 that maps articles to risk of bias assessments (low or high/unclear) (task 1) and simultaneously extracts sentences supporting these judgments (task 2). Multi-task learning refers to scenarios where the aim is to induce classifiers for multiple, related classification problems or 'tasks'. 15 Here, bias assessment for the respective domains constitute our related tasks. 16 Our approach includes two novel components. First, we explicitly incorporate features derived from sentences that support risk of bias assessment into the document-level model that predicts the RoB.
Second, we jointly model RoB across all domains, for both sentence and document-level predictions.
The document-level feature space comprises the uni-and bigrams of the full-text documents as a foundation. To incorporate supporting sentence information into this basic document-level model, we append interaction features for each sentence, which cross the sentence relevance to the domain of interest (relevant v not relevant) with the textual features from that sentence (uni-and bi-grams). We have presented details of this method elsewhere. 7 Intuitively, the word 'computer' appearing in a sentence concerning randomization may strongly indicate a low risk of bias, but the same word elsewhere in the document may have little predictive power. The interaction features we introduce aim to capture such information. To create these interaction features when training the model, we use data on sentence relevance taken directly from the CDSR. At test time, however, we do not know which sentences support risk of bias assessments for the respective domains, and we therefore use the sentences predicted as relevant by our sentence classifier.
Here we extend this model to borrow strength across risk of bias domains, using a multi-task approach for both sentence and document-level classifications. Specifically, we introduce 'interaction features' that represent the intersection of domains and token (word) indicators. Denoting the number of domains by k, we insert k copies of each feature vector x (one per domain) for each instance, in addition to a shared copy of x common to all domains (see Figure 3 for a schematic of this approach). For example, there will be a feature corresponding to the presence of the word 'computer' and the target  domain randomization. This will be non-zero only in columns that comprise the copy of x specific to randomization. Note that this can be viewed as an instantiation of Daumé 's frustratingly easy domain adaptation approach. 17 Our sentence model is thus trained jointly across all risk of bias domains; the shared component enables information sharing between them. We adopt this approach for both the sentence and the document-level models.
For a new article (at test time), we then use the multi-task sentence model to generate predictions for each sentence regarding whether it is likely to support assessments for the respective domains. Before a document-level RoB prediction is made, indicators corresponding to the tokens comprising sentences predicted to be relevant are inserted into the vectors representing documents. This is done for each domain. These document representations include a shared component across domains (again enabling borrowing of strength). Therefore, sentence-level predictions (made via a multi-task sentence-level model) directly inform our multi-task document-level model. This realizes a joint approach to predicting sentence-level relevance and document-level assessments across related tasks.
For both the sentence-and document-level model, we adopt a linear classification model defined by a weight vector w, such that y ¼ L(wÁx), where L maps the continuous score to a categorical label of 0 (when wÁx < 0) or 1 (wÁx ! 0). We use the hinge-loss function, which imposes no loss when the model prediction is correct and a loss proportional to the magnitude of wÁx when the prediction is incorrect. We combine this with a squared L2 penalty term on the model parameters to form our objective, which describes a linear-kernel soft-margin SVM. 14 This objective is shown in Equation 1. For the sentence-level model, we sum the loss over the sentences comprising each distantly labeled document. The joint, multi-task modeling we have proposed is effectively realized by augmenting the feature space; for example, by inserting into document or sentence vectors shared and domain-specific copies of token indicators. Note that in our multi-task model, there is only a single, shared w for document classification (and another for sentence classification), whereas in our previous approach we fit separate weight vectors for each domain for both the document and sentence models.
The objective function we minimise is shown in equation 1. The general objective function we minimize. This is the form we use across all domains for both the document and sentence-level models, however for the latter we need to sum over the sentences comprising documents. Note that the innovations we have proposed (multi-task learning and a joint model incorporating sentence labels or predictions into the document-level model) are realized by manipulation of the feature spaces, hence the form the objective is unchanged.
We fit this model (estimate w) by minimizing our objective via Stochastic Gradient Descent (a standard optimization procedure in which one optimizes parameters to minimize an objective function by following the gradient as approximated by iterative evaluation on individual instances). The objective includes a hyper-parameter a, which trades regularization strength (model simplicity) against empirical loss. We tune a via line search ranging over values from 10 -4 to 10 -1 , equidistant in log-space, and select the value that performs best on average according to nested cross-fold validation.
All models we consider leverage token-based features encoded in a binary 'bag-of-words' representation. For sentence prediction, our vocabulary V comprises all unique uni-and bi-grams present in at least two supporting sentences. For document prediction, V comprises all uni-and bi-grams that appear in at least two articles. We preprocessed the text by removing English 'stop words' (uninformative words like "the" and "as") and converting to lowercase. Because using fulltext and interaction features produces a very large feature space, we use the 'feature hashing' trick to keep the model tractable. The hashing trick maps strings to vector indices via a hashing function. This has been shown to work well for large-scale multi-task text classification. 18 To summarize, we use a novel model for risk of bias prediction that 1) jointly makes article-and sentence-level predictions, and 2) borrows strength across the related risk of bias assessment tasks via multi-task learning, for both article and sentence predictions.
We have made the code used for the entire distant supervision pipeline and evaluation available at https://www.github.com/ijmarshall (under cochrane-nlp and cochrane-nlp-experiments). Systematic reviewers might instead use our prototype web-based tool which generates and presents RoB assessments for articles uploaded by users ( Figure 5). 19 Evaluation Task 1: Document Prediction For task 1 (document judgments) we exploited the fact that many trials (ranging from 239-1148 trials for each bias domain) are described in Figure 3: A schematic depiction of our multi-task learning approach. We define a joint classification model across domains. To achieve this, we include k representations of each instance (e.g., document) d in the design matrix, one per risk of bias domain. We construct the target vector y with the corresponding per-domain labels for d (derived from the CDSR). Each of the k entries for d comprises kþ1 concatenated vectors of length equal to the vocabulary size (V), where for the document model this vocabulary is the union of unique uni-and bi-grams appearing in at least two articles, and for the sentence model V is the union of uni-and bigrams appearing in at least two supporting sentences. The first copy of V in each row is shared across all entries representing the corresponding article; the remaining are domain specific copies. Thus in any given row, all but two of these sub-vectors will be zero vectors. Specifically, each row i will contain two non-zero vectors that are copies of the bag-ofwords representation (binary indicators for V) for instance i (x i ): one shared and the other domain specific. The shared component allows borrowing of strength across domains, while the domain-specific components enable the model to learn words that signal low risk of bias (or the likelihood of supporting RoB assessment, for the sentence prediction task) only in specific domains. more than one Cochrane review. We could therefore access two independently conducted RoB assessments for each of these trials. We held out this set of trials to use as test data.
We were therefore able to evaluate both the accuracy of the automated RoB assignment and agreement among human reviewers on the same set of trials. This is important because agreement between human reviewers is imperfect (see Table 1). This establishes an upper bound on the performance we might hope to achieve using an automated approach. Figure 4 depicts our evaluation setup schematically.
For document prediction, we consider three models. Model 1 is a standard uni-and bi-gram model using the entire document text; each RoB domain is modeled independently in this approach. Models 2 and 3 are the multi-task models that we have proposed in this work, which incorporate supporting text (output from task 2) and also share features across domains for document-level prediction. Model 2 uses the multi-task approach to jointly model all 6 domains. Model 3 is identical to model 2, except that it excludes incomplete reporting of outcomes and selective reporting. We excluded these domains post hoc under the assumption that predictions in these noisy domains were adversely affecting performance in other domains in the case of our multi-task model. Specifically, incomplete reporting of outcomes would typically involve calculation of withdrawal and dropout rates (which is not possible using bag-of-words modeling), and assessing selective reporting would usually require reference to a trial protocol. Finally, for comparison, we report a baseline result, where all documents are labeled with the majority class for the domain.

Task 2: Sentence Prediction
For task 2, we automatically labeled sentences comprising held-out documents using the trained model. We considered the evaluation criteria: top1, where the top scoring sentence per document was identified, and top3, where the top three scoring sentences were identified. The latter is particularly relevant to semi-automating RoB assessment; if relevant sentences are amongst the top three, then we can rapidly guide reviewers to these sentences, thus expediting assessment.
We used two control strategies for comparison: cochrane, where the text justifying a bias decision was taken directly from the published review in the CDSR describing the trial (to estimate human performance; mean 1.3 sentences per trial), and baseline where a sentence was drawn randomly from the document. We aim to assess the comparative relevance of sentences selected by our automated approach and the two control strategies. We cannot derive this information from the CDSR, however, and thus need to rely on human expertise to manually assess relevance. To this end, we recruited a panel of 20 systematic review authors (median published reviews per author: 19, IQR 7.5 to 51.5); all members had substantial experience of the Cochrane RoB tool.
Sentences identified by each strategy were presented to the panel members, who were blinded to the text source (i.e., whether sentences were selected by human, at random, or by our algorithm). The assessors were asked to assess sentence relevance to a particular domain in the RoB tool using a Likert-like scale (3¼highly relevant, 2¼somewhat relevant, and 1¼not relevant). The assessors were provided with additional definitions for each of these categories. Two assessors piloted the evaluation and had substantial agreement (Kappa¼0.79). We calculated based on pilot data that at least 350  trials would be needed to detect a 10% difference in model output quality with 80% power with significance of P <0.05. We collected a total of 1731 judgments from 20 experts from 371 trials.

Document-Level Results
Document-level results are presented in Table 1. Model 2 judgments were less accurate than those from published reviews, though the difference was <10% (overall accuracy 71.0% with ML v 78.3% with CDSR; P < 0.001). Model 1 (which does not include supporting sentences and models each domain separately) achieved substantially greater accuracy than baseline. Model 2 (which jointly models all domains, and incorporates information about whether sentences are judged relevant) improved performance compared with Model 1 in all but one domain (blinding of outcome assessment). Model 3, which ignores the noisy selective reporting and incomplete reporting of outcomes domains, resulted in uniform improvement compared to Model 1 across all the domains it included. Our study is not powered to assess the significance of differences between the three models; a larger dataset would be needed to determine whether these apparent differences are statistically significant.

Sentence-Level Results
The results from task 2 (identifying sentence with information about RoB) are presented in Tables 2 and 3. The top1 model (which retrieves one sentence per study) performed substantially better than baseline, but produced text judged less relevant than that in the CDSR (10% fewer documents with highly relevant output, and 14% fewer documents with highly, or somewhat relevant output). The best text from the top3 model was rated as more relevant than text from the CDSR overall, and in individual domains, but differences were not statistically significant.

DISCUSSION
We report the development and evaluation of RobotReviewer, a system for automating RoB assessment. Our system determines whether a  trial is at low risk of bias for each domain in the Cochrane RoB tool, and identifies text that supports these judgments. We demonstrated strong performance on these tasks. Automatic document judgments were of reasonable accuracy, lagging our estimate of human reviewer accuracy by $7%. Our automated approach identified text supporting RoB judgments of similar quality to that found in published systematic reviews. While our algorithm is not ready to replace manual RoB assessment altogether, we envisage several ways it could reduce author workload in practice. Since justifications are provided, most errors should be easy to identify, meaning reviewers need consult the full paper only where judgments are not adequately justified (see Box 3). This should mitigate a common concern about automation technologies: that they act as black boxes. 21 Elsewhere, we have described a prototype tool which presents the model predictions to the user directly within the original PDF document. 19 Figure 5 shows the system in use. This has the additional advantage of preserving the link between published reviews and their source data, a key omission in current practice. 20 Alternatively, this system could also be used to draw reviewers' attention to sentences that are likely to be relevant, leaving them to make the final judgment: this would expedite RoB assessment. Finally, we note that current practice is for two reviewers to assess the RoB of trials independently, then reach a consensus. An alternative workflow would be to replace one of these two reviewers with the automated approach, thus still having a second independent assessment.
One potential weakness is that our corpus comprises a fraction of the studies reported in the Cochrane Library. The 12,808 PDFs were a convenience sample comprising PDFs available via university library subscriptions. Studies with unobtainable PDFs might have increased RoB, particularly those in lesser-known journals, those reported as conference abstracts, and older study reports. Post-hoc, we found these were 6% less likely to be at low RoB. However, our system is designed for PDF use, and our corpus should generalize to the types of PDFs that researchers use.
In this work we sought to estimate the accuracy of manual RoB assessment by making use of trials with 2 or more RoB assessments in the CDSR. Our approach makes the assumption that discrepancies between RoB assessments represent errors or disagreements. In

Domain
Random sequence generation Risk of bias Low Text justifying judgment

Sequence generation
Assuming an average of 10 individuals per group, the project leader generated a random sequence of 20 sessions through an online program, with the criterion that the occurrence of both interventions had to be balanced (i.e., 10 sessions per intervention) Figure 5. Example of our prototype system showing the bias assessment for random sequence generation and a supporting sentence.
practice, RoB judgments may be influenced by individual review question, or by outcome. However, our estimate of human performance shows substantially greater agreement for this task than previous estimates. Where multiple Cochrane reviews contain the same trial, it is likely that they were produced by the same review group, and will share editors and/or author teams who may reach similar bias decisions more often than independent groups.
There are several promising routes for improving performance. First, our model was trained on trials from any clinical specialty. Since reviewer agreement increases when review specific guidance is given, 5 training a model on trials relevant to a review of interest may improve performance.
Second, the Cochrane Handbook recommends that some domains of bias should be assessed per outcome, rather than per study. 16 While our tool should identify text that is relevant to bias for all outcomes assessed, it produces one overall judgment per paper. This may partly explain relatively poor performance in the domain blinding of outcome assessment, which seems likely to vary substantially for different outcomes in the same trial. Most Cochrane reviews at present (and hence our training data) still assess all RoB domains per study, but we aspire to judge bias for each outcome in future.
Third, the domain selective outcome reporting requires reference to a trial protocol, to find if any pre-specified outcomes were not reported in the final paper. That our model performed better than baseline for this domain is probably explained by correlations between selective reporting bias and other biases that are more readily determined from the trial publication. Compulsory registration of clinical trials means that trial protocols are now easily obtainable: the WHO collates protocols from international registries in machine-readable format, including lists of outcomes. Linking with this dataset may yield better performance.
Finally, by incorporating reviewer corrections into the model, we could make use of online learning. This strategy may improve model performance with use. This would make it possible to stay up-to-date with changes in research methodology or bias assessment practice over time, and would learn to assess bias in a way which is tailored to the needs of a particular clinical area, review group, or individual authors.

CONCLUSION
We have outlined a method for automating RoB assessment, including the tasks of categorizing articles as at low or high/unclear risk and extraction of sentences supporting these categorizations, across several domains. While model performance lags human accuracy, it can identify text in trial PDFs with similar relevance to that in published reviews. Future methodological improvements are likely to close the gap between automated and human performance, and may eventually be able to replace manual RoB assessment altogether.

FUNDING
This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.