{'fraction_non_alphanumeric': 0.027406907479679568, 'fraction_numerical': 0.015082602189031367, 'mean_word_length': 5.24633431085044, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Effect of a comprehensive deep-learning model on the accuracy of chest x-ray interpretation by radiologists: a retrospective, multireader multicase study

clinical findings predicted by the model and was non-inferior to unassisted radiologists for all other clinical findings. Interpretation This study shows the potential of a comprehensive deep-learning model to improve chest x-ray interpretation across a large breadth of clinical practice.


Introduction
Chest x-ray is the most frequently used medical imaging test worldwide. 3 This relatively simple method has allowed investigation of chest pathology, including infection, cardiac pathology, chest trauma, and malignancy, in almost every country worldwide. Advances in digital image acquisition and safe principles of ionising radiation use have led to improved image quality, reduced radiation burden, and wide availability.
However, diagnostic use of chest x-rays has some limitations. Assessment of soft tissue contrast is limited by two-dimensional projection of x-rays through multiple organs, with superimposed densities leading to reduced sensitivity for subtle findings. 4 90% of cases in which a lung cancer diagnosis was missed were due to errors in the interpretation of chest x-rays. 5 Human error, due to factors such as fatigue or interruptions, and reader inexperience contribute to inaccuracy; 4,6 however, few experienced thoracic radiologists (radiologists who are fellowship trained and have >5 years of post-training experience) are avail able. For these reasons, several attempts have been made to create artificial intelligence (AI) systems to aid radiologists in the interpretation of chest x-rays. 7,8 Deep-learning diagnostic image processing algorithms based on convolutional neural networks have shown strong performance. 9 However, although usually highly accurate, most deeplearning systems have a narrow scope and are often limited to a single finding or a small number of findings, 7,10 restricting their use in clinical practice. For example, a decision support system that finds a pneumothorax but misses a pulmonary mass is of questionable clinical benefit. Furthermore, lateral radiographs are often not assessed, despite clear evidence that they contain clinically important information. 11 Deep-learning chest x-ray analysis systems have been developed to automate lung segmentation and bone exclusion; 12 diagnose tuberculosis; 13 detect pneumonia, 14,15 COVID-19, 16 pneumothorax, 17 pneumoconiosis, 18 and lung cancer; 19 identify the position of feeding tubes; 20 and to predict temporal changes in imaging findings. 21 Deeplearning diagnostic tools have also been shown to improve the classification accuracy of radiologists in the detection of pulmonary nodules, 22 pneumoconiosis, 18 pneumonia, 14,15 emphysema, 7 and pleural effusion. 23 Tschandl and col leagues 24 showed that coupling AI models with clinicians can lead to higher diagnostic accuracy than either AI or physicians alone. Some evidence suggests that use of AI can reduce reporting time 14 and that adjunct diagnostic tools are especially useful for junior clinicians. 25 Several studies have compared the accuracy of AI and radio logists; however, the number of radiologists included in these studies was often less than ten. 14,26,27 Here we evaluate a deep-learning model designed to assist clinicians in the interpretation of chest x-rays, encompassing the full range of clinically relevant findings on frontal and lateral chest x-rays.

Study design and participants
In this retrospective, multireader multicase study we evaluated the diagnostic accuracy of 20 radiologists with and without the aid of a deep-learning system. Radiologists interpreted cases without access to the deep-learning tool, then interpreted the same cases with the support of the deep-learning tool after a 3-month washout period. We assessed the change in diagnostic accuracy of radiologists when the deep-learning model was used as a decision support, and also compared the performance of the model alone with that of unassisted radiologists.
Model development and evaluation involved three groups of radiologists (147 fully accredited radiologists in total): 120 consultant radiologists from Vietnam labelled the training dataset, seven specialist thoracic radiologists from Australia did ground truth labelling for the test dataset, and a third group of 20 consultant radiologists from Vietnam interpreted cases in the test dataset. Training dataset labelling defined the radiological findings present on each case in the training dataset. Ground truth labelling defined the radiological findings present in the test dataset.
An overview of the study design is presented in figure 1. This study was reviewed and approved by the human research ethics committee at the University of Notre Dame Australia (Sydney, NSW, Australia; approval number 2020-127S).

Research in context
Evidence before this study Deep learning has the potential to improve the accuracy and speed of chest x-ray diagnosis. We searched PubMed and Google Scholar from Jan 1, 1999, to Oct 1, 2020, using search terms "machine learning chest x-ray", "deep learning", "artificial intelligence", "chest x-ray", "chest radiography", and "automat* detect*". Our search identified 559 machine learning studies in chest x-ray diagnostics, which were mostly small proof-ofconcept studies. Most previously developed deep-learning chest x-ray interpretation models did not compare clinician accuracy with and without the use of artificial intelligence, focused on only one or a few clinical findings, solely used public prelabelled datasets, did not appropriately address hidden stratification, or involved only a small group (less than ten) of clinical radiologists. We therefore evaluated the effects of a comprehensive deep-learning model on the interpretation of chest x-rays by radiologists.

Added value of this study
Our study evaluated the effects of a deep-learning model for 127 clinical findings on the accuracy of chest x-ray interpretation by radiologists. This model is, to our knowledge, the most comprehensive to date, and was trained on a labelled dataset larger than that used in previous studies (821 681 chest x-rays from 520 014 cases). Diagnostic accuracy was compared with a robust ground truth. When comparing the performance of radiologists with and without assistance from the deep-learning model, we found that the model improved performance across most chest x-ray clinical findings. We report the full underlying ontology tree, which represents the comprehensive chest x-ray clinical interpretation framework of a practising radiologist, to enable future research. Our model has been developed into a clinical decision support tool.

Implications of all the available evidence
Radiologist accuracy improved across a large number of clinical chest x-ray findings when assisted by the deep-learning model. Effective implementation of the model has the potential to augment clinicians and improve clinical practice. The labelled training dataset continues to grow, and research is being done to iteratively and progressively improve the model over time. Detailed subpopulation and error analyses are also being done to enable model development. Research is underway to assess the generalisability of results to various clinical environments and health systems.

Data sources and processing
Chest x-rays used for the training dataset were obtained from multiple datasets: I-MED Radiology Network (I-MED; Australia), MIMIC (Beth Israel Deaconess Medical Center, Boston, MA, USA), ChestX-ray14 (NIH Clinical Center, Bethesda, MD, USA), CheXpert (Stanford University Medical Center, CA, USA), and PadChest (Hospital San Juan, Spain; appendix p 4). [28][29][30][31] Images from patients (≥16 years) who had at least one frontal chest x-ray were included in the test dataset. Selected cases were from inpatient, outpatient, and emergency settings. Digital Imaging and Communications in Medicine tags were removed. Protected health information (excluding age and sex) was removed from reports through an automated deidentification process, and patient and case identification were anonymised to deidentify patients while retaining the temporal and logical association between cases and patients. Image data were preserved at the original resolution and bit-depth.
127 chest x-ray findings were identified prospectively by three clinical experts. A chest x-ray finding ontology tree was developed to evaluate and develop the test set (appendix p 5). Each of the findings was defined by consensus between three Australian radio logists, including one subspecialist thoracic radiologist. All participating radiologists in the labelling and evaluation phases were trained to identify chest x-ray findings according to these definitions.
Each case in the training set was independently labelled by three radiologists. Cases were randomly shuffled and placed in a queue. After the radiologist labelled a case, they were allocated the next case according to the random queue order. If a case had already been labelled by that radiologist, the next case in the queue was drawn instead. This ensured that each case was labelled by three different radiologists. Each case consisted of multiple images and the clinical report, which was consistent for each radiologist. Each radiologist was masked to the labels of the other two.
For the training dataset, clinical reports, age, and sex were provided, together with frontal and lateral chest x-rays. Each finding was assigned a present or absent label. Labels consisted of both classification labels on a case level, indicating whether each finding was present in the entire case (multiple images) and each segment for relevant findings. The consensus for each finding for each tripleread case was generated as a consensus score between 0 and 1 using the Dawid-Skene consensus algorithm, 32 which considers the relative accuracies of each labeller for each finding. Segmentation overlays were generated by a single radiologist to localise and depict pathology. This was used to train the model to produce overlay outputs.
In addition to training the model on the original labels, derived training labels were created based on the ontology tree. For example, focal airspace opacity and airspace opacity-multifocal each belonged to the same group of findings: airspace opacity. Because of this, any case that was labelled with either focal airspace opacity or airspace opacity-multifocal was also automatically labelled with airspace opacity. This meant that the model learnt not just from the original labels but from the structure of the ontology tree. It was penalised less if it classified the original label incorrectly but still correctly classified the parent label.
Test dataset cases were excluded from the model training process such that no patient within the test dataset was present within the training dataset. The test dataset included cases from the I-MED and MIMIC datasets only, and was designed to contain approximately 50% of cases from I-MED and 50% from the MIMIC dataset. Cases were randomly drawn to achieve the target number of cases per finding, while keeping the total number of cases as low as possible. Commonly co-occurring findings were controlled so that episodes of co-occurrence comprised no more than 50% of all cases of that finding within this dataset or did not exceed the baseline co-occurrence rate in the training dataset by more than 10%.
Ground truth labels for the test dataset were determined by consensus between three specialist thoracic radiologists from Australia drawn from a pool of seven who did the ground truth labelling. Radiologists had access to anonymised clinical information, past and future chest x-ray images and reports, and, where available, relevant chest CT reports. They did not have access to the outputs of the deep-learning model. The ground truth labels were derived from a Dawid-Skene 33 consensus algorithm from independent labelling of the cases by the three radiologists.

Figure 1: Study design
An ontology tree containing 127 clinical findings was developed, datasets were gathered and aggregated, and clinical findings were labelled by a large team of radiologists. The test set contained past and future images, together with clinical reports, which facilitated robust ground truth labelling by three thoracic subspecialist radiologists. The deep-learning model was trained with five-fold cross-validation. The test set was assessed by 20 radiologists both with and without deep learning assistance. *Data sources were I-MED and MIMIC-CXR. †Data sources were I-MED, MIMIC-CXR, NIH ChestX-ray14, CheXpert, and PadChest (appendix p 4). 4568  Before labelling or ground truth annotation, radiologists underwent rigorous training and a screening examination, which involved familiarisation with the annotation tool, reviewing definitions of each clinical finding, and training on a dataset of 113 chest x-rays from the I-MED database covering most findings on the ontology tree. Each labeller was then assessed with the F1 metric (the harmonic mean of precision [positive predictive value] and recall [sensitivity]). 33 Each ground truth labeller had an F1 score averaged across all findings exceeding 0·5, and each training data labeller an F1 score exceeding 0·45. 20 radiologists, each with 5-25 years of clinical experience (median 10·5 years [IQR 6·75-16·75]) after completing radiology specialist training, interpreted all cases in the test dataset. Patient age and sex were shown, but no radiological report or other comparison images were provided. As such, radiologists did a research read in both study groups rather than a clinical read (ie, radiologists identified findings in a research database, but did not write a detailed clinical report). Radiologists were asked to rate their confidence in the presence of each of the 127 findings using a five-point scale (from 0 [unlikely] to 4 [consistent]; appendix p 4) and did not have access to their labelling results when they reviewed cases for the second time after the 3-month washout period.
Labelling, ground truth annotation, and interpretation were done with the same custom-built, web-based digital imaging and communications in medicine viewer (Annalise Web Labelling Tool, Annalise.ai, Sydney, NSW, Australia). Radiologists viewed images and recorded responses on diagnostic quality monitors and hardware, and interpretation times were recorded by the software platform.

Deep learning model
The deep-learning tool consisted of three convolutional neural networks designed for clinical decision support: an image projection classification model (attributes model), a clinical finding classification model (classification model), and a clinical finding segmentation model (segmentation model). The attributes and classification models were based on the EfficientNet architecture, 34 whereas the segmentation model was based on the U-Net 35 architecture with an EfficientNet backbone. Focal loss 36 was minimised with respect to the Dawid-Skene consensus labels. Class-balanced loss weighting 37 accounted for class imbalance and models were trained using five-fold cross-validation.
We assessed the accuracy of the classification model (version 1.2.0; figure 2). Segmentation output was displayed with the model output, but segmentation was not directly evaluated. The attributes model was not directly evaluated, but it prevented the system from producing output for cases that did not have a recognisable frontal anterior-posterior or posterioranterior image.

Statistical analysis
For the test dataset, we calculated that a minimum dataset of 2568 cases was required to detect a mean difference in area under the receiver operating characteristics (ROC) curve (AUC) of 0·02 in the diagnostic accuracy of at least 18 radiologists labelling all 127 findings (alpha=0·05, beta=0·8). 20 radiologists were recruited to mitigate the risk of dropouts; all of the radiologists completed the assessments.
We first assessed the change in AUC when radiologists were assisted by the model. Clinical findings for which model output was insufficiently powered were retained for this analysis, but discarded for the comparison between the model alone and unassisted radiologists.
The positive predictive value, sensitivity, and specificity for each finding were estimated to assess performance. AUC ROC curves were plotted. The generalised Roe and Metz model and US Food and Drug Administration iMRMC (version 4.0.1) software were used to analyse radiologist accuracy (measured as the AUCROC) with and without the assistance of the model. 38,39 The Matthews correlation coefficient represents the quality of a binary classifier, ranging from -1 (total disagreement) to +1 (total agreement). 40,41 An AUC difference more than 0·05 and a Matthews correlation coefficient difference more than 0·1 were considered to be superior. 42,43 Therefore, findings for which the lower bound of the 95% CI was less than -0·05 were considered inconclusive, findings for which the lower bound was between -0·05 and 0·0 were considered non-inferior, and findings for which the lower bound was higher than 0·0 were considered superior to unaided radiologists.
Positive predictive value, sensitivity, specificity, and Matthews correlation coefficients for each radiologist were calculated by binarising confidence scores for each finding. Any finding with a rating of one or more was considered positive. Bootstrapping was used to assess statistically significant differences in the average Matthews correlation coefficient across the 20 radiologists for each finding between assessment with and without the deep-learning tool. When bootstrapping was done, 10 000 bootstraps of all cases were drawn with resampling to estimate the empirical distribution of the parameter concerned. A two-way repeated measures ANOVA was used to compare mean interpretation time, accounting for between-labeller variation.
A model that has high performance for many findings, but fails to accurately identify crucial findings is not useful in a real-world setting. A subset of 34 crucial findings were identified before the start of the study by a subspecialist thoracic radiologist (appendix p 4). These crucial findings were determined before the analysis and were used to determine the utility of the model. These crucial findings represented the findings most likely to be clinically relevant.
The AUC of the deep-learning model was compared with the average radiologist AUC for each finding with a bootstrapping technique. The Benjamini-Hochberg procedure 39 was used to control the false positive rate, accounting for multiple comparisons.
Significance testing and data processing were done with Python (version 3.7.

Role of the funding source
Employees of the funder (Annalise.ai) were involved in study design, data collection, data analysis, data interpretation, and writing of the report. The two co-chief executive officers of the funder were not involved in data analysis, data interpretation, or writing of the report, but one was involved in study design and one oversaw data collection.

Results
A total of 821 681 images from 520 014 cases were labelled and included in the training dataset (table 1). The median number of model training cases per clinical finding was 5427 (IQR 1515-18 804). 4568 images from 2568 cases were included in the test dataset (table 1). All 2568 cases were classified by the radiologists and 2551 cases were classified by the model. 17 (0·6%) cases were not interpreted by the attributes model: nine were rejected because no frontal image was recognised by the model, four were rejected because no chest x-ray image was found by the model, three raised a processing error, and one had missing data.
Initially, 127 clinical findings were identified on the ontology tree. However, review of the training and test datasets showed that suboptimal intercostal catheter position, pneumobilia, and portal venous gas were infrequently present in both datasets. Pneumobilia and portal venous gas were therefore not included in the secondary outcome analysis. For intercostal catheter position, the initially separate labels of suboptimal intercostal catheter and satisfactory intercostal catheter were merged to create a single label to identify the presence of an intercostal catheter, which was sufficiently prevalent in the test dataset for analysis. Four findings were dropped (pneumobilia, portal venous gas, in-position intercostal catheter and suboptimal intercostal catheter) and one parent finding was added (intercostal catheter), resulting in the change from 127 to 124 findings. To alleviate concerns regarding multiple comparisons, these three additional comparisons were adjusted using the Benjamini-Hochberg procedure for 127 comparisons for the assessment of change in AUC when radiologists were assisted by the model. 124 clinical findings were predicted by the model, which formed the basis of the comparison between the performance of the model alone and that of unassisted radiologists.
100 findings had a statistically significant improvement in Matthews correlation coefficient when radiologists used the deep-learning model (appendix pp 7-8). 24 of the remaining findings were statistically non-inferior. Three findings (image obscured, portal venous gas, and overexposed) were inconclusive because the lower bounds of the 95% CI were less than -0·1 and the upper bounds were more than 0·0. Additionally, Matthews correlation coefficients for the detection of any crucial finding on a given case improved by 0·082 (95% CI 0·030-0·139), from 0·491 to 0·574, when radiologists used the deep learning model. Sensitivity for crucial findings also significantly improved from 0·890 to 0·956, and positive predictive value decreased slightly from 0·905 to 0·899 (figure 4). Most findings showed improved sensitivity,

Figure 3: Change in AUC when radiologists were aided by the deep-learning model
Mean change in AUC and adjusted 95% CI is shown for each clinical finding. Findings for which the lower bound of the 95% CI crosses -0·05 are considered inconclusive, findings for which the lower bound is between -0·05 and 0·0 are considered non-inferior, and findings for which the lower bound is to the right of the 0·0 are superior to unaided radiologists. The numbers of positive cases in the testing dataset for each finding are presented. AUC=area under the receiver operator characteristic curve. ICC=intercostal catheter. *Crucial clinical finding.  Mean interpretation time per case when radiologists used the deep-learning model (107 s; SD 35·6) was significantly lower than when they did not use the model (122 s; 37·4; p=0·0045).
AUCs for the deep-learning model were statistically superior to those for unassisted radiologists for 117 (94%) of 124 clinical findings and statistically non-inferior for all other clinical findings (appendix pp 9-10). The seven remaining clinical findings (shoulder fixation, rib fixation, oesophageal stent, gastric band, pulmonary arterial catheter, clavicle fixation, and shoulder replacement) were non-inferior because the lower bounds of the change in AUC lay between -0·05 and 0·0. ROC curves comparing the model alone with unassisted radiologists are reported in the appendix (pp 18-28). AUCs for unassisted radiologists and the model alone are reported for the subset of 34 crucial findings in table 2.

Discussion
In this retrospective, multireader multicase study, the accuracy of radiologists assisted by a deep learning model was superior to that of unassisted radiologists for 80% of chest x-ray findings and non-inferior for 95% of findings. For the remaining 5% of findings, results were inconclusive. Model-assisted radiologists did not have an inferior performance on any findings compared with unassisted radiologists. Of note, human performance was markedly increased with model assistance.
The deep-learning model alone was either superior or non-inferior to unassisted radiologists for 124 clinical findings. The diagnostic accuracy of the model also compared favourably with that of previously published models (eg, mean ChestNet AUC 0·78). 7,8,10,22,23,44 The accuracy of the model can be at least partly attributed to the large number of cases labelled by radiologists for model training. The evaluated chest x-ray model was trained on more than 800 000 images, each labelled by radiologists using a prospectively defined ontology tree of chest x-ray findings. Many other largescale attempts to train deep-learning models on chest x-ray data have relied on text mining from the original radiology reports, 8,45 a process that has been criticised for inconsistency and inaccuracy. 46 Furthermore, the model uses all common chest x-ray projections (anteriorposterior, posterior-anterior, and lateral), which represents the standard of care in real-world settings.
The mechanism underlying improved accuracy for model-assisted radiologists might be complex. When multiple findings are present, radiologists are less likely to perceive them all. 47 In general, missed findings on radiology reports have been attributed to satisfaction of search, difficulties in inter preting technically suboptimal imaging, and human error. 4,6 Overall, the model provided additional information to radiologists, facilitating improved decision making and making interpretation more efficient.
The validity of any diagnostic assessment is dependent on the quality of the ground truth. The ideal ground truth would include cross-sectional imaging, correlation  with clinical notes, and follow-up imaging. This is not practical across a large dataset, and many chest x-ray findings are never correlated or followed up. In this study, three subspecialist thoracic radiologists reviewed each case, with access to previous and follow-up chest x-rays and CT imaging, where available, with the corresponding reports. The consensus between the three ground truth radiologists was calculated for each clinical finding using the Dawid-Skene algorithm, which reflects a clinical gold standard for ground truth labels.
The number of radiologists included in the study in both the comparison between the deep-learning model alone and unassisted radiologists and assisted versus unassisted radiologists, was carefully deter mined, considering the possible effects of heterogeneous interpretation of x-rays. Other chest x-ray AI decision support studies have generally included a much smaller number of radiologists (less than ten radiologists), resulting in a reduction in power that is often unaccounted for in statistical analyses. 13,14 Given the substantial number of findings assessed in our study, we included 20 experienced radiologists.
Hidden stratification is a well known risk of deeplearning models applied to medical imaging, 48 in which visually distinct groups of imaging findings within the broad label categories of a dataset can produce unexpectedly poor accuracy for clinically relevant subgroups. The archetypal example is that of detecting pneumothoraces without chest drains. Because chest drains are visually obvious and inserted to treat pneumothoraces, deep-learning models trained to detect pneumothoraces often rely on the presence or absence of the chest drain for high accuracy. However, when tested on images of pneumothoraces without chest drains, models often perform poorly. Meaningful disease variation must be appropriately described in the dataset used to train the algorithm, 48 as faulty AI models can mislead clinicians. 24 This key issue was addressed by our comprehensive labelling process. Public datasets, such as the National Institutes of Health ChestXray14 dataset, 29 are labelled with a small number of broad finding classes, which do not account for important subsets; therefore, models trained on these datasets cannot effec tively evaluate this issue. By com prehensively labelling the training and test datasets, we showed that the high accuracy of our model is maintained across various clinically important subclasses. 48 Our study has several limitations. For some clinical findings, the AUC for unassisted radiologists was lower than in previous studies (eg, lung nodule detection AUC 0·742, 49 compared with 0·662 in this study). This might relate to differences in study design. Previous studies on diagnostic accuracy for specific chest x-ray findings generally include a higher proportion of positive cases than seen in this study or in clinical practice, with fewer other distracting findings present. This could be explained by the choice of labelling and participant radiologists. However, given that radiologists were required to pass a screening examination showing their understanding of definitions of clinical findings and their ability to detect them, we believe that the relatively lower initial radiologist accuracy is best explained by the difficult task of labelling all 127 findings concurrently without the benefit of other clinical information.
Radiologists ground-truth labelling the test set had this additional information, which would usually be available in clinical practice. The ontology tree was highly specific and contained many findings. Radiologists were required not only to detect findings, but also to characterise them.
There was a risk of spectrum bias from the 20 study radiologists, who were from Vietnam and interpreted x-rays in a research setting. Although fully qualified specialist radiologists, their findings might not be representative of radiologists elsewhere. More research is underway to test the generalisability of all of our findings.
Although a 3-month washout period and randomised ordering of cases were used, recall bias cannot be completely eliminated. Washout periods in similar studies range from a minimum of 3 h to a maximum of 2 months, with a median of 1 month. 50 The washout period implemented in our study exceeded these benchmarks, minimising the risk of recall bias.
Our study did not include data on ethnicity and patient demographics beyond age and sex. We recognise this as an important bias mitigation issue and work is underway to explore the generalisability of the model in different geographical settings and people of different ethnicities.
Furthermore, the non-clinical, retrospective design of this study might have influenced the interpretation of chest x-rays. The dataset was enriched with a higher prevalence of rare findings than in normal clinical practice. These factors might restrict direct applicability to the clinical setting.
This study has shown the potential of a deep-learning model to improve the accuracy of chest x-ray interpretation. However, AI will have little effect on practice unless it is validated and implemented in usable tools. 51 A strength of our model is that it has been developed into a ready-to-implement tool that can determine that the input data are appropriate, analyse the images, and present the findings to reporting radiologists. Research investigating the clinical applicability of the model in real-world settings, including effects on patient outcomes, is required. Subset analyses of findings in clinically relevant situations are underway. In future research we intend to explore whether results would differ if radiologists were from the same settings as the data sources. This issue was mitigated in our study by implementing a rigorous training procedure for each of the reading and labelling radiologists, including an assessment of accuracy on a separate set of cases ground truth labelled in a similar manner. Radiologists were required to show competency in this assessment before labelling or reading. Research is also underway to investigate the effect on radiologist workflow and their attitudes towards an AI diagnostic adjunct. Additionally, analysis of the effect of the model on the interpretation of chest x-rays by non-radiologist clinicians will be required, as this system has the potential to improve chest x-ray interpretation in settings where radiologists are scarce. 52 This diagnostic accuracy study showed that radiologist performance improved when assisted by a comprehensive chest x-ray deep-learning model. The model had a similar or better accuracy than the radiologists for most findings when compared with high-quality, gold standard assessment techniques. Research is underway to confirm the applicability of this model as a diagnostic adjunct in the clinical setting.

Contributors
JCYS, CHMT, BH, SJFH, BPJ, PB, and CMJ conceptualised and designed the study. JCYS, CHMT, and QDB searched the literature. JCYS, CHMT, HP, and PB acquisitioned the data. All authors analysed, visualised, and interpreted the data. JCYS, CHMT, QDB, XGH, PB, and CMJ drafted the manuscript. All authors critically revised, reviewed, and approved the manuscript. All authors had access to the datasets. JCYS, CHMT, JBW, BH, and XGH verified the data. An external biostatistician verified all data and statistical analyses. JCYS and CHMT independently did the statistical analyses. JCYS, CHMT, and CMJ verified the statistical analyses.

Declaration of interests
This study was funded by Annalise.ai. JCYS, CHMT, QDB, XGH, JBW, AA, HA, HP, JFL, BH, SJFH, BPJ, LO-R, PB, and CMJ were employed by or seconded to Annalise.ai and report personal fees from Annalise.ai during the study and outside the submitted work. CMJ was employed by I-MED. All other authors declare no competing interests.

Data sharing
The research team will make the model and radiologist performance datasets and the test dataset available to interested research partners with the goals of supporting the research community and making collaborative contributions to the literature. Requests for access can be made through the Annalise.ai website. The model is publicly available as a commercial software product. Free web-based demonstration can be accessed online.