{'fraction_non_alphanumeric': 0.04475376068909556, 'fraction_numerical': 0.02983584045939704, 'mean_word_length': 5.059757942511347, 'pattern_counts': {'":': 0, '<': 10, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Diagnostic performance of CO-RADS for COVID-19: a systematic review and meta-analysis

Objectives To investigate the diagnostic performance of the coronavirus disease 2019 (COVID-19) Reporting and Data System (CO-RADS) for detecting COVID-19. Methods We searched PubMed, EMBASE, MEDLINE, Web of Science, Cochrane Library, and Scopus database until September 21, 2021. Statistical analysis included data pooling, forest plot construction, heterogeneity testing, meta-regression, and subgroup analyses. Results We included 24 studies with 8382 patients. The pooled sensitivity and specificity and the area under the curve (AUC) of CO-RADS ≥ 3 for detecting COVID-19 were 0.89 (95% confidence interval (CI) 0.85–0.93), 0.68 (95% CI 0.60–0.75), and 0.87 (95% CI 0.84–0.90), respectively. The pooled sensitivity and specificity and AUC of CO-RADS ≥ 4 were 0.83 (95% CI 0.79–0.87), 0.84 (95% CI 0.78–0.88), and 0.90 (95% CI 0.87–0.92), respectively. Cochran’s Q test (p < 0.01) and Higgins I2 heterogeneity index revealed considerable heterogeneity. Studies with both symptomatic and asymptomatic patients had higher specificity than those with only symptomatic patients using CO-RADS ≥ 3 and CO-RADS ≥ 4. Using CO-RADS ≥ 4, studies with participants aged < 60 years had higher sensitivity (0.88 vs. 0.80, p = 0.02) and lower specificity (0.77 vs. 0.87, p = 0.01) than studies with participants aged > 60 years. Conclusions CO-RADS has favorable performance in detecting COVID-19. CO-RADS ≥ 3/4 might be applied as cutoff values given their high sensitivity and specificity. However, there is a need for more well-designed studies on CO-RADS. Key Points • CO-RADS shows a favorable performance in detecting COVID-19. • CO-RADS ≥ 3 had a high sensitivity 0.89 (95% CI 0.85–0.93), and it may prove advantageous in screening the potentially infected people to prevent the spread of COVID-19. • CO-RADS ≥ 4 had high specificity 0.84 (95% CI 0.78–0.88) and may be more suitable for definite diagnosis of COVID-19.

Computed tomography (CT) is a widely available, timesaving, and non-invasive approach for detecting COVID-19 [12]. It could be an efficient tool for diagnosing COVID-19 with high sensitivity and monitoring disease course [10,[13][14][15]. Accordingly, it has been recommended for identifying patients with COVID-19 [16][17][18]. Nevertheless, the use of CT scanning for diagnosing COVID-19 given the lack of accurate, consistent evaluation across radiologists and medical centers is controversial [19]. There has been heterogeneity among previous studies on chest CT for detecting COVID-19, with sensitivity and specificity of 0.19 to 1.00 [20] and 0.25 to 0.70 [21], respectively. This was mainly attributed to the lack of a specific reporting system in various centers with different radiologists [20][21][22][23].
The COVID-19 Reporting and Data System (CO-RADS) was established by the Dutch Radiological Society to standardize the assessment scheme and simplify reporting using a 5-point scale of suspicion (from 1 [very low suspicion] to 5 [very high suspicion]) for COVID-19 pneumonia on chest CT images [24]. CO-RADS has excellent diagnostic performance for COVID-19 through interpretation of chest CT images, with moderate to substantial interobserver agreement [24][25][26]. A meta-analysis on the diagnostic performance of CO-RADS in detecting COVID-19 [27] only included six studies up to August 24, 2020, with a small sample size. This systematic review and meta-analysis aimed to provide an updated overview of the diagnostic performance of CO-RADS in COVID-19, with a focus on the possible practical application of CO-RADS.

Materials and methods
This meta-analysis was conducted following the Preferred Reporting Items for Systematic Reviews and Meta-Analysis [28], with a standardized review and data extraction protocol. The primary study outcome was the diagnostic performance of CO-RADS for COVID-19. Additionally, we discussed the clinical application of CO-RADS.

Search strategy and selection criteria
A computerized search of PubMed, EMBASE, MEDLINE, Web of Science, Cochrane Library, and Scopus was performed up to September 21, 2021, without restrictions regarding regions or publication types. We manually reviewed the reference lists of all retrieved studies, review articles, and conference abstracts to identify additional potential studies. We used the following search terms: ([COVID-19 reporting-anddata system] or [COVID-RADS]

Inclusion and exclusion criteria
The inclusion criteria were (1) studies using CO-RADS in COVID-19 diagnosis, (2) studies using RT-PCR as the reference standard, (3) original scientific articles, and (4) sufficient data for reconstructing the 2 × 2 tables for diagnostic accuracy. The exclusion criteria were (1) duplication of publications; (2) nonoriginal studies, including editorials, letters to the editor, review articles, and case reports; (3) non-English articles; and (4) studies with insufficient information for reconstructing a 2 × 2 table.

Data extraction
We extracted data from the included studies using a standardized form: (1) true positives, false positives, true negatives, false negatives; (2) study characteristics, including first author, publication year, country, study design, cutoff values, study duration, recruitment, number of centers, number of readers, setting, and sample size; and (3) patient characteristics including age, sex ratio, and symptoms. When different readers in a single study reported more than one result, we extracted the consensus result or the average result if there was no consensus result.

Quality assessment
The methodological quality of the studies was assessed using the Quality Assessment of Diagnostic Accuracy Studies-2 tool [29]. Each study was rated with respect to the following domains: patient selection, index test, reference standard, and flow and timing. Each domain was assessed based on the risk of bias and the first three domains, including applicability.

Statistical analysis
We created 2 × 2 tables to calculate the pooled sensitivity, specificity, and corresponding 95% confidence interval (CI) using a bivariate random-effects model; further, we calculated the diagnostic odds ratio (DOR). The results were graphically shown in the forest plots. We constructed hierarchical summary receiver operating characteristic (HSROC) models; further, we calculated the area under the curve (AUC). We performed Deeks' funnel plot asymmetry test to evaluate the possible presence of publication bias, with p < 0.10 indicating a possibility of publication bias [30]. The heterogeneity of the included studies was evaluated using the inconsistency index (I 2 ) and Q statistic of the chi-square value test [31].
Statistical analyses were performed using STATA 16.1 and RevMan 5.3. Statistical significance was set at p < 0.05.

Characteristics of included studies
The 24 included studies comprised 8382 patients. More than half of studies were published in 2021. The studies were conducted in various countries, including Italy, French, the Netherlands, and China. There were 3 and 17 prospective and retrospective studies, respectively, with the remaining 4 studies not providing information on study design. Fourteen studies consecutively enrolled participants, while the remaining 10 studies did not describe participant enrollment. Among the 24 studies, over half were single-center studies while six were multi-center studies and three studies did not provide relevant information. Regarding the number of readers, 12 and 4 studies included ≥ 3 and < 3 radiologists, respectively, while 8 studies lacked information. Ten studies were conducted in the ED. Five studies included both symptomatic and asymptomatic patients while 19 studies only included symptomatic patients. The sample size significantly   (Table 1). Figure 2 shows the quality variables of 24 included studies. Ten studies showed an unclear risk of bias since there was no information suggestive of consecutive or random sampling. Regarding the index test domain, only one study had an unclear risk of bias since it did not provide information regarding blinding to the reference standard. In the reference standard domain, two studies showed a high risk for bias since the reference standard was not blinded to the index test results. Additionally, four studies had an unclear risk of bias since they did not mention whether the reference standard was performed blinded to the index test results. Regarding the flow and timing domain, four studies had a high risk of bias since some patients were not included in the analysis while seven studies had an unclear risk of bias since they did not provide clear information regarding the time interval between the reference standard test and index test. Due to concerns regarding applicability, unclear risk ratings were assigned in two, one, and two studies in the patient selection, index test, and reference standard domains, respectively.

Meta-regression
We performed meta-regression to explore the heterogeneity causes among 20 and 23 studies using CO-RADS ≥ 3 and CO-RADS ≥ 4, respectively (     Abbreviations: LR+, likelihood ratio positive; LR-, likelihood ratio negative; DOR, diagnostic odds ratio; CI, confidence interval  Table 4 summarizes the results of subgroup analyses of the age, sex, and symptoms.

Discussion
Inconsistent reports regarding sensitivity and specificity impede our ability to determine the diagnostic accuracy of CO-RADS for detecting COVID-19. Few relevant systematic reviews or meta-analyses have assessed the diagnostic utility of CO-RADS for COVID-19 [27]. This is the first systematic review and meta-analysis to directly evaluate the accuracy of CO-RADS for COVID-19 and discuss the practical application of CO-RADS. Our findings demonstrated that CO-RADS has great utility in diagnosing COVID-19; moreover, CO-RADS ≥ 3/4 showed outstanding ability.
The AUC of CO-RADS revealed that CO-RADS ≥ 3/4 displayed favorable performance in detecting COVID-19. CO-RADS ≥ 3 might be preferred when priority is placed on the sensitivity of the examination. In case COVID-19 outbreaks re-occur in some countries, there is a need to rapidly   Considerable among-study heterogeneity was observed. When the cutoff value was 3, there were significant differences in sensitivity according to publication year, study design, enrollment, number of centers, and setting; moreover, there were significant differences in specificity according to the setting. This suggests that the diagnostic performance of CO-RADS ≥ 3 may have been overestimated in studies with characteristics such as publication in 2020, failure to mention enrollment and setting, and multi-center settings, which yielded limitations in this meta-analysis. This suggests that CO-RADS ≥ 3 might facilitate quick detection of COVID-19 without geographical limitations in case of reoccurrence of COVID-19 outbreak. Additionally, the performance of CO-RADS ≥ 3 in readers has been relatively consistent which demonstrates the consistency and stability of CO-RADS ≥ 3. However, multi-center studies had significantly higher sensitivity than single-center studies (0.91 vs. 0.89, p < 0.01). Our findings provide additional evidence for exploring the consistency of CO-RADS ≥ 3 among different centers.
Meta-regression analysis of CO-RADS ≥ 4 revealed an association of study heterogeneity with publication year, study design, enrollment, number of centers, number of readers, and study settings, which indicated that the diagnostic performance of CO-RADS ≥ 4 may have been overestimated to some degree in studies published in 2021 without setting information, which led to several limitations in this study. Specifically, we focused on the consistency among various centers with different readers. Multi-center studies had significantly higher sensitivity than single-center studies (0.85 vs. 0.82, p < 0.01); moreover, studies with ≥ 3 readers showed significantly lower sensitivity than studies with < 3 readers (0.81 vs. 0.84; p = 0.03). Our findings provide additional evidence for exploring the consistency of CO-RADS ≥ 4 among different centers and whether the cutoff value of CO-RADS ≥ 4 should be recommended when there are < 3 readers. All the included studies recruited patients between January and June 2020, when there were rapid changes in the COVID-19 epidemic in different counties [59,60], which may cause detection difficulties and finally lead to errors or heterogeneity. Moreover, not all studies were well-designed. Therefore, there is a need for better-designed studies on CO-RADS.
Subgroup analysis according to age revealed lower sensitivity in the older group with CO-RADS ≥ 4. Age affects the diagnostic performance for COVID-19 since older adults often have chronic lung diseases, including chronic obstructive pulmonary disease and interstitial pulmonary fibrosis [61].
Further, COVID-19 presentation in older patients is often atypical with less unspecific symptoms [62,63], which may impede diagnosis through CO-RADS. Subgroup analysis according to sex revealed that most studies included more male participants, which is consistent with the sex-based susceptibility for COVID-19 infection [64][65][66]. Studies that included fewer male participants had higher sensitivity and specificity than those with more male participants, which may have translated to potential factor, including smoking, that affects CT scanning [66,67], but there is a need for more well-designed studies to confirm the lack of statistical significance. Regarding the symptom subgroup, there was a significant difference in the sensitivity of CO-RADS ≥ 4; however, we could not determine the group with higher sensitivity. Further related studies are warranted. Moreover, studies enrolling both symptomatic and asymptomatic patients showed higher specificity for CO-RADS ≥ 3/4 than those only enrolling symptomatic patients, which is consistent with a previous report of higher specificity in asymptomatic patients (0.89) than in symptomatic patients (0.73) [25]. This could be attributed to patients with chronic lung diseases, who have respiratory symptoms similar to those of COVID-19, being included in the symptomatic group, which decreases the specificity [25,68]. Moreover, this result could be attributed to subjective factors among observers and the local COVID-19 prevalence [24].
This study has several limitations. First, there was considerable heterogeneity among the included studies. Although we performed both meta-regression and subgroup analyses, there might have been insufficient exploration. Therefore, heterogeneity might be a concern. Second, some potential factors may remain unknown. Third, although we included 24 studies on CO-RADS, the available studies in each category were < 24 since some studies only provided data on one category. Finally, most of the included studies were retrospective, which led to a high risk of bias in the patient selection domain.
Our findings demonstrated that CO-RADS has favorable performance in detecting COVID-19. CO-RADS ≥ 3 and CO-RADS ≥ 4 might be suitable in different situations given their high sensitivity and specificity, respectively. However, future better-designed studies are warranted to further explore the consistency of CO-RADS among different centers and readers.
Funding This study was supported by (1) 1.3.5 project for disciplines of excellence, West China Hospital, Sichuan University (ZY2016104), and (2) Chinese National Natural Science Foundation (31801066).

Declarations
Guarantor The scientific guarantor of this publication is Jiaming Liu.

Conflict of Interest
The authors declare no competing interests.

Statistics and biometry
No complex statistical methods were necessary for this paper.
Informed consent None; this study is a systematic review and metaanlysis.
Ethical approval Institutional Review Board approval was not required because this studies was not related with human nor animal.