{'fraction_non_alphanumeric': 0.0426972031841982, 'fraction_numerical': 0.02566940530415904, 'mean_word_length': 5.06871609403255, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Interobserver agreement for contrast-enhanced ultrasound of liver imaging reporting and data system: A systematic review and meta-analysis

BACKGROUND Hepatocellular carcinoma is the most common primary liver malignancy. From the results of previous studies, Liver Imaging Reporting and Data System (LI-RADS) on contrast-enhanced ultrasound (CEUS) has shown satisfactory diagnostic value. However, a unified conclusion on the interobserver stability of this innovative ultrasound imaging has not been determined. The present meta-analysis examined the interobserver agreement of CEUS LI-RADS to provide some reference for subsequent related research. AIM To evaluate the interobserver agreement of LI-RADS on CEUS and analyze the sources of heterogeneity between studies. METHODS Relevant papers on the subject of interobserver agreement on CEUS LI-RADS published before March 1, 2020 in China and other countries were analyzed. The studies were filtered, and the diagnostic criteria were evaluated. The selected references were analyzed using the “meta” and “metafor” packages of R software version 3.6.2. RESULTS Eight studies were ultimately included in the present analysis. Meta-analysis results revealed that the summary Kappa value of included studies was 0.76 [95% confidence interval, 0.67-0.83], which shows substantial agreement. Higgins I2 statistics also confirmed the substantial heterogeneity (I2 = 91.30%, 95% confidence interval, 85.3%-94.9%, P < 0.01). Meta-regression identified the variables, including the method of patient enrollment, method of consistency testing, and patient race, which explained the substantial study heterogeneity. CONCLUSION CEUS LI-RADS demonstrated overall substantial interobserver agreement, but heterogeneous results between studies were also obvious. Further clinical investigations should consider a modified recommendation about the experimental design.


INTRODUCTION
Hepatocellular carcinoma (HCC) is the most common primary liver malignancy, and it is the second-most frequent cause of cancer-related deaths [1,2] . HCC often occurs in patients with risk factors, such as chronic hepatitis and cirrhosis [3] .
The prognosis of patients with advanced HCC is poor, and curative treatments in patients with early stage HCC are needed [4] . Unlike other systemic malignancies, HCC is diagnosed noninvasively based on imaging characteristics without mandatory pathological confirmation in at-risk patients [5] . Countries with high incidence rates of HCC traditionally screen for high-risk patients using imaging examination, and these patients are closely followed. The focal liver lesions (FLLs) found in HCC screening on various imaging examinations are definitively diagnosed using contrast-enhanced diagnostic imaging examinations, including contrast-enhanced ultrasound (CEUS), contrast-enhanced computed tomography (CECT), and contrast-enhanced magnetic resonance imaging (CEMRI) [6] .
In the abovementioned contrast-enhanced imaging examinations, CEUS allows noninvasive assessments of the contrast enhancement model of HCC without the use of ionizing radiation and with a much higher temporal resolution than computed tomography (CT) and magnetic resonance imaging (MRI) [7,8] . CEUS continues to gain traction as a technique that complements traditional B-mode and Doppler ultrasound in the evaluation of the liver and other organs [9] . CEUS shows changes in microvascular flow mechanics in the focus using real-time imaging of tissue perfusion, which also yields supplementary information, including flow in the microvasculature, slow flow, and perfusion kinetics [10] . CEUS exhibits high accuracy in the differential diagnosis of FLLs in cirrhosis and non-cirrhotic livers [11,12] . CEUS exhibits the same sensitivity and specificity for the differential diagnosis of FLLs, but it is more economical and effective than CECT and CEMRI [13] .
Because imaging is important for the diagnosis and treatment decisions in HCC, it is necessary to standardize the imaging diagnosis of HCC and improve its diagnostic accuracy [14,15] . Considering this background, a group of international experts convened by the American College of Radiology (ACR) proposed the Liver Imaging Reporting and Data System (LI-RADS) to standardize the interpretation and reporting of HCC in 2014 [16] . The ACR released the CEUS LI-RADS in 2016, with revisions in 2017, and it has become a standardized system for the technique, interpretation, reporting, and data collection for CEUS exams in patients who are at risk for developing HCC [17] . CEUS LI-RADS integrates with the previously released CT/MRI LI-RADS, which provides the criteria for ordinal categories and definitions of the major and ancillary features for HCC [18] . CEUS allows radiologists to (1) use consistent terminology, (2) reduce variability and mistakes in imaging interpretation, (3) promote communication with referring clinicians, and (4) facilitate research and quality assurance [18] . Therefore, the standardized diagnosis helps promote standardization and reproducibility across institutions and radiologists [19] .
However, a unified conclusion on the interobserver stability of this innovative ultrasound imaging has not been determined. Several previous studies reported research on the repeatability of LI-RADS on CEUS. The results of some clinical trials revealed strong controversies in interobserver agreement. Schellhaas et al [20] demonstrated that the Kappa value was just 0.39, and Tan et al [21] showed a Kappa value of 0.94 using LI-RADS on CEUS.
Based on the abovementioned studies, the present meta-analysis examined the interobserver agreement of CEUS LI-RADS to provide some reference for subsequent related research.

Literature retrieval
The meta-analysis was performed on relevant literature that was published as late as March 1, 2020 in the databases of PubMed, Web of Science, Embase, China Biology Medicine disc, Cochrane Library, Google Scholar, China National Knowledge Infrastructure, WANFANG databases, and ClinicalTrials.gov. No restriction on language was applied. Search keywords were HCC, CEUS, LI-RADS, and their synonyms: 'liver neoplasm' or 'liver cancer' or 'liver malignancy' or 'hepatocellular carcinoma' or 'HCC'; 'Liver Imaging Reporting and Data System' or 'LI-RADS'; and 'contrast-enhanced ultrasound' or 'CEUS'.

Inclusion and exclusion criteria
Inclusion criteria: Research that met all of the following criteria were included: (1) Study types: Observational studies, such as retrospective or prospective; (2) Population: patients at-risk of HCC who needed regular observation, such as patients with cirrhosis and hepatitis B virus carrier; (3) Index tests: CEUS; and (4) Outcomes: sufficiently detailed information to evaluate interobserver agreement for CEUS of LI-RADS.
Exclusion criteria: Papers were excluded if they met the following conditions: (1) Editorials, comments, letters, cases reports, and reviews; (2) Experimentation on animals; (3) Repetitive studies and research topic of documents without meeting the requirements; (4) Studies that were not related to the field of interest of the present November 26, 2020 Volume 8 Issue 22 research; and (5) Laboratory studies.

Data extraction
Two reviewers independently screened the appropriate articles according to the inclusion and exclusion criteria detailed above. Discrepancies in opinion between the two reviewers were resolved via consult with an additional researcher for reevaluation at a consensus meeting.
The following data were distilled from the included studies using predefined data formats: (1) Article characteristics, including authors, publication years, and study designs (prospective or retrospective); (2) Process characteristics, including enrollment method of patients (selective or consecutive), the number of patients, age and gender ratio, the number of FLLs, and ratio of benign to malignant; (3) Ultrasound (US) system; (4) LI-RADS version; and (5) Reference standard, including pathology and synthesized clinical reference standard (SCRS). The Kappa value for categorical variables was extracted for each major feature and LI-RADS categorization.

Literature quality evaluation
The bias risk of the included papers was assessed using QUADAS-2 domains [22] . The answers to the symbolic questions of each of the five sections were either "yes", "no", or "unclear" corresponding to the judgment of the risk level of bias as "low", "high", or "uncertain". If the answers to every question were "yes", the study was at "low risk", meanwhile, if the answers to all questions were "no" or "unclear", the study was judged as "high risk". If one of these answers was "no" or "unclear", the study was placed under "uncertain". Revman 5.3, special software for Cochrane collaborative network was used to output the results of QEDAST.

Statistical analysis
To calculate meta-analysis summary estimations, the Kappa value with standard error categorization was summarized. We estimated standard error from the 95% confidence interval (CI) if it was not mentioned in the original studies. The metaanalysis pooled Kappa value with 95%CI was calculated using the DerSimonian-Laird model with Knapp and Hartung adjustment [23] . According to Landis and Koch, Kappa value was categorized as follows: < 0.20, poor; 0.21-0.40, fair; 0.41-0.60, moderate; 0.61-0.80, substantial; and 0.81-1.00, almost perfect agreement [24] . Substantial heterogeneity existed if the value of I 2 statistics exceeded 50% and the P value did not exceed 0.10 using the Cochran Q-test. A sensitivity analysis or subgroup analysis was performed when heterogeneity was noted, and data synthesis was selected for the random effect model. If the reasons for heterogeneity required further exploration, then a metaregression analysis was performed using covariates in the bivariate model. Statistical significance was denoted at P < 0.05. Funnel plot was used for diagnostic metaanalysis to assess the publication bias of included articles, and significant asymmetry was denoted at P < 0.10 for the slope coefficient. We used the "meta" package and "metafor" package in R software version 3.6.2 for analysis and synthesis (R Foundation, Vienna, Austria).

Results of the literature search
A detailed flow chart of the study selection process is shown in Figure 1. An aggregate of 129 articles was originally identified using the search strategy, and 54 articles were filtered after excluding duplicates. Twenty-two of the remaining studies were removed, including 19 articles that were unrelated to the field of interest and three review articles. The full texts of the remaining 13 studies were obtained. After review of the full transcripts, five other articles were excluded. Eight studies were ultimately eligible for meta-analysis.

Characteristics of eligible studies
The meta-analysis included eight studies (five in English, three in Chinese) with a total of 1177 patients and 1379 FLLs [3,6,20,21,[25][26][27][28] . The major characteristics and basic information of the included articles are detailed in Table 1. The publication dates were 2017 to 2020. The patients who conformed to the inclusion and exclusion criteria from seven studies were Easterners [3,6,21, [25][26][27][28] , and only one article included Westerners [20] . Only one article was a cohort study [26] , and the others were retrospective

Reference standard
Chen et al [25] 2017 studies [3,6,20,21,25,27,28] . The enrollment method of one article was selective cohort [21] , and the other articles used consecutive cohorts [3,6,20,[25][26][27][28] . The classification standard for CEUS of the four articles was the LI-RADS 2016 version [20,25,27,28] , and the other studies selected the LI-RADS 2017 version [3,6,21,26] . The two articles published in the same year were from the same first author, but the samples and methods selected were not the same [3,6] . To distinguish these studies in the present meta-analysis, the article published in the Journal of Ultrasound in Medicine is identified using the first author (Wang et al [6] ), and the article published in Ultrasound Medicine Biology is identified using the corresponding author (Cui et al [3] ). Three articles were dissertations [25][26][27] . Only one study used pathology as a reference standard [28] , and the other seven studies used a combination of pathology and SCRS [3,6,20,21,[25][26][27] .

Literature quality evaluation
Results of the quality assessment of the included articles are shown in Figure 2. The results revealed that relatively acceptable quality evaluations could be acquired from the involved studies.  in Table 2 and Table 3. Tan et al [21] concluded that the interobserver agreement of LI-RADS was in near perfect agreement (Kappa value = 0.94; 95%CI, 0.89-0.97). The Higgins I 2 statistics indicated substantial heterogeneity in the summary Kappa value (I 2 = 91.30%, 95%CI, 85.3%-94.9%, P < 0.01). Therefore, the pooled calculation used a random effect model. The pooled Kappa value from the random effect model was 0.76 (95%CI, 0.67-0.83), which showed substantial agreement. The forest plot of summary Kappa values is shown in Figure 3.

Publication bias
The P value of the linear regression test of funnel plot asymmetry was 0.39, and we determined that no publication bias existed in the statistics. The Egger's funnel plot is shown in Figure 4.

Sensitivity analysis
One study at a time was omitted from analysis, and the results of the sensitivity analysis are shown in Table 4, which showed no literature influence. Exclusion of the included articles one by one revealed that the Higgins I 2 also did not change significantly. The consolidation result was relatively stable.

Meta-regression analysis
Because of the strong heterogeneity of pooled studies in the merger statistics, a metaregression was used. The meta-regression analysis analyzed some clinically relevant variables of investigation, including study design (retrospective or prospective), method of patient enrollment (consecutive or selective), LI-RADS version (2016 or 2017), number of interobservers (two or three), number of US systems used (one or more), race of chosen patients (Easterners or Westerners), number of diagnosing FLLs (less than 100 or more than 100), and reference standard (only pathology or combination of pathology and SCRS). The results of regression analysis are exhibited in Table 5. The variables of method of patient enrollment (P < 0.01), number of November 26, 2020 Volume 8 Issue 22 Table 2 Analysis of interobserver agreement for included studies

DISCUSSION
Meta-analysis of interobserver agreement of LI-RADS on CEUS was not reported previously. The summary Kappa value for the eight included studies was 0.76 (95%CI, 0.67-0.83) in our study, which showed substantial inter-reader agreement for the use of LI-RADS on CEUS. The CEMRI is another common noninvasive imaging method to assess benign and malignant FLLs. Notably, the LI-RADS on MRI to evaluate FLLs was developed in 2011 and recently updated in 2018 [29] . The meta-analysis of Kang et al [30] revealed that summary interobserver agreement of LI-RADS on MRI was 0.70 (95%CI, 0.56-0.85). Another multicenter international study, which used a large number of readers and a mixture of all LI-RADS category assignments, obtained a similar result (Kappa value = 0.73; 95%CI, 0.68-0.77) [19] . The interobserver agreement of LI-RADS on CEUS seems better than that of LI-RADS on MRI. Notably, CEUS avoids the disadvantages of MRI, such as high expense and a long inspection time. Some researchers demonstrated that the sensitivity of CEUS in the observation of arterial hypervascularity from nodules in liver cirrhosis was significantly higher than that of MRI [31][32][33] . Two recent meta-analyses of CEUS showed excellent diagnostic accuracy in differentiating malignant from benign FLLs with a summary sensitivity of 0.92 and summary specificity of 0.87, and the sensitivity of CEMRI was slightly weaker than that of CEUS with a pooled sensitivity of 0.86 and pooled specificity of 0.89 [34,35] . November 26, 2020 Volume 8 Issue 22

Figure 2 Bias risk of the included studies (QUADAS 2 criteria)
. 1 This is our study.
The majority of HCCs are not suitable for curative resection at the time of treatment, and difficulties of surgical resection may be related to size, site, and number of tumors, vascular and extrahepatic involvement as well as liver function of the patient [36] . Radiofrequency ablation (RFA) is another effective treatment for liver cancer, and it has emerged in clinical practice to expand the pool of patients considered for liverdirected therapies [37] . Traditionally, RFA is usually performed under B-mode US guidance. In recent years, some scholars have reported the treatment technique of RFA guided by CEUS for HCC. Miyamoto et al [38] exhibited the complete ablation rate after a single treatment session was significantly higher in CEUS group than in the B-mode US group. Moreover, Masuzaki et al [39] reported in a large-scale study that the detectability of tumor nodules was 83.5% in B-mode US and 93.2% in CEUS (P = 0.04). Therefore, the use of CEUS guidance in RFA for liver cancer is an efficient approach.
The eight studies included in the present meta-analysis also exhibited some problems. For example, most studies did not list the interobserver agreement for the major features of CEUS in detail. Notably, the CEUS LI-RADS criteria requires the combination of two major features, including arterial phase hyperenhancement (APHE) and washout, to distinguish benign and malignant FLLs [17] . Unfortunately, just two articles mentioned the Kappa value of APHE and washout [3,20] . Therefore, it is recommended that further research on the interobserver agreement of CEUS LI-RADS add an extra consistency test of the major features of CEUS to increase the persuasiveness of the research.
The present meta-analysis used meta-regression to compensate for the high heterogeneity and analyzed existing covariates due to several potential causes. The method of patient enrollment had a significant impact on the Kappa value. Tan et al [21] used selective screening as an inclusion standard rather than consecutive screening, which was different from the other seven articles, and the Kappa value of his research was obviously higher than the summary values of the other articles (0.94 vs 0.73, P < 0.01). However, selective screening of patients caused a small sample size of FLLs, which may introduce potential confounders and bias.
Consistency tests of the eight included studies revealed that the number of reviewers also affected heterogeneity. Schellhaas et al [20] used two reviewers for interobserver agreement and achieved a satisfactory Kappa value using pairwise comparisons, and this Kappa value was lower than in the other articles (0.39 vs 0.79, P November 26, 2020 Volume 8 Issue 22  < 0.01). The low Kappa value may be because the calculation of Kappa relies on the assumption that a significant proportion of agreement is due to chance, and if a feature is observed very frequently, then a low Kappa value between the observers results [40] . The author indicated that the reason for the low Kappa value was that the calculation of Cohen's Kappa was influenced by the frequency of a certain feature being observed [20] . Therefore, the use of intraclass coefficient correlation rather than Cohen's Kappa to represent interobserver agreement in multiple observers would be more satisfactory. infection led to HCC in eastern Asia and sub-Saharan Africa, and non-alcoholic fatty liver disease was the major precipitating factor of HCC in Western countries [41] . Nonalcoholic fatty liver diseases (NAFLD) are one of the most prevalent causes of chronic liver diseases in Western countries, with an estimated prevalence of 20%-40% [42] . In the context of fatty liver, the diagnosis of HCC that progressed from NAFLD may be more difficult. However, there is no relevant literature on this issue to research in the databases. Therefore, the diagnosing of HCC progressed from NAFLD and viral stimulation on CEUS is a hot issue. Because most of the included articles were retrospective studies, the US system and reference standard of these studies were partially diversified. However, the metaregression analysis showed that the study design (0.78 vs 0.61, P = 0.26), the number of US systems (0.77 vs 0.76, P = 0.93), the number of FLLs (0.72 vs 0.82, P = 0.16), and reference standard (0.69 vs 0.77, P = 0.58) did not reach statistical significance for the heterogeneity, which confirmed that the use of LI-RADS on CEUS for the diagnosing of FLLs was stable laterally.
Notably, reliable interobserver agreements for LI-RADS categorization on CEUS were also observed for LI-RADS version 2016 and version 2017 (0.74 vs 0.78, P = 0.66). Because LI-RADS included lexicons, minute definitions and illustrations for imaging features, a high interobserver agreement may be achieved. The two versions of LI-RADS on CEUS define APHE and washout clearly. For example, a feature of APHE may be considered present if it is demonstrated in the entire nodule or only a portion of the nodule [43] . CEUS characterization of washout requires assessment of its onset (late vs early) and degree (mild vs marked), not just its presence [17] . Generally, early (< 60 s) and/or marked washout is a major feature for LR-M, and late (≥ 60 s) and mild washout is a major feature for HCC [44,45] . The diversity of version did not cause heterogeneity of statistical significance, and it was sufficient to authenticate the diagnosis stability of the two versions of LI-RADS on interobserver agreement.
Measurements may differ due to bias between reviewers, which may be explained by the use of the reader's judgment of test results of a measurement [46] . The bias may result from differences in training, learning, and experience between reviewers. The consistency test demonstrated that reviewers in similar working environments used diagnostic reasoning, which may be reduced with continuous education and updated definitions in LI-RADS on CEUS.
The present meta-analysis has several limitations. First, substantial research heterogeneity was mentioned. However, three significant factors for study heterogeneity were found using meta-regression. Second, the different study designs and participants from diverse geographic locations led to the heterogeneous distribution of the disease.

CONCLUSION
In conclusion, the present meta-analysis produced some results. Summary results showed substantial interobserver agreement for LI-RADS on CEUS. The heterogeneity factors included the method of enrolling patients, the method of consistency testing, and the race of patients, which should be considered in subsequent study design. Certainly, a large, prospective, and multicenter study is also needed to confirm our results.

Research background
From the results of previous studies, Liver Imaging Reporting and Data System (LI-RADS) on contrast-enhanced ultrasound (CEUS) for diagnosing hepatocellular carcinoma (HCC) has shown a satisfactory diagnostic value. However, a unified conclusion on the interobserver stability of this innovative ultrasound imaging has not been determined. The present meta-analysis examined the interobserver agreement of CEUS LI-RADS to provide some reference for subsequent related research.