{'fraction_non_alphanumeric': 0.03399660926206835, 'fraction_numerical': 0.020121352725974838, 'mean_word_length': 5.193699917104173, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 2, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Diagnostic Accuracy of Subtraction Coronary CT Angiography in Severely Calcified Segments: Comparison Between Readers With Different Levels of Experience

Purpose Subtraction coronary CT angiography (CCTA) may reduce blooming and beam-hardening artifacts. This study aimed to assess its value in improving the diagnostic accuracy of readers with different experience levels. Method We prospectively enrolled patients with target segment who underwent CCTA and invasive coronary angiography (ICA). Target segment images were independently evaluated by three groups of radiologists with different experience levels with CCTA using ICA as the standard reference. Diagnostic accuracy was measured by the area under the curve (AUC), using ≥50% stenosis as the cut-off value. Results In total, 134 target segments with severe calcification from 47 patients were analyzed. The mean specificity of conventional CCTA for each group ranged from 22.4 to 42.2%, which significantly improved with subtraction CCTA, ranging from 81.3 to 85.7% (all p < 0.001). The mean sensitivity of conventional CCTA for each group ranged from 83.3 to 88.0%. Following calcification subtraction, the mean sensitivity decreased for the novice (p < 0.001) and junior (p = 0.017) radiologists but was unchanged for the senior radiologists (p = 0.690). With subtraction CCTA, the mean AUCs of CCTA significantly increased: values ranged from 0.53, 0.54, and 0.61 to 0.70, 0.74, and 0.85 for the novice, junior, and senior groups (all p < 0.001). Conclusion Subtraction CCTA could improve the diagnostic accuracy of radiologists at all experience levels of CCTA interpretation.


INTRODUCTION
Coronary computed tomography angiography (CCTA) has become a cost-effective first line technique in the diagnosis of patients with chest pain (1,2). The diagnostic performance of CCTA varies depending on the level of experience of the reading physician. It has been proven that a greater level of experience leads to improvements in the diagnostic accuracy of CCTA (3, 4).
In addition, inter-reader variability in CCTA interpretation also differs according to the readers' levels of experience. Even with the use of the Coronary Artery Disease Reporting and Data System (CAD-RADS), the standardized reporting system for coronary artery diseases, the inter-reader agreement of expert readers was stronger than that of early career readers (5). Severe coronary calcification is a major challenge for readers in interpreting CCTA. The "blooming" and beam-hardening artifacts of severely calcified plaques may cause difficulties for readers in estimating coronary luminal stenosis. The degree of coronary artery stenosis with severe calcification is often overestimated or underestimated, and in these segments, the diagnostic accuracy of CCTA subsequently decreases (6,7). To address this problem, a subtraction CCTA method was proposed to allow elimination of the artifacts from calcified plaques (8). Several studies indicated that subtraction CCTA improved the diagnostic accuracy and increased the diagnostic confidence of readers when evaluating severely calcified segments (9-14). However, to our knowledge, whether the diagnostic accuracy and diagnostic confidence of subtraction CCTA varies among readers with different levels of experience remains unknown.
To address this issue, our research aimed to evaluate the value of subtracting CCTA in diagnosing severely calcified coronary artery stenosis by investigating the impact of this method on the diagnostic accuracy and diagnostic confidence of readers with different levels of experience.

Study Population
Patients with suspected coronary artery disease were prospectively selected to undergo subtraction CCTA at our institution from October 2019 to June 2020. The inclusion criteria were as follows: (1) age ≥ 50 years, (2) basal or drug control heart rate (HR) ≤65 bpm, (3) no contraindications for iodinated contrast material, (4) no history of cardiac surgery, (5) sinus rhythm, no history of arrhythmia or heart failure, and (6) willingness to provide written informed consent. The exclusion criteria were as follows: (1) no severe calcifications, (2) no invasive coronary angiography (ICA) or ICA performed more than 1 month after subtraction CCTA, and (3) severe motion artifacts. This study was approved by the ethics committee of our hospital, and all patients provided informed consent.

Conventional CCTA Data Acquisition
The CT scanner used in this study was a 320-detector row CT scanner (Aquilion ONE Vision Edition, Canon Medical Systems) with 0.5 mm detector elements, a rotation time of 275 ms, and the Adaptive Iterative Dose Reduction 3D (AIDR 3D) algorithm. A two-breath hold acquisition technique was used (8). A cardioselective beta-blocker (metoprolol 25-150 mg) was administered orally 1 h before scanning for individuals with a heart rate > 65 bpm and no contraindications to the substance.
Tube voltage and tube current were determined by an automated exposure control function ( SURE Exposure) with target image noise with a standard deviation (SD) of 22 Hounsfield units. Depending on the body mass index (BMI), 100 or 120 kV was used for both scans. The phase window was set at 70-80% of the R-R interval. Iodinated contrast medium with an iodine concentration of 370 mg I/ml (Iopamidol-370, Bayer Schering Pharma, Berlin, Germany) was injected at a rate of body weight (kg) × 0.07 mL/s in 10 s (fixed), followed by a 50-mL saline chaser bolus.

Subtraction CCTA
Canon Sure Subtraction Coronary software was used for subtraction CCTA as previously reported (8). First, non-contrast and contrast images were reconstructed at 70-80% phases at 1% intervals to obtain the best-quality image sequence. Non-contrast and contrast image sequences without motion artifacts were selected as the best-quality image sequences for registration. The registration was performed in two steps. In the first step, a global non-rigid registration was performed. In the next step, a local rigid registration for the targeted coronary segment followed. Consequently, the subtraction images obtained. An example is shown in Figure 1.

Image Analysis
Canon Workstation perfusion software (Vitrea fX, Version 40693, Vital Images, Minnetonka, MN and Canon Medical Systems, Otawara, Japan) was used to post-process the conventional and subtraction CCTA images. Eight radiologists, including three novices (who had performed between 50 and 100 CCTA interpretations), two junior radiologists (who had performed ∼500 CCTA interpretations) and 3 senior radiologists (who had performed >1,000 CCTA interpretations), independently read the CCTA images twice. The first time, only conventional CCTA images were available; the second time, both conventional and subtraction CCTA images were available. All images were anonymized and presented in random order. As the memory washout period, the interval between the two readings was at least 2 weeks. All readers were blinded to the patient's clinically relevant information and coronary angiography results. Coronary artery stenosis was assessed in segments with a diameter of ≥ 1.5 mm based on the 18-segment Society of Cardiovascular Computed Tomography (SCCT) model (15). The estimation of the coronary stenosis was assessed visually. The degree of lumen stenosis was divided into the following 7 categories: 1-normal: no plaque and no luminal stenosis; 2-slight stenosis: plaque present, luminal stenosis: <25%; 3-mild stenosis: 25-49%; 4-moderate stenosis: 50-69%; 5-severe stenosis: 70-99%; 6-occlusion; 7-uninterpretable due to the presence of calcification. Patients with a percentage stenosis of <50% were classified as the stenosis (-) group, while those with a percentage stenosis of ≥50% were classified as the stenosis (+) group. Uninterpretable segments were recorded and considered to be stenotic in the accuracy analysis. Image quality was evaluated by using a 4-point scale. Severe calcification was defined as a cross-section calcification artifact arc ≥180 • (16). Diagnostic confidence in the assessment of the degree of luminal stenosis was rated using a 5-point Likert scale as follows: 1-very low; 2-low; 3-moderate; 4-high; and 5-very high.
All ICA images were interpreted by two expert interventional cardiologists with more than 10 years of clinical ICA experience who were blinded to all patient characteristics and CCTA findings. Any discrepancy between the observers was settled by consensus. Stenosis equivalent to at least 50% of the diameter was defined as obstructive CAD for ICA and CCTA.

Statistical Analysis
Statistical analysis was performed by using SPSS software (version 26). Diagnostic confidence was computed for each radiologist and each reading group with different levels of experience. Group differences in diagnostic confidence were compared by using Student's t-test. Taking the ICA results as the standard reference, the sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) of conventional and subtraction CCTA were calculated for each radiologist and each reading group. Because of the presence of paired samples, the McNemar test was applied to explore group differences in sensitivity and specificity. The overall diagnostic accuracy was measured with respect to the ICA results by the AUC using the total number of points. The AUCs were compared by using the Hanley-McNeil method (17). All tests were two tailed, and the significance threshold was p < 0.05.

Baseline Information
Forty-seven patients (26 men and 21 women) were enrolled, with a mean age of 67.1 ± 7.3 years. The median Agatston score was 427.1 (IQR: 235.4-886.3). The baseline characteristics of the patients are summarized in Table 1. After exclusion at the patient level, we performed another exclusion at the segment level. Out of a total of 568 segments, 36 segments had diameters <1.5 mm, and 19 segments were excluded because the image quality was <2 points. Of the remaining 513 segments, 379 segments were excluded because there were no target segments. Finally, 134 severely calcified target segments were analyzed, including 49 eccentric calcification and 85 concentric calcification. Among the 134 target segments in this study, 87 segments were well-coregistered, 47 segments had different degrees of misregistration. The enrollment workflow is summarized in Figure 2.

Diagnostic Confidence for Novice, Junior, and Senior Radiologists
For the conventional CCTA data, the mean diagnostic confidence was 2.6 ± 1.0, 3.2 ± 0.5, and 2.7 ± 0.9 for the novice, junior, and senior reading groups, respectively. Comparatively, each reading group showed significantly stronger diagnostic confidence with subtraction CCTA data, with mean values of 3.4 ± 0.8, 4.0 ± 0.7, and 4.5 ± 0.7, respectively (all p < 0.001, Figure 3). The diagnostic confidence of each reader significantly increased after calcifications were subtracted (all p < 0.001, Table 2).
Comparison Between the Sensitivity, Specificity, PPV, and NPV of Conventional and Subtraction CCTA for Radiologists With Different Levels of Experience When using conventional CCTA, 9, 4, and 9 segments were rated as uninterpretable by at least one reader in the novice, FIGURE 2 | Flowchart of patient enrollment. CCTA, coronary computed tomography angiography; ICA, invasive coronary angiography; AF, atrial fibrillation; N represents the number of patients; n represents the number of segments.
FIGURE 3 | Mean diagnostic confidence with conventional and subtraction coronary computed tomography angiography (CCTA) for novice, junior, and senior radiologists. The chart shows that the mean diagnostic confidence of subtraction CCTA is significantly higher than that of conventional CCTA, with p < 0.001 for all three groups. The * symbol indicates the value p < 0.01. junior and senior reading groups, respectively. However, when information from subtraction CCTA was included, the number of uninterpretable segments decreased to 4, 2, and 0 for each group. As shown in Table 3, the mean sensitivity of conventional CCTA for each group ranged from 83.3 to 88.0%. Following calcification subtraction, the mean sensitivity decreased for the novice (p < 0.001) and junior radiologists (p = 0.017) but remained similar for the senior radiologists (p = 0.690). The mean specificity of conventional CCTA for each group ranged from 22.4 to 42.2%, which significantly improved following the inclusion of subtraction CCTA, ranging from 81.3 to 85.7% (all p < 0.001). The mean PPVs and NPVs of subtraction CCTA were both higher than those of conventional CCTA.
As shown in Table 5, in the subgroup analysis of eccentric calcification and central calcification, we found that for concentric calcification, subtraction can significantly improve the AUC for novice, junior and senior radiologists, and for eccentric calcification, the AUC for senior radiologists was significantly improved, while the AUC for novice and junior radiologists was not significantly improved.

DISCUSSION
Since calcified lesions are associated with blooming and beamhardening artifacts and partial volume effects, the assessment of luminal stenosis is often difficult or impossible (7,(18)(19)(20). Some previous studies have shown that the application of subtraction CCTA can reduce the number of uninterpretable segments, improve image quality and improve diagnostic performance (9, 21). However, in those studies, CCTA data were analyzed exclusively by experienced cardiovascular radiologists, and it remains to be ascertained whether high diagnostic performance can also be achieved in readers with limited CTA experience. In this article, we simulate scenarios of the application of subtraction CCTA for readers with different levels of experience. The most significant findings of the present study were that subtraction CCTA reduced the number of uninterpretable segments with severe calcification. Furthermore, subtraction CCTA improved the diagnostic accuracy and confidence of readers with different levels of experience in coronary angiography interpretation at both the individual and group levels.
Research by Amanuma et al. (10) shows that when the information provided by subtraction CCTA was added, the number of non-assessable segments was reduced from 74 to 18. Consistent with previous findings, we demonstrated that the numbers decreased from 9, 4, and 9 to 4, 2, and 0 for each group.
De Santis et al. (22) scored diagnostic confidence using 5-point Likert scales. Their research shows that the calcium subtraction algorithm improved diagnostic confidence and that the mean score increased from 3.1 to 4.0. From our research, we can also see that calcification had an impact on physicians' diagnostic confidence. Regardless of whether they were experienced, the confidence in the CCTA diagnosis of severely calcified segments was not high. After calcium subtraction, the diagnostic confidence significantly improved at both the individual and group levels.
The quality of CCTA interpretation increases with the level of experience (23). It is unclear whether there are risks or benefits to expanding the clinical use of subtraction CCTA among inexperienced readers. According to our research results, with subtraction CCTA information, the mean specificity of senior radiologists was significantly improved, and the sensitivity was not decreased, which was similar to most previous research results. The study by Amanuma et al. (10) showed that the specificity was increased from 54.5 to 78.5%, but the sensitivity was not changed. A multicenter study (9) showed that the subtraction procedure lowered the false-positive rate in target segments without misregistration from 72 to 33% (false positives equal to 1-specificity) at the expense of 7% false negatives in subtraction CCTA. Different from these findings, it was also observed in our study that the novice and junior radiologists showed significant increases while the sensitivity was decreased. We deem the reason   for this phenomenon is that all segments interpreted in our study were severely calcified segments; however, with the conventional CCTA images, the novice and junior radiologists overestimated the degree of stenosis, and the incidence of coronary artery stenosis in severely calcified segments was relatively high, resulting in exaggerated sensitivity. The previous literature shows that the sensitivity of inexperienced radiologists is significantly lower than that of experienced radiologists in the interpretation of routine CCTA (23), while the sensitivity of novice and junior radiologists in our results was no less than that of senior radiologists because the target segments in our study were severely calcified. Severe calcification results in increased false positives (low specificity) and decreased false negatives (high sensitivity). Therefore, the sensitivity of novice and junior radiologists in our study was exaggerated when only the conventional CCTA images were read. When adding subtraction information, the calcification was removed, and the lumen was exposed to all readers, whose sensitivity and specificity reflected their true ability to interpret CCTA. Significant increases in AUC values were observed in novice, junior and senior radiologists, suggesting that FIGURE 4 | The overall diagnostic accuracy of conventional and subtraction coronary computed tomography angiography (CCTA) for novice, junior, or senior radiologists. The overall diagnostic accuracy was calculated by the area under the curve (AUC). The solid line represents subtraction CCTA, and the dashed line represents conventional CCTA. The colors green, blue and purple represent the novice, junior, and senior radiologists, respectively. The AUC of subtraction CCTA was significantly higher than that of conventional CCTA in each radiologist group stratified by working experience (all p < 0.001).
calcification subtraction was beneficial for all radiologists with different experience levels. Senior readers performed better than novices in evaluating lumen stenosis on subtraction CCTA images, especially in eccentrically calcified segments. For eccentric calcification, part of the lumen was usually obscured by calcification artifacts, which required multiangle observation, which depended on the experience of the reader.
Our current study is unique in that we analyzed the influence of subtraction CCTA on the performance of readers at various experience levels with CCTA interpretation. Following the inclusion of subtraction CCTA images, the specificity and PPV were significantly improved in the novice, junior and senior radiologist groups, and the AUC and diagnostic confidence were significantly improved at the individual and group levels. This means that subtraction CCTA can benefit readers of all levels. Calcification subtraction not only further improves the diagnostic accuracy of senior radiologists, but also novice and junior radiologists to achieve a higher level of accuracy in the diagnosis of severe calcification lesions. Therefore, we recommend the subtraction CCTA to all levels of readers. In subtraction CCTA, there are two methods for obtaining the non-contrast CT image data. One is a single breath-hold method and the other is a two breath-hold method. Both methods have their own advantages and disadvantages (12,13). The single breath hold method would require a single breath-hold of 20-40 s. Considering that long-term breath-holding may be difficult for elderly patients with severe coronary calcification, we used the two breath-hold method described by Yoshioka et al. (8). Among the 134 target segments in this study, 87 segments were well-coregistered, and 47 segments had different degrees of misregistration. The misregistration artifacts in this study was 35% (47/134), which was significantly lower than that of the multicenter study by Fuchs et al. (35 vs. 50%) (9), and close to that of the study by Yi et al. (35 vs. 32.5%) (24), both of which used the two breath-hold method. When the readers interpreted CCTA for the second time, the subtraction CCTA image was used as a supplement to the conventional CCTA image. Therefore, we did not exclude misaligned segments. This may be a limitation of this study, but this is more concordant with the clinical routine. Whether it is the one-breath-hold method or the twobreath-hold method, misregistration is inevitable. We should take measures to minimize the misregistration, but despite the presence of misregistered segments, readers with different experience can benefit from subtraction CCTA, which shows that subtraction CCTA is useful. As dual-energy CT without the need for additional image acquisition, the problems regarding misregistration artifacts could be solved (22). The second limitation is that the target segment in this paper is the severely calcified segment defined by visual assessment. The Agatston scores of some patients were not high. Although the Agatston score is more objective than visual methods, the segment calcium severity is not linearly associated with the patient's Agatston score (25). The third limitation is that the sample size of interpreters was not large enough. In future research, we will enroll more samples and conduct a multicenter study.

CONCLUSION
Subtraction CCTA could improve the diagnostic accuracy and diagnostic confidence of readers with different levels of experience in coronary CT angiography interpretation. Not only the senior readers with rich experience, but also readers with less experience.

DATA AVAILABILITY STATEMENT
The original contributions presented in the study are included in the article/supplementary materials, further inquiries can be directed to the corresponding authors.

ETHICS STATEMENT
The studies involving human participants were reviewed and approved by Ethics Committee of Beijing Friendship Hospital, Capital Medical University. The patients/participants provided their written informed consent to participate in this study.

AUTHOR CONTRIBUTIONS
YH, ZY, ZW, and YX were contributed to the conception, design, and analyzed the data. FL and QH were contributed to the experiments and wrote the draft manuscript. LX was contributed to methodology statistics and edited the manuscript. YZ and YS were contributed to study implement. All authors approved the final manuscript submitted.