{'fraction_non_alphanumeric': 0.02672147995889003, 'fraction_numerical': 0.017361620907355747, 'mean_word_length': 4.885720457982286, 'pattern_counts': {'":': 0, '<': 8, '<?xml version=': 0, '>': 14, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Does training and experience influence the accuracy of computed tomography colonography interpretation?

AIM
To evaluate the effect of experience on the accuracy rate of computed tomography colonography (CTC) interpretation and patient preferences/satisfaction for CTC and colonoscopy.


METHODS
A prospective, non-randomized, observational study performed in a single, tertiary care center involving 90 adults who underwent CTC followed by colonoscopy on the same day. CTC was interpreted by an abdominal imaging radiologist and then a colonoscopy was performed utilizing segmental un-blinding and re-examination as required. A radiology resident and two gastroenterology (GI) fellows blinded to the results also interpreted the CTC datasets independently. Accuracy rates and trend changes were determined for each reader to assess for a learning curve.


RESULTS
Among 90 patients (57% male) aged 55 ± 8.9 years, 39 polyps ≥ 6 mm were detected in 20 patients and 13 polyps > 9 mm in 10 patients. Accuracy rates were 88.9% (≥ 6 mm) and 93.3% (> 9 mm) for the GI Radiologist, 89.8% (≥ 6 mm) and 98.9% (> 9 mm) for the Radiology Resident and 86.7% and 95.6% (≥ 6 mm) and 87.8% and 94.4% (> 9 mm) for each of the GI fellows respectively. The reader's accuracy rate did not change significantly with the percentage change rate ranging between -1.7 to 0.9 (P = 0.12 to 0.56). Patients considered colonoscopy more satisfactory than CTC (30% vs 4%, P < 0.0001), they felt less anxiety during colonoscopy (36% vs 7%, P < 0.0001), they experienced less pain or discomfort during colonoscopy compared to CTC (69% vs 4%, P < 0.0001) and colonoscopy was preferred by 77% of the participants as a repeat screening test for the future.


CONCLUSION
No statistically significant learning curve was identified in CTC interpretation suggesting that further study is required to identify the necessary training to adequately interpret CTC scans.

tion of colonic neoplasia utilizing computed tomography colonography (CTC) as a screening tool. The optimal number of scans required to achieve satisfactory performance in CTC interpretation was not identified suggesting that larger scale studies are required to identify the training requirements necessary for adequate CTC interpretation. Patient preferences for CTC and colonoscopy were assessed by means of a self completion survey and patients were found to prefer colonoscopy to CTC as a screening tool, likely because of the use of conscious sedation.

INTRODUCTION
Colorectal cancer (CRC) is the third most common cause of cancer in the United states [1] and Canada [2] . Colorectal cancer is preventable through screening and removal of the precursor lesions; adenomatous polyps [3] . Despite its risk, inconvenience and cost, colonoscopy remains the optimal screening test for colorectal adenomas and cancer. Adherence to colonoscopy screening programs is low [4] and computed tomography colonography (CTC) has been proposed as an alternative to colonoscopy in part, because it may increase participation in colorectal cancer screening programs.
CTC every five years was adopted as a screening strategy in 2008 by the multi-society task force in their colorectal cancer screening guidelines [5] but was not included in other guidelines [6,7] . CTC has a similar sensitivity and accuracy for the detection of colorectal cancer and adenomas greater than 6 mm in diameter when interpreted by trained and experienced readers [8] . Furthermore, following training, CTC has been interpreted by non-radiologist and inexperienced readers with similar sensitivity to that of experienced radiologists [9,10] . However, despite adoption of CTC by many radiology departments, the type and duration of training and ongoing experience needed to accurately detect colorectal neoplasia by CTC has yet to be determined [11,12] .
This study sought to assess the performance for several readers of differing levels of expertise in CTC interpretation for the detection of colorectal neoplasia. As well, we wished to evaluate the learning curve for each reader in an attempt to delineate the number of scans required to reach a plateau in the accuracy rate of CTC interpretation. Finally, through a patient satisfaction survey, we sought to compare patient preferences and satisfaction for CTC and Colonoscopy.

MATERIALS AND METHODS
A single center, non-randomized, evaluator blinded study of 90 patients at average or high risk of developing colorectal neoplasia referred for colonoscopy. The study was approved by the institutional review board of the University of British Columbia and Providence Health Care and was registered at clinicaltrials.gov (NCT01181739).

Study group
Patients 19 years of age or older referred to a gastroenterologist for colonoscopy for the detection of colorectal neoplasia were invited to participate. Indications included: asymptomatic patients over the age of 50 years, a prior history of colorectal neoplasia, family history of CRC, positive screening test for CRC (fecal occult blood tests, or flexibile sigmoidoscopy), and symptoms of CRC. Exclusion criteria included: patients with a history of colonic biopsy or polypectomy in the previous year, diverticulitis in the previous 8 wk, colonic obstruction, colonic ischemia, inflammatory bowel disease and current pregnancy. Each patient provided written informed consent.

Design
Participants were given the CTC specific bowel preparation instructions used at our site as outlined. They were also given instructions regarding the intake of oral barium and gastrograffin which were used as stool tagging agents. The same bowel preparation was used for both CTC and colonoscopy. Patients were allowed sips of clear fluids and their usual medications on the morning of the procedure but were allowed nothing by mouth thereafter until completion of both procedures.

CTC
Preparation for CTC included bowel purging, stool and fluid tagging as outlined. Colonic distention was achieved with the use of an automated low-pressure delivery of carbon dioxide using the EXEM Protocol Colon insufflator. Buscopan 10 mg iv was used unless there were contraindications, however no sedation was used or offered for CTC.
Images were obtained in both the prone and supine position using the GE VCT LightSpeed Scanner. Images were acquired with the following specifications: collimation 40 mm; pitch 1.375:1; matrix 512 × 512; field of view to fit the patient; MA Noise index 35; tube rotation of 0.5 s and peak voltage 120 kV. Images were reconstructed using a standard algorithm with thicknesses of 1.25 mm. The quality and adequacy of images were assessed before the patients left the radiology department to await colonoscopy. Where necessary, right and left decubitus images were obtained.

Interpretation CTC reader training
The CTC scans were initially read by a radiologist experienced in gastrointestinal imaging and CTC interpretation.
Upon completion of the CT scans and prior to colonoscopy, the gastroenterology (GI) radiologist interpreted the images using the 2D images first and the 3D images for confirmation and recorded the data on a collection sheet. The size of any polyps identified was measured using the electronic measuring tool from the optimum view and the location was recorded according to one of 4 bowel segments: ascending colon, transverse colon, descending colon or sigmoid and rectum. Next extra-colonic findings were recorded as well as the total time required to review the images. The data was recorded on the data collection sheet according to the CRADS reporting system as proposed elsewhere [13] . The data collection sheet was sealed and given to a research assistant who then attended the colonoscopy.
The scans were later read by two first year GI fellows and a third year Radiology resident all of whom initially had no prior experience with CTC interpretation. However, one of the GI fellows had attended an American Gastroenterology Association introductory course in CTC interpretation. This course provided an overview of CTC in general, but no direct teaching of CTC interpretation. Prior to reading the patient CT scans, these three readers completed 30 training CTC datasets provided by GE Imaging Advantage 4.0 colon vcar software. These scans are self-learning modules, which teach a specific feature of CTC interpretation. The GI fellows and the Radiology resident read the CTC images at a later time, after the completion of both procedures and the findings were recorded in the same manner using the same data collection sheet used by the GI Radiologist. The colonoscopy findings were reviewed after each set of CTC images were interpreted and therefore, these readers were able to verify the accuracy of their CTC interpretation.

Colonoscopy
Within two hours of completion of the CTC imaging, participants underwent colonoscopy with conscious sedation using a combination of midazolam and merperidine or fentanyl. Colonoscopy was performed by an experienced gastroenterologist using a video colonoscope (PCF or HCF 180 series, Olympus, NY, United States). The endoscopist was blinded to the results of the CTC and performed colonoscopy in the standard fashion until the cecum was reached. During withdrawal, once the colonoscopist declared the examination of each segment complete, the results of the CTC findings of that segment were made known to the colonoscopist who would re-examine that segment if necessary. This technique of "segmental unblinding" has been described previously [14] . Polyps detected at colonoscopy were removed in the usual fashion with snare polypectomy, with or without cautery, or biopsy forceps and submitted for histopathological examination. Lesion size was estimated by comparison to the biopsy forceps or snare prior to removal. The findings at colonoscopy were recorded in a similar fashion to the CTC findings with the use of a graphic data collection sheet.

Patient questionnaires
Following CTC and prior to colonoscopy, all patients were given an 11 question written survey for selfcompletion. Another 13 question survey was given after recovery from the sedation in the endoscopy suite, immediately prior to the patient's discharge from the hospital. Both surveys were adapted from those published previously that were used to assess patient satisfaction with endoscopy [15] .
In order to assess the test/retest reliability of the survey, a random sample of 25% of the patients were mailed a second survey for completion, 3-4 wk after the day of the procedures.

Lesion matching
Tissue samples removed at colonoscopy were sent for examination by an experienced pathologist at St. Paul's Hospital. Lesion size was estimated at colonoscopy and the lesions found at CTC and colonoscopy were matched based on the colonoscopy and pathology reports. Lesions within one colonic segment were considered a match [16] . Matching was performed by the first author and the research assistant who was present at the endoscopy. Discrepancies were resolved through a consensus opinion.

Statistical analysis
Mean and standard deviation were calculated for continuous variables where appropriate. Otherwise, median and interquartile ranges were provided. Categorical variables were summarized with count and percentage.
The colonoscopy results after segmental unblinding and the pathological examination of removed tissue specimens was considered the "gold standard" for determining size, location and histologic type. Lesions < 5 mm in diameter were not included in the analysis. Lesions were grouped according to size as 6-9 mm or > 9 mm. Reader performance was analyzed separately for the two lesion groups, except for the analysis of a learning curve in accuracy rate which was evaluated only for 6-9 mm lesion group.
Sensitivity, false positive rate and accuracy rate were calculated on per-patient basis. Each patient was classified as "polyps present" or "no polyps present" based on the reference standard. The sensitivity for each read was calculated as the proportion of true positive findings, the false positive rate was the proportion of false positive findings and the accuracy rate was the proportion of correct findings. The four readers were compared pair-wisely in performance measures with the use of McNemar's test. The Joinpoint regression model was used to assess the trends and determine statistically significant trend changes in reader's performance during the study period, including reading time and accuracy rate. Statistically significant trend changes are detected using the Monte Carlo permutation method [17] . We defined a reading sequence with 90 time points to represent the order of the 90 scans accordingly. The logarithm of reading time was modeled against the reading sequence using joinpoint re-39 polyps 6-9 mm were found in 20 patients, and 13 polyps > 9 mm were found in 10 patients. Some of the patients had multiple polyps and in total, 70 patients did not have any polyps. The sensitivity, false positive and accuracy rates for each of the readers on a per-patient basis are shown in Table 1. On a per polyp basis, the sensitivity rates for polyps 6-9 mm and > 9 mm were 59% and 53.8% for the Radiologist and the Radiology resident, 66.7% and 84.6% for GI fellow 1, 66.7% and 69.2% for GI fellow 2 and 97.4% and 100% for colonoscopy, respectively (first pass, before unblinding and histopathology confirmation of adenomatous polyps). Pairwise comparisons between each of the readers showed a significant difference only for polyps > 9 mm, where GI fellow 1 detected significantly more polyps than the radiologist (P = 0.05) and the radiology resident (P = 0.05) but not GI fellow 2 (P = 0.16).

False positives
There were 70 patients with no polyp detected based on the reference standard. The false positive rate for the detection of polyps between 6-9 mm was 10% for the radiologist and the radiology resident, 11.4% for GI fellow 1 and 10% for GI fellow 2. For polyps > 9 mm, the false positive rates were 3.8% for the Radiologist, 0% for the Radiology resident, 3.8% for the GI fellow 1 and 2.5% for GI fellow 2. The false positive rate for colonoscopy was 2.86% (6-9 mm) and 1.25% (> 9 mm). Pairwise comparisons did not reveal any significant differences between the readers for either size of polyp.

Learning curve
Due to the small number of polyps > 9 mm, the learning curve was assessed based on detection of polyps ≥ 6 mm in size. Using the polyp as the experimental unit, the sensitivity rate for each reader was compared for the gression under the assumption of a normal distribution. The trend was described by percentage change rate, the same as annual the percentage change described previously [17] where a negative value indicates a decrease in reading time and a positive value suggests an increasing reading time. For accuracy rate, we redefined the reading sequence using 9 time points where each point represents a reading set of 10 patients. The accuracy rate for each reading set was calculated for each reader and fitted to the joinpoint regression model under the assumption of a normal (poisson) distribution.
Patient satisfaction results from the surveys of CTC and colonoscopy were summarized in a contingence table for each question. Proportions of patients with a difference in experience between the two approaches were calculated and compared using McNemar's test. The test, re-test reliability of the survey was assessed using the percent agreement between the replies for the first and second survey for the patients who completed the follow up survey. Following the guideline proposed by Fleiss [18] , the Kappa coefficient was calculated and classified as "excellent" (≥ 0.75), "good" (74-40) or "poor" (< 40).

RESULTS
A total of 90 patients of whom 57% were male, aged 55 ± 8.9 years were enrolled in the study. All of the patients initially enrolled in the trial were able to complete the trial with the exception of one patient in whom the colonoscopy was delayed, however, it was eventually performed. The majority (74%) of the patients were of average risk for colorectal cancer and were referred for screening purposes. The remainder were referred for bleeding (12%), a change in bowel habit (9%) and abdominal pain (4%). All of the patients completed both procedures on the same day with the colonoscopy occurring within two hours of the CTC, except for one patient. This patient, despite being asymptomatic, had extra luminal air following the CTC, and colonoscopy was performed 37 d later without complication. There were no other complications encountered during this trial.

Sensitivity
Using the gold standard of combined colonoscopy results after segmental unblinding and pathology results,

Reading time
The median time required to read one scan was 10. Using joinpoint regression, the reading time stayed stable for the radiologist with a percentage change rate of -0.1 (-0.4, 0.1), P = 0.36. However, a significant reduction in reading time was observed in the three readers with less experience ( Figure 1B). For the radiology resident and the two GI fellows, the percentage change rates were -0.8 (-1.3, -0.4), -1.1 (-1.6, -0.6) and -0.6 (-0.9, -0.4) respectively.

Extra-colonic findings
The radiologist identified 30 extra-colonic findings the majority of which were cystic appearing lesion of the kidneys or liver and 8 required additional investigations. The radiology resident identified 66 findings while the first GI fellow identified 8 and the second GI fellow identified 1. No further analysis was performed as the GI fellows were not trained in abdominal CT interpretation which likely explains the variability in findings between the readers. Furthermore, the radiology resident appeared more likely to "over call" extracolonic findings and thus found a substantially larger number than the radiologist. The majority of the extra colonic findings were not clinically significant and the differences in the detection of extra-colonic findings was not an endpoint of this study and therefore, this data was not evaluated further.
The average sedation used at colonoscopy was 4 mg of intravenous midazolam and the average analgesia was 78.5 mg of intravenous fentanyl (64 patients) or 54.8 mg of intravenous meperidine (26 patients).

Patient satisfaction surveys
All 90 patients enrolled completed the initial surveys and 20 of them also received follow up surveys. All 20 patients (100%) returned the mail-in surveys but not all of the questions were answered (≥ 85%). The agreement rate for each question ranged between 0.35 and 0.89 for CTC and between 0.56 and 1.00 for Colonoscopy. The Kappa Coefficients were "good" for most of the questions.
Although most participants felt their pain (or discomfort) was adequately controlled during both procedures, 69% experienced less pain or discomfort during colonoscopy compared to CTC and 4% felt less discomfort with CTC (P < 0.0001). Additionally, 36% of the participants felt less anxious during colonoscopy compared to CTC and 7% of subjects felt less anxious with CTC (P < 0.0001). Twenty-six percent of the participants preferred the physical environment of the endoscopy suite, whereas, 14% preferred the CTC exam room (P = 0.06). Overall, colonoscopy was considered more satisfactory than CTC by 30% of participants and 4% subjects held the opposite opinion (P < 0.0001). Colonoscopy was preferred by 77% of the participants as a repeat screening test for the future, mostly because there was less pain and discomfort during procedure. Of the 23% who preferred CTC, the majority made this decision based on the

A B
shorter time requirement for CTC.

DISCUSSION
There was no learning curve demonstrated for any of the readers in this study. In fact, although not statistically significant, the accuracy rates for three of the readers declined slightly with increasing experience. One of the GI fellows detected two more polyps > 9 mm than the other readers. Beyond this, there was no difference between the readers for accuracy rates or interpretation times. This is in keeping with previous evidence, that with training, non-radiologist readers can learn to interpret CTC scans with a similar degree of accuracy to that of experienced radiologists [9,10,19] . Experience alone, is not sufficient to achieve competence in CTC interpretation [20] and clearly some training is required, however, how much training has yet to be clearly defined. The lack of a learning curve may suggest that the 30 training scans was enough for the readers to reach a plateau in their interpretation skills or, alternatively, that many more scans are necessary to see the effect of training on CTC interpretation. Exposure to an introduction to CTC interpretation conference by one of the GI fellows also did not make a difference to the learning curve. Similarly, the order of review of 2D or 3D images did not appear to impact the learning curve. The accuracy rates and learning curves do vary widely amongst readers [12] . Guidelines from The American College of Radiology have recommended at least 50 cases are required for training [21] while the American Gastroenterology Association has recommended at least 75 cases in their guidelines [22] showing that there is still no consensus regarding the amount of training necessary for adequate training for CTC interpretation [8] . These findings suggest that more study is clearly required to determine the specifics regarding learning curves in the appropriate interpretation of CTC. The sensitivity rates ranging from 54% to 84% for the detection of polyps > 6 mm were lower in our study than seen previously in some studies [14] but similar to those seen in others [23][24][25] . This is consistent with prior meta analyses which have shown that there is a wide variability in the sensitivity of CTC for detecting polyps [26,27] . This variability in detection rates is one of the potential problems with CTC as a screening test [12] . On the other hand, the accuracy rates of our study were similar to those reported elsewhere [24] . GI fellow 1 detected 2 more polyps > 9 mm than the other readers, however, this reader also had a slightly higher false positive rate and subsequently, a slightly lower overall accuracy rate. This would not be unexpected in a novice reader as it could be anticipated in an effort not to miss any significant pathology that a reader might "overcall" a polyp on CTC. The radiology resident had the greatest decrease in reading time over the course of the study while the experienced radiologist had little change in the reading time over the course of the study. This is not surprising given the familiarity with interpreting CT scans on the part of the radiologist, while the novice readers would be expected to show decreases in reading time as they became more familiar and proficient in the necessary steps to interpret the scans. The accuracy rates for the three novice readers also declined slightly over time and it may be that the decrease in reading time comes at a cost of a decrease in accuracy. That is to say, that as the reading time goes down so does the accuracy. This seems intuitive and would be analogous to the finding that adenoma detection rates at colonoscopy decrease when the withdrawal time is less than 6 min. Further research comparing polyp detection rates with the amount of time spent interpret CTC scans may help to better define any potential association.
Patients preferred colonoscopy over CTC in our study based primarily on decreased levels of anxiety and discomfort. This is similar to one previous study [28] but contrary to other previous reports where patients preferred CTC [4,23,[29][30][31] . The preference for colonoscopy in our study is most likely based on the sedation received during colonoscopy. Patient preference is important for the adherence to any screening program. The uptake of all screening for CRC has not been as high as anticipated and this in part, may be related to a negative patient perception of both colonoscopy and CTC.
The main limitation of this single centre trial was the small sample size of 90 patients. We encountered difficulty enrolling patients for tandem procedures and it was difficult to coordinate the two exams on the same day. The one patient who was found to have a small amount of extra luminal air remained asymptomatic and at no point did she develop any clinical sign of perforation. The etiology of the air remained unclear but it was likely related in some way to the insufflation of the carbon dioxide. The extra colonic findings seen by the radiologist was not seen by either of the GI fellows (novice readers) and this highlights one proposed benefits of CTC, being the ability to pick up incidental pathology on the CT scan of the abdomen. This point is somewhat controversial as the extracolonic findings could also lead to additional unnecessary investigations. The inexperienced readers were clearly not able to adequately interpret the scans for the presence of extracolonic findings. Some bias may have been introduced by the order of examinations with CTC always being performed first. As a result, patients did not realize the full benefit of the shorter examination and lack of recovery time for CTC when performed alone.
In this study, four readers with differing levels of experience had similar sensitivity and accuracy rates for the detection of adenomatous polyps in patients undergoing colonoscopy for the detection of colorectal neoplasia. There was no learning curve for CTC interpretation for any of the readers and patients demonstrated a preference for colonoscopy over CTC. Larger, multi-center studies are needed to better define the training requirements and performance standards for CTC credentialing, particularly given the variability in accuracy with different readers.

Background
Computed tomography colonography (CTC) has previously been recommended as a screening test for adenomatous neoplasia in the colon as a precursor to colorectal cancer. While several previous articles have attempted to address the sensitivity of CTC for the detection of colonic neoplasia, few have looked at the necessary training require to adequately interpret CTC.

Research frontiers
As well as looking at a possible learning curve in the interpretation of CTC, this paper also evaluated patient preferences for CTC and colonoscopy.

Innovations and breakthroughs
After 30 training cases and 90 study cases, no learning curve was identified in the interpretation of CTC. Patients experienced less pain or discomfort and less anxiety with colonoscopy over CTC. The majority of patients preferred colonoscopy over CTC and would choose colonoscopy for repeat screening if necessary.

Applications
Interpretation of CTC can be learned by novice readers with time. However, the optimal number of scans required to learn this skill has yet to be determined.

Terminology
CTC is a abdominal computed tomography scan which uses air and contrast to distend the bowel so that the lining of the bowel can be carefully examined. Colonscopy involves the use of a flexible tube with a fiber optic camera to visualize the inside of the colon.