{'fraction_non_alphanumeric': 0.03828365878725591, 'fraction_numerical': 0.024794450154162383, 'mean_word_length': 4.8140403286034354, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Training and validation of a commercial deep learning contouring platform

We present our experience with training and validation of a commercially available deep learning algorithm for organs at risk(OAR) auto-contouring. Computed tomography(CTs) with OARs from a cohort of 213 head and neck(H&N) patients were used for training the deep learning model. A separate cohort of 85 CTs and structure sets was used for validation. All OARs (13) were contoured by a single physician. Metrics such as the DICE similarity coefficient (DSC), Jaccard similarity coefficient (JSC), and volumetric difference (VD) were used to analyze contouring variation. Mean DSC and JSC values ranged 0.48-0.89 and 0.32-0.8, respectively, depending on OAR. A DSC value ≥0.7 indicated low inter-observer variability. In our study, all but one of the contours were above this threshold. DSC for the middle pharyngeal constrictor had the lowest value of all the contours. This may be due to the small volume of this structure. Qualitative assessment of auto-segmented structure samples confirmed the reliability of DSC by demonstrating the compatibility between the expert’s evaluation and DSC values. Overall, we found that deep learning auto contouring is a useful tool to speed up the process of contouring in radiotherapy treatment planning.


Introduction
Contouring of organs at risk (OARs) is a time-consuming part of radiotherapy treatment planning. There are many approaches to speed up this process. These range from atlas-based segmentation (ABS) algorithms, active contour model (ACM), machine learning, and others. ABS methods are often used and some software packages are commercially available; however, they are generally limited to certain OARs and local elastic registration is required for accurate results, which is time-consuming. [1][2] ACM delineates the outline of objects using energy constraint and image forces. Being a deformable model, ACM is frequently adopted in medical image processing; however, it is computationally intensive especially when the image size is large and accuracy demands aconvergence criteria. [3] Machine learning techniques, deep learning in particular, have been rapidly growing in the last few years in many industries and are being adopted in radiotherapy for many tasks such as clinical outcome prediction, medical image analysis, dose-response modeling and image segmentation. [4][5][6][7] The main advantage of this technique is its ability to learn the most suitable representation of data for given tasks. In this paper, we present our experience with training and validation of a commercial deep learning automatic contouring software for H&N OARs. Patients with tilted head position were excluded. Thirteen most common and important OARs in H&N radiation treatment planning were considered in this study ( Table 1). All OARs were contoured by a single physician and were used to generate the radiation treatment plans with which the patients were treated. In order to quantitatively evaluate the contours generated from the model against the expert's contours, the following metrics were used; DICE similarity coefficient (DSC),

Quantitative assessment
where and are the volume of organs of ground truth and auto-segmented contour.
A commercial software (StructSure, Standard Imaging Inc.) was also used to validate the contouring variation. The program gives score (0-100) based on the similarity of two contours by penalizing each missing and extra voxel as a function of distance from the primary contours with forgiveness threshold distances;

= + +
where A is a constant penalty, B is a linear penalty according to the distance, C is a base penalty for the exponential function, D is an exponential penalty rate, and d is distance. [8]

Qualitative assessment
A sample of 20 CTs was selected from the validation CT set. 256 automatically segmented structures were qualitatively evaluated by the single physician on a scale of 0-5; 0: ideal, exactly what the ground truth would be, 1: acceptable, no edits necessary, 2: too big, needs minor edits, 3: too small, needs minor edits, 4: not close enough to the ground truth, too big, 5: unacceptable (0-3: clinically useful, 4-5: not clinically useful) ( Table 2).

Results
Mean DSC values ranged from 0.48±0.14 for the middle pharyngeal constrictor to 0.89±0.03 for the cerebellum. Similarly, the JSC values ranged from 0.32±0.12 for the middle pharyngeal constrictor to 0.80±0.05 for the cerebellum. VD values ranged from 0.02±0.29 for the right submandibular gland to 17.17±59.40 for the middle pharyngeal constrictor (Table 1). For 20 CTs with 13 OARs, a total of 256 automatically generated contours were evaluated by a physician to determine if they were clinically applicable ( Table 2). None of the structures were evaluated to be ideal or acceptable without some edits, but 98% of contours, 251 out of 256, were still considered clinically useful with minor modifications. Among the clinically useful cases, 242 structures were bigger than the ground truth and the other 9 contours were smaller. Meanwhile, five automatically generated structures, 2% of all, were evaluated to be clinically unacceptable; one got a score of 4, much bigger than ground truth (Figure 1). The contour graded as 4 was the brainstem (Table 3). When compared to the ground truth contour slice by slice, overall contour lines were posteriorly expanded and shifted with 2 and 4 extra slices at the top and bottom. It had a DSC value of 0.69, StructSure score of 0.00, and VD value of 0.80.  Three of the lowest-rated structures were in the oral cavity. The oral cavity is an OAR with the largest volume in H&N cases with complex structures, thereby carries a larger element of risk of variation in contour. One of the three oral cavity contours had irregular shapes, with large missing (67.66cc) and extra (83.59cc) volume. VD was small, 0.07, but less important in this case as the VD formula only considers the quantitative difference between the volumes, not the spatial concordance. On the other hand, the StructSure score, which gives a penalty on the distance between each voxel, was 0.00. The other two had DSC>0.7 (0.81 and 0.80), but the StructSure scores were 32.03 and 49.96 (median .05) as one was missing 11 slices of contours (VD=-0.22) and the other had irregular shapes. The other lowest-rated contour was a left submandibular gland. In this case, the structure was adjacent to the target and DLC generated contour had 1 and 11 extra contoured slices at the top and bottom. Therefore Vauto was 5.7 times larger than the Vgt; VD=4.67. DSC value and StructSure score were 0.24 and 0.00, respectively, which showed that the two contours were substantially different. In most of the structures, except for the pharyngeal constrictors, DSC values agreed well with the score given by the physician. Only two out of 197 contours were evaluated as clinically useful when DSC values were below 0.7 (0.68 and 0.62). However, DSC value ≤0.4 was generally considered as a large variation, therefore those two contours also required expert evaluation. Thus, qualitative assessment substantiated the reliability of DSC by demonstrating the compatibility between the expert's evaluation and DSC values.

Conclusions
A DSC value ≥0.7 indicated good concurrence between automated segmentation and expert contours.
[9] DSC for middle pharyngeal constrictor had the lowest value of all the contours. This may be because of the small volume of this structure. In the majority of cases, DSC showed good agreement with the expert's judgment of the quality of the auto-segmented structures. In some cases of discordance, other metrics substantiated the qualitative evaluation as different metrics take different (a) (b)