{'fraction_non_alphanumeric': 0.036931818181818184, 'fraction_numerical': 0.001893939393939394, 'mean_word_length': 5.707936507936508, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
SU‐FF‐I‐87: DTA‐Based Metrics for the Evaluation of Autosegmentation Algorithms in Clinical Radiotherapy Workflow

Purpose: To establish quantitative metrics to assess the performance of autosegmentation programs in the context of clinical radiotherapy workflow. Method and Materials: One of the more time intensive and predominantly un‐automated tasks in radiation therapytreatment planning is the segmentation of the anatomical structures which are relevant for a particular intended radiotherapytreatment. We present performance metrics based on the distance‐to‐agreement (DTA) between manually‐contoured reference sets and the results of a generic autosegmentation platform. Results: To correlate to the time expended to correct erroneous auto‐contouring results we posit the slice‐by‐slice DTA = min { | x r,i − x t,i | } ∀ { x t,i } , where{x r,i} and {x t,i} are the vertices on the reference and test sets, respectively, on the ith axial CT slice. The histogram of DTAs (or simple pass/fail DTA thresholds) will assess the amount of manual correction required to bring an auto‐contoured structure into acceptable agreement with the reference set. Beyond this application, full automation of the contouring process (i.e. no manual correction) would require stricter, 3‐dimensional, and site‐specific DTA metrics, for which we propose generalized quality factors. As a test bed for these metrics, we have compared manually‐generated clinical contours to those created by CMS' Atlas‐Based Autosegmentation system and shown that, for some sites, the autosegmentation is nearly indistinguishable from the variability in our clinic's manual contouring. Conclusion: Depending on the desired use of an autosegmentation package, either slice‐by‐slice or 3D DTA metrics facilitate the assessment of clinical performance. Regardless of intended application, the DTA‐based metrics may be used as (a) a quantitative benchmark of an autosegmentation algorithm against the inherent variability in manual contouring and (b) a potential correlate to efficiency gains should the software be brought into full clinical use.