{'fraction_non_alphanumeric': 0.027901643521467703, 'fraction_numerical': 0.012421964581475348, 'mean_word_length': 5.2871445734881855, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Deep Quality Estimation: Creating Surrogate Models for Human Quality Ratings

Human ratings are abstract representations of segmentation quality. To approximate human quality ratings on scarce expert data, we train surrogate quality estimation models. We evaluate on a complex multi-class segmentation problem, specifically glioma segmentation, following the BraTS annotation protocol. The training data features quality ratings from 15 expert neuroradiologists on a scale ranging from 1 to 6 stars for various computer-generated and manual 3D annotations. Even though the networks operate on 2D images and with scarce training data, we can approximate segmentation quality within a margin of error comparable to human intra-rater reliability. Segmentation quality prediction has broad applications. While an understanding of segmentation quality is imperative for successful clinical translation of automatic segmentation quality algorithms, it can play an essential role in training new segmentation models. Due to the split-second inference times, it can be directly applied within a loss function or as a fully-automatic dataset curation mechanism in a federated learning setting.


Introduction
Large and long-standing community challenges, such as BraTS [2], have created a multitude of fully-automatic segmentation algorithms over the years. To fully exploit the potential of these task-specific algorithms, be it for clinical or scientific purposes, it is essential to understand the quality of their predictions and account for segmentation failures. 13 As individual segmentation quality metrics are only able to cover isolated aspects of segmentation quality, most segmentation challenges evaluate on a combination of metrics [15]. Human expert quality ratings have become a prominent tool to complement conventional analysis, among others [14,17,20] for segmentation outputs [13]. In contrast, to narrowly defined quality metrics, these more holistic measures capture various quality aspects. However, they are prohibitively expensive regarding data acquisition times and financials to be deployed regularly. Related work: Therefore, previous research tried to approximate segmentation performance by other means. Reverse classification accuracy (RCA) has been proposed to evaluate segmentation accuracy for applications of multi-organ segmentation in magnetic resonance imaging (MRI) [21] and cardiac MRI segmentation [19]. The authors used the maximum predicted segmentation quality metric estimated by a multi-atlas-based registration method as a proxy to estimate the true quality metric. However, the application of RCA is so far limited to organ segmentation. Further, the analysis is restricted to established segmentation quality metrics such as DSC and Hausdorff distance. It is unclear how it would generalize to expert scoring and to lesion segmentation.
A method to estimate the Dice score of an ensemble of convolutional neural networks (CNNs) for segmentation has also been proposed [7]. They propose to train a linear classifier regression model to predict the Dice score for every CNN in the ensemble. Here, the Dice scores for every pair of segmentation predictions of the models in the ensemble serve as input for the regressor. Further, Audelan et al. proposed an unsupervised learning method for predicting DSC using Bayesian learning [1].
The above methods are coupled with segmentation algorithms for the estimation of segmentation quality metrics. Unlike this, Fournel et al. predict the segmentation quality metric directly from the input image and segmentation map using a CNN [5]. Similarly, it is possible to predict an ensemble's segmentation performance from discord between individual segmentation maps, even when ignoring the image data [12].
The BraTS challenge [16] features the multi-class segmentation problem of glioma segmentation. Distinguishing between enhancing tumor, necrosis, and edema is a complex subtask scattered over multiple imaging modalities. It is evaluated using Sørensen-Dice coefficient (DSC) and Hausdorff distance (HD) for the whole tumor, tumor core and enhancing tumor channels. Previous research revealed that BraTS tumor segmentation algorithms typically perform well when monitoring established segmentation quality ratings such as DSC, HD, and others. However, when they fail, they fail spectacularly [4,2,11,12,16]. These findings reflecting multiple established segmentation quality metrics are supported by surveys with expert neuroradiologists [13]. Here, the multi-faceted concept of segmentation quality was condensed to a single expert quality rating. To this end, BraTS glioma segmentation is a good candidate for studying how a holistic rating can complement or replace them. Contribution: In contrast to the above-mentioned methods, which predict narrowly defined quality metrics such as DSC or HD, we focus on the approximation of more holistic expert neuroradiologists' ratings. We build surrogate regression models for these abstract human quality ratings to estimate the segmentation quality of MICCAI BraTS segmentation algorithms. A sophisticated augmentation pipeline compensates for the scarce 2D training data available. Despite these obstacles, our model manages to create robust estimates for 3D segmentation quality on an internal and external test set. While our models are agnostic to the segmentation method and are even compatible with 2D manual segmentations, split-second inference times enable broad downstream applications in scientific and clinical practice.

Methods: Network Training
We train multiple regression networks to approximate the human quality ratings. Segmentation quality rating: We use human quality ratings provided by Kofler et al. [13]. In this study, expert radiologists rated the quality of glioma segmentations in two experiments. In the first experiment, 15 neuroradiologists rated the segmentations' center of mass for 25 exams with four different segmentations from axial, sagittal, and coronal views, resulting in 300 trials. The experiment featured one manual and three computer-generated segmentations. In the second experiment, three neuroradiologists rated another 50 exams with one manual and five computer-generated segmentations only on axial views, again resulting in 300 trials. The rating scale ranges from 1 star for very bad to 6 stars for very good segmentations. Network input and output: To predict the above-mentioned quality rating, the network receives the four MR modalities, namely T1, T1c, T2, and FLAIR. The tumor segmentations are either supplied in a single label channel encoding the different tumor tissues or with three label channels following BraTS annota-tion concepts, as illustrated by Figure 1. We try this style of encoding the tumor segmentations as this approach has been proven successful for training BraTS segmentation algorithms [2]. Training constants: The dataset is randomly split into 80 percent (60) of the 75 exams for training and 20 percent (15) for testing. Batch size is kept constant at 80 and learning rate at 1e-3. To compensate for the scarce training data, we employ a heavy augmentation pipeline featuring Gaussian noise, flips, and random affine plus elastic transformations. Additionally, we use batchgenerators [9] to augment with contrast, brightness, gamma, low resolution, and rician noise. Further, we simulate MR artifacts with TorchIO [18], specifically motion, ghosting, spikes, and bias fields. We employ a Mean Square Error (MSE) loss for all training runs to especially penalize far-off predictions. The human quality ratings serve as reference annotations for the loss computations. Due to the scarcity of training data, we do not conduct model selection and use the last checkpoint after 500 epochs of training across all training runs. Training variations: Between training runs we experiment with three optimizers, namely Ranger21 [22], AdamW, and SGD with a momentum of 0.95. We use DenseNet121 and DenseNet201 [8] to investigate whether the performance prof-its from more trainable parameters. Moreover, we try a channel-wise min/max, and a nn-Unet [9] inspired percentile-based normalization using the 0.5 and 99.5 percentiles for minimum and maximum, respectively. Hardware: All computations take place on a small workstation with an 8-core Intel(R) Xeon(R) W-2123 CPU @ 3.60GHz with 256GB RAM and a NVIDIA Quadro P5000 GPU. Computation times: A training run takes approximately two hours. The above machine infers four 3D exams per second, including mass computation. Without the extraction of 2D slices, the inference performance increases to 25 per second. Note that the implementation is not fully-optimized for computation time, as the split-second inference times are in no way impeding our purposes. Memory consumption: With the batch size of 80 we use most of the 16gb CUDA memory of the NVIDIA Quadro P5000 GPU.

Evaluation Experiments
We conduct two experiments to evaluate the performance of our models. In the first experiment we identify and evaluate our best performing model. In the second experiment we validate its generalization capabilities on an external test set. In both experiments, we generate network predictions for the axial, coronal, and sagittal views and compute a mean rating for each exam.

Internal evaluation experiment
To investigate which of the six hyperparameter combinations works best by evaluating on an internal test set. Dataset: We use the previously held-back 20 percent of the training data for evaluation, as described in Section 2. Procedure: We run inference on the test set for each model. Following, we evaluate using established regression metrics, namely mean absolute error (MAE), root mean square error (RMSE). We further compute Pearson r (r) to measure the linear correlation between network predictions and ground truth labels. Results: Table 1 illustrates the performance differences between training runs. We observe that the simple DenseNet121 trained with percentile-based normalization, Ranger21 optimizer, and BraTS label encoding approximates the human rating best. Figure 2 visualizes the model's predictions compared to the averaged human star ratings. According to the scatter plot, illustrated in Figure 2, the model performs more accurately for better segmentations. This is also reflected by a Bland-Altman plot, see Figure 3. It is important to note that the variance in human quality assessment also increases for lower quality segmentations, see   3. Bland-Altman plot: Network predictions vs. human star rating. The model reveals higher prediction accuracy for better-quality segmentations. This is not surprising given that human raters also display higher agreement for such cases and that these are better represented in the training data, compare

External evaluation experiment
To better understand our model's generalizability, we further evaluate an external dataset. Data set: The dataset features manual annotations for 68 exams generated by two expert radiologists in consensus voting. It includes 15 high-grade glioma (GBM) and 13 low-grade glioma (LGG) from University Hospital rechts der Isar. Furthermore, 25 GBM and 15 LGG from the publicly available Rembrandt dataset [6] are added to the analysis. We obtain five segmentations from BraTS algorithms and four fusions from BraTS Toolkit [11]. This way, we have a total of 612 segmentations to evaluate. Procedure: We select the best model obtained from the first experiment. We feed 2D views of the 3D augmentations' center of mass to the network to obtain an axial, sagittal, and coronal quality rating. As there are no human quality ratings for this dataset, we measure segmentation performance using established quality metrics, namely Sørensen-Dice coefficient (DSC) and surface Dice coefficient (SDSC). Results: We find a strong correlation between the quality ratings predicted by the network and DSC, see Figure 5. We observe a Pearson r of 0.75 for the axial, 0.76 for the coronal, and 0.77 for the sagittal view, while the averaged rating across views has a 0.79 correlation. This is supported by a high correlation between the mean rating and SDSC (Pearson r: 0.85), suggesting that the model generalizes well to the external data set. Fig. 4. Network predictions vs. minimum, mean, and maximum human rating. For lower quality segmentations, human raters disagree more. In most cases, the network's predictions are in the range or close to the human ratings. As already visible in Figure 3, the network tends to overestimate the quality of bad segmentations. However, they are still assigned systematically lower scores. In practice, this difference is sufficient to distinguish between good and bad segmentations by employing a simple thresholding operation.

Discussion
We demonstrate that a simple DenseNet121 is able to serve as a surrogate model for the abstract human segmentation quality rating under a scarce training data regime. Notably, the mean absolute error deviation is lower than the difference Kofler et al. [13] reported for individual human raters from the mean human rating. Apparently, the 3D segmentation quality is sufficiently encoded in the 2D center of mass slices. Our experiments show that the choice of hyperparameters is not critical as all networks reach solid performance.
Expert radiologists are among the highest-paid doctors and are notoriously hard to come by. Given that the inference of the approximated quality rating only takes split seconds, there are broad potential applications for deep quality estimation (DQE) networks: One obvious application for DQE is quality monitoring during inference. Even though fully-automatic glioma segmentation algorithms, on average, tend to outperform human annotators [13], they sometimes fail spectacularly. For successful clinical translation of such algorithms, detection and mitigation of failure cases is imperative.
Another potential use case for DQE is dat aset curation. Data set curation is an important aspect of model training, as broken ground truth labels can destroy model performance. As we demonstrate in the evaluation experiments, DQE can differentiate between trustworthy and broken ground truth cases in a fully-automatic fashion. This property becomes especially valuable in a federated learning setting, where researchers have no access to the ground truth labels. DQE allows training models only on trustworthy exams by applying a simple thresholding operation on the estimated quality score. Limitations: It is unclear how well our approach generalizes to other (segmentation) tasks and quality metrics. Taking into account that glioma segmentation is a complex multi-class segmentation problem, the scarce training data, the 3D to 2D translation, and the abstract nature of the human-generated quality judgments, we believe there is a slight reason for optimism.
The proposed model performs better for predicting high-quality segmentations. This is perhaps not surprising given that humans agree more on the quality of such cases and that these are better represented in the training data, as visible in Figures 2 to 4. Nevertheless, in practice, simple thresholding of the predicted star ratings can sufficiently distinguish segmentation qualities. Outlook: As we demonstrated, DQE can approximate non-differentiable quality metrics, such as the abstract human segmentation quality rating, with a differentiable CNN. This promises the possibility of training new (segmentation) networks with surrogates of non-differentiable quality metrics by using DQE within the loss function. 14 Future research should address these open questions.