{'fraction_non_alphanumeric': 0.03950997184079891, 'fraction_numerical': 0.015269834818707, 'mean_word_length': 4.097366084640426, 'pattern_counts': {'":': 0, '<': 12, '<?xml version=': 0, '>': 14, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 1}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 12, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
On Image Segmentation With Noisy Labels: Characterization and Volume Properties of the Optimal Solutions to Accuracy and Dice

We study two of the most popular performance metrics in medical image segmentation, Accuracy and Dice, when the target labels are noisy. For both metrics, several statements related to characterization and volume properties of the set of optimal segmentations are proved, and associated experiments are provided. Our main insights are: (i) the volume of the solutions to both metrics may deviate significantly from the expected volume of the target, (ii) the volume of a solution to Accuracy is always less than or equal to the volume of a solution to Dice and (iii) the optimal solutions to both of these metrics coincide when the set of feasible segmentations is constrained to the set of segmentations with the volume equal to the expected volume of the target.


Introduction
One of the most central problems in medical image analysis is to identify the region of an image associated with a certain target structure. This problem, referred to as image segmentation or delineation, is often very time consuming to solve manually. Consequently, there is great interest in the development of methods that can assist in automation of the procedure. Since 2015, it has become increasingly popular to address the segmentation problem using machine learning based methods, and in particular, fully convolutions neural networks with U-net architecture. Such methods commonly dominate the winning submissions to segmentation contests and are backed by a large base of supporting literature [2,7,9,14,20,29,33].
Despite the success, performance of these methods will, like any machine learning method, depend on the quality of the available data [5,38]. Since it is well known that the data commonly used in practice is produced by medical practitioners that delineate structures in an inconsistent manner [26], it is important to understand the impact of label noise. One way to study the influence of label noise is to consider how the noise impacts segmentations that are theoretically optimal with respect to the metric used for measuring performance. Even if theoretically optimal solutions may not be attainable in practice, studying them gives important insights into what the effect label noise has on the particular metric and any training method that is designed to maximize it.
Arguably, the most simple and classical choice of metric is Accuracy; the fraction of the image that is correctly delineated. This metric is most commonly targeted by taking a 1/2-threshold of predictions from a model trained with the cross-entropy loss [25]. However, since Accuracy may not reflect the desired behaviour when the data is unbalanced, that is, when the target structure is much smaller than the background, alternative metrics are often preferred. The most popular such alternative is the Sørensen-Dice coefficient, or Dice for short and is related to the F 1 -metric used in binary classification. This metric is most commonly targeted by taking a 1/2-threshold of predictions from a model trained with the soft-Dice loss, a smoothed version of the Dice metric [25]. Other examples considered in the literature include the Jaccard index and variations of the Haussdorff metric [34].
In this work, we conduct a theoretical investigation of the effect label noise has on optimal segmentations with respect to the performance metrics Accuracy and Dice. Because the volume of a proposed segmentation may be used for important properties such as estimating the size of a tumor [10], we pay special attention to the effect noise may have on the volume of the optimal segmentations.
Contributions: A characterization of all optimal solutions to Accuracy and Dice when the target is noisy is provided. This characterization is used to analyze the volume of the optimal solutions and we prove: (i) sharp upper and lower bounds on the volume of optimal segmentations with respect to Accuracy and Dice, (ii) that the volume of the optimal solutions to Accuracy always is less or equal to the volume of the optimal solutions to Dice and (iii) that the optimal solutions to both metrics coincide when the volume is held fix. We also show the relevance of the problem in a practical setting by including experiments on data from the Gold Atlas project [26] and The Lung Image Database Consortium (LIDC) and Image Database Resource Initiative (IDRI) [1].

Related work
Deep learning methods are playing an increasingly vital part in the development of medical image analysis. However, deep learning models require large annotated data sets for successful training that rarely are available in the clinics. Even if some data sets are being curated for training deep learning models, it is generally difficult and expensive to accurately annotate large collections of medical images. Moreover, training data may include corrupted or noisy labels. This is particularly the case in image segmentation where different annotators may have different views on the correct delineation of a region of interest, leading to uncertainty about the true label, see e.g. [4,24]. Noisy labels may also appear due to automated systems or non-expert systems being used to annotate large volumes of data, see [6,28].
There is a large body of literature on the impact of label noise in image segmentation, see [35] and [15] for recent reviews. The proposed solutions to limit the loss of performance when the labels are noisy include label cleaning and pre-processing, e.g. [11], modification of network architechtures, e.g. [37], robustification of loss functions, e.g. [22], reweighting of training data, see [23,40], and many others. These approaches are of practical nature and generally address methodology that improve the performance on some chosen noisy data set. On the contrary, the literature that address the effect of label noise from a theoretical point of view in the context of image segmentation is rather limited. It was shown that the loss function soft-Dice, in contrast to the loss function cross-entropy, does not yield optimal predictions that coincide with the pixel-wise marginals and that the associated volume is biased [2,3]. This motivated methods for post-calibrating uncalibrated marginal estimates [30] and a more general investigation of the relationship between volume and marginal calibration [27]. Finally, an alternative volume preserving segmentation method based on optimal transport theory was proposed and investigated in [21].
Another domain of related work can be found in the binary classification literature. The connection between the study of solutions to Accuracy and Dice in segmentation and the study of Accuracy and F 1 in binary classification was investigated in [25]. Of importance to us are optimal plug-in classifiers to Accuracy and F 1 . That is, classifiers that are obtained by processing the posterior class probabilities or estimates thereof using a threshold. For Accuracy, this relates to the classical Bayes classifier which has been studied since the origin of the field [36]. Early work on the existence of such a classifier for F 1 , and the fact that this threshold is lower than the threshold for Accuracy can be traced to [39]. Further work showed this threshold to be equal to half of the maximal attainable F 1 -score [19]. Lots of extensions of these works have been proposed, but to the best of our knowledge no such extension is in the direction of our work.

Preliminaries
In our work we find that it is convenient to do the theoretical analysis over a continuous domain and where all encountered continuous spaces are equipped with their associated Borel σ-fields. Formally, let Ω = [0, 1] n ⊂ R n be the unit cube of dimension n ≥ 1 and λ be the associated standard normalized Lebesgue measure such that λ(Ω) = 1. The space of segmentations is denoted by S and is formally given by the space of measurable functions from Ω to the binary numbers {0, 1}. For any segmentation s ∈ S, s(w) = 1 implies that the object of interest occupies the site ω ∈ Ω. In a numerical setting, a discretization of the domain Ω is commonly used. For instance, 3D CT scans are often represented by a three dimensional voxel grid of an approximate order of 512 × 512 × 100. In such situations the space of segmentations are given by all possible binary functions defined on this voxel grid. Note that any segmentation on a discretized domain can be incorporated in our continuous framework by using appropriate step functions. Details on this can be found in the end of Section 4.
Beyond the space of segmentations, several other technical constructions are introduced. This includes the space of measurable functions from Ω to [0, 1] which we denote by M and refer to as the marginal functions. We also let f 1 .
= 1 − f , where f is any measurable function defined on Ω. Throughout we adopt the convention that two λ-measurable functions f, g are equal if they are equal λ-a.e. We will use I{·} to denote the identity function, and when F is a cumulative distribution function, we will denote the left limit F (t−) = lim s↑t F (s). Finally, for a given volume v ∈ [0, 1], we let S v . = {s ∈ S, s 1 = v} be the set of segmentations with volume v.
Classically, metrics in medical image segmentation are defined per image as functionals over two deterministic segmentations [34]. When noise is present, the label becomes a random variable and the metrics need to be extended to a functional over one deterministic segmentation and one random label segmentation. In this work, the soft labeling convention for this extension is adopted [12,16,17,18,32]. Also, since we do our analysis with respect to measurable functions on a continuous domain instead of functions on a finite index set, we replace the sums usually used with integrals. This, however, does not change the intuition of the metrics, and the definition using sums can be seen a special case. Definition 1. For any m ∈ M, Accuracy is given by Definition 2. For any m ∈ M, Dice is given by For a noisy segmentation L, that is, a random variable taking values in S, m can be taken to be the exact marginal success probability m(ω) = E[L(ω)], ω ∈ Ω. Such marginal functions are important in theory but can rarely be obtained in practice. Alternative choices of marginal functions include finite sample approximations, that is, point-wise averages over finite observations of L, and estimates of E[L] according to a single annotator [12,16,17,18,32]. These choices of m are important because they are sometimes used for training machine learning models. Finally, note that Because of the fact that m 1 = E[ L 1 ] when m is taken to be the exact marginal success probability, m 1 plays an important role in the medical segmentation context, either theoretically as the expected volume of the target or as an approximation thereof. Understanding how m 1 relates to s 1 , where s is an optimal segmentation to Accuracy or Dice will be central in our work. To the best of our knowledge, this has not been studied in prior work. Figure 1: To the left is Urinary bladder and to the right is Neurovascular bundles for one patient in the Gold Atlas data [26]. Each line is associated with the boundary of a segmentation produced by a particular annotator. The screenshots are taken with RayStation 12A (RaySearch Laboratories AB, Stockholm Sweden).

Main results
The objective of our analysis is to characterize the optimizers to A m and D m and give a detailed description of the volume of the optimal segmentations. That is, for a given m ∈ M, identify properties (e.g. volume) of the optimal segmentation s ∈ S that maximimize Accuracy or Dice. To this end, consider the probability measure on [0, 1] given by the push-forward measure λ •m −1 (·) and let F m denote its cumulative distribution function, The function F m (t) may be interpreted as the volume of the set of sites with non-success probability less than or equal to t. In other words, the volume of the sub-level set ofm at level t.
Since F m is the cumulative distribution function of a probability distribution on [0, 1], it has several well-known properties making it easy to work with, e.g., F m is non-decreasing and right-continuous with F m (1) = 1. Of particular interest to us is that it has a generalized inverse given by which can be interpreted as the minimum level at which the volume of the corresponding sub-level set ofm, is at least v. This function is often referred to as the quantile function and also has several well known properties; it is non-decreasing and left-continuous. Moreover, it allows us to define the following important class of segmentations for a given m ∈ M.
The described class S m,v is informally the set of segmentations with volume v that assigns 1 to sites ω where m(ω) is large.  A lot of our analysis can be simplified if F m is assumed to be invertible almost everywhere. However, this would require m to not have any non-neglible constant regions, which for instance excludes any m that is given by an empirical approximations using a finite number of samples. Consequently, in the sequel, we treat general F m . Our first result contains the essential ingredients for characterizing the optimizers to Accuracy and Dice.
and the elements where the supremum is attained is given by S m,v .
A complete proof is given in Appendix A.2 and outlined as follows. The first part shows that the class S m,v is the class of optimal solutions by showing that for any s * ∈ S m,v and s ∈ S v \ S m,v , The second part proves the equality (6) using an application of the quantile transform. That is, if U has uniform distribution on [0, 1] then F −1 m (U ) has cdf given by F m and Lemma 1 allows us to reduce the constrained optimization problem over the rather complicated space of segmentations, to a one-dimensional integral with respect to the quantile function. It is the starting point for our analysis.
In the remaining section our main theoretical results are presented. In Theorem 1, we provide a characterization of all of the optimal solutions to Accuracy based on volume. In addition, sharp upper and lower bounds on the volume of the associated optimal segmentations are provided.

Theorem 1. For any m ∈ M, the class of maximizers to A m is given by
Moreover, V Am satisfies the following bounds where the bounds are sharp in the sense that there for any The complete proof is given in Appendix A.3 and is outlined as follows. First, the function is introduced and then Lemma 1 is used to show that sup s∈S A m (s) = sup v∈[0,1] a m (v). Consequently, the class of optimal solutions to A m is given by ∪ v∈V Am S m,v , where V Am is the set of optimizers to a m . The rest of the proof consists of detailed analysis of a m and is composed of three parts. The first part is to show (7) by finding one optimal solution and then identifying all volumes that yield the same optimal value. The second part is to provide the lower and upper bounds on the elements of V Am in terms of m 1 given by (8). The third part is to provide examples of situations where the extreme cases occur (9). In Figure 3, a case that is extreme both in the lower and in the upper sense is illustrated. Figure 3: To the left is a particular quantile function F −1 m and to the right is the associated function a m given by (10).
and consequently is an extreme case to (9) in both the lower sense and the upper sense.
In Theorem 2, we provide a characterization of all of the optimal solutions to Dice based on volume. In addition, sharp upper and lower bounds on the volume of the associated optimal segmentations are provided.
Theorem 2. For any m ∈ M, the class of maximizers to D m is given by Moreover, V Dm satisfies the following bounds where the bounds are sharp in the sense that there for any v ∈ (0, 1] exist m 0 , m 1 ∈ M such that m 0 1 = m 1 1 = v and The complete proof is given in Appendix A.4 and is outlined as follows. First, the function is introduced and then Lemma 1 is used to show that sup s∈S D m (s) = sup v∈[0,1] d m (v). Consequently, the class of optimal solutions to D m is given by ∪ v∈V Dm S m,v , where V Dm is the set of optimizers to d m . The remaining proof consists of detailed analysis of d m and composed of three parts. The first part is to show (11) which is derived by careful investigation of the properties of the function , which has the same sign as ∂ v d m (v) and therefore can be used to identify optimal values of d m . The second part is to provide the lower and upper bounds on the elements of V Dm in terms of m 1 given by (12). The third part is to provide examples of situations where the extreme values occur (13). In Figure 4, a case that is extreme both in the lower and in the upper sense is illustrated. Figure 4: To the left is a particular quantile function F −1 m and to the right is the associated function d m given by (14).
and consequently is an extreme case to (13) in both the lower sense and the upper sense.
In Theorem 3, we relate the volume of the optimal segmentations of Accuracy and the optimal segmentations of Dice for a given marginal probability m ∈ M. Theorem 3. For any m ∈ M, V Am given by (7) and V Dm given by (11) satisfy The complete proof is given in Appendix A.5 and is outlined as follows. First note that D m (s) ≤ 1 for any s ∈ S and then consider separately the cases when D m (s) < 1 for all s ∈ S and when there exist some s ∈ S such that D m (s) = 1. For the first case, it is obvious that sup s∈S D m (s) For the second case, we show that the volume of the optimizers are uniquely given by In either case, (15) holds.
In Theorem 4, the set of optimal solutions to Accuracy and Dice when constrained to a specific volume is shown to coincide.
coincide and are given by S m,v .
The complete proof is given in Appendix A.6 and is a straightforward application of Lemma 1. Of particular interest is the case when v = m 1 , since this correspond to the situation when the metrics are maximized under the constraint that there should be no volume bias.
It follows from Theorem 1 and Theorem 2 that the optimizers to both Accuracy and Dice are of the for some t ∈ [0, 1]. This type of charecterization is practical for proving properties on volume, but inconvenient for other tasks. In Theorem 5, we provide an alternative charecterization using threshold segmentations of the form s(ω) = I{m(ω) > α} or s(ω) = I{m(ω) ≥ α}, for some α. Even if there exist optimal segmentations that are not necessarily of threshold type, they can always be bounded, above and below, by optimal segmentations of threshold type. Theorem 5. For any m ∈ M and t ∈ (0, 1], let The complete proof is given in Appendix A.7 and is outlined as follows. First note that s 0 1 = For the upper bound, with A = {ω : s(ω) > s 1 (ω)}, we first observe that I{ω ∈ A} = s(ω)s 1 (ω) and then, using the definition of S m,v we prove that The lower bound is similar, but slightly more involved.
In numerical applications, the continuum Ω is usually partitioned into a finite collection of voxels {Ω i } i∈I . Marginal functions are then constrained to the subset of M that is compatible with the voxelization in the sense that m is measurable with respect to the σ-field generated by the partition. Note that if s 1 (ω) = I{m(ω) ≥ 1 − t} and s 0 (ω) = I{m(ω) > 1 − t} for some t ∈ (0, 1], then also s 0 and s 1 are compatible with the voxelization. By Theorem 1 (Theorem 2) and Theorem 5, the segmentations with least and greatest volume that are optimal with respect to Accuracy (Dice) are compatible with the voxelization. For the metrics respectively, we denote the segmentations with the greatest volume by: Note that s Am is analogous to the Bayes classifier and s Dm is analogous to the threshold classifier described in [19]. Both of these are trivial to compute from a given marginal function m compatible to some voxelization and code for doing so is available in the Supplementary Material.

Experiments
The sharp bounds on volume in Thereom 1 and Theorem 2 implies that there exist marginal functions for which the volume of the optimal segmentations to Accuracy and Dice deviate significantly from the expected target volume. In this section we conduct experiments on marginal functions formed from real world data to compare the volume of optimal segmentations to the expected volume in practice.
For our experiments we investigate two data sets. The first data set (G) contains segmentations in the pelvic area and is part of the Gold Atlas project [26]. The data is in 3D with a resolution of 512 × 512 pixels per slice and consist of 19 patients with 9 different ROI's (region of interest), each of which have been delineated by 5 experts (see Figure 1 for an illustration of the segmentations associated with two different ROI's for one patient). The second data set (L) contains segmentations in the thorax area and is part of The Lung Image Database Consortium (LIDC) and Image Database Resource Initiative (IDRI) [1] and is hosted by TCIA [8]. The data is in 3D with a resolution of 512 × 512 pixels per slice and contains 1018 cases with lung nodules delineated by 4 experts. For each data set, ROI and patient, a marginal function m is formed by taking the fraction of which each pixel has been selected by the annotators, that is, a finite sample approximation is considered. The resulting marginal functions are then used to compute the segmentations s Am (17) and s Dm (18). For (G) we make use of the software Plastimatch [31] and for (L) we make use of the python package pylidc [13]. Details on the experiment, implementation and how to obtain the data is available in the Appendix B and the code can be found in the Supplementary Material.
From our experiments we report the quantities s Am 1 / m 1 and s Dm 1 / m 1 , which in a relative sense describe how much the volume of the computed optimal segmentations with respect to Accuracy  [26]. For each ROI and patient, the associated m is formed by finite sample approximation and used to compute s Am as defined by (17) and s Dm as defined by (18).  Figure 6: Results of our experiments on the lung nodules in the LIDC-IDRI data set [1]. For each patient, the associated m is formed by finite sample approximation and used to compute s Am as defined by (17) and s Dm as defined by (18). Table 1: Results of our experiments on the pelvic data in the Gold Atlas project (G) [26] and the lung nodule data in LIDC-IDRI (L) [1]. For each ROI and patient, the associated m is formed by finite sample approximation and used to compute s Am as defined by (17) and s Dm as defined by (18). and Dice deviate from the expected target volume. In Figure 5 and Figure 6, these quantities are illustrated for each patient and ROI in scatter plots. In Table 1, aggregated statistics of these quantities with respect to all patients are shown. By simple inspection it is clear that the volume of optimal segmentations to Accuracy and Dice often deviate significantly from the expected target volume. For

Conclusion
In this work, we have theoretically investigated the optimal segmentations with respect to the performance metrics Accuracy and Dice. We have given a detailed rigorous characterization of the optimizers and upper and lower bounds on the volume of optimal segmentations. Finally, we have shown the relevance of our theoretical observations in practice by comparing the volume of optimal segmentations with respect to the performance metrics to the expected volume, on two real world data sets. We conclude that noise may cause optimal segmentations to have a volume that deviates significantly from the expected target volume and that the reason for this may be what metric is chosen, or implicitly, what metric a chosen training method targets.
Broader impacts: Formalizing the evaluation process of automated segmentation methods can be done in many ways, each with its pros and cons. Even if this work can be interpreted as describing the problems with using Dice for this formalization, it can still paradoxically contribute to an unhealthy fixation of Dice as the gold standard for segmentation evaluation in medical image analysis. This in turn can lead to that medical practitioners put too much faith in models that have been shown to perform well with respect to the metric on some test data. One solution to this is to make sure that clinical practitioners using such models are educated in the problems associated with the metric.
Limitations: In order for the volume bounds to be sharp in Theorem 1 and Theorem 2, we construct extreme cases of m. These extreme cases might only be representable approximately with step functions for a particular choice of voxelization. Consequently, the most extreme cases we can construct in a numerical setting may not be as extreme as those we have constructed in the continuous setting. However, in medical image analysis it is common to deal with voxelizations of the order of 512 × 512 × 100 voxels which means that the approximation error would be negligible. Our work is also limited by the amount of experiments included. Additional numerical experiments on a wider range of data sets would give a more comprehensive picture on the impact of different noise distributions.

A.1 Generalities
A lot of our work will be devoted to analyzing cumulative distribution functions and associated quantile functions. There are several properties and identities that we will use that can be found in any standard textbook on the topic. Here is an incomplete list. Let F (t), t ∈ [0, 1] be a cumulative distribution and let F −1 (v) = inf{t : F (t) ≥ v} be the associated quantile function. Then the following holds.
1. F is non-decreasing and right-continuous 2. F (0−) = 0 and F (1) = 1 Proof. Our proof consist of two parts. In the first part, we show that the class of maximizers to is given by S m,v . In the second part, we show that if s * ∈ S m,v , then This together shows the statement.
Part 1, optimality. Let s * ∈ S m,v and s ∈ S v . Then s * and s are feasible. By construction, we have that there must exist some non-negative numbers 1 , 2 ≥ 0 such that If s ∈ S m,v , then 1 = 2 = 0 and consequently Otherwise, if s ∈ S v \ S m,v , then at least one of the inequaities 1 > 0 and 2 > 0 hold. If 1 > 0, it follows that and similarly, if 2 > 0, then In either case, at least one inequality is strict and it follows that We conclude that S m,v is the set of optimal segmentations where the supremum is attained.
Part 2, equality. Let s * ∈ S m,v and introduce and We now interpretm as a random variable on (Ω, λ) with cdf given by F m and then note by the quantile transform, that F −1 m (U ) also has cdf given by F m , where U is uniformly distributed random variable on [0, 1]. It follows for C 1 that and for C 2 that Together with the definition of S m,v this yields Consequently, This completes the proof.

A.3 Proof of Theorem 1
Proof. Consider the function where the supremum of A m is attained for segmentations in ∪ v * ∈V * S m,v * . Moreover, the function a m is differentiable a.e., with derivative given by Part 2, bounds. We now prove the lower bound for F m ( 1 2 −) and the upper bound for F m ( 1 2 ). For this, first note that if we interpretm as a random variable on (Ω, λ), with cdf given by F m , and then note by the quantile transform, that F −1 m (U ) also has cdf given by F m where U is a uniformly distributed random variable on [0, 1], then Now, for the lower bound, note that, for v ∈ [0, 1], which implies that Consequently, for m 1 ≥ 1/2, and For m 1 < 1/2, clearly F m ( 1 2 −) ≥ 0. The cases m 1 ≥ 1/2 and m 1 < 1/2 together means that For the upper bound, note that, for v ∈ [0, 1], which implies that and it follows that F m ( 1 2 ) ≤ 2 m 1 . For m 1 ≥ 1 2 we clearly have that F m ( 1 2 ) ≤ 1. The cases m 1 < 1/2 and m 1 ≥ 1/2 together means that Part 3, sharpness. Recall that the domain Ω = [0, 1] n for some n ≥ 1. We use (ω 1 , . . . , ω n ) = ω ∈ Ω to denote the components.
Note that Furthermore, which means that F m0 ( 1 2 −) = 0. Together, these cases say that for any v ∈ [0, 1], the m 0 formed by taking satisfies m 0 1 = v and To see that the upper bound is sharp, take v < 1 2 and m 1 ∈ M such that Note that Furthermore, which means that F m1 ( 1 2 ) = 2v. Now instead take v ≥ 1/2 and m 1 ∈ M given by Note that which means that F m1 ( 1 2 ) = 1. Together, these cases say that for any v ∈ [0, 1], the m 1 formed by taking satisfies m 1 1 = v and This completes the proof.

A.4 Proof of Theorem 2
Proof. Consider the function and note that by Lemma 1 where the supremum of D m is attained for segmentations in ∪ v * ∈V * S m,v * . Note that d m is continuous and hence attains its maximum on the compact set [0, 1]. By continuity, the set V * ⊂ [0, 1] where the maximum is attained is closed and therefore compact. Moreover, d m is differentiable a.e. with derivative given by Consider the function δ given by The interest in the function δ comes from the fact that it has the same sign as ∂ v d m but is somewhat easier to work with. Indeed, note that δ Using these properties, it follows that there are three possibilities. In each case V * is a closed subinterval of [0, 1]. Now we prove that it is exactly the interval denotes the left limit. To this end, consider the three cases (i), (ii), and (iii) separately.
Recall that δ(v) > 0 for v < 1 and δ is left-continuous it follows that δ(1) ≥ 0 and by definition of δ that There are two cases to consider. If for v < 1, and consequently, This completes the proof in case (i). In The first inequality is similar to the proof of case (i) above. For v < a, δ(v) > 0, and left-continuity of δ implies that δ(a) ≥ 0 and it follows that There are two cases to consider. If F −1 m (a) = 1 − d * m 2 , then δ(a) = 0 and we claim that To show the reverse inequality a ≥ F m ( which implies that δ(v) > 0. Consequently, v ≤ a and we conclude that To complete the proof in case (ii) it remains to show that By taking the limit as . For the reverse inequality, since The Part 2, bounds. We now show that V * ∈ [ m 2 1 , 1]. Note that the upper bound is the upper bound of the feasibility set v ∈ [0, 1] so it is already done. For the lower bound, note that which implies that Therefore, We conclude that δ(v) > 0 for v < m 2 1 , which completes the proof.
To prove that the bounds are sharp, it is enough to consider one example. Take v ∈ (0, 1] and m ∈ M such that Note that Furthermore, or equivelently, which means that Then for v ∈ [0, 1] Hence, δ(v) = 0 for v > m 2 1 and d attains its maximum on V * = [ m 2 1 , 1]. Since we we can choose any v ∈ (0, 1] so that m 1 = v , this completes the proof.

A.5 Proof of Theorem 3
Proof. If we interpretm as a random variable on (Ω, λ), with cdf given by F m , and then note by the quantile transform, that F −1 m (U ) also has cdf given by F m where U is a uniformly distributed random variable on [0, 1], it follows that Now, as in Theorem 2, consider the function and note that by Lemma 1 This means and consequently that d * m ≤ 1. We now show that separately for the cases d * m < 1 and d * m = 1. First, assume that d * m < 1. Then there exist some > 0 such that d * m < 1 − 2 , hence and since F m is non-decreasing Secondly, assume that d * m = 1. Then there must exist some v such that d m (v) = 1, or equivalently If which is a contradiction and if v > |m| then which also is a contradiction. Consequently, we must have that v = m 1 . This in turn means that F −1 m needs to satisfy which can only be the case if This means that and finally that This completes the proof.
A.6 Proof of Theorem 4 Proof. By Lemma 1, the maximizers to are given by S m,v , and the maximizers to are given by S m,v . This completes the proof.
We know from the definition of S m,v that We conclude that (135) holds, which completes the proof. The data consist of 19 patients with 9 ROIs (region of interest) where each ROI has been delineate by 5 separate practitioners. This leads to a total of 855 segmentations. Let 1 ≤ r ≤ 9 be an index of the ROIs, and 1 ≤ p ≤ 19 be an index of the patients. We then think of each ROI for each patient as a random segmentation L p,r taking values in S. For each such random segmentation, we have access to 5 observations 1 ≤ i ≤ 5, and denote each observation with l {( s Am p,r 1 / m p,r 1 , s Dm p,r 1 / m p,r 1 )} 1≤p≤19,1≤r≤9 . (143) In Figure 5 theses points are illustrated as scatter plots and in Table 1 aggregated statistics of these points are listed.

B.1.2 Comments on license, identifiability and consent
From what we can gather, there is no licence specified by the creators. However, it is specified in the article and in the download link (see below) that the data is free to use for non-commercial purposes.
As always is the case with medical image data, identifiable of the patients may be an issue. Different experts and legislators have various opinions as to what lengths researchers has to go in order for the data to be considered anonymized. However, in this case this is not an issue since all of the patients signed informed consent to be part of the data. and download the Radiologist Annotations/Segmentations (XML) file. In our experiments, we use version 3, which at the time of writing is the current version. Extract the zip file to the Experiments_ L/xml folder. Note that we do not need access to the DICOM data in order to run our experiments.
3. Python: We use Python 3.10.4 which can be found at the following address.
https://www.python.org/downloads/ It is recommended but not necessary that you create a virtual environment. Once in the right python environment, the necessary packages can be installed by using the provided requirements file.
pip install -r requirements.txt 4. Running the code: To execute the code, make sure your path is set to the Experiments_L. The code is then simply executed with the following. python main.py It will take approximately 30min on a descent desktop computer and should be no problems running on a laptop. No GPU computations are done. The code will use the pilidc library to query annotation data and generate masks. The masks are used to compute the discrete versions of the marginal functions and used to compute the relative volumes. When complete, the results of the experiment can be found under Experiments_L/results.