{'fraction_non_alphanumeric': 0.03676032020091038, 'fraction_numerical': 0.018113326008475907, 'mean_word_length': 4.400237328360739, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 9, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Marginal loss and exclusion loss for partially supervised multi-organ segmentation

Annotating multiple organs in medical images is both costly and time-consuming; therefore, existing multi-organ datasets with labels are often low in sample size and mostly partially labeled, that is, a dataset has a few organs labeled but not all organs. In this paper, we investigate how to learn a single multi-organ segmentation network from a union of such datasets. To this end, we propose two types of novel loss function, particularly designed for this scenario: (i) marginal loss and (ii) exclusion loss. Because the background label for a partially labeled image is, in fact, a `merged' label of all unlabelled organs and `true' background (in the sense of full labels), the probability of this `merged' background label is a marginal probability, summing the relevant probabilities before merging. This marginal probability can be plugged into any existing loss function (such as cross entropy loss, Dice loss, etc.) to form a marginal loss. Leveraging the fact that the organs are non-overlapping, we propose the exclusion loss to gauge the dissimilarity between labeled organs and the estimated segmentation of unlabelled organs. Experiments on a union of five benchmark datasets in multi-organ segmentation of liver, spleen, left and right kidneys, and pancreas demonstrate that using our newly proposed loss functions brings a conspicuous performance improvement for state-of-the-art methods without introducing any extra computation.


Introduction
Multiple organ segmentation has been widely used in clinical practice, including diagnostic interventions, treatment planning, and treatment delivery (Ginneken et al., 2011;Sykes, 2014). It is a time-consuming task in radiotherapy treatment planning, with manual or semi-automated tools ) frequently employed to delineate organs at risk. Therefore, to increase the efficiency of organ segmentation, auto-segmentation methods such as statistical models (Cerrolaza et al., 2015;Okada et al., 2015), multi-atlas label fu- Fig. 1. Five typical annotated images from five different datasets, one image per dataset. The colored edges show the annotated organ boundaries (red for liver, yellow for spleen, green for pancreas, blue for left kidney, and cyan for right kidney). The left image shows the case of a fully annotated data set and the amount of such data set is usually small. The right four images are partially labeled. Gibson et al., 2018). There is a rich body of subsequent works Chu et al., 2013;Suzuki et al., 2012;Liu et al., 2020;Gibson et al., 2018), focusing on improving existing frameworks by finding and representing the interrelations based on canonical correlation analysis especially by constructing and utilizing the statistical atlas.
However, almost all current segmentation models rely on fully annotated data Chen et al., 2018;Yang et al., 2017) with strong supervision. To curate a large-scale fully annotated dataset is a challenging task, both costly and time-consuming. It is also a bottleneck in the multi-organ segmentation research area that current labeled data sets are often low in sample size and mostly partially labeled. That is, a data set has a few organs labeled but not all organs (as shown in Fig. 1). Such partially annotated datasets obviate the use of segmentation methods that require full supervision.
It becomes a research problem of practical need on how to make full use of these partially annotated data to improve the segmentation accuracy and robustness. In the case of sufficient network model capabilities, a larger amount of data typically means that it is more likely to represent the actual distribution of data in reality, hence leading to better overall performance. Motivated by this, in this paper we investigate how to learn a single multi-organ segmentation network from the union of such partially labeled data sets. Such learning does not introduce any extra computation.
To this end, we propose two types of loss functions particularly designed for this task: (i) marginal loss and (ii) exclusion loss. Firstly, because the background label for a partially labeled image is, in fact, a 'merged' label of all unlabeled organs and 'true' background (in the sense of full labels), the probability of this 'merged' background label is a marginal probability, summing the relevant probabilities before merging. This marginal probability can be plugged into any existing loss function such as cross entropy (CE) loss, Dice loss, etc. to form a marginal loss. In this paper, we propose to use marginal cross entropy loss and marginal Dice loss in the experiment. Secondly, in multi-organ segmentation, there is a one-to-one mapping between pixels and labels, different organs are mutually exclusive and not allowed to overlap. This leads us to propose the exclusion loss, which adds the exclusiveness as prior knowledge on each labeled image pixel. In this way, we make use of the explicit relationships of given ground truth in partially labeled data, while mitigating the impact of unlabeled categories on model learning. Using the state-of-the-art network model (e.g., nnUNet (Isensee et al., 2018)) as the backbone, we successfully learn a single multi-organ segmentation network that outputs the full set of organ labels (plus background) from a union of five benchmark organ segmentation datasets from different sources. Refer to Fig. 1 for image samples from these datasets.
In the following, after a brief survey of related literature in Section 2, we provide the derivation of marginal loss and exclusion loss in Section 3. The two types of loss function can be applied to pretty much any loss function that relies on posterior class probabilities. In Section 4, extensive experiments are then presented to demonstrate the effectiveness of the two loss functions. By successfully pooling together partially labeled datasets, our new method can achieve significant performance improvement, which is essentially a free boost as these auxiliary datasets are existent and already labeled. Our method outperforms two state-of-the-art models Fang and Yan, 2020) for partially annotated data learning. We conclude the paper in Section 5.

Multi-organ segmentation models
Many pioneering works have been done on multi-organ segmentation, using traditional machine learning methods or deep learning methods. In (Okada et al., 2015;Xu et al., 2015;Tong et al., 2015;Suzuki et al., 2012;Shimizu et al., 2007;Wolz et al., 2013), a multi-altas based strategy is used for segmentation, which registers an unseen test image with multiple training images and use the registration map to propagate the labels in the training images to generate final segmentation. However, its performance is limited by image registration quality. In (Heimann and Meinzer, 2009;Cootes et al., 2001;Chen et al., 2012), prior knowledge of statistical models is employed to achieve multi-organ segmentation. There are also some methods that directly use deep learning semantic segmentation networks for multi-organ segmentation (Gibson et al., 2018;Wang et al., 2019;Kohlberger et al., 2011;Lay et al., 2013). Besides, there are prior approaches that combine the above-mentioned different methods Lu et al., 2012) to achieve better multi-organ segmentation. However, all these methods rely on the availability of fully labelled images.
2.2. Multi-organ segmentation with partially annotated data learning Very limited works have been done on medical image segmentation with partially-supervised learning. Zhou et al. ) learns a segmentation model in the case of partial labeling by adding a prior-aware loss in the learning objective to match the distribution between the unlabeled and labeled datasets. However, it trains separate models for the fully labeled and partially labeled datasets, and hence involves extra memory and time consumption. Instead, our work trains a single multi-class network. Since only two loss terms are added, it needs nearly no additional training time and memory cost. Dmitriev et al. (Dmitriev and Kaufman, 2019) propose a unified, highly efficient segmentation framework for robust simultaneous learning of multi-class datasets with missing labels. But the network can only learn from datasets with single-class labels. Fang et al. (Fang and Yan, 2020) hierarchically incorporate multi-scale features at various depths for image segmentation, further develop a unified segmentation strategy to train three separate datasets together, and finally achieve multi-organ segmentation by learning from the union of partially labeled and fully labeled datasets. Though this paper also uses a loss function that amounts to our marginal cross entropy, its main focus is on proposing the hierarchical network architecture. In contrast, we concentrate on studying the impact of the marginal loss including both marginal cross entropy and marginal Dice loss. Furthermore, it is worth mentioning that none of the above works considers the mutual exclusiveness, a well-known attribute between different organs. We propose a novel exclusion loss term, exploiting the fact that organs are mutually exclusive and adding the exclusiveness as prior knowledge on each image pixel.

Partially annotated data learning in other tasks
A few existing methods have been developed on classification and object detection tasks using partially annotated data. Yu et al. (Yu et al., 2014) propose an empirical risk minimization framework to solve multi-label classification problem with missing labels; Wu et al. (Wu et al., 2015) train a classifier with multi-label learning with missing labels to improve object detection problem. Cour et al. (Cour et al., 2011) propose a convex learning formulation based on the minimization of a loss function appropriate for the partially labeled setting. Besides, as far as semi-supervised learning is concerned, a number of researches have been developed to solve (He et al., 2019;Zhu et al., 2018;Xiao et al., 2019) classification problems or detection problems in the absence of annotations.

Method
The goal of our work is to train a single multi-class segmentation network Ψ by using a large number of partially annotated data in addition to a few fully labeled data for baseline training. Learning under such a setup is enabled by the novel losses we propose below.
Segmentation is achieved by grouping pixels (or voxels) of the same label. A labeled pixel has two attributes: (i) pixel and (ii) label. Therefore, it is possible to improve the segmentation performances by exploiting the pixel or label information. To be more specific, we leverage some prior knowledge on each image pixel, such as its anatomical location or its relation with other pixels, to facilitate the network for better segmentation; we also merge or split labels to help the network focus more on specific task requirements. In this work, we apply the two ideas on multi-organ segmentation tasks as follows. Firstly, due to a large amount of partially labeled images, we merge all unlabeled organ pixels with the background label, which forms a marginal loss. Secondly, regarding a well known prior knowledge that organs are mutually exclusive, we design an exclusion loss, which adds exclusion information on each image pixel, to further reduce the segmentation errors.

Regular cross-entropy loss and regular Dice loss
The loss function is generally proposed for a specific problem. A common idea for loss functions are based on classification tasks which optimize the intra-class difference and reduce the intra-class variation, for example contrastive loss (Hadsell et al., 2006), triplet Loss (Schroff et al., 2015), center loss , large margin softmax loss (Liu et al., 2016), angular softmax  and cosine embeding loss (Wang et al., 2018). The cross entropy loss (Long et al., 2015) is the most representative loss function, which is commonly used in deep learning. There are also some loss functions designed to optimize the global performance for semantic segmentation, such as Dice loss (Long et al., 2015), Tversky loss (Salehi et al., 2017), combo loss (Taghanaki et al., 2019), Lovasz-Softmax loss (Berman et al., 2018). Besides, some losses are proposed specifically to improve a given loss function, for example, the focal loss (Lin et al., 2017) is developed based on cross-entropy loss (Long et al., 2015) to better solve class imbalance problem. Here we focus on the cross-entropy loss and regular Dice loss that are most commonly used in multi-organ segmentation.
Suppose that, for a multi-class classification task with N labels with its label index set as Ω N = {C 1 , C 2 , . . . , C N }, its data sample x (i.e., an image pixel in image segmentation) belongs to one of N classes, say class C n , which is encoded as an Ndimensional one-hot vectorŷ n = [y 1 , y 2 , . . . , y N ] with y n = 1 and all others 0. A multi-class classifier consists of a set of response functions {a n (x); n ∈ Ω N }, which constitutes the outputs of the segmentation network Ψ. From these response functions, the posterior classification probabilities are computed by a softmax function, To learn the classifier, the regular cross-entropy loss is often used, which is defined as follows: (2) Besides, the Dice score coefficient (DSC) is often used, which measures the overlap between the segmentation map and ground truth. The dice loss is defined as 1 − DS C:

Marginal loss
For an image with incomplete segmentation label, it is possible that the pixels for some given classes are not 'properly' provided. To deal with such situations, we assume that there are a reduced number of M < N classes in a partiallylabeled dataset with its corresponding label index set as Ω M = {C 1 , C 2 , . . . , C M }. For each merged class label m ∈ Ω M , there is a corresponding subset Φ m ⊂ Ω N , which is comprised of all the label indexes in Ω N that can be merged into the same class m. Because the labels are exclusive in multi-organ segmentation, we have Ω N = ∪ m∈Ω M Φ m . Fig.2 illustrates the process of label merging, using an example of four organ classes C i , i = 1, 2, 3, 4. After the merging, there are two classes C 1 and C 2 , with C 1 and C 2 are combined together to form a new merged label C 1 and C 3 and C 4 to form a new label C 2 . Fig. 2. Venn Diagram to illustrate the marginal loss. The dataset contains three classes C 1 , C 2 , C 3 , C 4 , the partially labeled dataset only contains two class labels, with C 1 and C 2 merged together as C 1 and C 3 and C 4 merged together as C 2 .
The classification probability for the merged class m is a marginal probability Also, the one-hot vector for a class m ∈ Ω M is denoted asẑ m = [z 1 , z 2 , . . . , z M ], which is M-dimensional with z m = 1 and all others 0. Consequently, we define marginal cross-entropy loss and marginal Dice loss as follows: We use marginal cross entropy as an example to perform the gradient calculation. Firstly, referring to Eqs. (1) and (4), the gradient of the output m of a softmax node to the network node a j is: where π j (Φ m ) is a boolean indicator function that tells if j is in Φ m . p and q are the classification probabilities of regular and marginal softmax functions.The derivative gradient of L mCE to the network node a j is: wherem is the only class index that makes π j (Φ m ) = 0.

Exclusion loss
It happens in multi-organ segmentation tasks that some classes are mutually exclusive to each other. The exclusion loss is designed to add the exclusiveness as an additional prior knowledge on each image pixel. We define an exclusion subset for a class n as E n , which comprises all (or a part of) the label indexes that are mutually exclusive with class n. The exclusion label information is encoded as an N-dimensional vector e n = [e 1 , e 2 , . . . , e N ], which is obtained as: Note thatê n is still an N-dimensional vector, but it is not an onehot vector any more. Fig.3 shows the procedure of applying exclusion loss. Assuming that organ classes C 1 , C 2 and C 3 are mutually exclusive, the labels of C 2 and C 3 form the exclusion subset E C 1 . Fig. 3. Venn Diagram to illustrate the exclusion loss. There are three mutually exclusive classes C 1 , C 2 , and C 3 . The exclusion set for C 1 is We expect that the intersection between the segmentation prediction p n from the network and e n is as small as possible. Following the Dice coefficient, the formula for the exclusion Dice loss is given as: e n · p n e n + p n .
The exclusion cross-entropy loss is defined accordingly: where is introduced to avoid the trap of −∞. We set = 1.

Problem setting and benchmark dataset
We consider a partially-supervised multi-organ segmentation task that is common in practice (such as Fig. 1). For each partially annotated image, we restrict it with only one label. For clarity of description, we assume that F denotes the fullylabeled segmentation dataset and P i ; i ∈ {1, 2, . . . , C} denotes a dataset of partially-annotated images that contain only a partial list of organ label(s). The datasets P 1:C do not overlap in terms of their organ labels. For an image in P i , there is a 'merged' background, which is the union of real background and missing organ labels. We jointly learn a single segmentation network Ψ using F ∪ P 1 ∪ ... ∪ P C , assisted by the proposed loss functions.
For our experiments, we choose liver, spleen, pancreas, left kidney and right kidney as the segmentation targets and use the following benchmark datasets.
• Dataset F. We use Multi-Atlas Labeling Beyond the Cranial Vault -Workshop and Challenge (Landman et al., 2017) as fully annotated base dataset F. It is composed of 30 CT images with segmentation labels of 13 organs, including liver, spleen, right kidney, left kidney, pancreas, and other organs (gallbladder, esophagus, stomach, aorta, inferior vena cava, portal vein and splenic vein, right adrenal gland, and left adrenal gland) we hereby ignore.
• Dataset P 1 . We refer to the task03 liver dataset from the Decathlon-10 (Simpson et al., 2019) challenge as P 1 . It is composed of 130 CT's with annotations for liver and liver cancer region. We merge the cancer label into the liver label and obtain a binary-class (liver vs background) dataset.
• Dataset P 2 . We refer to the task09 spleen dataset from the Decathlon-10 challenge as P 2 . It includes 41 CT's with spleen segmentation label.
• Dataset P 3 . We refer to the task07 pancreas dataset from the Decathlon-10 challenge as P 3 . It includes 281 CT's with pancreas and its cancer segmentation label. The cancer label is merged into the pancreas label to obtain a binary-class (pancreas vs background) dataset.
• Dataset P 4 . We refer to KiTS (Heller et al., 2019) challenge dataset as P 4 . Since the offered 210 CT segmentation makes no distinction between left and right kidneys, we manually divide it into left and right kidneys according to the connected component. Cancer label is merged into the according kidney label.
The spatial resolution of all these datasets are resampled to (1.5 × 1.5 × 3)mm 3 . We split the datesets into training and testing. we randomly choose 6 samples from F, 26 samples from P 1 and 8 samples from P 2 , 56 samples from P 3 and 42 samples from P 4 as testing. The others are used for training. Table 2 also provides a summary description of the datasets.

Segmentation networks
We set up the training of 10 deep segmentation networks for comparison as in Table 1.
• Ψ mc F : a multiclass segmentation network based on F. • Ψ b P 1 : a binary segmentation network for liver only based on P 1 .
• Ψ b P 2 : a binary segmentation network for spleen only based on P 2 .
• Ψ b P 3 : a binary segmentation network for pancreas only based on P 3 .
• Ψ t P 4 : a ternary segmentation network for left kidney and right kidney only based on P 4 .
• Ψ b F+P 1 : a binary segmentation network for liver only based on F and P 1 . Note that the spleen, pancreas, left kidney and right kidney labels in F are merged into background.
• Ψ b F+P 2 : a binary segmentation network for spleen only based on F and P 2 . Note that the liver, pancreas, left kidney and right kidney labels in F are merged into background.
• Ψ b F+P 3 : a binary segmentation network for pancreas only based on F and P 3 . Note that the liver, spleen, left kidney and right kidney labels in F are merged into background.
• Ψ t F+P 4 : a ternary segmentation network for left kidney and right kidney only based on F and P 4 . Note that the liver, spleen, pancreas labels in F are merged into background.
• Ψ mc All : a multi-class segmentation network based on F, P 1 , P 2 , P 3 and P 4 .

Training procedure
For training the above networks except Ψ mc All , we use the regular CE loss, regular Dice loss, and their combination. For training the network Ψ mc All , when involves partial labels we need to invoke the marginal CE loss, marginal Dice loss, and their combination. Further, for Ψ mc All we experiment the use of exclusion Dice loss and exclusion CE loss.
Considering the impact of the varying axial resolutions of different data sets in the original CT image on the training process, we resample the 3D CT image to (1.5 × 1.5 × 3)mm 3 and then extract the patch with the shape [190,190,48] as input to illustrate the merit of our loss functions. For comparison, we use the same parameter settings in all networks; therefore there is no inference time difference among them. During training, we use 250 batches per epoch and 2 patches per batch. In order to ensure the stability of model training, we set the proportion of patches that contain foreground in each batch to be at least 33%. The initial learning rate of the network is 1e-1. Whenever the loss reduction is less than 1e-3 in consecutive 10 epochs, the learning rate decays by 20%.
We train 3D nnUNet (Isensee et al., 2018) for all segmentation networks. We choose the 3D nnUNet because it is known to be a state-of-the-art segmentation network. While there are Fig. 4. The comparison of results obtained by different segmentation networks. The red area represents the liver, the yellow area represents the spleen, the green area represents the pancreas, the cyan and blue areas represent the right and left kidneys, respectively. The edge with deeper color means the ground truth given by the dataset.
other network architectures (Fang and Yan, 2020) that might achieve comparable performance, we expect similar empirical observations from our ablation studies even based on the other networks.
For the network Ψ mc All , we train it in two stages in order to prevent the instability caused by large loss value at the beginning of the training. In the first stage, we only use the fully annotated dataset F. The goal is to minimize the regular loss function using the Adam optimizer. The purpose of the first phase is to give the network an initial weight on multi-class segmentation in order to prevent the large loss value when applying the marginal loss functions. In the second stage, each epoch is trained jointly using the union of five datasets. In each epoch, we randomly select 500 patches from each training dataset with a batch size of 2. Depending on the source of the slice, we use either the regular loss, if from F, or the marginal loss and the exclusion loss, if from P i (i ∈ {1, 2, 3, 4}). In actual experiment, the first stage consists of 120 epochs and the second stage 80 epochs.

Ablation studies
We use two standard metrics for gauging the performance of a segmentation method: Dice coefficient and Hausdorff distance (HD). A higher Dice coefficient or a lower HD means a better segmentation result. Table 3 shows the mean and standard deviation of Dice coefficients of the results obtained by the deep segmentation networks under different loss combinations and with different dataset usages, from which we make the following observations.
The effect of pooling together more data. The experimental results obtained by the models jointly trained from combinations of the datasets F and P i (i ∈ {1, 2, 3, 4}) are generally better than those by the models trained from a single labeled dataset alone. As shown in Table 3 and Table 4, when comparing the performance of Ψ b F+P i vs Ψ b P i (i ∈ {1, 2, 3, 4}), the former generally outperforms the latter. For example, when using rCE+rDC as the loss, the mean Dice coefficient is boosted from .851 to .900 (the according HD is reduced by 37.5%). When comparing the performance of Ψ mc F+P i (i ∈ {1, 2, 3, 4}) vs Ψ mc F , again the former is better than the latter, the mean dice coefficient is increased from .874 to .900 (the according HD is reduced by 28.7%).
The importance of CE and Dice losses. When comparing the importance of CE and Dice losses, in general, it is inconclusive which one is better, depending on the setup. For example, the Dice loss works better on liver segmentation while the CE loss significantly outperforms the Dice loss on left kidney segmentation. Also fusing CE and Dice losses is in general beneficial in terms of our results as it usually brings a gain in segmentation performance. For example, when using Ψ b F , the average dice loss reaches .874 for rCE+rDC, while that for rCE and rDC is .846 and .826, respectively.
The combined effect of data pooling and using marginal loss. It is evident that the segmentation network Ψ mc All exhibits a significant performance gain, enabled by joint training on the five datasets. It brings a 4.7% increases (.921 vs .874) in average dice coefficient for test images when compared with Ψ mc F , which is trained on F alone when using the dice loss and CE. The effect of exclusion loss. In addition, the exclusion loss brings significant performance boosting. The final results have been effectively improved by an average of 1.0% increases of Dice coefficient compared to the results obtained without the exclusion loss. This confirms that our proposed exclusion loss can promote the proper learning of the mutual exclusion between two labels. But it should be noted that exclusion loss is more like an auxiliary loss for partial label learning.
In sum, with the help of our newly proposed marginal loss and exclusion loss which enable the joint training of both fully labelled and partially labelled dataset, it brings a 3.1% increase (.931 vs .900) in dice coefficient. Such a performance improvement is essentially a free boost because these datasets are existent and already labeled.
Hausdorff distance. Table 4 shows the mean Hausdorff distance of the testing results, from which similar observations are made. Notably, jointly training from the five datasets, enabled by the marginal loss, can effectively increase the performances, especially it reduces the average distance from 9.33 to 4.43 (a 52.5% reduction) when using the Dice loss. Adding exclusion   2.17 ± 0.14 3.31 ± 0.07 8.50 ± 6.88 6.25 ± 6.81 9.33  dice can further improve the performances (4.43 to 4.02, another 9.3% reduction). The main reason for the big HD values for say spleen ∈ F is that sometime a small part of predicted spleen segmentation appears in non-spleen region. This does not affect the Dice coefficient but creates an outlier HD value. The impact of loss weight. In order to further explore the impact of marginal loss and exclusion loss on the performance, we set up the training of a series of models to understand the influence of the weight ratio of marginal and exclusion losses. All the models are trained on the union of F and all the partiallyannotated datasets. We experiment with ten different weight ratios: 4:1, 3:1, 2:1, 1:1, 1:2, 1:3, 1:4, 1:0, and 0:1. The dice coefficients and Hausdorff distances are reported in Table 5. Results demonstrate that a weight ratio of 1:2 achieves the best results on almost all the metrics. It is interesting to observe that, when only using exclusion loss (experiment with a weight of 0:1), there is nearly no performance improvement on pancreas and kidney comparing with Ψ mc F , which uses only F for training (as in Tables 3 and 4). This indicates that exclusion loss is more suitable as an auxiliary loss to be used with marginal loss together.
The effect of the number of annotations. Finally, we per-form a group of tests to measure the sensitivity of performance with the number of data annotation increases. We randomly split the fully annotated dataset F into a training set with 24 samples and a testing set with six samples and leave the testing set untouched. In the five sets of experiments reported in Table 6 , we alter the training set by replacing some fully labeled data with single labeled data, while keeping the total number of the training data unchanged. For example, for a '14/10' split, we have 14 fully labels images with 5 organs, and the rest of 10 images are further randomly divided into 5 single-label groups of 2 images. For the 1st group, we can use its liver annotation.
Similarly we use only the spleen, pancreas, left kidney, and right kidney labels for the 2nd to the 5th groups, respectively. As a result, we have a total of 14*5+2*5=80 annotated organs. Results in Table 6 confirm that the dice coefficient consistently decreases as the amount of annotation decreases, which is as expected.

Comparison with state-of-the-art
Our model is also compared with the other partiallysupervised segmentation networks. The results are shown in Table 7. The Prior-aware Neural Network (PaNN) refers to the   which adds a prioraware loss to learn partially labeled data. The pyramid input and pyramid output (PIPO) refers to the work by Fang et al. (Fang and Yan, 2020) which develops a multi-scale structure as well as target adaptive loss to enable learning partially labeled data. Our work achieves a significantly better performance than these two methods. The average Dice reaches 0.931 for our model, while that for PaNN and PIPO is 0.906 and 0.907, respectively. Our method also greatly reduce the mean Hausdorff distance by 24.0% comparing with PaNN and 40.0% comparing with PIPO. Specifically, our method achieves slight better (except for Liver∈ F) performance for large organs such as liver and spleen, but it brings a significant performance boost on small organs such as pancreas, left and right kidneys. Our work performs consistently better than the PIPO method on all the organs regardless the datasets, the improvement may be due to the use of 3D model as well as the exclusion loss. Fig. 4 presents visualization of sample results of different methods. With the assistance of auxiliary datasets, the performances are significantly improved. Especially, there are situations occurring on all the other methods that the predicted organ region enters a different organ, which results a large HD value. The exclusion loss used in our method can effectively reduce such an error and greatly improve the HD performance. Besides, our method can achieve more meticulous segmentation results on some small organs such as pancreas and kidney, es-pecially when there are small holes around the organ center.

Discussions and Conclusions
In this paper, we propose two new types of loss function that can be used for learning a multi-class segmentation network based on multiple datasets with partial organ labels. The marginal loss enables the learning due to the presence of 'merged' labels, while the exclusion loss promotes the learning by adding the mutual exclusiveness as prior knowledge on each labeled image pixel. Our extensive experiments on five benchmark datasets clearly confirm that a significant performance boost is achieved by using marginal loss and exclusion Table 7. Segmentation performance comparison in terms of Dice coefficients and Hausdorff distances between our proposed method and state-of-the-art methods.