{'fraction_non_alphanumeric': 0.034191612588095735, 'fraction_numerical': 0.0120717326076338, 'mean_word_length': 5.151072961373391, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Hierarchical Consistency Regularized Mean Teacher for Semi-supervised 3D Left Atrium Segmentation

Deep learning has achieved promising segmentation performance on 3D left atrium MR images. However, annotations for segmentation tasks are expensive, costly and difficult to obtain. In this paper, we introduce a novel hierarchical consistency regularized mean teacher framework for 3D left atrium segmentation. In each iteration, the student model is optimized by multi-scale deep supervision and hierarchical consistency regularization, concurrently. Extensive experiments have shown that our method achieves competitive performance as compared with full annotation, outperforming other state-of-the-art semi-supervised segmentation methods.


I. INTRODUCTION
Deep learning has been widely applied in the field of medical image processing, in which, segmentation techniques play an important role in diagnostic analysis and clinical treatment [1]. Segmenting left atrium (LA) structures from magnetic resonance (MR) images accurately can provide a pre-operative assessment of its anatomy, which is essential for treating various cardiovascular diseases, such as atrial fibrillation [2]. Despite the promising performance of deep learning on LA segmentation [3], [4], most of the existing methods rely on a great number of expensive annotations delineated by experienced experts, and the labeling process is tedious and laborious.
To ease the manual labeling burden, significant effort has been devoted to utilizing available annotations and resources efficiently to improve the segmentation performance [5], [6]. Noting that plenty of unlabeled data can be easily collected, many works focus on leveraging limited data with annotations and abundant data without annotations together for improving the segmentation performance in a semisupervised learning (SSL) manner [7], [8], [9], [10]. In recent years, the mean teacher (MT) model [11] has achieved great success in SSL, which enforces the consistency of the predictions with different perturbed inputs between student and teach models thereby improving the robustness of the student model. However, the consistency is only applied to the final predictions. Following [11], many perturbationbased SSL methods have been proposed, obtaining promising performance on LA segmentation [12], [13], [14]. These methods focus on uncertainty estimations to exploit rich unlabeled data from different perspectives with high computational complexity. For instance, UA-MT [12] estimates the uncertainty with multiple forward passes via Monte Carlo Dropout [15]. Another line of research [16], [17] tends to build task-level regularization by adding additional auxiliary tasks. In the absence of expert knowledge, these methods easily suffer from negative transfer.
In this work, a novel mean teacher-based SSL framework is presented for 3D LA segmentation, in which, we encourage the prediction consistency at different scales between the student and teacher networks. Concretely, we first extend the backbone framework to generate multi-scale predictions. Based on that, we inject multi-level deep supervision to different layers with labeled data to optimize the student network thereby producing more reliable targets. Meanwhile, we enforce the prediction consistency between student and teacher networks hierarchically to further regularize the student network with unlabeled data. Consequently, the proposed framework can efficiently take advantage of both labeled and unlabeled data for accurate SSL LA segmentation. Our framework is extensively evaluated on MICCAI 2018 Atrial Segmentation Challenge dataset [18]. Experimental results demonstrate that the proposed approach can effectively leverage unlabeled data for LA segmentation, and outperform many state-of-the-art SSL techniques.

II. RELATED WORK
Semi-supervised learning has achieved promising performance on medical image segmentation under the scarcity of labeled data. Bai et al. [7] involve a self-training process for cardiac MR image segmentation, where model updates and pseudo annotations are performed alternatively. Meanwhile, self-ensembling methods, particularly the mean teacher (MT) model [11], have received much attention on semi-supervised image segmentation. Specifically, the mean teacher (MT) structure [11] forces the consistency of the predictions with inputs under different perturbations between the student and teacher models and further boosts the model performance. Following that, Yu et al. [12] utilize uncertainty estimations with Monte Carlo sampling [15] to generate uncertainty maps for more reliable predictions. Beyond the prediction consistency, Li et al. [10] construct the transformation consistency into the self-ensembling model (TCSM) to improve the model generalization ability. Hang et al. [13] design the local and global structural consistency to extract structural information for semi-supervised LA segmentation. Li et al. [16] impose geometric constraints via an adversarial loss for shape-aware segmentation. Considering the consistency of different tasks, Luo et al. [17] apply a multi-task framework for segmentation and level set function regression together, enforcing the representation consistency at task level.
Despite the great success of semi-supervised segmentation, these methods suffer from high computational complexity, which increases the running time. It is observed that the consistency of existing methods is only applied to the final outputs of each voxel, leaving the abundant information from hidden layers unexplored. On the other hand, many works [19], [20] obtain multi-scale predictions from different layers for boosting the network performance. Inspired by this, we inject deep supervision into hidden layers for accelerating the convergence of model optimization, thereby improving the prediction accuracy. Moreover, we expect to leverage predictions at different scales to further regularize the mean teacher model hierarchically, while obtaining more reliable outputs via multi-level deep supervision with minimal extra computational cost.
III. METHODOLOGY Fig. 1 illustrates the proposed mean teacher-based SSL framework. Our method follows the basic spirit of the mean teacher and takes 3D medical images under different perturbations as inputs and encourages the consistency of the segmentation predictions. The student network and the teacher network utilize the same backbone structure. We introduce additional prediction layers to estimate the quality of the hierarchical hidden representations. On one hand, deeply supervised learning with labeled data is performed by minimizing the multi-level segmentation loss from the different hidden layers and the final layer. On the other hand, the multi-level segmentation prediction maps of the teacher network are taken as the learning goal of the student network, which is regularized by hierarchical consistency to effectively take advantage of unlabeled data.

A. Multi-scale Deeply supervised Network
In our framework, 3D V-Net [21] is employed as the backbone. To extract hierarchical hidden representations, we add auxiliary layers after each block of the decoding stage to form a hierarchical supervised component (HS). Specifically, the auxiliary layers consist of an upsampling layer, a 1 × 1 × 1 convolution, and a softmax layer to obtain the multi-scale predictions. For the labeled dataset D L = {(x i , y i ) , i = 1, . . . , L}, we consider the multi-scale predictions into model optimization for deeply supervised learning. We calculate the loss between predictions at different scales and ground truths respectively to obtain multiscale deep supervision. Formally, the hierarchical supervised segmentation loss function is defined as: where L ce and L dice are cross-entropy loss and dice loss, respectively. f s (·) denotes the segmentation prediction of HS at scale s. α s is the weights of HS component at scale s in the hierarchical network.

B. Hierarchical Consistency Regularized Mean Teacher
In SSL task, the training dataset consists of the labeled data D L = {(x i , y i ) , i = 1, . . . , L} and the unlabeled data H×W ×D is the 3D ground truth. L and U denote the number of 3D volumes of the labeled dataset and unlabeled dataset, respectively. Following the design of the mean teacher [12], the student network and the teacher network share the same network backbone, and the teacher network weights θ are updated with the exponential moving average (EMA) of the student network weights θ. The teacher network weights update strategy at step t is expressed as θ t = ηθ t−1 + (1 − η) θ t , where η is to control the EMA update rate, weighing the influence of the teacher network by the weights of current student network and the teacher network at step t − 1.
For the unsupervised dataset, we expect the segmentation results of the student network and teacher network to be consistent. To further enforce the consistency between the two networks, we design a hierarchical unsupervised consistency component (HU) to encourage the prediction consistent in the hidden feature space. In detail, based on the multiscale feature maps generated from the student and teacher networks, the hierarchical consistency regularization is introduced. The regularization effectively utilizes unlabeled data, and the consistency constraint between the student network and the teacher network is determined as the mean squared error (MSE) loss of multi-scale predictions between them, which is denoted as: where (θ, ξ) and (θ , ξ ) denote the different weights and input data with different perturbations of student and teacher networks, respectively. The unsupervised regularization loss can be combined with the deeply supervised loss to optimize the network together. The training objective can be formulated as: where L sup is the supervised term and L unsup is the unsupervised consistency regularization term, respectively. λ (t) is a time-dependent Gaussian weighting function that controls the supervised loss and the regularization term, which is defined as , where t and t max denote the current and maximum training steps, respectively.

A. Dataset and Experimental Settings
We evaluate our proposed semi-supervised method on the dataset of 2018 Atrial Segmentation Challenge for left atrium segmentation in 3D gadolinium-enhanced MR image scans (GE-MRIs) [18]. The dataset consists of 100 scans with segmentation masks and the resolution of the dataset is 0.625 × 0.625 × 0.625 mm 3 . Following the previous work [12], [16], [13], [14], [17], we select 80 samples for training and 20 for testing and use the same data preprocessing methods for a fair comparison.
The proposed architecture is implemented in Pytorch and updated by the SGD optimizer for 6000 iterations in total by setting the initial learning rate of 0.01 and a stepped decay by 0.1 every 2500 iterations in our experiments. Hierarchical classifier mechanism is introduced to the V-Net backbone and the number of scales S is 4. The scale-wise weights {α s , s = 0, . . . , 3} for supervised loss and unsupervised consistency loss are specified as {0.5, 0.4, 0.05, 0.05}, empirically. Following Tarvainen and Valpola [11], the decay parameter η of exponential moving average (EMA) is 0.99. We set the batch size to 4 and each batch consists of two labeled volumes and two unlabeled volumes. For evaluation, four metrics employed to quantitatively evaluate the segmentation performance are Dice, Jaccard, the average surface distance (ASD), and the 95% Hausdorff Distance (95HD). In our experiments, we use 16 out of 80 training samples as labeled dataset D L and the rest 64 samples are served as unlabeled dataset D U . The results in Table I compare our proposed approach and several other previous image segmentation methods. It is observed that there are significant improvements in segmentation performance for semi-supervised methods when compared to the supervisedonly method (V-Net) for the same amount of labeled data. In addition, the results show that our method achieved up to 90.04% in Dice score. It is noticed that our approach is superior to other semi-supervised methods and obtains comparable segmentation results using only 1/5 labeled data as compared with full annotation. Figure 2 exemplifies the visualization results of UA-MT [12], SASSNet [16], LG-ER-MT [13], DTC [17], our network and ground truth segmentation. It is observed that our method can obtain more reliable predictions which are closer to the ground truth segmentation labels. Remarkably, through our method, a more complete shape of the left atrium and more details are predicted, especially in the boundary area. This further demonstrates the effectiveness of our proposed strategy for 3D left atrium image segmentation. Furthermore, ablation studies are adopted to evaluate the effectiveness of the hierarchical supervised component (HS) and the hierarchical unsupervised consistency component (HU). We list the results in Table II. Compared with the V-Net trained on labeled data only, multi-scale deep supervision, i.e., V-Net + HS can efficiently improve the segmentation performance. And it can also be employed for SSL to better utilize the labeled data. Then we introduce HU and HS to the mean teacher model, individually, i.e., MT + HU and MT + HS, respectively. The results demonstrate the effectiveness of the two components. Lastly, combining two components together can achieve better performance.

V. CONCLUSION
In a nutshell, a novel and simple approach is proposed for semi-supervised 3D medical image segmentation. We present a framework based on the mean teacher architecture and it encourages the consistency of predictions between the student and the teacher networks with the same input under different perturbations. We perform the hierarchical prediction consistency based on hidden feature space to regularize the mean teacher model while obtaining more reliable outputs via multi-scale deep supervision. Experimental results on the Atrial Segmentation Challenge dataset demonstrate the superiority of our framework over the other state-of-the-art semi-supervised learning methods. Besides, the proposed method is simple and versatile, which can be widely applied to other segmentation tasks.