{'fraction_non_alphanumeric': 0.032806513409961685, 'fraction_numerical': 0.013136288998357963, 'mean_word_length': 4.968354430379747, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Cascaded Context Enhancement Network for Automatic Skin Lesion Segmentation

Skin lesion segmentation is an important step for automatic melanoma diagnosis. Due to the non-negligible diversity of lesions from different patients, extracting powerful context for fine-grained semantic segmentation is still challenging today. Although the deep convolutional neural network (CNNs) have made significant improvements on skin lesion segmentation, they often fail to reserve the spatial details and long-range dependencies context due to consecutive convolution striding and pooling operations inside CNNs. In this paper, we formulate a cascaded context enhancement neural network for automatic skin lesion segmentation. A new cascaded context aggregation (CCA) module with a gate-based information integration approach is proposed to sequentially and selectively aggregate original image and multi-level features from the encoder sub-network. The generated context is further utilized to guide discriminative features extraction by the designed context-guided local affinity (CGL) module. Furthermore, an auxiliary loss is added to the CCA module for refining the prediction. In our work, we evaluate our approach on four public skin dermoscopy image datasets. The proposed method achieves the Jaccard Index (JA) of 87.1%, 80.3%, 83.4%, and 86.6% on ISIC-2016, ISIC-2017, ISIC-2018, and PH2 datasets, which are higher than other state-of-the-art models respectively.


Introduction
Skin cancer is one of the most common human malignancies globally accounting for at least 40% of cases (Barata et al., 2017;Cakir et al., 2012). In the United States, there are more than five million new cases each year (Siegel et al., 2016). It has been a major public health problem and the incidence of skin cancer has been gradually rising (Guy Jr et al., 2015). Among them, melanoma is a highly malignant tumor, which is prone to metastasis and endangers the lives of patients. Meanwhile, melanoma is a kind of cancer with a high survival rate if treated promptly by chemotherapy, radiation therapy, and surgery at its early stage (Leachman et al., 2016). Therefore, early accurate diagnosis and screening are the keys to enhance the cure rate. As pigmented lesions occurring on the surface of the skin, melanoma is amenable to early detection by expert visual inspection. Dermoscope imaging technique is one of the effective noninvasive skin cancer detection methods, which can enhance the visualization of deeper levels of the skin and assist clinicians in diagnosis (Kittler et al., 2002;Silveira et al., 2009). Nevertheless, melanoma and other pigmented skin lesions are similar in color and texture. And they have obvious individual differences. Therefore, only experienced dermatologists can accurately identify them, while clinicians with limited experience may affect the optimal timing of treatment.
Also, the diagnostic process is laborious and time-consuming.
With the development of Artificial Intelligence, the computer-assisted diagnosis system can effectively avoid many of the above problems and help doctors improve the accuracy and efficiency of diagnosis. For automatic melanomas recognition, however, lesion segmentation as a fundamental component remains a challenging task (Abbas et al., 2013;. First, most melanomas consist of various colors from shades of brown to black, the size and shape of the lesions are also significantly distinct. Second, the contrast between the boundary part of the lesion and the normal skin is low and the difference is not obvious, which increases the difficulty of segmentation. At last, some artifacts including hair, ruler marks, and color calibration also affect the accuracy of recognition. Figure 1 shows some actual dermoscopy image examples with various lesion areas. Over the past decade, large numbers of automatic analysis algorithms for skin lesion segmentation have been proposed (Ganster et al., 2001;Chung & Sapiro, 2000;Xie & Bovik, 2013;Tang, 2009), which can be roughly classified into three categories: thresholding-based (Ganster et al., 2001), edge-based (Chung & Sapiro, 2000), and region-based methods (Yuan et al., 2009). Although these early attempts improve the accuracy of automatic lesion recognition through extracting pixel or region features, they still have some common defects: (1) low-level hand-crafted features (e.g. color, texture) restrict the performance of prediction in that other available cues in the original image are neglected.
(2) The robustness to artifacts, image quality, and contrast is low, which heavily depends on effective pre-processing.
Recently, compared with the approaches based on the hand-crafted features, significantly improved performance has been achieved on skin lesion segmenta-tion by deep convolutional neural network (CNN) (Esteva et al., 2017;Bi et al., 2019b), which mainly adopts the encoder-decoder structure, e.g. U-Net (Ronneberger et al., 2015), fully convolutional network (FCN) (Long et al., 2015). In the encoder stage, the high-level semantic feature representations are obtained through multiple convolution operations, then the top features are embedded into the decoder sub-network and generate the predicted segmentation mask. However, although high-level features have more abundant semantic features, much spatial detail is lost caused by consecutive convolution striding and pooling operations, which yields insufficient prediction performances. Additionally, capturing the long-range dependencies context plays an important role in promoting the performance of the segmentation. Many recent well-known works present different approaches to overcome these problems and get the informative context, such as dilated convolution (Lessmann et al., 2017), some non-local methods (Luo et al., 2017), and skip connection (Ronneberger et al., 2015). In order to obtain more semantic information, these approaches mainly focus on enlarging the local respective field in each stage of the encoder part, or bridging the feature maps from the encoder part with the same spatial resolution during the decoding process. However, for complex and fine lesions, they fail to reserve sufficient spatial details of the target and long-range dependencies contextual information. Overall, how to learn richer context and is still a challenge of the segmentation algorithm.
To address the above problems, we develop a novel neural network framework for skin lesion segmentation following the encoder-decoder structure. To embed more spatial details into high-level semantic features and encode long-range dependencies context, a cascaded context aggregation module (CCA) is designed by gradually and selectively aggregating original image and features from the encoder sub-network. The generated context of the CCA module is employed as a global context to guide the fine-grained mining of local context relationships.
In order to effectively integrate more global and complementary information for segmentation prediction, a new context-guided local affinity (CGL) module is proposed. Furthermore, an auxiliary loss module is added to the generated global context to refine the prediction performances. Finally, we evaluate our model on four public datasets, i.e. ISIC-2016 (Gutman et al., 2016), ISIC-2017 (Codella et al., 2018), ISIC-2018(Codella et al., 2019, and PH2 (Mendonça et al., 2013). The results demonstrate that our work has significant performance for skin lesion segmentation than other state-of-the-art methods. Our main contributions of this paper can be summarized as follows: (1) We present a cascaded context enhancement neural network, which employs a cascaded context aggregation module and a context-guided local affinity module to produce richer contextual information for skin lesion segmentation.
(2) We design a cascaded context aggregation module to embed additional spatial structural details into high-level features and generate richer non-local context. By designing a new gate-based information integration approach, multi-level features are well integrated.
(3) We propose a context-guided local affinity module that is used to extract discriminative features by leveraging the generated context to guide local affinity computation.
(4) We achieve outstanding state-of-the-art skin lesion segmentation performance on four public dermoscopy image datasets. The experimental results convince the efficiency of the proposed method.
The remainder of this paper is organized as follows: Section 2 is the related work with respect to skin lesion segmentation. In Section 3, we describe the details of our proposed segmentation neural network. Section 4 presents experimental results with different methods on four public datasets. And related analysis of our method is discussed. Finally, Section 5 concludes this paper and prospects some future works.

Encoder-decoder segmentation framework
In recent years, deep learning algorithms have wide applications in computer vision. Semantic segmentation is one of the fundamental vision tasks in daily life, which assigns semantic labels to every pixel in an image. Deep learning methods (Chen et al., 2017a;Lin et al., 2018) based on the FCN (Long et al., 2015) show striking improvement using encoder-decoder framework for many segmentation tasks. In the encoder sub-network, image content is encoded by multiple convolutional layers from low-level to high-level. And in the decoder part, the prediction mask is obtained by multiple deconvolutional (upsampling/uppooling) layers. In particular, image feature coding and context extraction are the keys for segmentation tasks. Some outstanding works have been developed by leveraging multi-scale convolution and attention mechanisms. For example, DeepLab (Chen et al., 2017a,b) designed an atrous spatial pyramid pooling module to encode the multi-scale contextual information by parallel dilated convolution. PSPNet (Zhao et al., 2017) adopted a pyramid pooling module to capture the rich context in the encoder part using different pyramid scales. Yang et al. (2018) proposed densely connected atrous spatial pyramid pooling by connecting each set of atrous convolutional layers in a dense way. PSANet (Zhao et al., 2018) used the point-wise spatial attention network to learn a pixel-wise global attention map. Through effective context feature extraction, the segmentation performance of the network can be greatly improved.

Skin lesion segmentation
For skin lesion segmentation, many achievements have been made based on deep learning over these few years. Most of these studies follow the deep fully convolutional neural network and encoder-decoder structure for lesions segmen-  based on encoder-decoder structure. And a loss function by combining both negative log-likelihood and end point error is adopted to accurately segment lesions with sharp boundaries. However, due to constrained local receptive fields and insufficient short-range contextual information, the performances of many segmentation approaches are limited. Overall, effective extraction of image context is crucial to enhance lesion segmentation performance.

Methodology
In this section, we first present a general framework of our method and then the details of the CCA module and CGL module are introduced which extract powerful global context and aggregate discriminative features for fine-grained semantic segmentation. Finally, we describe network optimization.

Overview
The overall framework design of the proposed model is illustrated in Figure   2. For simplicity, we denote training dataset indicates the background pixel and 1 denotes the foreground (lesion) pixel. The goal of deep learning based on skin lesion segmentation is to learn powerful contextual information for obtaining a fine-grain prediction mask. The proposed method is based on an encoder-decoder architecture. Specifically, we employ the ResNet (He et al., 2016) (pre-trained on ImageNet (Russakovsky et al., 2015)) as the backbone. And the atrous spatial pyramid pooling (ASPP) module (Chen et al., 2017a) is adopted after the last ResNet block to generate multilevel feature representations. The ASPP module consists of four parallel atrous convolutions with different atrous rates and one global average pooling, which is effective to capture multi-scale representations and encode context. The output features of ASPP are concatenated by upsampling and one 1 × 1 convolution (with 256 filters). Meantime, to maintain the large spatial resolution for the segmentation task, in our work, the last two blocks in ResNet are modified with atrous convolution (atrous rate = 2) and remove the pooling operation.
Thus, the sizes of these feature maps in each ResNet block are 1/4, 1/8, 1/8 and 1/8 of the input image. For the sake of encoding long-range dependencies context and embedding more spatial details to high-level features, a cascaded context aggregation module (CCA) is designed to sequentially enhance context features by integrating external origin image and features from each level of the encoder sub-network. Therefore, more spatial details are embedded into high-level semantic features and the context dependencies from low-to highlevel are recorded and mined. Furthermore, the generated context features of the CCA are employed to guide to extract discriminative features by the context-guided local affinity module (CGL). The CGL leverages the local affinity information and forms the finer context for the decoder part. In the decoder sub-network, we aggregate the output feature maps from the CCA module and CGL module, and further integrate them using a convolution layer for final skin lesion segmentation. Meantime, an auxiliary supervised module that utilizes the generated context features is added to the framework for assisting network optimization.

Cascaded context aggregation module
In the encoder sub-network, the low-level features are rich in spatial details, and the high-level features have more abundant semantic features  after consecutive pooling and convolution operations. Due to the lack of spatial detail and long-range dependencies context, however, the performance of decoding with high-level features alone is limited. In order to embed additional spatial structural details into high-level features and generate richer global context, a cascaded context aggregation (CCA) module is introduced by gradually aggregating the original image and encoder sub-network features of each stage.
The CCA module combines features from low-level to high-level and external raw image information adaptively, thus enhancing the ability of the context representation.
In our CCA module, the core component is the gate-based information integration unit, which is design for sequentially and selectively aggregating the message flows from different sources. For one image X (k) , without loss of generality, we denote the feature maps extracted from the last convolution layer of each residual convolutional block as f i , i ∈ {1, 2, 3, 4}. As illustrated in Figure   3, to extract powerful high-level features and capture long-range dependencies context, we use the external image information, the feature maps of front level and current level for refining and supplementing spatial details in each stage of the encoder sub-network. A gate-based information integration unit is designed to gradually aggregate the above message flows, which contains three inputs and one output setting. Specifically, for ith integration, the inputs are the generated context features f i−1,G of i − 1th integration, network features f i of current level and the image features f i,I . First, we obtain the output features f i of each stage using 1 × 1 convolution, which aims to further incorporate and reduce the dimension of features f i . It can be defined as follows: where F indicates one 1×1 convolution and θ i denotes the respective parameter.
And the original image X is resized to the same spatial resolution as features f i by downsampling operation. Then, features f i,I with same channel numbers as f i are extracted by a 3 × 3 convolution operation: where Down(·) refers to a downsampling operator. θ i,I is the convolution parameter. Thus, the addition-based fusion process in one gate-based information integration unit can be formulated as follows: where a i,I and a i,G denote the gate masks of features f i,I and f i−1,G respectively, which is used to screen important message from f i,I and f i−1,G adaptively. And ⊗ denotes the element-wise product. The gate function controls the message passing which selectively filters useful cues of front level and original image to current level. In our work, the gate function is designed using the 1 × 1 convolution and a sigmoid activation function to produce filter masks for f i,I and f i−1,G , which can be defined as follows: where C refers to the concatenation operation. θ i,I and θ i,G are the parameters of the 1 × 1 convolution. σ indicates the element-wise sigmoid function. For the sake of simplification, we define f G as the output of our CCA module. By the CCA module, we gradually enhance the context features and avoid introducing too much redundant content by the designed gate-based information integration unit. Meanwhile, the generated context f G can be viewed as the global feature maps to guide the local context learning, and improve the performance of segmentation as an auxiliary supervised branch.

Context-guided local affinity module
To further promote the features representation and discriminative capability, we propose a context-guided local affinity (CGL) module to capture the semantic relationship among different positions in local feature maps based on the position attention. In local feature maps, any two local positions or regions with similar features have correlated complementary improvement, which is helpful to improve the discriminative ability of the encoder features. Moreover, we employ the generated context f G for guidance, which can effectively where a L is the filter masks of generated context features f G . θ L denotes the parameters of 1 × 1 convolution. Then the obtained features f L ∈ R C×H×W with same channels as f A are used to encode finer local context by leveraging the spatial local affinity information. Given f p L , f q L ∈ R C , p, q ∈ {1, ..., HW } denote the features in the pth and qth position of the feature maps f L . The normalized similarity of local position features f p L and f q L can be obtained by softmax function: where d(·) denotes the metric function, in our work, we adopt the inner product of vectors as the metric. By calculating the similarity between any two spatial positions, a similarity matrix S ∈ R HW ×HW can be obtained. Therefore, the features f L can be updated by utilizing the similarity S and f A as follows: where f p L refers to the output of pth position of the CGL module. Therefore, the CGL module further aggregates the global and local features, and encodes more contextual relationships over local features.

Joint optimization with auxiliary loss
In the decoder sub-network, we incorporate extracted global context features f G and finer local context f L to f D by concatenation and convolution operation. Then the decoding feature f D is followed by a convolution layer and upsampling operator to generate the final prediction mapỸ . The calculation of the prediction mask can be written as: where θ D and θ Y are the corresponding convolution parameters. U p(·) denotes the bilinearly interpolation.
As illustrated in Figure 2, to learn powerful features and facilitate the training process, we add an auxiliary supervised module to the CCA branch and train the whole model by jointly optimizing the losses of two branches in an end-to-end manner. The introduced auxiliary supervised loss helps optimize the learning process while not influencing learning in the main branch. In our work, the weighted binary cross-entropy and dice loss are adopted as our objective function for each branch. For kth image, the total objective function is defined as follows: where b denotes the different branches (i.e. master branch and the auxiliary supervised branch). λ represents the balance parameters and is empirically set d denote the weighted binary cross-entropy (WBCE) loss function and dice loss respectively. The WBCE measures the accuracy of the segmentation, which are defined as follows: And L d is the dice loss which is suitable for imbalance samples and measures the coincidence of ground truth annotations and prediction results: where P (Y For fair comparisons, the compared results of these methods are generated by the original code released by the authors or taken from their respective publications.

Evaluation criteria
In this paper, five widely accepted metrics are employed as assessment criteria following the ISIC skin lesion segmentation challenges. They include Dice Similarity Coefficient (DI), Jaccard Index (JA), Accuracy (AC), Sensitivity (SE), and Specificity (SP). The details are as follows: where T P , T N , F P , and F N refer to the number of true positives, true negatives, false positives, and false negatives, respectively. They are all defined on the pixel level. Among these metrics, JA mainly assesses the overlapping between predictions and ground truth, which is the most important evaluation metric for segmentation tasks. Also, the participants of challenges are ranked based on the JA. In this work, we mainly refer to JA to evaluate the performance of all approaches.

Implementation details
The proposed method is implemented based on the Pytorch framework and is trained with one NVIDIA TITAN X GPU. In the model, we exploit the standard stochastic gradient descent optimizer with 0.9 momentum for training.
We set the initial learning rate as 10 −4 and adopt the "poly" learning rate policy following (Chen et al., 2017b). The learning rate is multiplied by (1− iter totaliter ) 0.9 after each iteration, eventually terminated at 150 epochs. All the training data is divided into mini-batches for network training, the mini-batch size is set as 8 during the training stage. In the CCA module, all feature maps are reduced to 256 channels with convolution operation for each integration. The backbone of the encoder sub-network is based on ResNet with the ASPP module pre-trained on ImageNet. The adopted ASPP consists of one global average pooling, one 1 × 1 convolution and three 3 × 3 convolutions with rates = (6, 12, 18). The output feature map is 1/8 size of the input image by our model. Then the predicted segmentation masks are bilinearly interpolated to target size directly.  Table 1 shows the performance of reference models and our work in the skin lesion segmentation task on the ISIC-2016 dataset. Compared to other segmentation approaches, our model outperforms existing approaches in almost all evaluation metrics. Specifically, the proposed method reaches an overall JA of 87.1% and DI of 92.6%. In comparison with the results of the top five teams of the challenge, the proposed architecture achieves JA increases of 2.8% than the best result of the competition (84.3%). Also, we compare the proposed algorithm with other state-of-the-art methods on the testing set. Our approach also obtains improvement among these methods to some extent. Compared with the classical FCN (Long et al., 2015), our method improves JA by 5.8%. And when compared with the recently published works (Bi et al., 2019b), (Xie et al.,  2020) and (Ibtehaz & Rahman, 2020), our model has also shown the obvious advantages with 1.2%, 1.3%, and 2.2% enhancement on metric JA, respectively. Table 2 compares the related metrics of reference models and our work in the ISIC-2017 segmentation task. As shown in the table, we can find that the proposed method has the highest DI of 87.8% and the best JA of 80.3% than other methods. In comparison with the results of classical U-Net, our work exceeds it 6.2% on metric JA, and has a 3.8% improvement than the best result of the competition (76.5%). Similarly, our work has considerable margin in terms of both JA and DI compared with some recently published methods, Lei    In addition, Table 3 class variation of the lesion, recognition of melanoma is more difficult than nonmelanoma cases. From the tables, we can observe that the proposed method achieves considerable gains in both cases, which suggests the effectiveness of our work on melanoma detection. Furthermore, Figure 5 provides a clear demonstration of the distribution in terms of JA for the whole case, melanoma case, and non-melanoma case on two datasets. From the figure, we can find that the performances of lesion segmentation with higher JA take up much of these two datasets by our work, which implies better utility and stability than other approaches.

Evaluation on ISIC-2018 dataset
For the ISIC-2018 dataset, our work also achieves state-of-the-art performance with the best average JA score of 83.4%. As shown in Table 5, the performances are averagely improved by 4.0% and 3.8% compared with the AG-Net and CE-Net, respectively. Compared to the latest DAGAN and CPF-  Net, benefiting from the guidance and integration of context, our method also shows a competitive advantage. This proves that effective long-range contextual information preservation has an important impact on the fine segmentation results.  In particular, the base model can extract coarse region contour on the whole.

Evaluation on PH2 dataset
However, for some images with low contrast or unclear boundary, the segmenta-tion performance is unsatisfactory. Compared with the baseline, the local affinity greatly enhances the expression of features, achieving better local connectivity of the lesion area. But, due to the lack of guidance of global information, many pseudo regions are identified as the lesion. With the CCA, the generated context adaptively integrates the original image and multi-level encoding features, making some lesion areas with low contrast recognized. However, the detail of the local area is not captured finely. Therefore, only integrating the CCA or CGL module, the performance of skin lesion boundary recognition is not accurate. With the generated context and the context-guided local relationship mining, some details and lesion region boundaries are clearer. Results show that the CCA module and CGL module bring great benefit to identify the lesion area, and the proposed method performs much better at details and lesion boundaries.

Effect of different numbers of training samples
In this section, the effect of different numbers of training samples is analyzed on the ISIC-2017 dataset. We conduct additional experiments by setting the number of training samples to 100, 500, 1000, 1500, and 2000. Experiment results are shown in Table 8. As we can see, the performances of the baseline and our method are generally improved with the growing numbers of training images in that more skin lesion content is captured. When the numbers increase from 100 to the whole training set, the overall DI and JA of our model obtain 8.2% and 9.7% gains on the testing set respectively. And compared with the baseline, the proposed method consistently provides superior performances under the different numbers of training samples in DI and JA. In particular, when only half of the whole training samples is used for learning, we achieve an average overall JA of 76.1%. And the better segmentation performance is obtained using three-quarters of samples (79.1%), which has outperformed other state-of-the- art methods in Table 2.

Effect of auxiliary loss
In our model, we add an auxiliary supervision module to the generated context features of the CCA module for assisting network training. To evaluate the effectiveness of the joint multi-loss optimization, we compare it with the model adopting single-loss in the main decoder branch. As shown in Table 7, both the average overall DI and JA increase by 0.3% with the auxiliary loss, which suggests that joint multi-loss optimization can improve the segmentation performance of the proposed method.

Conclusion
In this paper, we present a cascaded context enhancement neural network for automatic skin lesion segmentation, which produces a richer global context and adaptively integrates local contextual information. Specifically, a cascaded context aggregation module with a gate-based information integration approach is proposed in the encoder sub-network to adaptively capture richer long-range dependencies context. At the same time, a context-guided local affinity module is employed to extract discriminative features by leveraging the local context relationship. Compared with the state-of-the-art deep learning methods for skin lesion segmentation, our proposed approach can effectively achieve multilevel feature integration and generate discriminative feature representation. The experimental results convince the superiority of the proposed method. Based on the outstanding performance of our work, in the future, we will extend our work to support various medical image segmentation tasks including the 2D and 3D image datasets.