{'fraction_non_alphanumeric': 0.03613151872817054, 'fraction_numerical': 0.015897868240395036, 'mean_word_length': 4.9210839077727595, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Bend-Net: Bending Loss Regularized Multitask Learning Network for Nuclei Segmentation in Histopathology Images

Separating overlapped nuclei is a major challenge in histopathology image analysis. Recently published approaches have achieved promising overall performance on nuclei segmentation; however, their performance on separating overlapped nuclei is quite limited. To address the issue, we propose a novel multitask learning network with a bending loss regularizer to separate overlapped nuclei accurately. The newly proposed multitask learning architecture enhances the generalization by learning shared representation from three tasks: instance segmentation, nuclei distance map prediction, and overlapped nuclei distance map prediction. The proposed bending loss defines high penalties to concave contour points with large curvatures, and applies small penalties to convex contour points with small curvatures. Minimizing the bending loss avoids generating contours that encompass multiple nuclei. In addition, two new quantitative metrics, Aggregated Jaccard Index of overlapped nuclei (AJIO) and Accuracy of overlapped nuclei (ACCO), are designed for the evaluation of overlapped nuclei segmentation. We validate the proposed approach on the CoNSeP and MoNuSegv1 datasets using seven quantitative metrics: Aggregate Jaccard Index, Dice, Segmentation Quality, Recognition Quality, Panoptic Quality, AJIO, and ACCO. Extensive experiments demonstrate that the proposed Bend-Net outperforms eight state-of-the-art approaches.


Introduction
Histopathology nuclei segmentation aims to extract all nuclei from histopathology images, and it provides reliable evidence for cancer evaluation. Conventionally, pathologists examine the shapes and distributions of the nuclei under microscopes to determine the carcinoma and the malignancy level (He et al., 2011).
However, the large number of nuclei makes the whole process time-consuming, low-throughput, and prone to human error. Automated nuclei segmentation is highly desirable in clinical practice. Recently, with the growing interest of digital pathology, the whole slide scanner provides a solution that transfers glass slides digitally to whole slide images (Pantanowitz, 2010).
In an H&E-stained histopathology image, nuclei are the first and the most visible structures among tissues. Accurate nuclei segmentation is essential in the further quantitative analysis (Aeffner et al., 2019), e.g., movement tracking, morphological changing, and nuclei counting. Many computational approaches have been proposed for automatic nuclei segmentation in histopathology images. Some conventional approaches (Ali et al., 2012;Cheng et al., 2009;Yang et al., 2006) utilized thresholding and watershed algorithms to segment nuclei, but these approaches are not robust in handling images with various nucleus types, fat tissue, and staining procedures. In recent years, deep learning-based approaches have been thriving in numerous biomedical image processing tasks (Badrinarayanan et al., 2017;Ronneberger et al., 2015;Shelhamer et al., 2015), and have achieved promising results in nuclei segmentation (Chen et al., 2016;Graham et al., 2018;Graham et al., 2019;Koohbanani et al., 2019;Kumar et al., 2017;Naylor et al., 2019;Oda et al., 2018;Qu et al., 2019;Vu et al., 2019;Xing et al., 2015;Zeng et al., 2019;Zhou et al., 2019). Xing et al. (2015) proposed a convolution neural network (CNN) to produce probability maps, and improved the robustness by using postprocessing, e.g., distance transformation, H-minima thresholding and region growing algorithm. Kumar et al. (2017) demonstrated a three-class (instance, boundary, and background) CNN that computes the label for each pixel to segment the nuclei. Naylor et al. (2019) constructed a DIST-map that utilized the regression output of nuclei distance maps for accurate nuclei segmentation. Although these methods achieved better results compared to conventional approaches, however, it is still challenging to segment nuclei accurately due to the existence of a large amount of overlapped nuclei.
Overlapped nuclei segmentation is challenging because of the lack of clear boundaries among nuclei, similar background textures, and large size and morphology variations. In recent published deep learningbased approaches, three main strategies have been proposed to address this challenge. The first strategy utilized the neural network to split the overlapped nuclei by generating both nuclei regions and boundaries.
For example, Kumar et al. (2017) proposed a three-classes CNN, including instances, boundaries, and background to segment the overlapped nuclei. Chen et al. ( 2016) proposed a multitask learning framework that output instance map and boundary map in separate branches. Vu et al. (2019) constructed a multiscale deep residual network with instances and boundary classes to segment nuclei. The second strategy integrated features from overlapped nuclei to improve the overall segmentation performance. Zhou et al. ( 2019) proposed the CIA-Net that utilized spatial and texture dependencies between nuclei and contours to improve the robustness of nuclei segmentation. Koohbanahi et al. (2019) proposed a SpaNet that captures spatial features in a multiscale neural network. Graham et al. (2018) proposed a new weighted cross-entropy loss that was sensitive to the Hematoxylin stain. Qu et al. (2019) constructed a new cross-entropy loss to learn spatial features for improving localization accuracy. The third strategy utilized the Watershed algorithm to segment the overlapped nuclei. Naylor et al. (2019) constructed a regression network that generated markers for the Watershed algorithm to segment overlapped nuclei. Graham et al. (2019) proposed the HoVer-Net architecture to output the instance map and horizontal and vertical nuclei distance maps for obtaining the markers of the Watershed algorithm. According to the reported results, these approaches achieved better overall performance than conventional methods, but their ability to separate overlapped nuclei is still limited (Fig. 1).
To address the challenges above, we proposed a novel bending loss regularized deep multitask network for nuclei segmentation. First, the proposed multitask network consists of three decoder branches: 1) instance segmentation branch, 2) boundary-distance branch for all nuclei, and 3) boundary-distance branch for overlapped nuclei. The third branch is designed to identify overlapped nuclei. Second, we propose the bending energy-based regularizer to penalize large curvatures of nuclei contours. In histopathology images, the curvatures of nucleus contour points change smoothly; but if one contour contains two or multiple overlapped or touching nuclei, their touching points on the contour will have sharp curvature changes (Fig.   2). Inspired by this observation, we develop the bending loss to generate large penalties for contour points with large curvatures. Third, we propose two new metrics to evaluate overlapped nuclei segmentation.
Previous approaches evaluate overlapped nuclei segmentation using metrics for overall segmentation performance, which hides the real performance of the overlapped nuclei segmentation. Compared to the closest work, HoVer-Net , both the proposed approach and the HoVer-Net follow the multitask learning architecture and use ResNet-50 as the building blocks. There are two major differences between the two approaches: 1) the proposed method introduced a new decoder branch to give focuses on overlapped nuclei; and 2) we propose the bending loss to penalize large curvatures of nuclei contours.
The rest of the paper is organized as follows. Section 2 describes the proposed method, including the bending loss, multitask learning network, and the loss function of the proposed architecture. Section 3 firstly  describes the datasets and evaluation metrics in our experiments, then presents the implementation and training process. The experimental results are discussed in Sections 3.3 to 3.7. Section 4 is the conclusion and future work.

Method
The proposed method, namely Bend-Net, consists of two key components: the bending loss and multitask learning architecture. Firstly, we propose a bending energy-based regularizer for penalizing touching nuclei points. Secondly, we propose a multitask learning network with three decoder branches that focus on overlapped nuclei contours. The final loss function consists of the regular segmentation loss , overlapped nuclei loss, and the bending loss.

Bending loss
Bending energy has been widely applied in measuring the shapes of biological structures, e.g., blood cells (Canham et al., 1970), cardiac (Duncan et al., 1991), vesicle membranes (Du et al., 2006), and blood vessels (Stuhmer et al., 2013). Young et al. (1974) used the chain-code representations to model bending energy. For 2D digital images, a contour is composed of discrete pixels, and the curvature of a contour point is computed by using the vectors created by neighboring points on the contour. For histopathology images, a nucleus usually has a smooth contour, and the points on the contour have small curvature changes; the points on the contour with large curvature have high probability to be the touching points of two/multiple nuclei ( Fig. 2.). To split the touching nuclei, we define the bending loss that gives high penalties to the contour points with large curvatures, and small penalties to points with small curvatures. The proposed total loss is given by where L0 refers to conventional segmentation loss (Section 2.3); Lbe denotes the proposed bending loss; and the parameter α controls the contribution of the bending loss. Let = { } =1 be the set of contour points of nuclei in an image, and Lbe is defined by where BE(i) is the discrete bending energy at the point , For three consecutive pixels on a nucleus boundary with coordinates xi-1, xi and xi+1, v Operator |· | calculates the length of a vector. defines the weight for concave contour points.
The 8-neighborhood system is applied to search neighbors for contour points. Ideally, a contour point only has two neighboring points, and their coordinates are used to calculate the edge vectors in Eqs. (3)      concave points are likely to be overlapped contour points. The previous approach cannot distinguish convex and concave contour points and tends to over-segment nuclei.
A sample of overlapped nucleus contour is shown in Fig. 4. The red dots highlight the concave points and the green dots highlight the convex points. The concave points' bending loss values are 28.28. The mid-points of green dots are inside of the predicted nucleus, and they are convex points; their bending loss values are less than 1.41. In Fig. 4, the concave points with bending loss 28.28 and the convex points with bending loss 1.41 have the same curve pattern; however, the concave points produce 20 times as much loss as the convex points.
The proposed bending loss is rotation invariant since all patterns with the same angle between two edge vectors have the same bending loss. In practice, if two nuclei contours share some contour segments, one contour point may have more than two neighbors. In this scenario, we calculate the bending loss for all possible combinations, and choose the smallest loss as the discrete bending loss for the point.
As shown in Fig. 5, for poorly-segmented nuclei contours, all these touching contour points have relatively high (red and green points) bending losses. If the touching nuclei are well separated (Fig. 5(c)), and the bending loss of all contour points are less than 9.66.

Multitask learning network
The proposed multitask learning architecture is shown in Fig. 6. The network follows an encoder-decoder design, and has three decoder branches. The encoder employs ResNet-50 (He et al., 2016) as a feature extractor. In the first convolutional layer, 64 7×7 kernels with a stride of 1 are applied, but the following max-pooling layer is removed to preserve more information. The network has three decoders/tasks. The first task predicts the nuclei instance map (INST); the second produces each nucleus's horizontal and vertical boundary-distance map (HV); and the third ouputs the overlapped nuclei's horizontal and vertical boundary-distance map (OHV). All decoders in the three branches have the same sub-architectures and the dense units (Gao et al., 2017). The OHV and HV branches share weights through skip connections.
The weight-sharing among decoders are designed to use features learned from similar tasks. In traditional multitask learning networks, different branches typically addressed different tasks. However, in the proposed network, both the HV and OHV branches share some comment results; one for all nuclei, and the other for overlapped nuclei. To take advantage of the features from two similar tasks, we design the skip connections among two branches to share weights. Specifically, the network first learns the distance maps of overlapped nuclei and aggregates them through skip connections to distance maps in the HV branch.

Loss function
As shown in Fig. 6, the loss function of the proposed network has four terms: the losses from three different decoders and the proposed bending loss. Let denote the loss of the binary instance map; be the loss of the horizontal and vertical distance maps from the HV branch; and denote the loss of the horizontal and vertical distance maps from the OHV branch; is the bending loss. The proposed loss function also can split into the segmentation loss (L0) and the bending loss regularizer (Eq. (1)). The total loss is given by: where is the weight of the bending loss in all loss function. We follow the design in  to set the losses of the three branches to have equal contributions to the total loss.

Loss of the INST branch.
To segment the nuclei instance, we calculate the binary classification for each image pixel. I and I* are the predicted instance map and the ground truth instance map for all nuclei. The loss ( ) is a summation of the cross-entropy loss ( ) and Dice loss ( ). They are given by ( , * ) = − 1 ∑ * log( ) where is the class prediction at point i, and n denotes the number of pixels in an image patch. The INST branch separates the nuclei instance from the background.

Loss of the HV branch.
The loss function is to compare the predicted distance maps (D) with the ground truth distance maps (D*) for all nuclei. We employed the distance loss function in .

Loss of the OHV branch.
is also defined using mean square error and the mean square gradient error (Eq. (9)). But is calculated using the predicted distance maps (D) and the ground truth distance maps of overlapped nuclei.

Datasets and evaluation metrics
In this paper, we validate the proposed method using two histopathology nuclei datasets: CoNSeP  and MoNuSegv1 (Kumar et al, 2017 where Sj is the matched predicted segments that produce the largest Jaccard Index value with Gi; and U denotes the set of unmatched predicted segments, where the total amount of U is (M -N).
Dice coefficient (DICE) is utilized to evaluate overall segmentation performance, the DICE is given by where operator |· | denotes the cardinalities of the segments.
PQ is used to estimate both detection and segmentation results. RQ is the familiar F1-score, and SQ is known as the average Jaccard Index of matched pairs. RQ, SQ, PQ are defined as where Sj is the matched nucleus in S that produces the largest Jaccard Index value with Gi.
Let M be the number of matched nuclei pairs between the segmentation and ground truth, and O denote the total number of overlapped nuclei in an image. For each overlapped nuclei, we iterate all the predicted segments, and count two nuclei matched if their Jaccard Index value is larger than a threshold τ (0.5). The ACCO is given by

ACCO =
The two metrics are general and can be applied to other overlapped object segmentation.

Implementation and training
The proposed approach is trained by using an NVIDIA Titan Xp GPU. The encoder was pretrained on ImageNet; and we trained the decoder for 100 epochs to obtain the initial parameters for the decoder branches. The network was further fine-tuned for 100 epochs on the nuclei training set. The size of the final output images is 80×80 pixels, and these output images are merged to form images with the same size (1000×1000) as the original images. In experiments, the initial learning rate is 10 -4 and is reduced to 10 -5 after 50 epochs. The batch size is 8 for training the decoder and 2 for fine-tuning the network. Moreover, processing an image of size 1000×1000 with our architecture takes about one second.
The input dimensionality of the network is 270×270×3. We prepare the training, validation, and test sets by extracting patches from images with 270×270 pixels size. During the training stage, data augmentation strategies, i.e., rotation, Gaussian blur, and median blur, are utilized for generating more images. The ground truths of overlapped nuclei are two or multiple individual nuclei have connected-component labeling. An example histopathology image, the ground truth of all nuclei, and the overlapped nuclei are demonstrated in Fig. 7.
The proposed scheme comprises three stages: 1) preprocessing; 2) training of the proposed multitask learning network; and 3) postprocessing. The preprocessing performs color normalization (Vahadane et al., 2016) to reduce the impact of variations from the H&E staining and scanning processes. The postprocessing described in Graham et al. (2019) is employed in this study, which applies Sobel operators to the distance maps to generate initial contour map; then the difference of initial nuclei contour map and nuclei instance map is used to generate markers; and finally, the watershed algorithm is applied to generate nuclei regions.

Effectiveness of the network architecture
The proposed multitask learning architecture uses HoVer-Net as the backbone and integrates our newly proposed overlapped nuclei (OHV) branch and skip connections (Fig. 6). To demonstrate the effectiveness of the proposed architecture, we compare the proposed network with the single-task network (Instance-Net), and two-task network (HoVer-Net). To perform a fair comparison, the proposed bending loss is not used. The approaches are evaluated on CoNSeP dataset by using AJI, Dice, RQ, SQ, and PQ scores. As shown in Table. 1, the Instance-Net does not apply any strategy to separate the overlapped nuclei and achieved very limited performance, e.g., AJI is only 0.371. The proposed network with the OHV branch ('Ours-OHV') achieved better average performance than the Instance-Net and HoVer-Net. With the new skip connections between the HV and OHV branches, the AJI, RQ, and PQ scores of the proposed approach  ('Ours-skip') increased by 3.54%, 3.30%, and 4.04%, respectively. The AJIO scores demonstrate that the proposed approach outperforms HoVer-Net in separating overlapped nuclei.

Effectiveness of the Bending Loss
The newly proposed bending loss improves our original bending loss calculation in (Wang et al., 2020) by characterizing the difference between the concave and convex contour points. First, we compare the proposed multitask learning network without any bending loss, with the bending loss (Lbe v1) in (Wang et al., 2020), and with the newly proposed bending loss (Lbe v2). Second, we demonstrate the effectiveness of the bending loss by adding it to HoVer-Net. The CoNSeP dataset and AJI, Dice, RQ, SQ, and PQ scores are used in experiments. As shown in Table. 2, the proposed architecture with the v1 bending loss (Wang et al., 2020) achieves better performance than that of the network without any bending loss, and the proposed architecture with the newly proposed v2 bending loss outperforms the network with the v1 bending loss. The results demonstrate that the v2 bending loss can improve the overall performance (AJI: from 0.565 to 0.578) of nuclei segmentation. Meanwhile, adding the v1 or v2 bending losses to HoVer-Net improves its overall performance, which demonstrated the potential of applying the bending loss to improve performance of other approaches. In addtion, the AJIO scores demonstrate that the proposed approach with the v2 bending loss outperforms all other approaches in separating overlapped nuclei.

Parameter tuning
Two hyper-parameters, , exist in the proposed loss function. balances the bending loss and all other losses (Eq. (5)); and (Eq. (3)) gives different weights to the concave and convex contour points when calculating the bending loss. We conducted grid search for the two parameters on the CoNSeP dataset by using the AJI score. Fig. 8 shows the AJI results of nine parameter combinations ( : 10, 20, 40; α: 0.5, 1.0, 2.0). As shown in Fig. 8, the proposed approach achieved the best performance when is 20, and is 1.0. Therefore, the bending loss of a concave curve pattern is twenty times the quantity of the same convex curve pattern. Refer to Fig. 3 for the bending loss of different curve patterns.

Performance comparison of state-of-the-art approaches
We compared eight deep learning-based approaches, including three widely used biomedical segmentation architectures: FCN8 (Shelhamer et al., 2015), U-Net (Ronneberger et al., 2015), and SegNet (Badrinarayanan et al., 2017), and five state-of-the-art nuclei segmentation approaches: DCAN (Chen et al., 2016), DIST (Naylor et al., 2019), Micro-Net , HoVer-net , and BEND (Wang et al. 2020). Table 3 shows the overall performance of nine approaches on two public datasets (CoNSeP and MoNuSegv1) using five metrics AJI, Dice, RQ, SQ, and PQ. Note that all other approaches are tested using the described experiment settings, and therefore, the values in Table 3 may not be the same as those reported in the original publications. The Watershed algorithm is applied to FCN8, U-Net, and

Overlapped nuclei segmentation
We proposed two new metrics, AJIO and ACCO, to evaluate overlapped nuclei segmentation. Table 4 shows the performance of overlapped nuclei segmentation on the CoNSeP and MoNuSegv1 datasets by using AJIO and ACCO scores. The DIST, Micro-Net, HoVer-Net, and the proposed method applied strategies to separate overlapped nuclei; therefore, their performances are significantly better than FCN8, U-Net, and SegNet. Our method achieved the best AJIO and ACCO scores on two datasets. Fig. 9 shows segmentation examples of six image regions with overlapped nuclei from the CoNSeP and MoNuSegv1 datasets. In the ground truth images, overlapped nuclei are represented using different colors. In the result images, if an approach can separate two overlapped nuclei, the two nuclei should be in two different colors.
As shown in Fig. 9, FCN8, SegNet, and U-Net tend to miss small nuclei, and cannot separate overlapped nuclei. DCAN is slightly better in handling overlapped nuclei than FCN8, SegNet, and U-Net, but tends to miss nuclei that are not small. DIST separates overlapped nuclei better than the last four approaches, but it tends to over-segment nuclei and generate many small regions. Micro-Net performs well in segmenting overlapped nuclei; but it produces smaller nuclei regions than the ground truth and tends to over-segment nuclei. Hover-Net shows better segmentation results than other six approaches. It can segment out small nuclei, and the sizes of result nuclei regions are close to those of the ground truth. It outperforms other six approaches in segmenting overlapped nuclei; however, it has difficult in separating closely touched nuclei.
In Fig. 9, the proposed method achieves the most accurate results on six images. The proposed approach not only can segment out small nuclei, but also can separate closely touched nuclei accurately.

Conclusion
In this paper, we propose a novel deep multitask learning network to address the challenge of segmenting overlapped nuclei in histopathology images. Firstly, we propose the bending loss regularizer, which defines different losses for the concave and convex points of nuclei contours. Experimental results demonstrated that the bending loss effectively improves the overall performance of nuclei segmentation, and it can also be integrated into other deep learning-based segmentation approaches. Secondly, the proposed multitask learning network integrates the OHV branch to learn knowledge from the overlapped nuclei, which enhances the segmentation of touching nuclei. Thirdly, we proposed two quantitative metrics, AJIO and ACCO, to evaluate overlapped nuclei segmentation. The extensive experimental results on two public datasets demonstrate that the proposed Bend-Net achieves state-of-the-art performance for nuclei segmentation. In the future, we will extend the proposed approach to more challenging tasks, such as gland segmentation and semantic image segmentation.