{'fraction_non_alphanumeric': 0.03709914064137498, 'fraction_numerical': 0.012785579543072731, 'mean_word_length': 4.6281332940135655, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A Hybrid Framework for Tumor Saliency Estimation

Automatic tumor segmentation of breast ultrasound (BUS) image is quite challenging due to the complicated anatomic structure of breast and poor image quality. Most tumor segmentation approaches achieve good performance on BUS images collected in controlled settings; however, the performance degrades greatly with BUS images from different sources. Tumor saliency estimation (TSE) has attracted increasing attention to solving the problem by modeling radiologists' attention mechanism. In this paper, we propose a novel hybrid framework for TSE, which integrates both high-level domain-knowledge and robust low-level saliency assumptions and can overcome drawbacks caused by direct mapping in traditional TSE approaches. The new framework integrated the Neutro-Connectedness (NC) map, the adaptive-center, the correlation and the layer structure-based weighted map. The experimental results demonstrate that the proposed approach outperforms state-of-the-art TSE methods.


INTRODUCTION
Breast cancer is the most frequently diagnosed cancer and account for about 29% of all new female cancer cases [1]. Automatic BUS segmentation is a key component in computeraided diagnosis (CAD) systems and has the advantages of operator-independence and high reproducibility [2,3]. However, developing automatic segmentation approaches for BUS images is challenging due to the speckle noise, low contrast, weak boundary, and artifacts; furthermore, strong priors to object features such as tumor size, shape and echo strength vary considerably across patients and machine settings, and cannot work well on images from multiple sources [19].
Many automatic BUS segmentation approaches have been proposed in the last decade [3,18,21,[33][34][35][36]. The major strategy of the approaches is to locate tumors automatically by modeling domain-related priors. However, some strong constraints such as the number of tumors, tumor size, and predefined tumor locations, were utilized in the approaches, which result in dramatic performance degradation in clinical practice where BUS images could be collected under different settings or situations such as low image contrast, more artifacts, containing no tumor/more than one tumors per image, etc. Therefore, it is crucial to develop automatic BUS segmentation techniques that are invariant and robust to images settings.
Visual saliency estimation (VSE) is an important and popular method to achieve the unconstrained image segmentation [20][21][22] by modeling human visual mechanism. It measures the degrees of different image regions attracting human attention. The center, and the local and global contrast are typical clues modeled in VSE approaches. Generally, VSE approaches can be classified into two categories based on the ways of saliency generation. First, the directly mapping methods [4][5][6][7][8][9] transfer image features into saliency values using predefined mapping; second, the optimization models [10][11][12][13][14][15][16][17] focus on modeling different hypotheses into one framework, and the saliency values are generated by optimization techniques. The approaches in the former category are faster; however, they have lower accuracy because the directly mapping methods fail on images with low contrast and big objects. While approaches in the latter category can achieve better performance by automatically adapting and balancing different elements of the models. Most VSE models were proposed to process natural images utilizing bottom-up frameworks and cannot achieve good performance on BUS images (Fig.1 saliency estimation hypothesis and achieved very good performance using their BUS image dataset. However, it has two main drawbacks: 1) it always outputs a salient region and cannot deal with images without tumor; 2) the predefined mapping failed to handle the images with large tumors, shadows, and low contrast ( Fig. 1). Xie et al. [20] proposed to calculate TSE by modeling intensity, blackness ratio, and superpixel contrast; and the final saliency value of each pixel was the average of values of the three components. It shares the same drawbacks with [20] due to the nature of direct mapping and the strategy of "winner-take-all".
Xu et al. proposed a general bottom-up saliency estimation model [22] that integrated many robust hypotheses: the global contrast, adaptive center-bias, boundary constraint and the smoothness term based on the color statistic. The model is flexible, and the global optimum can be reached by using the primal-dual interior point method. However, the model always outputs a salient region and cannot handle BUS images without tumors.
To solve the problems, we propose a novel hybrid framework for TSE, which follows a two-step strategy. The first step determines if a BUS image has tumor (s) based on the weighted map by utilizing adaptive reference point (RP) generation [3]. In the second step, it formulates the TSE problem as a quadratic programming (QP) problem which integrates both high-level domain-knowledge and robust low-level saliency assumptions. In this framework, it incorporates Neutro-Connectedness (NC) [31] to generate more robust and accurate boundary connectedness, and to measure the corresponding degree of confidence simultaneously. The adaptive center-bias (AC), foreground map (FG Map), which is based on the weighted map, and regions' correlation hypothesis are also integrated into the framework. The flowchart of the proposed model is shown in Fig.2.

A. Tumor existence determination
Existing TSE methods assume that there exists a tumor in each BUS image and cannot handle the image without tumor; however, as an automatic tumor detection or segmentation system, it is important to identify whether there is a tumor or not. Besides, the convex optimization frameworks [22] cannot deal with the image without salient object. The equality constraint will force that there must be at least one salient object in an image.
In [3], Xian et al. proposed an algorithm to automatically generate the adaptive reference point (RP) based on the breast anatomy. The RP is generated accurately and fast and can detect the darker regions (candidates of the tumors). The weighted map is constructed based on the RPs and the intensity map. The region is nearer the RP, the intensity value of the region in the weighted map is higher, vice versa. As shown in Fig.3, the weighted map will enhance the low-intensity pixels near RP and decay the high-intensity pixels far away from RP.
Based on the observation, the weighted map of the BUS image without tumor is smoother than that of the BUS images with tumor. In Fig. 3, the four max intensities of the weighted maps are 0.043, 0.0152,0.9086 and 0.0035, listing from left to right, top to bottom respectively. It chooses the local maximum, mean, and standard deviation of the weighted map as the feature vector and applies the threshold or Decision Tree. The result and discussion are in Section III.

B. Tumor Saliency Estimation
Researchers have applied several saliency hypotheses to construct mathematics models for VSE, such as rarity hypothesis, center-bias hypothesis, correlation hypothesis, etc. In this work, it utilizes the adaptive center-bias, regions' correlation hypotheses, the boundary NC and weighted maps to model the TSE problem as a convex optimization problem.
Firstly, it used a quick shift algorithm in [24] to over-segment the image into N superpixels, noted as { } . Similar to the method in [21], it extracts regions' average intensities as the region features. To facilitate the discussion, it defines S = ( , , ⋯ , ) as a vector of saliency values for N image regions, where denotes the saliency value of the ith image region and ∈ [0, 1]. The optimization of the model is to assign the optimal saliency values for a set of image regions.

1) Problem formulation
The problem is formulated as (1) In Eq. (1), the term = ( , , ⋯ , ) denotes the NC map， and defines the NC between the ith region and the boundary; the term = ( , , ⋯ , ) is the distance map, and defines the distance between the ith region and the adaptivecenter; the term = ( , , ⋯ , ) is the FG map, and is the value of the ith region; the terms and define the similarity and the spatial distance between the ith and the jth regions, respectively. The term S T T defines the cost using the NC map, the term S T W defines the cost on the FG map, and the term S T C defines the cost based on the adaptive center-bias. The last term is the smoothness that forces the regions with similar features to have similar saliency values.
The formulated problem is a typical QP problem with linear equality and inequality constraints. The original problem can be rewritten as follows:

2) NC map generating
Boundary connectivity is an effective prior utilized in many visual saliency estimation models [22,[26][27][28][29][30]. Most models define the boundary connectivity by using the shortest path between the local regions and the boundary. However, such connectivity cannot handle noisy data well. The Neutro-Connectedness (NC) theory [31,32] introduces a new domain, the degree of confidence, to measure the confidence of the connectedness. The new domain is very useful to avoid the fake connectedness caused by the uncertainty, such as noise.
In [32], the NC of two region contains three parts: the degree of truth, the degree of confidence, and the degree of false, Here, NC map is defined on image region { } . To calculate the NC triplet between all the regions with the boundary set, it applies the definitions of NC and computation algorithm in [31,32]. For more details of NC theory, refer [31. 32]. The three basic ideas of NC are summarized as follows: NC of two adjacent regions i and j where is the regions average gray level of the ith region, = 0.5, and ℎ( ) is the inhomogeneity of the ith region [31,32 ].
NC of a path. The degree connectedness of a path is defined as the minimum value of along the path, and the confidence is the maximum value along the path.
NC of any two regions. The degree of connectedness is defined by the strongest path of all paths connecting the two regions. It uses the confidence of the strongest path as the degree of confidence of the two regions. For more information about how to deal with ties, refer [31,32].
As the particular characteristic that no tumor is touching the border, it sets the border regions as the background seeds to generate the NC map by using the algorithm in [31]. Fig. 4 shows some samples, T maps, which refers to the background map generated by the NC theory, and I maps, which refers to the confidence of the degree of NC between each local region and boundary set. Based on the three basic ideas discussed, the I map will be used to generate each path of NC map.
T and I maps of image regions employed in the framework are defined as = ( , , ⋯ , ) and = ( , , ⋯ , ) , and they are × 1 vectors.

3) Adaptive center bias and FG maps generation
Traditional saliency estimation models usually use the image center as an important visual clue to estimate the saliency map. However, it fails when objects are far away from the center. The approach in [22] solve this problem on natural images by estimating the adaptive center using the weighted local contrast map; but the local contrast map is sensitive to noise and cannot achieve good performance on BUS images. Instead of detecting the top and bottom lines of mammary layer [21], a new tumor detection approach was proposed by utilizing the RP and weighted map [31].
The FG map in the framework is defined as = ( , , ⋯ , ) is the mean value of the ith region in the weighted map, and σ = 0.5. is equal to √2/2.

4) Regions' correlation
It uses the region correlation hypothesis to force the closer similar regions to have similar saliency value.
It defines w as the similarity, and as the spatial distance between the ith and the jth regions.
where Gray is the regions average gray level vector, In Eq. (8), is the coordinate of the ith region's center and the value is

C. Optimization
It uses the primal-dual method to optimize the QP with linear inequality and equality constraints similar with [22]. It can obtain the global optimal value quickly. There are three important steps to apply the primal-dual interior point method: (1) modify the KKT conditions and obtain the dual, prime and centrality residuals; (2) obtain the primal-dual search direction; and (3) update and the dual variables.
In the primal-dual interior method, and are initialized as 1 and (1 ) ⁄ , respectively; and the dual residual, primal residual, and the centrality residual are updated in each iteration, and the optimization processing stops when the sum of the norms is less than 10 .

A. Datasets, metrics and setting
In this section, it validates the performance of the newly proposed method using a BUS image dataset containing 706 ultrasound images from heterogeneous sources, in which 96 images have no tumors, and 610 images have tumors [25]. All experiments are performed by using Matlab (R2014a, MathWorks Inc., MA) on a Windows-based PC equipped with a dual-core (3.6 GHz) processor and 8 GB memory.
Metrics of saliency estimation: the saliency estimation is evaluated using the 610 images with tumors. The precisionrecall (P-R) curve, F-measure and mean absolute error (MAE) are employed to evaluate the overall performance of saliency detection method. The precision and recall ratios are defined as follows: where SM denotes the binary saliency map, while GT is the ground truth binary map, and | | denotes the white pixel number of the saliency map. The P-R curve shows the mean precision and recall rate of all saliency maps on a dataset. For each method, the P-R curve is calculated by segmenting saliency map with threshold range from 0 to 255, and computing the precision and recall rates by comparing the thresholding result with the ground truth. To obtain the average precision and recall rates, it uses an adaptive thresholding method [8], which chooses two times the mean saliency value as the threshold. The F-measure [9] and MAE [26] are defined as where is set to 0.3 as suggested in published saliency detection methods, is the coordinate of the ith image pixel, S( ) is the saliency value of the ith pixel, and G is the binary ground truth.
Metrics of tumor existence determination: two metrics, precision ratio, recall ratio are utilized: where | | is the number of correct detected images with tumors, and | | is the total number of images detected with tumors; | | is the total number of images with tumors in the dataset.
Parameter setting: all the experiments are based on the parameters: = 10, = 2, and = 80.

B. Tumor existence determination
Based on the observation, the maximum value of the weighted map is very useful to identify whether there is a tumor. Simple thresholds were applied to the maximum value, and the result is shown in Table 1. There are no distinguish differences (less than 2%) between the values of PR and RR by using different threshold values because of the limitation by the number of images with no tumor. Although the difference is small, it will have a significant impact in the clinical practice. The mean accuracy is 100% if using the Decision Tree classifier with ten-fold cross-validation method.

C. Tumor saliency estimation evaluation
The proposed method is compared with most recently published methods SMTD [21], OMRC [22], RBD [26] and RRWR [17]. SMTD is the directly mapping method for tumor saliency estimation. And the other three are the bottom-up models and can achieve good performance in the natural images. The proposed method can make the high saliency value concentrate on the tumor and the background areas have low saliency value while the methods SMTD, RBD and RRWR made the background regions around the tumor have higher salient values. This situation will cause the recall ratio of the methods higher but the precision ratio lower; and the result are more accurate than those of other methods are. Especially, SMTD and OMRC cannot produce good saliency maps even miss the big tumors.
The performance of the proposed method is evaluated using the metrics and the dataset: MAE values, the F-measure values, and P-R curves. As shown in Figs. 5-6, the proposed method is better than other methods. The methods, SMTD, RBD and RRWR, can obtain relative high average recall ratios, but the precision ratios and F-measures are low. It is s because the saliency maps generating by those methods make the tumors as well as the background around the tumors have high saliency value.

D. The effectiveness of NC term
The NC map with the boundary connectivity based on the graph shortest path is computed. In this experiment, it used the algorithm of [29] and defined the edge weight for each pair of adjacent nodes as − to obtain the background map, where is the average gray level of the ith region. The examples of 5 images are shown in Fig. 7. The results in Fig.7 demonstrate that the two methods can both achieve better results on the smooth BUS images (the 3rd and 4th rows). The method based on graph shortest path (GS) fails to handle the BUS images with too small or too large tumors, or poor quality with noise. Moreover, the maps generated by NC method are much smoother than that of GS method.

E. Parameter tuning
, and . As presented in Section Ⅱ , the detection framework has 4 major parts. The NC map or weighted map cannot always provide the correct information to generate the saliency map (see . It is very important to balance the effect of each part.
The values of , and are used to balance the influence of the adaptive center-bias term, weighted map and smooth term, respectively. It evaluates the performance of the proposed method with ranging from 0 to 200, ranging from 0 to 200, Original images GS map NC map   and ranging from 0 to 200 on the randomly selected subset containing 20 images. There are three stages to choose the parameters. The first stage, it makes the step size of three parameters be 40 and roughly obtain the range of each parameter which can achieve better P-R curve performance and MAE value if the P-R curve is similar. The second stage, the step size is 10. The step size is 2 in the third stage. As shown in Fig. 8, we obtain better P-R curve when is close to 10 , is close to 2 and is close to 80, respectively.

IV. CONCLUSION
In this paper, we propose a hybrid optimization framework for tumor saliency estimation, which models both the domainrelated knowledge and low-level visual clue. There are three main reasons that the proposed method can achieve much better results than traditional approaches: (1) the proposed framework integrates the weighted map, NC map and reference point, robust low-level image features; (2) the proposed method model TSE as an optimzation problem which can balance each part of energy function automatically for different images; (3) large amount of BUS images is used to choose the proper parameters.