{'fraction_non_alphanumeric': 0.033627653578808946, 'fraction_numerical': 0.014089798985534472, 'mean_word_length': 4.883289124668435, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Global Context-Aware Progressive Aggregation Network for Salient Object Detection

Deep convolutional neural networks have achieved competitive performance in salient object detection, in which how to learn effective and comprehensive features plays a critical role. Most of the previous works mainly adopted multiple level feature integration yet ignored the gap between different features. Besides, there also exists a dilution process of high-level features as they passed on the top-down pathway. To remedy these issues, we propose a novel network named GCPANet to effectively integrate low-level appearance features, high-level semantic features, and global context features through some progressive context-aware Feature Interweaved Aggregation (FIA) modules and generate the saliency map in a supervised way. Moreover, a Head Attention (HA) module is used to reduce information redundancy and enhance the top layers features by leveraging the spatial and channel-wise attention, and the Self Refinement (SR) module is utilized to further refine and heighten the input features. Furthermore, we design the Global Context Flow (GCF) module to generate the global context information at different stages, which aims to learn the relationship among different salient regions and alleviate the dilution effect of high-level features. Experimental results on six benchmark datasets demonstrate that the proposed approach outperforms the state-of-the-art methods both quantitatively and qualitatively.


Introduction
Salient object detection aims to detect interesting regions that attract human attention in an image (Cong et al. 2018a). As an efficient preprocessing technique, salient object detection benefits a wide range of applications such as image understanding (Zhang, Du, and Zhang 2014), image retrieval (Gao et al. 2015), and object tracking (Hong et al. 2015).
In recent years, the development of deep learning, especially the emergence of Fully Convolutional Network (Long, Shelhamer, and Darrell 2015), has greatly boosted the progress of salient object detection (Zhao et al. 2015;Li and Yu 2016;Wang et al. 2016). Fully Convolutional Network (FCN) stacks multiple convolution layers and pooling  (Liu, Han, and Yang 2018); (e) BASNet (Qin et al. 2019).
layers to gradually enlarge the receptive fields of network and extracts high-level semantic information. As pointed out in previous works (Luo et al. 2017;Zhang et al. 2017), due to the pyramid-like CNNs structure, low-level features usually have larger spatial size and more fine-grained details, while high-level features tend to gain more semantic knowledge and discard some meaningless or irrelevant detail information. Generally speaking, the high-level features are beneficial to the coarse localization of salient objects, whereas the low-level features that contain the spatial structural details are suitable to refine boundaries. However, there remains several problems for the FCN-based methods: (1) Due to the gap between different level features, the simple combination of semantic information and appearance information is insufficient and lacks consideration of the different contribution of different features for salient object detection; (2) Most of the previous works ignored the global context information, which benefits for deducing the relationship among multiple salient regions and producing more complete saliency result.
To remedy the above mentioned issues, we propose a novel network named Global Context-Aware Progressive Aggregation Network (GCPANet), which consists of four modules: Feature Interweaved Aggregation (FIA) module, Self Refinement (SR) module, Head Attention (HA) module, and Global Context Flow (GCF) module. Considering the characteristics difference between multiple level features, we design the FIA module to fully integrate the high-level semantic features, low-level detail features, and global context features, which is expected to suppress the noises but recover more structural and detail information. Before the first FIA module, we add a HA module on the top layer of the backbone to strengthen the spatial and channel-wise response on the salient object. After aggregation, features will be fed into a SR module to refine the feature maps via leveraging the inner characteristics within features. Taken into account that the context information can benefit for capturing the relationship among multiple salient objects or different parts of salient object, we design a GCF module to exploit the relationship from global perspective, which is conducive to improving the completeness of salient object detection. Besides, as pointed out in (Liu et al. 2019), the high-level features will be diluted as they passed on the topdown pathway. By introducing GCF, the features containing global semantics are delivered to feature maps at different stages, which alleviates the effect of features dilution. As shown in Fig. 1, the proposed method can handle some challenging scenarios, such as complex scene understanding (the high-luminance ceiling interference), or multiple objects relationship reasoning (the ping-pong bat and ball).
From the above, the contributions of our work can be summarized as follows: 1. A global context-aware progressive aggregation network is proposed to achieve saliency detection, which includes the Feature Interweaved Aggregation (FIA) module, the Self Refinement (SR) module, the Head Attention (HA) module, and the Global Context Flow (GCF) module. 2. The FIA module integrates the low-level detail information, high-level semantic information, and global context information in an interweaved way, where the global context information is produced by the GCF module to capture the relationship among different salient regions and improve the completeness of the generated saliency map. 3. Compared with 12 state-of-the-art methods on six public benchmark datasets, the proposed network GCPANet achieves best performance in quantitative and qualitative evaluations.

Related Work
In this section, we will review the related works on deep learning based salient object detection methods, which have achieved remarkable progress on saliency detection thanks to its powerful representation capability. Inspired by image semantic segmentation, Zhao et al. (Zhao et al. 2015) proposed a fully connected CNN to integrate local and global features to predict the saliency map. Wang et al. (Wang et al. 2016) adopted a recurrent CNN to refine the predicted saliency map step by step. For further enhance the saliency map, several recent works Zhang et al. 2017;Deng et al. 2018;Hu et al. 2018;Li et al. 2018;Zhang et al. 2018a; integrate features in multiple layers of CNN to exploit the context information at different semantic levels. Among them, Hou et al. ) introduced short connections to the skiplayer structure for capturing fine details. Zhang et al. (Zhang et al. 2017) concatenated multi-level feature maps based on multiple resolution and introduced a boundary refinement strategy. Deng et al. (Deng et al. 2018) proposed an iterative method to optimize the saliency map, leveraging features generated by deep and shallow layers. Hu et al. ) recurrently concatenated multi-layer features for saliency detection. Li et al. (Li et al. 2018) proposed a contour-to-saliency transferring method that simultaneously predict the contours and saliency maps. Zhang et al. (Zhang et al. 2018a) built a bi-directional message passing model for better integrating multi-level features. Zhang et al. (Zhang et al. 2018b) designed an attention guided network that selectively integrates multi-level contextual information in a progressive manner. Lately, Wu et al. (Wu, Su, and Huang 2019) proposed a cascade partial decoder that utilizes attention mechanism to refine high-level features. Qin et al. (Qin et al. 2019) proposed a boundary-aware model to segment salient object regions and predict the boundaries simultaneously. Liu et al. (Liu et al. 2019) extended the FPN structure equipped with pyramid pooling module to fuse the coarselevel semantic features and fine-level features.

Methodology
In this section, we first outline the proposed network. Then, we elucidate how each component made up and illustrate its effect for saliency detection.

Overview of the Proposed Network
As Fig. 2 shows, the proposed network is a symmetrical encoder-decoder architecture, where the encoder component is based on ResNet-50 to extract the multi-level features, and the decoder component progressively integrates the multilevel comprehensive features to generate the saliency map in a supervised way. Specifically, we first use a HA module to strengthen the spatial regions and feature channels with high response on salient objects, and a SR module to generate the first-stage high-level features through the feature refinement and enhancement. Then, we progressively cascade a FIA module and a SR module in three times to learn more discriminative features and generate more accurate saliency map. In the FIA module, the low-level detail information, high-level semantic information, and global context information are fused in an interweaved way. The SR module successive to each FIA module is to refine the coarse aggregation features. Note that, the global context information is produced by the proposed GCF module, which captures the relationship among different salient regions and constrains more complete saliency prediction. To facilitate the optimization, we combine auxiliary loss branches of each sub-stage with dominant loss.

Feature Interweaved Aggregation Module
As we all know, low-level features include more detail information, such as texture, boundary, and spatial structure, but they also contain more background noises. By contrast, high-level features can provide abstract semantic information, which is beneficial to locate the salient object and suppress the noises. Thus, these two level features are always 3) denote the low-level detail, high-level semantic, and global context features, respectively. combined together to generate the complementary features. In addition to these two level features, the global context information is very useful to infer the relationship among different salient objects or parts from the global perspective, which is conducive to generate more complete and accurate saliency map. Moreover, using the context features can alleviate the effect of feature dilution. Hence, we develop the FIA module to fully integrate these three level features, which in turn produces a discriminative and comprehensive feature with global perception. Specifically, as shown in Fig.  3, the FIA module receives three parts input, i.e., the highlevel features from the output of the previous layer, the lowlevel features from the corresponding bottom layer, and the global context feature generated by the GCF module. Note that, the production of global context feature will be introduced in the latter subsection.
We first introduce the aggregation strategy for high-level features and low-level features. Different from previous works (Qin et al. 2019;Liu et al. 2019) that often simply fuse the high-level features after up-sampling with the low-level features by concatenation or addition operation, we adopt a more aggressive yet efficient operation, i.e., multiplication. The multiplication operation can strengthen the response of salient objects, meanwhile suppress the background noises. Specifically, for the consistency of multiplication operation, the low-level feature maps f t l (t = 1, 2, 3) are firstly fed into a 1 × 1 convolution layer conv 1 , which compress the features to have the same number of channels as of the highlevel features f t h . Then, a 3 × 3 convolution layer is applied to high-level features f t h to obtain a semantic mask W t h after up-sampling. Further, we multiply the mask W t h to the compressed low-level featuresf t l . Besides, considering high-level features will discard some detail information relevant to salient objects, we apply the above fusion strategy in a mirror way. The mirror path different from the above mentioned is that a detail mask W t l is generated by lowlevel features through a 3 × 3 convolution layer and then, the mask W t l is multiplied to the high-level featuresf t h af- ter up-sampling. The mirror path is supposed to add finegrained detail information to the predicted saliency maps. The above process can be described as wheref t l = conv 1 (f t l ) denotes the compressed low-level features, denotes element-wise multiplication, δ denotes the ReLU activation function, upsample is the up-sampling operation via bilinear interpolation, and t is the stage index. Further, to model the relationship between different parts of salient objects and alleviate the dilution process of highlevel features, we introduce the global context features f t g at each stage. We employ the global context features f t g to generate a context mask W t g . Then, the mask W t g is multiplied conv conv conv f out f in Figure 4: Structure of the SR module. to the compressed low-level featuresf t l .
Finally, these three level features are concatenated and then passed through a 3 × 3 convolution layer to obtain the final fusion features: Each of the above mentioned convolution layers except conv 2 , conv 3 , and conv 4 is equipped with a batch normalization layer and the ReLU activation function. The output of FIA module is then passed to the SR module.

Self Refinement Module
In FIA module, we combine the complementary characteristics between different level features and obtain the comprehensive feature expression. As a simple and intuitionistic way, one can directly apply a softmax layer after FIA module to obtain the saliency maps, while it still exists some defects. For instance, there are some holes in the predicted salient objects, which are caused by the contradictory response of different layers. Hence, we develop a SR module to further refine and enhance the feature maps after passing the HA module and FIA modules by utilizing the multiplication and addition operation (see Fig. 4). In detail, we firstly apply a 3 × 3 convolution layer to squeeze the input features f in into feature vectorf with the channel dimension of 256, meanwhile remaining useful information. Then, the featuref is fed into two convolution layers to obtain the mask W and bias b for multiplication and addition operation. The main process can be described as where f out is the refined feature maps.

Head Attention Module
Since the top layers features of the encoder component usually are redundant for salient object detection, we design a HA module following the top layer to learn more selective and representative features by leveraging the spatial and channel-wise attention mechanisms. Specifically, we first apply a convolution layer to the input feature maps F to obtain a compressed feature representa-tionF with 256 channels. Then, we generate a mask W and bias b as similar as the way used in the SR module. The output of the first stage is obtained by Further, the input feature F is down-sampled into a channel-wise feature vector f through average pooling, which has strong consistency and invariance. Then, two successive fully connected layers f c 1 (·), f c 2 (·) are applied to project the feature vector f into an output vector y. The final output feature maps F out will be obtained via weighting with vector y. The second stage can be described as the following equations, where f c i (·) denotes i-th FC layers, δ denotes the ReLU activation function, σ is the sigmoid operation, and • denotes function composition.

Global Context Flow Module
For the challenging scenarios in salient object detection, such as cluttered background, foreground disturbance, and multiple salient objects, simple integration of high-level and low-level features may fail to completely detect the salient regions due to lacking the global semantic relationship among different parts of salient object or multiple salient objects. Besides, since the top-down pathway is built upon the bottom-up backbone, the high-level features will be gradually diluted as they are transmitted to lower layers.
To remedy these issues, we design the GCF module to capture the global context information embedded into the FIA module at each stage. Different from (Liu et al. 2019), we take into account the different contributions at different stages. We firstly employ global average pooling (Lin, Chen, and Yan 2013) to obtain the global contextual information and then reassign different weights to different channels of the global contextual feature maps for each stage. More specifically, for each stage, the process can be described as where f top refers to the top layer features, and f gap refers to the features generated by the top layer features through global average pooling, which includes global contextual information. Then, the output f t g is fed into the FIA module, which has been elaborated in the previous section.

Loss Function
In saliency detection, binary cross-entropy loss is often used as the loss function to measure the relation between the generated saliency map and the ground truth, whcih can be formulated as where H, W denote the height and width of the image, respectively, G ij is the ground truth label of the pixel (i, j), and S ij represents the corresponding probability of being salient objects in position (i, j). To facilitate the optimization of the proposed network, we add auxiliary loss at three decoder stages. Specifically, a 3 × 3 convolution operation is applied for each stage to squeeze the channel of the output feature maps to 1. Then these maps are up-sampled to the same size as the ground truth via bilinear interpolation and sigmoid function is used to normalize the predicted values into [0, 1]. The total loss consists of two parts, i.e., the dominant loss corresponding to the output and the auxiliary loss of each sub-stage.
where λ i denotes the weight of different loss, and dom , i aux denote the dominant and auxiliary loss, respectively. The auxiliary loss branches only exist during the training stage, whereas they are abandoned when inference.

Experiments
In this section, we first describe the implementation details, introduce the benchmark datasets, evaluation metrics. Then, we conduct experiments on these datasets to evaluate the effectiveness of the proposed method.

Implementation Details
We adopt ResNet-50 (He et al. 2016) pretrained on ImageNet (Deng et al. 2009) as our network backbone. In the training stage, we resize each image to 320×320 with random horizontal flipping, then randomly crop a patch with the size of 288 × 288 for training. During the inference stage, images are simply resized to 320 × 320 then fed into the network to obtain prediction without any other post-processing (e.g., CRF). We use Pytorch (Paszke et al. 2017) to implement our model. Mini-batch Stochastic gradient descent (SGD) is used to optimize the whole network with the batch size of 32, the momentum of 0.9, and the weight decay of 5e-4. We use the warm-up and linear decay strategies with the maximum learning rate 5e-3 for the backbone and 0.05 for other parts to train our model and stop training after 30 epochs. The inference of a 320 × 320 image takes about 0.02s (over 50 fps) with the acceleration of one NVIDIA Titan-Xp GPU card. The code is now available. 1 Datasets We conduct experiments on six public saliency detection benchmark datasets, and the detailed introduction is provided as follows: • ECSSD  consists of 1,000 natural images which are manually collected from the Internet; • DUT-OMRON (Yang et al. 2013) contains 5,168 high quality images. Images of this dataset have one or more salient objects and relatively cluttered background, and thus salient object detection on this dataset is very challenging; • SOD (Movahedi and Elder 2010) is composed of 300 images, many of which contain multiple objects either with low contrast or touching the image boundary; • DUTS ) is currently the largest saliency detection benchmark dataset, which consists of 10,553 training images (DUTS-TR) and 5,019 testing images (DUTS-TE).
As with other works in salient object detection (Qin et al. 2019;Liu et al. 2019), we employ DUTS-TR as our training dataset and evaluate our model on other datasets.
Evaluation Metrics To quantitatively evaluate the effectiveness of our proposed model, we adopt precision-recall (PR) curves , F-measure (F β ) score and curves, Mean Absolute Error (MAE), and structural similarity measure (S m ) as our performance measures. With different thresholds, pairs of precision and recall value can be computed by comparing the binarized map with the ground truth. Then, we can plot the precision-recall curve (Cong et al. 2019). The second metric F-measure score takes both precision and recall into account, which is defined as F β = (1+β 2 )·P recision·Recall where β 2 is set to 0.3 to emphasize the precision over recall, as suggested in the previous work (Cong et al. 2018b). Larger F-measure score indicates better performance. For precision-recall pairs, we calculated each corresponding Fmeasure score and choose the maximum as the evaluation score on the whole dataset. Another metric MAE is defined as the average pixel-wise absolute difference between the prediction map and the ground truth (Cong et al. 2017), i.e., M AE = 1 H×W H y=1 W x=1 |S(x, y) − G(x, y)| where S denotes the predicted saliency map, G indicates the corresponding ground truth, and H, W are the height and width of the saliency map respectively. The smaller MAE indicates better performance. Since the F β and MAE are based on pixel-wise errors and ignore the structural similarities, we adopt the structural similarity measure proposed by (Fan et al. 2017) as one of our metrics. The structural similarity measure is defined as S m = α * S o + (1 − α) * S r , where α is set to 0.5 to balance the object-aware structural similarity (S o ) and region-aware structural similarity (S r ).

Compared with the State-of-the-arts
We compare the proposed model with 12 state-of-the-art methods, including Amulet (Zhang et al. 2017), C2S (Li et al. 2018), RANet (Chen et al. 2018), PAGR (Zhang et al. 2018b), PiCANet-R (Liu, Han, and Yang 2018), DGRL , R 3 Net (Deng et al. 2018), BMPM (Zhang et al. 2018a), RADF , CPD-R (Wu, Su, and Huang 2019), BASNet (Qin et al. 2019) and Pool-Net (Liu et al. 2019). For fair comparison, the saliency maps of different methods are provided by authors or obtained by running their released codes under the default parameters. Table 1: Performance comparison with 12 state-of-the-art methods on 6 benchmark datasets. The best results on each dataset are highlighted in boldface. (Zhang et al. 2017) 0 Quantitative Evaluation Table 1 shows the quantitative comparison results in terms of F-measure, S-measure, and MAE score. It's obvious that the proposed method achieves the best performance in terms of different measures, which demonstrates the effectiveness of the proposed model. In addition, as shown in Fig. 5, the PR curves and F-measure curves by our approach (the red curves) are outstanding in most cases compared with other previous methods under different thresholds, which is consistent with the measures reported in Table 1.
Qualitative Evaluation To further illustrate the advantages of the proposed method, we provide some visual examples of different methods. As Fig. 6 shows, our proposed method can handle various challenging scenarios, including fine-grained structures, cluttered background, foreground disturbance, objects concurrency, and multiple salient objects, etc. Compared with other previous methods, the saliency maps generated by our approach are more complete and accurate. Note that our approach is more robust to background/foreground disturbance (the second/third row) and can capture the relationship among multiple objects (the fifth row), which illustrates the power of the feature interweaved aggregation strategy and the introducing of global context information.

Ablation Study
In this part, we conduct the ablation study to verify the effectiveness of each key components designed in the proposed model. The ablation experiments are conducted on the EC-SSD dataset and ResNet-50 is adopted as the backbone. As shown in Table 3, the proposed model containing all components (i.e., FIA, SR, HA and GCF) achieves the best performance, which demonstrates the necessity of each component for the proposed model to obtain the best saliency detection results. We adopt the model like U-Net (Ronneberger, Fischer, and Brox 2015) that only concatenates high-level features after up-sampling and low-level features as the baseline model, then add each module progressively. From Table 3, the FIA module largely improves the baseline from 0.0456 to 0.0390 in terms of MAE. Furthermore, the MAE score is improved by 14% compared with the basic model after Figure 6: Qualitative comparison of the proposed model with other state-of-the-art methods. Obviously, saliency maps generated by our approach are more accurate and much close to the ground truth in various challenging scenarios.  adding the SR module. The combination of FIA and SR has already achieved well performance, while the addition of HA has a slight enhancement. Finally, we add the GCF to the model and obtain the best result. Moreover, we evaluate the effectiveness of the GCF module compared to another setting, in which the global context features are shared at all stages. From Table 2, the proposed GCF module outperforms the shared one. The potential reason behind this phenomenon is that the parallel scheme of the GCF modules can provide distinct features for different stages, which benefits to learn the comprehensive and discriminative features for salient objects.

Conclusion
In this paper, we propose a Global Context-Aware Progressive Aggregation Network (GCPANet) to achieve salient object detection. Considering different characteristics of different level features, we design a simple yet effective aggregation module to fully integrate different level features. We introduce global context information at different stages to capture the relationship among multiple salient objects or multiple regions of salient object and alleviate the dilution effect of features. Experimental results on six benchmark datasets demonstrate that the proposed network outperforms other 12 state-of-the-art methods under different evaluation metrics.