{'fraction_non_alphanumeric': 0.03443334876441152, 'fraction_numerical': 0.011705575028106607, 'mean_word_length': 4.858711093891257, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 2, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Learning to Detect Instance-level Salient Objects Using Complementary Image Labels

Existing salient instance detection (SID) methods typically learn from pixel-level annotated datasets. In this paper, we present the first weakly-supervised approach to the SID problem. Although weak supervision has been considered in general saliency detection, it is mainly based on using class labels for object localization. However, it is non-trivial to use only class labels to learn instance-aware saliency information, as salient instances with high semantic affinities may not be easily separated by the labels. As the subitizing information provides an instant judgement on the number of salient items, it is naturally related to detecting salient instances and may help separate instances of the same class while grouping different parts of the same instance. Inspired by this observation, we propose to use class and subitizing labels as weak supervision for the SID problem. We propose a novel weakly-supervised network with three branches: a Saliency Detection Branch leveraging class consistency information to locate candidate objects; a Boundary Detection Branch exploiting class discrepancy information to delineate object boundaries; and a Centroid Detection Branch using subitizing information to detect salient instance centroids. This complementary information is then fused to produce a salient instance map. To facilitate the learning process, we further propose a progressive training scheme to reduce label noise and the corresponding noise learned by the model, via reciprocating the model with progressive salient instance prediction and model refreshing. Our extensive evaluations show that the proposed method plays favorably against carefully designed baseline methods adapted from related tasks.


Introduction
Salient Object Detection (SOD) is a long-standing vision task that aims to segment visually salient objects in a scene. It often serves as a core step for downstream vision tasks like video object segmentation [50], object proposal generation [4], and image cropping [49]. Recent deep learningbased SOD methods have achieved a significant performance progress [47,74,56,42,68,17,48], benefited from the powerful representation learning capability of neural networks and large-scale pixel-level annotated training data. Since annotating pixel-level labels is extremely tedious, there are some works [47,60] that aim to explore cheaper image-level labels (e.g., class labels) to train SOD models in a weakly-supervised manner.
Salient Instance Detection (SID) goes further from SOD as it aims to differentiate individual salient instances. This instance-level saliency information can benefit vision tasks that require fine-grained scene understanding, e.g., object rank [41], image captioning [21], image editing [59] and semantic segmentation [43]. However, existing SID methods [11,25,62] still rely on large-scale annotated ground truth masks in order to learn how to segment salient instances with their boundaries delineated. Hence, it is worthwhile studying the SID problem from the weakly-supervised perceptive by using cheaper image-level labels.
A straightforward solution to the weakly-supervised SID problem is to use class labels for training, like the weaklysupervised SOD methods [47,60]. However, using just class labels to learn a SID model is non-trivial for two reasons. First, while class labels can help detect semantically predominant regions [70], there is no guarantee that the detected regions are visually salient. Second, objects of the same class may not be easily distinguished due to their high semantic affinity. We observe that subitizing refers to the number of certain objects and is therefore naturally related to saliency instance detection. By predicting the number of salient objects, we may use it as a global supervision to help separate instances of the same class while clustering different parts of an instance with diverse appearances into one. Inspired by this insight, we propose to learn a weaklysupervised SID network (denoted as WSID-Net) using class and subitizing labels.
Our WSID-Net consists of three synergic branches: a salient object detection branch is proposed to locate candidate salient objects while a boundary detection branch is proposed to delineate their boundaries, both by exploiting semantics from the class labels; and a centroid detection branch is proposed to detect the centroid of each salient instance, by leveraging saliency cues from the subitizing labels. This information is fused to obtain the salient instance map. To facilitate the learning process, we propose a Progressive Training Scheme (PTS) to reduce the noise generated in our salient object detection branch (e.g., incomplete object proposals and cluttered background objects), by reciprocally updating the branch using generated pseudo labels and refreshing the branch in a self-supervised manner.
To demonstrate the effectiveness of the proposed model, we compare it with a variety of baselines adapted from related tasks on the standard benchmark [25].
To summarize, this work has four main contributions: -To the best of our knowledge, we propose the first weakly-supervised method for salient instance detection, which only requires image-level class and subitizing labels to obtain salient instance maps. -We propose a novel network (WSID-Net), with a novel centroid-based subitizing loss to exploit salient instance number, a novel Boundary Enhancement module to learn instance boundaries, and a novel Cross-layer Attention module to enhance cross-layer context feature learning of centroids and the boundaries. -We propose a novel Progressive Training Scheme, to facilitate the learning of the saliency detection branch by reducing the noise in a self-supervised manner. -We conduct extensive experiments to analyze the proposed method, and verify its superiority against baselines adapted from related state-of-the-art approaches.
2 Related Work

Salient Instance Detection
Existing SID methods are fully-supervised. Zhang et al. [62] propose to detect salient instances with bounding boxes, and propose a MAP-based optimization framework to regress a large amount of pre-defined bounding boxes into a compact number of instance-level bounding boxes of high confidences. However, this method based on bounding boxes cannot detect salient instances with accurately delineated boundaries. Other works predict pixel-wise masks for the detected salient instances, and typically rely on large amount of manually annotated ground truth labels. Specifically, Li et al. [25] propose to first predict the saliency mask and instance-aware saliency contour, and then apply the Multiscale Combinatorial Grouping (MCG) algorithm [5] to extract instance-level masks. Fan et al. [11] propose an endto-end SID network based on the object detection model FPN [28], with a segmentation branch to segment the salient instances. Unlike these existing SID methods, we propose in this paper to train a weakly-supervised network, which only requires image-level class and subitizing labels.
The work presented in this paper extends our BMVC oral paper [45] in three aspects. First, we provide a more comprehensive literature survey on the weakly supervised salient instance detection task and other relevant works. Second, we note that the earlier method [45] typically suffers from the salient instance incompleteness problem, due to the noise generated in both salient object detection and boundary detection branches. To address this problem, we propose a Cross-layer Attention module here to learn boundary and centroid features, and a self-supervised Progressive Training Scheme to reduce the noise in the salient object detection branch. Third, we perform more experiments to analyze the properties of our method and show its effectiveness over existing state-of-the-art approaches.

Salient Object Detection
SOD methods aim at detecting salient objects in a scene without differentiating the detected instances. Liu et al. [30] formulate the SOD task as a binary segmentation problem for segmenting out the visually conspicuous objects of an image via color and contrast histogram based priors.
Traditional methods propose to leverage different handcrafted priors to detect salient objects, e.g., image colors and luminance [1], global and local contrast priors [37,8], and background geometric distance prior [57]. Recently, deep learning based SOD methods achieve superior performances on the standard SOD benchmarks [57,27,39,19,47,8]. Among them, several methods explore boundary information for salient object detection. Xu et al. [56] propose a CRF-based architecture to refine boundaries of both deep features and saliency maps in a coarse-to-fine manner. Some methods [74,71,53,42] propose to formulate saliency and edge detection with two network branches as multi-task learning. Feature fusion strategies have also been widely explored in salient object detection. DSS [17], DGRL [48], and MINet [35] integrate multi-level features in the top-down direction. In GBMPM [66] and PAGE-Net [51], multi-level saliency features are fused in both top-down and bottomup directions, to detect salient objects of varying scales. F3Net [52] and PFPN [46] propose to fuse features progressively, to enrich saliency features with recurrent feedback information. Attention mechanism has also been exploited to reweigh multi-scale features in order to suppress noise and enhance context learning, via dynamic weight decay scheme [13], mutual relation learning of object parts [6], gate-based interference control [69], and spatial-/channelwise attentions on different features [68]. In particular, He et al. [16] propose to leverage numerical representation of subitizing to enrich spatial representations of salient objects. These methods are typically benefited from the powerful learning ability of deep neural networks as well as largescale annotated ground truth data.
To alleviate the data annotation efforts, many weaklysupervised SOD methods are proposed, by investigating different approaches of generating pseudo saliency labels. A method leverage subitizing information alone for refining saliency prediction Some methods [61,65,31] propose to use traditional SOD methods to generate pseudo labels for training deep saliency models. Some other methods [47,60] propose to train weakly-supervised deep models using object class labels and class activation maps (CAMs) [70]. There are also some methods [26,64] that propose to combine pre-trained contour networks with segment propos-als [26] or scribbles [64] to generate pseudo labels for training saliency detection networks.
However, existing weakly-supervised SOD methods cannot be directly applied to our problem, as class labels along do not provide instance-level information. In this paper, we propose to use class and subitizing labels to train our SID model.

Noise Reduction
Noise commonly exists in a weakly-supervised setting, typically when the task is a pixel-level prediction and the supervision is provided at the image-level. Existing methods typically rely on auxiliary full-annotated labels (referred to as clean labels) or pre-trained models to regularize the noise. Hu et al. [18] formulate it as a multi-task learning problem, in which the networks trained on a small set of clean labels can help regularize the noise in the networks trained on a large set of weak labels. Zhang et al. [63] propose a noise-aware method for learning a disentangled clean saliency detector from noisy labels. Lu et al. [32] propose a sparse learning model to learn the noise statistics from oversegmented superpixels, while Zhu et al. [73] propose to filter out noisy segment proposals with low matching scores. Both methods rely on additional pre-trained models for noise reduction.
Unlike the above methods, we do not leverage clean labels or pre-trained proposals as assistances. We achieve this goal by training our salient object detection branch in a progressive manner, via model refreshing and pseudo label regeneration.

Methodology
Class labels are widely explored in weakly-supervised SOD methods for learning to localize candidate objects, based on the pixel-level semantic affinity derived from the network responses to the class labels. However, class labels lack instance-level information, causing over-and underdetection when salient instances are from the same category. We note that subitizing, which is a cheap image-level label that denotes the number of salient instances of a scene, can serve as a complementary supervision to the class labels to provide instance-related information. Hence, we propose to use both class and subitizing labels to address our weaklysupervised SID problem. To this end, we propose a weaklysupervised SID network (WSID-Net), as shown in Figure 2.
The proposed WSID-Net has three branches: Finally, we propose a novel Cross-layer Attention (CA) module for the Centroid Detection Branch and Boundary Detection Branch to learn the context information for detecting centroids and boundaries, respectively.

Centroid Detection Branch
Detecting object centroids is crucial to separating object instances in a weakly-supervised scheme. Unlike existing semantic (instance) segmentation methods [2,34,72,9,73,24] that detect the centroids based on network responses to the class labels, we propose to introduce subitizing information to explicitly supervise the salient centroid detection process.

Centroid-based Subitizing Loss
It has been shown that penalizing the centroid loss [2,34] helps cluster local pixels with high semantic affinities. However, this typically fails when salient instances from the same object category have varying shapes and appearances. The reason is that the clustering process of local pixels lacks global saliency supervision. Hence, we introduce the centroid-based subitizing loss L SU to resolve this problem. We use subitizing to explicitly supervise the number of predicted centroids, which is implicitly related to the learned offset vectors as L su is back-propagated to the centroid detection branch during training, to guide the centroid-aware pixel clustering process. The detailed formulation is discussed below. The Centroid Detection Branch predicts an offset vector map V ∈ R W ×H×2 , where W × H denotes the spatial size of the map, and each 2D vector v i ∈ V indicates the vertical and horizontal distances of the i th pixel from its associated instance centroid. We follow [2] to iteratively derive V as: where p i is the coordinates of the i th pixel, m is the iteration number, and v m i + p i indexes the the current centroid that the offset of the i th pixel points to. Ideally, Eq. 1 would converge within a few iterations when v m+1 i = v m i and the offset of the centroid is zero, and yields a set of centroids that represent the instances. Pixel i can then be assigned to its corresponding centroid c n by measuring its distance from the centroid, as described by: We then use the saliency map S of the saliency detection branch to filter out non-salient instances by computing their IoU = (SI n ∩ S)/SI n > θ, to obtain a set of saliency instances SI * = {SI 1 , SI 2 , ..., SI T * }, where T * represents the number of predicted salient instances. Finally, we use MSE to measure L su as: where T is the subitizing ground truth, and T * denotes the number of predicted centroids extracted from the offset vectors of the pixels in the salient region. Note that the loss L su is back-propagated to update the offset vectors only in the salient region, which avoids the learning process of instance centroid detection being distracted by the non-salient background. The gradient δ of L su is calculated as: where V * are the offset vectors in the salient region, and K is the total number of offset vectors in V * . Figure 3 visualizes the results from centroid detection and the corresponding instance segmentation, with and without using the centroid-based subitizing L SU loss function. We can see that the network groups the two dogs into one when not using L SU , as these two dogs have similar appearances and lie next to each other ( Figure 3(b,e)). By introducing L SU , the network is able to predict a correct number of centroids, and generate reasonable salient instance masks compared with the ground truth ( Figure 3(c,f)).

Network Structure
We adopt the image-to-image translation scheme, where our network outputs a 2D centroid map, in which the value of each pixel location indicates the offset vector to its instance centroid. The bottom part of Figure 2 shows the network structure of our centroid detection branch. Given an input image, we first extract multi-scale backbone features f 1 to f 5 and feed them to the Cross-layer Attention (CA) modules with boundary-aware features for joint refinement (to be discussed in Section 3.3). We then fuse the high-level features to obtain f h : f h = Conv(Concat(f 4 * , f 5 * )), which is further fused with the low-level features to produce the centroid Input Image: Fig. 2: Pipeline overview. Our SID model is trained using only image-level class and subitizing labels. It has three synergic branches: (1) a Boundary Detection Branch for detecting object boundaries using class discrepancy information; (2) a Saliency Detection Branch for detecting objects using class consistency information; and (3) a Centroid Detection Branch for detecting salient instance centroids using subitizing information. A random walk method is further applied to fuse these information to obtain a final salient instance mask.
(a) input image Fig. 3: Visualization of the centroid detection branch with and without L SU . Using class labels alone fails to train the network to detect instance centroids (denoted by red stars) if they have similar appearances (b), resulting in wrong segmentations (e). In contrast, our proposed subitizing loss can segment these salient objects in instance-level (f), by learning to identify the correct number of salient instances (c).

Boundary Detection Branch
Boundaries provide strong cues for separating salient instances. Unlike fully-supervised SID methods that learn boundary-aware information based on pixel-level ground truth masks, we propose the Boundary Enhancement module to leverage the Canny prior [20] to delineate continuous instance boundaries.

Boundary Enhancement (BE) Module
We apply a random walk algorithm to search a salient instance from a centroid to its boundary. However, it may fail when part of the boundary is discontinuous as the random walk algorithm will also search the region outside the boundary. Hence, we propose the BE module to incorporate the edge prior for learning continuous instance boundaries, as shown in Figure 4. Specifically, we first extract low-level features along the horizontal and vertical directions from the input image, by two 1 × 7 and 7 × 1 convolution layers. These low-level features are then fed into three Residual Blocks [15] for feature refinement, which are further concatenated with enriched edges computed from the Canny operator [20]. To compute the final enriched boundary features, another 1×1 convolution layer is applied.   visualizes two examples of boundary detection and the corresponding salient instance detection with and without the BE module. We can see that our BE module helps detect the boundaries between objects, which is crucial to salient instance segmentation. 5: Visualization of the boundary detection branch with and without the BE module, which shows the effectiveness of our proposed BE module in mining continuous boundary information for separating salient objects of same classes.

Network Structure
The top part of Figure 2 shows the architecture of the boundary detection branch. Given an input image I, the backbone network produces multi-scale features (f 1 to f 5 ), each of which is enhanced by a CA module (to be discussed in Section 3.3) before they are concatenated and computed to predict the boundary map. We also feed the input image into the BE module to obtain enhanced edge features f b . The output boundary map B is then computed as: is the sigmoid activation function.

Cross-layer Attention (CA) Module
Detecting instance centroids and boundaries are two highly coupled sub-tasks, i.e., they can influence each other and further affect the SID performance. To effectively learn these two sub-tasks, we propose the Cross-layer Attention (CA) module for refining backbone features before they are used for these two sub-tasks. Its design is based on two observations. First, low-level features contain high-resolution but noisy information for delineating salient instance boundaries, while high-level features have low-resolution but robust information for salient instance localization. Second, since salient instances may have various shapes and they may correspond to different class labels, we need to model both long-range spatial and cross-channel contextual information. Unlike existing dual attention mechanisms [54,12] that only enhance the feature representation capacity of one fixed layer, our CA module first incorporates a Cross-layer Feature Mixing (CFM) unit to enhance the communication across different levels of backbone features and then uses multiple Mutual Attention (MA) units to learn hierarchical channel-wise and spatial-wise attentive features for each sub-task. The CFM unit shares its parameters to allow information exchanges across the boundary and centroid branches. Figure 6 shows the module structure.  We first upsample f 2 , f 3 , f 4 , and f 5 such that the given feature maps have the same resolution, and apply 1×1 convolution on the five feature maps such that they have the same channel depth (256). We get features f 1 to f 5 that have the same shape for the following operation. We then apply CFM on the pyramid features to generate cross-layer features. CFM is implemented via a concatenation-splitconcatenation operation on the feature channels. We concatenate features f 1 to f 5 as f c , with 1280 (256×5) channels. The split-concatenation operation could be considered as a reshape-transpose-reshape process. We reshape channel dimension of f c to 2 dimensions (i.e., [5,256] Figure 8 shows the structure of our MA unit. The top and bottom branches are channel-wise and spatial-wise attention blocks, respectively. Specifically, given the input fea-tures f n , we compute the channel-wise attention features F c as:

Structure of MA Unit
where M axP ool c and AvgP ool c denote two channel-wise pooling operations, and MLP is the multi-layer perception with one hidden layer to generate the attention features. We also compute the spatial-wise attention features F s as: where Conv 7×7 is a convolutional layer with kernel size 7.
The final attention features f n * are then computed as: where × denotes the dot product operation, and + is the element-wise summation operation.

Progressive Training Scheme (PTS)
Weak annotated labels would inevitably introduce noise into the learning process. To reduce the noise, previous works propose to use temporal ensemble learning [23,44] in a semi-supervised setting, where latent knowledge learned from labeled data can be applied to noisy unlabeled data. We extend this idea in our weakly-supervised setting. Since we do not have fully-annotated data, we explore this ensemble learning strategy in a progressive self-supervised manner, i.e., by reciprocally training the salient object detection branch using newly predicted salient instance maps. It mainly contains two iterative steps: pseudo label generation, and model refreshing. Figure 10 shows the overview of the proposed Progressive Training Scheme, and Algorithm 1 summarizes the main steps.

Pseudo Label Generation
Since we do not have any fully-annotated labels to learn a noise-free representation, we propose to use our WSID-Net as a pseudo-label generator and refine its salient object detection branch in a self-supervised manner. This is because the output of our WSID-Net has more accurate boundaries with the help of other two branches, compared with its salient object detection branch. On the other hand, the retrained saliency detection branch can further boost the performance of our WSID-Net due to the improved salient object localization. To this end, we first forward the WSID-Net to generate salient instance masks, and use them as the initial pseudo training labels to update the network parameters of the salient object detection branch. In the next training iteration, we update the pseudo training labels using the retrained WSID-Net.

Model Refreshing
Before we update the pseudo training labels in the next iteration, we need to refresh our salient object detection branch. Note that we do not have any clean data (fully-annotated labels in our case) that can be used for learning noise-free features. It is possible that our model may overfit the noise in the pseudo labels and converge to a local minimum. To avoid these problems, we adopt the Exponential Moving Average method to update the model parameters with a weighted sum of model parameters in the current and former iterations. We define ω r as the model weight in iteration r, and the model refreshing is formulated as: where α is the smoothing hyper-parameter that balances the contributions of model parameters from different iterations. Input: Initial pseudo label ρ 1 , initial model weight ω 1 , training iterations R, and training epochs E per iteration Output: Final trained model weight ω f 1: for r = 1 to R do 2: for e = 1 to E do 3: update model weight ω r via backpropagating gradients and learning from pseudo labels ρ r 4: end for 5: if r > 2 then 6: update model weight ω r using Eq. 8 7: end if 8: generate pseudo label ρ r+1 using WSID-Net, where the saliency branch uses model weight ω r , and further uses CRF to refine the boundaries of the pseudo labels 9: end for 10: ω f = ω R : Illustration of how the proposed PTS benefits our task progressively in a self-supervised manner. While WSID-Net initially fails due to inaccurate detection (e), the output of WSID-Net (f) is still better than (e) with the help of the centroid (c) and boundary (d) detections. By using (f) to further refine the saliency detection branch (g), our WSID-Net could also enjoy the performance boost (h). Figure 11 shows one example of how the proposed PTS benefits our task in a self-supervised manner: given a challenging input image of two persons leaning on each other (a), our saliency detection branch fails to detect the majority of the salient regions due to the diverse foreground appearance and cluttered background (e), which further causes the failure of our WSID-Net (f). However, since (f) is still better than (e) due to the help of the centroid and boundary detections ((c) and (d)), after we have applied the proposed PTS training strategy, we can see that our saliency detection branch enjoys a better performance in detecting the saliency region as a whole (g), leading to an accurate salient instance detection result (h).

Implementation Details
We implement WSID-Net on the Pytorch framework [36]. Both training and testing are performed on a PC with an i7 4GHz CPU and a GTX 1080Ti GPU. CRF is used to generate or refine pseudo labels. The hyper parameters of CRF are set as w 1 = 4.0, w 2 = 3.0, σ α = 49.0, σ β = 5.0, and σ γ = 3.0. We choose ResNet50 as the backbone for all three branches in WSID-Net. The backbone are initialized as in [40]. Input images are resized to 512×512 resolution. To minimize the loss function, we use the SGD optimizer with batch size 6 and initial learning rate 0.01. The learning rate decreases following poly policy (lr itr = lr init (1 − itr maxitr ) γ ). We train our WSID-Net for 5 epoches. The proposed PTS training is further applied to refine the saliency detection branch for another 6 iterations, of which each iteration contains 8 epoches (R and E in Algorithm 1). α in Eq. 8 is set to r r+1 , where r is the current iteration index. The learning rate begins with 0.0001, and the decay follows the aforementioned poly policy.

Training and Evaluation Details
Datasets. Our WSID-Net is trained on two kinds of imagelevel labels, class and subitizing. We use the class labels from the PASCAL VOC 2012 [10] dataset (which is originally proposed for semantic and instance segmentation) to train our network. For subitizing labels, we count the numbers of salient instances from the ILSO [25] dataset, and train our network on this training set. For the proposed progressive training scheme, we augment the training data by using the unlabeled training data from C2SNet [26]. For testing, following the existing fully-supervised SID method [11], we perform SID evaluations on the test set of ILSO [25].
Training and Inference. We train the boundary and centroid branches together with different losses, and train the Table 1: Quantitative evaluation of our method against six baseline methods and state-of-the-art fully-supervised SID methods. For the compared methods, we show their original tasks, supervision types, training labels and auxiliary pre-trained models in the 2nd to 5th columns. SID, SOD, OD, and SIS represent salient instance detection, salient object detection, object detection and semantic instance segmentation, respectively. FS and WS denote Fully-Supervised and Weakly-Supervised. Best performances among the weakly-supervised methods are marked in red.

Methods
Original task saliency branch independently. We train the centroid detection branch using the proposed centroid-based subitizing loss together with the centroid loss introduced in [2,34]. We train the boundary detection branch using the boundary loss introduced in [3,2]. To train the saliency detection branch, we follow existing weakly-supervised SOD methods to use pseudo masks derived from class labels. Specifically, we first compute class activation maps via [70]. We then feed these maps together with the input image to a Conditional Random Field (CRF) [22] to generate pseudo object maps, and use these pixel-level pseudo labels to train the saliency detection branch. We further utilize the proposed PTS with model refreshing and self-generated pseudo labels to retrain the saliency detection branch. During inference, given an input image, WSID-Net first computes the centroids, boundaries, and saliency maps. We first obtain the initial saliency instance map SI * via the saliency map and centroid map, as discussed in Section 3.1.1. We then use the boundary map to refine the initial saliency instance map with the random walk algorithm. The transition probability matrix M is defined as: where H is the affinity matrix of the learned boundary map B, and D is a diagonal matrix relating to H. The element in H is defined as: of pixels on the line between boundary pixels x i and x j . In addition, H χ is the self production of H with power χ for affinity distillation, and D's diagonal element D ii equals to h χ ij for summarizing values of H χ by row. The random walk algorithm for instance-wise saliency value propagation is conducted as: where vec() refers to the vectorization of the matrix, and SI * n is our final saliency instance map.
Evaluation Metrics. We use the mean Average Precision (mAP) metric [14] to evaluate the SID performance. The IoU is set to 0.5 and 0.7 for this metric.

Comparing to the State-of-the-art Methods
As we are the first to propose a weakly-supervised SID method, we compare our method to 2 existing fullysupervised state-of-the-art SID methods: S4Net [11] and MSRNet [25]. We also prepare the following baselines from related tasks for evaluation. We choose 6 state-of-theart weakly-supervised methods, with two from the SOD task C2SNet [26] and NLDF [33]; one from the SID task MAP [62]; one from the object detection (OD) task, Deep-Mask [38]; and two from the Semantic Instance Segmentation task, PRM+D [9] and IRN [2]. We adapt them by adding different post-processing strategies to these methods for deriving instance-level saliency maps from their original outputs, or modifying their networks and retrain them using our training data. Details are summarized as follows: -We choose "MCG [5] + MAP [62]" as the postprocessing strategy for the weakly-supervised SOD methods (i.e., C2SNet [26] and NLDF [33]), inspired by the fully-supervised SID method MSRNet [25]. It has been shown in [25] that MCG [5] can be used to produce segments given the contour maps as input and assign these segments with confidence scores. Segments with low confidences can then be filtered out by MAP [62].
Considering that both C2SNet [26] and NLDF [33] learn to produce contour maps, we find this post-process strategy suitable for weakly-supervised SOD methods with contour predictions. -We select CRF [22] as the post-processing strategy for fully-supervised bounding-box-based SID method MAP [62], due to the fact that CRF is a popular graphical model used as post-processing for boosting segmentation performance. Considering that MAP [62] can generate instance-level bounding boxes, CRF can be used to obtain instance maps by refining the boundaries, which gives a performance boost of 3%. -We choose a weakly-supervised SOD method as postprocessing for filtering out non-salient segments produced by DeepMask [38], as DeepMask [38] is a classagnostic object detection method that is not aware of saliency information. We choose WSS [47] as the weakly-supervised SOD method for a fair comparison, as it performs closely to our Saliency Detection Branch in our preliminary experiment. This strategy improves the performance by 5%. -IRN [2] produces class-specific instance segmentation maps, which do not have saliency information. To adapt its results from class-specific to class-agnostic, we remove the CAM in their method and directly use their centroid and boundary maps to obtain instance maps. We then utilize WSS [47] to select salient instances. -PRM+D [9] is trained with class and per-class subitizing labels to predict semantic instance maps. However, this method can only response to the instances with predefined class labels. To adapt it to class-agnostic, we merge its per-class outputs (originally 20 output maps for 20 classes) into one class-agnostic map by adding an additional convolutional layer, and then retrain it using our training data.
Quantitative Comparisons. We quantitatively evaluate our method in Table 1 † . It is worth noting that our method achieves a significantly better performance of about 20% over the second-place weakly-supervised baseline, on the mAP@0.7 metric (which is very challenging as it requires the IoU value to be over 70%). These results show that our method achieves the best performance using just two imagelevel labels. † As of today, the codes for MSRNet [25] are still not available. Following [11], we directly copy the numbers reported in [25] to our submission for a quantitative comparison.
Qualitative Comparisons. We further qualitatively evaluate our method with fully-supervised methods and baselines in Figure 12. The visual results verify that our method is able to delineate the instance boundaries clearly, and output accurate numbers of segmented salient instances directly for different scenes, i.e. scenes with single instances, small instances, (non-)adjacent instances, similar/varied instances, and cluttered contents. In contrast, the compared methods exhibit different limitations as follows: -PRM+D and IRN fail to detect integral instances with inferior detected boundaries (e.g., rows 1, 13). -C2SNet and NLDF tend to recognize texture boundaries, causing fragmented instances (e.g., rows 10, 12, 14). -DeepMask and S4Net suffer from the over-detection problem, as they fail to distinguish instance proposals belonging to the same instance (e.g., rows 2, 3, 13). -MAP is a bounding-box based method. It fails to get clear instance boundaries even post-processed by CRF (e.g., rows 1, 2, 7).
Overall, our method outperforms all these baselines, as a result of the centroid-based subitizing loss, the carefully designed BE and CA modules, and the progress training scheme.

Ablation Study of Network Design
We begin by investigating the effectiveness of the proposed network design, including the proposed Boundary Enhancement module, Cross-layer Attention module, Progressive Training Scheme, and L SU loss. Table 2 shows the results. We can see that the SID performance would drop if we remove any of the components from the network. This shows that these components can help boost the performances of the saliency, centroid and boundary detection sub-tasks, which play a vital role in detecting salient instances. Figures 3, 5, and 9 provide additional visual comparisons to demonstrate the effectiveness of these components.  Fig. 12: Qualitative results of our method, compared with existing fully-supervised methods (S4Net [11] and MAP [62]) and modified baselines (PRM+D [9], DeepMask [38], C2SNet [26], NLDF [33], and IRN [2]). Refer to Section 4.3 and Table 1 on how we modify and train these baselines, in order to carry out a fair comparison.

Evaluation of the CA Module
We then investigate our CA module on its design choices and intermediate feature visualization. Design Choices: we examine our CA module against its variants, as reported in Table 3. First, we study the benefits of the CFM and MA units. Row 1 shows that removing the CFM unit leads to a performance drop, due to the reduction in communication enhancement among the pyramid feature layers. Row 2 also shows a performance drop without the MA unit, as we are not able to learn long-range dependencies. Second, we study the connection styles (parallel and cascade) of the attention mechanisms in the MA unit. Rows 3, 4 and 5 show that parallel connection of the spatial-and channel-wise attentions performs better than the cascade one. This may be because cascade connection may lose the learned context information of the former attention mechanism. Feature Visualization: we visualize multi-level intermediate features learned by our CA module in Figure 13. Given multi-level backbone features f 1 ∼ f 5 (1 st row), the CFM unit in the CA module first generates multilevel mixed context features f * 1 ∼ f * 5 (2 nd row), which are then used for learning boundary features f * 1⇒B ∼ f * 5⇒B (3 rd row) and centroid features f * 1⇒V ∼ f * 5⇒V (4 th row), respectively. First, we can see that our CFM unit is able to highlight the salient objects via aggregating multilevel backbone features, as shown in row 2. Second, the boundary-aware feature maps have high responses in different regions as shown in row 3, which suggests that determining the instance boundaries also require multi-level features. Third, in row 4, the visualization of centroid-aware features generally corresponds to the centroid map V, where the instance boundaries are generally highlighted, and pixel values of the centroid locations are close to zero. Overall, our proposed CA module is able to help adapt the backbone features into different task-specific features.
In addition, our CA module differs from CAM in two aspects. First, CAM is conditioned on the class label input, but our CA module learns class-agnostic attentions from pseudo labels. Second, CAM is unable to delineate clear boundary and instance information, while our CA module can learn this information to complement the CAM for detecting salient instances, as shown in Figure 14.

Evaluation of the BE Module
We conduct ablation studies to investigate the effect of the Canny filter in the Boundary Enhancement (BE) module. We compare our BE module to two ablated versions: removing the Canny filter from the BE module (denoted as BE Module w/o Canny), and using Canny filter only (denoted as Canny Only). Results are shown in Table 4. We can see that our method outperforms both ablated versions. The Canny filter is used to enrich the high-level boundary features with low-level edge information. Without the Canny filter, the BE module may not detect small boundaries accurately. However, relying only on the Canny filter cannot obtain high-level boundary information, which typically leads to over-segmentation of instances.

Evaluation of Parameter settings for the Canny Filter
We empirically set the thresholds (i.e., θ up and θ low for controlling the connectivity and density of the detected edges) in the Canny operator to be automatically determined by the channel median of the gray-scale image. We find that this works well in our experiments. To further investigate how these two thresholds affect the performance, we provide both qualitative and quantitative comparisons between our threshold choice with two manual choices described below: -Large range: we manually set θ low and θ up to 30 and 200, respectively, so that the Canny operator is sensitive to textures and can detect more edges. -Small range with large values: we manually set θ low and θ up to 230 and 260, respectively, so that only structural edges of objects can be detected. Table 5 shows that both manual strategies would degrade the performance. Figures 15 and 16 show two scenes that these manual strategies fail. In row 2 of Figure 15, if the Canny edge map provides insufficient object structure information and the learned boundaries are partially weak, our Fig. 13 Figure 15, and rows 2 and 3 in Figure 16. This visually verifies that the Canny edge generated under our automatic setting is more stable to provide pleasant instance boundaries.

Evaluation of PTS
We study how PTS helps improve the saliency detection performance iteration by iteration. Figure 17 and 18 show the progressively improving results over six training iterations.
(a) input:  Intermediate results in both figures verity that our PTS does not only penalize the background distraction but also recover the integral objects. Overall, our PTS is able to reduce noise and improve the performance in a self-supervised manner.

Evaluation of the Saliency Detection Branch
Since our WSID-Net relies on the performance of the saliency detection branch in detecting salient objects, we are particularly interested in the question of to what extent that the quality of the saliency object detection maps may affect the SID performance. To answer this question, we replace the outputs of our weakly-supervised salient object detection branch with five state-of-the-art full-supervised SOD methods (i.e., DSS [48], PiCANet [29], EGNet [67], ITSD [71], and SCRN [55]), as well as the ground truth saliency maps, to generate the salient instance maps. Results are reported in Table 6. We can see that the performance generally increases when inferring salient instance masks using the fully-supervised SOD results. This is because the fullysupervised methods are more robust to background distractions and able to delineate full object masks. However, we can still observe that the performance would not be saturated even if we feed the ground truth saliency maps to generate the SID maps. This is because the instance boundaries are still very difficult to detect, especially when these salient instances overlap each other. This suggests that developing an effective method for detecting salient instance boundaries in a weakly-supervised setting would be a promising solution. We can see that the detected objects are becoming more and more complete. Table 6: Investigation on how the SID performance is affected by the quality of the SOD maps. We show the performances by replacing the saliency maps (denoted Salmap) predicted by our saliency detection branch with saliency maps computed by different fully-supervised state-of-the-art SOD methods.

Evaluation of CRF in the Saliency Branch
CRF is used to produce pseudo ground truth saliency maps given the coarse CAM activation maps, so that the saliency detection branch can learn more accurate boundary information. Figure 19 shows that CRF helps produce more accurate pseudo ground truth saliency maps. We also provide quantitative results in Table 7, from which we can see that the performance drops without CRF refinement.  Fig. 19: Visualization of the effect of CRF on refining the coarse CAM.

Conclusion and Future Work
In this paper, we have proposed the first weakly-supervised SID method, called WSID-Net, which is trained on class and subitizing labels. Our WSID-Net learns to predict object boundaries, instance centroids, and salient regions. By using the proposed Boundary Enhancement module, Crosslayer Attention module, Progressive Training Scheme, and centroid-based subitizing loss, our method can identify and segment salient instances effectively. Both quantitative and qualitative experiments demonstrate the effectiveness of the proposed method compared with baseline methods. Our method does have its limitation. It may fail when the images are taken with improper exposures. Therefore, our method cannot detect salient objects/instances with low contrast to their surroundings. As a future work, we are currently exploring the use of a discriminative network of generative adversarial learning to overcome this limitation. We would also like to extend this work for videos.