{'fraction_non_alphanumeric': 0.03312659090094289, 'fraction_numerical': 0.012870612698526513, 'mean_word_length': 4.651134521880065, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Multi-Miner: Object-Adaptive Region Mining for Weakly-Supervised Semantic Segmentation

Object region mining is a critical step for weakly-supervised semantic segmentation. Most recent methods mine the object regions by expanding the seed regions localized by class activation maps. They generally do not consider the sizes of objects and apply a monotonous procedure to mining all the object regions. Thus their mined regions are often insufficient in number and scale for large objects, and on the other hand easily contaminated by surrounding backgrounds for small objects. In this paper, we propose a novel multi-miner framework to perform a region mining process that adapts to diverse object sizes and is thus able to mine more integral and finer object regions. Specifically, our multi-miner leverages a parallel modulator to check whether there are remaining object regions for each single object, and guide a category-aware generator to mine the regions of each object independently. In this way, the multi-miner adaptively takes more steps for large objects and fewer steps for small objects. Experiment results demonstrate that the multi-miner offers better region mining results and helps achieve better segmentation performance than state-of-the-art weakly-supervised semantic segmentation methods.


Introduction
Weakly-supervised semantic segmentation with imagelevel supervision is widely studied to relieve the scarcity of pixel-level annotations. Object region mining is the key step for recent weakly-supervised semantic segmentation methods [29,11,30,17,16], which aims to expand the sparse seed object regions localized by class activation maps [32,25].
In the existing weakly-supervised semantic segmentation methods, the regions of all the objects are mined in a monotonous manner, with a pre-fixed number of erasing steps [29,11,17] or randomly selected hidden units [16],  Our method adaptively takes more steps for large objects and fewer steps for small objects. (b) Comparisons among regions mined by our method and previous methods [30,11], and the original seed regions localized by CAM [32]. Our mined regions are more integral and finer. Better viewed in color.
etc. However, as the size of object regions varies for different objects in different images, the optimal number of region mining steps also differs. It can be observed that the mined regions of existing methods are often insufficient in number and scale for large objects and for small ones tend to be contaminated by surrounding backgrounds.
In this paper, we propose a novel multi-miner framework that can perform a region mining process fully adaptive to every single object. As shown in Fig. 1, our method adaptively takes different region mining steps for different objects, and thus offers more integral and finer region mining results than existing methods.
The key to such object-level adaptability is to automatically stop mining for an object when all of its regions are mined, and continue mining for an object whose regions are not completely mined. To achieve this, we leverage a parallel modulator, which can be simply implemented as a multi-label classifier, to control the region mining process by checking whether there are remaining object regions for each single object in an input image in parallel. The modulator then guides a category-aware generator to continue or stop mining regions for each object. With such a checkingmining mechanism, the number of region mining steps for each object adapts to its size. Intuitively, our multi-miner consists of multiple parallel "region miners", each of which performs the region mining sub-process for a single object. Furthermore, we also provide the theoretical basis of the multi-miner by revealing the connection between region mining and distribution mapping.
Some backgrounds would be included inevitably during region mining as the spatial resolution of input images is not preserved due to stacked convolutional layers. To alleviate this problem, we further propose a multi-scale training strategy to help our multi-miner progressively mine finer regions. Our motivation is, small-scale inputs provide global information about the whole objects, while largescale inputs provide information about details. As illustrated in Fig. 2, as the scale of the input image increases, the category-aware generator mines smaller but finer object regions. Thus, the multi-miner roughly mines the objects during early steps, and then further mines the small remaining regions in later steps. The regions mined in later steps are smaller and finer, and include fewer background regions.
We apply the multi-miner to mining object regions from input images and train a semantic segmentation model with our mined regions. Extensive experiments demonstrate that the multi-miner provides higher-quality region mining results and offers better segmentation performance, compared with well-established baselines. In particular, it helps achieve the mIoU of 65.9% and 66.1% on PASCAL VOC 2012 val and test sets.
To sum up, we make the following contributions: • We propose a multi-miner, which is the first model to perform region mining procedure that is fully adaptive to different objects. We also provide the theoretical basis for our multi-miner.
• We propose a multi-scale training strategy to work with the multi-miner to mine progressively finer object regions and prohibit backgrounds from being mined.
• Experiment results on PASCAL VOC 2012 segmentation benchmark show that our method establishes new state-of-the-art under the same weakly-supervised setting.

Related Work
A variety of weakly-supervised methods have been proposed to mitigate the insufficiency of pixel-level supervision in semantic segmentation. For example, Dai et al. [5] and Papandreou et al. [21] propose to leverage bounding boxes as supervision for semantic segmentation; Lin et al. [18] take scribbles as relatively coarse labels. Compared with bounding boxes and scribbles, image-level labels are easier to obtain and thus widely exploited. In this work we focus on weakly-supervised semantic segmentation with image-level supervision.
Some early methods propose to directly train a segmentation model with image-level labels [21,22,23,6] through Multiple Instance Learning (MIL) [19]. However, as image-level labels do not include sufficient information for learning a segmentation model, the performance of these methods is not satisfactory. Recently, most methods [14,29,2,11,16] propose to mine object regions by expanding the seed regions localized by the class activation maps [32,25]. These methods can be divided into the following two categories.
The first category, like [14,12,1], expand the object regions outside the training process of the classification model. SEC [14] proposes to refine the seed regions by an approximated CRF [15]. In [12], seed regions are refined in an unsupervised manner during the segmentation model training. AffinityNet [1] leverages the seed regions to train an additional network for learning pixel-level affinity, which is in turn used to expand the seed regions.
The second category, such as [29,11,17,30,28,16], mine object regions during the training process of the classification model. AE [29] expands the seed regions by iteratively erasing the newly found discriminative regions and then re-training the classification model. [17] modifies the erasing strategy to be end-to-end trainable. Furthermore, SeeNet [11] introduces two self-erasing strategies to keep unexpected background regions from being discovered. [30] revisits the dilated convolution [3] and mines object regions by merging the feature maps generated by multiple dilated convolutional blocks of different dilation rates. In [28], object regions are mined by iteratively training the classification model and the segmentation model. More recently, FickleNet [16] proposes to discover different object regions by randomly selecting various hidden units to generate attention maps. Figure 3. Model architecture of the multi-miner. The category-aware generator generates region maps to mine the object regions. The parallel modulator controls the whole region mining procedure by performing a multi-label classification task. Both the category-aware generator and the parallel modulator are trained via the loss of the modulator. A region map pool is maintained for each object to store all region maps.
All these methods mine the object regions for all the objects in a monotonous way. Different from them, our method adaptively takes different numbers of region mining steps for different objects.

Object-adaptive region mining
Architecture of the multi-miner The architecture of the proposed multi-miner is illustrated in Fig. 3. It consists of three components: a backbone feature extractor, a parallel modulator and a category-aware generator. The feature extractor is used to extract high-level features of the input images. The parallel modulator aims to control the region mining procedure by checking whether there are remaining object regions left for each category of each image, which can be exactly implemented by a multi-label classification model. The multi-label classifier is actually a combination of multiple independent binary classifiers, each of which corresponds to one semantic category. For each semantic category, if there are still remaining object regions, the classifier will recognize the object continuously based on the regions. Otherwise, the classifier will not be able to recognize this object. This characteristic can be leveraged to guide the category-aware generator to continue or stop mining object regions. The category-aware generator is fullyconvolutional. The number of channels of its last layer is equal to the number of all possible categories, such that it can generate a region map for each single category in the image. A region map is used to mine the object regions in the input image via element-wise multiplication; the pixel values of a region map are in the range [0, 1]. In a region map, the mined regions have low pixel values.
Region mining procedure The multi-miner adaptively takes different numbers of region mining steps for different objects. In each step, the multi-miner mines object regions via the following three stages. First, the parallel modulator is updated to evaluate the remaining object regions for all semantic categories in the input image. Second, under the guidance of the updated modulator, the category-aware generator is trained to continue or stop mining regions for each semantic category. Third, the updated category-aware generator is used to generate region maps. To reduce computation cost, we apply the region maps to the output feature maps of the feature extractor, instead of the input images.
Formally, let I = {(I i , y i )} N i=1 denote the training set of N images, where y i is the image-level label for image I i 1 . Denote the set of all possible object categories as C, and the set of image-level labels in a given image I as C pos . We use θ e , θ m and θ g to represent the parameters of the feature extractor, parallel modulator, and category-aware generator respectively. Additionally, we maintain a region map pool P j for each semantic category j ∈ C pos in each image I to store the region maps in all steps.
In the first stage of the t-th region mining step, the feature extractor takes in I and outputs the feature map F . We mask the previously mined regions from F , obtaining F t : Here denotes an element-wise multiplication, and M τ j denotes the region map of semantic category j generated in the τ -th iteration (τ = 1, · · · , t − 1). The minimization operation is always assumed to be conducted individually for every single spatial location unless stated otherwise.
In Eqn. (1), we adopt the minimization operation at every spatial location across all previously generated categoryspecific region maps for two reasons. First, object regions of one category are regarded as backgrounds for other categories. Therefore, such a Winner-Take-All merging strategy avoids object regions of one category from being overridden by region maps of other categories. This ensures that our multi-miner mines regions for all semantic categories independently. Second, this strategy prohibits interference of the regions mined in different steps. We train the parallel modulator by minimizing a multilabel classification loss. Let L cls (x) be the loss of the modulator when its input is x. We train our modulator by minimizing After optimizing Eqn. (2), the parallel modulator will be able to recognize the objects with remaining regions while being unable to recognize those whose regions are completely mined.
In the second stage, we train the category-aware generator so that it can continue or stop mining regions for each object according to whether there are remaining regions. Concretely, the category-aware generator takes in F t and outputs the region mapM t j for category j: Here H t j is output from the j-th channel of the last convolutional layer of the generator, the division is an elementwise operation, the maximization and minimization are conducted across all spatial locations of H t j , and takes a small value and is introduced for computational stability. In Eqn. (3), we use normalization rather than Sigmoid activation because the latter often causes training difficulty. We then further mask F t byM t j , obtainingF t : Then we maximize the classification loss of the parallel modulator withF t as its input. But there may be a trivial solution for the generator, i.e. to generate a region map masking the whole feature map. Such a region map actually mines everything from the input image. To avoid this issue, we add an additional regularization over the size of the mined regions: where · F denotes the F-norm and |C pos | counts the elements in C pos . Thus, the overall loss of training the category-aware generator is Algorithm 1 Region Mining procedure of Multi-Miner. Input: Training set I, factor λ, training epochs n m and n g , learning rate η. 1: Initialize θ e and θ m , then keep θ e fixed. 2: while not all object regions are mined do 3: Freeze θ g . 4: for n m epochs do Freeze θ m . 8: for n g epochs do 9: Update the θ g under the guidance the modulator: θ g ← L g − η∇ θg L g . 10: end for 11: for I i in I do 12: if M t j contains object regions then 13: Store M t j in P j . 14: end if 15: end for 16: end while Output: Region map pools P j for all images I i .
where λ is a trade-off parameter. Since we train the category-aware generator by maximizing the loss of the parallel modulator, we say the modulator "guides" the generator to mine regions.
For a given object, if its regions are not completely mined in previous steps, we will make the generator generate a region map to mine the remaining regions by minimizing Eqn. (6). If there is no object region left, i.e. the parallel modulator cannot recognize this object, the second term in Eqn. (6) dominates in the optimization process as the first term cannot guide the generator to update. Decreasing the second term, i.e. increasing the F-norm of generated region maps, pushes the category-aware generator to generate a region map whose all pixel values are equal to 1. Such a region map does not contain any object region, which indicates the region mining sub-process of this object should stop.
In the last stage, we use the updated category-aware generator to generate region maps M t j for each semantic category j in image I, and store them in their corresponding region map pool P j . Note that here M t j differs fromM t j in that M t j is generated by the updated generator, whileM t j is produced to train the generator.
Training process We first minimize the classification loss w.r.t. θ e and θ m over I to obtain the initial feature extractor and parallel modulator. Then, we only update θ m and θ g , keeping θ e fixed for computational efficiency. The whole region mining procedure is summarized in Alg. 1. After all the region mining sub-processes stop, we merge all the region maps for each object and obtain the mined regions where τ = 1, · · · , T j , and T j is the last step of category j.

Multi-scale training strategy
Due to the decreased spatial resolution with stacked convolutional layers, some background regions, particularly those surrounding the object regions, would be inevitably included during the region mining procedure. To mitigate this problem, we propose a multi-scale training strategy to progressively mine finer object regions.
Specifically, we start the region mining procedure with input images of a small scale, i.e. low resolution. This helps our multi-miner to obtain holistic information of the object regions, and mine the main part of the objects. Then, we increase the spatial resolution of the input images step by step, so that our multi-miner can progressively mine details of the object regions. In this way, the region maps become increasingly finer, and backgrounds are kept from being included. However, considering computational efficiency, the spatial resolution cannot be increased without any constraint. Thus we only use a set of K different spatial scales. From the first region mining step to the K-th step, we gradually increase the spatial resolution of the input images. From the K-th step, the spatial resolution stops changing.
Formally, let S = {s 1 , · · · , s K |s 1 < · · · < s K , K 2} denote the set of spatial resolution. The spatial resolution of the input images in the t-th step r t is given by Note that due to multi-scale training, in Eqn. (1), some of the previously generated region maps are of lower spatial resolution than the feature map F . We use bilinear interpolation to upsample these region maps so that they have the same spatial resolution as F .

Connection with distribution mapping
In this subsection, we reveal the connection between region mining and distribution mapping. With such a connection, we can explain why our multi-miner is able to perform region mining that is adaptive to each single object.
A multi-category region mining task can be recast as multiple binary region mining tasks of two categories, i.e. foreground and background. Without loss of generality, we consider a binary region mining task. The goal of such a task is to mine all foreground regions from the input image, and leave only background regions in the image. Namely, binary region mining aims to map the distribution of the images containing both foregrounds and backgrounds to the distribution of the images containing only backgrounds.
Formally, we define two original distributions: p 0 is the distribution of the images containing only backgrounds, and p 1 is the distribution of the images that contain both foregrounds and backgrounds. We denote an image as x. Then, region mining aims to find a mapping M(·) s.t. ∀x ∼ p 1 , M(x) ∼ p 0 . As proven in [8], this can be achieved by utilizing a generator G to implement the mapping M(·) and a discriminator D to differentiate samples from p 0 and samples generated by G. We further denote the distribution of the images produced by the generator G as q 1 , i.e. G(x) ∼ q 1 . Similar to [8], it can be proven that by solving the following minimax objective optimization problem: we reach an equilibrium where both G and D stop updating, and have q 1 = p 0 , meaning we obtain a generator G which maps all images containing both foregrounds and backgrounds to the images containing only backgrounds.
For a |C|-category region mining task, however, we do not need to instantiate |C| pairs of generator and discriminator for all categories. Instead, we can leverage two neural networks to model the |C| generators and the |C| discriminators respectively, which correspond to the category-aware generator and the parallel modulator in our multi-miner. Namely, our multi-miner is actually the combination of the |C| pairs of generator and discriminator for all categories. With such a mechanism, the multi-miner performs region mining sub-processes for each category in each image, and thus is adaptive to each single object.

Settings
Dataset and evaluation metrics We evaluate our multiminer for weakly-supervised semantic segmentation on the PASCAL VOC 2012 benchmark [7]. It provides images from 20 object categories and is split into training (1,464 images), validation (1,449 images) and testing (1,456 images). We use the augmented training set provided by [9] that contains 10,582 images to train our multi-miner based region mining model as well as the segmentation model. We compare our model with state-of-the-arts on both the validation and test sets. The segmentation performance is evaluated in terms of pixel Intersection-over-Union (IoU) averaged on 21 semantic categories. The results on the test set are obtained by submitting the predicted results to the official PASCAL VOC evaluation server.
Implementation Following previous works [29,11,17], we use VGG-16 [27] pre-trained on ImageNet [24] to build our multi-miner. Specifically, for its feature extractor, we remove the layers after conv5-3. Furthermore, we modify pool1, pool2 and pool3 by changing their kernel size to 3, stride to 2, and padding to 1. Similarly, we modify pool4 and pool5 so that their kernel size is 3, stride is 1, and padding is 1. With such modifications to the aforementioned pooling layers, given the same input image, the spatial resolution of the feature maps output by conv5 is 2× larger than that of the conv5 feature maps from the vanilla VGG-16. The parallel modulator of our multi-miner consists of three convolution layers, among which the first two have 1024 3×3 convolution kernels and the third one has 20 1 × 1 kernels, corresponding to 20 object categories. After the convolutional layers, there is a global average pooling layer to output the classification scores. The category-aware generator has the same convolutional layers as the parallel modulator and has a ReLU activation layer.
We initialize the feature extractor and parallel modulator with input size of 321 2 , batch size of 64, and weight decay of 0.0001. The initial learning rate is 0.001 for the feature extractor layers and 0.01 for the modulator layers, both divided by 10 after 30 epochs. We train this classification network for totally 50 epochs. The parameters of the feature extractor are frozen afterwards. Then we initialize our category-aware generator using the modulator parameters.
For computation efficiency, before the region mining procedure, we first extract and store the features output by the feature extractor, and then use the features to train the parallel modulator and the category-aware generator. Specifically, we train the modulator and the generator for 15 epochs and 1 epoch respectively, with weight decay of 10 −4 and learning rate of 10 −3 . As for multi-scale training, the spatial resolutions are 256 2 , 321 2 and 417 2 for the first, second and third step respectively, and we keep 417 2 from the third step. The corresponding batch size for the three scales are 256, 128 and 64.
For the segmentation task, to fairly compare with other works, we adopt the standard Deeplab-LargeFOV architecture [3] pretrained on ImageNet [24]. Following [11,2,12,28,16], we also use ResNet [10] version of Deeplab-LargeFOV architecture [4] and report the results of both versions. We use the same background cues as SeeNet [11]. When using Conditional Random Fields (CRF) for postprocessing, we adopt the same code as in [3].

Comparisons with state-of-the-arts
We conduct experiments to compare our proposed model with existing state-of-the-art weakly-supervised semantic segmentation ones under the same setting. Table 1 shows the experiment results.
We observe that our multi-miner outperforms all the baselines. Among all the baseline methods, the erasingbased methods, i.e. AE [29], GAIN [17] and SeeNet [11], share some similarities with ours. The difference is that

Ablation studies
Comparisons across steps In our experiment, we find that the longest region mining sub-process takes 7 steps.
After all region mining sub-processes stop, we merge the region maps generated from the first T steps (T =   1, · · · , 7) for each object, and train the standard Deeplab-LargeFOV architecture accordingly. Concretely, for the category j in image I, we merge {M τ j } Tj τ =1 if T j < T , and merge {M τ j } T τ =1 otherwise, where T j is last step of category j.
We summarize their performance on the PASCAL VOC 2012 validation set in Fig. 6. We can see that the performance increases as T increases from 1 to 7. This demonstrates our multi-miner gradually mines object regions from the 1st to the 7th step. In addition, we can see that the incre- ment of mIoU per step begins to decrease after the 4th step. The reason may be that a majority of objects are roughly or completely mined before the 4th step.
Moreover, we visualize some of the region mining subprocesses in Fig. 7. We observe that our multi-miner adaptively takes different region mining steps for different objects. Generally, larger objects need more steps. The first two rows and the 3rd-5th rows show two examples of images containing multiple objects. From them, we can observe that the number of region mining steps for different objects in the same image also differs.
Effectiveness of multi-scale training To prove effectiveness of our multi-scale training strategy, we compare it with 3 single-scale baselines. Specifically, we train 3 multiminer models with input images of spatial resolution fixed at 256 2 , 321 2 , 417 2 respectively, and compare their results with that of the multi-miner trained with multi-scale strategy. Fig. 8 shows some of the mined regions. Comparing the three baselines, we can observe with a fixed small scale i.e. 256 2 , our multi-miner can mine a large size of object regions, but the regions are coarse and include some irrelevant background regions, particularly for small objects. With a fixed large scale i.e. 417 2 , the mined regions are finer but  Figure 7. Regions mined in different steps. In each row, the first column is the input image, the second column is the output region mining result, and the following rows show the mined region in every step. Here "Stop" means the region mining sub-process of this object stops, and the semi-transparent region maps means there is no more remaining object region. Note that the number of region mining steps of different objects in the same image also varies. Please refer to Supplementary Material for more results.  less integral for large objects. In contrast, our multi-scale training strategy inherits the advantages of both small-scale and large-scale training, but does not have their drawbacks, and thus is good for both large and small objects and gives integral object regions with fine boundaries. Additionally, we train the standard Deeplab-LargeFOV architecture with object regions obtained with the above four training strategies. The segmentation results on the PASCAL VOC 2012 validation set are summarized in Table 2. It can be seen that our multi-miner achieves the best performance with multi-scale training, which further demonstrates the advantage of multi-scale training.

Conclusion
We proposed an object-adaptive multi-miner framework to mine integral and fine object regions. It adaptively takes different numbers of region mining steps for different objects, offering high-quality region mining results. Moreover, we proposed a multi-scale training strategy to mine progressively finer regions. We applied the mined regions to training a weakly-supervised semantic segmentation model. Experiment results have shown that our proposed model offers much higher-quality mined object regions than state-ofthe-arts.