{'fraction_non_alphanumeric': 0.03406597006125323, 'fraction_numerical': 0.01297127321563104, 'mean_word_length': 4.6286873156342185, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Weakly Supervised Semantic Image Segmentation with Self-correcting Networks

Building a large image dataset with high-quality object masks for semantic segmentation is costly and time consuming. In this paper, we reduce the data preparation cost by leveraging weak supervision in the form of object bounding boxes. To accomplish this, we propose a principled framework that trains a deep convolutional segmentation model that combines a large set of weakly supervised images (having only object bounding box labels) with a small set of fully supervised images (having semantic segmentation labels and box labels). Our framework trains the primary segmentation model with the aid of an ancillary model that generates initial segmentation labels for the weakly supervised instances and a self-correction module that improves the generated labels during training using the increasingly accurate primary model. We introduce two variants of the self-correction module using either linear or convolutional functions. Experiments on the PASCAL VOC 2012 and Cityscape datasets show that our models trained with a small fully supervised set perform similar to, or better than, models trained with a large fully supervised set while requiring ∼7x less annotation effort.


Introduction
Deep convolutional neural networks (CNNs) have been successful in many computer vision tasks including image classification [27,18,71], object detection [43,32,41], semantic segmentation [3,66,8], action recognition [13,24,47,52], and facial landmark localization [50,64,70]. However, the common prerequisite for all these successes is the availability of large training corpora of labeled images. Of these tasks, semantic image segmentation is one of the most costly tasks in terms of data annotation. For example, drawing a segmentation annotation on an object is on average ∼8x slower than drawing a bounding box and ∼78x slower than labeling the presence of objects in images [4]. As a result, most image segmentation datasets are orders of magnitude smaller than image-classification datasets. * Work done while interning at Quadrant.ai In this paper, we mitigate the data demands of semantic segmentation with a weakly supervised method that leverages cheap object bounding box labels in training. This approach reduces the data annotation requirements at the cost of requiring inference of the mask label for an object within a bounding box.
Current state-of-the-art weakly supervised methods typically rely on hand-crafted heuristics to infer an object mask inside a bounding box [39,11,25]. In contrast, we propose a principled framework that trains semantic segmentation models in a weakly supervised setting using a small set of fully supervised images (with semantic object masks and bounding boxes) and a larger set of weakly supervised images (with only bounding box annotations). The fully supervised set is first used to train an ancillary segmentation model that predicts object masks on the weakly labeled set. Using this augmented data a primary segmentation model is trained. This primary segmentation model is probabilistic to accommodate the uncertainty of the mask labels generated by the ancillary model. Training is formulated so that the labels supplied to the primary model are refined during training from the initial ancillary mask labels to more accurate labels obtained from the primary model itself as it improves. Hence, we call our framework a self-correcting segmentation model as it improves the weak ancillary labels based on its current probabilistic model of object masks.
We propose two approaches to the self-correction mechanism. Firstly, inspired by Vahdat [53], we use a function that linearly combines the ancillary and model predictions. We show that this simple and effective approach is the natural result of minimizing a weighted Kullback-Leibler (KL) divergence from a distribution over segmentation labels to both the ancillary and primary models. However, this approach requires defining a weight whose optimal value should change during training. With this motivation, we develop a second adaptive self-correction mechanism. We use CNNs to learn how to combine the ancillary and primary models to predict a segmentation on weakly supervised instances. This approach eliminates the need for a weighting schedule.
Experiments on the PASCAL VOC and Cityscapes datasets show that our models trained a with small portion of fully supervised set achieve a performance comparable to (and in some cases better than) the models trained with all the fully supervised images.

Related Work
Semantic Segmentation: Fully convolutional networks (FCNs) [35] have become indispensable models for semantic image segmentation. Many successful applications of FCNs rely on atrous convolutions [62] (to increase the receptive field of the network without down-scaling the image) and dense conditional random fields (CRFs) [26] (either as post-processing [5] or as an integral part of the segmentation model [68,31,46,34]). Recent efforts have focused on encoder-decoder based models that extract longrange information using encoder networks whose output is passed to decoder networks that generate a high-resolution segmentation prediction. SegNet [3], U-Net [44] and Re-fineNet [30] are examples of such models that use different mechanisms for passing information from the encoder to the decoder. 1 Another approach for capturing long-range contextual information is spatial pyramid pooling [28]. ParseNet [33] adds global context features to the spatial features, DeepLabv2 [6] uses atrous spatial pyramid pooling (ASPP), and PSPNet [66] introduces spatial pyramid pooling on several scales for the segmentation problem.
While other segmentation models may be used, we employ DeepLabv3+ [8] as our segmentation model because it outperforms previous CRF-based DeepLab models using simple factorial output. DeepLabv3+ replaces Deeplabv3's [7] backbone with the Xception network [9] and stacks it with a simple two-level decoder that uses lower-resolution feature maps of the encoder.
Robust Training: Training a segmentation model from bounding box information can be formulated as a problem of robust learning from noisy labeled instances. Previous work on robust learning has focused on classification problems with a small number of output variables. In this setting, a common simplifying assumption models the noise on output labels as independent of the input [38,37,40,49,65]. However, recent work has lifted this constraint to model noise based on each instance's content (i.e., input-dependent noise). Xiao et al. [60] use a simple binary indicator function to represent whether each instance does or does not have a noisy label. Misra et al. [36] represent label noise for each class independently. Vahdat [53] proposes CRFs to represent the joint distribution of noisy and clean labels extending structural max-margin models [54,55] to deep networks. Ren et al. [42] gain robustness against noisy labels by reweighting each instance during training whereas Dehghani et al. [12] reweight gradients based on a confidence score on labels. Among methods proposed for label correction, Veit et al. [56] use a neural regression model to predict clean labels given noisy labels and image features, Jiang et al. [23] learn curriculum, and Tanaka et al. [51] use the current model to predict labels on noisy instances. All these models have been restricted to image-classification problems and have not yet been applied to image segmentation.
Weakly Supervised Semantic Segmentation: The focus of this paper is to train deep segmentation CNNs using bounding box annotations. Papandreou et al. [39] propose an Expectation-Maximization-based (EM) algorithm on top of DeepLabv1 [5] to estimate segmentation labels for the weakly annotated images with box information. In each training step, segmentation labels are estimated based on the network output in an EM fashion. Dai et al. [11] propose an iterative training approach that alternates between generating region proposals (from a pool of fixed proposals) and fine-tuning the network. Similarly, Khoreva et al. [25] use an iterative algorithm but rely on GrabCut [45] and handcrafted rules to extract the segmentation mask in each iteration. Our work differs from these previous methods in two significant aspects: i) We replace hand-crafted rules with an ancillary CNN for extracting probabilistic segmentation labels for an object within a box for the weakly supervised set. ii) We use a self-correcting model to correct for the mismatch between the output of the ancillary CNN and the primary segmentation model during training.
In addition to box annotations, segmentation models may use other forms of weak supervision such as image pixel-level [57,59,21,2,16,58,14], image labellevel [63], scribbles [61,29], point supervision [4], or web videos [19]. Recently, adversarial learning-based methods [22,48] have been also proposed for this problem. Our framework is complimentary to other forms of supervision or adversarial training and can be used alongside them.

Methods
Our goal is to train a semantic segmentation network in a weekly-supervised setting using two training sets: i) a small fully supervised set (containing images, segmentation ground-truth and object bounding boxes) and ii) a larger weakly supervised set (containing images and object bounding boxes only). An overview of our framework is shown in Fig. 1. There are three models: i) The Primary segmentation model generates a semantic segmentation of objects given an image. ii) The Ancillary segmentation model outputs a segmentation given an image and bounding box. The model generates an initial segmentation for the weakly supervised set, which aids training of the pri-  Figure 1: An overview of our segmentation framework consisting of three models: i) Primary segmentation model generates a semantic segmentation of objects given an image. This is the main model that is subject to the training and is used at test time. ii) Ancillary segmentation model outputs a segmentation given an image and bounding box. This model generates an initial segmentation for the weakly supervised set, which will aid training the primary model. iii) Self-correction module refines segmentations generated by the ancillary model and the current primary model for the weakly supervised set. The primary model is trained using the cross-entropy loss that matches its output to either ground-truth segmentation labels for the fully supervised examples or soft refined labels generated by the self-correction module for the weakly supervised examples.
mary model. iii) The Self-correction module refines the segmentations generated by the ancillary and current primary model for the weakly supervised set. Both the ancillary and the primary models are based on DeepLabv3+ [8]. However, our framework is general and can use any existing segmentation model. In Sec. 3.1, we present the ancillary model, and in Sec. 3.2, we show a simple way to use this model to train the primary model. In Sec. 3.3 and Sec. 3.4, we present two variants of self-correcting model.
Notation: x x x represents an image, b b b represents object bounding boxes in an image, and y y y = [y y y 1 , y y y 2 , . . . , y y y M ] represents a segmentation label where y y y m ∈ [0, 1] C+1 for m ∈ {1, 2, . . . , M } is a one-hot label for the m th pixel, C is the number of foreground labels augmented with the background class, and M is the total number of pixels. Each bounding box is associated with an object and has one of the foreground labels. The fully supervised dataset is in- . We use p(y y y|x x x; φ φ φ) to represent the primary segmentation model and p anc (y y y|x x x, b b b; θ θ θ) to represent the ancillary model. φ φ φ and θ θ θ are the respective parameters of each model. We occasionally drop the denotation of parameters for readability. We assume that both ancillary and primary models define a distribution of segmentation labels using a factorial distribution, i.e., p(y y y|x x x; φ φ φ) = M m=1 p m (y y y m |x x x; φ φ φ) and p anc (y y y|x x x, b b b; θ θ θ) = M m=1 p anc,m (y y y m |x x x, b b b; θ θ θ) where each factor (p m (y y y m |x x x; φ φ φ) or p anc,m (y y y m |x x x, b b b; θ θ θ)) is a categorical distribution (over C + 1 categories).

Ancillary Segmentation Model
The key challenge in weakly supervised training of segmentation models with bounding box annotations is to infer the segmentation of the object inside a box. Existing approaches to this problem mainly rely on hand-crafted rulebased procedures such as GrabCut [45] or an iterative label refinement [39,11,25] mechanism. This latter procedure typically iterates between segmentation extraction from the image and label refinement using the bounding box information (for example, by zeroing-out the mask outside of boxes). The main issues with such procedures are i) bounding box information is not directly used to extract the segmentation mask, ii) the procedure may be suboptimal as it is hand-designed, and iii) the segmentation becomes ambiguous when multiple boxes overlap.
In this paper, we take a different approach by designing an ancillary segmentation model that forms a per-pixel label distribution given an image and bounding box annotation. This model is easily trained using the fully supervised set (F ) and can be used as a training signal for images in W. At inference time, both the image and its bounding box are fed to the network to obtain p anc (y y y|x x x (w) , b b b (w) ), the segmentation labels distribution.
Our key observation in designing the ancillary model is that encoder-decoder-based segmentation networks typically rely on encoders initialized from an imageclassification model (e.g., ImageNet pretrained models). This usually improves the segmentation performance by transferring knowledge from large image-classification datasets. To maintain the same advantage, we augment an encoder-decoder-based segmentation model with a parallel bounding box encoder network that embeds bounding box information at different scales (See Fig. 2). The input to the bounding box encoder is a 3D tensor representing a binarized mask of the bounding boxes and a 3D shape representing the target dimensions for the encoder output. The input mask tensor is resized to the target shape then passed through a 3×3 convolution layer with sigmoid activations. The resulting tensor can be interpreted as an attention map which is element-wise multiplied to the feature maps generated by the segmentation encoder. Fig. 2 shows two paths of such feature maps at two different scales, as in the DeepLabv3+ architecture. For each scale, an attention map is generated, fused with the corresponding feature map using element-wise multiplication, and fed to the decoder. For an image of size W × H × 3, we represent its object bounding boxes using a binary mask of size W×H×(C+1) that encodes the C + 1 binary masks. The c th binary mask at a pixel has the value 1 if it is inside one of the bounding boxes of the c th class. A pixel in the background mask has value 1 if it is not covered by any bounding box.
The ancillary model is trained using the cross-entropy loss on the full dataset F : which can be expressed analytically under the factorial distribution assumption. This model is held fixed for the subsequent experiments.

No Self-Correction
We empirically observe that the performance of our ancillary model is superior to segmentation models that do not have box information. This is mainly because the bounding box information guides the ancillary model to look for the object inside the box at inference time.
The simplest approach to training the primary model is to train it to predict using ground-truth labels on the fully supervised set F and the labels generated by the ancillary model on the weakly supervised set W. For this "no-selfcorrection" model the Self-correction module in Fig. 1 merely copies the predictions made by the ancillary segmentation model.
Training is guided by optimizing the following loss: w∈W y y y where the first term is the cross-entropy loss with one-hot ground-truth labels as target and the second term is the cross-entropy with soft probabilistic labels generated by p anc as target. Note that the ancillary model parameterized by θ θ θ is fixed. We call this approach the no self-correction model as it relies directly on the ancillary model for training the primary model for examples in W.

Linear Self-Correction
Eq. 2 relies on the ancillary model to predict label distribution on the weakly supervised set. However, this model is trained using only instances of F without benefit of the data in W. Several recent works [39,11,25,51,53] have incorporated the information in W by using the primary model itself (as it is being trained on both F and W) to extract more accurate label distributions on W.
Vahdat [53] introduced a regularized Expectation-Maximization algorithm that uses a linear combination of KL divergences to infer a distribution over missing labels for general classification problems. The main insight is that the inferred distribution q(y y y|x x x, b b b) over labels should be close to both the distributions generated by the ancillary model p anc (y y y|x x x, b b b) and the primary model p(y y y|x x x). However, since the primary model is not capable of predicting the segmentation mask accurately early in training, these two terms are reweighted using a positive scaling factor α: The global minimizer of Eq. 3 is obtained as the weighted geometric mean of the two distributions: Since both p anc (y y y|x x x, b b b) and p(y y y|x x x) decompose into a product of probabilities over the components of y y y, and since the distribution over each component is categorical, then q(y y y|x x is also factorial where the parameters of the categorical distribution over each component are computed by applying softmax activation to the linear combination of logits coming from primary and ancillary models using σ l l l m +α l l l anc m / α + 1 . Here, σ(.) is the softmax function and, l l l m and l l l anc m are logits generated by primary and ancillary models for the m th pixel.
Having fixed q(y y y|x x x (w) , b b b (w) ) on the weakly supervised instances in each iteration of training the primary model, we can train the primary model using: W y y y q(y y y|x x x (w) , b b b (w) ) log p(y y y|x x x (w) ; φ φ φ).
Note that α in Eq. 3 controls the closeness of q to p(y y y|x x x) and p anc (y y y|x x x, b b b). With α = ∞, we have q = p anc (y y y|x x x, b b b) and the linear self-correction in Eq. 5 collapses to Eq. 2, whereas α = 0 recovers q = p(y y y|x x x). A finite α maintains q close to both p(y y y|x x x) and p anc (y y y|x x x, b b b). At the beginning of training, p anc (y y y|x x x, b b b) cannot predict the segmentation label distribution accurately. Therefore, we define a schedule for α where α is decreased from a large value to a small value during training of the primary model. This corrective model is called the linear self-correction model as it uses the solution to a linear combination of KL divergences (Eq. 3) to infer a distribution over latent segmentation labels. 2 As the primary model's parameters are optimized during training, α biases the self-correction mechanism towards the primary model.

Convolutional Self-Correction
One disadvantage of linear self-correction is the hyperparameter search required for tuning the α schedule during training. In this section, we present an approach that overcomes this difficulty by replacing the linear function with a convolutional network that learns the self-correction mechanism. As a result, the network automatically tunes the mechanism dynamically as the primary model is trained. If the primary model predicts labels accurately, this network can shift its predictions towards the primary model. Fig. 3 shows the architecture of the convolutional selfcorrecting model. This small network accepts the logits generated by p anc (y y y|x x x, b b b) and p(y y y|x x x) models and generates the factorial distribution q conv (y y y|x x x, b b b; λ λ λ) over segmentation labels where λ λ λ represents the parameters of the subnetwork. The convolutional self-correction subnetwork consists of two convolution layers. Both layers use a 3×3 kernel and ReLU activations. The first layer has 128 output feature maps and the second has feature maps based on the number of classes in the dataset.
The challenge here is to train this subnetwork such that it predicts the segmentation labels more accurately than either p anc (y y y|x x x, b b b) or p(y y y|x x x). To this end, we introduce an additional term in the objective function, which trains the subnetwork using training examples in F while the primary model is being trained on the whole dataset: where the first and second terms train the primary model on F and W (we do not backpropagate through q in the second term) and the last term trains the convolutional selfcorrecting network. Because the q conv subnetwork is initialized randomly, it is not able to accurately predict segmentation labels on W early on during training. To overcome this issue, we propose the following pretraining procedure: 1. Initial training of ancillary model: As with the previous self-correction models, we need to train the ancillary model. Here, half of the fully supervised set (F ) is used for this purpose.
2. Initial training of convolutional self-correction network: The fully supervised data (F ) is used to train the primary model and the convolutional self-correcting network. This is done using the first and last terms in Eq. 6.
3. The main training: The whole data (F and W) are used to fine-tune the previous model using the objective function in Eq. 6.
The rationale behind using half of F in stage 1 is that if we used all F for training the p anc (y y y|x x x, b b b) model, it would train to predict the segmentation mask almost perfectly on this set, therefore, the subsequent training of the convolutional self-correcting network would just learn to rely on p anc (y y y|x x x, b b b) . To overcome this training issue, the second half of F is held out to help the self-correcting network to learn how to combine p anc (y y y|x x x, b b b) and p(y y y|x x x).  Figure 3: Convolutional self-correction model learns refining the input label distributions. This subnetwork receives logits from the primary and ancillary models, then concatenates and feeds the output to a two-layer convolution network.

Experiments
In this section, we evaluate our models on the PASCAL VOC 2012 and Cityscapes datasets. Both datasets contain object segmentation and bounding box annotations. We split the full dataset annotations into two parts to simulate a fully and weakly supervised setting. Similar to [8,39], performance is measured using the mean intersection-overunion (mIOU) across the available classes.
Training: We use the public Tensorflow [1] implementation of DeepLabv3+ [8] as the primary model. We use an initial learning rate of 0.007 and train the models for 30,000 steps from the ImageNet-pretrained Xception-65 model [8]. 3 For all other parameters we use standard settings suggested by other authors. At evaluation time, we apply flipping and multi-scale processing for images as in [8]. We use 4 GPUs, each with a batch of 4 images.
We define the following baselines in all our experiments: 1. Ancillary Model: This is the ancillary model, introduced in Sec. 3.1, predicts semantic segmentation labels given an image and its object bounding boxes. This model is expected to perform better than other models as it uses bounding box information.
2. No Self-correction: This is the primary model trained using the model introduced in Sec. 3.2.
3. Lin. Self-correction: This is the primary model trained with linear self-correction as in Sec. 3.3.
4. Conv. Self-correction: The primary model trained with the convolutional self-correction as in Sec. 3.4.
5. EM-fixed Baseline: Since our linear self-correction model is derived from a regularized EM model [53], we compare our model with Papandreou et al. [39] which is also an EM-based model. We implemented their EM-fixed baseline with DeepLabv3+ for fair comparison. This baseline achieved the best results in [39] for weakly supervised learning.
For linear self-correction, α controls the weighting in the KL-divergence bias with large α favoring the ancillary model and small α favoring the primary model. We explored different starting and ending values for α with an exponential decay between these endpoints. We find that a starting value of α = 30 and the final value of α = 0.5 performs well for both datasets. This parameter setting is robust as moderate changes of these values have little effect.

PASCAL VOC Dataset
In this section, we evaluate all models on the PASCAL VOC 2012 segmentation benchmark [15]. This dataset consists of 1464 training, 1449 validation, and 1456 test images covering 20 foreground object classes and one background class for segmentation. An auxiliary dataset of 9118 training images is provided by [17]. We suspect, however, that the segmentation labels of [17] contain a small amount of noise. In this section, we refer to the union of the original PASCAL VOC training dataset and the auxiliary set as the training set. We evaluate the models mainly on the validation set and the best model is evaluated only once on the test set using the online evaluation server.
In Table 1, we show the performance of different variants of our model for different sizes of the fully supervised set F . The remaining examples in the training set are used as W. We make several observations from Table 1: i) The ancillary model that predicts segmentation labels given an image and its object bounding boxes performs well even when it is trained with a training set as small as 200 images. This shows that this model can also provide a good training signal for the weakly supervised set that lacks segmentation labels. ii) The linear self-correction model typically performs better than no self-correction model supporting our idea that combining the primary and ancillary model for inferring segmentation labels results in better training of the primary model. iii) The convolutional self-correction model performs better than the linear self-correction while eliminating the need for defining an α schedule. Fig. 4 shows the output of these models. Table 2 compares the performance of the self-correcting models against different baselines and published results. In this experiment, we use 1464 images as F and 9118 images originally from the auxiliary dataset as W. Both selfcorrection models achieve similar results and outperform other models. The EM-fixed baseline [39] trained using the DeepLabv3+ [8] architecture achieves 79.25% on the validation set while the original paper has reported 64.6% using DeepLabv1 trained with the same split. We also train the DeepLabv3+ [8] Table 1: Ablation study of models on the PASCAL VOC 2012 validation set using mIOU for different sizes of F . For the last three rows, the remaining images in the training set is used as W, i.e. W + F = 10582.
Xception model using all the training data as the fully supervised set, and we observe that this model achieves 79.33%. 4 Surprisingly, our weakly supervised models outperform the fully supervised trained model using the whole training data. We hypothesize that this may be due to the label noise in the 9k auxiliary set [17] that negatively affects the performance of the Vanilla DeepLapv3+. Fig. 5 compares the output of the ancillary model with ground-truth annotations for a few examples in this set.
Comparing Table 1 and 2, we see that with F = 200 and W = 10382, our linear self-correction model performs similarly to vanilla DeepLabv3+ trained with the whole dataset. Using the labeling cost reported in [4], this theoretically translates to a ∼7x reduction in annotation cost.

Cityscapes Dataset
In this section we evaluate performance on the Cityscapes dataset [10] which contains images collected from cars driving in cities during different seasons. This dataset has good quality annotations, however some instances are over/under segmented. It consists of 2975 training, 500 validation, and 1525 test images covering 19 foreground object classes (stuff and object) for the segmentation task. However, 8 of these classes are flat or construction labels (e.g., road, sidewalk, building, and etc.), and a very few bounding boxes of such classes cover the whole scene. To create an object segmentation task similar to the PASCAL VOC dataset, we use only 11 classes (pole, traffic light, traffic sign, person, rider, car, truck, bus, train, motorcycle, and bicycle) as foreground classes and all other classes are assigned as background. Due to this modification of labels, we report the results only on the validation set, as the test set on server evaluates on all classes. We do not use the coarse annotated training data in the dataset. Table 3 reports the performance of our model for an increasing number of images as F , and Table 4 compares our models with several baselines similar to the previous dataset. The same conclusion and insights observed on the PASCAL dataset hold for the Cityscapes dataset indicating the efficacy of our self-corrective framework.  Input Image ground-truth Ancillary Model No Self-correction Lin. Self-correction Conv. Self-correction  Table 1. The Conv. Self-correction model typically segments objects better than other models.
Input Image Ground-truth Ancillary Heatmap Input Image Ground-truth Ancillary Heatmap

Conclusion
In this paper, we have proposed a weakly supervised framework for training deep CNN segmentation models using a small set of fully labeled and a large set of weakly labeled images. We introduced two mechanisms that enable the underlying primary model to correct the weak labels provided by an ancillary model. The proposed self-correction mechanisms combine the predictions made by the primary and ancillary model either using a linear function or trainable CNN. The experiments show that our proposed framework outperforms previous weakly supervised models on both the PASCAL VOC 2012 and Cityscapes datasets. Our framework can also be applied to the instance segmentation task [20,69,67], but we leave further study of this to future work.