{'fraction_non_alphanumeric': 0.01989925221363537, 'fraction_numerical': 0.007540439911141704, 'mean_word_length': 4.808104670179902, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Amodal Instance Segmentation

We consider the problem of amodal instance segmentation, the objective of which is to predict the region encompassing both visible and occluded parts of each object. Thus far, the lack of publicly available amodal segmentation annotations has stymied the development of amodal segmentation methods. In this paper, we sidestep this issue by relying solely on standard modal instance segmentation annotations to train our model. The result is a new method for amodal instance segmentation, which represents the first such method to the best of our knowledge. We demonstrate the proposed method's effectiveness both qualitatively and quantitatively.


Introduction
Consider the horse shown in the left panel of Figure 1. The task of instance segmentation requires marking the visible region of the horse, as shown in the middle panel, and has been tackled by several existing algorithms [18,6,19,27,7]. In this paper, we consider a different task, which requires marking both the visible and the occluded regions of the horse, as shown in the right panel. In keeping with terminology used in the psychology literature on visual perception [20], we refer to the former task as modal instance segmentation and the latter task as amodal instance segmentation. A natural question to ask is if the task of amodal instance segmentation is well-posed: given only the visible portions of an object, there are many possible configurations of the hidden portions of the object, all of which appear to be plausible hypotheses to a human. This is particularly true for articulated objects under heavy occlusion. For example, if the lower body of a person is blocked from view, there is no single correct hypothesis for the configuration of the person's legs -the person could be sitting or standing, and so hypotheses consistent with either pose would be equally valid. Despite this ambiguity, humans are capable of performing amodal completion and tend to predict the occluded regions with high degrees of consistency [39]. An amodal segmentation system would open the way to sophisticated occlusion reasoning. For instance, given an amodal segmentation mask, we can infer the presence, extent, boundary and region of occlusions by comparing it to the modal segmentation mask. We can also deduce the relative depth ordering by comparing the modal and amodal masks of the occluded and occluding objects. The information derived from the amodal segmentation mask can be further used downstream for a variety of interesting applications. For instance, we can estimate the physical dimensions of an object in the real world using its amodal bounding box, as demonstrated by Kar et al. [21] The fact that all these occlusion reasoning problems can be reduced to amodal instance segmentation implies that amodal instance segmentation is more challenging than all these problems combined. An amodal segmentation system must not only be capable of determining if an object is occluded, but also where it is occluded. It must be able to hypothesize the shape of the occluded portion even though it has never seen the whole object before. It must be sensitive enough to detect the diminished signal from the small part of the occluded object that remains visible, but must be robust enough to avoid being misled by strong signals from occluding objects.
Furthermore, additional complicating factors arise if one were to attempt training a model for the task, the chief among them being the lack of supervised training data. While efforts are underway to collect amodal segmentation annotations [39], no amodal segmentation data is publicly available at the time of writing. In the meantime, we need to devise a clever way of using existing data to train a model for this new task.
In this paper, we present a new method for amodal instance segmentation, which to the best of our knowledge is the first such method. We train our method purely from existing modal instance segmentation data, thereby sidestepping issues arising from the lack of supervised training data. We make a key observation: while it is not possible to compute the amodal mask of an object from the modal mask by undoing occlusion, it is easy to do the reverse. Instead of undoing existing occlusion, we add synthetic occlusion and retain the original mask, which essentially becomes the true amodal mask for the composite image. We train a convolutional neural net to recover the original mask from the generated composite image. We do not assume knowledge of the amodal bounding box at test time; instead, we infer it from the amodal segmentation heatmap using a new strategy, which we dub Iterative Bounding Box Expansion. We demonstrate that despite being trained on synthetic data, the resulting model is quite effective at predicting amodal masks on images with real occlusions.

Related Work
Efforts toward understanding the semantic meaning associated with free-form regions in images started with work on figure-ground segmentation, the objective of which is to identify the foreground pixels in typically object-centric images. Early methods [4,37,25,23,26] investigated ways of combining top-down and bottomup segmentation approaches and incorporating class-specific or object-specific templates of the foreground object's appearance. Later on, the focus shifted to the more general problem of semantic segmentation, which aims to identify the pixels that belong to each object category in more complex images. A diverse range of approaches for this problem have been developed; earlier approaches extend CRF-based formulations [24,22,3], consider combining object detections with region proposals [15], scoring groupings of over-segmented regions [35], aggregating information from multiple foreground-background hypotheses [5] and synthesizing scores from different overlapping regions to obtain a pixel-wise classification [1]. Later approaches [10,31,29,38] use feedforward or recurrent neural net models to extract features or to predict the final label of each pixel directly from images. In effort at achieving understanding of a scene at a finer level of granularity, recent work has focused on the task of instance segmentation, the goal of which is to identify the pixels that belong to each individual object instance. The predominant framework for this task is to find the bounding box of each instance using an object detector and predict the figure-ground segmentation mask inside each box. Earlier approaches rely on DPM [11] detections and predict segmentation masks using a simple appearance model [36,30,8], combine DPM detections and semantic segmentation predictions [12,9] or adopt a transductive approach [34]. More recent methods leverage the power of convolutional neural nets. SDS [18] and Dai et al. [6] use a neural net to compute features on region proposals and classifies them using an SVM, while the Hypercolumn net [19], Iterative Instance Segmentation [27] and Multi-Task Network Casades [7] predict the segmentation mask directly from the image patch. We view the task of amodal instance segmentation as the natural next step in this direction.
There has been relatively little work exploring amodal completion. Kar et al. [21] tackled the problem of predicting the amodal bounding box of an object. Gupta et al. [16] explored completing the occluded portions of planar surfaces given depth information. To the best of our knowledge, there has been no algorithmic work on general-purpose amodal segmentation. However, there has been work on collecting amodal segmentation annotations. Zhu et al. [39] collected amodal segmentation annotations on BSDS images, but has yet to make them publicly available. As far as we know, the proposed method represents the first method for amodal segmentation.

Overview
We generate amodal training data solely from standard modal instance segmentation annotations. In our case, we use the Semantic Boundaries (SBD) annotations [17] on the PASCAL VOC 2012 train set as the data source. We generate three types of data: image patches, modal bounding boxes and target segmentation masks. Image patches and modal bounding boxes are used as input to the model and target segmentation masks are used as supervisory signal on the output of the model.
The key observation we leverage is that the phenomenon of occlusion can be easily simulated by overlaying objects on top of other objects. More concretely, we first generate randomly cropped image patches that overlap with at least one foreground object instance, which we will refer to as the main object. We then extract random object instances from other images and overlay them on top of the randomly cropped patches with their modal segmentation masks serving as the alpha matte. Each overlaid object is positioned and scaled randomly in a way that ensures a moderate degree of overlap with the main object. Essentially, this procedure generates composite patches where the main object is partially occluded by other objects.
Next, for each composite patch, we find the smallest bounding box that encloses the portions of the main object that remain visible. This is essentially ground truth modal bounding box of the main object in the composite patch. To simulate noisy modal localization at test time, we jitter the bounding box randomly.
Finally, we generate the target segmentation masks corresponding to the composite patches produced above. For each patch, we take the corresponding part of its original modal segmentation mask and label the pixels belonging to the object as positive, pixels belonging to the background as negative and pixels belonging to other objects as unknown. This manner of label assignment captures what we know about the amodal mask given the modal mask -we know the visible portion of the object must be a part of the whole object and that the object cannot be occluded by the background. However, the object may be occluded by other objects in the image; consequently, the pixels belonging to other objects in the modal mask are labelled as unknown. Because the original modal mask is not affected by overlaid objects, this mask includes portions of the main object that were originally visible but are now occluded in the composite patch. Hence, the target mask is consistent with the true amodal mask.

Implementation Details
Data is generated on-the-fly during training. To generate a training example, we sample an image uniformly and then sample an object instance from the image, which will be referred to as the main object. We then randomly sample a bounding box that overlaps with the main object's bounding box along each dimension by at least 70%. The size of the sampled box is randomly chosen and the length of each dimension is between 70% and 200% of the length of the corresponding dimension of the object's bounding box. Next, we choose the number of objects to overlay onto the patch inside the bounding box randomly by picking an integer from 0 to 2. To select an object to overlay, we sample an image and then sample a random object instance from the image. The object is placed at a random location that overlaps with the main object and scaled randomly so that the length of its shortest dimension is 75% of the length of the corresponding dimension of the patch on average. After each of the above operations, we check if the proportion of the main object that remains visible falls below 30%. If it does, we undo the most recent operation and try again. Otherwise, we proceed to find the bounding box the encloses the portion of the main object that remains visible and randomly samples a box that overlaps with the bounding box by at least 75% along each dimension and can differ in size from the bounding box by at most 10% in each dimension.

Testing
We take the modal bounding box, that is, the bounding box of the visible part of the object, and the category of the object as given, which can be obtained from an object detector, like R-CNN [14], fast R-CNN [13] or faster R-CNN [32]. We then compute the modal segmentation heatmap using Iterative Instance Segmentation (IIS) [27], which is the state-of-the-art method for modal instance segmentation. The proposed algorithm proceeds to predict the amodal segmentation mask and bounding box in an iterative fashion using a new strategy that will be referred to as Iterative Bounding Box Expansion. Initially, the amodal bounding box is set to be the same as the modal bounding box. In each iteration, given the amodal bounding box, we feed the patch inside the amodal bounding box to a convolutional neural net, which predicts the amodal segmentation mask inside an expanded amodal bounding box that also includes areas immediately outside the original amodal bounding box. We compute the average heat intensity in the areas lie above, below, to the left and to the right of the original bounding box. If the average heat intensity associated with a particular direction is above a threshold, which is set to 0.1 in our experiments, we expand the bounding box in that direction and take this new bounding box to be the amodal bounding box used in the next iteration. This procedure is repeated until the average heat intensities in all directions are below the threshold. To obtain the final amodal segmentation mask, we colour in all pixels whose intensities in the corresponding heatmap exceed 0.7. Similarly, we obtain the modal segmentation mask by thresholding the modal heatmap at 0.8.

Image
Iteration 1 Iteration 2 Iteration 3 Fig. 4: Amodal segmentation heatmap after each iteration of Iterative Bounding Box Expansion. The green box denotes the modal bounding box and the blue box denotes the amodal bounding box. In each iteration, we use the average heat intensity outside the amodal bounding box to decide whether to expand the amodal bounding box in the next iteration.

Training
The convolutional neural net we use takes in an image patch, a modal segmentation heatmap and a category and outputs the amodal segmentation heatmap. The net has the same architecture as that used by IIS, which is based on the hypercolumn architecture introduced by Hariharan et al. [19]. This architecture is designed to take advantage of both low-level image features at finer scales and high-level image features at coarser scales. It does so by making the final heatmap prediction dependent on the summation of upsampled feature maps from multiple intermediate layers, which is known as the hypercolumn representation. The version of the architecture we use is based on the VGG 16-layer net [33], which is referred to as "O-Net" in [19]. The IIS architecture is a variant of this architecture that also takes in an initial heatmap hypothesis via an additional category-dependent channel as input, which can be set to constant if no initial heatmap hypothesis is available. If an initial heatmap hypothesis is provided, the model refines the heatmap hypothesis to produce an improved heatmap prediction. Using this architecture, IIS is able to iteratively refine its own heatmap prediction by feeding in its heatmap prediction from the preceding iteration as input.
Each training example consists of an image patch, a modal bounding box and a target amodal segmentation mask. To prepare input to the net, we take the part of image patch that lies inside the modal bounding box and scale it anisotropically to 224 × 224, feed it to IIS as input. We take the modal segmentation heatmap produced by one iteration of IIS, align it to the coordinates of the original image patch and upsample it to 224 × 224 using bilinear interpolation. Because the model should predict the mask corresponding to an area larger than the image patch it sees, we remove 10% of the image patch from each of the four sides and rescale it to 224 × 224. If less than 10% of the pixels in this new patch belong to the visible portion of the object, we reject the current sample and generate a new training example. We centre the data by subtracting the mean pixel from the image patch and transforming the modal segmentation heatmap element-wise to lie between −127 and 128. Finally, we feed the image patch, the modal segmentation heatmap and the target amodal mask to the model for training.
The model is trained end-to-end using stochastic gradient descent with momentum on mini-batches of 32 patches starting from weights of the model used by IIS. The loss function we use is the sum of pixel-wise negative log likelihoods over all pixels with known ground truth labels. We apply instance-specific weights that are inversely proportional to the factor by which each patch is upsampled. We train the model with a constant learning rate of 10 −5 , weight decay of 10 −3 and momentum of 0.9 for 50, 000 iterations.

Experiments
Because there is no existing dataset with amodal instance segmentation annotations, there is no ground truth against which the predictions can be evaluated, making it difficult to perform a quantitative evaluation. We first present qualitative results and then perform an indirect quantitative evaluation against coarselevel annotations on the full PASCAL VOC 2012 val set. To perform a direct quantitive evaluation, we annotated 100 randomly chosen occluded objects from the same dataset with amodal segmentation masks and evaluate the proposed method on this subset.

Qualitative Results
We use the proposed method to generate amodal segmentation mask predictions for objects in the PASCAL VOC 2012 val set. Because the focus of this paper is on the segmentation system, we take the modal bounding box and the category of the object of interest as given and obtain them from the ground truth. We show the amodal heatmap and mask predictions produced by the proposed method Figures 5, 6 and 7, along with the modal heatmap and mask predictions generated by IIS for comparison. As shown, the proposed method is generally quite effective. In particular, even though the synthetic occlusions generated for training purposes do not appear entirely realistic, the proposed method is able to devise plausible hypotheses for hidden portions of objects caused by real occlusions.
We classify occlusions into two types: cases where the occluding object is mostly contained inside the occluded object and cases where a significant portion of the occluding object lies outside the occluded object. We will refer to the former as interior occlusions and the latter as exterior occlusions. For interior occlusions, the goal is to predict the mask between visible portions of the object, whereas for exterior occlusions, the goal is to predict the mask beyond the visible portion. There is typically a single correct way to handle interior occlusions: if it results from a true occlusion, the corresponding hole should be filled in; otherwise, it should remain in the amodal mask. Therefore, the task in this case is less ambiguous and more straightforward. On the other hand, there are generally multiple equally valid ways to handle exterior occlusions: there are many possible ways to extend the visible portion that all lead to plausible amodal masks. As a result, the task in this case is more ambiguous and challenging. The method need to decide how much to extend the modal mask by in every direction. To do so, it must rely on the knowledge it learns about the general shape of objects of the particular category to produce a plausible amodal mask hypothesis.
We first examine examples of occluded objects for which the proposed method produces correct amodal segmentation masks, which are shown in Figure 5. As shown, the proposed method is able to successfully predict the amodal mask on images with interior or exterior occlusions. On images with interior occlusions, such as the ones depicting a person standing in front of a bicycle, a person lying on top of a car and a person sitting on a sofa, the proposed method is able to overlook the occluding person and predict the correct amodal mask. On images with exterior occlusions, like the ones depicting a person sitting at a dining table, a coffee table occluded by a person, a bottle occluded by other bottles and a boat occluded by a dock, the proposed method is able to guess the correct extent of the object and predict a reasonably plausible amodal mask. It is interesting to compare the two images that both depict a person sitting at a dining table: the proposed method's prediction for the occluded part of the person varies with the camera angle, in that the table is excluded from the amodal mask when the person sits sideways from the camera, but is included in the amodal mask when the person sits in front of the camera. Surprisingly, on the image with a dog on a folding chair, the proposed method is able to produce a fairly good amodal mask despite the poor quality of the modal mask in this case. Finally, the proposed method is able to produce remarkably good amodal masks on some challenging images with exterior occlusions, such as the image depicting a dog on a kayak and the images in the bottom two rows, suggesting that the proposed method is able to learn something about the general shape of objects.
Next, we take a look at examples of occluded objects for which the predicted amodal segmentation masks are incorrect, which are shown in Figure 6. For the first image on the left, even though the predicted amodal mask appears plausible and resembles a person, it is inconsistent with the image. A foot is incorrectly hallucinated near the bike pedal, even though the person's foot is actually on top of the bike. The scarcity of a unusual pose like this in the training set may have led to this mistake. For the second image in the same column, the predicted amodal mask fails to include the full extent of the person's legs and feet. It is interesting to observe that at locations where the person's legs could be positioned, the heat intensity is moderately high. This suggests that the mistake may be caused by learning difficulties that result from the large variation in the possible configurations of the lower body. Because there are multiple modes that all represent valid predictions, the model learns to hedge its bets among different modes, yielding uncertain predictions at test time. For the third image, the modal mask incorrectly includes a part of the adjacent cat, resulting in the amodal mask covering both cats. For the first image on the right, the shape of the amodal mask roughly resembles the shape of a quadruped, though the model seems unsure of the side the head should be on. Since it is difficult to tell from the image which way the horse is facing, such a mistake is not unreasonable. For the second image in the same column, the model seems to believe the ship faces right, even though it actually faces left. This may be caused by the visual resemblance of the visible portion of the ship to the front of a typical ship. For the last image, even though the model correctly predicts the location of the occluded arm, it misses a part of the woman's back. This is likely due to the erroneous exclusion of the back from the modal mask.
While the above examples show that the model is capable of performing amodal completion, we need to make sure that the model does not hallucinate when presented with unoccluded objects. We explore some examples of unoccluded objects and the corresponding mask predictions in Figure 7. Since the objects are unoccluded, amodal segmentation masks should be the same as modal segmentation masks. As shown, for images in the top row, the predicted modal and amodal masks are fairly similar. For images in the bottom row, the predicted amodal masks are more accurate than the predicted modal masks. Unlike the modal masks, the amodal masks successfully cover the objects in their entirety. This can be explained by the inherent robustness the model must have in order to predict amodal segmentations. By learning to be robust to occlusion, the model also learns to be robust to variations in low-level patterns in the image that may confuse a modal segmentation model.
We include additional examples of heatmap and mask predictions in the appendix.

Indirect Evaluation
We use the proposed method combined with a modal instance segmentation method like IIS to predict the presence or absence of occlusion. We do so by predicting the modal and amodal segmentation masks for each object in the PASCAL VOC 2012 val set and computing the following ratio: area (modal mask ∩ amodal mask) area (amodal mask) Intuitively, this ratio measures the degree by which an object is occluded. For an unoccluded object, because the amodal mask should be the same as the modal mask, this ratio should be close to 1. On the other hand, for a heavily occluded object, only a small proportion of the pixels inside the amodal mask should also be included in the modal mask; as a result, this ratio should be significantly less than 1. We will henceforth refer to this ratio as the area ratio.
We compare our predictions to the occlusion presence annotations in the PASCAL VOC 2012 val set, which are available for all object instances and specify whether they are occluded. First, we compute the modal and amodal mask predictions using IIS and the proposed method for all objects that are annotated as being occluded and plot the distribution of area ratios. We then do the same for objects that are annotated as being unoccluded. These two distributions are shown in Figure 8a. As shown, the distribution for unoccluded objects is heavily skewed towards high area ratios, whereas the distribution for occluded objects peaks at an area ratio of around 0.75. This is in line with our expectations and indicates the predicted amodal masks for occluded objects typically includes larger areas outside the modal mask than those for unoccluded objects. This verifies that the proposed method generally performs amodal completion only for occluded objects and not for unoccluded objects. The distribution for occluded objects is also flatter than that for unoccluded objects, which is also not surprising, since the amount by which an object is occluded by can vary significantly from object to object.
This difference in the two distributions suggests that area ratio can be used to predict the presence or absence of occlusion. We can consider a simple classifier that declares an object to be unoccluded if its area ratio is greater than a threshold. For each value of the threshold, we can compute the precision and recall of this classifier. We plot the precision and recall we obtain by varying this threshold in Figure 8b. We compute average precision and find it to be 77.17%.

Direct Evaluation
To evaluate the accuracy of the mask produced by the proposed method, we need ground truth amodal segmentation annotations. Because no such annotations  are publicly available, we collected a set of amodal segmentation masks on 100 objects. For each category, we selected five object instances randomly from the PASCAL VOC 2012 val set that are labelled as occluded and annotated them with amodal segmentation masks. For this purpose, we used the annotation tool for MS COCO [28], which is based on the Open Surfaces annotation tool [2].  Table 1: Performance comparison of IIS and the proposed method on the task of amodal instance segmentation.
We compare the overlap with ground truth achieved by the amodal mask predicted by the proposed method to that achieved by the modal mask predicted by the state-of-the-art modal segmentation method, IIS. In this setting, a modal instance segmentation system represents a fairly strong baseline since in cases where occlusion is not severe, it is possible to omit the occluded portion completely from the predicted mask without significantly lowering IoU with the ground truth.
Segmentation Performance We first evaluate the segmentation system in isolation by taking the modal bounding box and the category of the object of interest from ground truth. Each point represents an object instance and the points belonging to object instances in the same category share the same colour. Points that lie above the diagonal represent object instances whose amodal mask predictions are more accurate than their modal mask predictions. (b) Consider the setting where a predicted mask is deemed correct when it overlaps with the ground truth by at least a particular level of cutoff. This plot shows the proportion of object instances whose predicted masks are correct as a function of the cutoff.
As shown in Figure 9a, on most instances, the masks produced by the proposed method are significantly more accurate than the masks produced by IIS. Notably, the proposed method improves overlap compared to IIS by as much as 20% to 50% in many cases. Overall, the proposed method produces better masks than IIS on 73% of objects. Of the remaining 27% of objects on which the proposed method performs worse than IIS, the drop in overlap is less than 5% for the majority of objects. Hence, the masks produced by the proposed method are generally more accurate, sometimes by a sizeable margin.
In Figure 9b, we plot prediction accuracy if all masks with IoUs that exceed a particular cutoff are considered correct. As shown, predicting the mask using the proposed method consistently results in higher accuracy than using IIS at all levels of the cutoff. In Table 1, we report the accuracy of the proposed method and IIS at 50% IoU cutoff and 70% IoU cutoff. Additionally, we compute the area under the accuracy curve for both methods. We find that the proposed method performs better than IIS on all metrics.
Combined Detection and Segmentation Performance Next, we evaluate the performance of the combined detection and segmentation pipeline. We use faster R-CNN [32] as our detection system and compare the performance of the pipeline with the proposed method as the segmentation system to that of the pipeline with IIS as the segmentation system. We use the amodal segmentation annotations we collected as ground truth and measure performance using mean region average precision (mAP r ) [18], which is a common metric used for modal instance segmentation. Region average precision is defined analogously to the standard average precision metric used for detection, except that overlap is computed by finding the pixel-wise intersection-over-union (IoU) between the predicted mask and the ground truth mask. Because some instances in a given image are not annotated with ground truth amodal masks, we are unable to compute region overlap with some ground truth instances. Hence, we make a slight modification to the computation of the metric: we use bounding box overlap instead of region overlap to determine which ground truth instance a mask prediction is assigned to. However, we still use region overlap to determine if a mask prediction should be deemed correct.
As shown in Table 2, the pipeline with the proposed method outperforms the pipeline with IIS by 11.1 points at 50% overlap and 8.6 points at 70% overlap.  Table 2: Performance comparison of the combined detection and segmentation pipeline with faster R-CNN as the detection system and either IIS or the proposed method as the segmentation system.

Conclusion
We presented a new method for amodal instance segmentation, which represents the first such method to the best of our knowledge. We introduced a novel strategy for generating synthetic amodal instance segmentation data from modal instance segmentation annotations. This strategy enabled us to train a model for amodal instance segmentation despite the the lack of publicly available amodal segmentation data. Additionally, we presented a new approach for iteratively predicting the amodal bounding box from amodal segmentations. We demonstrated the effectiveness of the proposed method in predicting amodal segmentation masks both qualitatively and quantitatively.