{'fraction_non_alphanumeric': 0.034803840423770896, 'fraction_numerical': 0.0118357887766926, 'mean_word_length': 4.604128014842301, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Learning to Separate: Detecting Heavily-Occluded Objects in Urban Scenes

In the past decade, deep learning based visual object detection has received a significant amount of attention, but cases when heavy intra-class occlusions occur are not studied thoroughly. In this work, we propose a novel Non-MaximumSuppression (NMS) algorithm that dramatically improves the detection recall while maintaining high precision in scenes with heavy occlusions. Our NMS algorithm is derived from a novel embedding mechanism, in which the semantic and geometric features of the detected boxes are jointly exploited. The embedding makes it possible to determine whether two heavily-overlapping boxes belong to the same object in the physical world. Our approach is particularly useful for car detection and pedestrian detection in urban scenes where occlusions tend to happen. We validate our approach on two widely-adopted datasets, KITTI and CityPersons, and achieve state-of-the-art performance.


Introduction
Recent years have witnessed significant progress in object detection using deep convolutional neural networks (DCNNs) [5,10,30]. The approach taken by many state-of-the-art object detection methods [7,8,21,27,29] is to predict multiple bounding boxes for an object and then use a heuristic method such as non-maximum suppression (NMS) to remove superfluous bounding boxes that stem from duplicate detected objects.
The Greedy-NMS algorithm is easy to implement and tends to work well in images where objects of the same class do not significantly occlude each other. However, in urban scenes, where the task is to detect potentially heavily occluded cars or pedestrians, Greedy-NMS does not perform adequately. The decrease in accuracy is due to the fundamental limitation of the NMS algorithm that uses a fixed threshold: The algorithm cannot suppress duplicate bounding boxes belonging to the same object while preserving boxes belonging to different objects, where one object heavily occludes others. Soft-NMS [2] attempts to address this limitation by not removing overlapping boxes but instead lowering their confidence; however, all overlapping boxes are still treated as false positives regardless of how many physical objects are in the image.
The limitation of NMS could be circumvented with the aid of an oracle that assigns each bounding box an identifier related to its corresponding physical-world object. Then, a standard NMS algorithm could be applied per set of boxes with the same identifier (but not across identifiers), thus ensuring that false positives from one object do not result in suppression of a true positive from a nearby object.

SGEs of occluded objects are separated
SGEs of the same object are grouped together

Heavily Occluded Objects
Semantics-Geometry Embedding (SGE) Figure 1: Learned embedding, Semantics-Geometry Embedding (SGE) for bounding boxes predicted by our proposed detector on a KITTI image. Heavily overlapped boxes are separated in the SGE space according to the objects they are assigned to. Thus, distance between SGEs can be used for Non-Maximum-Suppression in heavy-occlusion scenes.
To approximate such an oracle, we can try to learn a mapping from boxes into a latent space so that the heavily overlapping boxes can be separated in that space. Naively, this mapping can be implemented by learning an embedding for every box based on its region features, for example, the pooled features after RoIPooling [7]. However, the usefulness of such an embedding would be limited because heavily overlapping boxes would tend to yield similar region features, thus would map to nearby points in the embedding space. In this paper, we demonstrate that by considering both the region features and the geometry of each box, we can successfully learn an embedding in a space where heavily overlapping boxes are separated apart if they belong to different objects; we call the learned embedding Semantics-Geometry Embedding (SGE). We also propose a novel NMS algorithm that takes advantage of the SGE to improve detection recall.
We summarize Semantics-Geometry Embedding (SGE) in Fig. 1 where boxes belonging to the same object are mapped to similar SGE and boxes belonging to different but occluded objects are mapped to SGEs that are far away. Although the embedding algorithm may assign boxes in disparate parts of an image to similar SGEs, that would not negatively impact our SG-NMS algorithm because those boxes can be easily separated based on their intersection-over-union (IoU). The SGE is implemented as an associative embedding [24] and learned using two loss functions, separate loss and group loss.
To train the SGE with the object detector end-to-end, we propose a novel Serial R-FCN where the geometric feature of each detected box is precisely aligned with its semantic features. In summary, we make three main contributions: 1. A novel bounding-box-level Semantics-Geometry Embedding (SGE) is developed, and a novel Non-Maximum-Suppression algorithm, Semantics-Geometry Non-Maximum-Suppression (SG-NMS), based on this embedding, is derived. The algorithm markedly improves object detection in scenarios with heavy intra-class occlusions.

2.
A Serial R-FCN with self-attention in each head is presented, which not only provides the capability to learn the above-mentioned SGE end-to-end, but also improves object detection accuracy.
3. The proposed model with SG-NMS achieves state-of-the-art performance on the task of car detection in the benchmark KITTI [6] dataset and the task of pedestrian detection in the CityPersons [41] dataset by dramatically improving the detection recall in heavily-occluded scenes.
Object Detection. The CNN based object detectors can be divided into one-stage and two-stage approaches. In one-stage detectors [19,21,27], the object's class and the bounding box regressor are directly predicted by sliding windows on the feature maps. In two-stage object detectors [3,4,7,29,42], regions of interest (RoIs) are first proposed [9,29,34,46], then the class label and bounding box regressor are predicted for each RoI. Although the two-stage approaches often achieve higher accuracy, they suffer from low computational efficiency. R-FCN [4] addresses this problem by replacing the computation in fully-connected layers with nearly cost-free pooling operations.
Non Maximum Suppression. NMS is widely used in modern object detectors to remove duplicate bounding boxes. It may mistakenly remove boxes belonging to different objects. Soft-NMS [2] was proposed to address this problem by replacing the fixed NMS threshold with a score-lowering mechanism. However, highly-overlapping boxes are still treated as false positives regardless of the semantic information. In Learning-NMS [12], a neural network is used to perform NMS, but the appearance information is still not considered. In Adaptive-NMS [20], the NMS threshold is learned with the object detector, but when the threshold is set too high, false positives may be kept. There are other improvements on NMS, but they are not designed for handling intra-class occlusion. In [13], the relation of bounding boxes is used to perform NMS by considering their appearance and geometric feature, but the method is not designed for handling occlusion. In [11,15,31,33], the localization quality of each box is learned to help NMS with keeping accurate boxes.
Other Occlusion Handling Approaches. There are many other methods designed to handle occlusion, including both intra-class or inter-class occlusion. Most of them focus on detecting pedestrians in crowd scenes. Repulsion Loss [36] is proposed to prevent boxes from shifting to adjacent objects. The occluded person is detected by considering different body parts separately [25,32,42,43,44]. Extra annotations such as head position or visible regions are used [26,40,45] to create robust person detectors. Although these approaches have been shown to be effective in detecting occluded persons, it is difficult to generalize them to other scenarios like car detection.  Figure 2: Overview of the Serial R-FCN pipeline. The classification head is placed after the class agnostic bounding box regression head, making the whole pipeline serial. An additional Semantics-Geometry Module, parallel to the classification head, is added to learn the Semantics-Geometry Embedding for each refined box to assist the proposed SG-NMS.

Methodology
In this section, we first introduce the proposed Semantics-Geometry Embedding (Sec. 3.1), then the Semantics-Geometry NMS (Sec. 3.2), and finally the proposed Serial R-FCN (Sec 3.3). The overview of our proposed model is shown in Fig. 2.

Semantics-Geometry Embedding
Our key idea for separating occluded objects in an image is to map each putative detection to a point in a latent space. In this latent space, detections belonging to the same physical object form a tight cluster; detections that are nearby in the image plane, but belong to different physical objects, are pushed far apart.
To implement this idea we consider an embedding for each bounding box that takes the form of a dot-product where g is the geometric feature and s is the semantic feature. The geometric feature has a fixed form g = (x, y, w, h) T , in which (x, y) is its center coordinate and (w, h) are the width and height of the bounding box. We tried different kinds of geometric features including the geometric feature used in [13] and feature vectors with higher dimensions produced by a fully-connected layer, but found that such complexity did not provide further significant improvement.
Unlike the geometric feature, the semantic feature s is a weight output by a function that yields a vector compatible with g; the function is implemented as a neural network, as shown in the Semantics-Geometry Head in Fig. 2. Note that the SGE is computed by the linear transformation of the geometric feature taking the learned semantic feature as weight. This can be interpreted as that the neural network automatically learns how to distinguish the bounding boxes belonging to different objects.
We train the SGE using the loss function defined in Eq. 1. The training is carried out end-to-end, jointly with the object-detection branch using the loss function defined later.
We derive the loss function for the SGEby extending associative embedding [17,24]. Specifically, we use a group loss to group the SGEs of boxes belonging to the same object, and use a separate loss to separate apart SGEs of boxes belonging to different objects. For one image, the ground-truth boxes are denoted by Thus the refined bounding box is divided into M + 1 sets: is the set of refined boxes that are not assigned to any ground truth box. Then the group loss  Fig. 2) share the same architecture. The feature map from the backbone network goes through two branches to yield two score maps. Then the Position Sensitive RoI-Pooling [4] is applied to produce two grids of k 2 position-sensitive scores: one for the score map, the other one is transformed through a softmax function into a discreate distribution over the k 2 grids. The k 2 scores are then aggregated by the distribution to yield the output. and the separate loss is defined as: where e j * is the SGE of ground truth box b * j ,b * i is the ground truth box with the second largest IoU to b i and its SGE isẽ i . σ is a hyper-parameter controlling the maximum distance of two SGEs. In separate loss, p * i is a indicator variable which would be

Semantics-Geometry NMS
We now derive our simple, yet effective NMS algorithm, SG-NMS, which takes advantage of the Semantics-Geometry Embedding. The SG-NMS algorithm is summarized in Algorithm 1.
First the box with the highest detection score is selected as the pivot box. For each of the remaining boxes, its IoU with the pivot box is denoted by τ , and the box will be kept if the τ < N t . When τ > N t , we check the distance between its SGE and the pivot box's SGE. If the distance is larger than Φ(τ ), the box will also be kept. Here Φ(·) is a monotonically increasing function, which means that, as τ increases, a larger distance is required to keep it. In this work, we consider three kinds of SG-NMS: SG-NMS-Constant, SG-NMS-Linear and SG-NMS-Square, which correspond to: where t c , t l and t s are hyper-parameters.
In practice, we may choose a Feature Pyramid Network (FPN) [18] as the backbone network to improve the detection performance for small objects. This would cause the SGEs of boxes belonging to different FPN layers to be incomparable. To address this problem, we adopt a simple two-stage NMS schema. We first apply SG-NMS to objects produced by the same FPN level. We then use Greedy-NMS with a higher threshold to merge boxes of different layers, in which a box would only be suppressed by boxes from other layers.

Serial R-FCN
In order to get SGEs that can capture the difference between geometric features of boxes belonging to different objects, we need to align extracted semantic features strictly with the refined boxes after bounding box regression. However, this cannot be achieved by normal two-stage CNN-based object detectors where the pooled feature is aligned with the RoI instead of the refined box because of the bounding-box regression.
To address this problem, we propose Serial R-FCN, see Fig. 2. In Serial R-FCN, the classification head along with the SG module is placed after the class-agnostic bounding box regression head [7]; thus, the whole pipeline becomes a serial structure. The classification head and the SG module use the refined boxes for feature extraction rather than the RoIs. Thus, the pooled features are strictly aligned with the refined boxes. A light-weight self-attention branch is added into each head, as in Fig. 3. The output of the attention head is a discrete distribution over the k 2 position-sensitive grid. The position-sensitive scores are then aggregated through a weighted sum based on that distribution.
Placing the classification head after the regression head can bring us another benefit: It enables us to train the classification head using a higher IoU threshold. This yields more accurate bounding boxes. Without the serial structure, setting the IoU threshold to such a high value would result in the shortage of positive samples.
The idea of Serial R-FCN is similar to Cascade R-CNN [3]. However, different from Cascade R-CNN where multiple classification and regression heads are stacked for better accuracy, here we only have one regression head and one classification head, thus no extra parameter is introduced. Although the serial structure can be used by any two-stage detector, it suit for R-FCN best since no extra operation is added so the computation for the refined box is nearly cost-free.
In practice, we find that simply adopting the serial structure would easily make the network overfit on the training data. The reason is that as training progresses, the regression head becomes more and more powerful so that the classification head cannot receive enough hard negative examples. For instance, boxes whose IoU with the ground truth box is slightly smaller than the training threshold. The result is that it cannot distinguish those hard negative examples and true positives when the model is tested. To alleviate the overfitting problem, we adopt a simple but effective approach that we add some noise to the refined bounding box so that the classification head can continuously obtain hard false examples. Formally, during training, a box b = (x, y, w, h) is transformed to b = (x , y , w , h ) to train the classification head and the SG module: where σ x , σ y , σ w , σ h are noise coefficient drawn from uniform distribution j=k j=1 (−ζ k , ζ k ) where the four dimensions correspond to x, y, w, h respectively. In practice we set ζ x = ζ y = 0.05 and ζ w = ζ h = 0.2.
The whole pipeline is trained end-to-end, and the total loss function is as below: L total = L rpn + αL det + βL SGE ; (6) L rpn = L cls-anchor + L reg-anchor (7) where the L rpn is the commonly used loss to train the Region Proposal Network (RPN) [29], L det is object detection loss [7] and L SGE is the loss to train SGE as described in Sec. 3.1. We use two hyper-parameter α and β to balance between losses. Inside the above formula, we also show which kind of box a loss is apply to: the RPN classification loss and RPN regression loss is applied to the anchor boxes, the regression loss is applied to RoIs, and the classification loss, group loss and separate loss is applied to the refined boxes.

Experiments
This section presents experiments we conduct on two standard datasets, KITTI [6] and CityPersons [41]; results demonstrate the effectiveness of our approach to detect heavily-occluded cars and pedestrians in urban scenes.

Implementation Details
We implemented our Serial R-FCN in TensorFlow [1] and trained it on 1 Nvidia Titan V GPU. For KITTI, we chose a ResNet-101 [10] based Feature Pyramid Network (FPN) as the backbone network and set the batch size to 4. The model is trained for 100000 iterations using the Adam [16] optimizer with learning rate of 0.0001. For CityPersons, we chose ResNet-50 [10] as the backbone network and trained the model for 240000 iterations with batch size of 4. The initial learning rate was set to 0.0001 and decreased by a factor of 10 after 120000 iterations. In all experiments, OHEM is adopted to accelerate convergence [4]. In both dataset, we set θ, σ, ρ to 0.7, 0.3, and 1.0, and set α and β to 1.

Experiments on KITTI
The KITTI dataset contains 7,481 images for training and validation, and another 7,518 images for testing. We evaluate our methods on the car detection task where intra-class occlusions tend to happen the most. The dataset has a standard split into three levels of difficulty: Easy, Moderate, and Hard, according to the object scale, occlusion level and maximum truncation. Average Precision (AP) is used to evaluate the performance [6]. We split the training set by randomly sampling 3,722 images for validation; we use the remaining 3,759 images for training in all experiments.

Effectiveness of SG-NMS
In   Table 3: Comparison of AP between the proposed SG-NMS with other commonly used NMS algorithms with different NMS thresholds on the KITTI validation set. The highest AP for each occlusion level is marked red. The SG-NMS is better than Greedy-NMS and Soft-NMS at detecting partially and heavily occluded objects.
Greedy-NMS and Soft-NMS by 2.33 and 1.39 on the Hard level where heavy intra-class occlusion occurs.
To further demonstrate how our proposed SG-NMS handles intra-class occlusions, we divide the KITTI validation set into disjoint subsets based on the max-mutual-IoU, denoted by MMIOU, between ground-truth boxes. The max-mutual-IoU of a ground-truth box is defined by its maximum IoU with other ground-truth boxes in the same category. We separate the validation set into 3 levels: Bare (0 < MMIOU ≤ 0.2), Partial ( 0.2 < MMIOU ≤ 0.5) and Heavy ( 0.5 < MMIOU).
Ground-truth boxes with height less than 25 pixels are ignored during our evaluation.
We apply the NMS algorithm with a different threshold on the same set of detected boxes produced by our Serial R-FCN and report the results in Table 3. Overall, all three variants of SG-NMS outperform Greedy-NMS and Soft-NMS for the Heavy and the Partial occlusion levels, while maintaining high performance on the Bare level. For the Heavy level, the best result is achieved by SG-NMS-Square that is 6.54 and 4.68 higher than the best result of Greedy-NMS and Soft-NMS. Although Greedy-NMS can achieve an AP of 55.46 in the hard level when N t is set to 0.6, the AP in bare and partial level drops significantly, due to the false-positive boxes it generates.
We also report the detection recall on different MMIOU intervals and show the results in Fig. 4. When MMIOU is less than 0.5, all of those algorithms achieve similar recall. When there is severe intraclass occlusion, i.e., MMIOU > 0.5, the recall of Greedy-NMS and Soft-NMS drops significantly. However, all three SG-NMS keep a relatively high recall. When MMIOU > 0.5, the difference in recall among three SG-NMS algorithms is caused by the different slope of their Φ(·) function.

Purely-Semantic Embedding
We compare our Semantics-Geometry Embedding (SGE) with the purely-semantic model where for every box a 1-d Semantic Embedding (SE) is computed directly from its pooled region feature. The results are reported in Table 2. Our SGE is consistently better than the SE in all three embeddingbased NMS algorithms. In fact, the two loss functions defined in Sec. 3.1 for the SE are very unstable during training, which means it is difficult for the neural network to learn such an embedding based on semantic features only and reveal the necessity of introducing geometric features.

Ablation Study
In Table 4 we present experiments that demonstrate how the different model components affect the overall detection performance. Our method is proposed for detecting occluded objects, thus the analysis is focused on the detection of objects at the Hard difficulty (in the official split) and the Heavy occlusion level.   To conclude, we note that self-attention is useful to capture the semantic difference between heavily overlapping boxes. The box noise can alleviate the overfitting problem so that the detection precision is improved and the SG-NMS can improve the detection performance for heavily occluded objects.

Comparison with SOTA Methods
We compare our model with other state-of-the-art models on the KITTI car detection leaderboard and results are in Table 5. Our Serial R-FCN + SG-NMS is ranked at the third place among the SOTA methods. The APs on the Moderate and Hard level are 0.90 and 3.76 higher than the fourthplace [39]. Although the the RRC [28] and sensekitti [38] are ranked higher than ours, the speed of our method is more than ten times faster than theirs, since we focus our main contribution on the post-processing steps rather than in the detection pipeline which may achieve higher accuracy in the cost of computational efficiency.

Experiments on CityPersons
In addition to car detection, pedestrian detection is another task where intra-class occlusions usually happen. In this section, we report on experiments that we conducted on the CityPersons dataset to show how our SG-NMS and Serial R-FCN perform in detecting occluded pedestrians. The CityPersons dataset contains 5,000 images (2,975 for training, 500 for validation and 1,525 for testing). The log-average Miss Rate (MR) is used to evaluate the performance. Following [36], we compare the detection MR in different occlusion degrees.
In Table 6, we compare the MR performance among different NMS algorithms on CityPersons validation set. We also compare our model with other SOTA methods. The NMS hyper-parameters are obtained from grid search and the best result for each algorithm is reported. With Greedy-NMS, our Serial R-FCN achieves MR of 11.7 and 52.4 on the reasonable and heavy level. Using Soft-NMS yields a slight improvement. SG-NMS-Linear and SG-NMS-Square yield 0.2 and 0.7 improvement on the reasonable difficulty, but using SG-NMS-Constant harms the performance on reasonable difficulty because a single threshold cannot handle all the complex occlusion situations. All three SG-NMS improve the performance on the heavy and partial occlusion levels. Especially, the SG-NMS-Square improves the MR to 10.7 and 51.1 on partial and heavy occlusion levels, making our methods superior than other SOTA methods on those two levels. This means our method excels at handling occlusions.

Conclusion
In this paper, we presented two contributions, a novel Semantics-Geometry Embedding mechanism that operates on detected bounding boxes and an effective Semantics-Geometry Non-Maximum-Suppression algorithm that improves detection recall for heavily-occluded objects. Our approach achieves state-of-the-art performance on KITTI and CityPersons datasets by dramatically improving the detection recall. In the future, we plan to extend our method to detecting objects in video.