{'fraction_non_alphanumeric': 0.03743355826139405, 'fraction_numerical': 0.009876729370075771, 'mean_word_length': 4.484391151540211, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 8, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
End-to-End Integration of a Convolutional Network, Deformable Parts Model and Non-Maximum Suppression

Deformable Parts Models and Convolutional Networks each have achieved notable performance in object detection. Yet these two approaches find their strengths in complementary areas: DPMs are well-versed in object composition, modeling fine-grained spatial relationships between parts; likewise, ConvNets are adept at producing powerful image features, having been discriminatively trained directly on the pixels. In this paper, we propose a new model that combines these two approaches, obtaining the advantages of each. We train this model using a new structured loss function that considers all bounding boxes within an image, rather than isolated object instances. This enables the non-maximal suppression (NMS) operation, previously treated as a separate post-processing stage, to be integrated into the model. This allows for discriminative training of our combined Convnet + DPM + NMS model in end-to-end fashion. We evaluate our system on PASCAL VOC 2007 and 2011 datasets, achieving competitive results on both benchmarks.


Introduction
Object detection has been addressed using a variety of approaches, including sliding-window Deformable Parts Models [5,19,6], region proposal with classification [7,16], and location regression with deep learning [13,14]. Each of these methods have their own advantages, yet are by no means mutually exclusive. In particular, structured parts models capture the composition of individual objects from component parts, yet often use rudimentary features like HoG [2] that throw away much of the discriminative information in the image. By contrast, deep learning approaches [9,18,13], based on Convolutional Networks [10], extract strong image features, but do not explicitly model object composition. Instead, they rely on pooling and large fully connected layers to combine information from spatially disparate regions; these operations can throw away useful finegrained spatial relationships important for detection.
In this paper, we propose a framework (shown in Fig. 1) Figure 1. An overview of our system: (i) a convolutional network extracts features from an image pyramid; (ii) a set of deformable parts models (each capturing a different view) are applied to the convolutional feature maps; (iii) non-maximal suppression is applied to the resulting response maps, yielding bounding box predictions. Training is performed using a new loss function that enables back-propagation through all stages.
that combines these two approaches, fusing together structured learning and deep learning to obtain the advantages of each. We use a DPM for detection, but replace the HoG features with features learned by a convolutional network. This allows the use of complex image features, but still preserves the spatial relationships between object parts during inference.
An often overlooked aspect of many detection systems is the non-maximal suppression stage, used to winnow multiple high scoring bounding boxes around an object instance down to a single detection. Typically, this is a postprocessing operation applied to the set of bounding boxes produced by the object detector. As such, it is not part of the loss function used to train the model and any parameters must be tuned by hand. However, as demonstrated by Parikh and Zitnick [11], NMS can be a major performance bottleneck (see Fig. 2). We introduce a new type of image-level loss function for training that takes into consideration of all bounding boxes within an image. This differs with the losses used in existing frameworks that consider single cropped object instances. Our new loss function enables the NMS operation trained as part of the model, jointly with the Convnet and DPM components.

Related Work
Most closely related is the concurrent work of Girshick et al. [8], who also combine a DPM with ConvNet features in a model called DeepPyramid DPM (DP-DPM). Their work, however, is limited to integrating fixed pretrained ConvNet features with a DPM. We independently corroborate the conclusion that using ConvNet features in place of HoG greatly boosts the performance of DPMs. Furthermore, we show how using a post-NMS online training loss improves response ordering and addresses errors from the NMS stage. We also perform joint end-to-end training of the entire system.
The basic building blocks of our model architecture come from the DPMs of Felzenszwalb et al. [5] and Zhu et al. [19] [1], and the ConvNet of Krizhevsky et al. [9]. We make crucial modifications in their integration that enables the resulting model to achieve competitive object detection performance. In particular, we develop ways to transfer the ConvNet from classification to the detection environment, as well as changes to the learning procedure to enable joint training of all parts.
The first system to combine structured learning with a ConvNet is LeCun et al. [10], who train a ConvNet to classify individual digits, then train with hand written strings of digits discriminatively. Very recently, Tompson et al. [15] trained a model for human pose estimation that combines body part location estimates into a convolutional network, in effect integrating an MRF-like model with a ConvNet. Their system, however, requires annotated body part locations and is applied to pose estimation, whereas our system does not require annotated parts and is applied to object detection.
Some recent works have applied ConvNets to object detection directly: Sermanet et al. [13] train a network to regress on object bounding box coordinates at different strides and scales, then merge predicted boxes across the image. Szegedy et al. [14] regress to a bitmap with the bounding box target, which they then apply to strided windows. Both of these approaches directly regress to the bounding box from the convolutional network features, potentially ignoring many important spatial relationships. By contrast, we use the ConvNet features as input to a DPM. In this way, we can include a model of the spatial relationships between object parts.
In the R-CNN model, Girshick et al. [7] take a different approach in the use of ConvNets. Instead of integrating machines to form complete object detectors, as seen in Figure 2 (a). The various human tasks involved are summarized in Figure 2 (b). As before, we conducted these human studies on Amazon Mechanical Turk.
Our experiments concluded that part detection is the weakest link for challenging person detection datasets. Non-maximal suppression and context can also significantly boost performance. However, the use of human or machine spatial models does not significantly or consistently affect detection accuracy. A summary of the results can be seen in Figure 2 (c). This was the first analysis of its kind that provided the community valuable and concrete feedback about which specific problem to focus on to improve overall performance: in this case, classifying local image patches into one of six person-part categories.

Challenges
The key idea behind human-debugging is to replace isolated components of a machine pipe-line with human subjects. This necessitates designing studies that require humans to perform very specific tasks; whose input and outputs precisely match those used by the equivalent machine implementation. This leads to several interesting challenges.
Accessing isolated human-models: It is crucial for the information available to humans to be equivalent to that available to the machine implementation. This often involves providing information in 3 Figure 2. Reproduced from Parikh and Zitnick [11]: an ablation study of the stages in a DPM model [5] . Their figure shows how significant performance improvements could be obtained by replacing the parts detection and non-maximal suppresssion stages with human subjects. This suggests that these stages limit performance within the model. Our work focuses on improving each of these, replacing the part detectors with a Convnet and integrating NMS into the model. a location regressor into the network, they instead produce candidate region proposals with a separate mechanism, then use the ConvNet to classify each region. However, this explicitly resizes each region to the classifier field of view (fixed size), performing significant distortions to the input, and requires the entire network stack to be recomputed for each region. Instead, our integration runs the features in a convolutional bottom-up fashion over the whole image, preserving the true aspect ratios and requiring only one computational pass.
End-to-end training of a multi-class detector and postprocessing has also been discussed in Desai et al. [3]. Their approach reformulates NMS as a contextual relationship between locations. They replace NMS, which removes duplicate detections, with a greedy search that adds detection results using an object class-pairs context model. Whereas their system handles interactions between different types of objects, our system integrates NMS in a way that creates an ordering of results both of different classes and the same class but different views. In addition, we further integrate this into a full end-to-end system including ConvNet feature generation.

Model Architecture
The architecture of our model is shown in Fig. 3. For a given input image x, we first construct an image pyramid (a) with five intervals over one octave 1 We apply the Con-vNet (b) at each scale x s to generate feature maps φ A (x s ). These are then passed to the DPM (c) for each class; as we describe in Section 3.2, the DPM may also be formulated as a series of neural network layers. At training time, the loss is computed using the final detection output obtained after NMS (d), and this is then back-propagated end-to-end NMS responses for each scale Figure 3. Our model architecture, with Convolutional Network (left), Deformable Parts Model (right) and non-maximal suppression (top) components. An input x is first repeatedly downsampled to create an image pyramid (a). We run the convolutional network on each scale, by performing four layers of convolution and maxpooling operations (b.ii -b.iv). This produces a set of appearance features φA(xs) at each scale, which are used as input to a DPM (c.i). Each object class model has three views of object templates (c.ii), each of which is composed of a root filter and nine parts filters. These produce a response map Fv for each view (c.iii), which are then combined using a pixel-wise max (c.iv) to form a final activation map for the object class, F (xs, y). We then perform NMS (d) across responses for all scales. To generate bounding boxes, we trace the activation locations back to their corresponding boxes in the input. through the entire system, including NMS, DPM and Con-vNet.

Convolutional Network
We generate appearance features φ A (x) using the first five layers of a Convolutional Network pre-trained for the ImageNet Classification task. We first train an eight layer classification model, which is composed of five convolutional feature extraction layers, plus three fully-connected classification layers 2 . After this network has been trained, 2 The fully connected layers have 4096 -4096 -1000 output units each, with dropout applied to the two hidden layers. We use the basic model we throw away the three fully-connected layers, replacing them instead with the DPM. The five convolutional layers are then used to extract appearance features.
Note that for detection, we apply the convolutional layers to images of arbitrary size (as opposed to ConvNet training, which uses fixed-size inputs). Each layer of the network is applied in a bottom-up fashion over the entire spatial extent of the image, so that the total computation performed is still proportional to the image size. This stands in contrast to [7], who apply the ConvNet with a fixed input size to different image regions, and is more similar to [13].
Applying the ImageNet classification model to PASCAL detection has two scale-related problems that must be addressed. The first is that there is a total of 16x subsampling between the input and the fifth layer; that is, each pixel in φ A corresponds to 16 pixels of input -this is insufficient for detection, as it effectively constrains detected bounding boxes to a lie on a 16-pixel grid. The second is that the Im-ageNet classifier was trained on objects that are fairly large, taking up much of the 224x224 image area. By contrast, many target objects in PASCAL are significantly smaller.
To address these, we simply apply the first convolution layer with a stride of 1 instead of 4 when combining with the DPM (however, we also perform 2x2 pooling after the top ConvNet layer due to speed issues in training, making the net resolution increase only a factor of 2). This addresses both scale issues simultaneously. The feature resolution is automatically increased by elimination of the stride. Moreover, the scale of objects presented to the network at layers 2 and above is increased by a factor of 4, better aligning the PASCAL objects to the ImageNet expected size This is due to the fact that when the second layer is applied to the output of the stride-1 maps, their field of view is 4x smaller compared to stride-4, effectively increasing the size of input objects.
Note that changing the stride of the first layer is effectively the same as upsampling the input image, but preserves resolution in the convolutional filters (if the filters were downsampled, these would be equivalent operations; however we found this to work well without changing the filter size, as they are already just 11x11).

Part Responses
The first step in the DPM formulation is to convolve the appearance features with the root and parts filters, producing appearance responses. Each object view has both a root filter and nine parts filters; the parts are arranged on a 3x3 grid relative to the root, as illustrated in Fig. 4. (This is similar to [19], who find this works as well as the more complex placements used by [5]). Note that the number of root and parts filters is the same for all classes, but the size of each root and part may vary between classes and views.
Given appearance filters w root A,y,v for each class y and view v, and filters w part A,y,v,p for each part p, the appearance scores are: Part responses are then fed to the deformation layer.

Deformation Layer
The deformation layer finds the optimal part locations, accounting for both apperance and a deformation cost that models the spatial relation of the part to the root. Given appearance scores F part v,p , part location p relative to the root, and deformation parameters w D,v,p for each part, the deformed part responses are the following (input variables (x s , y) omitted): T is the shape deformation feature. w part D,y,v ≥ 0 are the deformation weights. Note the maximum in Eqn. 3 is taken independently at each output spatial location: i.e. for each output location, we find the max over possible deformations (δ i , δ j ). In practice, searching globally is unnecessary, and we constrain to search over a window [−s, s] × [−s, s] where s is the spatial size of the part (in feature space). During training, we save the optimal (δ i ,δ j ) at each output location found during forward-propagation to use during backpropagation.
The deformation layer extends standard max-pooling over (δ i , δ j ) with (i) a shift offset (p i , p j ) accounting for the part location, and (ii) deformation cost w T D φ D (δ i , δ j ). Setting both of these to zero would result in standard maxpooling.

AND/OR Layer
Combining the scores of root, parts and object views is done using an AND-like accumulation over parts to form a score F v for each view v, followed by an OR-like maximum over views to form the final object score F : (a) w root F (x s , y) is then the final score map for class y at scale s, given the image x as shown in Fig. 5(left).

Bounding Box Prediction
After obtaining activation maps for each class at each scale of input, we trace the activation locations back to their corresponding bounding boxes in input space. Detection locations in F (x s , y) are first projected back to boxes in φ A (x s ), using the root and parts filter sizes and inferred parts offsets. These are then projected back into input space through the convolutional network. As shown in Fig. 5(right), each pixel in φ A (x s ) has a field of view of 35 pixels in the input, and moving by 1 pixel in φ A moves by 8 pixels in the input (due to the 8x subsampling of the convolutional model). Each bounding box is obtained by choosing the input region that lies between the field of view centers of the box's boundary. This means that 17 pixels on all sides of the input field of view are treated as context, and the bounding box is aligned to the interior region.

Non-Maximal Suppression (NMS)
The procedure above generates a list of label assign- ..|B| } for the image, where b i is a bounding box, and y i and r i are its associated class label and network response score, i.e. r i is equal to F (x, y i ) ...  and θ = 0.5.

Motivation
Our second main contribution is the use of a finalprediction loss that takes into account the NMS step used in inference. In contrast to bootstrapping with a hard negative pool, such as in [19] [5], we consider each image individually when determining positive and negative examples, accounting for NMS and the views present in the image itself. Consider the example in Fig. 6: A person detector may fire on three object views: red, green, and blue. The blue (largest in this example) is closest to the ground truth, while green and red are incorrect predictions. However, we cannot simply add the green or red boxes to a set negative examples, since they are indeed present in other images as occluded people. This leads to a situation where the red view has a higher inference score than blue or green, i.e. r(red) > r(blue) and r(red) > r(green), because red is never labeled as negative in the bootstrapping process. After NMS, blue response will be suppressed by red, causing a NMS error. Such an error can only be avoided when we have a global view on each image: if r(blue) > r(red), then we would have a correct final prediction.

Loss Function
Recall that the NMS stage produces a set of assignments predicted by the model A = {(b i , y i , r i ) i=1...n } from the set B of all possible assignments. We compose the loss using two terms, C(A) and C(A ). The first, C(A), measures the cost incurred by the assignment currently predicted by the model, while C(A ) measures the cost incurred by an assignment close to the ground truth. The current prediction cost C(A) is: Figure 6. Three possible bounding boxes: blue, green and red (blue closest to the ground truth). However, green and red should not be considered negative instances (since they may be positive in other images where the person is occluded). Thus, we want r(blue) > r(red) r(blue) > r(green) where H(r, y) = I(y > 0) max(0, 1 − r) 2 + I(y = 0) max(0, r + 1) i.e. a squared hinge error. 3 We construct a constrained inference assignment A close to A gt by choosing for each b gt i the box (b i , y i , r i ) = argmax (b,y,r) r, where the argmax is taken over all overlap(b, b gt i ) ≥ θ ; that is, the box with highest response out of those with sufficient overlap with the ground truth. (θ = 0.7 in our experiments.) Similarly to before, the cost C(A ) is: Thus we measure two costs: that of the current model prediction, and that of an assignment close to the ground truth. The final discriminative training loss is difference between these two: Note this loss is always greater than 0 because the constrained assignment always has cost at least as large as the unconstrained one, and L(A, A ) = 0 when A = A , i.e. when we produce detection results which are consistent with the ground truth A gt . 3 where I is an indicator function that equals 1 iff the condition holds  . 7); thus we can compute the loss looking only at these neighborhood sets. Figure 7. Illustration of ground-truth-constrained assignment A and unconstrained assignments A from the model, along with associated neighborhoods. Note neighborhoods are actually dense, and we show only a few boxes for illustration.

Interpretation and Effect on NMS Ordering
As mentioned earlier, a key benefit to training on the final predictions as we describe is that our loss accounts for the NMS inference step. In our example in Fig. 6, if the response r(red) > r(blue), then red ∈ A and blue ∈ A . Thus L P (A, A ) will decrease r(red) and increase r(blue). This ensures the responses r are in an appropriate order when NMS is applied. Once r(blue) > r(red), the mistake will be fixed.
The term L N in the loss is akin to an online version of hard negative mining, ensuring that the background is not detected as a positive example.

Soft Positive Assignments
When training jointly with the ConvNet, it is insufficient to measure the cost using only single positive instances, as the network can easily overfit to the individual examples. We address this using soft positive assignments in place of hard assignments; that is, we replace the definition of C P in Eqn. 6 used above with one using a weighted average of neighbors for each box in the assignment list: Note a similar strategy has been tried in the case of HoG features before, but was not found to be beneficial [5]. By contrast, we found this to be important for integrating the ConvNet. We believe this is because the ConvNet has many more parameters than can easily overfit, whereas HoG is more constrained.

Training
Our model is trained in online fashion with SGD, with each image being forward propagated through the model and then the resulting error backpropagated to update the parameters. During the fprop, the position of the parts in the DPM are computed and then used for the subsequent bprop. Training in standard DPM models [5] differs in two respects: (i) a fixed negative set is mined periodically (we have no such set, instead processing each image in turn) and (ii) part positions on this negative set are fixed for many subsequent parameter updates.
We first pretrain the DPM root and parts filters without any deformation, using a fixed set of 20K random negative examples for each class. Note that during this stage, the ConvNet weights are fixed to their initialization from Ima-geNet. Following this, we perform end-to-end joint training of the entire system, including ConvNet, DPM and NMS (via the final prediction loss). During joint training, we use inferred part locations in the deformation layer.
The joint training phase is outlined in Algorithm 1. For each training image sample, we build an image pyramid, and fprop each scale of the pyramid through the ConvNet and DPM to generate the assignment list A 0 . Note A 0 is represented using the output response maps. We then apply NMS to get the final assignments A, as well as construct the ground-truth constrained assignments A . Using the final prediction loss L(A, A ) from Eqn. 9, we find the gradient and backpropagate through the network to update the model weights. We repeat this for 15 epochs through the training set with a learning rate η = 10 −3 , then another 15 using η = 10 −4 .
At test time, we simply forward-propagate the input pyramid through the network (ConvNet and DPM) and apply NMS.

Experiments
We apply our model to the PASCAL VOC 2007 and VOC 2011/2012 object detection tasks [4]. Table 1 shows Algorithm 1 Training algorithm for each image 1: Input: Image X with ground truth assignment A gt 2: Build image pyramid X → X 1 , X 2 , . . . , X s 3: A 0 = {} 4: for X i ∈ X 1 , X 2 , . . . X s do

5:
A 0 = A 0 ∪ assignments from responses F (X i ; w) 6: end for 7: find A = N M S(A 0 ) 8: find A using A 0 and A gt 9: for X i ∈ X 1 , X 2 , . . . X s do 10: find gradient at scale i: Note that this allows the post-NMS loss to account for objects of different classes as well as locations and views within classes, and also makes inference faster due to the shared features. We call this model "conv-dpm+FT-all", and the separate-class set of models "conv-dpm+FT". Comparisons with other systems are shown in Tables 2  (VOC 2007)  . Notably, we outperform the DP-DPM method (45.2% vs. our 46.9%), due to our integrated joint training and online NMS loss. In addition, our final model achieves comparible performance to R-CNN [7] with a similar feature extractor using pool 5 features (46.9% vs. 47.3%). Recent version of R-CNN achieve a better performance 54.2% using a more complex network which includes fully connected layers (f c 7 ); extending our model to use deeper networks may also provide similar gains from better feature representations. Table 3 shows our system performance on VOC2011. Here our system outperforms comparison methods, and in particular DP-DPM, achieving 43.7% mAP versus 29.6% for HoG-DPM [5] and 41.6% for DP-DPM [8]. Finally, we provide examples of detections from our model in Figures 8, 9 and 10. Detection results are either show in green or red with ground truth bounding box in blue. Figure 10 illustrates training with our new loss function helps model fix problem for both inter-class and intraclass NMS. Our loss allows the larger view of the train to be selected in (b), rather than the more limited view that appears in more images. However, the gains are not limited to selecting larger views: In (d), we see a cat correctly selected at a smaller scale. Finally, there are also examples of inter-class correction in (g), e.g. "train" being selected over "bus". Figure 8 shows the effect of using a DPM with parts over just the root-only model.

Discussion
We have described an object detection system that integrates a Convolutional Network, Deformable Parts model and NMS loss in an end-to-end fashion. This fuses together aspects from both structured learning and deep learning: object structures are modeled by a composition of parts and views, while discriminative features are leveraged for appearance comparisons. Our evaluations show that our model achieves competitive performance on PASCAL VOC 2007 and 2011 datasets, and achieves substantial gains from integrating both ConvNet features as well as NMS, and training all parts jointly.  Table 3. Mean AP on PASCAL VOC 2011 Figure 9. Examples of correct (green) and incorrect (red) detections found by our model.