{'fraction_non_alphanumeric': 0.029894213884962953, 'fraction_numerical': 0.02107156623409996, 'mean_word_length': 4.510974746282748, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Single-Shot 3D Detection of Vehicles from Monocular RGB Images via Geometry Constrained Keypoints in Real-Time

In this paper we propose a novel 3D single-shot object detection method for detecting vehicles in monocular RGB images. Our approach lifts 2D detections to 3D space by predicting additional regression and classification parameters and hence keeping the runtime close to pure 2D object detection. The additional parameters are transformed to 3D bounding box keypoints within the network under geometric constraints. Our proposed method features a full 3D description including all three angles of rotation without supervision by any labeled ground truth data for the object's orientation, as it focuses on certain keypoints within the image plane. While our approach can be combined with any modern object detection framework with only little computational overhead, we exemplify the extension of SSD for the prediction of 3D bounding boxes. We test our approach on different datasets for autonomous driving and evaluate it using the challenging KITTI 3D Object Detection as well as the novel nuScenes Object Detection benchmarks. While we achieve competitive results on both benchmarks we outperform current state-of-the-art methods in terms of speed with more than 20 FPS for all tested datasets and image resolutions.


I. INTRODUCTION
Object detection -both in 2D as well as in 3D -is a key enabler for autonomous driving systems. To this end, autonomous vehicles that are currently in development as well as consumer cars that provide advanced driver assistance systems are equipped with a decent set of sensors such as RGB cameras, LiDARs as well as Radar systems. While the accurate distance measurement of LiDAR sensors enables robust 3D bounding box detection, the high cost may prohibit their use in series production vehicles. 3D object detection from monocular RGB cameras thus became a focus in recent computer vision research. In contrast to LiDAR measurements, RGB images provide rich semantic information that can be used to boost object classification. One of the most challenging problems in 3D object detection from monocular RGB images is the missing depth information. A neural network thus needs to accurately estimate depth from monocular RGB images. Furthermore, real-time performance is required to enable the use of an algorithm in an autonomous vehicle.
In this paper we present a novel technique to detect vehicles as 3D bounding boxes from monocular RGB images by 1   transforming a set of predicted regression and classification parameters to geometrically constrain 3D keypoints called 3D-GCK. In contrast to other 3D bounding box estimators 3D-GCK is capable of predicting all three angles of rotation (θ, ψ, φ) which are required for a full description of a 3D bounding box. 3D-GCK focuses only on keypoints in the image plane and exploits the projection properties to generate 3D orientation information. Hence, no labeled ground truth for the angles of rotation is required to train 3D-GCK which facilitates the collection of training data.
We use a standard single-shot 2D object detection framework -in our case SSD [2] -and add the proposed extension to lift the predicted 2D bounding boxes from image space to 3D bounding boxes. Lifting 2D bounding boxes to 3D space can be done with minimal computational overhead leading to real-time capable performance.
We summarize our contributions as follows: 1) We introduce 3D-GCK which can be used with all current stateof-the-art 2D object detection frameworks such as SSD [2], Yolo [3] and Faster-RCNN [4] to detect vehicles and lift their 2D bounding boxes to 3D space. 2) We exemplary extend SSD with the proposed 3D-GCK architecture to accentuate the practical use of 3D-GCK. 3) We evaluate 3D-GCK on 4 challenging and diverse datasets which are especially tailored for autonomous driving: KITTI, nuScenes, A2D2 and Synscapes. We achieve competitive results on the publicly available KITTI 3D Object Detection and nuScenes Object Detection benchmarks. At the same time 3D-GCK is the fastest 3D object detection framework that relies exclusively on monocular RGB images.

II. RELATED WORK
Our work is related to different approaches for monocular 3D object detection. In the following section, we divide the relevant methods into three typical streams of research. First, methods that suggest novel 3D prior bounding box generations or losses. Second, methods that explore geometric reasoning and shape reconstruction. Third, methods that transform input data or feature representations.

A. Novel 3D Prior Bounding Box Proposals or Losses
Mono3D [5] exhaustively generates possible 3D region proposals based on predictions from class semantics, contour, shape and location priors, leading to slow inference speed. ROI-10D [6] concatenates features from a 2D Region Proposal Network (RPN) as proposed in [4] and depth network in a differentiable Region Of Interest (ROI) lifting layer. These features are used to generate sparse 3D prior boxes, which are optimized w.r.t. their ground truths in 3D space. [7] proposes 3D bounding box priors by utilizing 2D prior bounding boxes for object centroid estimation and predicted point clouds for object scale and shape estimation. M3D-RPN [8] simultaneously creates 2D and 3D bounding box priors by pre-computing the mean statistics of 3D parameters for each prior individually and learns location-specific features by using 2D convolutions of non-shared weights for joint prediction of 2D and 3D boxes. [9] conducts two-stage 3D object detection by disentangling the dependencies of different parameters. This is achieved by handling groups of parameters individually at loss-level.
Most relevant to our work is GS3D [10], where visible surface features are extracted from the 2D image plane and perspectively projected to regular shape. In combination with 2D bounding box features, these features are then used for refining the rough initial 3D bounding box proposals, which are based on the estimation of the class, size and local orientation of 2D bounding boxes.
Compared with GS3D [10], our novel keypoint representation serves twofold functions. First, it enhances the initialization of the 3D bounding box by leveraging knowledge from 2D image plane that we use to dynamically determine the prior global orientation and dimension of the 3D bounding box. Moreover, these keypoints are employed as geometric constraints to optimize the 3D bounding box estimation.

B. Geometric Reasoning and Shape Reconstruction
Geometry is often employed as a constraint to alleviate the 3D object detection problem. Deep3DBox [11] and Shift R-CNN [12] leverage translation constraints provided by the predicted 2D bounding box by forcing the predicted 3D bounding box to fit tightly within the 2D bounding box using a system of linear equations. In addition to the prediction of dimensions and orientation of the 3D objects, [13] also classifies the viewpoint and regresses on the center projection of the bottom face of the 3D bounding box, which are used to jointly fit the projection of 3D bounding boxes to predicted 2D bounding boxes. Mono3D++ [14] performs 3D vehicle detection by optimizing both coarserepresented 3D bounding box and fine-grained morphable wireframe. To this end, a ground plane assumption and vehicle shape priors are used. Instead of per-pixel depth estimation, MonoGRNet [15] performs instance-level depth estimation for 3D detection. It explores geometric features to refine the 3D center location of objects by backprojecting the 2D projection of the estimated 3D center to the 3D space based on the estimated depth. [16] employs the multi-view representation to regress projected 3D bounding boxes into 2D.
In BS 3 D [17] the bounding shape representation, consisting of four coarsely constrained key points, is learned end-to-end for reconstructing 3D bounding box in a postprocessing step. [18] predicts coordinates and a visibility state for 14 predefined 2D keypoints and estimate local orientation based on multi-bin classification. It then learns to estimate correspondences between the 2D keypoints and their 3D counterparts annotated on 3D CAD models for solving the object localization problem. DeepManta [19] uses 3D CAD models and annotated 3D parts in a coarse-to-fine localization process, while [20] jointly learns the detailed amodal 3D shape and pose by exploring inverse graphics.
In contrast, we select four keypoints directly from the set of 3D bounding box vertices and hence do not rely on keypoints on object level. As a result, we do not depend on any CAD models. These four selected keypoints serve as geometric constraints jointly optimizing the localization, dimensions, and 3D orientation of the 3D bounding box detection.

C. Input Data or Feature Representation Transformation
3D object detection can be facilitated by directly transforming the input data representation -such as from monocular RGB images to 3D pseudo point clouds -or feature representation. [21] restores distance information by projecting the front image onto a bird's-eye-view (BEV) image after correcting the motion change of the ego vehicle by utilizing the extrinsic parameters of the camera and inertial measurement unit information. A one-stage object detector, Yolov3 [3], is then used to determine the class, location, width and height and the orientation of objects. [22] uses a GAN [23] to generate 3D data. [24], [25] focus on generating a pseudo point cloud by using stereo disparity estimation. [26] transforms the estimated depth into a point cloud with the assistance of camera calibration, followed by a multimodal feature fusion module embedding the complementary RGB cue into the generated point cloud representation. [27] extracts a set of point cloud frustums based on the generated pseudo point clouds and 2D object proposals in the input image, which are passed to train a state-of-the-art two-stage LiDAR 3D bounding box detector [28]. [29] introduces an orthographic feature transform operation that populates the 3D voxel feature map with features from the image-based feature map. [30] performs multi-level fusion both on feature and data representation for 3D object detection.

III. MODEL
While standard 2D object detection frameworks are solely built upon a classification and a bounding box regression part, several adjustments are required to lift 2D bounding boxes to 3D ones. Fig. 2 illustrates an overview about the additional prediction modules that are used in our network architecture. In addition, it depicts the required steps to generate a 3D bounding box using these predictions and their corresponding loss function.
In general 3D-GCK follows a 3-step-scheme to lift a 2D detection into 3D: 1) Estimation of a 3D bounding box initialization.
2) Refinement of the aspect ratios and size.
3) Refinement of the angles of rotation.

A. Coordinate Frame
We define a coordinate frame for the object of interest with the setup shown in Fig. 3. O is located on the bottom of the object and is chosen to be the point which is closest to the camera. A is defining the width of the object relative to O while B and C are used to determine the height and the length of the object, respectively.

B. Network Prediction
In 3D-GCK there are 3 different types of prediction modules as shown in Fig. 2 (left): 2D Bounding Boxes: Box Init is a standard 4-parameter 2D bounding box regressor that is defined by A x , B y , C x , O y as shown in Fig. 3. In addition, Box Pix will be trained to predict only the visible parts of the actual vehicle. However, this bounding box regressor is solely used for selecting the best 2D predictions in the underlying 2D object detection framework using vg-NMS [31] and is not mandatory for the actual 3D bounding box generation.
Regression: In addition to the standard 2D bounding box regression, multiple additional parameters are regressed in 3D-GCK.
The side ratio (S-Ratio) determines the extent to which the side of the vehicle is visible in contrast to the front or the back as shown in Fig. 3 with a value range between 0 and 1. This value is used to generate an initialization for the yaw angle of the actual 3D bounding box. While the initial yaw angle is dynamically calculated, the initial pitch and roll angles are fixed to the camera extrinsics.
In a next step, the initial angles are refined. ∆ Angles is a 3 dimensional vector that represents the offsets of these initially calculated 3D orientations given in radians. ∆ Aspect is a 2 dimensional vector that is used for adjusting the aspect ratios w/h as well as l/h where l, h, w refer to the object's length, height and width, respectively.
As the 3D bounding box requires a valid depth and 3D-GCK operates exclusively on monocular RGB images, the depth of the bounding box needs to be estimated as well. To this end, the inverse euclidean distance from the camera to the origin of the vehicle O is estimated.
Classification: In addition to the actual object's classifi-

C. 3D Box Generator
In the 3D Box Generator of the 3D-GCK architecture, the predicted parameters are transformed to a real 3D bounding box for each detection. To this end three matrices are calculated that represent scaling S, translation T and rotation R. The dependencies for each calculation are illustrated in Fig. 2 (middle).
Translation: For each Box Init the 2D coordinates of the origin O are calculated as follows: with w 2D being the width of Box Init . To determine the 3D position (δ x , δ y , δ z ) of O, inverse graphics is used under the constraint that the distance of O 3D is the predicted depth d. For inverse graphics the camera projection matrix P is needed. Finally, T = 1|(δ x , δ y , δ z , 1) T is the homogeneous translation matrix. Scaling: As the 3D position of O is known, Box Init,ymin can be used to calculate the 3D height h: However, Eq. 3 is an approximation that is only valid for small pitch and roll angles ψ and φ which is a reasonable assumption for most of the driving scenarios occurring in real world situations. To calculate both length l and width w of the 3D bounding box, class dependent prior aspect ratios AR Prior are used that In our experiments we use the prior aspect ratios AR Prior,l ≡ l/h = 2.8 and AR Prior,w ≡ w/h = 1.1 for the class Car.
The homogeneous scaling matrix S is finally calculated as S = 1 · (l, h, w, 1) T . Rotation: Analogous to the scaling values, prior rotation values are estimated that are refined using ∆ Angles . In an first order approximation the street in front of the ego vehicle is parallel to vehicle coordinate frame. Hence, prior values for pitch ψ Init and roll φ Init are set to the inverse of the camera extrinsics -i.e. if the camera is mounted with a pitch value of 10 • the prior pitch for the 3D bounding boxes is −10 • .
While these assumption are valid for both pitch as well as roll, calculation of the prior value for yaw θ Init directly involves the object of interest. If S-Ratio=0 the object's heading is close to the angle of the camera ray θ Cam . If S-Ratio=1, it is close to θ Cam ± π 2 . Hence, it is Finally, the prior angles are refined using ∆ Angles to determine the actual angles of the 3D bounding box. Furthermore, the yaw angle θ is corrected by +π in case the front of the object is visible: The joint rotation matrix R is then calculated and the homogeneous rotation matrix R computes to R = R 0 0 1 . 3D Bounding Box: As a last step an initial 3D bounding box with size of 1 m×1 m×1 m is transformed using S, R and T . This 3D bounding box is projected in the image plane and the xand y-offsets for A, B, C and O of the predicted 3D bounding box are calculated which will be later used in the loss calculation. The offsets are values between 0 and 1 and determine the relative position of the keypoints in Box Full . Box Full is the 2D bounding box enclosing the 8 projected vertices of the 3D bounding box.

D. Loss
For standard 2D object detection the overall loss is defined as L = τ L cls + αL loc (12) with L cls and L loc denoting the classification and the box localization loss. We extend the loss by adding further loss terms corresponding to the predictions and the final bounding boxes as well as the offsets: L = L boxes + L params + L class (13) L boxes = αL loc,init + βL loc,full + γL loc, pix (14) L params = ζL s-ratio + ηL depth + κL offsets + µL a-ratios (15) L class = νL F/B + ξL L/R + τ L cls .
For all 2D box regression losses we use cos(IoU) and for all parameters specific losses we use L 2 , except for the depth regressor where a smooth L 1 loss is used. For L F/B and L L/R , sigmoid cross entropy loss and for the object's classification L cls focal loss [33] is employed as shown in the network overview Fig. 2 (right).

IV. EXPERIMENTS A. Dataset
We evaluate 3D-GCK using several different and challenging datasets tailored especially for autonomous driving purposes.
KITTI [34] contains approx. 15 000 images and offers a public benchmark for 3D object detection. However, the images have a rather small resolution compared to more recent datasets like nuScenes [1] or A2D2 [35]. The nuScenes dataset features 1000 distinct scenes in 40 000 annotated frames and additionally offers a public benchmark for monocular 3D object detection. A2D2 contains 12 000 annotated frames.
In contrast to KITTI, nuScenes and A2D2, Synscapes is a synthetic dataset with 25 000 photo-realistic rendered images. While real world datasets usually only provide annotations for the yaw angle θ, synthetic datasets like Synscapes provide a full description of the 3D object including pitch and roll orientations. As 3D-GCK is capable of predicting all 3 angles of rotation, we also evaluate on Synscapes.
Since A2D2 and Synscapes do not offer an official trainval split and a test set, we split the data such that 80 % of all images are used for training while 20 % are used for evaluation. We make sure that all images of a recorded sequence are contained either in the train or the val set.

B. Experimental Setup
For each dataset we train a single network such that we do not share data of different datasets across the experiments. We use InceptionV1 [36] as the backbone network for our extended 3D-GCK SSD architecture. While other backbone networks like ResNet [37] may result in superior detection performances [38], [39], InceptionV1 offers a good trade-off between speed and accuracy. We use the Adam optimizer with an initial learning rate of lr = 0.0001. We found that a loss weight of 1 for all loss weights except for the class and depth weights τ = 2 and η = 0.5 yields high detection rates and stable training. However, we schedule the training such that the network first focuses on pure 2D SSD box detection and therefore set all weights initially to 0 except for α and τ . The other weights are logistically increased during the first 100k iterations until they reach their final value. This ensures a high 2D detection performance which is required for a good performance in 3D object detection as 3D-GCK lifts 2D detections to 3D space. Furthermore, we add a cosine learning rate decay to improve the overall network performance and stability [40]. MonoDis [9] AM3D [26] GS3D [10] ROI-10D [6] 3D intersection over union (IoU) of at least 70 % is required to accept a 3D bounding box as True Positive (TP). For a 3D bounding box representing an average car with dimensions (l/w/h) = (4.7 m/1.8 m/1.4 m) a shift of 13 cm in each direction already results in an IoU < 0.7 leading to both a F P and a F N . While such an accurate localization estimation is easy to achieve for LiDAR based methods such as [41], [42], it is significantly more difficult for approaches that are solely built upon monocular RGB input as predicting depth from monocular images is a challenging task [43], [44]. Furthermore, Average Orientation Similarity (AOS) is used to assess the ability of 3D-GCK to correctly predict the object orientation [32]. AOS is calculated similar to AP but uses a normalized cosine similarity s(r) instead of p(r) for p interp in Eq. 18:

C. Metrics
where D(r) is the set of all detections at recall r.
In addition, we evaluate our approach on the public nuScenes Object Detection benchmark. In contrast to the KITTI benchmark, a prediction is accepted as T P if the 2D center distance d is below a certain threshold t ∈ {0.5, 1, 2, 4} m.

D. Results
In the KITTI 3D Object Detection benchmark we achieve a 3D AP for class Car of 2.52 % and an AOS of 78.44 % in the moderate category which is competitive to other methods that purely process monocular RGB images and lift 2D detections to 3D space. In general, architectures that enrich and lift 2D detections are comparable fast: 3D-GCK runs in 24 ms (> 40 FPS) on an NVidia V100 Tesla GPU without further hardware optimization. 3D-GCK therefore

Dataset Resolution Runtime
KITTI [32] 1245 px×375 px 24 ms nuScenes [1] 1600 px×900 px 43 ms A2D2 [35] 1920 px×1208 px 46 ms Synscapes [45] 1440 px×720 px 36 ms offers an exceptional trade-off between speed and detection performance as shown in Fig. 5. While other approaches achieve better detection performance they suffer from a slow inference speed as these methods directly explore in 3D space leading to an increased complexity compared to 3D-GCK.
Quantitative results on the KITTI 3D Object Detection benchmark is listed in Tab. I. In the official nuScenes Object Detection benchmark, 3D-GCK achieves 11.4 % AP for class Car.
We applied the same evaluation metrics as used in KITTI for the nuScenes val, A2D2 val and Synscapes val datasets and obtain similar results as illustrated in Tab. II. However, we combined the categories moderate and hard as not all required ground truth information is available for each dataset. Tab. III illustrates the runtime of 3D-GCK for the different datasets. 3D-GCK is able to run also with more than 20 FPS on datasets with high resolution images like nuScenes and A2D2.
Qualitative results for each dataset are shown in Fig. 6.

V. CONCLUSIONS
In this paper we proposed a novel method for detecting vehicles as 3D bounding boxes for autonomous driving purposes via geometrically constrained keypoints. The basis for this approach is a 2D bounding box detection framework. The 2D detections are then lifted into 3D space. The proposed method can be universally applied to current state-of-the-art 2D object detection frameworks with little computational overhead keeping the runtime close to pure 2D object detection. We added the proposed 3D-GCK extension to SSD and evaluated our approach on different datasets and benchmarks and achieved competitive results on the challenging KITTI 3D Object Detection and the novel nuScenes Object Detection benchmarks. At the same time our approach is extremely fast and runs with more than 20 FPS even on high resolution images without GPU Fig. 6. Qualitative results for KITTI (top, taken from test set), nuScenes (2nd row, from test set), A2D2 (3rd row, from val set) and Synscapes (bottom, from val set). For each image the corresponding bird's-eye-view shown on the right. specific hardware optimization. Hence, our approach offers an exceptional trade-off between speed and accuracy and is suitable for being used in productive autonomous systems.