{'fraction_non_alphanumeric': 0.029184605777763172, 'fraction_numerical': 0.011272882637131495, 'mean_word_length': 4.468727534148095, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
3D Structure from Detections

We present a novel method to infer, in closed-form, the 3D spatial occupancy of a collection of rigid objects given 2D image detections from a sequence of images. In particular, starting from 2D ellipses fitted to bounding boxes, this novel multi-view problem can be reformulated as the estimation of a quadric (ellipsoid) in 3D. We show that an efficient solution exists in the dual-space using a minimum of three views. This algebraic solution can be negatively affected in the presence of gross inaccuracies in the bounding boxes estimation. To this end, we also propose a robust regularization method and a robust ellipse fitting algorithm able to improve performance in the presence of errors in the detected objects. Results on synthetic tests and on different real datasets, involving real challenging scenarios, demonstrate the applicability and potential of our method.

Localising objects in generic scenes is a fundamental step for higher level scene understanding. This task is of such importance that major efforts have been put in Computer Vision research in order to obtain efficient detectors from single images. These methods give remarkable results in finding several objects categories in natural scenes and thus putting the basis for the understanding of the visual world. However, while these results are certainly compelling, detection algorithms have been mostly restricted to the estimation of the objects position in 2D. When imaging the same scene from different viewpoints, a crucial question is how to deal with both 3D geometrical reasoning and higher-level object representation.
Recent works have clearly pointed out that this lack of 3D reasoning is limiting and that bridging the gap between object detection and multi-view geometry might provide surprising improvements in classical approaches. Starting from the work of Hoeim et al. [13], the inclusion of 3D scene reasoning and rules has provided higher detection accuracies. Notably, attempts of unifying geometry and object Figure 1: Example of a set of images of a given 3D rigid scene taken from a camera at different viewpoints. The problem is to recover the 3D occupancy of each object given the 2D BBs detected at each image frame. representation have been achieved by defining elaborated Maximum a Posteriori (MAP) inference [2] or bundle adjustment with objects [7]. This way of pursuing high-level object reasoning in multi-view geometry has also inspired novel methods in Simultaneous Localisation and Mapping (SLAM) [16] where 3D objects are detected and localised in real time. Semantic information has been used, on the other hand, to infer the 3D shape of objects and the cameras viewpoints [20].
Differently, several works discarded multi-view relations attempting to solve 3D pose and shape estimation from a single image only. This severely under-constrained problem has to be solved with the use of strong semantic information (geometrical and physical constraints, i.e. object lying on the ground plane), 2D appearance and shape in the form of 3D wireframe or CAD models [17,22,23,21]. Hejrati and Ramanan [10] instead use an analysis by synthesis approach that, guided by visual evidence, selects matching HOG patches which best represent a 3D object.
This work takes a different path from previous methods by showing that there is a sound and effective solution for the 3D localisation of objects given only a set of image detections, i.e. 2D bounding boxes (BBs), using multi-view geometry constraints. To the best of the authors knowledge, this work is the first to answer this problem in 3D scene understanding. The solution proposed here recalls standard Structure from Motion (SfM) approaches [19,18] where multiple views of the same rigid scene are used to obtain a 3D reconstruction given a set of 2D feature points measurements. Similarly we define a novel problem, namely Structure from Detections (SfD), that attempts to estimate an occupancy boundary that localise an object in 3D from a set of BBs obtained by general purpose detectors applied on multiple views (see Fig. 1 for a graphical representation). It is worth noting that the occupancy boundary encapsulates the estimation of object center, pose, size and aspect ratio in 3D. Moreover, differently from the large majority of works, we do not need geometric or semantic priors out of 2D bounding boxes from detections, nor advanced detectors yielding pose categorization.
In particular, we show that there exists an efficient closed-form solution to 3D object localisation if the problem is reformulated as the estimation of a quadric in 3D. This solution is not viable to derive using BBs (which is a piecewise-defined curve), but is mathematically feasible given a set of ellipses fitted to the original 2D BBs as extracted by the detectors. Given that the positions of the BBs can often be inaccurate, we propose a regularization method, based on priors on the object's aspect ratios that fixes the possible ill-conditioning of the problem. Moreover we introduce a robust segmentation algorithm that can increase the reliability of the 2D ellipse fitting stage by finding the object profile inside the BBs. In such a way, improved results are achieved in challenging real scenarios on different freely available datasets.
The paper is structured as follows. Section 1 defines the problem and the related mathematical formalisation. Section 2 presents the SfD solution while Section 3 describes the regularization method and robust ellipse fitting approach. Experiments on real and synthetic data are discussed in Section 4 and then followed by concluding remarks in Section 5.

SfD problem statement
Let us consider a set of image frames f = 1 . . . F representing a 3D scene under different viewpoints. A set of i = 1 . . . N rigid objects is placed in arbitrary positions and each object can be detected in each of the F images. Each object i in each image frame f is identified by a 2D BB B if given by a generic object detector. The BB is defined by a triplet of parameters: where w if and h if are two scalars defining the BB height and width respectively and c if is a 2-vector defining the BB center.
Our goal is to estimate the position and volume occu- pancy of each object in the 3D scene given the 2D BBs using multi-view constraints. In order to ease the mathematical formalization of the problem, we move from a BB representation of an object to an ellipsoid one. This is done by associating at each B if an ellipse fittingĈ if that inscribes the BB, as shown in Fig. 2.
The aim of our problem is to find the 3D ellipsoids Q i whose projections on the image planes, associated to each frame f = 1 . . . F , best fit the 2D ellipsesĈ if in the image plane . This will solve for both the 3D localisation and occupancy of each object starting from the image detections in the different views.
In the following, we represent each ellipse using the homogeneous quadratic form of a conic equation: where u ∈ R 3 is the homogeneous vector of a generic 2D point belonging, to the conic defined by the symmetric ma-trixĈ if ∈ R 3×3 . The conic has five degrees of freedom, given by the six elements of the lower triangular part of the symmetric matrixĈ if except one for the scale, since Eq. (1) is homogeneous in u [9]. Similarly to the ellipses, we represent the ellipsoids in the 3D space with the homogeneous quadratic form of a quadric equation: where x ∈ R 4 represents an homogeneous 3D point belonging to the quadric defined by the symmetric matrix Q i ∈ R 4×4 . The quadric has nine degrees of freedom, given by the ten elements of the symmetric matrix Q i up to one for the overall scale. Each quadric Q i , when projected onto the image plane, gives a conic denoted with C if ∈ R 3×3 . The relationship between Q i and C if is defined by the projection matrices P f ∈ R 3×4 associated to each frame. Such matrices can be estimated from the image sequence using standard selfcalibration methods [9,15].

Dual space fitting
Since the relationship between Q i and C if is not straightforward in the primal space, i.e. the Euclidean space of 3D points (2D points in the images) , it is convenient to reformulate it in dual space, i.e. the space of the planes (lines in the images) [3]. In particular, the conics in 2D can be represented by the envelope of all the lines tangent to the conic curve, while the quadrics in 3D can be represented by the envelope of all the planes tangent to the quadric surface. Hence, the dual quadric is defined by the matrix where adj is the adjoint operator, and the dual conic is defined by C * if = adj(C if ) [9]. Considering that the dual conic C * if , like the primal one, is defined up to an overall scale factor β if , the relation between a dual quadric and its dual conic projections C * if can be written as: In order to recover Q * i in closed form from the set of dual as the vectorization of symmetric matrices Q * i and C * if respectively 1 . Then, let us arrange the products of the elements of P f and P f in a unique matrix G f ∈ R 6×10 as follows [11]: where ⊗ is the Kronecker product and matrices D ∈ R 6×9 and E ∈ R 16×10 are two matrices such that vech(X) = D vec(X) and vec(X) = E vech(X) respectively, where X is a symmetric matrix 2 .
Given G f , we can rewrite Eq. (3) as: In order to get a unique solution for v * i at least three image frames are needed. Therefore, stacking column-wise Eqs. (5) for f = 1 . . . F , with F ≥ 3, we obtain the following linear system: where the matrix M i ∈ R 6F ×(10+F ) and the vector w ∈ R 10+F are defined as follows: 1 The operator vech serializes the elements of the lower triangular part of a symmetric matrix, such that, given a symmetric matrix X ∈ R n×n , the vector x, defined as x = vech(X), is x ∈ R g with g = n(n+1) 2 . 2 The operator vec serializes all the elements of a generic matrix.
and 0 d being a zero column vector of length d.
Note that in real cases the ellipsesĈ if computed by a general purpose object detector might be inaccurate regarding the location of the BB and the window size. Likewise, this will have an effect on the ellipse fitting, inducing an error on the C if . For this reason, ifM i is the matrix given by real object detections we can find the solution by minimizing:w were the equality constraint w i 2 2 = 1 avoids the trivial zero solution. The solution to the minimization problem in Eq. (9) can be obtained via SVD on theM i matrix, taking the right singular vector associated to the minimum singular value. The first 10 rows ofw i are the vectorized elements of the estimated dual quadric denoted byṽ * i . To get back the estimated matrix of the quadric in the primal space, we obtain first the dual estimated quadric byQ * , and subsequently apply the following relation:

Regularisation and robust ellipse fitting
Image detectors can provide inaccurate results given occlusions, illumination variations and complex object poses. To this end, we propose a regularisation approach for Eq. (9) and a method to re-adjust the object orientation and size from the BBs extracted in the image sequence.

Regularized cost function
As seen in the previous section, moving to the dual space allows an efficient linearisation of the problem. However this implies a drawback: the algebraic minimization is carried on the dual quadrics in Eq. (9) and the primal one is obtained by a matrix inversion as in Eq. (10). In the presence of noise the matrixQ * i may become ill-conditioned and therefore small errors may cause relatively large errors in the estimated primal quadric. This problem is particularly evident when few camera views are spanning a limited range of angles. In such a case, the estimated quadric may result in a nearly degenerate ellipsoid, with dramatically unbalanced axes lengths, or even in wrong quadrics such as an hyperboloid.
To tackle this problem, we propose to add a regularization term that penalizes the departure of the estimated quadric from a sphere of a given size and center. In other words we enforce a prior on the aspect ratios of the objects modeled by the ellipsoids. A sphere in the dual space, centered at the origin, can be represented as a 4×4 diagonal matrix with the first three diagonal elements positive and equal each other and the fourth diagonal element negative. To account for arbitrary translations, it is sufficient to pre-and post-multiply the diagonal matrix by a translation matrix. Hence a generic sphere in the dual space can be written as a function of a vector of five parameters s = [t 1 , t 2 , t 3 , a, b] as: with a > 0 and b > 0. Thus, the regularization term can be defined as: wherev * i is the vectorized i-th dual quadric normalized by minus its last (tenth) element giving: Such normalization avoids the trivial solution when minimizing for Eq. (13). The minus sign has been set in order to preserve the coherence with the constraint b > 0.
Finally, the solution to the regularized problem can be found solving for: Note that the normalization in Eq. (14) avoids the need for the quadratic equality constraint w i 2 2 = 1 in Eq. (15). Finally, the cost function in Eq. (15) is minimized with a nonlinear least squares procedure.
Initialisation. Since the convexity of the cost function cannot be guaranteed, a good initialization is mandatory, in particular for the elements of the dual quadricv * i . First of all we evaluated the sphere with the same center and the same volume of the quadric represented byṽ * i . Next, we initialized bothv * i and the parameters s according to such a sphere. In detail, let us denote the starting guess values as a (0) and b (0) given by: where e i1 , e i2 and e i3 are the three elements of a vector e i defined as: where eig() is the operator that computes the eigenvalues of a square matrix andQ i,3×3 is the 3 × 3 principal submatrix of the primal quadricQ i found by Eq. (10) 3 . If the initialQ i is not an ellipsoid, obviously the concept of volume preservation does not hold any more. However the initialization strategy, thanks to the modulus in Eq. (17) guarantees to start from a feasible solution corresponding to a sphere. The initialized translation terms t

Ellipse fitting from image segmentation
In general, the BBs from generic object detectors are not precisely aligned with the true object center and often they include a relevant portion of background. More importantly, BB axes are aligned by construction to the image axes. Thus, the related ellipses are aligned to these axes as well: this results in a relevant rotation mismatch between the real ellipse enclosing the object in the image and the one given by the detector. This situation is even more complex when the ground truth ellipses have both strong axes rotation and eccentricity: the ground truth ellipses are badly approximated by the ellipse obtained from the BB, both in terms of rotation, shape and area (as in Fig. 3(a)).
(a) (b) Figure 3: (a) Example of mismatch between ground truth ellipse (blue) and ellipse from BB (red); (b) segmented object using SCA algorithm with ellipse fit to the segmentation (red) and ground truth ellipse (blue).
To cope with these limitations, we design an approach to obtain a tighter and more reliable ellipse fitting on the object inside the BB. In particular, we adopt a variation of the method proposed in [14], based on spatial correlations 3 We recall that the volume of an ellipsoid is proportional to 1 |e i1 e i2 e i3 | and the volume of a sphere is proportional to among different image patches, to produce a binary segmentation mask. Here we exploit the fact that the pixel color and intensity distribution of the object in a set of frames are often sufficiently different from the background to allow an accurate object segmentation. Moreover as images come from a video sequence, the pose of the objects smoothly changes from one frame to another and these are the optimal conditions for stel component analysis -SCA algorithm [14]. SCA is a co-segmentation algorithm that segments images in S segments by learning K components of the model. Such components represent peculiar poses of an object and they are blended together to create a segmentation prior that well adapts to the different poses of the object in the image. The final segmentation is obtained by applying this flexible prior to a GMM-based segmentation. Given the binary mask localising the object in each BB, we extract the object orientation and size from the covariance matrix Cov if of the 2D spatial distribution of the pixels belonging to the object i in frame f . Now we can fit a 2D ellipse with axes aligned along the two eigenvectors of Cov if and with their magnitude proportional to the square root of its eigenvalues (i.e. λ j if with j = 1, 2). In particular the magnitude of the ellipse's axes is set making the second central moments of the region enclosed by the ellipse equal to the second central moments of the segmented region. An example of this segmentation process is displayed in Fig.  3(b) where the matching between ground truth and fitted ellipses is by far improved with respect to the ellipse from BB in Fig. 3(a).

Experiments
The proposed method has been tested on a synthetic scenario and on three real datasets. In all the tests, the accuracy of the results was measured by the volume overlap between ground truth and estimated ellipsoids (O 3D ) defined as follows: where Q i andQ i denote the volume of the ellipsoids given by the matrices Q i andQ i respectively. When the i-th estimated quadric is not an ellipsoidQ i is set to zero. Notice that, when camera views are restricted to a small range of angles or they are related to quasi-planar trajectories, O 3D could give poor results even with a small algebraic error in Eq. (9). This might happen because of the ill-conditioning of the problem, yielding near degenerate solutions corresponding to ellipsoids with strong eccentricity. Nevertheless, we chose such metric since it measures in a direct way the success of the algorithm in recovering the 3D position and occupancy of an object. In the following we will denote the non regularized solution obtained with SVD (Sec. 2) as SfD, the regularized solution (Sec. 3.1) as SfD+REG, the solution obtained from ellipses fitted to segmented images (Sec. 3.2) as SfD+ES (Ellipse Segmentation) and finally the solution implying both regularization and image segmentation as SfD+ES+REG.

Synthetic setup
The synthetic environment contains 50 objects modeled by ellipsoids randomly placed inside a cube of side 20 units. The length of the largest axis ranges from 3 to 12 units, according to a uniform Probability Density Function (PDF). The lengths of the other two axes are proportional to the largest axis with a proportionality factor ranging from 0.3 to 1 units. Finally, axes orientation was randomly generated.
A set of 20 views were generated with a camera distance from the cube center of 200 units and a trajectory was computed so that azimuth and elevation angles span the range [0 • , 60 • ] and [0 • , 70 • ] respectively 4 . Given the projection matrix P f of each camera frame, ground truth ellipses given by the exact projections of the ellipsoids were calculated.
Synthetic tests were aimed at validating the robustness of the proposed method against common inaccuracies affecting object detectors, such as coarse estimation of the object center, tightness of the BB with respect to the object size and variations over the object pose. Thus, each ellipse was corrupted by three errors, namely translation error (TE), rotation error (RE) and size error (SE), and fed to the proposed algorithm. To impose such errors, the ellipses centers coordinates c 1 , c 2 , the axes length l 1 , l 2 and the orientation α of the first axis were perturbed as follows 5 : where ν c j , ν α and ν l are random variables with uniform PDF and mean value equal to zero, andl = (l 1 + l 2 )/2. In order to highlight the specific impact of each error, they were applied separately. Error magnitudes were set tuning the boundary values of the uniform PDFs of ν c j , ν α and ν l . In Fig. 4 the average accuracy O 3D of the proposed algorithm, for both SfD and SfD+REG, is displayed versus the error magnitude (i.e. the boundary value of the uniform PDF), for RE ( Fig. 4(a)), SE (Fig. 4(b)) and TE (Fig. 4(c)). Concerning SfD+REG, the weighting parameter λ has been kept equal for all the tests. The results for SfD are perfect in absence of errors but undergo a significant drop as the error increases. This confirms an expected sensitivity of the closed form solution to mismatches between fitted and ground truth ellipses. Nevertheless, when errors are moderate, the accuracy achieved is remarkable.
On the contrary, SfD+REG exhibits a very smooth decrease in accuracy versus the errors magnitudes, and a def- Notice that both the methods are generally more robust toward RE and SE (Figs. 4(a), (b) ), than toward TE. The higher robustness toward RE and SE is quite important since such kind of errors are likely to happen very frequently whenever ellipses are fitted to BBs. Even if the detector is accurate, the BB quantizes the object alignment at steps of 90 • , yielding a maximum RE of 45 • . This tends to overestimate the object area, thus affecting SE, whenever the object is not aligned to the BB axes (see Fig. 3(a)). Concerning TE, we noticed that the robustness to such kind of errors decreases when ellipses are far from the image center and small in comparison to the image size. This is mainly due to the structure of the 3 × 3 matrix of the conic, whose entries become strongly unbalanced whenever the translation terms in the third row and column increase in respect to the 2×2 principal sub-matrix, leading likely to numerical problems. However, the maximum degree of failure for the regularized solution is reasonable considering that an object detector placing the BB with a TE of 0.3 is considered to fail the detection 6 .

ACCV dataset evaluation
The ACCV dataset [12] contains 15 sequences, each related to a single object laying on a table at different camera viewpoints (from 100 to 1000 per sequence). We selected the subset of 8 sequences for which the 3D point cloud of the object is provided, and limit the number of views to 100 for each sequence. For each object we evaluated the ground truth ellipsoid as the envelope of the 3D point cloud. More- 6 The TE is applied independently to both the horizontal and vertical components of the ellipse center, i.e. see Eq. (20), resulting in a maximum overall translation of 0.3 √ 2.
over, for each frame and each object, we generated a 2D BB by simulating the output of a multi-scale object detector providing BBs with variable aspect ratio, like the well known Deformable Part Model (DPM) [6]. In Fig. 5 an example of the reconstruction performance, for SfD method, is displayed for the "duck" sequence. The estimated ellipsoid almost perfectly fits the ground truth one in respect to location, size, eccentricity and alignment, as it can be seen in three frames from the sequence and in the overall 3D reconstruction. In Table 1 the accuracy for each sequence, for SfD, is reported in terms of O 3D In general, the accuracy is lower for objects with a strongly not convex shape, like the driller, and higher for ellipsoid-like objects like the duck.
Due to the large number of views and to the centering of each object in each frame (see discussion on synthetic results), the accuracy is on average quite good even without regularization and ellipses from BBs, with an O 3D of 0.60 For this reason, either adding regularization or fitting ellipses to segmented images did not improve significantly the results on this dataset.

TUW dataset evaluation
The TUW dataset [1] has 15 annotated sequences showing a table with different sets of objects deployed over it. The number of frames per sequence ranges from 6 to 20. A 3D point cloud for each object is also provided. As for the ACCV dataset, we obtained the ground truth ellipsoids for each object and the 2D BBs. We discarded sequences with strong occlusions that cannot be handled by current object detectors, and sequences where objects appear for a number of frames lower than 3, thus retaining 5 sequences. In Fig.  6 an example of the reconstruction performance with the SfD+ES+REG method for a selected sequence is displayed. The accuracy in the estimation of object location is remarkable for all the objects. The estimation of object size and shape, in term of size, eccentricity and alignment of ground truth ellipsoids, is also very accurate for the majority of objects. All the selected sequences have been tested with the four methods SfD, SfD+REG, SfD+ES and SfD+ES+REG. The accuracy for each sequence is reported in Table 2, ac-cording to O 3D . It can be noticed that the regularization yields a sharp improvement in the accuracy with respect to non regularized methods on all the tested sequences. In particular, the accuracy is even doubled moving from SfD to SfD+REG. The ellipse fitting from segmentation yields a further improvement, leading a 5% increment in accuracy moving from SfD+REG to SfD+ES+REG and a 3% increment moving from SfD to SfD+ES. Remarkably, the improvement from SfD+REG to SfD+ES+REG is achieved in every sequence.

KITTI dataset evaluation
The KITTI dataset [8] is composed by a set of sequences taken from a camera mounted on a moving car in an urban environment. The dataset provides full annotations for cars appearing in each frame from which GT ellipsoids can be computed. We sampled 6 sub-sequences displaying parked cars 7 . As for the previous datasets, we generated 2D BBs simulating a multi-scale object detector for each car and each frame. We pruned out those cars characterized by strong occlusions for which a reliable detection is unlikely. The total number of remaining cars was of 34 and the average number of views in which a car is visible is 10. Ellipsoids estimation is particularly challenging on this dataset, since the camera motion is almost planar. Moreover, cars are usually placed at the street borders and the camera moves straight in most of the sequences. In such a condition the range of angles between car and camera spanned by the sequence of camera views is very narrow and almost limited to the azimuth plane. Finally, each car appears in a limited subset of frames. We did not apply segmentation on this dataset due to the extreme difficulty in segmenting some of the cars that are partially overlapping each other. In Table 3 quantitative results are displayed for the six selected sequences. Despite the difficulty of the dataset, SfD achieves a reasonable result of O 3D = 0.17 and the use of regularization almost doubles the accuracy, yielding O 3D = 0.36. The result for SfD+REG is visually confirmed by looking at Fig. 7: all the seven cars are correctly located in the 3D space, with a reasonable precision 7 The selected sequences (Seq.) and the corresponding frames (Fr.) defining the sub-sequences are the following: Seq 5 (Fr. 142 -153); Seq.  concerning the object size. The estimated aspect ratio and orientation of the cars is less accurate but acceptable considering the challenging conditions characterizing this dataset. Besides the object occupancy in 3D, our algorithm achieves excellent 3D localization performance. In Table 4 the percentage of estimated object centroids, corresponding to the ellipsoid centers, within 1 m and 2 m from the GT centroids is reported. As it can be seen, with SfD+REG almost all the cars (93%) are within 2 m and 76% are within 1 m. Moreover, SfD alone achieves very good results (83% < 2 m and 81% < 1 m on average). The lower average value of SfD+REG w.r.t. SfD can be explained by the fact that regularization enforces the sphere-likeness of the quadric, and due to the non-linearity of the regularization term, this may sometime decrease the accuracy in the 3D center estimation. Notice that, even for those estimated quadrics not corresponding to feasible ellipsoids, a quadric center can be still computed and also in this case its position is most of the times within 2 m of the GT centroids, thus contributing to explain the very good performance of SfD.

Discussions and Future Work
This paper presented a closed-form solution to recover the 3D occupancy of objects from 2D detections in multiview. This algebraic solution is achieved through the estimation of a 3D quadric given 2D ellipsoids fitted at the objects detectors BBs. Moreover a regularization approach was devised to cope with possible ill-conditioning of the problem. The approach was tested against the common inaccuracies affecting object detectors such as coarse estimation of the object center, tightness of the BB in respect to the object size and variations over the object pose. To strengthen further the approach, a robust ellipse fitting method was introducing using a segmentation algorithm over the detected BBs in multi-view. Experiments show that even with relevant errors, the estimated quadrics are able to localise the object in 3D and to define a reasonable occupancy. Moreover, the proposed estimation of object orientation, by means of a segmentation algorithm, can be used in order to increase accuracy and the percentage of the reconstructed quadrics.
The solutions of this problem has strong practical breakthroughs given the recent evolution of recognition algorithms. In particular, object detection is certainly going towards increased generality, so providing detectors for several object classes [4]. Thus, the proposed method can provide a quick and very efficient solution to leverage the 2D information for 3D scene understanding where objects can be inter-related given their position in the metric space. This will inject important 3D reasoning in classic frameworks for object detection that are mostly restricted to 2D reasoning.
Regarding future work, mis-detections (i.e. outliers) might affect negatively the estimation of the quadrics. Thus, including further robustness in the optimization, through ad hoc regularization terms in the cost function, might improve the performance of the system. Moreover, the ellipsoid orientations given the BBs can be further improved, especially when objects occlusions or similar appearance textures are present.