{'fraction_non_alphanumeric': 0.02385668253170544, 'fraction_numerical': 0.013302982824371064, 'mean_word_length': 4.827390180878553, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
HomebrewedDB: RGB-D Dataset for 6D Pose Estimation of 3D Objects

One of the most important prerequisites for creating and evaluating 6D object pose detectors are datasets with labeled 6D poses. In the advent of deep learning methods, demand for such datasets is consinuously arising. Despite the fact that some of those exist, they are scarce and typically have restricted setups, e.g. a single object per sequence, or focus on specific object types, such as textureless industrial parts. Besides, two significant components are often ignored: training only from available 3D models instead of real data and scalability, i.e. training one method to detect all objects rather than training one detector per object. Other challenges, such as occlusions, changing light conditions and object appearance changes, as well as precisely defined benchmarks are either not present or scattered among different datasets. In this paper we present dataset for 6D pose estimation that covers the above-mentioned challenges, mainly targeting training from 3D models (both textured and textureless), scalability, occlusions, light and object appearance changes. The dataset features 33 objects (17 toy, 8 household and 8 industry-relevant objects) over 13 scenes of various difficulty. Moreover, we present a set benchmarks with the purpose of testing various desired properties of the detectors, particularly focusing on scalability with respect to the number of objects, resistance to changing light conditions, occlusions and clutter. We also set a baseline for the presented benchmarks using a publicly available state of the art detector. Considering difficulties in making such datasets, we plan to release the code allowing other researchers to extend this dataset or make their own datasets in the future.


Introduction
Detection of 3D objects in images and recovery of their 6D pose is crucial for a large range of computer vision tasks. In robotics it is essential to determine 6DoF of the object for the tasks of object grasping, manipulation and automatic assembly. Also, precise 6D poses in RGB images are of utmost importance in Augmented Reality (AR) applications, where overlaying 3D models of the virtual objects on top of the real objects is critical for AR driven assembly or repair tasks. With the rise of deep learning methods, demonstrating better performance than traditional template matching approaches or methods based on handcrafted features, it is essential to have appropriate datasets, which would encourage development and thorough evaluation of the new approaches.
There is a number of 6D pose object datasets, each of them focusing on one of the aspects of this challenging task. For example, datasets like T-LESS [17] and LineMOD [16] cover textureless and target-specific object types in particular scenarios. LineMOD contains mainly toys and household objects of distinct geometry, concentrating on object detection in cluttered environments with non-existent or minor occlusuions. There are in total 15 separate image sequences in this dataset, however each of them features 6D pose annotations only for a single object. T-LESS is a much bigger dataset, focusing exclusively on textureless industrial objects, which exhibit strong inter-object similarities and symmetries. In T-LESS, separate training and test images are explicitly provided, while in case of LineMOD it remains unclear how to sample or produce the training data. This ambiguity has led to wide inconsistencies in presented results, making it hard to make a comprehensive assessment of the developed methods. Striving for the best possible results, researchers tend to train their detectors on subsets of provided real images with ground truth poses very similar to those in the test split. This strategy inevitably leads to overfitting to a particular dataset, restricts applicability of the detectors and undermines fair comparison of various approaches.
One crucial aspect that has often been neglected by 6D pose detectors is scalability. The majority of the datasets contains a rather small number of objects, and this is natural, since producing large number of 3D models and scenes with annotated poses is a tedious task, especially when poses of all the objects in all frames must be available. When it comes to deep learning methods, it is known that training detectors on real data demonstrates the best results. However, the fact that 3D models of the objects are available, and training data can be synthesized by rendering these models, is used in only a few works, most notably in SSD6D [20], AAE [28] and recent DPOD [33] detectors. It is remarkable that all deep learning 6DoF object detectors trained either on real or synthetic data use one neural network per object, in contrast to deep-learning-based 2D object detectors, such as YOLO [23], SSD [21] or R-CNNs [13,12,24,14], which are trained with one network for all object classes. One of the major reasons for this issue is unavailability of the proper dataset with variety of sequences and well-defined benchmarks, causing a fallback to the well-studied LineMOD dataset with single object annotation per scene. Nevertheless, training one network per object defeats the scalability aspect naturally brought by deep neural networks. Therefore, the central aspects of our proposed dataset is to test scalability of methods by introducing corresonding benchmarks and also encourage training of detectors on renderings of 3D models instead of the real data.
Our dataset may be considered to be aligned with the LineMOD OCCLUSION dataset [4], which contains poses of all the objects present in each frame. However, instead of just a single sequence with 1214 frames, our dataset contains 13 full-circle scenes filmed with both PrimeSense Carmine 1.09 (structured light) and Microsoft Kinect 2 (time-of-light) RGB-D cameras resulting in total of 34830 fully annotated frames with poses for all objects in all frames. The complexity of the scenes increases from simple (several separated objects per scene) to heavily cluttered and occluded (objects close or on top of each other, also mixed with other objects not present as 3D models). Another aspect that is not addressed in the other datasets is strong illumination variations, including not only changes in light intensity, but also in light color. Finally, driven by the fact that in industry objects can undergo severe appearance changes, we created a benchmark where the object appearance is altered compared to the one in available 3D models.
In the following sections, besides describing the steps of the dataset creation pipeline, we also present detection and 6D pose estimation results on all of the newly-defined benchmarks of one of the recently introduced methods -Dense Pose Object Detector (DPOD) [33]. This time the method is trained for all the objects at once or on all the objects present in the test scene on strictly synthetic renderings of provided 3D models. The aim of this experiment is not only to test scalability of the current methods, but also to demonstrate that even the best performing methods trained on real images with one network per object obtain mediocre results when trained on multiple objects. This establishes a baseline and opens a new challenge to the community related mainly to sclability and synthetic training data. Light and appearance changes are another novel aspect of this challenging dataset.

Related Datasets
Given the fact that determining ground truth 6D pose in RGB images is an ambiguous task requiring manual interventions, it is not surprising that the majority of 6D pose datasets are made with the use of RGB-D cameras. Since these cameras provide depth images aligned with color images, the task of 6D pose estimation becomes simpler and more automated. In the following paragraphs we list the most popular datasets for 6D pose estimation.
LineMOD Dataset. One of the most widely used 6D pose datasets is LineMOD of Hinterstoisser et al. [16]. Essentially, it contains objects embedded in cluttered scenes. It is acquired with PrimeSense Carmine RGB-D sensor and in total comprises 15 objects, two of them being symmetric. For each sequence, poses of only one object are annotated. Moreover, the target objects with the existing ground truth poses are either not occluded at all, or are subject to very slight occlusions. The original template matching method [16], which was published alongside the dataset, performed poorly on RGB images and depth images separately, while the results obtained on RGB-D images were extremely good and still remain hard to achieve for many modern deep-learning-based methods. However, this method suffers from a couple of drawbacks -it scales poorly and cannot handle occlusions. LineMOD OCCLU-SION dataset was created by Brachmann et al. [4] in order to address the lack of occluded test data. In this extension of the original LineMOD, additional manual annotations of 6D poses for all the objects in each frame were performed. Due to necessity of intensive manual labour, it was done only for a limited number of frames. Availability of both RGB and depth images and 3D CAD models with original object colors resulted in probably the widest use of this dataset of all those targeting 6D pose estimation. Deep learning methods use RGB images of this dataset for object detection and 6D pose estimation. The state of the art results for real and synthetic data were achieved by the recently introduced DPOD method [33].
T-LESS Dataset. T-LESS [17] is one of the recent datasets which is gaining popularity. It contains 30 textureless industrial objects and 20 RGB-D scenes captured with three synchronized cameras -PrimeSense Carmine 1.09, Kinect 2 (both RGB-D cameras) and Canon RGB camera. The objects in this dataset have strong inter-object similarity. The acquired scenes vary from simple, with less clutter and several objects per scene, to the heavily-cluttered, with piles of objects, mimicking typical robotic bin-picking scenario. Both hand-designed CAD models and reconstructed 3D models are available. Training images contain isolated objects on black background, while test images capture entire scenes with labeled 6D poses for each object in each frame. The structure of this dataset is exceptional, but due to symmetry, low texture and industrial nature of the objects it is very challenging, which might be the reason why it does not gain popularity at the pace it deserves. We are truly inspired by T-LESS [17] and prepared our dataset similar to this one in terms of structure. However, our dataset covers a wider range of object types, spanning over toys, industrial and household objects. Moreover, we were aiming for a dataset that brings occlusion challenges, and goes well beyond OCCLUSION, providing many more frames and scenes for this task.
YCB-Video dataset. This dataset is very recent and came out with PoseCNN detector [31]. Unlike T-LESS, which contains images of the scenes from all the viewpoints, this one resembles LineMOD, containing short videos depicting several household objects in the scene. However, a large number of video sequences (92) as well as presence of occlusions in the test scenes make this dataset quite attractive.
Other datasets. The following datasets have been sporadically used in some publications, but their systematic use was not noticed. Tejani et al. [29] contains 2 textureless and 4 textured objects with 700 frames of test images. Characteristic for this dataset are multiple instances of the same object. Clutter and occlusions are moderate, which makes it not particularly challenging. Doumanoglou et al. [9] present bin-picking dataset with 183 test images and only two objects from Tejani dataset, but multiple instances of them. The Challenge and Willow datasets [32] contain a larger number of objects, but a relatively small number of test images: 176 and 353 respectively. TUW dataset [2] is similar with 17 objects in 224 test images. Additional datasets, which are suited for robotic tasks are Rutgers dataset [25], Amazon Picking Challange [7], BIG-Bird [26] etc. Datasets introducing light change challenge are TUD-Light and Toyota Light, both used and referenced in the benchmark paper of Hodan et al. [18]. Our dataset could be seen between LineMOD OCCLUSION and T-LESS. It contains high-quality annotations, multitude of objects and a large number of scenes and test images. In contrast to the recent benchmark of Hodan et al. [18] that tries to unify 8 different datasets and concentrates on evaluation of RGB-D methods, we are more interested in RGB methods, even though full RGB-D images are available in our dataset. With scalbility at its core, we design benchmarks in which one network is trained either for all the available objects or only for the objects present in a particular scene. Additionally, we introduce scenes with severe environment changes, including changes of light colors and intensities as well as changes of object appearance. We believe that this dataset will push forward research on scalable object detection and domain adaptation.

HomebrewedDB Dataset Creation
Within HomebrewedDB dataset we introduce: 1. 33 highly accurately reconstructed 3D models of toys, household objects and low-textured industrial objects of sizes varying from 10.1 to 47.7 cm in diameter.
2. 13 sequences, each containing 1340 frames filmed using two different RGB-D sensors. All the scenes have different complexity ranging from simple ones with 3 objects on plain background to highly occluded ones with 8 objects and presence of extensive clutter (see Figure 3). Moreover, two sequences feature drastic light changes and contain objects with altered textures respectively (see Figure 4).
3. Precise 6D pose annotations for dataset objects in the scenes, which were obtained using an automated pipeline described in the section 3.5. The following sections describe a detailed pipeline of the dataset creation, including calibration of RGB-D sensors, reconstruction of 3D models, depth correction, acquisition of image sequences and creation of ground truth annotations. We believe that the described steps would serve as a sufficient guide on how to extend an existing or create a new dataset for 6D pose estimation.

Calibration of RGB-D sensors
For the footage of test sequence we used two RGB-D sensors: one is structured-light PrimeSense Carmine 1.09, and the other is time-of-flight Miscrosoft Kinect 2. Intrinsics and distortion parameters of both sensors were estimated during the calibration procedure. We used ArUco board [11], which has demonstrated to give better calibration results in comparison with the classical checkerboard pattern, and a corresponding intrinsic calibration module from OpenCV [5]. As a result of calibration, the root-mean squared reprojection error calculated at the corners of the ArUco markers are below 0.5 and 0.3 pixels for Carmine and Kinect 2 respectively. Intrinsic and distortion parameters for both sensors are provided with the dataset. Since depth and color images are observed from two different cameras, there arises a need for depth-to-color registration, which was performed with the help of OpenNI 2.2 Driver for Carmine and Windows SDK 2.0 for Kinect 2. Since the scenes were recorded independently with each of the sensors, there was no need in extrinsic calibration of the cameras.

Acquisition of the sequences
In total we acquired 1 handheld and 2 turntable sequences for each of the scenes with each RGB-D sensor. Turntable sequences capture a full 360 • rotation of a markerboard with objects on it using a camera mounted on a tripod. Each turntable sequence has 170 RGB and depth images filmed with elevation angles of either 30 • or 45 • . These turntable sequences accompanied by the ground truth 6D pose labels are provided as validation data. The test sequences, on the contrary, were recorded in a handheld fashion. There were two major reasons for recording in a handheld mode instead of introducing a controlled setup similar to those in T-LESS [17] or BigBird [26]. The first clear advantage is its close resemblance to the natural use of cameras, while the second one is ability to introduce more variation in camera poses, namely considerable scale changes as well as significant in-plane rotations within a single sequence. For the test sequences, in total 1000 RGB and corresponding depth images were captured with each sensor for each of the scenes. While shooting each of those, a full pass around the markerboard was made. Within the test sequences distance from the camera to an object varies from 0.42 to 1.43 meters, while the elevations stay between 11 • and 87 • . Both color and depth images in the sequences recorded with Carmine sensor have a default resolution of 640x480 pixels, while RGB images of Kinect 2 are of size 1920x1080 pixels and depth image of 512x424, the latter getting upscaled and resized to the dimension of RGB image during the registration process. For each of the scenes (sample images are shown in Figures 3 and 4) the target objects were placed on the markerboard with ArUco markers facilitating camera pose estimation. In simpler scenarios we placed the objects onto a monochrome Lambertian surface (either white or dark-blue), while in the more complicated ones the objects resided on multi-color reflective sur-  face, being occasionally on top of each other. Also, more complicated sequences feature severe occlusions as well as objects not present in the dataset to make the scene more cluttered. One novelty introduced in HomebrewedDB is a sequence aiming to test robustness of detection and pose estimation methods to significant changes in lighting conditions. To create it, we employed a spotlight repeatedly projecting series of light patterns with different colors and intensities onto the scene. Besides, we created another sequence where we altered the textures of the objects by selectively painting some part of them with chalks of various colors. This sequence was created with a main goal to evaluate robustness to considerable texture changes.

Reconstruction of 3D models
To obtain the 3D models presented in the Home-brewedDB dataset (depicted in Figure 2), we first scanned each object form multiple viewpoints using structured light scanner Artec Eva [1] . We opted for this scanner since it provides precise depth measurements as well as high resolution texture maps, which are crucial for reconstructing highquality 3D models. The following pipeline of converting the scans to complete reconstructed 3D models proceeded using Artec Studio software. Firstly, raw meshes were reconstructed for each of the scanned viewpoints. Secondly, we manually removed unnecessary parts of the meshes and then aligned them. This was followed by removal of outliers and minor artifacts on the models. After that we proceeded with global optimization of the mesh structure, including inpainting minor holes in the model as well as inducing smoothness of the mesh. Finally, high-resolution textures were back-projected onto the resulting 3D model. Afterwards we used MeshLab [6] to center and and axis-align the models. Finally, the surface normals were computed as a weighted sum of normals of the incident faces as suggested in [22].

Depth correction
It was observed similarly to [27] and [17] that depth measurement of both Carmine and Kinect 2 have systematic errors, since the measured depth values were always slightly different from the depth values calculated from the markers in images captured with calibrated RGB cameras. Though in [27] it is reported that a single correction multiplier is sufficient to account for depth measurement error, we found similarly to Hodan et al. [17] that first degree polynomial works better as a correction function for the depth measurements in our setup. Using regularized least squares to account for noise in the measurements we ended up with the following linear correction models : d c = 1.0391 · d − 15.8 for Carmine and d c = 1.0186 · d − 13.1 for Kinect 2 (depth measured in millimeters). After applying the correction model the mean absolute difference from expected depth reduced from 14.7mm to 2.03 mm and from 5.81 mm to 2.66 mm for Carmine and Kinect 2 respectively. We applied the correction model to the dataset, so that no further user action is required.

Creation of ground truth annotations
The estimation of 6D ground truth objects poses for each frame in the sequence proceeded as follows. Firstly, the markerboard pose was estimated, providing us with the camera trajectory around the scene. Then we obtained a dense 3D reconstruction by SDF (Signed Distance Field) fusion of depth maps of a scene with a method of Cureless and Levoy [8], utilizing all the images of a sequence. The next step was to estimate a rigid body transformation of a 3D model from its own coordinate system to the coordinate system of the scene (i.e markerboard). Since the locations of the objects with respect to the markerboard did not change when filmed with different sensors, it was sufficient to get the object pose in markerboard coordinate system and then transfer it to a new sensor/camera coordinate system, avoiding performing reconstruction for each new sensor. For the purpose of estimating 3D model pose in the reconstructed scene we employed a method of Drost et al. [10], which utilizes Point Pair Feature representation of a target model used for the local matching via an efficient voting scheme on a reduced two-dimensional search space. Having a camera pose in each image estimated from the markerboard as well pose of an object in the markerboard coordinate system, 6D object poses for each of the frames can be easily computed. However, rendering the 3D models on top of RGB images revealed that even though the method of Drost et al. [10] gives a good initial estimate of a pose, in many cases there still exist visible discrepancies, which must be mitigated in the process of further refinement. To improve the poses, we opted for 2D edge-based ICP refinement. For each object in the sequence we automatically selected RGB images where the object is not occluded. This was done by rendering all the objects in the scene with the estimated initial poses and calculating the fraction of visible pixels for the target object. Multi-view consistency was enforced in a way that all the camera poses stayed fixed, and optimization was only done for the object pose in the scene coordinate system. Edge-based refinement was performed on RGB images due to the fact that depth measurements were relatively noisy, and minor misregistrations between depth and RGB images were observed, particularly  Table 1: Differences between the depth of object rendered models at the ground truth poses and the captured depth (in mm). µ δ and σ δ is the mean and the standard deviation of the differences, µ |δ| and med |δ| is the mean and the median of the absolute differences for those captured with Kinect 2.

Accuracy of ground-truth poses
To evaluate accuracy of the computed ground truth poses, we followed the same procedure as introduced by Hodan et al. [17]. We rendered the 3D objects using computed ground truth poses and for each pixel pair with valid depth in both rendered and captured images we compute the difference δ = d c − d r , where d c and d r are captured and rendered depth values correspondingly. The statistics obtained over the whole test set are presented in the Table  1. As in T-LESS [17], differences exceeding 5 cm were omitted from the statistics as outliers. In case of Home-brewedDB, such measurements amounted to 3.7%, most of them caused by the clutter objects occluding the target objects, sensor measurement noise or minor discrepancies between the reconstructed 3D models and real-world objects.
From the presented results it can be seen that rendered depth maps align well with the depth maps obtained with Carmine, resulting in mean depth difference close to zero and absolute mean of differences less that 2 mm. While for Kinect 2 depth difference mean and absolute mean values also stay very close to zero, it can be seen that absolute median value is notably higher than the absolute mean, signifying that the distribution of the depth differences is leftskewed. As noted in T-LESS [17], this might be caused by slight misregistration of RGB and depth images captured by Kinect 2, as well as higher magnitude of noise in the measurements of this depth sensor based on time-of-flight principle.

Benchmark and Experiments
In this section, we present a set of benchmarks assessing the performance of a detector with respect to a variety of different conditions. In particular, such aspects as scalability, resistance to occlusions, different illumination conditions and object texture wearing are tested.

Evaluation Metrics
We use the standard metrics for evaluatuing performance in object detection in 2D: Precision, Recall and mean Aver- Texture / Illumination   Scene ID  0  1  2  3  4  5  6  7  8  11  12  3  9 Table 2: Result of object detection and pose estimation presented on two benchmarks: (1) per scene benchmark spanning over 11 scenes of the dataset and the (2) domain adaptation benchmark evaluating the detector's generalization capabilities.

Per Scene
age Precision (mAP). Conventionally, we consider an object to be correctly detected if the intersection over union (IoU) between the ground truth and predicted bounding boxes is not lower than 0.5.
To evaluate correctness of the estimated 6DoF poses similarly to [15,20,33] we employ ADD score, which is defined as average Euclidean distance between the model vertices transformed with ground truth and predicted poses, i.e.
where M is a set of vertices of a 3D model, (R, t) and (R,t) are ground truth and predicted rotation and translation correspondingly. As mentioned in [15], a predicted pose in considered to be correct if ADD calculated with this pose is less than 10% of a model diameter. However, in case of more complicated scenes, there is only a small fraction of poses falling into this category. Therefore we also report ADD for the thresholds of 30% and 50% to give a broader overview of pose quality as well as to give an estimate of proportion of the poses which could be still a subject for further refinement.

DPOD: Dense Object Pose Detector
For the performance evaluation we decided to opt for the recent Dense Pose Object Detector [33] due to its excellent performance on LineMOD and LineMOD OCCLU-SION datasets and the ability to be trained on synthetic data. Other detectors could be also used, and we highly encourage their authors to perform evaluation on HomebrewedDB and research in the direction of scalability and training from CAD models.
DPOD detector follows the idea of DensePose [3] applied for human pose detection and formalizes the object detection problem as dense correspondence estimation problem. From the input image DPOD outputs the multilabeled object segmentation mask and UV correspondence map. UV map has been automatically created by spherical or cylindrical projection during training for each ob-ject. Since the values of UV map are limited to a discrete range, it is much easier to regress them than 3D object coordinates [19]. UV coordinates represent a direct mapping between the input image and vertices of 3D object model. Therefore, 2D-3D correspondences are directly obtained and can be further used as an input to PnP and RANSAC for pose estimation. Large number of correspondences makes the detector less prone to wrong matches and allows easier and more accurate pose estimation.

Scalability Benchmark
The first benchmark is introduced with a goal to evaluate the method's scalability with respect to the number of objects. The main requirement is to train a single network for all the objects available in the dataset and test it on a set of sequences containing all the objects. Sprecifically, for this purpose we jointly evaluate a method on the sequences from 0 to 7 (inclusive), which form a minimal subset of sequences with all the 33 objects present. In this benchmark object detection and pose estimation results are reported separately for each object. From the results presented in Table 3 it can be seen that the best detection and pose estimation performance is achieved for the bigger objects with distinct texture and geometrical features (e.g. 27,29), while detection of smaller low-textured or glossy industrial objects (e.g. 11, 12, 13) is a considerable challenge for DPOD. Besides, pose estimation results demonstrate that DPOD detector does not scale particularly well for this task -for 17 objects out of 33 ADD with 10% threshold is under 10%, and there are no instances for which score higher than 50% was achieved.

Scene Benchmarks
Our dataset presents a collection of 13 scenes with a varying degree of difficulty, and each of these scenes represents a separate benchmark, where all the objects it contains are used for training. We include all the scenes except for those with altered illumination conditions and texture changes (i.e. 9 and 10) into per scene benchmarks. All the object detection and pose estimation scores are averaged over all the objects present in the scene.
In the Table 2 the performance in objects detection and pose estimation per scene is demonstrated. As expected, DPOD demonstrates significantly better performance in both tasks in the sequences with smaller number of objects, as well as less significant occlusions and no clutter. Besides, low scores in both detection and pose estimation are reported for the scene 8 composed exclusively of industrial objects.

Domain Adaptation Benchmark
The main goal of introducing domain adaptation benchmark is to test robustness of a method to significant changes in lighting conditions and objects' texture. It is composed of three scenes (4, 9 and 10) with the same set of objects, while being different in terms of illumination and color of object surfaces. The scene 4 was captured in usual lighting condition with no alteration of objects' appearance. On the contrary, in scene 9 we use a spotlight to project light of different colors and intensity onto the filmed objects to introduce considerable variations in illumination, while in scene 10 we applied paint on the object' surfaces to alter their texture. In this benchmark object detection and 6D pose estimation scores are presented per scene.
As it can be seen from results in Table 2, performance in both detection and pose estimation is notably better in case of no illumination or object texture changes. In normal conditions the resulting poses are 37% more accurate based on ADD with 10% threshold as compared to those in altered conditions. Also, performance in object detection falls far behind in case of altered conditions as compared to normal ones, resulting in 46% and 59% lower mAP scores for the scenes with variations in light and texture respectively. Judging by these results, one may conclude that there exists a lot of space for further improvements for DPOD detector in the direction of adaptation to changing environments.

Drawbacks of Training on Real Data
The main reason we exclusively use synthetic data for training in our benchmarks comes from the inability of the detectors trained on real data to generalize to the different environments. This includes background, illumination, texture and other changes. To support our claim, we have selected 2 recent state of the art detectors, YOLO6D [30] and DPOD [33], trained on real LineMOD data and tested them on our new sequence containing 3 LineMOD objectsbenchvise, driller and phone (Scene 1 in Figure 3). This sequence contains the minimal number of occlusions and no clutter, and may be regarded as one of the simplest scenes in the dataset. Besides, this sequence was captured with the same camera as LineMOD sequences (PrimeSense Carmine 1.09), making it similar in terms of color scheme and noise.   Table 4: Pose estimation results in terms of ADD 10% metric on LineMOD sequences (LM) and HomebrewedDB (HB) sequence with the same objects.
Performance comparison of both detectors on LineMOD sequences and our sequence can be seen in Table 4. In spite of demonstrating very good performance when tested on sequences from LineMOD, when run on our sequence with LineMOD objects, both detectors experience a significant drop in the pose estimation accuracy in terms of ADD 10% metric for all the objects. Specifically, pose estimation accuracy of DPOD [33] detector dropped more than 2 times for the phone, while for YOLO6D [33] there were nearly no correctly predicted poses for the driller. Such poor performance of detectors trained on real data from LineMOD points to the fact that they fail to generalize well, and even a seemingly insignificant changes in environment turn out to be a decisive factor leading to a drastic decrease in pose estimation accuracy.

Conclusion
In this work, we have presented a new challenging dataset for 6DoF object detection, covering the most important properties a solid object detector should have, namely scalability with respect to the number of objects, robustness to occlusions, illumination and appearance changes. The dataset contains 33 objects spanned over 13 scenes of various difficulty. To be able to compare the detectors to each other, we define a set of benchmarks that test all of the above mentioned properties. Finally, we presented a comparably simple, yet robust and fully-automated pipeline that we used for the acquisition of our dataset. We plan to release the code for the other researchers to be able to create their own datasets and thus promote the research in 6D pose estimation.