{'fraction_non_alphanumeric': 0.03248299319727891, 'fraction_numerical': 0.015022675736961451, 'mean_word_length': 4.732087733549959, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 2, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Templates for 3D Object Pose Estimation Revisited: Generalization to New Objects and Robustness to Occlusions

We present a method that can recognize new objects and estimate their 3D pose in RGB images even under partial occlusions. Our method requires neither a training phase on these objects nor real images depicting them, only their CAD models. It relies on a small set of training objects to learn local object representations, which allow us to locally match the input image to a set of"templates", rendered images of the CAD models for the new objects. In contrast with the state-of-the-art methods, the new objects on which our method is applied can be very different from the training objects. As a result, we are the first to show generalization without retraining on the LINEMOD and Occlusion-LINEMOD datasets. Our analysis of the failure modes of previous template-based approaches further confirms the benefits of local features for template matching. We outperform the state-of-the-art template matching methods on the LINEMOD, Occlusion-LINEMOD and T-LESS datasets. Our source code and data are publicly available at https://github.com/nv-nguyen/template-pose


Introduction
3D object pose estimation has significantly improved over the past decade in terms of both robustness and accuracy [18,30,34,20,44]. In particular, the robustness to partial occlusions has greatly increased [28,17,24], and the need for large amounts of real annotated training images has been relaxed thanks to domain transfer [1], domain randomization [36,19,31], and self-supervised learning [33] techniques that leverage synthetic images for training.
Nevertheless, the use of image-based 3D object pose estimation remains limited in the industry, despite its huge potential for robotics and augmented reality. Scalable industrial applications would, for example, require the ability to handle arbitrary, previously-unseen objects without retraining and with access only to the objects' CAD  Figure 1: Our method can estimate the 3D pose of new objects in query images by matching them with templates created from their 3D models. These new objects can be very different from the ones, and can be partially occluded in the query images. models, thus saving both training and data capture time. While a few works have already tackled this challenging task [31,29,39,2], most of them impose some additional constraints by assuming that the new objects belong to a known category [38], remain similar to the training ones as in the T-LESS dataset [31], or have prominent corners [29].
By contrast, template-based approaches [39,2] offer the promise of generalizing to arbitrary new objects by learning an image embedding used to match the input image to a series of templates generated from their CAD models. Unfortunately, their use with new objects has been demonstrated only anecdotally, and we show in our experiments that these methods struggle in this challenging scenario, particularly in the presence of occlusions. We indeed notice that the global representations used in [39,2] to compare the input image to the CAD-generated templates have two limitations. First, they generalize poorly to new objects in the presence of a cluttered background, and result in inaccurate pose estimation even for uniform background. Furthermore, they are ill-suited to handle occlusions.
These observations motivate us to keep the 2D structure of the images for a template-based approach. More precisely, given a small set of training objects, we learn local features that can be used to reliably match real images and synthetical templates. Relying on local features allows us to discard the background: While the object's mask in the input image is not available at run-time, we can use the template's mask, thus solving the first limitation of global representations. Note that using the template's mask to instead remove the background in the real image before computing the image global representation requires us to recompute the input image representation for each template, which would result in very slow matching.
As will be shown by our experiments, using local features also results in much more accurate poses. This can be explained by the fact that we do not use pooling operations, which remove critical information about the poses, especially for new objects. Finally, yet another advantage is that our method can be robust to partial occlusions. To do so, we introduce a measure to evaluate the similarity between two images that explicitly takes into account the object's mask in the template and the possible occlusions in the query image.
We demonstrate the benefits of our approach on the LINEMOD [12], Occlusion-LINEMOD [3], and T-LESS [14] datasets. It consistently outperforms previous works [39,2,32,31] on new objects by a large margin. In summary, our contributions are: • A failure-case analysis of previous template-based methods when testing on new objects; • A method that can predict the pose of new objects from their CAD models, without training on these objects nor restricting these objects to be similar to the training ones; • A method robust to occlusions even in the challenging scenario when objects are both new and occluded.

Related Work
Our goal is to develop a method able to estimate the 3D pose of previously-unseen objects while having access only to their 3D model. It should be noted that early approaches to 3D pose estimation already targeted this goal [22]. However, these approaches, based on image edges and object contours, proved to be very fragile. As discussed below, with the use of deep learning, methods have become much more robust but typically require many training images.
Pose estimation for known objects. Many 3D object pose estimation methods use a deep model trained on real images or synthetic renderings of these objects, [18,30,20,34,21,44,26,16]. Some also show remarkable robustness to partial occlusions of the objects [24,28,17]. Such an approach however requires long expensive training and data acquisition/generation time, which we would like to avoid. For example, the state-of-the-art method [19] on standard benchmarks [15] requires almost a day on 32 GPUs for training. While some works have attempted to reduce the burden of registering real images by learning to generate new images from real ones [27], their cost remains too cumbersome for many practical applications.
Category-level pose estimation. One way to avoid retraining on new object instances is to consider object categories, and train a model on target categories that will generalize to new instances of these categories [45,38]. While such an approach can be useful in some applications, such as scene understanding, in many others, the new objects do not belong to a known category. By contrast, our approach generalizes to new objects that bear no similarity in shape with the known objects used to train the initial model.
Unseen object pose estimation. [39] proposed to learn discriminative representations of templates, which are images of objects associated with the corresponding 3D poses. Pose estimation could then be achieved by matching the input image against these templates in an image-retrieval manner. In this context, [2] then showed how to obtain more discriminative representations. While the ability to consider unseen objects by using their 3D models seems to be the motivation for these works, this was only superficially demonstrated, and our experiments show that these methods perform poorly on unseen objects.
More recently, [31] proposed an extension of [32] to generalize to unseen objects. This method introduces a novel architecture with multiple decoders to adapt to different object types. While their results indeed show generalization to unseen objects, these objects must remain similar to the training ones. As a consequence, this method has been demonstrated only on the T-LESS dataset, which depicts different kinds of electrical appliances that bear strong visual similarities.
In any event, as we will discuss in detail in Section 3.1, these methods rely on a global representation of the templates. We will show that our local representation-based framework has significant advantages in terms of generalization to new objects and of robustness to occlusions.
[29] also considers local representations but in a way that is very different from us: [29] learns to detect specific 2D object locations in the image together with a descriptor for each such location to match them with 3D points on the object's 3D model. This matching, however, is done indepen-dently for each location, making it highly ambiguous, and resulting in a combinatorial matching cost and frequent failures. By contrast, we extract local representations in a grid structure and learn to match all local input and template representations jointly. To achieve this, we rely on contrastive learning, which we discuss below.
A different and interesting take was proposed in [43], where the embedding of the object's 3D model was used as input in addition to the input image to predict the 3D pose. However, this work considers only pose regression and assumes the object is already known in order to use the right 3D model.
Contrastive learning. Given a collection of images, contrastive learning aims to learn an embedding space where similar images are close to each other while dissimilar ones are far apart. [13,40,25,35,10,5] leverage unlabeled images and strong data augmentation to learn powerful image features that achieve results competitive with those of supervised learning on various downstream tasks.
In our context, [42] exploits a form of contrastive learning, leveraging the pose labels to learn a pose-aware embedding space for class-agnostic 3D object pose estimation. One limitation of [42] is that different objects can be mixed with each other in the embedding space, thus making it impossible to recognize the correct object instance from the input image. Moreover, like [43], [42] does not attempt to recognize the object.
By contrast, [39,2] rely on contrastive learning to learn an embedding space that is variant to both the object pose and the object instance. To this end, they rely on a triplet loss for learning object-discriminative features, together with a pairwise loss for pose-discriminative features. Similarly, we use contrastive learning to extract a discriminative feature representation, but we show that the InfoNCE [25] loss is the most simple and effective choice. Our experiments also show that most of the performance of our method in terms of generalization and robustness to occlusions come from our use of local representations.

Method
Our goal is to recognize new objects in color images and predict their 3D poses. We do this by matching the color image of the object with a set of templates. A template is a rendered image of a 3D model in some 3D pose. For each new object, the template set contains many templates, rendered from different views sampled around its 3D model. As the templates are annotated with the object's identity and pose, the method returns the identity and pose of the template most similar to the input image.
The challenge then is to measure the similarity between templates and input images. This should be done reliably despite that no real images of the new objects have been seen beforehand, the objects can be partially occluded, the Seen objects Unseen objects After masking [2] Our method lighting differs between the templates and the real images, and the object's background is cluttered in the real images.
In this work, motivated by the better repeatability and robustness to occlusions of local representations compared to global ones, we measure the similarity between an input image and a template based on local image features extracted using a deep model. We train this model using pairs made of a real image and a synthetic image from a small set of training objects. Note that these training objects can be very different in appearance from the new objects.
We start this section with an analysis of the limits of global representations in Section 3.1. We then detail in Section 3.2 our training procedure. It relies on a similarity measure that compares the local features of real images and synthetic templates. At run-time, we use an extended version of this similarity function that explicitly estimates which local features in the input image are occluded and discards them. We discuss this in Section 3.3. Finally, we detail how we generate the templates in Section 3.4.

Motivation and Analysis
Here, we present two experiments that point out the main drawbacks of global representations in template matching when working with unseen objects.

Cluttered Background
A first drawback of global representations is their poor ability to represent unseen objects on cluttered backgrounds. To show this, we plot in Figure 2 the t-SNE visualization [37] of the global representations learned by [2] and local representations learned by our method for real images of training and new objects of the LINEMOD dataset.

Training
Incoming new objects  Figure 3: At training time, we use pairs made of a real image and a synthetic template to train a network to compute local features, from which the similarity between the two images can be predicted. At run-time, we apply this network to images of objects not seen during training to compute their local features. We can then retrieve the object pose by matching the image against the database of templates.
The first column of Figure 2 shows that both representations manage to cluster the images of each training object together, despite the fact that the images of the objects are captured with a cluttered background. The second column shows that global representations of [2] cannot disentangle the images of unseen objects, while our representations can. To better understand the reason behind this, we remove the background in the images by replacing it with a uniform color using the ground-truth object masks. As shown in the third column, the representations are now disentangled. This shows the influence of the background on the global representations for unseen objects, and that our representations are robust to cluttered backgrounds.

Pose Discrimination
A second drawback of global representations is their poor reliability when matching the real image of an unseen object with the synthetic template for the corresponding 3D pose, even when the object identity is known and the background is uniform. This can be explained by the fact that the pooling layers remove important information. This information loss appears to be compensated by the rest of the architecture for the training objects, but this compensation does not generalize to unseen objects.
To show this, we visualize in the supplementary material the correlation between pose distances and representation distances for unseen objects, as done in [39,2]. While both representations result in a strong correlation for training objects, this correlation is lost when considering unseen objects for the global representations but not for ours. Even without background, the correlation is still very low for global representations [2].

Framework
In each training iteration, we sample N positive pairs, where pair i is composed of a real image q i depicting a training object and of a synthetic template t i of the same object in a similar 3D pose. Following [39], we deem the two viewpoints similar if the angle between them is less than 5 degrees. All the pairs composed by a real image and a synthetic image of different objects or dissimilar poses (larger than 5 degrees) are defined as negative pairs. Triplet loss. [39] proposed a metric learning approach based on the intuition that the distance between feature descriptors for positive pairs should be closer in the learnt embedding space than negative pairs. To learn this property, [39] used a training loss L = L triplet + L pair where: • L triplet is the triplet term, which allows the network to learn features such that the distance in the learned embedding space between the positive pairs ∆ (i) + is lower than the distance between the negative pairs ∆ (i) − within the limits of the margin m. This triplet term is defined as + is the pairwise term, to minimise distances between two images of identical poses but different viewing conditions. [2] made an extension of this work by proposing a triplet loss which focuses only on learning object-discriminative features while using a pairwise loss to learn an embedding space analogous to the pose differences.
While these two losses work well, we experimentally show that the recent standard contrast loss InfoNCE [25] is the most simple and effective choice.
InfoNCE loss. For each real image q i , we also create N − 1 negative pairs by combining it with synthetic templates t k of other pairs in the current batch, with 1 ≤ k ≤ N, k = i. Altogether, this yields N positive pairs and (N − 1) × N negative pairs for each batch. We train our model to maximize the agreement between the representations of samples in positive pairs, while minimizing that of negative pairs with the InfoNCE loss function [25]: where sim(q, t) measures the similarity between the local image features q and t computed by the deep model for real image q and template t, and τ = 0.1, is a temperature parameter. As shown in Figure 3, q and t retain a grid structure and are 3-tensors. In practice, their dimensions depend on the size of the input image, ranging from 25 × 25 × C to 28 × 28 × C, with C = 16.
Local feature similarity. While previous works on contrastive learning [25,35,23,4,10,7,5,6] focused mostly on image classification and define the similarity metric sim(., .) using a global representation of the two images, we found such a representation to only classify well either known objects or images with a clean background, as discussed in Section 3.1.1. To effectively handle new objects and complex backgrounds, we use a metric based on a pairwise comparison of the local features in q and t. Specifically, we define where S is a local similarity metric, M is a 2D binary visibility mask for template t, and index l indicates a 2D grid location. q (l) and t (l) are thus local features of dimension C. Considering the template mask allows us to discard the background in the real image. Note that the mask does not account for possible occlusions in the real image as it corresponds to the object's silhouette in the template. Occlusions will be considered in the next subsection. As a local similarity metric S, we use the cosine similarity We empirically observed that measuring the similarity as the opposite of the L1 and L2 norms of the differences yields the same performance as the cosine similarity.

Run-time and Robustness to Occlusions
At run-time, given a real query image q, we retrieve the most similar template in a template set. To be robust to occlusions that can occur in the query image, we modify sim(q, t) as:  Figure 4: Illustration of feature similarity when not using the occlusion mask O (second row) and when using it. As discussed in Section 3.3, using O allows "turning off" the possible occluded local features in the similarity score.
where O (l) = 1 S(q (l) ,t (l) )>δ with δ a threshold applied to the cosine similarity to "turn off" the occluded local features as shown in Figure 4. In practice, we set this threshold δ = 0.2 through ablation study. Note that Eq. (5) can be written as the element-wise product and can be computed efficiently with:

Template Creation
On LINEMOD [12] and Occlusion-LINEMOD [3] datasets, we follow the protocol of [39] to sample the synthetic templates. More precisely, the viewpoints are defined by starting with a regular icosahedron and recursively subdividing each triangle into 4 smaller triangles. After applying this subdivision two times and removing the lower half-sphere, we end up with 301 templates per object.
On T-LESS [14], we follow the protocol of [31] by using a dense regular icosahedron with 2'536 viewpoints and 36 in-plane rotations for each rendered image. Altogether, this yields 92'232 templates per object. Besides, we also show our results with a coarser regular icosahedron with 602 viewpoints, which results 21'672 templates per object.
We use BlenderProc [9] to generate templates with realistic rendering for both settings.

Experiments
In this section, we first describe the experimental setup (Section 4.1). Then, we compare quantitatively and qualitatively our method with previous works [39,2,32,31] on both training (or seen) and unseen objects of the LINEMOD (LM) [12], Occlusion-LINEMOD (O-LM) [3] and T-LESS [14] datasets (Section 4.2). Finally, we provide  On T-LESS [14], we follow the evaluation protocol of [31] by training only on objects 1-18 under randomized backgrounds of SUN397 [41] and testing on the complete T-LESS primesense test set. More details about training set of T-LESS can be found in the supplementary material.
Evaluation metrics. For the LM and O-LM datasets, the pose error is measured by the angle between the two positions on the viewing half-sphere. We also treat the "Eggbox" and "Glue" objects as symmetric around the z-axis as done in [39,2].
In the case of known object pose estimation, the recognition score is almost 100% on LM and O-LM. Previous works [39,2] that focused on known objects thus only evaluate the pose error without considering whether the retrieved object is actually correct. In the case of unseen objects, we found that retrieving correctly both pose and class is important as the model can still get correct poses but from another object. Therefore, we propose using the Acc15 metric, which measures how often the pose error is less than 15 degrees and the predicted object class is correct. We also report the pose error in the supplementary material.
As most objects in T-LESS [14] are symmetric, we report the recall under the Visible Surface Discrepancy (err vsd ) metric at err vsd < 0.3 with tolerance τ = 20mm and > 10% object visibility as done in [32,31]. Unless otherwise stated in previous works [32,31], only templates of the same object are used at testing time (in other words, the class of the object is assumed to be known before testing). Please note that for the evaluation on the T-LESS dataset, we also predict the translation by using the same formula "projective distance estimation" of SSD-6D [18] as done in [32,31]. This translation is deduced from the retrieved template and the input bounding box of query image. More details can be found in the supplementary material.

Implementation details.
For a fair comparison, in the evaluation on LM and O-LM, we consider two different backbones: (i) "Base" -the simple backbone used in [39,2]; (ii) ResNet50 -the standard backbone used in recent contrastive learning methods [10]. We reimplemented [39,2] to get quantitative results in both seen and unseen objects. Our implementations get very similar performance when evaluated on the same data as the original papers on seen objects (see Table 2), validating our reimplementation.
We also follow [39,2] when testing with the "Base" backbone by using the same input image of size 64×64. While testing with ResNet50, we use a larger input size of 224×224. In both settings, we slightly change the architecture by removing all the pooling, FC layers and then replace them by two 1 × 1 convolution layers to output the desired local feature of size 16. As done in [39,2], we use the ground-truth pose to crop the input image at the center of objects and do not consider in-plane rotation (more details can be found in the supplementary material). On the T-LESS dataset, we use the same backbone ResNet50 and crop the input image with ground-truth bounding box as done in [32,31].
For both evaluations, we train our networks using Adam with an initial learning rate of 1e-2 for the "Base" backbone and of 1e-4 for ResNet50. Training takes less than 5h for all splits on a single V100 GPU when training on LM [12] and around 12h when training on T-LESS [14]. Table 2 presents the results of our method compared with previous work [39,2]. With either the "Base" or ResNet50 backbones, our method based on local feature similarities achieves the best overall performance in almost all settings compared to previous methods that compute the feature similarity between global image representations. While [39,2] explored carefully designed pairwise and triplet losses for learning an embedding space that is both objectdiscriminative and pose-discriminative, we find that using  Table 2: Comparison of our method with [39] and [2] on seen and unseen objects of LM and O-LM under the three different splits detailed at the beginning of Section 4.1. We report Acc15 ↑, the accuracy of predicting correctly the object identity and its pose with an error less than 15 degrees. We are on par on the "easy" case and outperform them by a large margin on the 3 other configurations. Using the InfoNCE loss rather than the loss from [2] brings some improvement, but the main improvement comes from our approach based on local features.
the InfoNCE loss as defined in Eq. (2) boosts the performance of all methods, in particular for our method based on local feature similarities.
When the objects are occluded, the accuracy of [39,2] drops to below 70% for training objects, while our method can still maintain a relatively high accuracy. This shows the robustness of local image features rather than global image representations that are much more strongly affected by the occlusions. Furthermore, the prediction accuracy of our method on unseen objects is clearly higher than that of previous methods, regardless of the objects being occluded or not. This indicates that matching based on local features is not only robust to occlusions, but also generalizes better to unseen objects. More importantly, this improvement on unseen objects holds still in the presence of occlusions.

T-Less Results
In Table 3, we shown that our proposed approach outperforms the state-of-the-art methods [32,31] on the T-LESS dataset by a large margin on both seen and unseen objects. While [31] carefully designed single-encoder-multidecoder network that allows sharing a latent space for all objects and having each decoder only reconstructs views of a single object, we find that using our method and Info-NCE loss is much more simple but also boosts significantly the performance in the same setting.

Ablation Study
We present several ablation evaluations on LINEMOD and Occlusion-LINEMOD.    Table 6: Average run-time of our method on a single GPU V100 and CPU Intel Xeon.
Effectiveness of feature masking. Table 4 shows the effectiveness of using the template masks M in Eq. (6) for unseen objects. Removing M results in a dramatic degradation for our method on all the three splits.
Influence of the threshold δ. Table 5 shows the influence of the threshold δ in Eq. (5) for estimating the occlusion mask O. Using O brings improvements on large objects ("Can", "Driller", and "Eggbox"). This can be explained by the fact that the occlusions can be very large in O-LM, especially on small objects, as shown in Figure 7.
Influence of the local feature dimensions. Figure 6 shows the pose error as a function of the dimension C of the local features and of the resolution of the feature maps and masks M. While C is not a critical value, the resolution is more important, as higher resolution allows discarding the background more precisely. Furthermore, this hyperparameter has a much stronger influence on the performance on the unseen objects compared to the seen objects.
Run-time. Table 6 provides run-times on CPU and GPU. Figure 6: Influence of the local feature dimension C and of the resolution of the local features and masks. Using a good resolution is much more important than using highdimensional local features as this allows discarding background more precisely when computing the similarity score.

Failure Cases
When evaluated on O-LM, both our method and [39, 2] fail on the "Cat" object. As shown in Figure 7, this object is small and particularly heavily occluded in this dataset.

Conclusion
We have presented an efficient approach to 3D object recognition and pose estimation that can generalize to new objects without the need for retraining and that is robust to occlusions. Our analysis has shown that a global representation, which discards the grid structure of images, is not robust to clutter and results in inaccurate pose predictions. Our method, based on local representations, has much better properties and can be made robust to occlusions. We hope that our analysis and our new approach will guide the development of more practical systems.

Pose Discrimination
As discussed in Section 3.1.2 of the main paper, a drawback of global representations is their poor reliability to represent the real image of a new object even when the object identity is known and the background is uniform. To illustrate this, we show in Figure 8 the correlation between pose distances and representation distances as in [39,2]. [39,2] provided such plots only for seen objects and RGB-D data, we consider here objects that have been seen or unseen and we use RGB data only. As in [39,2], the plots of Figure 8 are obtained by considering all possible pairs made of real images and synthetic images for a given object.
Ideally, the plots should exhibit a diagonal pattern, in the region closed to the (0, 0) point on the bottom-left of the graph. This region corresponds to the critical region for correct image/template matching. A diagonal pattern corresponds to a strong correlation between pose differences and distances between representations.
More plots are given in Section 9 and they all yield to the same conclusion: • The first column of Figure 8 shows that both representations result in a strong correlation for an seen object.
• The second column shows this correlation is lost when considering a new object for the global representation but not with ours.
• To check if this was due to the presence of clutter in the background of the real images, we removed the background by using the ground truth mask of the objects. The third column of Figure 8 shows that even without background, the correlation is still very poor for global representations. This can be explained by the fact that the pooling layers remove important information for unseen objects. We postulate that the rest of the architecture, in particular the fully connected layers learns to compensate this loss of information for seen objects, but such compensation is not possible for unseen objects.

Training details
Cropping on LINEMOD. Unless otherwise stated in previous works [39,2], the cropping on LINEMOD and Occlusion-LINEMOD is done by virtually setting a box, 40 cm in each dimension, centered at the object as shown in Figure 9. When all the patches are extracted, we normalize them to the desired crop size. Please note that with this cropping, we do not consider in-plane rotations, in other words, we omit one additional degree of freedom.
Seen object Unseen object After masking [2] Our method Figure 8: Visualization of the correlation between pose distances and representation distances, for understanding the discriminative power of different image representations for pose retrieval. First row is for [2], second row is for our method. Please see Section 6. Figure 9: Cropping on LINEMOD. the cropping on LINEMOD and Occlusion-LINEMOD is done by virtually setting a box centered at the object. Please see Section 7. Figure 10: Examples of training images. We use randomized background from SUN397 [41] with data augmentation, including Gaussian blur, contrast, brightness, color and sharpness filters from the Pillow library [8].
Data augmentation on T-LESS. As done in [32,31], we also apply data augmentation to the input images of T-LESS during training. We use Gaussian blur, contrast, brightness, color, and sharpness filters with the Pillow library [8]. Some training samples can be seen in Figure 10.
Pre-trained Features. We intialize the network ResNet50 with MOCOv2's features [7]. It has been shown in [42] that this initialization can improve both the   [39] and [2] on seen and unseen objects of LINEMOD (LM) and Occlusion-LINEMOD (O-LM) for the three splits detailed in Section 4.1 of the main paper. We report here the pose error, measured by the angle between the positions on the half-sphere for the ground truth pose and the predicted pose.

Projective distance estimation
As done in [32,31], we estimate 3D translation in the query image from the retrieved template and the input bounding box as detailed in Section 3.6.2 of [33]. More precisely, given known camera intrinsic of both real sensor K query and of the synthetic view K temp , we estimate the distancet query,z of real image: where ||bb (.) || is the diagonal of the bounding box and ||f (.) || is the focal length. Then, we can estimate the vector to transform from the object center in the synthetic view to the query image: ∆t =t query,z K −1 query bb query,c −t temp,z K −1 temp bb temp,c (8) where bb (.),c is the bounding box centers in homogeneous coordinates.
Finally, the 3D translation in the query imaget query can be estimated as :t query =t temp + ∆t (9) wheret temp = (0, 0,t temp,z ), the translation from camera to object center in the synthetic view.

Quantitative Results
We show in Table 7 quantitative results with pose error, measured by the angle between the two positions on the viewing half-sphere.

[2]
Unseen [2] Our method Our method Figure 13: Visualization of the correlation between pose distances and representation distances on Split #1: of unseen objects of LINEMOD (two first rows and two first columns from the left of two last rows) and unseen objects of LINEMOD (fours last columns from the left of two last rows). Ideally, the plots should exhibit the diagonal pattern at the region closed to the (0, 0) point on the bottom-left that corresponds to the critical region for correct image/template matching, showing a strong correlation between pose differences and representation distances. The plots of seen objects of LINEMOD show that both representations result in a strong correlation for training objects. The plots of unseen objects of LINEMOD show this correlation is lost when considering a new object for the global representation [2] but not with ours.  Seen [2] Our method

[2]
Unseen [2] Our method Our method Figure 15: Visualization of the correlation between pose distances and representation distances on Split #2: seen objects of LINEMOD (two first rows and two first columns from the left of two last rows) and unseen objects of LINEMOD (fours last columns from the left of two last rows).  Our method Figure 17: Visualization of the correlation between pose distances and representation distances on Split #3: seen objects of LINEMOD (two first rows and two first columns from the left of two last rows) and unseen objects of LINEMOD (fours last columns from the left of two last rows).