{'fraction_non_alphanumeric': 0.034659773546437546, 'fraction_numerical': 0.0152910765646048, 'mean_word_length': 4.849233390119251, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Finer-Grained Correlations: Location Priors for Unseen Object Pose Estimation

We present a new method which provides object location priors for previously unseen object 6D pose estimation. Existing approaches build upon a template matching strategy and convolve a set of reference images with the query. Unfortunately, their performance is affected by the object scale mismatches between the references and the query. To address this issue, we present a finer-grained correlation estimation module, which handles the object scale mismatches by computing correlations with adjustable receptive fields. We also propose to decouple the correlations into scale-robust and scale-aware representations to estimate the object location and size, respectively. Our method achieves state-of-the-art unseen object localization and 6D pose estimation results on LINEMOD and GenMOP. We further construct a challenging synthetic dataset, where the results highlight the better robustness of our method to varying backgrounds, illuminations, and object sizes, as well as to the reference-query domain gap.


Introduction
6D object pose estimation is a fundamental step in many computer vision and robotics tasks, such as augmented reality [1], autonomous driving [5,7,18,36], and robotic manipulation [6,27,40]. As novel (unseen during training) objects are ubiquitous in real scenarios, we have witnessed an increasing number of methods designed to generalize to such novel objects without re-training or finetuning [17,20,23,25,38].
Typically, generalizable methods follow a two-stage strategy, i.e., first detecting the object in the image and then estimating the object pose from the cropped detected region. Unfortunately, for novel objects, the detection problem remains far from being solved. Specifically, [9,20,38] rely on the ground-truth object bounding boxes even at test time. Although some unseen object detectors have been presented in [17,23], the performance of 6D object pose estimation undergoes a dramatic drop when using a trained object detector instead of the ground truth. In this paper, we therefore focus on localizing novel objects in RGB images for 6D object pose estimation. Existing methods [17,19,23] handle a novel object by comparing the input query image with a set of reference images which depict the object in different orientations. The features extracted from the references are convolved with those of the query. The object is then located in the query region with highest response. We argue that previous methods suffer from mismatched object scales. As illustrated in Fig. 1, the object scales in images typically differ. In such context, the convolution responses tend to be unreliable because the query regions could be dominated by the background (Fig. 1  (b)) or contain only a part of the object (Fig. 1 (c)). In principle, this problem can be addressed by scaling the query image to different resolutions and performing convolutions on all resized images. However, due to the limitation of computational resources, previous works [17] resort to using only a few query scales. The problem of object size mismatches still exists.
We tackle this problem by introducing a new correlation module, where we perform a finer-grained yet computationally-friendly scale adaptation via adjustable convolutional receptive fields. Specifically, we leverage the fact that a convolution kernel, i.e., reference feature map, always contains information about the whole object regard-less of the spatial resolution, because the kernel is converted from a reference image which only depicts the object. By contrast, the size of a query region covered during each convolution is related to the kernel size. Therefore, We propose to adjust the receptive field of the reference-query convolution over query images by resizing the reference kernels to different scales. Note that the reference kernel is much smaller than the query image, and resizing reference feature maps only requires a single forward pass through the backbone network. Our strategy is thus more efficient and scalable than resizing query images. The 2D object center location and 2D object size are predicted from the estimated correlations, which are represented as a heat map and a scalar, respectively. Intuitively, the correlation score of the object center should be always maximum in the heat map, which is then insensitive to object size variations. By contrast, predicting the 2D object size requires the awareness of size variations. Therefore, we feed the correlation maps into two separate heads that extract scale-robust and scale-aware features to estimate the 2D object location and size, respectively.
Our experiments on LINEMOD [10], GenMOP [17], and a challenging synthetic dataset evidence that our method achieves considerably better localization accuracy than previous methods for novel objects. We extend our method to 6D object pose estimation by combining it with the offthe-shelf 3D rotation estimator and 6D pose refiner of [17]. Ours results demonstrate that the object location priors obtained with our method yield a significant improvement in unseen object 6D pose estimation accuracy. Our approach is also more robust to phenomena such as varying backgrounds, illuminations, and object scales, as well as to the reference-query domain gap.
Our contributions can be summarized as follows: • We introduce a generalizable object detector which can robustly locate novel objects in RGB images.
• We design a correlation module that captures finergrained query-reference similarities with adjustable receptive fields.
• We propose to predict the 2D object location and size by extracting scale-robust and scale-aware features from the correlations, respectively.

Related Work
Instance-specific 6D pose estimation. 6D object pose estimation has been widely studied in the literature. Most existing deep learning methods are instancespecific [21,28,30,33], which means the training data and testing data contain the same object instances. Although each object instance is observed under different poses during training and testing, the multi-view images in the training set make the deep networks able to memorize the object patterns. However, such assumption conflicts with the demand in real applications, where objects not included in the training data inevitably exist. The appearance of the novel objects could significantly differ from the appearance of the training ones. Therefore, the ability of instance-specific approaches to handle novel objects is seriously impeded [38], Generalizable 6D object pose estimation. Recently, some efforts have been made towards unseen object pose estimation [17,31]. In practice, the unseen objects can be either new instances from seen categories, or objects from entirely novel categories. In the former case, existing approaches [3,13,31] estimate normalized object coordinate space (NOCS) maps from RGBD images, and compute the 6D object pose from 3D-3D correspondences. As the training and testing data contain the same object categories, offthe-shelf detection or segmentation methods [8,16,22] can be employed to locate the objects. However, the categorylevel approaches have difficulty in generalizing to objects from new categories. Moreover, the required depth information is not always available, making the RGBD-based methods restricted to specific scenarios. In such context, solutions that handle objects from new categories using RGB images have been proposed. However, the vast majority [9,20,35,38] assumes that the ground-truth object bounding boxes are available even at test time. The approaches [17,23] independent to ground-truth bounding boxes build upon a template matching strategy, which suffers from inaccurate object localization because of the object size mismatches. Developing a detector capable of accurately locating novel objects in RGB images, is therefore pivotal to generalizable 6D object pose estimation methods.
Object localization. Most object detection and segmentation methods, such as Mask-RCNN [8] and YOLO [22], follow a category-level learning pipeline, which limits their generalization ability to objects from new categories. To address this problem, some methods [4,17,23,39] utilize a template matching strategy. Specifically, they assume the availability of reference images which contain the novel object as templates. Given a query image, correlations between the query and references are computed, and the object is expected to be located in the query region with the highest correlation score. Such template matching approach has shown promising generalization ability to unseen objects when the objects contained in the query and references are in similar sizes [17]. Note that the correlations are estimated with the same receptive fields over the query and references. Therefore, it becomes less effective when the object scales differ between the query and reference images. By contrast, in this paper, we introduce a finer-grained correlation module with adjustable receptive fields, which makes our

Feature Extraction Correlation Estimation Module Translation Estimator
Network architecture. Our network takes a query and a set of references as input. The feature extraction backbone is shared by the query and references. We introduce a finer-grained correlation module to compute correlations between the query and reference feature maps with adjustable receptive fields (indicated by different colors). MP and CAT represent max pooling and concatenation, respectively. The correlations are then fed into a scale-robust head and a scale-aware head, which output the 2D object location and size, respectively. The 3D object translation is estimated by using Eq. 5.
approach more robust to object size mismatches.

Problem Formulation
Given an RGB query image I q , we aim to locate an object O, with the goal of later using the object location information for 6D object pose estimation. We detect an object by predicting the 2D location of the object center c q and the object size s q . In contrast to the definition in the object detection domain [8,16,22], the ground-truth object size is defined as s * q = 2f /d in the context of 6D object pose estimation, wheref is a virtual focal length [17] and d is the depth of the object center. In this paper, we focus on unseen object localization, which means that the objects contained in the training and testing sets differ, i.e., O train ∩ O test = ∅. Following [17,23]

Motivation
In essence, the problem formulation described above follows a template matching approach which compares the input query to template images. Such a strategy has shown promising generalization ability to unseen objects [17,23,38]. Typically, existing methods employ a Siamese network [12] to extract features from the query and reference.
The reference feature map is then used as a template, convolved with the query feature map. The object is detected by finding the query region with the highest response. However, as shown in Fig. 1, the convolution tends to be affected by the object size mismatches between query and reference images. Previous work [17] proposes to alleviate this problem by resizing the query to different resolutions and feeding all resized images to the network. Due to limited computational resources, only a few resolutions are selected, e.g, 4 resolutions with the largest scale factor of √ 2 in [17]. Therefore, such resizing process still has difficulty in aligning the mismatched object scales. This observation motivates us to adjust the receptive fields while performing the convolutions, which we achieve by resizing the convolution kernels. As the varied receptive fields correspond to query regions with different resolutions and interpolation over small convolution kernels is much more efficient than over the full query image, our approach has better potential for robust unseen object localization. We present the details of our approach in the following sections. It takes I q and I r as input and estimates correlations aŝ C q = F (g(I q , Θ), g(I r , Θ)), where g(·, ·) represents the feature extraction module, Θ indicates the set of learnable parameters shared by I q and I r , and F (·, ·) denotes the correlation estimation module. As illustrated in Fig. 3, the correlations are computed by convolving the reference features Figure 3. Finer-grained correlation estimation module. The reference kernels are resized to different resolutions (indicated by different colors) by using bilinear interpolation when performing convolutions over the query feature map. The convolved results (c1, c2, c3) capture information from query with varied receptive fields as represented by the blue boxes, which facilitates the robustness against object size variations.  with the query features. This can be formalized as

Finer-Grained Correlations
where F r ∈ R C×Hr×Wr is a reference kernel, F q ∈ R C×Hr×Wr indicates the region in the query feature map covered by the kernel at each spatial location during the convolution, and u · v denotes a dot product. Eq. (1) shows that the receptive field on the query image is related to the spatial size (H r , W r ) of F r . To adjust this receptive field, we propose to resize F r to different resolutions r , by using bilinear interpolation. We then stack the resulting correlations as where c ∈ R Nc and CAT represents concatenation. As shown in Fig. 3, each c i ∈ c (represented by different colors) correspond to a specific resolution in the query image indicated by the blue box. Therefore, c accounts for object size variations in the query. By resizing the reference kernels instead of the query image, each image is only fed once through the feature extraction module g(·, ·), yet multiple correlations c ∈ R Nc can be computed. Consequently, our method is computationally more efficient than resizing the query image N c times. In Fig. 4, we provide a visualization example where we resize the convolution kernel computed from a reference to three different spatial resolutions, i.e., 9 × 9, 7 × 7, and 5 × 5. One-time forwarding through g(·, ·) results in three different correlation maps, in which the location with the maximum correlation score varies, as indicated by the red dash circles. By jointly considering the correlations, our method can robustly capture object location information. Given N r reference images, we compute the corresponding correla- where C ∈ R Nc×Nr . Note that the reference images are captured from different viewpoints, some of which are far from the query viewpoint. The appearance of objects in different poses could be dissimilar, which would result in low correlation scores. Therefore, we only keep the correlations with the most similar reference by max pooling C toĈ: R Nc×Nr → R Nc .

Decoupled Translation Estimator
Given the correlationsĈ for each spatial location in the query feature map, we represent the complete correlation map asĈ q ∈ R Nc×Hq×Wq . A naive way to locate the object would consist of usingĈ q as input to jointly predict the 2D object location and size. However, we argue that these two tasks have conflicting requirements. Let us explain the intuition behind by taking the images shown in Fig. 5 as an example.
As will be described in detail below, we predict the object location as a heat map. Therefore, we expect that the response of the object center (x 1 , x 2 , and x 3 in Fig. 5) is always maximum regardless of the object size. Recall that the convolution receptive field over the query varies when resizing a reference kernel to different resolutions. The correlations with improper receptive fields may lead to low correlation scores such as (Ĉ 31 ,Ĉ 32 ), and would thus interfere with the object location prediction. To alleviate this issue, we define a scale-robust head that performs a max pooling over the correlation mapĈ q : R Nc×Hq×Wq → R Hq×Wq . The max pooling only retains the largest correlation, which we expect to come from the reference kernel in the correct size.
By contrast, to predict the object size, one needs to reason about the object patterns at different scales. Such information is contained in the multiple correlation scores

Relative Scales
Scale-Aware Head Figure 5. Decoupled Translation Estimator. Given Nq query images (I 1 q , I 2 q , I 3 q ) contain an object in different sizes,Ĉ1,Ĉ2, andĈ3 are the estimated correlations, corresponding to the object center locations x1, x2, and x3 in the queries, respectively. The scale-robust features and scale-aware features are separately extracted by using max pooling and learnable linear transformation when computing the object location and size.
(Ĉ 31 ,Ĉ 32 ,Ĉ 33 ). This property, however, would be lost after the max pooling layer used in the scale-robust head. Therefore, we design a scale-aware head that exploits the full information available inĈ ∈Ĉ q via a learnable linear transformation defined as where w ∈ R Nc×D indicates the learnable parameters. Let us denote the resulting scale-robust and scale-aware feature maps as Y rb ∈ R Hq×Wq and Y aw ∈ R D×Hq×Wq , respectively. We then predict the object center heat map M q from Y rb using three convolution layers, and extract the 2D object center location as Consequently, we employ c q to locate the object features y q on Y aw , and compute the object size as s q = s * s r , where s is a scalar learned from y q by using convolutions. Given the predicted object location c q and size s q , we compute the 3D object translation as where K is the known camera intrinsic matrix, andĉ q = [c q , 1] T .

Experiments
Implementation details. Following the implementation in [17], we employ a pretrained VGG-11 [24] network as the feature extraction module. The detail of our network architecture is provided in the supplementary material. We train our network for 300, 000 iterations using the Adam optimizer [11] with a batch size of 8 and a learning rate LINEMOD GenMOP Figure 6. Query and reference images on LINEMOD [10] and GenMOP [17]. Images in the left column are the queries and the image groups in the right column are the references.

Experimental setup.
We conduct experiments on two public datasets, i.e., LINEMOD [10] and GenMOP [17]. Some examples of the query and reference images in these two datasets are shown in Fig. 6. To achieve a fair comparison, we follow Gen6D's [17] settings and train our network using the same training data as Gen6D. Note that all objects in the testing phase are previously unseen, i.e., not included in the training data. We compare our method with existing ones from three perspectives, i.e., unseen object localization, robustness, and unseen object 6D pose estimation.
Performance of unseen object localization on LINEMOD [10]. mAP@[.5:.95] (%) [15] Table 1 and Table 2 provide the mAP@[.5:.95] (%) [15] of the evaluated methods on LINEMOD and GenMOP, respectively. We compare our approach to Gen6D [17], which, as mentioned before, is the most related work to ours. In addition to Gen6D, we report the results of OSOP [23], which is an advanced version of one-shot object detection methods such as OS2D [19], and achieves better performance as shown in [23]. We employ the pretrained model provided by the authors for Gen6D. As the OSOP code was not released, we re-implement and train OSOP from scratch, following the same empirical settings as for Gen6D and our approach. We sample the same 32 reference images for all evaluated methods, which leads to a fair comparison. As shown in Table 1 and Table 2, our approach achieves better mAPs in all cases, e.g., 12.96% increase on LINEMOD and 5.82% increase on GenMOP on average. Fig. 7 depicts some qualitative results. As explained in Sec-  .95] (%) [15] under different scale ratios. tion 3.1, the definition of ground-truth bounding boxes differs from the one in the object detection domain [8,16,22]. Therefore, all visualized object bounding boxes are square. Note that all bounding boxes predicted by our method are closer to the ground-truth ones.

Robustness Evaluation
As shown in Fig. 6, in Gen6D's experimental setup, the references are real images taken from different viewpoints. The object size and background in the reference images are therefore similar to the ones in the query images. In real applications, however, these factors tend to differ. Moreover, OSOP Gen6D Ours Figure 9. Qualitative results on synthetic LINEMOD. The ground-truth and predicted bounding boxes are colored by green and blue, respectively. the reference images may often be synthetic ones when the 3D object model is available [23,38], which leads to a domain gap between the query and references. Taking all these possible challenges into account, we construct a synthetic dataset to evaluate the robustness of the approaches. Specially, we keep the original reference images, but render the 3D object models of LINEMOD from different viewpoints with diverse illumination conditions to generate synthetic query images. We randomly assign a background to each synthetic query using images from SUN2012 [34]. Some exemplar query images are shown in Fig. 8(a). Note that we only employ this dataset for testing without any retraining or fine-tuning on it. The results on the synthetic dataset are provided in Table 3. The mAPs of all evaluated methods decrease compared with those in Table 1, which demonstrates that our synthetic dataset is more challenging. Nevertheless, our method still surpasses previous works by a large margin (at least 16.51%). The qualitative results in Fig. 9 evidence that our method is able to consistently detect the objects, while the competitors fail in some cases.
To further analyze robustness to object size variations, we resize the objects in the synthetic query images with different scale ratios. This can be formalized asŝ q ∈ [s q /p, s q * p], where p represents the scale ratio. We sample 6 scale ratios p ∈ {1.0, 1.2, 1.4, 1.6, 1.8, 2.0} and report the mAP@[.5:.95] in all these cases. The dataset becomes more challenging as p increases. As shown in Fig. 8(b), the performance of our method is consistently better than that of previous works when the scale ratio varies. This observation further demonstrates the better robustness of our method to various noise sources.

Unseen 6D Object Pose Estimation
We extend our method to 6D unseen object pose estimation by computing 3D object translation using Eq. 5 and estimating 3D rotation employing an off-the-shelf ro- Table 4. Performance of 6D pose estimation for unseen objects on LINEMOD [10]. We employ ADD-0.1d (%) [10] as the evaluation metric. "Retrain" indicates if the method is retrained on the unseen objects.

Method
Retrain Refine Avg. PVNet [21] 38.79 RLLG [2] 2.71 ObjDesc [32] 8.55 Gen6D [17] 17.90 Ours 26.87 Gen6D [17] Volume [17] 50.39 Ours Volume [17] 50.61 tation estimator [17]. In Table 4 and Table 5, we report the ADD-0.1d [10] for all evaluated methods on LINEMOD and GenMOP, respectively. As the instance-specific methods such as PVNet [21] cannot generalize to unseen objects,  the models are retrained on testing objects. Therefore, we only compare our method with other generalizable competitors that are not retrained. Note that we assume that the depth information is unavailable, so LatentFusion [20] and category-level approaches [3,13,31] are not evaluated. We also exclude OSOP [23] because it requires 90, 000 reference images for each object during object pose estimation, which becomes intractable for the 3000 objects in the training set used in [17]. Our method yields better ADDs than existing generalizable approaches when no refinement is performed, which truly shows that the 6D object poses obtained with our method are more accurate. When combined with the same refinement approach as [17], our method also performs better, which leads to competitive results compared with instance-specific methods. These observations evidence that 6D unseen object pose estimation can benefit from accurate object localization.

Ablation Studies
To shed more light on the effectiveness of each component in our method, we perform ablation studies using our constructed synthetic dataset, which lets us evaluate both object detection accuracy and robustness to noise. We evaluate two variants of our approach: One where we only convolve the original reference kernels with the queries, i.e., excluding the kernel resizing process; and one where we exclude the decoupled translation estimator from our framework, replacing it with a coupled one. Specifically, we ex-tract features from the estimated correlations by using Eq. 3, and jointly learn 2D object location and size from the features. As shown in Table 6, every component results in an increase of mAP, and the best performance is achieved by combining all components.
Note that, in our method, the receptive fields are adjusted by resizing the reference kernels when computing correlations. We claimed in Section 3 that such a design is more computationally friendly than resizing the query images. To evaluate the efficiency, we conduct an experiment comparing different types of resizing from the perspectives of multiply-accumulate operations (MACs) and speed. Specifically, we resize the query image, reference image, or reference kernel to different scales with the number of scale factors changing from 1 to 5. As shown in Fig. 10, MACs of the network significantly increase and the speed is much slower when multi-scale queries or references are used. By contrast, the network with our "kernel resizing" approach is faster and requires fewer MACs. The observations can be explained from two perspectives. First, estimating correlations with M receptive fields only requires one forward pass through the feature extraction module g(·, ·) in our framework, whereas M forward passes are needed when resizing the image. Second, the reference kernel is much smaller than the original query or reference image, e.g., 5 × 5 vs. 480×640. Consequently, the resizing process is much faster for reference kernels. We provide more results related to the effect of kernel resizing in the supplementary material.

Conclusion
In this paper, we have presented a generalizable object detector, which can be used to provide location priors for previously unseen objects in a 6D object pose estimation pipeline. To handle the object size mismatches between query and reference images, we have introduced a finergrained correlation estimation module, which efficiently captures object information with adjustable receptive fields. We have also proposed to predict the 2D object location and size in a decoupled manner, by separately extracting scale-robust and scale-aware features from the correlations. We have conducted comprehensive experiments on LINEMOD, GenMOP, and a challenging synthetic dataset, where our method achieves state-of-the-art performance and better robustness to typical noise sources.
Limitations and discussions. We see two main limitations of our method, which mainly come from the template matching paradigm, including the requirement of access to a number of reference images and the difficulty of locating multiple objects simultaneously from a single input. Therefore, in future work, we plan to study alternative approaches for unseen object localization and 6D pose estimation, going beyond the template matching strategy.