{'fraction_non_alphanumeric': 0.032165052348692685, 'fraction_numerical': 0.012625272941044735, 'mean_word_length': 4.927161108345778, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
ACR-Pose: Adversarial Canonical Representation Reconstruction Network for Category Level 6D Object Pose Estimation

Recently, category-level 6D object pose estimation has achieved significant improvements with the development of reconstructing canonical 3D representations. However, the reconstruction quality of existing methods is still far from excellent. In this paper, we propose a novel Adversarial Canonical Representation Reconstruction Network named ACR-Pose. ACR-Pose consists of a Reconstructor and a Discriminator. The Reconstructor is primarily composed of two novel sub-modules: Pose-Irrelevant Module (PIM) and Relational Reconstruction Module (RRM). PIM tends to learn canonical-related features to make the Reconstructor insensitive to rotation and translation, while RRM explores essential relational information between different input modalities to generate high-quality features. Subsequently, a Discriminator is employed to guide the Reconstructor to generate realistic canonical representations. The Reconstructor and the Discriminator learn to optimize through adversarial training. Experimental results on the prevalent NOCS-CAMERA and NOCS-REAL datasets demonstrate that our method achieves state-of-the-art performance.


Introduction
6D object pose estimation is at the core of many applications, such as robotic grasping [3,32,39], augmented reality [2,4,41], and autonomous driving [16,35,36]. Given an observation (RGB, RGBD, or LiDAR scan), 6D object pose estimation aims to estimate the rotation and translation of the target object in the camera coordinate system. Over the last decade, numerous 6D object pose estimation works [13,21,29,34,37,47,51] have emerged and been deployed in industrial products, demonstrating the usefulness of this line of research. However, most of the existing methods are instance-level approaches, i.e., a well-trained model works only for a particular object. This prohibits large-scale applications in the wild, as it is both memory-inefficient and labor exhausting to train and deploy models per object.
To address this issue, category-level 6D object pose estimation has become increasingly popular recently among academia and industry. Given a RGBD observation as input, a deep learning network is often used to predict a canonical representation [43] of the target object. The canonical representation refers to a normalized 3D model that integrates a category of objects' most representative characteristics, independent of the object's pose, such as the NOCS [43] and the CASS [5] representation. Once the canonical representation is obtained, it can be used to align with the backprojected object depth observation to recover the object's pose. Therefore, the key to successful category-level object pose estimation is to accurately reconstruct the canonical representation.
Although existing methods [5,6,23,38,43] have achieved promising 6D object pose estimation performance through the canonical representation, their reconstruction quality still needs improvement. Specifically, previous methods have the following drawbacks: 1) They are ineffective in learning canonical-related features due to their sensitivity to pose-related characteristics. 2) They ignore the critical relational information between different input modalities. 3) Some of their reconstruction results appear to be unrealistic.
In this paper, we propose an Adversarial Canonical Representation Reconstruction pipeline called ACR-Pose, which aims at reconstructing high quality canonical representations for precise 6D object estimation as shown in Fig. 1. In ACR-Pose, we first detect and segment out objects from the RGBD images. Then, taking the image patch and depth region of an object as input, we train a Reconstructor and a Discriminator in an adversarial manner to generate realistic canonical representations.
Specifically, as the Reconstructor plays a significant role in generating high-quality canonical representations, two novel sub-modules are proposed to learn more discriminative and robust features within the Reconstructor. First, we note that observations of the objects contain both poserelated information (rotation and translation) and shape information, and existing canonical representation reconstruction models [38,43] are sensitive to pose-related characteristics. However, since the goal of reconstruction is canonical, the rotational and translational cues in the object observation must be filtered out, while only the shape information should be preserved. To achieve this, we introduce a Pose-Irrelevant Module (PIM) so as to learn canonicalrelated features only. Second, inspired by [38], we take a learned shape prior as an input of our network, in addition to the RGB and depth images. We hypothesize that the relational information between these three sources of information is important for high-quality reconstruction. Therefore, a Relational Reconstruction Module (RRM) is introduced to learn this relational information of the three inputs. Finally, utilizing the features learned from the RRM, the canonical representation can be reconstructed using a Shape Prior Deformation step [38]. Afterwards, 6D object pose can be easily solved by aligning the canonical representation with the back-projected object depth using the Umeyama algorithm [40].
Furthermore, we find that the reconstruction results could be unrealistic in some cases, which may cause inaccurate alignment results. To resolve this issue, we adopt the idea of Generative Adversarial Network (GAN) [55,56] in our pipeline and propose to use a Discriminator to adversarially guide the Reconstructor, making it more creative so that it can produce more realistic canonical representations. As a result, experiments on the NOCS-CAMERA and NOCS-REAL datasets show that our ACR-Pose can achieve state-of-the-art performance in both synthetic and real scenarios.
To summarize, our contributions are: 1) We propose ACR-Pose, an Adversarial Canonical Representation Reconstruction Network, for learning highquality canonical representations trained in an adversarial manner. To the best of our knowledge, we are the first to introduce adversarial training for category-level 6D object pose estimation.
2) We propose a Pose-Irrelevant Module and a Relational Reconstruction Module to form the ACR-Pose's Reconstructor, through which high-quality canonical-related features and relational features are effectively learned for credible reconstruction.
3) We conduct extensive experiments to evaluate our model and compare it with previous works, where our model achieves state-of-the-art performance on both synthetic and real world datasets.

Related Work
In this section, we briefly review recent methods in 6D object pose estimation. Since our model is a deep learning model, we mainly introduce the deep learning-based counterparts. Then, we present works related to adversarial training.

6D Object Pose Estimation
Instance-level 6D object pose estimation only estimates the 6D pose of a particular object can be divided into five parts: direct-methods [17,47,53], keypoint-based methods [29,34,37], dense coordinate-based methods [22,27,51], refinement based methods [21,25,48] and self-supervised methods [33,42]. There are also many methods propose to utilize RGBD data as input for instance-level object pose estimation [12,13,31,46]. For more details, we refer readers to Fan et al. [8] for a comprehensive overview. Though promising, generalization of instance-level methods is limited because we have to train different models for different objects even though they belong to the same category. This requires huge computational costs at both training and inference time. In this paper, we explore category-level 6D object pose estimation owing to its better generalization ability towards different objects.
NOCS [43] is the first deep learning-based categorylevel method. It proposes the Normalized Object Coordinate Space (NOCS) representation to handle huge shape variance between different objects. Then, Chen et al. [5] propose another canonical representation named Canonical Shape Space (CASS). Both NOCS and CASS attract the interests of the following works. For example, SPD [38] proposes a shape prior deformation method for accurate NOCS representation reconstruction utilizing a category-prior as an additional model input. Lee et al. [19] propose to simultaneously predict the NOCS representation and the metricscale object shape from the RGB observation for categorylevel object pose detection, trying to remove the dependence on the depth information. However, its performance still lags behind RGBD based methods. Recently, Dual-PoseNet [23] proposes a Dual-Pose Network to learn pose Figure 2. The pipeline of ACR-Pose. We use the image patch, depth region, and shape prior as inputs at Initialization Phase. We predict the NOCS representation using our Reconstructor. We adopt adversarial training using the Discriminator. consistency to improve the pose estimation accuracy, and FS-Net [6] introduces a novel data augmentation method for improving models' performance. Nevertheless, no matter what efforts they make to improve performance, most of the existing methods adopt the idea of reconstructing the canonical representation in their models.
Our method also reconstructs the NOCS representation for category-level 6D object pose estimation. We argue that the potential of the NOCS representation has not been fully explored. Therefore, we propose ACR-Pose to improve the reconstruction quality through adversarial training, which will be presented in detail later.

Generative Adversarial Networks (GANs)
Generative Adversarial Network (GAN) [9,26] is widely used in many generation tasks like style translation [15], image generation [24], human reconstruction [18], etc. Usually, GAN consists of a Generator and a Discriminator. The former is the main branch to predict the task goal, while the latter is used to distinguish real sample (ground-truth) and fake sample (output of the generator). The key insight of GAN is to use the Discriminator to guide the Generator through adversarial training. CGAN [9] introduces the conditional version of GANs, which makes the generation process of GANs being controlled. DCGANs [50] proposes to use GANs for unsupervised learning with CNNs. WGAN [1] improves the stability of learning, getting rid of problems like mode collapse. CycleGAN [56] proposes the concept of utilizing cycle consistency for training GANs in an unsupervised manner. Recently, there are aslo many works utilizing GANs for 3D vision tasks. For example, Li et al. [20] introduces PUGAN for point cloud up-sampling. Zhang et al. [52] introduces an unsupervised point cloud completion method mainly benefits from GAN inversion. There also exists methods like [28] adopts the idea of GANs to predict dense-coordinates for instance-level 6D object pose estimation.
In this paper, we propose a novel and effective approach called ACR-Pose. Its Reconstructor taking use of two delicately designed submodules: PIM and RRM, to generate pose insensitive and essential relational information to get accurate reconstruction results. The Reconstructor and the Discriminator are optimized adversarially. As far as we know, we are the first to adopt adversarial training in canonical representation reconstruction.

ACR-Pose
In this section, we introduce ACR-Pose in detail. We first illustrate the overview of our pipeline (Sec. 3.1). Then we introduce at length the Pose-Irrelevant Module (Sec. 3.2) and Relational Reconstruction Module (Sec. 3.3) respectively. Next, we shortly depict the Shape Prior Deformation step [38] (Sec. 3.4) we used for reconstruction. After that, we describe how the Discriminator and adversarial training (Sec. 3.5) improve the reconstruction reality. Finally, we present the loss function.

Pipeline
In this paper, we aim to reconstruct the 3D canonical representation, for example, NOCS [43] representation from a RGBD input. Fig. 2 illustrates the overview of our pipeline. It can be divided into three parts.
The first part is the initialization phase. A detector is used to get the image patch and depth region of the target object from the RGBD input. The image patch and depth region are the main input of the following networks. Similar to [38], we train an encoder-decoder network to learn a shape prior for each category and take the learned shape prior as one of the following network's input. Totally, we have three modalities of input.
The second part is the Reconstructor. In this part, the im-  Then, for each point xij in this group, nine low-level rotationinvariant geometry features (angles a1 to a4 and distances d1 to d5) and two spherical signals are calculated and utilized to learn the rotation-invariant convolutional kernels. Finally, a PointConv operation is employed to apply the learned kernels to the input point-wise features for feature updating.
age patch is fed into an Image Encoder backbone to learn instance RGB image features f I ∈ R U ×V ×C , where U and V are the image size. The depth region is back-projected into a point cloud (observed points) and fed into the PIM to learn canonical-related instance point features f P ∈ R Np×C , where N p is the number of observed points. The shape prior is fed into a simple Shape Prior Encoder to learn highdimensional category prior features f c ∈ R Nc×C . Then, learned features of the three modalities are integrated and enhanced by the RRM, which is another key component of our work. Finally, taking features learned by both modules as input, a Shape Prior Deformation [38] step is employed to reconstruct the NOCS representation.
In the Discriminator, the third part, we utilize an adversarial scheme during training. Specifically, the reconstructed NOCS representation and the ground-truth are fed into a Discriminator. The Discriminator would then judge the realistic level of the reconstructed NOCS presentations as well as the ground-truth, hence guiding and encouraging our Reconstructor to generate NOCS representations as realistic as possible. Thus, the Reconstructor and the Discriminator would compete with each other to become stronger. We call it Adversarial Reconstruction.
At inference time, given the reconstructed NOCS representation and the back-projected observed object depth, we use the Umeyama algorithm [40] to recover the 6D object pose. Next, we introduce the PIM and the RRM in the Reconstructor first.

Pose-Irrelevant Module in Reconstructor
The Reconstructor takes the back-projected object depth as one of its main inputs. A normalization filter is occupied to eliminate the impact of translation and a Rotation-Invariant Network is utilized to filter the impact of rotation. These two modules together comprise the PIM.
Specifically, denoting the back-projected point cloud of the object as P , we first filter out the impact of the translation by subtracting its geometric center, i.e.P = P − P , where P is the mean geometric center. Then, we inputP into 4 layers of Rotation-Invariant Convolution for learning rotation-invariant features, called canonical-related features. The Rotation-Invariant Convolution is illustrated in Fig. 3. Following [7], given the input point cloud and its corresponding point-wise features as input, we choose nine low-level rotation-invariant geometry features (see Fig.  3 middle) as well as two spherical signals [49] to learn rotation-invariant convolutional kernels using a MLP and a SE-block [14]. Then we use a PointConv [45] to apply learned kernels to point cloud features for feature update.
The output of the PIM is f P ∈ R Np×C , where N p is the number of observed points in the point cloud or the depth region. It will be incorporated with features of the other two modalities to achieve the NOCS representation reconstruction task.

Relational Reconstruction Module in Reconstructor
As mentioned before, the Reconstructor takes three modalities of data as input, they all contribute to reconstructing the NOCS representation. Therefore, to learn high-quality features for high-quality NOCS representation reconstruction, we propose the RRM, which consists of three different graph convolutional networks to learn and integrate three kinds of relational features in the feature space. Details of the three kinds of relational features are below: Relational instance feature is the point-wise feature of the observed object, including the texture feature and the geometry feature. To learn it, we first index each point's point feature and RGB image feature from f P and f I , and concatenate them to form a N p × 2C features matrix. A MLP is used to reduce the feature dimension to N p × C. Then, we construct a graph G(V, E) in the feature space using the feature matrix, where V represents nodes in the graph, a node means an observed point, and E represents edges. The adjacent relation in the graph is built using the K-Nearest-Neighbors (KNNs) algorithm. The KNN algorithm would first compute the pair-wise distance in the feature space for each point pair. Then, it chooses each point's K nearest neighbors to build adjacent edges. Next, an Edge-Conv [44] operation is adopted to learn features taking the graph as input. Finally, we use another MLP to increase the per-point feature dimension back to 2C. We denote the relational instance feature as f ins ∈ R Np×2C .
Relational deformation feature is the second kind of feature our network learns. It is mainly used to deform the shape prior into a canonical instance model of the object in the Shape Prior Deformation step (detailed in Sec 3.4). To achieve this goal, we should make the feature be able to implicitly reflect the relational context between the shape prior and the observed object. Therefore, as before, we use a graph network to learn this kind of relational information. Specifically, we use a MLP and an adaptive average pooling operation to embed the relational instance feature f ins into a global feature vector v ins . The global prior feature vector v c is obtained in the same way. Then, we repeat v ins and v c for N c times and concatenate them with the prior features f c . The concatenated features are fed into another MLP for dimension reduction. After that, we build a graph utilizing the output and pass messages between nodes using an Edge-Conv. The final output is the relational deformation feature.
We denote it as f d ∈ R Nc×C .
Relational assignment feature is the third kind of feature we learn. In the Shape Prior Deformation step, we need to assign the above-mentioned deformed shape prior to the NOCS representation (detailed in Sec 3.4). Since the deformed shape prior and the NOCS representation are from different parameter fields, we need to establish a oneto-many correspondence (an assignment matrix) between them. To build the correspondence, we should mainly take advantages of two kinds of data, i.e., the observed instance points and the shape variation. The former works for providing instance characteristics, and the latter works for reducing shape variations. To better utilize their features and flowing information between them, we repeat v ins and v c for N p times and concatenate them with the relational instance feature f ins . Then, we use a graph network as before to learn the assignment feature f a ∈ R Np×C .
The Relational Reconstruction Module uses graphs to learn high-quality features by integrating relational information between features of different modalities. Constructing graphs in the feature space is quite beneficial in learning more semantics about the object shape and category specialities. The learned features will be further used in the Shape Prior Deformation step for the final reconstruction (Sec. 3.4), which would all contribute greatly to highquality NOCS representation reconstruction, verified by experimental analysis.

Shape Prior Deformation in Reconstructor
After all necessary features are learned. We adopt the Shape Prior Deformation approach to the reconstruct the NOCS representation in the Reconstructor. Shape Prior Deformation is first proposed by Tian et al. [38]. For the integrity of the paper, we briefly introduce it here.
The Shape Prior Deformation step requires us to learn two matrices to transform the shape prior into the NOCS representation. They are the deformation field M d ∈ R Nc×3 and the assignment matrix M a ∈ R Np×Nc . We feed the relational deformation feature into a MLP for learning M d and use another MLP to take the relational assignment features as input to learn M a . The NOCS representation can be reconstructed by: where P c is the shape prior. Using the two-stage transformation operation, P c would first be transformed into the observed points' canonical instance model and then be transformed to the NOCS representation. The Shape Prior Deformation step ensures the robustness of the reconstruction. It serves as the final step of our Reconstructor.

Discriminator and Adversarial Training
Though the NOCS representation can be predicted by only using the Reconstructor, we observe that some reconstruction results are quite unrealistic, which may severely affect the 6D object pose estimation accuracy. In this paper, we propose to use an adversarial reconstruction strategy to solve the problem. Specifically, we propose to use a Discriminator to guide the Reconstructor. The Discriminator in our work consists of a MLP, a global max-pooling and two fully connected layers. It takes the reconstructed NOCS representation and the real one as input at training time. And the output is a probability value distributed from 0-1, where a higher value means a larger probability it regards the input point cloud as a real NOCS representation and vice versa. The optimization goal of the Discriminator is to minimize: where P nocs is the reconstructed NOCS representation andP nocs is the ground-truth.
Correspondingly, the optimization goal of the Reconstructor is to minimize: During training, the Discriminator would try its best to judge the difference between real and fake, while the Reconstructor would try its best to confuse the Discriminator. In the fierce competition, both of them will gradually become stronger and stronger, hence increasing the reality of the reconstructed NOCS representation.

Loss Function
Except for the adversarial loss L d and L g described before, we use the smooth L1 loss L corr between the reconstructed NOCS representation and ground-truth to encourage better one-to-one correspondence. We use the chamfer distance loss L cd between the deformed shape prior and the object's canonical instance model to preserve appearance information. We also use a cross-entropy loss L entro to encourage peak distribution of the assignment matrix M a and a L2 regularization loss L reg on M d to avoid collapsing deformation. The final loss function is: where γ 1 to γ 6 are balance terms.

Datasets
We evaluate our method on the NOCS-CAMERA dataset and NOCS-REAL dataset proposed by Wang et al. [43], which currently are the most widely used and authoritative datasets for benchmarking category-level 6D object pose estimation methods. The NOCS-CAMERA dataset is a synthetic dataset that contains 300K RGBD images (with 25K for evaluation) generated by rendering and compositing synthetic objects into real scenes. The NOCS-REAL dataset is a real-world dataset that contains 4.3K real-world RGBD images from 7 scenes for training, and 2.75K realworld RGBD images from 6 scenes for evaluation. Both datasets consist of six categories, i.e., bottle, bowl, camera, can, laptop and mug.

Implementation Details
We choose Mask-RCNN [10] as the detector. The Image Encoder is a PSP-Net [54] with a ResNet-18 [11] backbone. The Shape Prior Encoder is a MLP. We first train our model for 50 epochs on the NOCS-CAMERA train set and evaluate it on the NOCS-CAMERA val set. Then, we fine-tune it on the NOCS-CAMERA train set and NOCS-REAL train set for 10 epochs. And we evaluate our fine-tuned model on the NOCS-REAL test set. Note we also have to recover the object size, we simply use the average size of P c + M d as the result following [38]. More details are in the SuppMat.

Comparison with State-of-the-art
We compare the performance of ACR-Pose with some recent state-of-the-art methods including NOCS [43], CASS [5], SPD [38], DualPoseNet [23] and FS-Net [6], to verify the effectiveness of our method. All of them are strong baselines. For comparison, we compute the IoU50, IoU75, 5 • 2cm, 5 • 5cm, 10 • 2cm and 10 • 5cm metrics and report the mean Average Precision (mAP) to measure the performance.
Results on NOCS-CAMERA: Table 1 shows the result of our method and the results of all competitors on the val set of NOCS-CAMERA. It is obvious that our method outperforms all the strong baselines for a large margin. For example, SPD [38] shares the same backbone and the same Shape Prior Deformation step as our work, however, it is outperformed by our ACR-Pose by 16.3%, 15.1%, 9.3% and 6.3% in terms of the 5 • 2cm, 5 • 5cm, 10 • 2cm and 10 • 5cm, respectively, which is a significant improvement. The comparison between SPD [38] and our method demonstrates that the improvement really comes from our adversarial reconstruction scheme, which also benefits from the two proposed modules in the Reconstructor. They have increased our model's 6D object pose estimation performance by increasing the reconstruction quality and reality of the NOCS representations through adversarial training. Our method also outperforms the second-best method Du-alPoseNet [23] by 5.7%, 3.4%, 4.6%, 3.1% at the above four metrics respectively, which is also a great improvement. This further proves the effectiveness our method.
Results on NOCS-REAL: We also compare ACR-Pose with other methods on the NOCS-REAL test set. Results are shown in Table 2. Our method is defeated by FS-Net [6]

Versions
IoU50 IoU75 Table 3. Results of ablation study. We gradually add our designs one by one to the baseline to investigate their impacts.
at the IoU50 metric and defeated by DualPoseNet [23] at the 10 • 5cm metric. However, these two metrics are relatively loose metrics. Besides, the training setting of FS-Net [6] is strongly different from all the other baselines. For example, it uses the YOLO v3 [30] object detector to detect and crop image patches and depth regions, while all the other baselines use the Mask-RCNN detector. When it comes to other strict metrics, our method outperforms other baselines for a large margin. The excellent performance of our method on the NOCS-REAL dataset at more strict metrics demonstrate that ACR-Pose is powered with strong generalization ability and owns the potential of being deployed into real-world industry products. We contribute the superiority of our model into PIM, RRM and the adversarial reconstruction scheme.
We also illustrate the average precision (AP) vs. different thresholds on 3D IoU, rotation error, and translation error of our method on Fig. 5. We compare our method with the mean result of [44]. Both versions of our method achieve excellent performance. And our method performs well for all categories.   Qualitative results: To better exhibit the performance of ACR-Pose, we visualize some prediction results on Fig.  4. It can be seen that our model can predict accurate 6D object pose because the visualized bounding boxes have high overlap with the ground-truth and they can frame the target objects tightly. Our method achieves excellent performance in both synthetic and real scenarios. Fig. 6 also illustrates some failure cases. When there exists truncation or occlusion, our model sometimes fails. And some objects may be missed by the detector. We will explore how to solve these limitations in our future work. More visualization results can be found in the SuppMat.

Ablation Study
In this section, we conduct experiments on the NOCS-CAMERA dataset to investigate the effectiveness and necessity of our several design choices. In Table 3, we replace all of our novel network designs in ACR-Pose as MLPs and then gradually add them back one by one to study their impact on the final 6D object pose estimation results.
Impact of the Relational Reconstruction Module: The RRM is proposed to learn relational information between three different modalities in our Reconstructor. It includes three graphs that learn relational instance features, relational deformation features and relational assignment features respectively. From row 4, row 6 and row 7 of Table 3, we can find that all the three kinds of features contribute to the accuracy improvement because adding them one by one cumulatively increases the model's performance. The improvement comes from the message passing through graphs in the feature space. Using them, powerful semantic features can be learned by the model to better reconstruct the canonical representation. These semantics can effectively describe both object-specified shape and common categorylevel characteristics (category specialities), therefore, highquality NOCS representations can be reconstructed. The above results demonstrate that relational features are essential for high-quality NOCS representation reconstruction.

Impact of the Pose-Irrelevant Module:
The PIM contributes to learning canonical-related features to filter rotation-related and translation-related information that is irrelevant to high-quality NOCS representation reconstruction in the Reconstructor. When it is added to our baseline, all evaluation metrics are increased by 1% to 3%. That is because, without the interference of rotation and translation, the model can learn more information about the shape of the object's observed part, which is critical for reconstructing the NOCS representation.
Impact of Adversarial Reconstruction: The adversarial reconstruction scheme plays a role of increasing reconstruction reality. Besides, since the Discriminator is only required at the training time, it is an very efficient strategy for improving performance. The 5th row of Table 3 evidently verifies that the adversarial training scheme would help the model learn better, more realistic and more highquality canonical representations because adding it brings significant performance gains. That is because the reconstructed canonical representations would be more realistic after adding the Discriminator. Further, the following Umeyama algorithm [40] would also perform much better and more robust if it receives more realistic canonical representations.

Conclusion
In this paper, we present an Adversarial Canonical Representation Reconstruction Network, the ACR-Pose, for accurate category-level 6D object pose estimation. Our work focuses on reconstructing high-quality canonical representations for the observed objects, given the RGB patch, the depth region, and the learned shape prior. To achieve so, we propose an adversarial reconstruction scheme, which competitively trains a Reconstructor and a Discriminator to improve reconstruction performance. Besides, within our reconstructor, a Pose-Irrelevant Module is introduced to better extract the shape information of the objects, as well as a Relational Reconstruction Module that leverages the relational information among the three input sources. Experiments are conducted on synthetic and real datasets, where the effectiveness of each novel module is demonstrated, and our overall model achieves state-of-the-art performance.
One of the limitations of our method is that it may fail in the case of truncation or occlusion, which we aim to address as part of our future works.
[56] Jun-Yan Zhu, Taesung [11] backbone. The Shape Prior Encoder is a MLP. The category instances are sampled from the ShapeNet dataset. The feature dimension C is set to 64. During training, we alternately update the parameters of the Reconstructor and the Discriminator. The initial learning rates for training the Reconstructor and the Discriminator are 0.0001 and 0.00001 respectively. When training on the NOCS-CAMERA train set only, the learning rate is decayed at the 10th, 30th and 40th epoch. The decay rates are 0.5, 0.1, 0.01 w.r.t the initial learning rate repetitively. When fine-tuning, the initial learning rates are also 0.0001 and 0.00001, and they are decayed by half at the 5th epoch. The image patch is resized to 192 × 192. The number of object points (back-projected object depth) P is 1024. The number of shape prior points is also 1024. The number of adjacent neighbors K in the feature graphs is 36. The balance terms γ 1 to γ 6 in the loss function are 0.1, 0.1, 1.0, 5.0, 0.0001 and 0.01, respectively. When fine-tuning, we re-initial the discriminator and select data randomly from NOCS-CAMERA and NOCS-REAL at a ratio of 3:1 to train the network, following [38,43]. All experiments are conducted on a single A6000 GPU.

Overall accuracy
Except for the mAP metric we mainly report, we also calculate the overall accuracy at the threshold of IoU50, IoU75, 5 • 2cm, 5 • 5cm, 10 • 2cm and 10 • 5cm. We show the results on Table 4 and compare our ACR-Pose with SPD [38]. Our method outperforms SPD for a large margin in terms of overall accuracy. This is consistent with the result of using mAP as an evaluation metric, which further indicates the superiority of our method.

Generalization ability evaluation
To evaluate the generalization ability, we evaluate ACR-Pose on the NOCA-REAL test set using the model only trained on the synthetic NOCS-CAMERA dataset. We also compare our method with SPD here. The evaluation metric is mAP. Results are shown in Table 5. Our method demonstrates better generalization ability towards real-world scenarios compared to SPD. However, there is still much room for performance improvement compared to the model trained on both NOCS-CAMERA and NOCS-REAL.

Reconstruction quality evaluation
To verify whether our model can reconstruct high-quality NOCS representations, we evaluate the reconstruction quality of the results of SPD and ACR-Pose using the chamfer distance. Results are shown in Table 6. Our method significantly outperforms SPD. Since we use the same Shape Prior Deformation step and Umeyama algorithm for NOCS representation reconstruction and object pose estimation, we claim that the mAP improvement really comes from the higher quality NOCS representation. Note our work mainly benefits from the Pose-Irrelevant Module, the Relational Reconstruction Module and the adversarial reconstruction scheme, while SPD doesn't use these modules.

Impact of adjacent neighbors
For the Relational Reconstruction Module, in each graph, we use KNN to search for the adjacent neighbors for each node. In Table 7, we show the impact of the number of adjacent neighbors K. We conclude that setting K as 36 is the best choice. When K is too small, it may cause a small receptive field for learning relationships between different modalities. While when it is too large, it may cause overfitting and additional unnecessary computational costs.

More visualization results
To better understand the performance of our method. We illustrate more visualization results on Fig. 7 and Fig. 8. It can be seen that no matter in real-world scenarios or in synthetic scenarios, our model can accurately detect all the objects and recover their poses. We also show more failure cases in Fig. 9. Except for truncation and occlusion, our method may also sometimes suffers from ambiguities caused by symmetry. In our future work, we will investigate how to solve these disadvantages.

Methods
IoU50 IoU75 Table 7. Impact of the number of adjacency neighbors.