{'fraction_non_alphanumeric': 0.03289654901034903, 'fraction_numerical': 0.01145083225121001, 'mean_word_length': 4.944795321637427, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
DeepRM: Deep Recurrent Matching for 6D Pose Refinement

Precise 6D pose estimation of rigid objects from RGB images is a critical but challenging task in robotics and augmented reality. To address this problem, we propose DeepRM, a novel recurrent network architecture for 6D pose refinement. DeepRM leverages initial coarse pose estimates to render synthetic images of target objects. The rendered images are then matched with the observed images to predict a rigid transform for updating the previous pose estimate. This process is repeated to incrementally refine the estimate at each iteration. LSTM units are used to propagate information through each refinement step, significantly improving overall performance. In contrast to many 2-stage Perspective-n-Point based solutions, DeepRM is trained end-to-end, and uses a scalable backbone that can be tuned via a single parameter for accuracy and efficiency. During training, a multi-scale optical flow head is added to predict the optical flow between the observed and synthetic images. Optical flow prediction stabilizes the training process, and enforces the learning of features that are relevant to the task of pose estimation. Our results demonstrate that DeepRM achieves state-of-the-art performance on two widely accepted challenging datasets.


I. INTRODUCTION
Detecting objects and estimating their 6 dimensional pose (x, y, z, roll, pitch, yaw) in 3D space is a fundamental task in the field of computer vision and robotics. As such, it has many applications, the most common of which is robotic manipulation. For a robot to be able to effectively interact with an object, it must know the object's pose in relation to itself. In the case of robotic grasping, the object's position is used to determine the input to the inverse kinematic solver, which can then calculate the joint states necessary to grasp the object. Augmented reality has also recently become a popular field requiring very precise pose estimation [1]. In this setting, pose estimation enables humans to interact with both physical and virtual objects in a seamless manner. Applications range across industries such as healthcare, manufacturing, education, and gaming.
Estimating the 6D pose from a single RGB image is an ill-posed problem due to the loss of dimensionality in projecting the 3D scene onto the 2D image sensor. Because of this, many solutions rely on depth sensors to recover the lost depth information. Depth sensors, however, are typically limited by factors such as cost, power, form factor, range, resolution, frame rate, and sensitivity to external factors such as sunlight [2], [3]. Furthermore, recent advancements in computer vision and AI are enabling RGB only solutions to approach the same levels of accuracy as RGB-D. In the 2020 BOP Challenge on 6D Object Localization [4], CosyPose [5], an extension of DeepIM [2], relied only on RGB data and outperformed all but two RGB-D approaches. Our intention in this work is to close the gap between RGB and RGB-D approaches by focusing on pose refinement with RGB only data, enabling our solution to be used across a wider range of applications.
In this paper, we introduce DeepRM, a 6D pose refinement technique for rigid objects. An illustration of the iterative pipeline is provided in Fig 1. Inspired by DeepIM [2], DeepRM uses an iterative render-and-compare approach to incrementally refine an initial pose estimate. Given an initial coarse pose, a target object can be rendered with the same camera intrinsics as the original observation. The rendered image can then be matched with the observed image to predict the rigid transform that aligns the object in the two images. By leveraging the geometric information implicitly contained within the 3D model of the object, updates to the 6D pose can be inferred without external depth information.
The proposed DeepRM method improves upon DeepIM [2] with several innovations, such as high resolution cropping, disentangled loss, variable renderer brightness, a scalable backbone based on EfficientNet [6], and most notably a recurrent network architecture. DeepRM is the first work that both leverages a recurrent neural network to directly regress 6D pose of rigid objects and provides a scalable framework for this task. Utilizing a recurrent architecture allows additional information to be propagated through each refinement step, significantly improving performance over non-recurrent methods. To summarize, the main contributions of this work are as follows: 1) A simple recurrent neural network architecture, DeepRM, for 6D object pose refinement that requires only a single RGB image as input. 2) A scalable solution that can be adapted based on computational constraints in realworld scenarios. 3) State-of-the-art results on the challenging YCB-Video [7] and Occlusion LINEMOD [8] datasets.

II. RELATED WORK
As the field of 6D pose estimation is vast, we limit the discussion of related works to methods based on RGB data.

A. 6D Object Pose Estimation
The purpose of 6D object pose estimation is to determine an object's fully constrained pose within 3D space. Traditional methods approached this problem with template matching techniques [9], or by matching hand crafted feature points to a 3D CAD model, and solving the Perspective-N-Point (PnP) problem [10]. With the recent advances in Fig. 1. Overview of the DeepRM method. An initial pose estimate is used to render a target object. The observed and rendered images are passed through a convolutional neural network to predict a se(3) transformation that updates the previous pose estimate. This process is repeated multiple times to incrementally refine the estimate. In addition to the updated pose estimate, hidden states from recurrent LSTM modules are propagated to each iteration. machine learning, virtually all new techniques leverage deep learning architectures. Early deep learning based methods built upon the two-stage approach of feature detection followed by PnP. BB8 [11] first used this technique to regress the 8 corners of the bounding cuboid in 2D, and then solve for pose via PnP. Similar methods followed the same approach, but addressed other limiting factors such as efficiency [12] and robustness to occlusion [13].
To further address the problem of occlusion, PVNet [14] introduced a pixel-wise voting network using RANSAC, resulting in an estimator that is capable of detecting keypoints, even when they are not visible due to occlusion or truncation. Although PVNet adopts the approach of using keypoints on the model surface to achieve a lower variation in the predictions, best results were achieved with 8 keypoints similarly to methods using bounding boxes. The sparsity of the keypoints in such approaches limits functionality under high levels of occlusion and truncation. To address this, a different line of research attempts to predict 3D coordinates for every pixel in the target image. By drastically increasing the number of 2D-3D correspondences, performance is maintained even under high occlusion. To handle the additional noise inherent to the dense predictions, PnP+RANSAC is needed to achieve robustness to outliers. Dense correspondence methods include DPOD [15], EPOS [16], and ZebraPose [17].
Recent works such as PoseCNN [7] attempt to directly regress the pose of objects from RGB images. PoseCNN uses a VGG16 [18] backbone to extract high dimensional feature maps. These shared feature maps are then utilized by three downstream tasks: semantic segmentation to localize and distinguish objects, translation prediction, and rotation prediction. The translation and rotation predictions are directly regressed by passing flattened feature maps through fully connected layers. The benefit of direct approaches is that they can be fully trained end-to-end, without surrogate loss functions as in the two-stage approaches.
To achieve the end-to-end differentiability of direct meth-ods, the geometry-guided accuracy of PnP methods, as well as the robustness of dense methods, Wang et al. developed a Geometry-guided Direct Regression Network (GDR-Net) [19]. GDR-Net predicts dense pixel-wise correspondences, but then instead of using a non-differentiable PnP solver, it uses a a convolutional Patch-PnP network to directly regress pose. SO-Pose [20] further extends this approach by leveraging self occlusion information to enforce cross-layer consistencies across the correspondence field, self-occlusion information, and 6D pose, resulting in a direct method that performs comparably to many refinement based techniques.

B. 6D Object Pose Refinement
Although recent methods such as GDR-Net [19] and SO-Pose [20], achieve high levels of accuracy compared to prior works, the ill-posedness of the problem still makes this task very challenging for RGB-only methods. As a result, refinement techniques are necessary to achieve the performance requirements of high-precision applications. Similar to traditional pose estimation techniques, early methods used either hand crafted feature descriptors, or template based matching techniques for refinement. DeepIM [2] then introduced a novel neural network architecture to iteratively refine the pose of an object in a target image by matching it to a rendered image of the object's initially estimated pose. DeepIM is based on the FlowNetS [21] optical flow architecture, and directly regresses the translational and rotational updates necessary to minimize the difference in the observed and rendered images.
Recent state-of-the-art works improve upon DeepIM by addressing a variety of factors, but virtually all of them follow the same basic render-and-compare approach. Cosy-Pose [5] for example replaces the FlowNetS backbone with EfficientNet. [6], removes the optical flow head, and directly regresses rotation in a 6D rotation parameterization [22] as opposed to a quaternion. Trabelsi et al. [3] introduce a combined pose proposal and refinement network. Focusing on the refinement network, [3] extracts and warps feature maps based on the optical flow between observed and rendered images. The warped feature maps then pass through a spatial multi-attention layer to highlight important features, before directly regressing the pose update.
RNNPose [23] is a recent work on RGB pose refinement that uses an architecture inspired by RAFT [24] for optical flow, but extends it significantly for the task of pose estimation. RNNPose is the first work to leverage Gated Recurrent Units (GRUs) during the iterative process of pose refinement. However, pose is optimized by a Levenberg-Marquardt (LM) algorithm on an estimated correspondence field, and therefore RNNPose is not considered a purely direct approach.
Following RNNPose, Lipson et al. [25] also use a RAFT inspired architecture, but solve for pose using a novel "Bidirectional Depth-Augmented PnP (BD-PnP)" solver. This technique extends the standard PnP process by additionally minimizing the reprojection errors of the rendered image, as well as the inverse depth. Like RNNPose, this method predicts a 2D-3D correspondence field and then solves for pose, therefore we do not consider it a direct approach.

III. METHOD
An overview of the proposed DeepRM method is illustrated in Fig. 1. Inspired by DeepIM [2], it follows an iterative render-and-compare approach to refine the pose of an object in a single RGB input image. Given an initial pose estimate of a target object, an image of the target object is rendered. The rendered image is then matched with the real image of the object to predict a se(3) transform to the initial pose estimate that better aligns the rendered object to the observed image. The se(3) transform consists of a translation and rotation vector, where the rotation is represented as a normalized unit quaternion. Se(3) denotes the Special Euclidean group, which refers to the set of proper rigid transformations within the Euclidean group. Such transforms within the Euclidean group preserve the Euclidean distance between transformed points. Because each update reduces the error between the rendered and observed images, this process can be repeated iteratively to incrementally refine the result. This method compensates for lack of external information such as depth by leveraging preexisting geometric and visual properties of target objects, i.e. textured CAD models. By rendering objects in a way that is geometrically consistent with the observed scene, 3-D spatial information can be recovered from the RGB only image data.

A. Network Architecture
The DeepRM neural network architecture is illustrated in Fig. 2. The observed and rendered RGB images are concatenated channel-wise to form a 240×320×6 dimensional tensor. The 6-channel tensor is passed as input to the backbone convolutional neural network to extract feature maps, where the final 8×10×384 feature map from the backbone is flattened and passed through three shared, fully-connected, LSTM layers before the final translation and rotation heads. The multi-scale feature maps from the backbone network are also used in an auxiliary optical flow head during training to stabilize the training process, and enforce the learning of features which are relevant to the task of pose estimation. The structure of the optical flow head is the same as FlowNetS [21], however both the spatial and channel dimensions are modified to match the corresponding layers in the backbone network. We use the B3 version of EfficientNet [6] as the backbone feature extractor to achieve the best balance between performance and model size, but also demonstrate state-of-the-art results with smaller versions of EfficientNet such as B0 and B2. Because we match the dimensions of the optical flow head to those of the backbone, our architecture can be scaled as a whole, using the same hyperparameter used by the EfficientNet backbone, called φ. Additional information on the use of φ as a model scaling hyperparameter is available in [6].

B. Recurrent Fully Connected Layers
While many other works [2], [5], [3] in pose refinement leverage an iterative process to incrementally improve upon an initial coarse estimate, most do not leverage any type of recurrent network features. However, recurrent architectures have been successfully used to improve the iterative processes of other visual processing tasks, such as optical flow prediction [24], saliency detection [26], and instance segmentation [27]. We hypothesize that adding gated recurrent mechanisms such as GRUs and LSTMs to iterative processes should generally maintain or improve their current levels of performance. Considering the case where all gated connections are disabled, we simply have the original network configuration, where each iteration is independent of the previous. We can then enable the recurrent connections to enforce continuity across iterations, improving performance with each iteration. Based on our hypothesis, we apply this theory to the task of 6D pose refinement and present a novel recurrent network architecture suited for this task.

C. High Resolution Cropping
To improve upon the cropping strategy of DeepIM [2], we choose to follow an approach similar to CosyPose [5]. This process consists of cropping to the region of interest around the object, based on the estimated pose, and then resizing this crop to 320×240 before passing it to the network. This cropping strategy has several benefits: a) it reduces background clutter b) it leverages the higher input image resolution. c) it reduces the memory and computational requirements of the network. We note that the only difference between our approach and [5] is that we generate the rendered image at the full 640×480 resolution, and use the same crop as the target image, rather than adjusting the camera parameters and rendering directly to 320×240.

D. Transformation Parameterization
Following DeepIM [2], the network does not directly predict the translational update as a vector in meters, but rather a 2D translation in pixel space, along with a relative change in depth, corresponding to the projected centerpoint of the target object. Given the initial pose of the object, and the pixel space displacements, the 3D translation can be recovered via the thin lens equation. This parameterization enables the network to perform simplified reasoning in 2D, as opposed to modeling the complex relationship between 3D object geometry and the camera intrinsics.

E. Rotation Parameterization
To regress rotation, the network predicts the four quaternion components, which are then normalized to form a unit quaternion. The advantage of normalizing the output is that the network only needs to learn the ratios between components.

1) Disentangled Point Matching Loss (DPML):
To learn 3D pose, we use the point matching loss (PML) function as in [2], but disentangle the translational components as in [28]. PML incorporates both rotational and translational error in a single scalar metric, conveniently eliminating the need to balance the separate elements. Additionally, the disentangled formulation isolates the influence of the xy translation with the relative change in depth. For a ground truth pose p = [R|T ], and an estimated pose p = [ R| T ], the point matching loss is defined as the average 1 norm of a subset of n model points: where x i denotes the i-th model point.
Extending this formula to disentangle the translational components, we first split the ground truth translation and the predicted translation into their respective components, i.e. T = [x, y, z] and T = [ x, y, z]. We then utilize a combination of the ground truth and predicted translations as input to the PML function as shown in Eq. (2).
Our formulation is slightly different than [28] and [5] in that it does not disentangle the rotation component. This was found experimentally to be much more stable during training, and provides better results than the fully disentangled representation. For the auxiliary optical flow head, we use the same multi-scale endpoint error (MS-EPE) loss as [21], and refer to this work for additional details. The disentangled point matching pose loss is then weighted with the mask loss as follows: Total Loss = DPML + 0.1 MS-EPE.

IV. EXPERIMENTS
A. Datasets 1) YCB-Video: The YCB-Video dataset [7] is a a large scale dataset, with a total of 133,827 images over 92 unique scenes. Images contain labeled 6D poses of 21 target objects. The majority of images contain 4-5 objects in the scene, resulting in high levels of occlusion, as well as a variety of challenging lighting conditions. The 21 objects are a diverse selection of common household items, which include various levels of symmetry (i.e. non-symmetric, discretely symmetric, and continuously symmetric objects). For consistent comparison, we use the same exact real data, synthetic data, and data split as DeepIM [2].
2) Occlusion-LINEMOD: The Occlusion LINEMOD dataset [8] is an extension upon the original LINEMOD dataset [29]. LINEMOD consists of 13 common household objects, split into 13 cluttered scenes. Roughly 1000 images are provided for each object. Many target objects   are present in each image, however only a single object is labeled per image. The target object in each image is also generally very visible. To create a more challenging dataset, Occlusion LINEMOD was introduced. Occlusion LINEMOD provides ground truth labels for all objects in one of the 13 scenes. This results in high levels of partial occlusion, significantly increasing the difficulty of the dataset. Following the convention of other works such as [19], [20], we train on LINEMOD, and evaluate on Occlusion LINEMOD. Although, due to the limited amount of real data provided in LINEMOD, we additionally augment the training data with physically-based rendering (pbr) images that are publicly available from the 2020 BOP Challenge [4].

B. Evaluation Metrics
To evaluate the performance against other state-of-the-art methods, we follow [2], [14], [5], [3], [20], [19] and use the ADD metric [29]. More specifically, we use two specific variations upon it, depending on the dataset, ADD(-S) 10% for Occlusion LINEMOD and area under the curve (AUC) ADD(-S) for YCB-Video. For the sake of brevity, we refer readers to prior works such as [29] and [7] for a more detailed description of these metrics.

C. Implementation Details
DeepRM is implemented in PyTorch, and uses the same OpenGL based renderer as [2]. Unlike other works [2], [5] that use a consistent light source, we manually tuned the renderer brightness so that it properly exposes each object that is matched in the target scene. This step can be automated by using an average metering algorithm, similar to what is used in digital photography for auto-exposure. Since the rendered object is always drawn on a black background, the render brightness for each object can be pre-determined offline, and used throughout all training and testing. This simple modification improved the baseline results by 0.4%. For both YCB-Video and Occlusion LINEMOD datasets, we use the ADAM optimizer [32], with a base learning rate of 1e-4. Although due to the differences in each dataset, we use different batch sizes, training durations, and learning rate schedules for each dataset.

D. Comparison to State-of-the-Art
Results on YCB-Video dataset. Table I presents the results of DeepRM compared to the current state-of-the-art on the YCB-Video dataset for the AUC ADD(-S) metric. Initial predictions are obtained from PoseCNN [7], where DeepRM outperforms the leading state-of-the-art method, ZebraPose [17], by 1.5%.
Results on LM-O dataset. Table II presents the results of DeepRM compared to the current state-of-the-art on the Occlusion LINEMOD dataset for the ADD(-S) 10% metric. Initial predictions are obtained from PVNet [14], where DeepRM outperforms all existing methods except for ZebraPose [17]. Although ZebraPose provides superior performance, it requires a different model for each object and runs at a significantly reduced frame rate compared to DeepRM.

E. Ablation Study on YCB-Video
Table III displays network performance in terms of accuracy and frames per second (FPS) as a function of various backbone architectures, fully-connected layer types, fullyconnected layer dimensions, and number of trainable parameters for the YCB-Video dataset. Due to resource and time constraints, results are limited to 8 refinement iterations. All tests were performed on a desktop workstation with a single NVIDIA RTX 3060 GPU and an Intel i7-11700K CPU.
Highest accuracy is observed for the EfficientNet-B3 backbone using 8 refinement iterations. This configuration achieves 86.6% at 9.5 FPS on the AUC ADD(-S) metric for YCB-Video. However, the number of refinement iterations can be decreased to 4 to achieve 18 FPS while still maintaining superior accuracy to all state-of-the-art methods.
Table III also demonstrates that the fully connected layers in our architecture can be scaled along with the EfficientNet backbone, using the same scaling parameter, φ. Using this technique, the model can be adapted to meet real-world execution time or resource constraints. This flexibility along with the accuracy and efficiency of our method provide a solution that is well-suited to practical robotics applications.
Finally, to support our claim that recurrent network features improve the performance of this task, Table III displays the impact of recurrent fully-connected layers compared to standard fully-connected ones. We find that LSTMs provide a significant increase of 1.6%, whereas GRUs provide a more moderate improvement of 0.5% over the standard fullyconnected baseline.

F. Ablation Study on Refinement Iterations for YCB-Video
The process of iterative refinement is heavily dependent on the number of iterations performed. As such, we investigate the impact of training and testing on a variety of refinement iterations. All tests were performed with the EfficientNet-B3 backbone on the YCB-Video dataset. AUC ADD(-S) results are reported in Table IV. Best performance is achieved when training with 6 iterations, and testing with 12 iterations, although we find 8 testing iterations to provide the best balance of accuracy and execution time.

G. Ablation Study on Optical Flow
In addition to the recurrent structure, the auxiliary optical flow head is one of the main features that distinguishes our work from others such as CosyPose [5]. We find that the auxiliary optical flow head provides an accuracy improvement of 1.8% on the EfficientNet-B3 backbone configuration of our network, clearly demonstrating its benefit. Furthermore, this improvement only costs a 5% increase in parameters during training. At inference, this portion of the network is removed. Results are shown for 6 training iterations, and 8 testing iterations on the YCB-Video dataset.

V. CONCLUSIONS
In this work, we introduce DeepRM, a novel method for precise 6D pose estimation of rigid objects from RGB only data. DeepRM improves upon existing render-and-compare approaches by leveraging several unique elements, such as an optical flow enforced learning process, an efficient and scalable backbone, and an LSTM enhanced iterative refinement mechanism. DeepRM outperforms all existing state-of-the-art methods on the challenging YCB-Video and Occlusion LINEMOD datsets.