{'fraction_non_alphanumeric': 0.04822315916277669, 'fraction_numerical': 0.01599909652160819, 'mean_word_length': 4.495448903599503, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 2, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Sparse Steerable Convolutions: An Efficient Learning of SE(3)-Equivariant Features for Estimation and Tracking of Object Poses in 3D Space

As a basic component of SE(3)-equivariant deep feature learning, steerable convolution has recently demonstrated its advantages for 3D semantic analysis. The advantages are, however, brought by expensive computations on dense, volumetric data, which prevent its practical use for efficient processing of 3D data that are inherently sparse. In this paper, we propose a novel design of Sparse Steerable Convolution (SS-Conv) to address the shortcoming; SS-Conv greatly accelerates steerable convolution with sparse tensors, while strictly preserving the property of SE(3)-equivariance. Based on SS-Conv, we propose a general pipeline for precise estimation of object poses, wherein a key design is a Feature-Steering module that takes the full advantage of SE(3)-equivariance and is able to conduct an efficient pose refinement. To verify our designs, we conduct thorough experiments on three tasks of 3D object semantic analysis, including instance-level 6D pose estimation, category-level 6D pose and size estimation, and category-level 6D pose tracking. Our proposed pipeline based on SS-Conv outperforms existing methods on almost all the metrics evaluated by the three tasks. Ablation studies also show the superiority of our SS-Conv over alternative convolutions in terms of both accuracy and efficiency. Our code is released publicly at https://github.com/Gorilla-Lab-SCUT/SS-Conv.


SE(3)-equivariant deep networks
have shown the promise recently in some tasks of 3D semantic analysis, among which 3D Steerable CNN [25] is a representative one. 3D Steerable CNNs employ steerable convolutions (termed as ST-Conv) to learn pose-equivariant features in a layer-wise manner, thus preserving the pose information of the 3D input. Intuitively speaking, for a layer of ST-Conv, any SE(3) transformation (r, t) applied to its 3D input would induce a pose-synchronized transformation to its output features, where r ∈ SO(3) stands for a rotation and t ∈ R 3 for a translation. Fig. 1 (a) gives an illustration where given an SE(3) transformation of the input, the locations at which feature vectors are defined are rigidly transformed with respect to (r, t), and the feature vectors themselves are also rotated by ρ(r) (ρ(r) is a representation of rotation r). This property of SE(3)-equivariance enables the steerability of feature space. For example, without transforming the input, SE(3) transformation can be directly realized by steering in the feature space. To produce steerable features, ST-Conv confines its feature domain to regular grids of 3D volumetric data; it can thus be conveniently supported by 3D convolution routines. This compatibility with 3D convolutions eases the implementation of ST-Conv, but at the sacrifice of efficiently processing 3D data (e.g., point clouds) that are typically irregular and sparse; consequently, ST-Conv is still less widely used in broader areas of 3D semantic analysis.
In this paper, we propose a novel design of Sparse Steerable Convolution (SS-Conv) to address the aforementioned shortcoming faced by ST-Conv. SS-Conv can greatly accelerate steerable convolutions with sparse tensors, while strictly preserving the SE(3)-equivariance in feature learning; Fig. 1 (b) gives the illustration. To implement SS-Conv, we construct convolutional kernels as linear combinations of basis ones based on spherical harmonics, which satisfy the rotation-steerable constraint of SE(3)-equivariant convolutions [25], and implement the convolution as matrix-matrix multiply-add operations on GPUs only at active sites, which are recorded along with their features as sparse tensors.
Although SE(3)-equivariant feature learning is widely used in 3D object recognition, its potentials for other tasks of 3D semantic analysis have not been well explored yet. In this work, we make the attempt to apply our proposed SS-Conv to object pose estimation in 3D space. To this end, we propose a general pipeline based on SS-Conv, which stacks layers of SS-Conv as the backbone, and decodes object poses directly from the learned SE(3)-equivariant features. A novel Feature-Steering module is also designed into the pipeline to support iterative pose refinement, by taking advantage of the steerability of the learned features. We conduct thorough experiments on three tasks of pose-related, 3D object semantic analysis, including instance-level 6D pose estimation, category-level 6D pose and size estimation, and category-level 6D pose tracking. Our proposed pipeline based on SS-Conv outperforms existing methods on almost all the metrics evaluated by the three tasks; the gaps are clearer in the regimes of high-precision pose estimation. Ablation studies also show the superiority of our SS-Conv over alternative convolutions in terms of both accuracy and efficiency.  [25] is proposed to work on regular volumetric data, so that basis kernels with respect to regular grids could be pre-computed; however, it still encounters challenging computational demands due to the ignorance of data sparsity. Compared to the above methods, our proposed sparse steerable convolution aims at efficient SE(3)-equivariant representation learning for volumetric data, which is realized with sparse tensors to accelerate the computation.

Estimation and Tracking of Object Poses in 3D Space
In the context of pose estimation, instancelevel 6D pose estimation is a classical and well-developed task, for which a body of works are proposed. These works can be broadly categorized into three types: i) template matching [12] by constructing templates to search for the best matched poses; ii) 2D-3D correspondence methods [1,14,16,19,17], which establish 2D-3D correspondence via 2D keypoint detection [19,17] or dense 3D coordinate predictions [1,14,16], followed by a PnP algorithm to obtain the target pose; iii) direct pose regression [26,13,23] via deep networks. Recently, a more challenging task of category-level 6D pose and size estimation is formally introduced in [24], aiming at estimating poses of 3D unknown objects with respect to a categorical normalized object coordinate space (NOCS). The early works [24,21] focus on regression of NOCS maps, and the poses can be obtained by aligning NOCS maps with the observed depth maps. Later, methods of direct pose regression are proposed thanks to the special designs of fusion of pose-dependent and pose-independent features [2], decoupled rotation mechanism [4], or dual pose decoders [15]. Motivated by [24], Wang et. al propose the task of category-level 6D pose tracking, aiming for the small change of object poses between two adjacent frames in a sequence; they also present 6-PACK, a pose tracker estimating the change of poses by matching keypoints of two frames.
In general, the transformation π n (g) is a group representation of SE(3), which satisfies π n (g 1 g 2 ) = π n (g 1 )π n (g 2 ). If g is decomposed into a 3D rotation r ∈ SO(3) and a 3D translation t ∈ R 3 , written as g = tr, π n (g) can be defined as follows: where ρ n (r) : R Kn → R Kn is an SO(3) representation. The illustration is given in Fig. 1.

Rotation-Steerable Constraint
To guarantee SE(3)-equivariance in Eq. (2), it can be derived that the kernel κ of 3D convolution must be rotation-steerable [25], which satisfies the following constraint: Irreducible Feature ρ n (r) is an SO(3) representation, which can be decomposed into F n irreducible representations as follows: where Q is a K n × K n change-of-basis matrix, D li (r) is the (2l i + 1) × (2l i + 1) irreducible Wigner-D matrix [8] of order l i (l i = 0, 1, 2, ...), and represents block-diagonal construction of {D li (r)}, so that K n = Fn i=0 2l i + 1. Based on Eq. (3), f n (x) can be constructed by stacking is associated with a D li (r). When l i = 0, D 0 (r) = 1, so that f i n (x) is a scalar invariant to any rotation; when l i > 0, f i n (x) is a vector which can be rotated by D li (r).

Sparse Steerable Convolution
3D STeerable Convolution (ST-Conv) enjoys the property of SE(3)-equivariance; however, as discussed in Sec. 1, it suffers from heavy computations as conventional 3D convolution does. Motivated by recent success of SParse Convolution (SP-Conv) [9], we propose a novel design of Sparse Steerable Convolution (SS-Conv) with sparse tensors, which takes the natural sparsity of 3D data into account, while strictly keeping the steerability of features.
Specifically, assuming κ is a discretized, s × s × s, cubic kernel with grid sites S = {− s−1 2 , ..., −1, 0, 1, ..., s−1 2 } 3 (s is an odd), our proposed SS-Conv can be formulated as follows: where σ n (x) represents the state of site x in the feature space R Kn . 1 σ n (x) = 0 denotes an inactive state at x, where f n (x) is in its ground state; when f n (x) is beyond the ground state, this site would be activated as σ n (x) = 1. In SS-Conv, we set the ground state as a zero vector.
Compared with ST-Conv, our sparse version is accelerated in two ways: i) convolutions are conducted at activated output sites, not on the whole 3D volume, where the number of active sites only takes a small proportion; ii) in the receptive field of each activated output site, only active input features are convolved. For these purposes, we represent the input and output features as sparse tensors (H n , F n ) and (H n+1 , F n+1 ), respectively. H n and H n+1 are hash tables recording the coordinates of active sites only, while F n and F n+1 are feature matrices. For a sparse tensor, its hash table and feature matrix correspond to each other row-by-row; that is, if r n+1,x is the row number of In this respect, the goal of SS-Conv is to convolve 3) for the output feature matrix F n+1 . We will introduce the detailed implementations shortly.

Rotation-Steerable Kernel Construction
The key to satisfy the rotation-steerable constraint (4) is to control the angular directions of feature vectors, and a recent research shows that spherical harmonics Y J = {Y J j } J j=−J give the unique and complete solution [25]. Linear combination of the basis kernels based on spherical harmonics produces the rotation-steerable convolutional kernel κ.
For simplicity, we firstly consider both input and output features as individual irreducible ones of orders l and k, respectively; the kernel κ kl : R 3 → R (2k+1)×(2l+1) is parameterized as a linear combination of basis kernels κ kl,Jm : where Q kl j is a (2k + 1) × (2l + 1) change-of-basis matrix, also known as Clebsch-Gordan coefficients, and ϕ m is a continuous Gaussian radial function: . In the basis kernel κ kl,Jm (8), Y J controls the angular direction, while ϕ m controls the radial one; then {κ kl,Jm } are linearly combined by learnable coefficients {w kl,Jm } as in Eq. (7) to further adjust the radial direction, which is the only degree of freedom in the process of optimization. Accordingly, the angular direction is totally controlled by Y J , such that the rotation-steerable constraint is strictly followed. In addition, the total number of learnable parameters in Eq. (7) is M [2min(k, l) + 1] (M is the number of selected {m}), which is, in practice, marginally less than that of conventional 3D convolution, which has (2k + 1)(2l + 1) parameters.
Finally, assuming that the input and output features are stacked irreducible features, whose orders are {l 1 , · · · , l Fn } and {k 1 , · · · , k Fn+1 } respectively, the rotation-steerable kernel of SS-Conv can be constructed as follows: with the size of

Site State Definition
The key to enable the efficiency of SS-Conv lies in the definition of site state. In general, for an output grid site x, if any of input sites in its receptive field are active, this site will be activated, and convolution at this site will be conducted; otherwise, this site will keep inactive, meaning that its feature will be directly set as a zero vector (representing the ground state) without convolutional operation. We formulate the above definition of state at site x as follows: The output hash table H n+1 is then generated as {x : σ n+1 (x) = 1}.
The number of active sites defined in (10) will increase layer-by-layer, enabling long-range message transfer. However, if dozens or even hundreds of convolutions are stacked, the rapid growth rate of active sites would result in heavy computational burden and the so-called "submanifold dilation problem" [9]. To alleviate this problem, we follow [9] and consider another choice of state definition in our SS-Conv, which keeps the output state consistent with the input one at a same grid site, i.e., σ n+1 (x) = σ n (x), such that H n+1 = H n . This kind of SS-Conv without dilation makes it possible to construct a deep but efficient network for sparse volumetric data, and we term it as "Submanifold SS-Conv".
In practice, we mix general SS-Convs and Submanifold SS-Convs in an alternating manner to achieve high accuracy and efficiency.

Sparse Convolutional Operation
After obtaining H n+1 , the next target is to compute the values of F n+1 . Specifically, we firstly initialize F n+1 to zeros; then the feature vectors in F n+1 are updated via the following algorithm: for y in H n do 4: if s = x − y ∈ S: 5: Append (r n+1,x , r n,y ) to R s . / / r n+1,x is the row number of x in H n+1 . 6: end for / / r n,y is the row number of y in H n . 7: end for 8: for R s in R do / / Update F n+1 . 9: for (r n+1,x , r n,y ) in R s do 10: This process can be divided into two substeps. The first one is to construct a rule book R = {R s : s ∈ S}, where an active output site x is paired with an active input y in each R s , if x − y = s. The

Feature-Steering Module
Tensor-to-Point second one is to update F n+1 according to the paired relationships recorded in R; for example, if the paired relationship of output x and input y is recorded in R s , the current f n+1 (x) will be updated by adding the multiplication of f n (y) and κ(s). In this process, the construction of R is very critical, which helps to implement the second substep by matrix-matrix multiply-add operations on GPUs efficiently.

Normalization and Activation
As conventional CNNs do, SS-Convs are also followed by normalization and activation, i.e., Activation(Norm([κ f n ](x))). Those operations of normalization and activation are required to be specially designed, not to break the SE(3)-equivariance of features. Since each SE(3)-equivariant feature is formed by stacking irreducible ones, without loss of generality, we take as an example an irreducible feature f (x) with order l, so that the normalization can be formulated as follows: where E[·] and Var[·] are population mean and variance, respectively. is a very small constant.
For the activation of f (x), if l = 0, ReLU can be chosen to increase non-linearity; if l > 0, we follow [25] and multiply to f (x) a scalar, which is learned by a SS-Conv and applied to the Sigmoid function: The above normalization and activation operations are both SE(3)-equivariant, since a feature vector multiplying any scalar keeps its equivariance; when applying them to features formed by numerous irreducible ones, we treat each irreducible member individually to ensure the equivariance.

Instance-level 6D Object Pose Estimation
Given an RGB-D image of a cluttered scene, instance-level 6D pose estimation is to estimate the 6D poses of known 3D objects with respect to the camera coordinate system. As introduced in Sec. 3.1, a 6D pose g ∈ SE(3) can be decomposed into a 3D rotation r ∈ SO(3) and a 3D translation t ∈ R 3 , which makes sparse steerable convolutional network well suited for this task, due to: i) SS-Convs extract strong SE(3)-equivariant features to decode a precise 6D pose; ii) the steerability of feature maps helps to enable a second stage of pose refinement. Therefore, we propose an efficient general pipeline based on SS-Convs for 6D pose estimation, as depicted in Fig. 2.
Specifically, we firstly segment out the objects of interest via an off-the-shelf model of instance segmentation, assigning each object with an RGB segment and a cropped point cloud; then each 3D object is voxelized and represented by a sparse tensor (H 0 , F 0 ), where each feature in F 0 is a 4−dimensional vector, containing RGB values and a constant "1". For the input tensor, we set the site active if the quantified grid centered at this site encloses any points, and average point features of those enclosed by a same grid. (H 0 , F 0 ) is then fed into our pipeline in Fig. 2, where the pose estimation could be achieved in the following two stages.
In the first stage, we construct an efficient SS-Conv-based backbone, which extracts hierarchical SE(3)-equivariant feature maps, represented in the form of sparse tensors {(H n , F n )}. Those feature tensors are used for interpolation of multi-level point-wise features by using a Tensor-to-Point module, proposed in [10], transforming features of discretized grid sites to those of real-world point coordinates. Each point feature is fed into two separate MLPs, regressing a point offset and a rotation, respectively; the addition of the point coordinate and its offset generates a translation. The initially predicted pose (r 1 , t 1 ) of this stage is obtained by averaging point-wise predictions.
In the second stage, we refine the pose (r 1 , t 1 ) by learning a residual pose (r 2 , t 2 ), wherein a Feature-Steering module is designed, generating transformed features {(H n , F n )} by efficiently steering hierarchical backbone features {(H n , F n )} individually with (r 1 , t 1 ). Again we interpolate point-wise features from {(H n , F n )}, and average point-wise predictions to obtain (r 2 , t 2 ). Finally, the predicted 6D pose is updated as (r 1 r 2 , t 1 +r 1 t 2 ). In addition, owing to the novel Feature-Steering modules, this stage can be iteratively repeated, generating finer and finer poses.

The Feature-Steering Module
Feature-Steering module in the pipeline is to transform (H n , F n ) of the backbone to (H n , F n ), where a rigid transformation of H n with (r, t) and a rotation of F n with ρ(r) are included. Specifically, for F n , we compute ρ(r) as defined in (5) and rotate F n by matrix multiplication; for H n , we convert the sites in it to the real-world point coordinates, which are then applied to a rigid transformation of (r, t) and re-voxelized as grid sites. The same new sites are merged to a unique one, while their features are averaged. We also use two another SS-Convs, each followed by steerable normalization and activation, to enrich the new features and generate the final steered (H n , F n ).

Category-level 6D Object Pose and Size Estimation
Category-level 6D pose and size estimation is formally introduced in [24]. This is a more challenging task, which aims to estimate categorical 6D poses of unknown objects, and also the 3D object sizes. To tackle this problem, we use a similar network as that in Fig. 2, and make some adaptive modifications: i) for each stage in Fig. 2, we add another two separate MLPs for point-wise predictions of 3D sizes and point coordinates in the canonical space, respectively; ii) in each Feature-Steering module, the real-world coordinates of all 3D objects are also scaled by their predicted 3D sizes to be enclosed within a unit cube, for estimating more precise poses.

Category-level 6D Object Pose Tracking
Motivated by the above task of categorical pose estimation, category-level 6D pose tracking is also proposed to estimate the small change of 6D poses in two adjacent RGB-D frames of an image sequence [22]. Due to the available pose of the previous frame, the target object can be roughly located in the current frame, avoiding the procedures of object detection or instance segmentation in images. However, without a precise mask, the estimation of small pose change from noisy 3D data is a big challenge for deep networks. Our sparse steerable convolutional network also surprisingly performs well in such noisy data, even though we only conduct one-stage pose estimation that achieves real-time tracking. For more details, one may refer to the supplementary material.

Experiments
Datasets We conduct experiments on the benchmark LineMOD dataset [11] for instance-level 6D pose estimation, which consists of 13 different objects. For both category-level 6D pose estimation and tracking, we experiment on REAL275 dataset [24], which is a more challenging real-world  Figure 3: Training of Plain12 based on different convolutions on the LineMOD dataset [11].
dataset with 4, 300 training images and 2, 750 testing ones, containing object instances of 6 categories. Following [24,22], we augment the training data of REAL275 with synthetic RGB-D images.
Evaluation Metrics For instance-level task, we follow [23] and evaluate the results of LineMOD dataset on ADD(S) metric. For the category-level tasks, we report the mean Average Precision (mAP) of intersection over union (IoU) and n • m cm, following [24]; mean rotation error (r err ) in degrees and mean translation error (t err ) in centimeters are also reported for pose tracking. Additionally, we compare the numbers of parameters (#Param) and the running speeds (FPS) for different models.
Testing is conducted on a server with a GeForce RTX 2080ti GPU for a batch size of 32, and FPS is computed by averaging the time cost of forward propagation on the whole dataset.

Comparisons with Different 3D Convolutions
We firstly conduct experiments to compare our proposed SS-Conv with other kinds of 3D convolutions, including conventional 3D convolution (Dense-Conv), sparse convolution (SP-Conv) [9], and steerable convolution (ST-Conv) [25], on the LineMOD dataset for instance-level 6D pose estimation. Among those convolutions, SP-Conv improves the speed of Dense-Conv by considering data sparsity and turns out to be efficient in some tasks of 3D semantic analysis(e.g., 3D object detection), while ST-Conv constructs rotation-steerable kernels and then realizes the convolution based on Dense-Conv.
To meet various computational demands of different convolutions, those experiments are conducted on a light plain architecture, termed as Plain12, in the same experimental settings, for a fair comparison. The architecture consists of 12 convolutional layers, of which the kernel sizes are all set as 3 × 3 × 3; for SS-Convs and ST-Convs, we set the superparameters of the radial function ϕ m in Eq. (8) as {m} = {0, 1} and = 0.6. We use ADAM to train the networks for a total of 30, 000 iterations, with an initial learning rate of 0.01, which is halved every 1, 500 iterations. We voxelize the input segmented objects into 64 × 64 × 64 dense/sparse grids, and set the training batch size as 16.
Quantitative results of different convolutions are listed in Table 1, which confirms the advantages of our SS-Conv in both accuracy and efficiency. In terms of accuracy, SS-Conv achieves comparable results on ADD(S) metric as ST-Conv does, which significantly outperforms those of Dense-Conv and SP-Conv, indicating the importance of SE(3)-equivariant feature learning on pose estimation;with preservation of relative poses of features layer-by-layer, the property of SE(3)-equivariance makes feature learning capture more information of object poses. We also visualize the behaviors of the four convolutions in the process of training in Fig. 3, where the learning based on SS-Conv/ST-Conv converges better and faster than that of Dense-Conv/SP-Conv.     Table 1. More results of FPS with improved sizes of data batches are given in Fig. 4, where ST-Conv can be only run at the extreme batch size of 48 on the GPU with 12G memory, while SS-Conv costs much less memory and can thus support a batch size as large as 512; running with larger batch sizes further improves the efficiency of our proposed SS-Conv (FPS goes to 725 when running with the batch size of 512 on Plain12). We also compare the efficiency of ST-Conv and SS-Conv on two other deeper networks (dubbed Plain24 and ResNet50, respectively); as shown in Fig. 4, SS-Conv consistently improves FPS over ST-Conv on these two architectures, with less GPU memory consumption.

Comparisons with Existing Methods
Instance-level 6D Object Pose Estimation For the instance-level task, we compare the results of our SS-Conv-based pipeline with existing methods on LineMOD dataset [11]. Quantitative results  are shown in Table 2, where our two-stage pipeline outperforms all the existing methods and achieves a new state-of-the-art result of 99.2% on mean ADD(S) metric. We can also observe that the second stage of pose refinement with Feature-Steering modules in our pipeline indeed improves the predictions in the first stage, benefitting from the steerability of the feature spaces in SS-Convs.

Category-level 6D Object Pose and Size Estimation
We conduct experiments on REAL275 [24] for the more challenging category-level task. Quantitative results in Table 3 confirm the advantage of our pipeline in the high-precision regime, especially on the precise metric of 5 • 5cm, where we improve the state-of-the-art result in [15] from 35.9% to 43.4%. The second stage of pose refinement also plays an important role in this task, achieving remarkable improvements over the first stage.
Category-level 6D Object Pose Tracking We compare the results of our one-stage tracking pipeline with the baseline of 6-PACK [22] on REAL275 [24]. In 6-PACK, the relative pose between two frames is computed based on predicted keypoint pairs inefficiently, while our pipeline regresses the pose in a direct way. The results in Table 4 show that our pipeline outperforms 6-PACK on all the evaluation metrics, demonstrating the ability of SS-Conv-based network for fine-grained pose estimation in noisy input data.
More implementation details and qualitative results are shown in the supplementary material.

Broader Impact
The studied problems of object pose estimation and tracking in 3D space are very important to many real-world applications, including augmented reality, robotic grasping, and autonomous driving. By precisely predicting object poses in the 3D space, virtual contents could be seamlessly embedded in real environments, creating fascinating personal experience; on the contrary, less precise predictions may cause property loss and even life threat, especially in autonomous driving. The contributed solution based on SS-Conv would improve the overall level of safety.