{'fraction_non_alphanumeric': 0.03562408613585006, 'fraction_numerical': 0.01114360405866454, 'mean_word_length': 4.956584850884139, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 0, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Occlusion-Invariant Rotation-Equivariant Semi-Supervised Depth Based Cross-View Gait Pose Estimation

Accurate estimation of three-dimensional human skeletons from depth images can provide important metrics for healthcare applications, especially for biomechanical gait analysis. However, there exist inherent problems associated with depth images captured from a single view. The collected data is greatly affected by occlusions where only partial surface data can be recorded. Furthermore, depth images of human body exhibit heterogeneous characteristics with viewpoint changes, and the estimated poses under local coordinate systems are expected to go through equivariant rotations. Most existing pose estimation models are sensitive to both issues. To address this, we propose a novel approach for cross-view generalization with an occlusion-invariant semi-supervised learning framework built upon a novel rotation-equivariant backbone. Our model was trained with real-world data from a single view and unlabelled synthetic data from multiple views. It can generalize well on the real-world data from all the other unseen views. Our approach has shown superior performance on gait analysis on our ICL-Gait dataset compared to other state-of-the-arts and it can produce more convincing keypoints on ITOP dataset, than its provided"ground truth".

The area of vision-based 2D pose estimation has shown remarkable achievements recently [5], [6]. However, precise 3D skeleton estimation remains a challenging task caused by the ambiguity when lifting monocular-derived 2D skeleton estimation to 3D. To reduce such ambiguity, multi-view constraints [7] and temporal constraints [8] have been utilized. Another line of research targets at 3D sensors such as depth cameras utilizing available 3D information. Several deep learning based methods have been proposed for depth based 3D pose estimation as discussed in Section II-B.
Albeit the 3D information provided by depth images, it is still challenging to perform 3D pose estimation in cross-view contexts [9]. As the observed images demonstrating distinct characteristics in terms of different viewpoints [9], [10], the single viewpoint observation leads to two fundamental problems, i.e., occlusion and rotation variations, as shown in Fig. 1. On one hand, the 3D data encoded in the depth maps will be significantly affected by occlusions caused by self-occlusion and partial-observation. Therefore, from a single viewpoint, estimating anatomical joint positions (underneath the surface) from occluded 3D surface data is very difficult. If the observed 3D data is rotated to a canonical coordinate system, as shown in the first row of Fig. 1, the final estimated poses are expected to stay the same even with different occlusions, called occlusion "invariant".
On the other hand, the body orientations of the depth images captured from a single camera are transformed along with the rotation of the viewpoints (cameras), as shown in the second row of Fig. 1. Under such changes, the predicted skeleton is expected to undergo the same rotation operation, which can be called rotation "equivariant". Hence, the final 3D pose estimation model is expected to be both occlusion-invariant and rotation-equivariant, as depicted in the bottom of Fig. 1. Recent research has proposed to utilize synthetic data generated by sophisticated virtual human model to enhance the generalization capacity [11]. However, there are two issues associated with utilizing synthetic data for healthcare applications with high-precision requirements, such as biomechanical analysis of human gait [12]. On one hand, the defined joint positions in these synthetic human models are not the same as those defined in clinical gait models [13]. On the other hand, the observed human surface data tends to exhibit slight differences compared to the naked synthetic human model due to cloth/shoe distortions. Consequently, direct supervised training with labelled synthetic data would not overcome the heterogeneity between real and synthetic data, and thus not be able to derive convincing clinically related biomechanical indexes for real-world data.
Therefore in this paper, to achieve robust depth based 3D lower-limb pose estimation, we propose a novel multi-view occlusion-invariant semi-supervised framework built upon a rotation-equivariant backbone. To facilitate rotation equivariance, the Cylindrical representation was utilized with 2D heatmap estimation. To achieve the invariance against multi-view occlusions, unlabelled synthetic data from random viewpoints was generated to achieve consistency across occlusions. Training labelled real data from a single view, together with unlabelled synthetic data from multiple views, form our semi-supervised cross-view generalization framework. In this work, we targeted at one healthcare application requiring precise pose estimation, i.e. biomechanical gait analysis. To achieve this, a multi-view dataset was collected, with lower-limb depth images and joint positions regressed from clinical gait models recorded simultaneously.
Our contributions are three-fold: Rotation Equivariant Pose Estimation Our proposed network architecture enables rotation equivariant by performing 3D keypoint estimation under Cylindrical coordinates. Under such coordinate system, rotation group equivariance can be directly realized by conventional planar convolutions with the property of translation group equivariance.
Cross-View Pose Estimation Generalization Our proposed framework enables cross-view generalization of pose estimation. With the use of unlabelled multi-view synthetic data and the rotational equivariant backbone, a semi-supervised learning framework can enable both occlusion-invariance and rotation-equivariance.
Multi-View Depth Pose Dataset A multi-view dataset with depth data is collected from multiple viewpoints for cross-view human pose estimation benchmarking. Precise lower-limb joints derived from Vicon motion capture system using a clinical gait model were recorded as the ground truth for accurate validation, which is available upon request https://xiaogu.site/ICL gait.

II. RELATED WORKS
A. Multi-View Pose Estimation a) Datasets: Existing multi-view human motion datasets include Human3.6M [14], Total Capture [15], MPII-INF-3DH [16], ITOP [9], etc. Most datasets target at 2D RGB images, whereas ITOP captures depth images. Their ground truth skeleton data comes from either manual annotations or tracked marker trajectories, which are not the actual anatomical positions. In fact, a commonly used metric for ITOP dataset is mean average precision (detected ratio of all joints below 10cm) [9], [17]. However, this threshold is too high to satisfy the precision required for biomechanical analysis of gait, where the subtle changes between different walking conditions are difficult to recognize. b) Cross-View Consistency: Based on the above existing multi-view RGB datasets, recent work either made use of epipolar geometry to predict 3D poses from multiple images [18], or proposed semi-/weakly supervised learning strategies to enhance the estimation in 2D/3D images while reducing the dependency on ground truth annotations [19], [7]. The key idea of these work is to ensure the estimation consistency across multiple views. Our multi-view framework is inspired by this line of research, as an extension to 3D data. Differently, to avoid the light projection interference across different depth sensors in the real world, we utilized synthetic data to achieve occlusion-invariance.
B. Depth Based Pose Estimation a) Computational models: For depth based pose estimation, deep learning based approaches have demonstrated superior performance compared to conventional ones. Based on the input representations (depth maps, volume, point sets, etc.), these approaches apply Conv2D [20], [21], Conv3D [22], [17], MLPs [23], [24], Transformers [25], etc. to automatically extract spatial features. Among these models, it has been validated that direct 3D convolutions [17] can better preserve the spatial information, yet is computationally inefficient. b) Regression or Detection: Depth based pose estimation can be considered as a regression or detection task in terms of the output categories. Directly performing 3D coordinate regression [22], [23] was utilized by most relevant research.
To better capture the spatial information, Xiong et al. [20] proposed an anchor-to-joint framework by firstly generating 2D joint heatmaps for anchor points and subsequently estimating the joint depth offsets from the surrounding depth values. However, with regards to the accuracy, it is still inferior to the state-of-the-art V2V estimating per-voxel probabilities, via an hourglass-like model composed of a series of 3D convolutions [17]. c) Training with Synthetic: It has been a popular method to incorporate simulators and 3D hand/human models, with the purpose of generating synthetic data for pose estimation training [26], [11]. Different from previous work that used synthetic data for supervised training, we do not rely on the keypoints extracted from synthetic models. As discussed in Section I, these keypoint positions may be biased from their counterparts defined in clinical gait models [13].

C. Cross-View Generalization
In fact, cross-view generalization problem arises in a variety of applications, such as face recognition [27], gait recognition [28], pedestrian re-identification [29], with series of solutions proposed. The main idea of these solutions is to abstract view-invariant features [27], [29] or to realize robust adaptation to the novel view with transfer learning or cross-view transformation [28], [30]. They all focused on cross-view invariant representation learning, limited in our pose estimation task, since the pose estimation model should be equivariant, not invariant, to the rotational transformations.
Especially, for pose estimation task, Haque et al. [9] proposed a local-patch view invariant model to iteratively refine the results. Although it highlighted the cross-view generalization issue in depth based pose estimation, its model did not perform well when performing cross-view validation.

D. Equivariance Network
Extracting equivariant feature representations under different data transformations is an expected behaviour for several computer vision tasks such as pose estimation [31] and semantic segmentation [32]. With such equivariance, the input transformations and feature-space transformations can be explicitly associated.
For deep neural networks, the translation-equivariance is "hard-baked" into the planar translation convolutions. In pursuit of the equivariance of other transformations like rotations, most widely used method is to train with aggressive data augmentation in a data-driven manner. This might force the network into learning equivariant output, but cannot ensure equivariant representations of the intermediate layers. Existing work further explored the convolution operations, either setting harmonics constraints to filter structures [31] or performing group convolutions on an equivariant filter orbit [33], [34].
Esteves et al. [35] proposed a simple yet effective Polar transformer network for rotation equivariant 2D/3D recognition problem. It performs convolution in canonical Polar coordinate systems, with the success of converting translation-equivariance in Polar coordinates to rotation-equivariance in Cartesian coordinates. Our work is inspired by it. Differently, we extended it for pose estimation with 3D data, with heatmap prediction in θ-ρ and θ-z planes under Cylindrical representations with the goal of realizing rotation equivariance.

III. MULTI-VIEW LOWER-LIMB POSE DATASET A. Real-World Datasets
To benchmark the multi-view depth based pose estimation algorithms, we built a multi-view gait pose dataset 1 with high-quality anatomical joint positions annotated.
As shown in Fig. 2, the Subjects (6 males, 2 females) were instructed to present different walking styles on a treadmill by supplementary wedged insoles or imitation. It follows the experimental settings similar to existing gait research [2], [36], [37], [38]. In total, five Conditions were considered 1 Experiments were approved by the College Ethics Committee of Imperial College London with the reference No. 18IC4915. (normal, supination, pronation, toe-in, and toe-out). The subjects walked on a treadmill, with a calibrated RGBD camera (around 20Hz) and a multi-camera motion capture system (Mocap; 100Hz) collecting data simultaneously. The RGBD camera was positioned in one of the annotated Viewpoints in Fig. 2, towards the subject without strict requirements on positions and orientations. Reflective markers were placed on the lower-limb surfaces based on the conventional gait model 2 and we followed the standard biomechanic analysis pipeline to derive regressed anatomical joint positions. The RGBD camera and Mocap were synchronized by internet triggered timestamps. The recorded RGB images were fed into the human parsing algorithm [26] to generate lower-limb masks for segmenting valid areas of interest. 100 frames of each S-V-C were provided for training if recorded.
In our real-world dataset, data from different viewpoints was not collected simultaneously to minimize light projection interference between cameras (depth-depth & depth-Mocap) and to avoid the intractable synchronization issue between multiple depth cameras [39].

B. Synthetic Datasets
As per our experiment settings, the real-world data is only available from single viewpoint A. To learn occlusion-invariant feature representations, we generated multi-view synthetic data based on a synthetic human mesh model SMPL (skinned multi-person linear model [40]). The kinematics derived from our real-world dataset together with varied realistic body shape parameters were transferred to the model. Hidden point removal was applied to simulate occlusions. For each subject, we generated 1000 samples with simultaneously five viewpoints recorded. They are similar to, but not exactly the same as the five views in our real-world settings.
The real-world dataset, together with the synthetic data generation program based on our derived kinematics, are available upon request https://xiaogu.site/ICL gait. Please refer to the Supplementary Material for more details.

A. Method Overview
For real-scanned depth maps, the occlusion problem P and rotation problem R are entangled, associated with the given viewpoint. In this work, we divide these two problems apart, and introduce our occlusion-invariant and rotation-equivariant approaches to handle them respectively.
Without loss of generality, in this section, the original lower limb representation is denoted as x under canonical coordinates, which can be of different representations such as Euclidean volume R D×W ×H , non-Euclidean point sets R n×3 , etc. The effect of the rotation and occlusion under viewpoint m is denoted as [R m • P m ](x), which can be viewed as the raw data we actually get from the camera. The corresponding pose is denoted as y ∈ R j×3 . The objective of our model F is as below, We denote the viewpoint in the real-world training set as A, and the set composed of all other testing views as X. The set of all the viewpoints as V, where V = {A} ∪ X. The i−th element in Set V or Set X is denoted as V i , X i , separately. Our final objective is to make a model trained on A work on X as well. The superscript r or s of x (as appears in Eq. 5&6&7) represents real or synthetic, respectively. a) Occlusion-Invariant: The first issue to address is the occlusion problem, which is formulated as below, The rotation-equivariance property is formulated as below, It should be noted that in our overall method, occlusion-invariance is data-driven, whereas rotation-equivariance is model-inherent. With these two properties, it is easy to sort out the following equations,

B. Semi-Supervised Occlusion-Invariant Framework
To achieve cross-view occlusion-invariance, we propose a semi-supervised framework to train labelled real data from one viewpoint and unlabelled synthetic data with different occlusions. The reason for such semi-supervised training manner mainly comes from two aspects. On one hand, in real-world settings, it is practically difficult to synchronize multiple depth cameras. This makes sense for our cross-view generalization purposes, as the training procedure is not supposed to see any real data from the other views. On the other hand, synthetic data remains inevitable heterogeneity compared to real data, in terms of both the input data and keypoint positions. Directly mixing these two would lead to poor model performance due to the data divergence.
Inspired by existing work [7], [19] utilizing multi-view consistency for RGB based pose estimation, our framework applies multi-view occluded point clouds in the simulators to reach cross-view occlusion-invariance. As shown in Fig. 3, our framework mainly consists of three components as introduced in the following.

1) Supervised Single-View Pose Estimation Loss:
Based on the available single-view data from the real world, we apply the supervised loss based on available ground truth data, formulated as below, where L dist refers to the explicit loss functions depending on the output formats. It can be L 2 loss between ground truth and the predicted heatmaps, the Euclidean distance between poses, etc. This is to make our semi-supervised framework compatible with other pose estimation methods. Specifically, it is L 2 loss in our case for multi-planar heatmap estimation as depicted in Section IV-C.
2) Unsupervised Multi-View Consistency Loss: On the basis of the straightforward principle that the estimated poses should be the same under different occlusions, we leverage synthetic data from multiple viewpoints and apply the following loss to learn such property.
3) Regularization Loss: There exists heterogeneity between real and synthetic data, in terms of data quality and shape distortions. Under such situations, directly applying L M plus L S would cause overfitting, where L M might force the network into a degenerate trivial solution, generating multi-view consistent but drifted poses. Similar issues were also mentioned in previous RGB based pose estimation work [19], [7], [41]. They applied either regularization loss or front-layer freezing as solutions. In our paper, following the regularization loss proposed in [19], we employed a similar loss to enforce the constraint.
As observed from the training with only real-world data x r , testing results on the synthetic data are reasonably acceptable for those with similar views to the real-world viewpoint A. Therefore, we save the model fully trained with only x r as γ, and the regularization for synthetic data pose estimation is formulated as below, In practice, we did a progressive training procedure, performing training with only L S on x r during the first n epochs, and subsequently applying semi-supervised training with L S &L M &L reg on x r &x s .

C. Rotation-Equivariant Backbone
To this end, it is expected that the cross-view pose estimation problem can be addressed by aforementioned occlusion-invariant framework with random rotation augmentations. However, applying aggressive data augmentation is not a sound solution and the vanilla operations on 3D data like MLPs [42] or convolutions [43] are inherently sensitive to rotational transformations. To address this, we introduce our rotation-equivariant backbone, which not only realizes the rotation equivariance by convolution in the novel coordinate system, but also is more efficient than direct 3D convolution. We firstly have a revisit of group equivariance and then introduce our rotation-euqivariant backbone.
1) Group Equivariance Theorem: The most straightforward group equivariant operation is the 2D planar convolution. 2D images/filters can be considered as planar signals, f, h : R 2 → R. The planar convolution can be viewed as the coordinate value addition over a group of translations on these planar signals [34].
It can be extended to the generalized group convolution for any transformation group G and its value functions f, h : It can be easily proved that the group convolution is equivariant to the operations belonging to this group, where L λ f (g) = f (λ −1 g) refers to a group operation of group λ performed on signal f . In our case, the rotation R around z axis can be viewed as the SO(2) rotation group [35], and our objective is to perform convolution on the SO(2) group with the goal of realizing rotational equivariance 3 .
2) Rotation-Equivariant Convolution with Cylindrical Representations: Instead of directly applying group convolution on SO(2) in the original Cartesian coordinates, we convert (x, y, z) to Cylindrical coordinates (θ, ρ, z), from (x, y, z) to (ρcos(θ), ρsin(θ), z). Considering the group convolution over the translation of θ axis, it can be directly associated with SO(2) equivariance under Cartesian coordinates (explained in Fig. 5) as shown below, 3) θ-r θ-z Heatmap Estimation: By voxelizing the point sets under Cylindrical coordinates, 3 , we can generate the volumetric representation, where ρ 0 and z 0 are predefined boundary parameters, and C is the converted cube length. Till now, architectures similar to V2V can be applied to estimate per-voxel probability via convolutions. As discussed in Section II-B, estimating 3D heatmap demonstrates superior performance compared to direct coordinate regression. However, such manner is computationally expensive and not suited for deployment in mobile devices [44].
To compensate the computational inefficiency of 3D heatmap prediction, while preserving the advantage of utilizing heatmaps as output, we converted the original 3D volumetric data to θ-r and θ-z via anisotropic 3D convolutions and subsequently applied 2D convolutions by periodic padding on those two planes to estimate corresponding 2D heatmaps. Both heatmaps are fused together to estimate the 3D keypoint positions by (sinusoidal) soft-argmax, as shown in Fig. 4. The model details are described as below.
4) Model Details: a) Anisotropic 3D Convolution: Anisotropic 3D convolution proposed in [45] is featured by its elongated anisotropic kernel to aggregate long-range interactions along the elongated axis. The anisotropic probing can "project" 3D data into 2D planes while encoding the 3D structure information into the feature channel. As shown in Fig. 4, the anisotropic 3D convolution along z and r axis converts original volume data to θ-r and θ-z planes separately. ResNet based pose estimation networks are thereafter applied on them to estimate corresponding 2D heatmaps. We applied ResNet-50 as the backbone in practice.
b) Periodic Padding: Normally, for convolution operations, commonly applied padding operations are zero padding. This is not ideal for our case, as there exists periodic property along the θ axis. Differently, periodic padding allows to pad the boundary values on one side to the other side over θ axis, which enables continuous cyclic sliding of kernel filters even at the boundary.
c) Sinusoidal Soft-Argmax: Similarly, according to the periodicity, we applied sinusoidal soft-argmax as proposed in [46] to derive the targeted θ positions formulated as below,  where p(θ) denotes the fused probability of two branches. For ρ&z, we applied the original soft-argmax manner to derive the estimated positions.

A. Experimental Settings
Our experiments were conducted with Pytorch + GeForce RTX 2080Ti GPUs. The architecture details can be found in our Supplementary Material. For our proposed method, the 3D data was firstly applied with min-max normalization and the cube size was set as 128. Subsequently, slight rotational augmentation around x/y axes [-5 • , 5 • ], slight 3D translational augmentation [-5, 5] cube units, and Chirality augmentation [47] were applied. We did not exert additional rotational augmentation around z axis for our method to highlight the rotational equivariance of our backbone. Further details can be found in our Supplementary Material. 1) Evaluation Protocols: To highlight the cross-view cross-subject robustness of our framework and the novel backbone, we introduce three evaluation protocols as mentioned below. The training subset is further divided to training (85%) and validation (15%).
• Cross-Subject (CS) The cross-subject validation protocol was used to evaluate the performance of our backbone model compared to other methods under intra-view settings. We split data into two folds, with four subjects each, and then apply training on one-fold and testing on the other. • Cross-View (CV): Train-A-Test-X To match the practical scenarios where the model trained on one view is expected to generalize to other arbitrary views, our cross-view validation is conducted under the extreme condition by leave-one-view-out for training rather than the commonly used leave-one-view-out for testing. We denote this protocol as cross-view (CV). We use mix the data of all the subjects in the CV only settings. • Cross-View Cross-Subject (CV-CS): Train-A-Test-X -Cross-Subject On the basis of CV, CV-CS further split data into two groups by the cross-subject setting of CS protocol.

2) Compared Methods:
• PN++ [23] This approach is based on PointNet++ [48] extracting hierarchical-level information from input point clouds. It is highlighted by its normalization procedure to minimize difference across viewpoints. Our implementation is based on its official codes 4 . • SO-Net [24] SO-Net is a multi-task framework with additional point cloud self-reconstruction branch for hand pose estimation. We implemented its fully training version 5 by simultaneously minimizing the self-reconstruction and pose estimation losses with all the labelled data. • A2J [20] A2J stands for anchor-to-joint, which firstly estimates 2D heatmaps from the depth maps to derive anchors, and subsequently predicts the joint depth offsets based on surrounding depth values. It can be viewed as a hybrid detection-regression approach 6 . • V2V [17] V2V stands for voxel-to-voxel, which directly estimates 3D heatmaps by an hourglass-like architecture composed of 3D convolutions. Our implementation is based on its Pytorch version 7 .
In the CS session, only the real-world data was used for training. In the CV and CV-CS session, our semi-supervised training was based on labelled real data from view A [R A • P A ](x r ) and unlabelled data from all the views [R A •P Vi ](x s ). For compared methods, largely degraded performance is inevitable if directly training on real data from A and testing on X. We applied a mixed supervised training 8 with real data from A [R A • P A ](x r ), and labelled synthetic data from all views [R Vi •P Vi ](x s ); they were evenly sampled in each batch. Our semi-supervised framework was also applied on V2V with random rotation augmentations for comparison, denoted as V2V-Semi. A similar mixed supervised training based on our backbone model was also implemented, called Proposed-Mix.
3) Metrics: We applied two commonly used 3D pose estimation metrics for evaluation. The first is the 3D mean Euclidean distance error (Dist), which calculates the average 3D Euclidean distance per joint. The second is the mean average precision (mAP), which measures the detection ratio within the precision threshold. Different from the 10cm utilized in [9], we applied 5cm to set a stricter threshold as required for biomechanical gait analysis applications.

B. Quantitative Results
The quantitative results of pose estimation under different protocols are presented in Table I&II&III respectively, with both the error plot of each view and error curve plot attached alongside the corresponding table.
a) CS Validation: It is observed in Table I that V2V and our proposed backbone perform better than the others. This is mainly due to the heatmap output used in V2V and ours, rather than the coordinate regression used in the regression (PN++, SO-Net) or semi-regression (A2J) methods. It is also found that our method performs better than V2V, which demonstrates the effectiveness of our backbone by directly regressing on two projected planes. As the viewpoints and shapes might slightly differ across subjects, our method on Cylindrical representations are more robust against such variations.
Meanwhile, the input resolution of our Cylindrical representations is Cube-128, whereas V2V under Cartesian coordinates applies Cube-88 by its default settings. Direct comparison between these two partitioning resolution is not reasonably fair. Therefore, we also compared the variant version of our proposed model with a cube size of 88, referred to as Proposed-88. Proposed-88 presents a slight difference compared to the original version, yet still outperforms V2V significantly.
It should be noted that V2V applied 3D Residual as its backbone whereas our method applied 2D ResNet-50 as the backbone of each planar branch. Although they share similar residual architectures, the 3D backbone of V2V is rather shallower than ResNet-50. The performance discrepancy might be caused by this; however, the V2V is memory-expensive due to 3D convolutions, and therefore is not feasible for a much deeper design in practice.

b) CV & CV-CS Validation:
The cross-view validation is applied to evaluate the cross-view generalization performance. As shown in Table II & Table III, our method outperforms all other supervised methods for both metrics. V2V-semi and our proposed perform better than V2V and Proposed-Mix respectively, which shows our semi-supervised framework can overcome the domain heterogeneity between real and synthetic datasets without any domain adaptation techniques.
It is observed that for those compared methods, i.e. PN++, SO-Net, A2J, V2V, Proposed-Mix, they all demonstrate inferior results. This validates the inferior performance of directly mixing real and synthetic data for training. Although synthesizing data from novel viewpoints could provide supplementary information of these unseen views, there exists substantial domain discrepancy between real and synthetic data even under the same viewpoint. One one hand, noises, shape variances, cloth/shoe distortions contribute to the difference of data distribution between two domains. On the other hand, the joint data-label distribution is also biased, as the keypoint positions in SMPL are not exactly the same with those defined in clinical gait models. Please refer to our Supplementary Materials.
It is noteworthy that in both the CV and CV-CS settings, the proposed method have better qualitative and quantitative results for Hip, Knee, and Ankle. However, compared to V2V-semi, our method shows slight inferior performance regarding mAP (5cm) of Toe. It is probably caused by the coordinate conversion from Cartesian to Cylindrical, which leads to sparser grid sampling in peripheral areas (with larger radius) of Cartesian space. Toes tend to locate in these peripheral areas during walking, and such sparse sampling would decrease the precision of keypoint estimation in Cartesian space.
C. Qualitative Results a) Pose Estimation Results: In Fig. 6, qualitative pose estimation results from CV-CS validation are displayed. We selected results from different conditions C and viewpoints V to make a comprehensive comparison. It is shown that our method can generate visually better results in most cases, and is more robust against occlusions.
b) Visualizations of occlusion-invariance and rotation -equivariance: The features from intermediate layers between the Anisotropic Conv3D and PoseRes2D, as well as the final estimated heatmap are displayed in Fig. 7. It presents the results of same occlusions & different rotations and different occlusions & same rotations. As shown in Fig. 7, for different occlusions, the generated heatmaps keep consistent, whereas for different rotations, the generated heatmap is periodically shifted across the θ axis as compared to the original one.
c) t-SNE Plot across Walking Conditions C: To validate that the features unique to each C are preserved during pose estimation and to avoid the heterogeneity caused by individual differences [2], we applied our model from CV-CS validation on the complete data of one representative subject. The estimated skeletons were rotated to canonical coordinates, segmented into each gait cycle (from right heel-strike to next heel-strike), and resampled to the same length, similar to [2]. We directly performed t-distributed stochastic neighbor embedding (t-SNE) on the extracted gait cycle samples, and the results are shown in Fig. 8. It can be observed that our model can generate slightly better results compared to

Methods
Time (ms) PN++ [23] 18.86 SO-Net [24] 13.27 A2J [20] 33.72 V2V [17] 43.41 V2V-Semi, in terms of the discernment of estimated skeletons from different C. Although as observed in Fig. 6 the proposed method demonstrates only slight visual performance gains compared to V2V-Semi, these subtle differences would influence the discernment of different gait types as shown in Fig. 8. This emphasizes the necessity of ensuring high precision in biomechanical gait analysis.

a) Computation Efficiency Comparison:
The comparison of the testing time for each sample is presented in Table IV, which highlights the efficiency of our model compared to V2V. b) Results on ITOP Dataset: We reported the results from Side-Test set of ITOP dataset to demonstrate the generalization capability of our method. The "ground truth" in ITOP dataset is semi-manually annotated and does not include toe points. Some qualitative results are displayed in Fig. 10.   We also superimpose the 'ground truth' (gray dashed line) for comparison. It can be observed that our method is able to actually derive the actual joint positions compared to the original labels provided by the dataset. Nonetheless, the qualitative results based on its given "ground truth" are reported in terms of the metric mAP, which is a common metric associated with this dataset [9]. Given the low quality and bias of ITOP GT (see Fig. 10), we apply mAP(15cm) as the metric. They are all around 90% for the mAP(15cm) metric. This is consistent with the qualitative results, demonstrating better quality of our estimated lower limb keypoints even without training on this specific dataset. c) Ablation Study: Here we show the results of without applying L reg (w/o L reg ) and of applying default zero padding instead of periodic padding (i.e., w/o period). We also compared the results of different ratios (0.1 & 0.5) of synthetic data for training. They are compared under one session of CV-CS validation. The better performance of our model compared to these ablated ones can be observed in Fig. 9. The extremely low performance of w/o L reg demonstrates that the model will degenerate to trivial solutions without the constraint enforced by L reg . Periodic padding also plays an important role in our method, as shown by the result of w/o period.

VI. DISCUSSION AND CONCLUSION
To achieve cross-view generalization for depth based pose estimation, we proposed a novel approach based on a multi-view occlusion-invariant framework built on a novel rotation-equivariant backbone model. The occlusion-invariant framework leverages 3D data with different occlusions to achieve occlusion-invariance learning. The rotation-equivariant backbone converts the rotation group equivariance to translational group equivariance by coordinate transformation. Furthermore, it performs 3D pose estimation with a dual-orthogonal-2D-heatmap estimation method, which is more computationally efficient than conventional 3D convolutions as in the state-of-the-art method. We validated our proposed method in a gait dataset and it is demonstrated that the lower-limb skeleton data could be extracted more precisely with discriminative gait features preserved.
Although this work focuses on lower limbs, it opens up the possibility of integrating equivariant geometric learning to 3D human pose estimation to enable the generalization in novel viewpoints. Further work should be aimed at further leveraging synthetic data by sim2real adaptation and generalizing our method to 3D rotations, as well as a more general framework for full-body pose estimation.

VII. DATASET DETAILS A. Experiment Settings
The experiment settings have been briefly introduced in the main paper. Here, a more detailed illustration of the experiment is provided as shown in Fig. 11. For each sample, RGB images, depth images, kinematics, and ground truth skeletons were recorded. The RGB image was processed by the human parsing algorithm cross-domain-complementary-learning (CDCL [26]) to generate masks of lower limbs for background removal and lower limb segmentation. The segmented depth maps can be applied to generate 3D point clouds by the camera intrinsic parameters. Each trial lasts about 30s, and the first 100 frames of each trial are included in the public dataset. The illustration of different walking conditions is shown in Fig. 12. The 3D data directly converted from the depth maps is denoted as [R Vi • P Vi ](x r ).
For each subject, we randomly selected 1000 kinematics samples from the recorded kinematics sequence for synthetic data generation. The skinned multi-person linear model (SMPL [40]) is a realistic articulated human body model and has been applied in existing pose estimation research [11]. It is controlled by two groups of parameters, the shape parameters and the pose parameters. The pose parameters are based on the kinematics derived from our real-world dataset, whereas the shape parameters are sampled from [49]. We applied the hidden point removal provided by Open3D 9 to simulate the occlusions contributed from different viewpoints. They were sampled from a canonical coordinate system for calculating multi-view consistency loss, as shown in the top layer of the synthetic block of Fig. 11. For the methods without applying the multi-view consistency constraints, their inputs were sampled from their local camera coordinate system, as shown in the bottom layer of the synthetic block of Fig. 11.
It should be noted that for the viewpoint V i of synthetic and real data, although they share the same subscript, are roughly but not exactly the same, as we did not set any strict requirement for the extrinsic parameters of the camera.

B. Subject Details
The details of the data from each subject recruited in our experiments are presented in Table VI. In total eight subjects (2 females and 6 males) participated in our experiments and the sample size of each subject is given in Table VI.  Fig. 11. Illustration of the data recording and processing procedures of our dataset. The depth camera and Mocap system simultaneously collected RGBD and ground truth skeleton/kinematics. The RGB image was processed by [26] to generate semantic labels for segmenting the lower limbs. The kinematics derived from Mocap was used to generate synthetic point clouds of different occlusions and rotations.  to the whole lower limb joints, although the stimulation was raised from feet. We also provide the distributions of two selected subjects in Fig. 13(e). It is observed that although intra-subject features present distinct distributions between different classes, inter-subject biases dominant the feature distribution. Such biases pose the challenge associated with overcoming individual difference when performance cross-subject validation.

VIII. MODEL DETAILS
As introduced in the main paper, based on voxelized cylindrical representations, each branch of our pose estimation model is composed of two parts, the 3D anisotropic convolution part and the 2D heatmap estimation part. There are two branches to handle θ-r and θ-z planes separately.
The details of our network architectures are shown in Fig. 14 (Anisotropic Conv3D) and Fig. 15 (PoseRes2D). The 3D anisotropic convolution aims to project the original 3D data into planar signals. Take the upper branch from 3D θ-r-z to 2D θ-r as an example. Although the point cloud is sparse, directly encoding the z value of each occupied voxel into the planar grid might cause conflict problems. Therefore, we applied the 3D anisotropic convolution to capture and encode the long-range spatial features across z axis. This 3D anisotropic convolution can be viewed as 2D convolutions by considering the z as the feature channel dimension. The details of this part are described in Fig. 14.
After the "projection", a heatmap prediction model modified from [50] was applied to generate the heatmap for each plane, as shown in Fig. 15. The downsampling part is based on ResNet-50 (w/o maxpooling after the first convolution layer) whereas the upsampling part is built on the deconvolution operations. All the involved padding operations for θ axis are based on the periodic padding according to the periodicity across θ axis.
The ResNet-50 part of PoseRes2D was initialized by its pre-trained weights. The remaining convolution/deconvolution weights were initialized by normal distribution of std 0.001 and bias by 0. The weights from Batch-Norm layer were initialized by constant values 1 and bias by 0.

A. Cross-View Validation
The training details of cross-view (CV) & cross-view cross-subject (CV-CS) validation protocols are given below.
1) Proposed Method: For our proposed method, we applied the Adam optimizer with the learning rate initialized as 1e-4. The learning rate was decayed by 0.5 per 20 steps. We started the semi-supervised training from Epoch 25 (epoch S ) and the epoch number of the whole training is 45. Our whole training procedure is described in Algorithm 1. It should be noted that one epoch represents a complete iterative sampling of our real-world dataset x r . Empirically, this group of hyper-parameters was stable for all the validation protocols. Fig. 14. Details of the anisotropic 3D convolution network architectures. In each step, elongated anisotropic kernels (1×k×1) were applied to capture the long-distance spatial relationships along the axis to be "projected" through. Each convolution operation is followed by a Batch-Norm and ReLU layer. The input is a volumetric cube with a length of 128, while the output can be viewed as a 128×128 planar image with a channel number of 3.  We applied min-max normalization for all the input to normalize the center and scale of the input, and added Gaussian noises to the synthetic input to simply augment the realism. Only [-5 • , 5 • ] rotation around x/y axis and slight 3D translation augmentation were then applied.
2) Compared Methods: For compared methods, we followed the default settings based on their online available codes.
The input for PN++, SO-Net, V2V, and Proposed-Mix was {[R A • P A ](x r )} and {[R Vi • P Vi ](x s )}. Only the synthetic

B. Cross-Subject Validation
For cross-subject validation, data from all the viewpoints of the training subjects was used for training. The input of all the methods was {[R Vi • P Vi ](x r )}.
For our proposed method, we applied the same hyper-parameters for training without using L M and L reg , with the same preprocessing and augmentations as in the CV & CV-CS session.
For A2J compared in the CS protocol, we adopted its default settings including the data augmentations. The center utilized in its data was based on the lower-limb masks generated from [26]. We did not compare A2J [20] in CV & CV-CS protocols, since we directly generated synthetic point clouds.
For PN++, SO-Net and V2V, their training was the same as that in the CV & CV-CS protocols, except for the absence of synthetic data for training.

A. Joint Difference Visualization
To illustrate the domain shift between real-world data and synthetic data, we derived the pose estimation results of synthetic data based on the model directly trained with the real-world data. The predefined keypoint positions and estimated positions are visualized in Fig. 16. The differences of these two groups of keypoints are resulted from not only the original position deviations but also the heterogeneity between the real and synthetic input data caused by shapes, cloth/shoe distortions, noises, etc.

a) Comparison between w/ & w/o rotation augmentations:
To highlight the rotation-equivariance of our proposed backbone, we did not exert additional z-axis rotational augmentations during the training. Here we also compare the version with additional aggressive rotation augmentations and w/o rotation augmentations. As shown in Table VII, no significant difference (paired t-test, p<0.05) can be seen between the results w/ and w/o rotational augmentations.
On the other hand, whilst additional aggressive rotational augmentations can achieve reasonably good performance of V2V (i.e. V2V-Semi), the performance contributed by such augmentations is still inferior to that of realizing the equivariance by the network itself (i.e. Our proposed).