{'fraction_non_alphanumeric': 0.03654105254562528, 'fraction_numerical': 0.015106356918303188, 'mean_word_length': 4.660734920268085, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
DECA: Deep viewpoint-Equivariant human pose estimation using Capsule Autoencoders

Human Pose Estimation (HPE) aims at retrieving the 3D position of human joints from images or videos. We show that current 3D HPE methods suffer a lack of viewpoint equivariance, namely they tend to fail or perform poorly when dealing with viewpoints unseen at training time. Deep learning methods often rely on either scale-invariant, translation-invariant, or rotation-invariant operations, such as max-pooling. However, the adoption of such procedures does not necessarily improve viewpoint generalization, rather leading to more data-dependent methods. To tackle this issue, we propose a novel capsule autoencoder network with fast Variational Bayes capsule routing, named DECA. By modeling each joint as a capsule entity, combined with the routing algorithm, our approach can preserve the joints' hierarchical and geometrical structure in the feature space, independently from the viewpoint. By achieving viewpoint equivariance, we drastically reduce the network data dependency at training time, resulting in an improved ability to generalize for unseen viewpoints. In the experimental validation, we outperform other methods on depth images from both seen and unseen viewpoints, both top-view, and front-view. In the RGB domain, the same network gives state-of-the-art results on the challenging viewpoint transfer task, also establishing a new framework for top-view HPE. The code can be found at https://github.com/mmlab-cv/DECA.


Introduction
Human pose estimation is key for many applications, such as action recognition, animation, gaming, to name a few [16,29,28]. State of the art methods [2,32] that rely on RGB images can correctly localize human joints (e.g. torso, elbows, knees) in images, also in presence of occlusions. However, they tend to fail when dealing with challenging scenarios. The top-view perspective, in particular, turns out to be a difficult task; on the one hand, it causes the largest amount of joints occlusions, and on the other hand, it suffers the scarcity of suitable training data, as shown in Fig. 1.
When presented with unseen viewpoints, humans display a remarkable ability to estimate human poses, even in the presence of occlusions and unconventional joints configurations. This is not always true in computer vision. In fact, available methods are trained in relatively constrained settings [15], with a limited variability between different viewpoints. Limited data, especially from the topviewpoint, along with limited capabilities of modeling the hierarchical and geometrical structure of the human pose, results in poor generalization capabilities.
This generalization problem, known as the viewpoint problem, depends on how the network activations vary with the change of the viewpoint, usually after a transformation (translation, scaling, rotation, shearing). Convolutional Neural Networks (CNNs) scalar activations are not suitable to effectively manage these viewpoint transformations, thus needing to rely on max-pooling and aggressive data augmentation [4,9,22,36]. By doing so, CNNs aim at achieving viewpoint invariance, defined as f (T x) = f (x) (1) According to this formulation, applying a viewpoint transformation T on the input image x, does not change the outcome of the network activations.
However, a more desirable property would be to capture and retain the transformation T applied to the input image x, thus obtaining a network that is aware of the different transformations applied to the input. Being able to model network activations that change in a structured way according to the input viewpoint transformations is also called viewpoint equivariance and it is defined as: This is achieved by introducing capsules: groups of neurons that explicitly encode the intrinsic viewpoint-invariant relationship existing between different parts of the same object. Capsule networks (CapsNets) can learn part-whole relationships between so-called entities across different viewpoints [12,26,13], similarly to how our visual cortex system operates, according to the recognition-by-components theory [1]. Unlike traditional CNNs, which usually retain viewpoint invariance, capsule networks can explicitly model and jointly preserve a viewpoint transformation T through the network activations, achieving viewpoint equivariance (Eq. 2).
Developing viewpoint-equivariant methods for 3D HPE networks leads to multiple advantages: (i) the learned model is more robust, interpretable, and suitable for realworld applications, (ii) the viewpoint is treated as a learnable parameter, allowing to disentangle the 3D data of the skeleton from each specific view, (iii) the same annotated data can be used to train a network for different viewpoints, thus less training data is required.
In this work, we address the problem of viewpointequivariant human pose estimation from single depth or RGB images. Our contribution is summarised as follows:

• We present a novel Deep viewpoint-Equivariant
Capsule Autoencoder architecture (DECA) which jointly addresses multiple tasks, such as 3D and 2D human pose estimation.
• We show how our network works with limited training data, no data augmentation, and across different input domains (RGB and depth images).
• We show how the feature space organization, defined by routing the input information to build capsule entities, improves when the tasks are jointly addressed.
• We evaluate our method on the ITOP [9] dataset for the depth domain and on the PanopTOP31K [5] dataset for the RGB domain. We establish a new baseline for the viewpoint transfer task and in the RGB domain.
Viewpoint-invariant HPE from RGB images. 3D HPE usually leverages on additional cues, such as 2D predictions [32,34,30], multiple images [38], pre-trained models [17] and pose dictionaries [27]. Other recent works aim at endto-end, learning-based 3D HPE [25,31,21]. In the RGB domain, common HPE datasets such as Human3.6M [14], provide images from multiple views, like front-view or sideview, while the top-view component is generally missing. It is then evident that the lack of suitable multi-view (topview in particular) data implies that state-of-the-art methods [2,32,19,18] necessarily perform poorly when presented with an unseen viewpoint at test time, as shown in Fig. 1(a).
Viewpoint-invariant HPE from depth images. Viewpoint invariant HPE methods have been developed using depth images [9,22,36] from top-view and side-view, using datasets like the K2HPD Body Pose Dataset [35] and the ITOP dataset [9]. To take advantage of the 3D information encoded in 2D depth images, one recent research trend is to resort to 3D deep learning. The paid efforts can be generally categorized into 3D CNN-based and point-set-based families. To enhance the 3D proprieties of depth data and compute more significant features, current methods rely on 3D CNNs [9,22] or 2D CNNs with dense features [36].
3D CNN-based methods [9,22] perform a voxelization operation on pixels to transform them into 3D objects. To process the 3D data, each network performs costly 3D convolutions on the input data. These operations are responsible for the high computational burden and the difficulty to properly tune a high number of parameters in 3D CNNs. In the domain of 2D CNNs, Xiong et al. [36] capture the 3D structure by computing dense features in an ensemble way, thus avoiding computationally intensive CNN layers, but they still rely on a backbone pre-trained network to extract 2D features. Still, the above-mentioned approaches usually achieve weak viewpoint-invariance but fail to model viewpoint-equivariance. Moreover, we argue that the 3D geometry of the data should be interpreted by the network without relying on the voxelization embedding, or a 2D pretrained feature extraction network.
Capsule networks for HPE. Capsule networks have shown the ability to model the geometric nature of training data thanks to the network structure and features [26,13,20]. Sabour et al., introduce a routing algorithm for vector capsules, called routing-by-agreement as a better maxpooling substitute. Hinton et al. [13] further improve accuracy through a more complex matrix capsule structure and an Expectation-Maximization routing (EM-routing) for capsules. Unfortunately, the EM-routing and the 4 × 4 pose matrix embedded in the capsule contribute to increasing the training time, when compared to both CNNs and vector CapsNets. Kosiorek et al. [20] introduce for the first time an unsupervised capsule-based autoencoder. Ribeiro et al. in [24] build upon the EM-routing version of capsule by proposing for the first time a Variational Bayes capsule routing (VB routing) fitting a mixture of transforming Gaussians. They present state-of-the-art results using ∼ 50% fewer capsules, achieving both performance gain and network complexity reduction. However, all the mentioned works only consider small datasets, such as MNIST, small-NORB, and CIFAR-10 for benchmarking.
In the RGB domain, Ramírez [23] tackles the problem of RGB HPE using dynamic vector capsule networks [26] to solve the 3D HPE problem in an end-to-end fashion. However, their work only exploits lateral viewpoints from the Human3.6M dataset and only considering RGB data.
In this work, we use matrix capsules [13], along with a different capsule routing algorithm and a new encodingdecoding pipeline with GELU activations. We argue that matrix capsules are better suited than vector capsules for the 3D HPE task, as the 4 × 4 pose matrix used for the routing can capture 3D geometry better than a dynamic vector structure.

Method
We now analyze the proposed autoencoder, DECA, starting with the capsule encoder and the multi-task decoders. DECA can be trained end-to-end, without any pre-training or data augmentation, and it works in real-time in the inference phase. An overview of the proposed architecture is shown in Fig. 2.

Capsule encoder
The encoding module of the network (light blue in Fig.  2) is divided in: (i) an input pre-processor I, (ii) a CNN encoder E and (iii) four layers of Matrix Capsules with Variational Bayes Routing [24].
(i) I is a layer which normalizes the different type of data (RGB images, depth images, top-view, side-view, freeview) in the interval [0, 1].
(ii) The normalised input is then forwarded to a CNN encoder E, built using four convolutional layers with inputs [N ch , 64, 128, 256], instance normalisation and GELU activations [11], as shown in Eq. 3. N ch is the number of channels, which may vary depending on the input.
(iii) The output of the CNN encoder E feeds our capsule layers. It has been shown in previous works [26,13,20] that capsules provide a superior understanding of the viewpoint and the relationship between parts and parent objects, thus aiming at true viewpoint equivariance. Given the multiple degrees of freedom of each joint, we adopt the matrix capsules model [13] instead of vector capsules [26], enriching the description of single joints as hierarchically linked capsule entities. We deploy the novel capsule routing based on Variational Bayes (VB) [24], which is proven to speed up the training of our matrix capsules layers, at the same time improving performances. The last iteration of the VB routing is also called ClassRouting and it is used to route the highest-level information to the last layer of capsules before the feature space F.
In our CapsNet, we employ four layers: a primary capsules layer encapsulates the output features of E into 16dimensional capsules, two convolutional capsules layers refine the capsule features, and a final class capsules layer encodes the output into a J-dimensional features in the latent space F, where J is the number of joints, also called entities.
Given each lower-level capsule i and the corresponding higher-level capsule j, we define M i as the proposed lower level pose matrix and W ij ∈ R 4×4 as a trainable viewpointequivariant transformation matrix such that: where V j|i is the vote coming from lower capsules i for higher capsules j. The voting procedure takes place inside the VB routing and it allows each lower capsule i to route its information to a higher capsule j of its choice, thus allowing to build the hierarchical structure typical of CapsNets.
To promote the viewpoint equivariance in Eq. 2, we introduce an inverse matrixŷ W in the class capsules, which aims at satisfying the Inverse Graphics constraint: meaning that the learned inverse matrixŷ W effectively acts as an approximated inverse of the rendering operation, as it is commonly found in computer graphics [12].
At the output of the encoder, each entity corresponding to each joint of the skeleton is defined by a flattened vector of 16 elements, or, in other words, a 4 × 4 matrix, which is sufficient to grasp the complete pose (translation + rotation) of each joint.
An overview of the capsule encoder is shown in Algorithm 1. In the algorithm, s 3D , s 2D , s DM , s W are weights used for the self-balancing of the loss, w c are the convolutional layer weights, a are the activations of each Capsule layer, and {·} represents parameters used only when in the RGB domain.

Multi-task decoders
Starting from the 16-dimensional entities in the capsule feature space F, we design a decoding module (light orange block in Fig. 2) that allows us to simultaneously retrieve multiple predictions for different tasks from the same feature space F. Each decoder D τ in the decoding module is configured as an independent fully connected block, with 0.5 Dropout and GELU activations [11]. We employ no weight sharing or layer sharing across the decoders to enforce the multi-task loss, as explained in section 3.3.
We define different tasks (τ ) with different objectives: • 3D: minimise the distance between ground truth and predicted 3D joints in 3D spaceŷ 3D ; • 2D: as above, but without relying on 3D joints predictions, and rather predicting 2D jointsŷ 2D as seen from the current viewpoint in camera frame coordinates; • DM: reconstruct the depth mapŷ DM of the input RGB image. It is used only in the RGB domain; • W Inverse Graphics loss : learn the inverse graphics matrixŷ W to promote the de-rendering of input pixels into isolated capsule entities, as explained in Sec. 3.1, Eq. 5.
For each task τ = 3D, 2D, DM, a decoder D τ takes as input the feature space F and it outputs the predictionŝ Y = [ŷ 3D ,ŷ 2D , {ŷ DM }] to the loss function. For W, thê y W matrix is forwarded to the loss function directly from the encoder.
An overview of the capsule decoders is shown in Algorithm 2.

Algorithm 2: Capsule decoders
3.3. Self-balancing multi-task loss Tasks are associated to the different input domains, as follows: Each task is assigned a loss L τ , defined as: • L 2D , L 3D : Mean Square Error (MSE) loss for the 3D and 2D joints prediction tasks.
• L DM : masked L1 loss for the depth estimation task DM, in the RGB domain, where mask is a function that applies the L1 loss only on pixels over a certain depth threshold, to promote the depth estimation over non-background areas.
• L W : inverse graphics loss W, which role is to enforce invertibility for the capsule weight matrices. The notation . F defines the Frobenius norm of a matrix.
Considering T as the set of the employed tasks τ , the overall balanced loss for all the tasks is expressed as: where s τ = [s 3D , s 2D , s DM , s W ] are the trainable weights associated with each loss in T , initialised to 1 in algorithm 1, and L τ is each loss of the enabled decoders, as defined in Eq. 6.

Datasets
ITOP Dataset of depth images. The ITOP dataset [9] contains depth images from top and front view. The training split and the test split consist of 40k and 10k images, respectively. The depth images display 15 videos of 20 actors in a constrained setting. The dataset is recorded using two Axus Xtion Pro cameras. The 3D skeleton model consists of 15 joints.
PanopTOP31K dataset of depth and RGB images. The PanopTOP31K dataset [5] consists of 34k top-view and 34k front view images coming from video sequences of 24 different actors, available both in the RGB and depth domain, for a total of 68k images. The ground truth 3D skeleton consists of 19 joints.

Evaluation metrics
Following the works of [9,22,36], we choose the mean average precision (mAP) as the evaluation metric for the depth domain. It is defined as the percentage of all predicted joints which fall in an interval smaller than 0.10 meters. In the RGB domain, we use the Mean Per Joint Position Error (MPJPE) in millimeters as in many HPE works [2, 32, 23].

Implementation details
Our network is trained in an end-to-end fashion using Pytorch Lightning. Input images are normalized in the interval [0, 1] with a resolution of 256x256 pixels for depth  Figure 3: 2D representation on the 16-dimensional latent space obtained using t-SNE [33]. Each dot corresponds to an entity E jt representing a joint jt of the skeleton from the test set of ITOP [9]. V2V network [22] relies on CNNs, thus is not able to cluster together samples corresponding to the same entity (a). When trained to satisfy only the 3D prediction constraint our DECA-D1 network performs slightly better than V2V (b). The 15 clusters, corresponding to the 15 joints of the skeleton model, are clearly distinguishable in DECA-D2 (c) and DECA-D3 (d), with (d) displaying better cluster separation and fewer outliers.
images and 256x256 pixels for RGB ones. We do not perform any augmentations on the input datasets. The batch size is set to 128 for ITOP and 128 for PanopTOP31K. We initialize the weights with the Xavier initialization [6]. The learning rate is set to 1e −5 , the weight decay is set to 0, and Adam is the optimizer of choice. We train our network for 20 epochs on the ITOP dataset and 15 epochs on Panop-TOP31K.

Feature space entities and ablation study
We report experiments on the top-view of the ITOP dataset [9] to validate the 3D representation provided by our network and to show how the multi-tasks decoder influences the overall performances.
To do so, we deploy 4 configurations, 3 on depth data and 1 on RGB data, with different sets of tasks T of our method: where the letter D or R indicates the depth or RGB domains, and the number defines how many tasks are assigned to the network. Since we are evaluating the performances on the 3D HPE, the τ = [3D] is used for all the different configurations.
Loss effectiveness analysis. The results are reported in the last 3 columns of Table 1. As shown in the Table, increasing the number of tasks in T generally leads to an increase in the network's performances. DECA-D1 already achieves similar results to the state-of-the-art, thanks to the CapsNets' capability to interpret the geometrical nature of the input data. When the inverse graphics loss W is employed (DECA-D2 and DECA-D3), the enforced invertibility of the weights matrix leads to an immediate gain in performances. In DECA-D3, the introduction of the 2D loss leads to an additional improvement in terms of accuracy. Hence, we argue that the network performances improve when more tasks are given because we achieve a better representation of the entities in the latent space.
Latent space analysis. To analyze the latent space, we use the features of the test set extracted after the capsule modules. Each feature f ∈ F is linearised to obtain a vector of length L f eat . At this stage, each entity E jt corresponding to each joint jt is defined by dividing each feature vector by the number of joints, resulting in vectors of length L f eat #of joints . For visualisation purposes, we use t-SNE [33] to project the entities on a 2-dimensional space. The results are displayed in Fig. 3. We compare our latent space against the publicly available version of the V2V [22] encoder/decoder structure. We show how our DECA network can better cluster and separate each entity E jt with respect to V2V. Our solution provides a better organization of the latent space, with bigger inter-class margins and fewer outliers. The latent space organization improves drastically when we employ the τ = W task (DECA-D2), thus enforcing the inverse graphics constraint. In DECA-D3 we add the τ = 2D task. The resulting organization of the latent space improves, thus further establishing a correlation between the growing number of tasks and the improvement in performances.

Comparison with state-of-the-art methods
Depth data: ITOP dataset. We compare our DECA against common state-of-the-art method for human pose estimation on depth images [28,37,3,9,8,22,36]. The results are reported in Tab ing methods on the front-view task, improving the accuracy by a wide margin on the more challenging top viewpoint.
In general, we consistently perform better than other methods on most of the joints and the average. The gain of our method is particularly large when dealing with the lower body, which is often occluded in the top-view.
Depth data: Viewpoint-equivariant ITOP. We test DECA on the viewpoint transfer task, meaning training on one viewpoint, either top-view or front-view, and testing on the other one, unseen at training time. The comparison against available state-of-the-art methods [28,37,3,9] are reported in Tab. 2. We consistently outperform other methods by a wide margin, thus making a step forward toward viewpoint equivariance. While other methods provide only the best subset of viewpoint transfer results (Tab. 2), omitting entirely the train on top and test on front scenario, we provide results for all the joints and all the viewpoint transfer combinations in Tab. 3. Our DECA achieves better results than the top-most of the other methods on many different joints (e.g. shoulders, lower body). In Tab 3, training DECA on top-view or front-view achieves comparable lower body accuracy. This means that when the network is trained on top view, where the lower body is mostly occluded, it can retrieve the occluded joints from previously unseen front views, and vice versa. This shows how our network has learned the viewpoint as a parameter, and it is thus able to generalize in a similar fashion in all the viewpoint transfer combinations.
RGB data: Viewpoint-equivariant PanopTop31K. To the best of our knowledge, we are the first to tackle the problem of viewpoint transfer between top-view and front-view in the RGB domain. We report results with training and testing on both seen and unseen viewpoints in Tab. 4. The chosen metric is the mean per-joint projection error (MPJPE). We report results with and without the Procrustes alignment [7] of the predicted poses. It is interesting to notice how DECA can reduce the gap between the same viewpoint results and the results of the viewpoint transfer tasks. In the case of viewpoint transfer, we train on viewpoint A, validate on the same viewpoint A and test on viewpoint B.

Qualitative results
In Fig. 4 we show some qualitative results from DECA-R4 configuration on RGB data. We deploy our network training and testing on all the possible viewpoint combinations. The network takes as input either the top-view RGB (Fig. 4a) image or the front view (Fig. 4b) one. When trained and tested on the same viewpoint (Fig. 4d, 4e), the network produces similar outputs, thus confirming its abil-   ity to deal with the challenging top-view scenario. When training on the top view and testing on the front one (Fig.  4f), the network can accurately retrieve the positions of the lower body joints. DECA can retrieve parts of the body mostly occluded ad training time, thus displaying its generalization capabilities. When training on the front view and testing on the top one (Fig. 4g), the network can retrieve the positions of the upper body joints, which are visible in both images but from different perspectives, proving that DECA can internally model the viewpoint.

Conclusions
We presented DECA, a deep viewpoint-equivariant method for human pose estimation on single RGB/depth images using capsule autoencoders. We show how Cap-sNets are better suited to deal with the 3D nature of raw data and how they allow taking a step forward to viewpoint equivariance. We have shown how our method can effectively generalize and achieve state-of-the-art results in both RGB and depth domains, as well as in the viewpoint transfer task. In future work, we aim at improving hands pose estimation and employing matrix capsules on bigger RGB datasets.