{'fraction_non_alphanumeric': 0.03811466267248797, 'fraction_numerical': 0.021383727970935976, 'mean_word_length': 4.782200110558319, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
MotionBERT: Unified Pretraining for Human Motion Analysis

We present MotionBERT, a unified pretraining framework, to tackle different sub-tasks of human motion analysis including 3D pose estimation, skeleton-based action recognition, and mesh recovery. The proposed framework is capable of utilizing all kinds of human motion data resources, including motion capture data and in-the-wild videos. During pretraining, the pretext task requires the motion encoder to recover the underlying 3D motion from noisy partial 2D observations. The pretrained motion representation thus acquires geometric, kinematic, and physical knowledge about human motion and therefore can be easily transferred to multiple downstream tasks. We implement the motion encoder with a novel Dual-stream Spatio-temporal Transformer (DSTformer) neural network. It could capture long-range spatio-temporal relationships among the skeletal joints comprehensively and adaptively, exemplified by the lowest 3D pose estimation error so far when trained from scratch. More importantly, the proposed framework achieves state-of-the-art performance on all three downstream tasks by simply finetuning the pretrained motion encoder with 1-2 linear layers, which demonstrates the versatility of the learned motion representations.


Introduction
Perceiving and understanding human activities have long been a core pursuit of machine intelligence. To this end, researchers define various sub-tasks to abstract semantic representations from videos, e.g. skeleton keypoints [13,31], action labels [53,107], and surface meshes [41,59]. While existing studies have made significant progress in each of the sub-tasks, they tend to confine and model the problems in a disjoint manner. For example, Graph Convolutional Networks (GCN) have been applied to modeling spatio- 1 Project page: https://motionbert.github.io/ temporal relationship of human joints in both 3D pose estimation [12,25,101] and action recognition [83,107], but their connections are barely explored. Intuitively, these models should all have learned to capture typical human motion patterns despite being designed for different problems. Nonetheless, current methods fail to mine and utilize such commonalities across the sub-tasks.
One major obstacle to unifying the sub-tasks of human motion analysis is data resource heterogeneity. Mocap systems [34,63] offer high-fidelity 3D motion with markers and sensors, but the appearances of captured videos are usually constrained to simple indoor scenes. Action recognition datasets provide annotations of the action semantics, but they either contain no human pose labels [15,82] or feature limited motion of daily activities [52,53,81]. In-the-wild human videos are massively accessible with diverse appearance and motion, but acquiring precise 2D pose annotations needs non-trivial efforts [3], and getting ground-truth (GT) 3D joint locations is almost impossible. Consequently, existing studies basically focus on one specific sub-task using an isolated type of motion data.
In this paper, we introduce a new framework named Mo-tionBERT for human motion analysis across various subtasks, as illustrated in Figure 1. It consists of a unified pretraining stage and a task-specific finetuning stage, motivated by the recent successful practices in natural language processing [11,26,78] and computer vision [7,32]. During pretraining, a motion encoder is trained to recover 3D human motion from corrupted 2D skeleton sequences, which could incorporate different kinds of human motion data sources. This challenging pretext task intrinsically requires the motion encoder to i) infer the underlying 3D human structures from its temporal movements; ii) recover the erroneous and missing observations. In this way, the motion encoder captures the human motion commonsense such as joint linkages, anatomical constraints, and temporal dynamics. In practice, we propose Dual-stream Spatio-temporal Transformer (DSTformer) as the motion encoder to capture the long-range  Figure 1. Framework overview. The proposed framework consists of two stages. During the pretraining stage, we extract the 2D skeleton sequences from a variety of motion data sources. We then corrupt the 2D skeletons by applying random masks and noises, from which the motion encoder is trained to recover the 3D motion. At the finetuning stage, we adapt the learned motion representations to different downstream tasks by jointly optimizing the pretrained motion encoder and a few linear layers. relationship among skeleton keypoints. We suppose that the motion representations learned from large-scale and diversified data resources could be shared by all the relevant downstream tasks and benefit their performance. Therefore, for each downstream task, we adapt the pretrained motion representations with a simple regression head using taskspecific training data and supervisory signals.
In summary, the contributions of this work are three-fold: 1) We propose a unified pretraining method to leverage the large-scale yet heterogeneous human motion sources and learn generalizable motion representations. Our approach could take advantage of the precision of 3D mocap data and the diversity of in-the-wild RGB videos at the same time. 2) We design a dual-stream Transformer network with cascaded spatio-temporal self-attention blocks that serves as a general backbone for human motion modeling. 3) The proposed unified pretraining approach empowers a versatile human motion representation, which could be transferred to multiple downstream tasks even with scarce labeled data. Without bells and whistles, MotionBERT outperforms the state-of-the-art methods on 3D pose estimation, skeletonbased action recognition, and mesh recovery, respectively.

Related Work
Learning Human Motion Representations. Early works formulate human motion with Hidden Markov Models [46,94] and graphical models [44,86]. Kanazawa et al. [38] design a temporal encoder and a hallucinator to learn representations of 3D human dynamics. Zhang et al. [114] predict future 3D dynamics in a self-supervised manner. Sun et al. [89] further incorporate action labels with an action memory bank. From the action recognition perspective, a variety of pretext tasks are designed to learn motion representations in a self-supervised manner, including future prediction [87], jigsaw puzzle [49], skeleton-contrastive [93], speed change [88], cross-view consistency [51], and contrastreconstruction [102]. Similar techniques are also explored in tasks like motion assessment [29,72] and motion retargeting [110,121]. These methods leverage homogeneous motion data, design corresponding pretext tasks, and apply them to a specific downstream task. In this work, we propose a unified pretrain-finetune framework to incorporate heterogeneous data resources and demonstrate its versatility in various downstream tasks.
3D Human Pose Estimation. The 3D human pose estimation methods can be divided into two categories. The first is to estimate 3D poses with CNN directly from images [69,91,118]. One limitation of these approaches is that there is a trade-off between 3D pose precision and appearance diversity under current data collection techniques. The second category is to extract the 2D pose first, then lift the estimated 2D pose to 3D with a separate neural network. The lifting can be achieved via Fully Connected Network [65], Temporal Convolutional Network (TCN) [21,77], GCN [12,25,101], and Transformer [48,116,117]. Our framework is built upon the second category as we use the proposed DSTformer to accomplish the 2D-to-3D lifting pretext task.
Skeleton-based Action Recognition. Early works [62,100,111] point out the intrinsic relationship between human pose estimation and action recognition. Towards modeling the spatio-temporal relationship among human joints, previous studies employ LSTM [85,120] and GCN [20,47,57,83,107]. Most recently, PoseConv3D [28] proposes to apply 3D-CNN on the stacked 2D joint heatmaps and achieves improved results. In addition to the fully-supervised action recognition task, NTU-RGB+D-120 [53] brings attention to the challenging one-shot action recognition problem. To this end, SL-DML [68] applies deep metric learning to multimodal signals. Sabater et al. [80] explores one-shot recognition in therapy scenarios with TCN. We demonstrate that the pretrained motion representations could generalize well to action recognition tasks, and the pretrain-finetune framework is a suitable solution for the one-shot challenges.
Human Mesh Recovery. Based on the parametric human models such as SMPL [59], many research works [37,70,106,115] focus on regressing the human mesh from a single image. Despite their promising per-frame results, these methods yield jittery and unstable results [41] when applied to videos. Several works [22,38,41,92] take video clips as input and produce smoother results by exploiting the temporal cues. Another common problem is that paired images and GT meshes are mostly captured in constrained scenarios, which limits the generalization ability of the above methods. To that end, Pose2Mesh [23] proposes to first extract 2D skeletons using an off-the-shelf pose estimator, then lift them to 3D mesh vertices. We adopt the 2D skeletons as the intermediate representation and further extend them to video inputs, which reduces the ambiguity and could further benefit from our pretrained motion representations.

Overview
As discussed in Section 1, our approach consists of two stages, namely unified pretraining and task-specific finetuning. In the first stage, we train a motion encoder to complete the 2D-to-3D lifting task, where we use the proposed DSTformer as the backbone. In the second stage, we finetune the pretrained motion encoder and a few linear layers on the downstream tasks. We use 2D skeleton sequences as input for both pretraining and finetuning because they could be reliably extracted from all kinds of motion sources [3,9,63,73,90], and is more robust to variations [18,28]. Existing studies have shown the effectiveness of using 2D skeleton sequences for different downstream tasks [23, 28,77,95]. We will first introduce the architecture of DSTformer, and then describe the training scheme in detail. Figure 2 shows the network architecture for 2D-to-3D lifting. Given an input 2D skeleton sequence x ∈ R T ×J×Cin , we first project it to a high-dimensional feature F 0 ∈ R T ×J×Cf , then add learnable spatial positional encoding P S pos ∈ R 1×J×Cf and temporal positional encoding P T pos ∈ R T ×1×Cf to it. We then use the sequence-to-sequence model DSTformer to calculate F i ∈ R T ×J×Cf (i = 1, . . . , N ) where N is the network depth. We apply a linear layer with tanh activation to F N to compute the motion representation E ∈ R T ×J×Ce . Finally, we apply a linear transformation to E to estimate 3D motionX ∈ R T ×J×Cout . Here, T denotes the sequence length, and J denotes the number of body joints. C in , C f , C e , and C out denote the channel numbers of input, feature, embedding, and output respectively. We first introduce the basic building blocks of DSTformer, i.e. Spatial and Temporal Blocks with Multi-Head Self-Attention (MHSA), and then explain the DSTformer architecture design.

Network Architecture
Spatial Block. Spatial MHSA (S-MHSA) aims at modeling the relationship among the joints within the same time step. It is defined as where W P S is a projection parameter matrix, h is the number of the heads, i ∈ 1, . . . , h. We utilize self-attention to get the query Q S , key K S , and value V S from input per-frame spatial feature F S ∈ R J×Ce for each head i , where W are projection matrices, and d K is the dimension of K S . We apply S-MHSA to features of different time steps in parallel. Residual connection and layer normalization (LayerNorm) are used to the S-MHSA result, which is further fed into a multilayer perceptron (MLP), and followed by a residual connection and LayerNorm following [98]. We denote the entire spatial block with MHSA, LayerNorm, MLP, and residual connections by S.
Temporal Block. Temporal MHSA (T-MHSA) aims at modeling the relationship across the time steps for a body joint. Its computation process is similar with S-MHSA except that the MHSA is applied to the per-joint temporal feature F T ∈ R T ×Ce and parallelized over the spatial dimension.  Figure 2. Model architecture. We propose the Dual-stream Spatio-temporal Transformer (DSTformer) as a general backbone for human motion analysis. DSTformer consists of N dual-stream-fusion modules. Each module contains two branches of spatial or temporal MHSA and MLP. The Spatial MHSA models the connection among different joints within a timestep, while the Temporal MHSA models the movement of one joint.

2D Skeletons 3D Motion DSTformer
where i ∈ 1, . . . , h, Q T , K T , V T are computed similar with Formula 2. We denote the entire temporal block by T .
Dual-stream Spatio-temporal Transformer. Given spatial and temporal MHSA that captures the intra-frame and inter-frame body joint interactions respectively, we assemble the basic building blocks to fuse the spatial and temporal information in the flow. We design a dual-stream architecture with the following guidelines: 1) The two streams could both model the comprehensive spatio-temporal context. 2) The two streams are specialized for different spatio-temporal aspects.
3) The two streams are fused together, and the fusion weights are dynamically balanced according to the input spatio-temporal characteristics. Therefore, we stack the spatial and temporal MHSA blocks in different orders, forming two parallel computation branches. The output features of the two branches are fused using adaptive weights predicted by an attention regressor. The dual-stream-fusion module is then repeated for N times: where F i denotes the feature embedding at depth i, • denotes element-wise production. Orders of S and T blocks are shown in Figure 2, and different blocks do not share weights. Adaptive fusion weights α ST , α TS ∈ R N ×T ×J are given by where W is a learnable linear transformation.

Unified Pretraining
We address two key challenges when designing the unified pretraining framework: 1) How to learn a powerful motion representation with a universal pretext task. 2) How to utilize large-scale but heterogeneous human motion data in all kinds of formats.
For the first challenge, we follow the pretrained models in language [11,26,78] and vision [7,32] modeling to construct the supervision signals, i.e. mask part of the input and use the encoded representations to reconstruct the whole input. Note that such "cloze" task naturally exists in human motion analysis, that is to recover the lost depth information from the 2D visual observations, i.e. 3D human pose estimation. Inspired by this, we leverage the large-scale 3D mocap data [63] and design a 2D-to-3D lifting pretext task. We first extract the 2D skeleton sequences x by projecting the 3D motion orthographically. Then, we corrupt x by randomly masking and adding noise to produce the corrupted 2D skeleton sequences, which also resemble the 2D detection results as it contains occlusions, detection failures, and errors. Both joint-level and frame-level masks are applied with certain probabilities. We use the aforementioned motion encoder to get motion representation E and reconstruct 3D motion X. We then compute the joint loss L 3D betweenX and GT 3D motion X. We also add the velocity loss L O following previous works [77,116]. The 3D reconstruction losses are given by For the second challenge, we notice that 2D skeletons could serve as a universal medium as they can be extracted from all sorts of motion data sources. We further incorporate in-the-wild RGB videos into the 2D-to-3D lifting framework for unified pretraining. For RGB videos, the 2D skeletons x could be given by manual annotation [3] or 2D pose estimator [13,90], and the depth channel of the extracted 2D skeletons is intrinsically "masked". Similarly, we add extra masks and noises to degrade x (if x already contains de-tection noise, only masking is applied). As 3D motion GT X is not available for these data, we apply a weighted 2D re-projection loss which is calculated by wherex is the 2D orthographical projection of the estimated 3D motionX, and δ ∈ R T ×J is given by visibility annotation or 2D detection confidence. The total pretraining loss is computed by where λ O is a constant coefficient to balance the training loss.

Task-specific Finetuning
The learned feature embedding E serves as a 3D-aware and temporal-aware motion representation. For downstream tasks, we adopt the minimalist design principle, i.e. implementing a shallow downstream network and training without bells and whistles. In practice, we use an extra linear layer or an MLP with one hidden layer. We then finetune the whole network end-to-end.
3D Pose Estimation. As we utilize 2D-to-3D lifting as the pretext task, we simply reuse the whole pretraining network (motion encoder + linear). During finetuning, the input 2D skeletons are estimated from videos without extra masks or noises.
Skeleton-based Action Recognition. We simply perform a global average pooling over different persons and timesteps. The result is then fed into an MLP with one hidden layer. The network is trained with cross-entropy classification loss. For one-shot learning, we apply a linear layer after the pooled features to extract clip-level action representation. We introduce the detailed setup of one-shot learning in Section 4.4.
Human Mesh Recovery. We use SMPL [59] model to represent the human mesh and regress its parameters. The SMPL model consists of pose parameters θ ∈ R 72 and shape parameters β ∈ R 10 , and calculates the 3D mesh as M(θ, β) ∈ R 6890×3 . To regress the pose parameters for each frame, we feed the motion embeddings E to an MLP with one hidden layer and getθ ∈ R T ×72 . To estimate shape parameters, considering that the human shape over a video sequence is supposed to be consistent, we first perform an average pooling of E over the temporal dimension and then feed it into another MLP to regress a singleβ and then expand it to the entire sequence asβ ∈ R T ×10 . The shape MLP has the same architecture as the pose regression one, and they are initialized with the mean shape and pose, respectively, as in [41]. The overall loss is computed as where each term is calculated as Note that each 3D pose in motion X m at frame t is regressed from mesh vertices by X m t = JM(θ t , β t ), where J ∈ R J×6890 is a pre-defined matrix [9]. λ m 3D , λ θ and λ β are constant coefficients to balance the training loss.

Implementation
We implement the proposed motion encoder DSTformer with depth N = 5, number of heads h = 8, feature size C f = 256, embedding size C e = 512. For pretraining, we use sequence length T = 243. The pretrained model could handle different input lengths thanks to the Transformerbased backbone. During finetuning, we set the backbone learning rate to be 0.1× of the new layer learning rate. We introduce the experiment datasets in the following sections respectively. Please refer to Appendix A.1 for more experimental details.

3D Pose Estimation
We evaluate the 3D pose estimation performance on Hu-man3.6M [34] and report the mean per joint position error (MPJPE) in millimeters, which measures the average distance between the predicted joint positions and the GT after aligning the root joint. We use the Stacked Hourglass (SH) networks [73] to extract the 2D skeletons from videos, and finetune the entire network on Human3.6M [34] training set. In addition, we train a separate model of the same architecture, but with random initialization rather than pretrained weights. Both of our models outperform the state-of-the-art methods, as shown in Table 1. The model trained from scratch demonstrates the effectiveness of the proposed DSTformer in terms of learning 3D geometric structures and temporal dynamics. The proposed pretraining stage additionally lowers the error, which proves the advantage of the unified pertaining framework.

Skeleton-based Action Recognition
We further explore the possibility to learn action semantics with the pretrained motion representations. We use the human action dataset NTU-RGB+D [81] which contains 57K videos of 60 action classes, and we follow the data splits Cross-subject (X-Sub) and Cross-view (X-View). The dataset has an extended version, NTU-RGB+D-120 [53], which contains 114K videos of 120 action classes. We follow the suggested One-shot action recognition protocol on NTU-RGB+D-120. For both datasets, we use HRNet [90] to extract 2D skeletons following [28]. Similarly, we train a scratch model with random initialization for comparison. All the models are evaluated without ensemble. As Table  2 shows, our methods are comparable or superior to the state-of-the-art approaches. Notably, the pretraining stage accounts for a large performance gain.
In addition, we study the one-shot setting which is of practical importance. In real-world deployment, action recognition in certain scenarios (e.g. education, sports, healthcare) are highly demanded. However, only scarce annotations are available for the novel action classes that are unseen in the public datasets. As proposed in [53], we report the results on the evaluation set of 20 novel classes using only 1 labeled video for each class. The auxiliary set contains the other 100 classes, and all samples of these classes can be used for learning. We train the model on the auxiliary set using the supervised contrastive learning technique [39]. For a batch of auxiliary data, samples of the same class are pulled together, while samples of different classes are pushed away in the action embedding space. During the evaluation, we calculate the cosine distance between the test examples and the exemplars, and use 1-nearest neighbor to determine the class. Table 2 illustrates that the proposed models outperform state-of-the-art by a considerable margin. It is also worth noting the pretrained model achieves optimal performance with only 1-2 epochs of finetuning. The results imply that the pretraining stage indeed helps to learn a good motion representation which could generalize to novel downstream tasks even with limited data annotations.

Human Mesh Recovery
We conduct experiments on Human3.6M [34] and 3DPW [99] datasets. For Human3.6M, we follow the conventional training and test split [65] as in Section 4.2. For 3DPW, we add COCO [50] dataset for training following [23]. We report results with and without the 3DPW training set as in [22,41]. Following the common practice [37,41,43], we report MPJPE (mm) and PA-MPJPE (mm) of 14 joints obtained by JM(θ, β). PA-MPJPE calculates MPJPE after aligning with GT in translation, rotation, and scale. We further report the mean per vertex error (MPVE) (mm) of the mesh M(θ, β), which measures the average distance between the estimated and GT vertices after aligning the root joint.

Ablation Studies
Model Architecture. We first study the design choices of DSTformer and report the results in Table 6. From For all the variants, we control the total number of self-attention blocks to be the same, so the network capacity would not be the defining factor. The result confirms our design principles that both streams should be capable and meanwhile complementary, as introduced in Section 3.2. In addition, we find out that  pairing each self-attention block with an MLP is crucial, as it could project the learned feature interactions and bring nonlinearity. From (g) to (k), we trial different model sizes.
In general, we optimize the model architecture design based on their performance for the 3D human pose estimation task when trained from scratch, and apply the design to all the tasks without additional adjustment.
Pretraining Strategies. We evaluate how different pretraining strategies influence the performance of downstream tasks. Starting from the scratch baseline, we apply the proposed strategies one by one. As shown in Table 4, a vanilla 2D-to-3D pretraining stage brings benefits to all the downstream tasks. Introducing corruptions additionally improves the learned motion embeddings. Unified pretraining with inthe-wild videos enjoy higher motion diversity, which further helps several downstream tasks.
Partial Finetuning. In addition to end-to-end finetuning, we experiment to freeze the motion encoder backbone and only train the downstream network for each subtask. In order to verify the effectiveness of the pretrained motion representations, we compare the pretrained motion encoder and a randomly initialized motion encoder. The results are shown in Table 5. It can be seen that based on the frozen pretrained motion representations, our method could still achieve competitive performance on multiple downstream tasks. The performance advantages over the random initialized ones demonstrate that generalizable motion representations are learned via pretraining. In real-world applications, it is often desirable to simultaneously obtain predictions for multiple sub-tasks (e.g. action + mesh). Existing methods require separate models to handle specific sub-tasks. Pretraining and partial finetuning make it possible for all the downstream tasks to share the same backbone, which largely reduces the computation overhead.

Conclusion
In this paper, we present MotionBERT to unify various sub-tasks related to human motion with the pretrain-finetune paradigm. We design the 2D-to-3D lifting pretext task for pretraining on large-scale human motion data. We also propose the DSTformer as a universal human motion encoder. Experimental results on multiple benchmarks demonstrate the effectiveness of our method. As for the limitations, by now this work mainly focuses on learning from the skeletons of a single person. Future work may explore fusing the learned motion representations with image features and explicitly model human interactions. Pretraining. For AMASS [63], we first render the parameterized human model SMPL+H [79], then extract 3D keypoints with a pre-defined regression matrix [9]. We extract 3D keypoints from Human3.6M [34] by camera projection [24]. We sample motion clips with length T = 243 for 3D mocap data. For 2D data, we utilize the provided annotations of PoseTrack [3]. We further include 2D motion from an unannotated video dataset InstaVariety [38] extracted by OpenPose [13]. Since the valid sequence lengths for in-the-wild videos are much shorter, we use T = 30 (PoseTrack) and T = 81 (InstaVariety). We set the input channels C in = 3 (x, y coordinates and confidence) following [28,107]. Random horizontal flipping is applied as data augmentation. The whole network is trained for 80 epochs with learning rate 0.0005 and batch size 64 using an Adam [40] optimizer. The weights of the loss terms are λ O = 20. We set the 2D skeleton masking ratio = 15%, same as BERT [26]. More specifically, we use 10% framelevel masks and 5% joint-level masks. We vary the proportion of the two types of masks and only observe marginal differences. We follow [16] to fit a mixture of distributions and sample per-joint noises from it. To keep the noise smooth and avoid severe jittering, we first sample the noise z ∈ R T K ×J for T K = 27 keyframes, then upsample it to z ∈ R T ×J , and add a small temporal gaussian noise N (0, 0.002 2 ).
3D Pose Estimation. The 2D skeletons are provided by 2D pose estimator trained on MPII [4] and Human3.6M [34] following the common practice [24,77]. For training from scratch, we train for 60 epochs with learning rate 0.0005 and batch size 32. For finetuning, we load the pretrained weights and train with learning rate 0.0003 and batch size 32.
Skeleton-based Action Recognition. We use the 2D skeleton sequences extracted with HRNet [90] provided by PYSKL [27]. We then upsample the skeleton sequences to a uniform length T = 243. For NTU-RGB+D, the DSTformer output after global average pooling is fed into an MLP including dropout p = 0.5, a hidden layer of 2048 channels, BatchNorm, and ReLU. We train for 200 epochs with learning rate 0.001 and batch size 32 for training from scratch. We set learning rate 0.0001 for the backbone, and learning rate 0.001 for the downstream MLP for finetuning. For oneshot recognition on NTU-RGB+D-120, we apply dropout p = 0.1, and use a linear layer to get action representation of size 2048. We train with batch size 16, and each batch includes 8 action pairs. Samples from the same action class are set as positives against the negatives from the remainder of the batch. We use the supervised contrastive loss [39] with temperature 0.1.
Human Mesh Recovery. For the Human3.6M dataset, we use the same 2D skeleton sequences as the 3D pose estimation task. The SMPL ground-truth (GT) parameters are fitted with SMPLify-X [75] provided by [70]. For the 3DPW dataset, we obtain its detected 2D skeleton sequences from DarkPose [113] provided by [23]. The SMPL GT parameters of 3DPW are obtained using IMUs [99]. For 3DPW benchmark, we use Human3.6M and COCO [50] datasets for training, following [23]. The pseudo-GT SMPL parameters of COCO are provided by [36]. 3DPW training data is used optionally according to different settings. Following [41,42], we use the 6D continuous rotation representations [119] instead of original axis angles when estimating pose parameters. The shape and pose MLPs are the same, including one hidden layer of 2048 channels, BatchNorm, and ReLU. We sample motion clips with length T = 81 and stride 27 for both datasets. We train the whole network end-to-end for 140 epochs with batch size 32, with learning rates 0.0001 for the backbone, and 0.001 for the downstream network, respectively.

A.2. Additional Ablation Studies
Finetune vs Scratch. We compare the training progress of finetuning the pretrained model and training from scratch. As Figure 3 shows, models initialized with pretrained weights enjoy better performance and faster convergence on all three downstream tasks. It indicates that the model learns transferable knowledge about human motion during pretraining, facilitating the learning of multiple downstream tasks.
Mask Ratio. We study the effect of different masking ratios on both pretraining and downstream finetuning. We measure the performance by MPJPE on the Human3.6M test set. Frame-level mask and joint-level mask are fixed to 2 : 1. As Figure 4 shows, when the masking ratio is less than 45%, the pretrained models could learn 2D-to-3D lifting well, and pretraining improves the finetuning performance (better than the scratch baseline). As we continue to increase the masking ratio, the pretrained models perform much worse at 2D-to-3D lifting, and pretraining is no longer beneficial for the finetuning performance. Our observation is slightly different from MAE [32] where a high masking ratio (75%) works well. One possible explanation is that  skeletons are highly abstract representations of human motion. A high masking ratio leaves too few cues to complete the whole sequence, therefore the model tends to overfit. It is also worth noting that in our pretraining task (2D-to-3D lifting), the depth channel ( 1 3 of original data) is intrinsically masked.