{'fraction_non_alphanumeric': 0.03825544326120871, 'fraction_numerical': 0.01929729363087567, 'mean_word_length': 4.807957455190073, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
TrajectoryCNN: A New Spatio-Temporal Feature Learning Network for Human Motion Prediction

Human motion prediction is an increasingly interesting topic in computer vision and robotics. In this paper, we propose a new 2D CNN based network, TrajectoryNet, to predict future poses in the trajectory space. Compared with most existing methods, our model focuses on modeling the motion dynamics with coupled spatio-temporal features, local-global spatial features and global temporal co-occurrence features of the previous pose sequence. Specifically, the coupled spatio-temporal features describe the spatial and temporal structure information hidden in the natural human motion sequence, which can be mined by covering the space and time dimensions of the input pose sequence with the convolutional filters. The local-global spatial features that encode different correlations of different joints of the human body (e.g. strong correlations between joints of one limb, weak correlations between joints of different limbs) are captured hierarchically by enlarging the receptive field layer by layer and residual connections from the lower layers to the deeper layers in our proposed convolutional network. And the global temporal co-occurrence features represent the co-occurrence relationship that different subsequences in a complex motion sequence are appeared simultaneously, which can be obtained automatically with our proposed TrajectoryNet by reorganizing the temporal information as the depth dimension of the input tensor. Finally, future poses are approximated based on the captured motion dynamics features. Extensive experiments show that our method achieves state-of-the-art performance on three challenging benchmarks (e.g. Human3.6M, G3D, and FNTU), which demonstrates the effectiveness of our proposed method. The code will be available if the paper is accepted.


Introduction
Pose prediction has became an increasingly interesting topic in the computer vision, which is widely applied to humancomputer interaction,video surveillance, etc (Chiu et al. 2019). The problem is very important in real applications. For example, if the robot can predict the future dynamic of humans, they can plan ahead and interact with humans to avoid dangers. As is shown in Figure 1, the goal of pose prediction is to predict future poses based on the observed poses. Moreover, the movement of joints position can efficiently and intuitively reflect the movement of the human body (Holden, Saito, and Komura 2016), and the 3D skeletal data can be acquired easily and cheaply (Han et al. 2013;Newell, Yang, and Jia 2016). Consistent with (Liu et al. 2019a), in this paper, we propose to predict future poses with a previous skeletal sequence in an on-going activity.
Coupled spatio-temporal modeling plays a key role to predict future poses (Butepage et al. 2017;Liu et al. 2019a;Xu et al. 2018). Two main methods of spatio-temporal modeling are based on RNN (Recurrent Neural Network) (Chiu et al. 2019;Martinez, Black, and Romero 2017;Gui et al. 2018b;Jain et al. 2016; Gopalakrishnan et al. 2019) and CNN (Convolutional Neural Networks) (Xu et al. 2018;Butepage et al. 2017;Shih et al. 2012;Liu et al. 2019a). RNN models have shown its power in temporal modeling, while it is inherently weak in spatial modeling. For example, (Martinez, Black, and Romero 2017) proposed a sequence-to-sequence architecture to predict future poses entirely based on GRU (Gated Recurrent Unit), which mainly focuses on capturing temporal information and ignores the spatial structure of the human body to some extent. CNN models have successfully used to predict human poses, while may fail to capture the spatio-temporal structure information well. For example, (Xu et al. 2018;Liu et al. 2019a) proposed a CNN based model to predict future poses, which can model the spatio-temporal information of previous frames. But the authors model the spatial and temporal information separately, which may fail to capture the spatio-temporal structure information. However,the spatial and temporal evolution in the on-going activity is an organic whole, which indicates that the spatial and temporal information are coupled in essence. Therefore, this motivates us to explore a new efficient spatio-temporal network to predict future pose sequence.
Another important aspect is the co-occurrence long-term temporal modeling, which is critical to predict more accurate poses. Most of the existing works either model the long-term temporal information of previous frames (Xu et al. 2018;Liu et al. 2019a;2019b) or model the co-occurrence spatial information of the human pose (Li et al. 2018a;Wang et al. 2016), which may ignore the co-occurrence long-term modeling of the previous sequence. For example, (Xu et al. 2018;Liu et al. 2019a;2019b) proposed to capture the longterm information hierarchically, but it is difficult to capture the co-occurrence temporal information of the previous sequence. Zhu et al.  and Li et al. (Li et al. 2018a) proposed to model the co-occurrence spatial information to characterize actions, which ignores the cooccurrence temporal information of the motion sequence. Therefore, we propose to model the global co-occurrence temporal information of previous frames.
To model the coupled spatio-temporal and long-term cooccurrence temporal information, a novel trajectorylet descriptor is introduced to describe the static and dynamic information of the previous sequence. Specifically, local spatial features can be extracted by convolutional filters, and global co-occurrence temporal features can be captured by aggregating the frame-level features gradually. The similar descriptor proposed in (Qiao et al. 2017) describes the local static and dynamic information of joint trajectories within a short interval by manual features (i.e. positions, displacement and velocity of joints), but it may be difficult to capture the co-occurrence long-term temporal pattern of pose sequence. To generate the trajectorylet descriptor and predict future poses, a new two-step network, Trajectorylet-Net, is proposed entirely based on CNN. The first step aims to generate trajectorylet descriptors that characterizes the local spatial and global co-occurrence temporal information of the previous sequence. The second step aims to predict future poses by comprehensively aggregating trajectorylet descriptors automatically. Finally, we evaluate the performance of the proposed framework on G3D (Bloom, Makris, and Argyriou 2012) and FNTU (Liu et al. 2019a), and achieve state-of-the-art results.
In summary, our main contributions are as follows:1) We propose a new coupled spatio-temporal descriptor, trajectorylet, which can model the local spatial information and global co-occurrence temporal information simultaneously. 2) We propose a new simple but efficient two-step network, Trajectorylet-Net, entirely based on CNN, which can extract trajectorylet descriptors and predict future poses end-to-end. 3) Experimental results show that the proposed method achieves state-of-the-art performance on two benchmark datasets, G3D and FNTU, which demonstrates the effectiveness of our proposed method.

Related Work
In this section, we review related works from two perspectives: the spatio-temporal modeling of sequence-tosequence models and the long-term modeling of pose prediction. The spatio-temporal modeling of sequence-to-sequence models: most of the sequence-to-sequence models capture the spatio-temporal information of previous frames using RNN model (Chiu et al. 2019;Martinez, Black, and Romero 2017;Gui et al. 2018b;Fragkiadaki et al. 2015;Jain et al. 2016;Gopalakrishnan et al. 2019;Liu et al. 2019b;Kundu, Gor, and Babu 2019;Wang and Feng 2019;Guo and Choi 2019). For example, (Martinez, Black, and Romero 2017) and (Gui et al. 2018b) proposed to predict future dynamics using a GRU-based encoder-decoder network. The authors model the spatial and temporal information of previous poses with GRU cells, and predict future poses using GRU cells followed by a linear layer. Although RNN models have shown its power in temporal modeling, but it is relatively weak in spatial modeling. Therefore, many works were proposed to address the spatio-temporal modeling of sequence-to-sequence models using CNN models (Cho and Foroosh 2018;Butepage et al. 2017;Vukotić et al. 2017;Pavllo et al. 2019;Li et al. 2018b). There are two schemes to model the spatio-temporal information: modeling the spatial and temporal information separately, and modeling the spatial and temporal equally. For example, (Cho and Foroosh 2018) proposed to model the spatial and temporal information of previous frames separately. In detail, the author first model the spatial information using VGG-16, and then use T-CNN (Temporal Convolutional Neural Network) to model the dynamics information of previous frames. (Butepage et al. 2017) and (Li et al. 2018b) proposed to simultaneously model the spatial and temporal information using CNN and fully connected networks respectively, which may take the spatial and temporal information equally and ignore some spatio-temporal information. The long-term modeling of pose prediction: it is very important to capture the long-term dynamic of previous poses to predict accurate future poses. However, most of the existing pose prediction models are proposed based on RNN (such as LSTM, GRU, i.e.), which is inherently hard to capture the long-term dependency of previous frames using its recurrent unit (Martinez, Black, and Romero 2017;Gui et al. 2018b). Some works were proposed to model the long-term dynamic of previous poses hierarchically (Chiu et al. 2019;Liu et al. 2019b;Xu et al. 2018;Liu et al. 2019a). For example, (Liu et al. 2019b) proposed to predict future human poses using a hierarchical motion recurrent (HMR) model that can simultaneously capture the local the global temporal context of previous poses, which ignores the spatial information of previous poses. Besides, (Xu et al. 2018) and (Liu et al. 2019a) proposed to encode the global temporal information of previous poses using a CNN based block (CMU) (Xu et al. 2018). Although these hierarchical models can capture the global temporal information of all input frames, similar to the commonly used RNN-based models, it is difficult to capture the co-occurrence temporal information of all previous frames. Moreover, other works can model the global temporal evolution of all history poses (Butepage et al. 2017;Li et al. 2019;Pavllo et al. 2019). (Butepage et al. 2017) proposed to capture the temporal information of all input frames using multiple fully connected layers. (Li et al. 2019) proposed to model the long-term information of previous frames by mapping the input sequence into deep features using autoencoder (P. Kingma and Max 2014). (Butepage et al. 2017) and (Li et al. 2019) treated the spatial and temporal equally, which may ignore the strong temporal correlation of all input poses. Recently, some works were proposed to model the temporal information using dilated convolutions. For example, (Pavllo et al. 2019) proposed a Quater-Net to predict future human motion by modeling the longterm temporal information of input sequence hierarchically using dilated convolutions. Although great success has been made in long-term modeling, rare works consider the cooccurrence temporal information of all joints of the previous skeletal trajectory. Therefore, in this work, we proposed to model the co-occurrence temporal information.

Methodology Problem formulation
Pose sequence can be represented by a trajectory of a set of 3D joints coordinate. Given a pose sequence S = {p 1 , p 2 , · · · , p N1 } with length N 1 and a future pose sequence S = { p 1 , p 2 , · · · , p N2 } with length are the i 1 th pose of S and i 2 th pose of S, respectively, is the k 2 th joint of the i 2 th pose of S, J is the number of joints. Then pose prediction can be formulated as a mapping S → S from the previous pose sequence to the future pose sequence.
Most of the existing methods were commonly proposed based on the Encoder-Decoder framework. The encoder was usually used to encode previous frames into an intermediate representation that characterizes the spatio-temporal information of the previous pose sequence, and the decoder was used to restore the spatial and temporal information of future poses. Therefore, the pose prediction can be considered as a two-step problem, which can reformulate as: S → T → S where T is an intermediate representation that characterizes the spatio-temporal information of S. In this paper, we introduce a novel descriptor T , trajectorylet, to learn the intermediate representation of the previous pose sequence. The first step is to learn the mapping φ(·): S → T from the previous pose sequence to the intermediate representation; the second step is to learn the mapping ψ(·): T → S from the intermediate representation to the future pose sequence. We will describe these two steps in detail in the following section.

Trajectorylet descriptor
If one movement pattern occurs, another special movement pattern may occur with a high probability. For example, as is shown in Figure 1, if sequence S 1 occurs, sequence S 2 may occur later. To capture these co-occurrence movement patterns, a novel descriptor, trajectorylet, is introduced to simultaneously model the spatial structure of the human pose and the long-term temporal dependency of the previous pose sequence. Due to the complexity and randomness of the human motion, one trajectorylet may only capture one special spatio-temporal evolution, which can not characterize the pose sequence well. Therefore, as is shown in Figure 2, we propose to use multiple trajectorylet descriptors to describe the same pose sequence, which can capture the rich spatiotemporal context of the pose sequence better.
Given the previous sequence S = {p 1 , p 2 , · · · , p N1 }, the trajectorylet descriptors T = {t j } n j=1 can be formulated as: Where t j is the jth trajectorylet descriptor, n is the number of trajectorylet descriptors. φ(·) is a mapping from the previous sequence to the trajectorylet descriptors, which can be learned using our proposed network in the following section.

Trajectorylet-Net
In this section, a new two-step, Trajectorylet-Net, is proposed to predict future poses shown in Figure 3. The first step is the trajectorylet generator, which aims to find a mapping φ(·) from the input sequence to the trajectorylet descriptors; the second step is the pose generator, which aims to find a mapping ψ(·) from the trajectorylet descriptors to the future poses. Inputs and outputs processing: In this paper, we use the skeletal representation proposed in (Liu et al. 2019a). Given a pose sequence S = {p 1 , p 2 , · · · , p N1 } with length N 1 , the pose sequence can be represented by a tensor X = [p 1 ; p 2 ; · · · , p l , · · · , p N1 ] shown in equation 2. Here, p l is the lth pose of the sequence S. And the tensor shape is with N 1 × J × D. Where J is the number of joints, D is the dimension of each joint.
x l,J y l,J z l,J     , l = 1, 2, · · · , N 1 (2) 1) Inputs processing: Inspired by (Li et al. 2018a), we propose to transpose the temporal dimension N 1 in the last channel that all poses of the previous sequence can be covered by the convolutional filter, which can conveniently model the global co-occurrence temporal features of previous poses.
2) Outputs processing: Finally, an additional transpose operation is applied to recover the skeletal frames. Trajectorylet generator: Inspired by the symmetric topology of the human body, a symmetric trajectorylet generator is proposed to generate trajectorylet descriptors that aims to find a mapping φ(·) by mining trajectorylet descriptors from the input sequence as is shown in Figure 3, which mainly consists of five convolutional layers. Inspired by (Xie et al. 2018;Wang et al. 2018), a filter with size (n × n × N ) can be divided into two filters: (n × n × 1) and (1 × 1 × N ). Similarly, as is shown in the right of Figure 2, the process of trajectorylet generator can be considered as two parts: local structure modeling and co-occurrence temporal modeling.
1) Local structure learning of the human body: From the perspective of the spatial modeling, the horizontal axis of the input tensor represents by D (dimension of each joint), and the vertical axis of the input tensor represents by J (the number of joints). As is shown in the top right of Figure 2, when the filter size n > 1, we can model the local structure of the human body by covering multiple adjacent joints with a convolutional filter.
2) Global co-occurrence temporal modeling of the previous skeletal sequence: From the perspective of the temporal modeling, the horizontal axis of the input tensor represents by J (the number of joints) or D (dimension of each joint), and the vertical axis of the input tensor represents by N (the length of input sequence). As is shown in the bottom right of Figure 2, when the filter size is 1 × N (N is the length of the input sequence), the convolutional filter covers x, y or z of each joint of all inputs poses. Therefore, we can conveniently model the co-occurrence temporal features of input poses automatically.
To sum up, we use convolutional filters to model the local structure information of the human body, and gather the pose-level features gradually to capture the global cooccurrence temporal information of the previous pose sequence. To capture rich spatio-temporal contexts of the previous sequence, different filters are applied to generate different trajectorylet deacriptors, which can characterize vary spatio-temporal evolutions of the previous sequence.
Pose generator: Pose generator aims to find a mapping ψ(·) to generate future poses from trajectorylet descriptors, which can be formulated as equation 3. To achieve this, we use a convolutional layer to learn this mapping, which can generate future poses by aggregating different trajectorylet descriptors gradually. Finally, one additional convolutional layer with a 1 × 1 convolutional filter is used to smooth the trajectory of future poses. p i = ψ(t 1 , t 2 , · · · , t n ), i = 1, 2, · · · , N 2 (3)

Experiments
In this section, we first introduce the datasets used in our experiments and implementation details of our work. Then we compare our method with state-of-the-arts. Next we carry out some ablative experiments to analyze the contributions of the proposed method. Next we show the generalization of our proposed method. Finally we visualize the predictive performance of our model. In experiments, all models are implemented by TensorFlow. We use fewer channels in a lower layer, and more channels in the deeper layer. Moreover, we find that more channels may improve the performance of our network. Considering the efficiency of the network, we design a symmetric network by using 300, 500, 1000, 500, 300 channels at each layer, respectively. Since the joints value can be negative, positive and zero, Leaky ReLU is selected as our activation function. All models are trained with Adam optimizer, and the learning rate is initial with 0.0001. We use MSE (Mean Squared Error) and MAE (Mean Absolute Error) for per sequence as our metrics as described in (Liu et al. 2019a).

Comparison with state-of-the-arts
To evaluate the effectiveness of our proposed method, we compare it with state-of-the-art methods ( PredCNN (Xu et  (1) From the perspective of the accuracy: Compared with state-of-the-art methods, our model reaches the best performance, which demonstrates the effectiveness of our proposed methods. For example, compared with (Liu et al. 2019a), the MSE and MAE decrease by 0.0194 and 0.1270, respectively. The possible reason is that our model can simultaneously capture the local structure of the human body and global co-occurrence temporal features of previous poses, while (Xu et al. 2018) modeled the spatial and temporal information separately.
(2) From the perspective of the efficiency: Compared with (Xu et al. 2018) and (Liu et al. 2019a), the training time and test time of our model are significantly decreased, which demonstrates that our model better meets the real-time requirements. Because our model is very simple that mainly consists of six convolutional layers and two transpose operations, while (Xu et al. 2018) and (Liu et al. 2019a) are more complex with many building blocks. Compared with (Butepage et al. 2017), the efficiency of our model is slight less efficient but still meets the requirement of real-time.
To verify the predictive performance at each time-step, the frame-wise performance is evaluated. As is shown in Figure  4, our method outperforms all the baselines on both MAE and MSE, especially for the long-term prediction, which demonstrates the effectiveness of our proposed method. Different from other methods, our method can capture the global co-occurrence temporal information automatically. This may be the reason that our model can achieve higher performance at a later time step. Results on FNTU: to further show the performance on a larger dataset, we evaluate our proposed method on FNTU, and quantitative results reported in Table 2, which is consistent with the results on G3D. Compared with (Liu et al. 2019a), our error is slightly smaller, but the training time and the test time are significantly decreased, which shows that our method can simultaneously achieve accuracy and efficiency. Comparing the experimental results of Table 1 and  Table 2, the accuracy and efficiency of our model are more stable on the larger dataset, which is different from other methods. On one hand, the performance of our method surpasses state-of-the-art methods, which further demonstrates 1 Original PredCNN is used in video prediction, we think it can also be used to predict human poses. Therefore,we name it PredCNN in this paper   (1) our model can capture spatio-temporal features well with our proposed trajectorylet descriptors; (2) the complexity of our network is relatively low.
The frame-wise performance on FNTU is reported in Figure 5. Our method outperforms state-of-the-art methods on both MSE and MAE, which is consistent with the quantitative results reported in Table 1. This further demonstrates the effectiveness of our proposed method.

Ablation analysis
In this section, we verify our network from these aspects: architecture, spatio-temporal modeling, activation function. Evaluation of architecture: There are two interesting findings: (1) following the common setting in the convolutional network, using fewer filters in the lower layer and more filters in the higher layer can significantly improve the efficiency of the network, but the performance will not decrease too much. (2) Increasing the filer number at each layer can improve the accuracy of the network. To verify this, we carry out three experiments: 1) the channel number of each layer is set to 1000. 2) The channel number of each layer is set to 300, 500, 1000, 500, 300, respectively. 3) The channel number of each layer is set to 100, 300, 600, 300, 100, respectively. Experimental results are given in Table 3. #1 and #2 show the effectiveness of finding (1). The MSE and MAE of #2 are slightly worse than #1 on G3D, while the MSE and MAE of #2 are slightly better than #1 on FNTU. Compared with #1, the training time and test time of #2 are halved on both datasets. Therefore, experimental results of #1 and #2 can support the finding (1), which can explain the rationality of our network setting. #2 and #3 show the effectiveness of the finding (2). When decreasing the channel numbers, the accuracy is decreased, but the training time and test time are less. Therefore, consideration from accuracy and efficiency, our final network is set as #2. Evaluation of spatio-temporal modeling: To demonstrate the importance of simultaneously model the spatial and temporal information and the effectiveness of global cooccurrence temporal information, we carry out three experiments reported in Table 4. Specifically, #1 and #3 show the importance of simultaneously model the spatial and temporal information; #2 and #3 show the effectiveness of global co-occurrence temporal information. In #1, the filter size is set to 1, which can only capture the global co-occurrence temporal (CoT) information; in #2, we remove the transpose operation in the Trajectorylet-Net and set the filter size to 3, which can capture the local spatial (LoS) and local temporal (LoT) information; in #3, the filter size is set to 3, which can model the local spatial and global co-occurrence temporal (CoT) information. Compared with #1, the MSE of #3 decreases by 0.0555 and 0.0418 on G3D and FNTU, respectively, and the MAE of #3 decreases by 0.1010 and 0.0840 on G3D and FNTU, respectively, which demonstrates that simultaneously model the spatial and temporal features are critical for the network performance. Compared with #2, #3 achieves the best performance on both datasets, which demonstrates the effectiveness of global cooccurrence temporal modeling. Evaluation of activation function: For our problem, we find that different activation functions affect the network performance differently. Therefore, in this section, we evaluate the performance of sigmoid, tanh, ReLU and Leaky ReLU, respectively. And experimental results are shown in Table 5. The performance of Leaky ReLU achieves the best results on both MSE and MAE on two datasets, while the sigmoid function gains the worse performance. The possible reasons are: (1) For Leaky ReLU, when the value is negative, it will be changed to a larger negative number. (2) For ReLU, when the value is negative, it will be set to 0. (3) For tanh, the small negative value will be changed to a smaller number; the large positive value will be changed to a larger number. (4) For sigmoid, all the values will be changed to between 0 and 1. However, the joints value of the human body can be negative, zero and positive (it may be larger than 1). Therefore, Leaky ReLU is more suitable for our problem, while sigmoid is unsuitable. Finally, we select Leaky ReLU as the activation function in our final network.

Evaluation of generalization
To show the generalization performance of our proposed method, we pre-trained our model on FNTU, and directly test on G3D. As is shown in Table 6, our method achieves the best performance on unseen data. For example, compared with the best prior method, the MSE and MAE of our network decrease by 0.0129 and 0.1221 respectively. The possible reason is that our model simultaneously considers global co-occurrence temporal information and coupled spatio-temporal information, which can capture the general representation of the previous poses better.

Visualization of experimental results
To show the qualitative performance of our proposed network, we also visualize the predictive results frame-byframe in Figure 6. On G3D, our method achieves the best visualization performance overall. For example, for the top right group sequence, our predictive sequence is more similar to the groundtruth, which shows the best performance both at short-term and long-term predictions. For the bottom left group sequence, our predictive pose sequence seems more reasonable, especially at the long-term prediction. This may be the reason that our model can handle the global cooccurrence temporal information and consider the spatiotemporal correlation of the previous sequence. On FNTU, our method achieves a similar performance. In general, our visualization performance is slightly better than state-of-thearts, which is consistent with the quantitative results shown in Table 2. For example, for the bottom left group sequence, our performance seems more similar to the groundtruth, especially for the long-term prediction. For the top left group sequence, compared with S-TE, the temporal evolution of our predictive poses is more plausible, such as the hands of the human body, which demonstrates that our network can capture the spatio-temporal information well. The possible reason is that S-TE uses fully connected layer to model the spatial and temporal information equally, which may ignore the difference of spatial and temporal information and lose some spatio-temporal structure information in some extent, while our method models the local spatial information and global co-occurrence temporal information together, which is different from S-TE.

Conclusion
In this work, we propose an efficient spatio-temporal network, Trajectorylet-Net, to predict future poses, which can model the local spatial and global temporal information of the previous pose sequence. Different from prior works, we    introduce a novel descriptor, trajectorylet, to simultaneously capture the coupled spatio-temporal information and longtern temporal dependency of previous poses. Evaluations are carried out on two benchmark datasets, and our method achieves state-of-the-art performance on both datasets. Experiments show that simultaneously model the spatial and temporal information are critical to the final performance of the network. However, the proposed network ignores global spatial information, which is important to predict more plausible poses. In the future, we plan to add the global spatial information of the human body to Trajectorylet-Net to reach a more accurate prediction.