{'fraction_non_alphanumeric': 0.03815780741920499, 'fraction_numerical': 0.018216205454907427, 'mean_word_length': 4.7848368522072935, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Learning to Recognize 3D Human Action from A New Skeleton-based Representation Using Deep Convolutional Neural Networks

Recognizing human actions in untrimmed videos is an important challenging task. An effective 3D motion representation and a powerful learning model are two key factors influencing recognition performance. In this paper we introduce a new skeleton-based representation for 3D action recognition in videos. The key idea of the proposed representation is to transform 3D joint coordinates of the human body carried in skeleton sequences into RGB images via a color encoding process. By normalizing the 3D joint coordinates and dividing each skeleton frame into five parts, where the joints are concatenated according to the order of their physical connections, the color-coded representation is able to represent spatio-temporal evolutions of complex 3D motions, independently of the length of each sequence. We then design and train different Deep Convolutional Neural Networks (D-CNNs) based on the Residual Network architecture (ResNet) on the obtained image-based representations to learn 3D motion features and classify them into classes. Our method is evaluated on two widely used action recognition benchmarks: MSR Action3D and NTU-RGB+D, a very large-scale dataset for 3D human action recognition. The experimental results demonstrate that the proposed method outperforms previous state-of-the-art approaches whilst requiring less computation for training and prediction.


Introduction
Human Action Recognition (HAR) [1] is an important topic in machine vision. HAR methods have been widely applied in building electronic systems based on machine intelligence such as intelligent surveillance [2, 3], human-machine interaction [4,5], health care [6], and so on. Although significant progress has been made in the last few years, HAR is still a challenging task due to many obstacles, e.g. camera viewpoint, occlusions, or large intra-class variations [7].
There are two main types of intelligence that are artificial intelligence and machine intelligence. HAR can be considered as an electronic system based on machine intelligence. The first stage of any HAR system is data acquisition. Nowadays, a variety of electronic imaging systems can be used for this task such as traditional optical cameras, RGB-D sensors, Thermal Infrared (IR) sensors, and Synthetic Aperture Radar (SAR) sensors. For instance, IR sensors have been exploited in HAR [8]. These sensors are able to generate images based on the heat radiated by the human body and to work independently of lighting conditions. SAR imaging has also been widely used for detecting and analyzing human activities [9]. This type of sensor is able to operate far away from potential targets and functions during the daytime as well as nighttime, under all weather conditions. In particular, SAR Image Segmentation (SIS) techniques [10][11][12] can segment humans from other components (e.g. objects, background) in the scene. As a result, the use of SAR sensors offers big advantages for human action recognition, in particular for human target detection and identification in a complex environment such as in military applications.
Traditional studies on HAR mainly focus on the use of RGB sequences provided by 2D cameras. These approaches typically recognize actions based on hand-crafted local features such as Cuboids [13], HOG/HOF [14], HOG-3D [15], which are extracted from the appearance and movements of human body parts. However, it is very hard to fully capture and model the spatial-temporal information of human action due to the absence of 3D structure from the scene. Therefore, RGB-D cameras are becoming one of the most commonly-used sensors for HAR [16][17][18]. Recently, the rapid development of cost-effective and easy-to-use depth cameras, e.g. Microsoft Kinect TM sensor [19,20], ASUS Xtion PRO [21], or Intel R RealSense TM [22] has helped dealing with problems related to 3D action recognition. In general, depth sensors are able to provide detailed information about the 3D structure of human body. Therefore, many approaches have been proposed for 3D HAR based on data provided by depth sensors such as RGB sequences, depth, or combining these two data types (RGB-D). Furthermore, these devices have integrated real-time skeleton tracking algorithms [23] that provide high-level information about human motions in a 3D space. Thus, exploiting skeletal data for 3D HAR is an active research topic in computer vision [24][25][26][27][28].
In recent years, approaches based on Convolutional Neural Networks (CNNs) have achieved outstanding results in many image recognition tasks [29][30][31]. After the success of AlexNet [29], a new direction of research has been opened for designing and optimizing higher performing CNN architectures. As a result, there are substantial evidences that seem to show that the learning performance of CNNs may be significantly improved by increasing the number of hidden layers [32? , 33]. In the literature of HAR, many studies have indicated that CNNs have the ability to learn complex motion features better than hand-crafted approaches. However, most previous works have just focused on exploring simple CNNs such as AlexNet [29] and have not exploited the potential of recent state-of-the-art very deep CNN architectures (D-CNNs), e.g. Residual Networks (ResNet) [34]. In addition, most existing CNN-based approaches limit themselves to using RGB-D sequences as the input to learning models. Although RGB-D images are informative for understanding human action, the computation complexity of learning models increases rapidly when the dimension of the input features is large. Therefore, representation learning models based on RGB-D modality become more complex, slower and less practical for solving large-scale problems as well as real-time applications.
Different from previous works, to take full advantages of 3D skeletal data and the learning capacity of D-CNNs, this paper proposes an end-to-end deep learning framework for 3D HAR from skeleton sequences. We focus on solving two main issues: First, using a simple skeleton-to-image encoding method to transform the 3D coordinates of the skeletal joints into RGB images. The encoding method needs to ensure that the image-based representation of skeleton sequences is able to effectively represent the spatial structure and temporal dynamics of the human action. Second, we design and train different D-CNNs based on Residual Networks (ResNets) [34] -a recent state-of-the-art CNN for image recognition, to learn and classify actions from the obtained image-coded representations.
This paper is an extended version of our work published in the 8th International Conference of Pattern Recognition Systems (ICPRS 2017) [35]. It is part of our research project about investigating and developing a low-cost system based on RGB-D data provided by depth sensors for understanding human actions and analyzing their behaviors in indoor environments such as inside home, offices, residential buildings, buses, trains, etc. In general, three observations motivate our exploration of using ResNet [34] for 3D HAR from skeleton sequences, including: (1) human actions can be correctly represented as the movements of skeletons [27] and the spatialtemporal dynamics of skeletons can be transformed into 2D image structure, which can be effectively learned by D-CNNs; (2) skeletal data is high-level information with much less complexity than RGB-D sequences, this advantage makes our action learning model much simpler and faster; (3) many evidences [32,33] show that deeper convolutional model, especially ResNet [34] can boost the learning accuracy in image recognition tasks.
We evaluate the proposed method on two benchmark skeleton datasets (i.e. MSR Action3D [36] and NTU-RGB+D [37]). Experimental results confirmed the above statements since our method achieveds state-of-the-art performance compared with the existing results using the same evaluation protocols. Furthermore, we also indicate the effectiveness of our learning framework in terms of computational complexity.
In summary, two main contributions of this paper include: • First, we introduce a new skeleton-based representation (Sec. 3.1) and an end-to-end learning framework * (Sec. 3.2) based on D-CNNs to learn the spatio-temporal evolutions of 3D motions and then to recognize human actions; • Second, we show the effectiveness of our method on HAR tasks by achieving state-of-the-art performance on two benchmark datasets, including the most challenging skeleton benchmark currently available for 3D HAR [37] (Sec. 5).
The advantage of the proposed method is that it has high computational efficiency (Sec. 5.4). Moreover, this approach is general and can be easily applied to other time-series problem, e.g. recognizing human actions with mobile devices with integrated inertial sensors.
The rest of the paper is organized as follows: Section 2 discusses related works. In Section 3, we present the details of our proposed method, including the color-encoding process from skeletons to RGB images and deep learning networks. Datasets and experiments are described in Section 4. Experimental results are reported in Section 5 with a detailed analysis of computational efficiency. Finally, Section 6 concludes the paper with a discussion on the future work. * Codes and pre-trained models will be shared to the community at https://github.com/huyhieupham/ after publication.

Related Work
Skeleton-based Action Recognition (SBAR) using depth sensors has been widely studied in recent years. In this section, we briefly review existing SBAR methods, including two main categories that are directly related to our work. The first category is approaches based on hand-crafted local features of skeletons. The second is approaches based on deep learning networks, especially Recurrent Neural Networks with Long Short Term Memory (RNN-LSTMs).
Approaches based on hand-crafted local features: Human motion can be considered as a spatio-temporal pattern. Many researchers have built hand-crafted feature representations for SBAR and then used temporal models for modeling 3D human motion. For instance, Wang et al. [24] represented the human motion from skeletal data by means of the pairwise relative positions of the key joints. The authors then employed Hidden Markov Model (HMM) [38] for modeling temporal dynamics of actions. A similar approach has been presented by Lv et al. [39] where the 3D joint position trajectories have been mapped into feature spaces. The dynamics of actions was then learned by one continuous HMM. Xia et al. [25] proposed the use of histogram-based representation of human pose, and then actions were classified into classes by a discrete HMM. Wu and Shao [40] also exploited HMM for recognizing actions from high-level features of skeletons. Vemulapalli et al. [27] represented the 3D geometric interactions of the body parts in a Lie Group. These elements were processed with a Fourier Temporal Pyramid (FTP) transformation for modeling temporal evolutions of the original motions. Different from the studies above, Luo et al. [41] proposed a new dictionary learning model to learn the spatio-temporal information from skeleton sequences. Although promising results have been achieved from approaches based on hand-crafted local features and probabilistic graphical models, they have some limitations that are very difficult to overcome. E.g., most of these approaches are data-dependent and require a lot of hand-designing features. HMM-based methods require preprocessing input data in which the skeleton sequences need to be segmented or aligned. Meanwhile, FTP-based approaches can only utilize limited contextual information of actions.
Approaches based on deep learning: Recurrent Neural Networks with Long Short-Term Memory Network (RNN-LSTMs) [42,43] are able to model the contextual information of the temporal sequences as skeleton data. Thus, many authors have explored RNN-LSTMs for SAR. For instance, Du et al. [44] proposed an end-to-end hierarchical RNN-LSTM for modeling local motions of a body part in which all skeleton frames were divided into five parts according to the human physical structure. Each part of a skeleton was then fed into an independent RNN and then fused to be the inputs of higher layers. The final representation was used for the classification task. Zhu et al. [45] proposed a LSTM network with a mixed-norm regularization term to cost functions in order to learn the co-occurrence of discriminative features from skeletal data. Liu et al. [46] introduced a new gating mechanism with LSTM network to analyze the reliability of input skeleton sequences. In another study, Shahroudy et al. [37] used a part-aware LSTM network in which the memorycell was divided into sub-cells such that each sub-cell can model long-term contextual representation of a body part. Finally, all these sub-cells were concatenated to the final output. Although RNN-LSTMs are able to model the long-term temporal of motion and experimental results provided that RNN-LSTMs have outperformed many other approaches, RNN-based approaches just consider skeleton sequences as a kind of low-level feature by feeding directly the raw 3D joint coordinates carried in skeletons into the network input. The huge number of input parameters may make RNNs become very complex and easily lead to overfitting due to insufficient training data. Moreover, many RNN-LSTMs act as a classifier and cannot extract high-level features [47] for recognition tasks.
representations and learning geometric features carried in skeleton sequences with simple CNN architectures. In contrast, in this paper we concentrate on proposing a new and simple skeleton-based representation and exploiting the power of D-CNNs for action recognition. Our experiments on two public datasets, including the MSR Action3D dataset [36] and the NTU-RGB+D dataset [37] show state-of-the-art performance on both datasets. Moreover, in terms of computational cost, our measurement and analyses show that the proposed model is fast enough for many real-time applications.

Proposed Method
This section presents the proposed method. We first describe the color-encoding process that allows us to represent the spatial structure and temporal dynamics of skeleton sequences as the static structure of a 2D color image. We then review the main idea of ResNet [34] and propose five different ResNet configurations, including 20-layer, 32-layer, 44-layer, 56-layer, and 110-layer networks to learn motion features carried in the image-based representations and perform action classification.

Building image-based representation from skeleton sequences
Currently, skeletal data that contains 3D joint coordinates can be obtained from depth cameras via real-time skeleton estimation algorithms [50,51]. This technology allows to extract the position of the key joints in the body, which is suitable for 3D HAR problems. E.g., the latest version of the Microsoft Kinect TM sensor [19,20] (Kinect v2 sensor) can track the main 25 joints of human body in real-time speed as shown in Figure 1.  [37]. Skeleton sequences can be recorded frameby-frame at 30 FPS.
In this paper, instead of feeding raw 3D joint coordinates directly to RNN-LSTMs as in many previous works [37,[44][45][46], we propose the use of D-CNNs for learning motion features from skeleton sequences. The first step is to find a skeleton-based representation that can effectively capture the spatio-temporal evolutions of skeletons and that can also be easily learned by learning method as D-CNNs. One solution for this problem is to transform each skeleton sequence into a single color image (e.g., RGB image) in which the pixel values must have the ability to represent the movement of skeletons.
To this end, we transform the 3D joint coordinates of each skeleton into a new space by normalizing their coordinates using a transformation function. Specifically, given a skeleton sequence S with N frames [F 1 , F 2 , ..., F N ], let (x i , y i , z i ) be the 3D coordinates of each joint in frame {Fn} ∈ S, n ∈ [1, N ]. A normalization function F(·) is used to transform all 3D joint coordinates to the range of [0, 255] as follows: where (x i , y i , z i ) is 3D joint coordinate in the normalized space S . The max{C} and min{C} are the maximum and minimum values of all coordinates, respectively. To preserve the spatio-temporal information of skeleton movements, we stack all normalized frames according to the temporal order S = [F 1 , F 2 , ..., F N ] to represent the whole action sequence. These elements are quantified to RGB color space and can be stored as RGB images. In this way, we convert the skeletal data to 3D tensors that will can then be fed into deep learning networks as the input for feature leaning and classification. Figure 2 illustrates the skeleton-to-image transformation process.

Fig. 2:
Illustration of the data transformation process. Each skeleton sequence is encoded into a single color image that represents the spatio-temporal evolutions of the motion. Here, N denotes the number of frames in each sequence while K denotes the number of joints in each frame. The value of K depends on each RGB+D dataset. E.g., K is equal to 20 for MSR Action3D dataset [36]. Meanwhile, K is equal to 25 for NTU-RGB+D dataset [37]. Naturally, the human body is structured as four limbs and one trunk (see Figure 1). Simple actions can be performed through the movement of a limb, e.g., hand-waving, kicking forward, etc. More complex actions combine the movements of a group of limbs or the whole body, e.g., running or swimming. To keep the local characteristics of the human action [53,54], we divide each skeleton into five parts, including two arms (P 1 , P 2 ), two legs (P 4 , P 5 ), and one trunk (P 3 ). In each part, the joints are concatenated according their physical connections as shown in Figure 3. These parts are then connected in a sequential order: P 1 → P 2 → P 3 → P 4 → P 5 . The whole process of rearranging all frames in a sequence can be done by rearranging the order of rows of pixels in the color-based representation. This process is illustrated in Figure 4. Like that, we have encoded skeleton sequences into RGB images. Figure 5 shows some examples of RGB images obtained from input sequences of MSR Action3D dataset [36]. These images will be learned and classified by D-CNN models. This way, the original skeleton sequence will be recognized via the corresponding image.

Deep residual networks for skeleton-based human action recognition
In this section, we introduce our network designs for recognizing human action from image-based representation obtained in Section 3.1. Before that, in order to put our method into context, we will briefly review the key idea of ResNet [34].
3.2.1 Residual learning method: Very deep neural networks, especially D-CNNs have demonstrated to have a high performance on many visual-related tasks [32,55]. However, D-CNNs are very difficult to optimize due to the vanishing gradients problem. If the network is deep enough, the error signal from the output layer can be completely attenuated on its way back to the input layer. In addition, another problem called "the degradation phenomenon [56]" also impedes the convergence of deeper networks. More specially, adding more layers to a deep network can lead to higher training/testing error. To overcome these challenges, ResNet [34] has been introduced. The key idea behind ResNet architectures is the presence of shortcut connections between input and output of each ResNet building block. These shortcut connections are implemented by identity mappings and are able to provide a path for gradients to back propagate to early layers in the network. This idea improves the information flow in ResNets and helps them to learn faster. Not only that, experimental results on standard datasets such as CIFAR-10 [57], and ImageNet [58] for image classification tasks showed that the use of the shortcut connections in ResNet architectures can boost recognition performance. Figure 6 shows information flow in a ResNet building block and their implementation details.

Fig. 6: A building block of ResNet: (a) Information flow in a
ResNet building block; (b) Implementation details of a ResNet block introduced in the original paper [34]. The symbol + denotes element-wise addition.
Mathematically, a layer or a series of layers in a typical CNN model will try to learn a mapping function y = F(x) from input feature x. However, a ResNet building block will learn the function The additional element id(x) is the key factor that improves the information flow through layers during training ResNets.  [59] and non-linear activation function ReLU [60] are applied after each Conv. To reduce possible overfitting, we add a Dropout layer [61] with a rate of 0.5 into each ResNet block, located between two Conv layers and after ReLU. Finally, another ReLU layer [60] is used after the element-wise addition. The proposed ResNet unit is shown in Figure 7. We refer the interested reader to the Supplementary Materials to see details of the proposed network architectures.
In this learning framework, all networks are designed for accepting images with size 32 × 32 pixels and trained in an end-to-end manner using Stochastic Gradient Descent (SGD) algorithm [62] from scratch. The last fully-connected layer of the network represents the action class scores and its size can be changed corresponding to the number of action classes.

3.2.3
Learning to recognize actions from skeleton-based representations: In order to recognize an action from a given skeleton sequence S with N frames [F 1 , F 2 , ..., F N ], we firstly encode all frames of S into a single RGB image I RGB as mentioned in section 3.1 where all elements of F i|i∈[1,N ] were normalized to the range of [0, 255] as color pixels and rearranged according to their physical structure (see Figure 4). We then propose the use of ResNets to learn and classify the obtained color images. During the training phase, we minimize the cross-entropy loss function L X (y,ŷ) between the action labels y of I RGB and the predicted action labelsŷ by the network over the training set X . In other words, the network will be trained to solve the following problem where W is the set of weights that will be learned by the network, M denotes the number of samples in training set X and C is the number of action classes.

Experiments
In this section, we evaluate the effectiveness of the proposed method on two benchmark datasets: MSR Action3D [36] and NTU-RGB+D [37]. To have a fair comparison with state-of-the-art approaches in the literature, we follow the evaluation protocols as provided in the original papers [36,37]. Recognition performance will be measured by average classification accuracy. The NTU RGB+D data was collected by the Microsoft Kinect v2 TM sensor. Each skeleton frame provides the 3D coordinates of 25 body joints. Two different evaluation criteria have been suggested, including Cross-Subject and Cross-View. In the Cross-Subject settings, 20 subjects (with IDs: 1, 2, 4, 5, 8, 9, 13, 14, 15, 16, 17, 18, 19, 25, 27, 28, 31, 34, 35, and 38) are used for training and the rest are used for test. For Cross-View settings, the sequences provided by cameras 2 and 3 are used for training and sequences provided by camera 1 are used for test.

Data augmentation
Very deep neural networks as ResNets [34] require a lot of data to optimize. D-CNNs cannot learn and classify images effectively when training with an insufficient data amount. Unfortunately, we have only 557 skeleton sequences (less than 30 images per class) on MSR Action3D dataset [36]. Therefore, data augmentation has been applied to prevent ResNets from overfitting. We used random cropping, flip horizontally and vertically techniques to generate more training samples. More specifically, 8 × cropping has been applied on 40 × 40 images to create 32 × 32 images. Then, their horizontally and vertically flipped images are also created. Figure 8 illustrates some data augmentation techniques on an image sample obtained from the MSR Action3D dataset [36]. For the NTU-RGB+D dataset [37], we do not apply any data augmentation method due to its very large-scale.

Implementation details
The image-based representations are computed directly from the original skeletons without any data preprocessing technique, e.g., transforming the coordinate system, using a fixed number of frames, or removing noise. In this project, the MatConvNet toolbox [63] was used to design and train our deep learning networks. During the training phase, we use mini-batches of 128 images for ResNet-20, ResNet-32, ResNet-44, and ResNet-56 networks. For ResNet-110 network, we use mini-batches of 64 images. We initialize the weights randomly and train all networks for 120 epochs on a computer using Geforce GTX 1080 Ti GPU with 11GB RAM from scratch. The initial learning rate is set to 0.01 and is decreased to 0.001 at epoch 75.
The last 45 epochs use a learning rate of 0.0001. The weight decay is set at 0.0001 and the momentum is 0.9.

Result on NTU-RGB+D dataset
As shown in Table 4, the proposed learning framework reaches competitive results with an accuracy of 73.40% on Cross-Subject evaluation and an accuracy of 80.40% on Cross-View evaluation. The obtained results indicate that our method can deal with large intraclass variations and multiple viewpoints dataset as NTU-RGB+D [37]. Table 5 provides a comparison with published studies [36, 64-72, 74, 75, 79, 88]. It is clear that our method surpasses many  Table 3 Comparison with the state-of-the-art approaches on MSR Action3D dataset [36]. The best performances are in bold.
previous approaches in the same experimental conditions. The learning curves of all ResNet configurations on two evaluation settings are shown in Figure 9d and Figure 9e.  Table 4 Test accuracies (%) of our proposed networks on Cross-Subject and Cross-View settings [37]. The best results and configuration are in bold.

5.3
Convergence rate analysis Figure 9 shows the learning curves of the five proposed deep learning networks on the MSR Action3D [36] and the NTU-RGB+D [37] datasets. We can find that the training convergence rate of the networks are different on the two datasets. More specifically, on MSR Action3D dataset [36], the proposed networks exhibit rapid convergence after 50 epochs at the beginning of the training process. Meanwhile, these networks can only start to converge after near 80 epochs on the NTU-RGB+D dataset [37]. This phenomenon can be explained by the complexity of feature spaces generated by the two datasets. With more than 56,000 videos collected from 40 subjects for 60 action classes, it is clear that the NTU-RGB+D dataset [37] is more complex than the MSR Action3D dataset [36]. This leads to deep learning networks needing more iterations to start convergence. As shown in Table 2 and Table 4, the best recognition accuracy and configuration are highlighted in bold. More specifically, our experimental result on the MSR Action3D indicated that the baseline ResNet-32 achieved the best overall accuracy (99.53%). On the NTU RGB+D dataset, the ResNet-32 network was the best version on the Cross-View setting (80.40%). However, it worked worse than the ResNet-20 on the Cross-Subject setting. This is a weak aspect of the proposed approach. This limitation could be overcome by using ensemble learning techniques [90].

Analysis of training and prediction time
We take the AS1 subset of the MSR Action3D dataset [36] and the proposed ResNet-32 network for illustrating the computational efficiency of our deep learning framework. As shown in Figure 10, the proposed framework contains three main stages, including the encoding process from skeleton sequences into RGB images (Stage A), the supervised training stage (Stage B), and the prediction stage † More details about the GPU specification, please refer to: https://www.nvidia.com/en-us/geforce/products/ 10series/geforce-gtx-1080-ti/.
sequence (see Table 6). These results verify the effectiveness of our proposed method, not only in terms of accuracy but also in terms of computational efficiency.

Conclusion and future work
In this paper, we have proposed a new 3D motion representation and an end-to-end deep learning framework based on ResNets [34] for human action recognition from skeleton sequences. To this end, we transformed the 3D joint coordinates carried in skeleton sequences into RGB images via a color encoding process. We then designed and trained different D-CNNs based on the ResNet architecture for learning the spatial and temporal dynamics of human motion from image-coded representations. The experimental results on two wellestablished datasets, including the largest RGB-D dataset currently available, demonstrated that our method can achieve state-of-the-art performance whilst requiring a low computation cost for the training and prediction stages. For future work, we aim to extend this research by investigating a new skeleton encoding method, in which the Euclidean distances and orientation relations between joints will be exploited. In addition, we plan to build a better feature learning and classification framework by experimenting some new and potential architectures based on the idea of ResNet such as Inception Residual Networks (Inception-ResNets) [91] or Densely Connected Convolutional Networks (DenseNet [92]). In order to overcome the limitations of Kinect sensors when dealing with HAR in which the lighting conditions are available, we will also consider some new approaches for estimating human body key-points such as deep learning based approaches [93,94], KWE and KCE methods [10]. We hope that our study will open a new door for the computer vision community on exploiting the potentials of very deep convolutional neural networks and skeletal data for 3D human action recognition.