{'fraction_non_alphanumeric': 0.039767007047317705, 'fraction_numerical': 0.01813965194879908, 'mean_word_length': 5.099177631578947, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Human Action Recognition from Various Data Modalities: A Review

Human Action Recognition (HAR) aims to understand human behavior and assign a label to each action. It has a wide range of applications, and therefore has been attracting increasing attention in the field of computer vision. Human actions can be represented using various data modalities, such as RGB, skeleton, depth, infrared, point cloud, event stream, audio, acceleration, radar, and WiFi signal, which encode different sources of useful yet distinct information and have various advantages depending on the application scenarios. Consequently, lots of existing works have attempted to investigate different types of approaches for HAR using various modalities. In this paper, we present a comprehensive survey of recent progress in deep learning methods for HAR based on the type of input data modality. Specifically, we review the current mainstream deep learning methods for single data modalities and multiple data modalities, including the fusion-based and the co-learning-based frameworks. We also present comparative results on several benchmark datasets for HAR, together with insightful observations and inspiring future research directions.


INTRODUCTION
H UMAN Action Recognition (HAR), i.e., recognizing and understanding human actions, is crucial for a number of real-world applications. It can be used in visual surveillance systems [1] to identify dangerous human activities, and autonomous navigation systems [2] to perceive human behaviors for safe operation. Besides, it is important for a number of other applications, e.g., video retrieval [3], human-robot interaction [4], and entertainment [5].
In the early days, most of the works focused on using RGB or gray-scale videos as input for HAR [6], due to their popularity and easy access. Recent years have witnessed an emergence of works [7], [8], [9], [10], [11], [12], [13], [14], [15], [16] using other data modalities, such as skeleton, depth, infrared sequence, point cloud, event stream, audio, acceleration, radar, and WiFi for HAR. This is mainly due to the development of different kinds of accurate and affordable sensors, and the distinct advantages of different modalities for HAR depending on the application scenarios.
Specifically, according to the visibility, data modalities This work is supported by National Research Foundation, Singapore under its AI Singapore Programme (AISG Award No: AISG-100E-2020-065), and SUTD SRG. This work is also supported by TAILOR, a project funded by EU Horizon 2020 research and innovation programme under GA No 952215.
This manuscript has been accepted by IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) -DOI: 10.1109/TPAMI.2022.3183112. This is the up-to-date version (updated in June 2022). We plan to update this arXiv version yearly to cover the latest advances in the field of human action recognition.
can be roughly divided into two categories, namely, visual modalities and non-visual modalities. As shown in Table  1, RGB, skeleton, depth, infrared sequence, point cloud, and event stream are visually "intuitive" for representing human actions, and can be seen as visual modalities. Generally, visual modalities are very effective for HAR. Among them, RGB video data is the most common data type for HAR, which has been widely used in surveillance and monitoring systems. Skeleton data encodes the trajectories of human body joints. It is succinct and efficient for HAR when the action performing does not involve objects or scene context. Point cloud and depth data, which capture the 3D structure and distance information, are popularly used for HAR in robot navigation and self-driving applications. In addition, infrared data can be utilized for HAR in even dark environments, while the event stream keeps the foreground movement of the human subjects and avoids much visual redundancy, making it suitable for HAR as well.
Meanwhile, audio, acceleration, radar, and WiFi, etc. are non-visual modalities, i.e., they are not visually "intuitive" for representing human behaviors. Nevertheless, these modalities can also be used for HAR in some scenarios which require the protection of privacy of subjects. Among them, audio data is suitable for locating actions in the temporal sequences, while acceleration data can be adopted for fine-grained HAR. Besides, as a non-visual modality, radar data can even be used for through-wall HAR. More details of different modalities are discussed in Section 2.
Single modality-based HAR has been extensively investigated in the past decades [6], [17], [18], [19], [20], [21], [22]. However, since different modalities have different strengths and limitations for HAR, the fusion of multiple data modalities and the transfer of knowledge across modalities to enhance the accuracy and robustness of HAR, have also received great attention recently [23], [24]. More specifically, fusion means the combination of the information of two or more modalities to recognize actions. For example, audio arXiv:2012.11866v5 [cs.CV] 21 Jun 2022 data can serve as complementary information of the visual modalities to distinguish the actions "putting a plate" and "putting a bag" [25]. Besides fusion, some other methods have also exploited co-learning, i.e., transferring knowledge across different modalities to strengthen the robustness of HAR models. For example, in [26], the skeleton data serves as an auxiliary modality enabling the model to extract more discriminative features from RGB videos for HAR.
Considering the significance of using different single modalities for HAR and also leveraging their complementary characteristics for fusion and co-learning-based HAR, we review existing HAR methods from the perspective of data modalities. Specifically, we review the mainstream deep learning architectures that use single data modalities for HAR, and also the methods taking advantage of multiple modalities for enhanced HAR. Note that in this survey, we mainly focus on HAR from short and trimmed video segments, i.e., each video contains only one action instance. We also report and discuss the benchmark datasets. The main contributions of this review are summarized as follows.
(1) To the best of our knowledge, this is the first survey paper that comprehensively reviews the HAR methods from the perspective of various data modalities, including RGB, depth, skeleton, infrared sequence, point cloud, event stream, audio, acceleration, radar, and WiFi.
(2) We comprehensively review the multi-modality-based HAR methods, and categorize them into two types, namely, multi-modality fusion-based approaches and crossmodality co-learning-based approaches.
(3) We focus on reviewing the more recent and advanced deep learning methods for HAR, and hence provide the readers with the state-of-the-art approaches. (4) We provide comprehensive comparisons of existing methods and their performance on several benchmark datasets (e.g., Tables 2,3,4,5), with brief summaries and insightful discussions.
The rest of this paper is organized as follows. Section 2 reviews HAR methods using different single data modalities. Section 3 introduces multi-modality HAR methods. The benchmark datasets are listed in Section 4. Finally, potential future development of HAR is discussed in Section 5.

RGB MODALITY
The RGB modality generally refers to images or videos (sequences of images) captured by RGB cameras which aim to recreate what human eyes see. RGB data is usually easy to collect, and it contains rich appearance information of the captured scene context. RGB-based HAR has a wide range of applications, such as visual surveillance [1], autonomous navigation [2], and sport analysis [36]. However, action TABLE 1 Action samples of different data modalities (with pros and cons).

Modality Example Pros Cons
Visual Modality

RGB
Hand-waving [27] · Provide rich appearance information · Easy to obtain and operate · Wide range of applications · Sensitive to viewpoint · Sensitive to background · Sensitive to illumination 3D Skeleton Looking at watch [28] · Provide 3D structural information of subject pose · Simple yet informative · Insensitive to viewpoint · Insensitive to background · Lack of appearance information · Lack of detailed shape information · Noisy

Depth
Mopping floor [29] · Provide 3D structural information · Provide geometric shape information · Lack of color and texture information · Limited workable distance Infrared Sequence Pushing [30] · Workable in dark environments · Lack of color and texture information · Susceptible to sunlight Point Cloud Bending over [31] · Provide 3D information · Provide geometric shape information · Insensitive to viewpoint · Lack of color and texture information · High computational complexity Event Stream Running [32] · Avoid much visual redundancy · High dynamic range · No motion blur · Asynchronous output · Spatio-temporally sparse · Capturing device is relatively expensive

Non-visual Modality
Audio Audio wave of jumping [33] · Easy to locate actions in temporal sequence · Lack of appearance information Acceleration Acceleration measurements of walking [34] · Can be used for fine-grained HAR · Privacy protecting · Low cost · Lack of appearance information · Capturing device needs to be carried by subject Radar Spectrogram of falling [35] · Can be used for through-wall HAR · Insensitive to illumination · Insensitive to weather · Privacy protecting · Lack of appearance information · Capturing device is relatively expensive WiFi CSI waveform of falling [35] · Simple and convenient · Privacy protecting · Low cost · Lack of appearance information · Sensitive to environments · Noisy recognition from RGB data is often challenging, owing to the variations of backgrounds, viewpoints, scales of humans, and illumination conditions. Besides, RGB videos have generally large data sizes, leading to high computational costs when modeling the spatio-temporal context for HAR. In this subsection, we review the methods which use the RGB modality for HAR. Specifically, since videos contain the temporal dynamics of human motions that are often crucial for HAR, most of the existing works focused on using videos to handle HAR [37], and only a few approaches utilized static images [38], [39], [40]. Therefore, we focus, in the following, on reviewing RGB video-based HAR methods.
In the pre-deep learning era, many hand-crafted featurebased approaches, including the space-time volume-based methods [41], space-time interest point (STIP)-based methods [42], and trajectory-based methods [43], were welldesigned for RGB video-based HAR. Recently, with the great progress of deep learning techniques, various deep learning architectures have also been proposed. Due to the strong representation capability and superior performance of deep learning-based methods, current mainstream research in this field focuses on designing different types of deep learning frameworks. Consequently, we review, in the following, the advanced deep learning works for RGB-based HAR, which can be mainly divided into four categories, namely, two-stream 2D Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), 3D CNN, and Transformer-based methods, as shown in Table  2. In particular, Section 2.1.1 mainly reviews the two-stream methods and their extensions (e.g., multi-stream architectures), which use 2D CNNs as their backbone models. Section 2.1.2 mainly reviews RGB-based HAR methods which mainly employ RNN models together with 2D CNNs as feature extractors. In Section 2.1.3, we review the 3D CNNbased methods and their extensions. Section 2.1.4 introduces recent advances in Transformer-based methods for HAR.

Two-Stream 2D CNN-Based Methods
As the name suggests, the two-stream 2D CNN framework generally contains two 2D CNN branches taking different input features extracted from the RGB videos for HAR, and the final result is usually obtained through fusion strategies, as shown in Fig. 1(a). In this section, we review the classic two-stream methods [44], [45] and also their extensions [46].
As a classic two-stream framework, Simonyan and Zisserman [44] proposed a two-stream CNN model consisting of a spatial network and a temporal network. More specifically, given a video, each individual RGB frame and multi-frame-based optical flows were fed to the spatial stream and temporal stream, respectively. Hence, appearance features and motion features were learned by these two streams for HAR. Finally, the classification scores of these two streams were fused to generate the final classification result. In another classic work, Karpathy et al. [45] fed lowresolution RGB frames and high-resolution center crops to two separate streams to speed up the computation. Different fusion strategies were investigated to model the temporal dynamics in videos. Several studies endeavored to extend and improve over these classic two-stream CNNs, and they are reviewed below.
To produce better video representations for HAR, Wang et al. [47] fed multi-scale video frames and optical flows to a two-stream CNN to extract convolutional feature maps, which were then sampled over the spatio-temporal tubes centered at the extracted trajectories. Finally, the resulting features were aggregated using Fisher Vector representation [48] followed by a SVM for HAR. Chéron et al. [49] used the positions of human body joints to crop multiple human body parts from the RGB and optical flow images, which were passed through a two-stream network for feature extraction followed by a SVM classifier for HAR.
There are also several other works [50], [51], [52], [53], which extended the two-stream framework to extract longterm video-level information for HAR. Wang et al. [50] divided each video into three segments and processed each segment with a two-stream network. The classification scores of the three segments were then fused by an average pooling method to produce the video-level prediction. Instead of fusing the scores of the segments as in [50], Diba et al. [52] aggregated the features of the segments by elementwise multiplication. Girdhar et al. [51] extracted features from sampled appearance and motion frames based on a two-stream framework, and used the vocabulary of "action words" to aggregate the features into a single video-level representation for classification. Feichtenhofer et al. [53] multiplied the appearance residual features with the motion information at the feature level for gated modulation. Zong et al. [54] extended the two-stream CNN in [53] to a three-stream CNN by adding the motion saliency stream to better capture the salient motion information. Bilen et al. [46] constructed dynamic images from RGB sequences and optical flow sequences to summarize the long-term global appearance and motion via rank pooling [55]. The dynamic images together with the original RGB images and optical flow information were then fed to a multi-stream framework for HAR. Dynamic images have also been adopted by some other works as representations of videos for HAR [56], [57], [58]. Besides, the two-stream network has also been extended to a two-stream Siamese network to extract features from the frames before an action happens (precondition) and the frames after the action (effect), and the action was then represented as a transformation between the two sets of features [59].
To tackle the high computational cost of computing accurate optical flow, some works [60] aimed to mimic the knowledge of the flow stream during training, in order to avoid the usage of optical flow during testing. Zhang et al. [60] proposed a teacher-student framework, which transfers the knowledge from the teacher network trained on optical flow data to the student network trained on motion vectors which can be obtained from compressed videos without extra calculation. Specifically, the knowledge was transferred by using the soft labels produced by the teacher model as an additional supervision to train the student network. Different from [60], Piergiovanni and Ryoo [61] proposed a trainable flow layer which captures the motion information without the need of computing optical flows.
There have also been some works extending the twostream CNN architectures in other aspects. Wang et al. [62] found that many two-stream CNNs were relatively shallow, and thus, they designed very deep two-stream CNNs for achieving better recognition results. Considering many frames of a video sequence could be irrelevant or useless for HAR, Kar et al. [63] recursively predicted the discriminative importance of each frame for feature pooling. To perform HAR on low spatial resolution videos, Zhang et al. [64] proposed two video super-resolution methods producing high resolution videos, which were fed to the spatial and temporal streams to predict the action class. In the work of [65], several fusion strategies were studied, showing that it is effective to fuse the spatial and temporal networks at the last convolution layer hence reducing the number of parameters, yet keeping the accuracy.
The two-stream 2D CNN architectures, that learn different types of information (e.g., spatial and temporal) from the input videos through separate networks and then perform fusion to get the final result, enable the traditional 2D CNNs to effectively handle the video data and achieve high HAR accuracy. However, this type of architectures is still not powerful enough for long-term dependency modeling, i.e., it has limitations in effectively modeling the video-level temporal information, while temporal sequence modeling networks, such as LSTM, can make up for it. (c) Schema of 2D convolution.
(e) Schema of Transformer-based methods.

RNN-Based Methods
RNNs can be used to analyze temporal data due to the recurrent connections in their hidden layers. However, the traditional vanilla RNN suffers from the vanishing gradient problem, making it incapable of effectively modeling the long-term temporal dependency. Thus, most of the existing methods have adopted gated RNN architectures, such as Long-Short Term Memory (LSTM) [67], [68], [69], [70], to model the long-term temporal dynamics in video sequences. As shown in Fig. 1(b), RNN-based methods usually employ 2D CNNs, which serve as feature extractors, followed by an LSTM model for HAR. Donahue et al. [7] introduced the Long-term Recurrent Convolutional Network (LRCN), which consists of a 2D CNN to extract frame-level RGB features followed by LSTMs to generate a single action label. Ng et al. [71] extracted frame-level RGB and "flow image" features from pre-trained 2D CNNs, and then fed these features to a stacked LSTM framework for HAR. In the work of [72], an encoder LSTM was used to map an input video into a fixed-length representation, which was then decoded through a decoder LSTM to perform the tasks of video reconstruction and prediction in an unsupervised manner. Wu et al. [73] leveraged two LSTMs, which operate on coarsescale and fine-scale CNN features cooperatively for efficient HAR. Majd and Safabakhsh [74] proposed a C 2 LSTM which incorporates convolution and cross-correlation operators to learn motion and spatial features while modeling temporal dependencies. Some other works [75], [76], [77] adopted the Bi-directional LSTM, which consists of two independent LSTMs to learn both the forward and backward temporal information, for HAR.
The introduction of attention mechanisms, in terms of spatial attention [78], [79], [80], [81], temporal attention [82], [83], and both spatial and temporal attention [84], [85], have also benefited the LSTM-based frameworks to achieve better HAR performance. Sharma et al. [78] designed a multi-layer LSTM model, which recursively outputs attention maps weighting the input features of the next frame to focus on the important spatial features, leading to better recognition performance. Sudhakaran et al. [80] introduced a recurrent unit with built-in spatial attention to spatially localize the discriminative information across a video sequence. LSTM has also been used to learn temporal attention to weight features of all frames [82]. Li et al. [85] proposed a Video-LSTM, which incorporates convolutions and motion-based attention into the soft-attention LSTM [86], to better capture both spatial and motion information.
Besides using LSTM, some works [87], [88], [89], [90] have used Gated Recurrent Units (GRUs) [91] for HAR, which can also alleviate the vanishing gradient problem of the vanilla RNN. Compared to LSTM, GRU has fewer gates, leading to fewer model parameters, yet it can usually provide a similar performance of LSTM for HAR [92].
Several other works [76], [93], [94] focused on the hybrid architectures combining two-stream 2D CNN and RNN models for HAR. For example, Wu et al. [93] utilized twostream 2D CNN models to extract spatial and short-term motion features, which were fed to two LSTMs respectively to model longer-term temporal information for HAR.
Generally, most of the RNN-based methods operate on the CNN features of frames instead of the high-dimensional frame images for HAR. Instead of using 2D CNN, some other works [95], [96], [97], [98] used 3D CNNs as their feature extractors. In the following section, we review the 3D CNN-based methods.

3D CNN-Based Methods
Plenty of researches [66], [99], [100], [101] have extended 2D CNNs ( Fig. 1(c)) to 3D structures ( Fig. 1(d)), to simultaneously model the spatial and temporal context information in videos that is crucial for HAR. As one of the earliest works, Ji et al. [99] segmented human subjects in videos by utilizing a human detector method, and then fed the segmented videos to a novel 3D CNN model to extract spatio-temporal features from videos. Unlike [99], Tran et al. [66] introduced a 3D CNN model, dubbed C3D, to learn the spatio-temporal features from raw videos in an end-to-end learning framework. However, these networks were mainly used for clip-level learning (e.g., 16 frames in each clip) instead of learning from full videos, which thus ignored the long-range spatio-temporal dependencies in videos. Hence, several approaches focused on modeling the long-range spatio-temporal dependencies in videos. For example, Diba et al. [102] extended DenseNet [103] with 3D filters and pooling kernels, and designed a Temporal 3D CNN (T3D), where the temporal transition layer can model variable temporal convolution kernel depths. T3D can densely and efficiently capture the appearance and temporal information at short, middle, and also long terms. In their subsequent study [104], they introduced a new block embedded in some architectures such as ResNext and ResNet, which can model the inter-channel correlations of a 3D CNN with respect to the temporal and spatial features. Varol et al. [105] proposed a Long-term Temporal Convolution (LTC) framework, which increases the temporal extents of 3D convolutional layers at the cost of reducing the spatial resolution, to model the long-term temporal structure. Hussein et al. [106] proposed multi-scale temporal-only convolutions, dubbed Timeception, to account for large variations and tolerate a variety of temporal extents in complex and long actions. Wang et al. [107] proposed a non-local operation, which models the correlations between any two positions in the feature maps to capture long-range dependencies. Besides, Li et al. [101] proposed a Channel Independent Directional Convolution (CIDC) which can be attached to I3D [108] to better capture the long-term temporal dynamics of the full input video.
To enhance the HAR performance, several other works [108], [109], [110], [111], [112] have investigated 3D CNN models with two-stream or multi-stream designs. For example, Carreira and Zisserman [108] introduced the twostream Inflated 3D CNN (I3D) inflating the convolutional and pooling kernels of a 2D CNN with an additional temporal dimension. Wang et al. [98] integrated a two-stream 3D CNN with an LSTM model to capture the long-range temporal dependencies. Feichtenhofer et al. [112] designed a twostream 3D CNN framework containing a slow pathway and a fast pathway that operate on RGB frames at low and high frame rates to capture semantic and motion, respectively. Inspired by [113], the features of the fast pathway were fused (by summation or concatenation) with the features of the slow pathway at each layer. Li et al. [114] introduced a two-stream spatio-temporal deformable 3D CNN with attention mechanisms to capture the long-range temporal and long-distance spatial dependencies. Besides, deep learning frameworks combining 2D CNNs and 3D CNNs have also been investigated for HAR [115], [116], [117]. For example, the ECO architecture [115] uses 2D CNNs to extract spatial features, which are stacked and then fed to 3D CNNs to model long-term dependencies for HAR. Some works focused on addressing certain other problems in 3D CNNs. For example, Zhou et al. [118] analyzed the spatio-temporal fusion in 3D CNN from a probabilistic perspective. Yang et al. [119] proposed a generic featurelevel Temporal Pyramid Network (TPN) to model speed variations in performing actions. Kim et al. [120] proposed a Random Mean Scaling (RMS) regularization method to address the problem of overfitting. The viewpoint variation problem has also been investigated [121], [122]. Inspired by [123], Varol et al. [122] proposed to train 3D CNN on synthetic action videos to address the problem of variations in viewpoints. Piergiovanni and Ryoo [121] proposed a geometric convolutional layer to learn viewinvariant representations of actions by imposing the latent action representations to follow 3D geometric transformations and projections. Knowledge distillation has also been investigated to improve motion representations of 3D CNN frameworks. For example, Stroud et al. [124] introduced a Distilled 3D Network (D3D) consisting of a student network and a teacher network, where the student network was trained on RGB videos and it also distilled knowledge from the teacher network that was trained on optical flow sequences. Crasto et al. [125] transferred the knowledge of a teacher model trained on the optical flows to a 3D CNN student network operating on RGB videos, by minimizing the mean squared error between the feature maps of the two streams. Following the research line of handling the large computational cost of obtaining optical flow, Shou et al. [126] proposed an adversarial framework with a lightweight generator to approximate flow information by refining the noisy and coarse motion vector available in the compressed video. Differently, Wang et al. [127] adopted an efficient learnable correlation operator to better learn motion information from 3D appearance features. Fayyaz et al. [128] addressed the problem of dynamically adapting the temporal feature resolution within the 3D CNNs to reduce their computational cost. A Similarity Guided Sampling (SGS) module was proposed to enable 3D CNNs to dynamically adapt their computational resources by selecting the most informative and distinctive temporal features.
The 3D CNN-based methods are very powerful in modeling discriminative features from both the spatial and temporal dimensions for HAR. However, many 3D CNNbased frameworks contain a large number of parameters, and thus require a large amount of training data. Therefore, some studies [109], [129], [130], [131], [132] aimed to factorize 3D convolutions. Sun et al. [129] proposed a Factorized spatio-temporal CNN (F st CN), factorizing the 3D convolution to 2D spatial convolutional layers followed by 1D temporal convolutional layers. Similarly, Qiu et al. [130] decomposed the 3D convolution into a 2D convolution for the spatial domain, followed by a 1D convolution for the temporal domain, to economically and effectively simulate 3D convolutions. Xie et al. [131] explored the performance of several variants of I3D [108] by utilizing a combination of 3D and 2D convolutional filters in the I3D network, and introduced temporally separable convolutions and spatio-temporal feature gating to enhance HAR. Yang et al. [133] proposed efficient asymmetric one-directional 3D convolutions to approximate the traditional 3D convolution. Besides, some other works [134], [135], [136], [137] also utilized 2D CNNs with powerful temporal modules to decrease the computational cost of 3D CNNs. Lin et al. [134] proposed a Temporal Shift Module (TSM), which shifts a part of the channels along the temporal dimension to perform temporal interaction between the features from adjacent frames. Unlike parameter-free temporal shift operations in [134], Sudhakaran et al. [135] introduced a lightweight Gate-Shift Module (GSM), which uses learnable spatial gating blocks for spatial-temporal decomposition of 3D convolutions. Wang et al. [136] designed a two-level Temporal Difference Module (TDM) to capture both finer local and long-range global motion information. Wang et al. [137] introduced an ACTION module leveraging multipath excitation to respectively model spatial-temporal, channelwise, and motion patterns.
The 3D CNN-based HAR methods generally perform spatio-temporal processing over limited intervals via the window-based 3D convolutional operations, where each convolutional operation only attends to relatively shortterm context in videos. Meanwhile, RNN-based methods process video sequence elements recurrently and thus cannot model relatively long-term spatio-temporal dependency. However, Transformers can directly attend to complete video sequences via its scalable self-attention mechanism, and thus can effectively learn long-range spatio-temporal relationships in videos. Hence, many recent works have also investigated Transformer-based HAR in RGB videos. In the following section, we review the Transformer-based methods.

Transformer-Based Methods
Transformer [171] is a novel deep learning model leading the machine learning field recently, due to its strong capabilities and promising prospects. As shown in Fig. 1(e), Transformer is composed of an encoder and a decoder. The encoder mainly consists of several self-attention blocks to encode the input sequence. The decoder shares a similar architecture as the encoder except an extra encoderdecoder attention mechanism in each block. This design enables Transformer to perform well concerning long-term dependency modeling, multi-modal fusion, and multi-task processing [172], [173]. Inspired by the success of Transformers in Natural Language Processing (NLP), plenty of works [160], [167] employed Transformers for HAR from videos. Girdhar et al. [151] proposed an Action Transformer network consisting of a Faster-RCNN [174] style model followed by a Transformer for action localization and recognition. Particularly, the Faster-RCNN [174] style network composes of an I3D model [108] as the base to generate initial spatio-temporal features and a Region Proposal Network to obtain object proposals for action localization. The Transformer treats the feature map of each particular subject as the query and features from the neighboring frames as the key and value. Bertasius et al. [155] extended ViT [175] to videos by decomposing each video into a sequence of framelevel patches. Then, a divided attention mechanism was proposed to separately apply spatial and temporal attentions within each block of the model. Arnab et al. [159] proposed pure-Transformer network variants by factorizing different components of the Transformer encoder along the spatial and temporal dimensions. Truong et al. [166] proposed a spatio-temporal directed attention architecture (DirecFormer), which learns the right order of frames within the action video by exploiting the direction of attention in addition to the magnitude of attention between frames. Patrick et al. [157] injected a trajectory attention block into the Transformer network to enhance the robustness of HAR in dynamic scenes. The trajectory attention block first generates a set of trajectory tokens in the spatial dimension, and then performs pooling operation over the temporal dimension to aggregate information along the motion trajectories. Fan et al. [158] proposed a multi-scale pyramid of features, which hierarchically expands the channel capacity while reducing the spatial resolution in videos via a multi-head pooling attention. Yan et al. [169] introduced Multiview Transformers, consisting of multiple individual encoders, each of which is specialized for a single input representation. Lateral connections between individual encoders are employed to effectively fuse information from different representations of the input video. Guo et al. [163] proposed an Uncertainty Guided Probabilistic Transformer (UGPT) by extending the standard Transformer to a probabilistic one. Particularly, UGPT treats the attention values as random variables and models their distribution as Gaussian distribution allowing the model to quantify the uncertainties of the prediction. Then, an uncertainty-guided training algorithm is utilized to separately train two models that respectively focus on low-uncertainty and high-uncertainty data. During the inference, these two models are dynami-cally combined to produce the action class. Ranasinghe et al. [170] proposed a Self-supervised Video Transformer (SVT) that learns cross-view and motion dependency from video clips with varying spatial sizes and frame rates.
Although video Transformers have acquired promising results, they also suffer from severe memory and computational overhead. Researchers thus have made plenty of efforts to decrease computation complexity [153], [156], [161], [164], [168], [176] and memory cost [154], [161], [164], [165], [176]. Fan et al. [168] presented an efficient Super Image for Action Recognition (SIFAR) approach, which transforms the 3D video frames into 2D super images as the input to the Transformer network. Zha et al. [161] introduced a Shifted Chunk Transformer (SCT), which is composed of a frame encoder and a clip encoder. The frame encoder relies on the image chunk and the shifted multi-head self-attention modules to capture fine-grained intra-frame representation and inter-frame variances, respectively. Specially, each individual frame is split into several local patches, which are fed into a hierarchical image chunk Transformer employing Locality-Sensitive Hashing (LSH) to significantly reduce the memory and computational consumption of matrix product in self-attention. Besides, the clip encoder is used to capture long-term temporal dependency. Neimark et al. [153] proposed a Video Transformer Network (VTN) to achieve a trade-off between speed and accuracy. VTN mainly relies on a spatial backbone (e.g., a 2D network) for frame-level feature processing and a temporal attention-based encoder, i.e., the Longformer model [177], for temporal modeling. Pan et al. [176] proposed an Interpretability-Aware REDundancy REDuction (IA-RED 2 ) framework, which learns to find the uncorrelated redundant patches, and hierarchically and dynamically drop them, resulting in a considerable reduction of computational complexity. IA-RED 2 is modelagnostic, task-agnostic, and inherently interpretable showing promising results in HAR. A local-window temporal attention strategy [156] and a separable attention mechanism [154] have also achieved a considerable shrinkage of computational complexity and memory overhead without loss of accuracy, respectively. Yang et al. [165] proposed a Recurrent Vision Transformer (RViT), which can be applied on both fixed-length and variant-length video clips without requiring huge memory. Specifically, RViT employs an attention gate mechanism, which operates in a recurrent manner, i.e., the output and current hidden state are determined by the current input frame and previous hidden state. Such a hidden state can capture spatio-temporal feature representations. Chen et al. [164] proposed a Regional-to-local attention-based Vision Transformer (RegionViT), which first generates regional and local tokens from each frame, and then performs interaction between a regional token and the associated local ones via information exchanging. Thus the local tokens can focus on a local field with global information, allowing the network to achieve a better tradeoff between accuracy and complexity. Finally, the dividedspace-time attention in TimeSformer [155] is adopted to model temporal variation.
In general, RGB data is the most commonly used modality for HAR in real-world application scenarios, since it is easy to collect and contains rich information. Besides, RGBbased HAR methods can leverage large-scale web videos to pre-train models for better recognition performance [140], [142]. However, it often requires complex computations for feature extraction from RGB videos. Moreover, HAR methods based on the RGB modality are often sensitive to viewpoint variations and background clutters, etc. Hence, HAR with other modalities, such as 3D skeleton data, has also received great attention, and is therefore discussed in the subsequent sections.

SKELETON MODALITY
Skeleton sequences encode the trajectories of human body joints, which characterize informative human motions. Therefore, skeleton data is also a suitable modality for HAR. The skeleton data can be acquired by applying pose estimation algorithms on RGB videos [182], [183] or depth maps [5], [184]. It can also be collected with motion capture systems. Generally, human pose estimation is sensitive to viewpoint variations. Meanwhile, motion capture systems that are insensitive to view and lighting can provide reliable skeleton data. However, in many application scenarios, it is not convenient to deploy motion capture systems. Thus many recent works on skeleton-based HAR used skeleton data obtained from depth maps [185] or RGB videos [186].
There are many advantages of using skeleton data for HAR, due to its provided body structure and pose information, its essentially simple and informative representation, its scale invariance, and its robustness against variations of clothing textures and backgrounds. Due to these advantages and also the availability of accurate and low-cost depth sensors, skeleton-based HAR has attracted much attention in the research community recently.
Early works focused on extracting hand-crafted spatial and temporal features from skeleton sequences for HAR. The hand-crafted feature-based methods can be roughly divided into joint-based [187] and body part-based [188] methods, depending on the used feature extraction techniques. Due to the strong feature learning capability, deep learning has been widely used for skeleton-based HAR, and has become the mainstream research in this field. Consequently, we review, in the following, the deep learning methods, which can be mainly divided into four categories: RNN, CNN, Graph Neural Network (GNN) or Graph Convolutional Network (GCN), and Transformer-based methods, as shown in Fig. 2. Table 3 shows the results achieved by different skeleton-based HAR methods on two benchmark datasets.

RNN-Based Methods
As mentioned in Section 2.1.2, RNNs and their gated variants (e.g., LSTMs) are capable of learning the dynamic dependencies in sequential data. Hence, various methods [189], [190], [191], [192] have applied and adapted RNNs and LSTMs to effectively model the temporal context information within the skeleton sequences for HAR.
In one of the classical methods, Du et al. [189] proposed an end-to-end hierarchical RNN, dividing the human skeleton into five body parts instead of inputting the skeleton in each frame as a whole. These five body parts were then separately fed to multiple bidirectional RNNs, whose output representations were hierarchically fused to generate high-level representations of the action. Differential RNN (dRNN) [193] learns salient spatio-temporal information by quantifying the change of the information gain caused by the salient motions between frames. The Derivative of States (DoS) was proposed inside the LSTM unit to act as a signal controlling the information flow into and out of the internal state over time. Zhu et al. [194] introduced a novel mechanism for the LSTM network to achieve automatic cooccurrence mining, since co-occurrence intrinsically characterizes the actions. Sharoudy et al. [195] proposed a Partaware LSTM (P-LSTM) by introducing a mechanism to simulate the relations among different body parts inside the LSTM unit. Liu et al. [8], [196] extended the RNN design to both the temporal and spatial domains. Specifically, they utilized the tree structure-based skeleton traversal method to further exploit the spatial information, and trust gates to deal with noise and occlusions. In their subsequent study [197], an attention-based LSTM network named Global Context-Aware Attention LSTM (GCA-LSTM) was proposed to selectively focus on the informative joints using the global context information. The GCA-LSTM network contains two LSTM layers, where the first layer encodes the skeleton sequence and outputs a global context memory, while the second layer outputs attention representations to refine the global context. Finally, a softmax classifier outputs the HAR result. In the work of [198], a two-stream RNN structure was proposed to model both the temporal dynamics and spatial configurations. Deep LSTM with spatio-temporal attention was proposed in [199], where a spatial attention sub-network and a temporal attention subnetwork work jointly under the main LSTM network. To model variable temporal dynamics of skeleton sequences, Lee et al. [200] proposed an ensemble Temporal Sliding LSTM (TS-LSTM) framework composed of multiple parts, containing short-term, medium-term, and long-term TS-LSTM networks. IndRNN [201] not only addresses the gradient vanishing and exploding issues, but it is also faster than the original LSTM.

CNN-Based Methods
CNNs have achieved great success in 2D image analysis due to their superior capability in learning features in the spatial domain. However, when facing skeleton-based HAR, the modeling of spatio-temporal information becomes a challenge. Nevertheless, plenty of advanced approaches have been proposed, which apply temporal convolution on skeleton data [202], or represent skeleton sequences as pseudo-images that are then fed to standard CNNs for HAR [203], [204]. These pseudo-images were designed to simultaneously encode spatial structure information in each frame and temporal dynamic information between frames.
Hou et al. [203] and Wang et al. [204] respectively proposed the skeleton optical spectra and the joint trajectory maps. They both encoded the spatio-temporal information of skeleton sequences into color-texture images, and then adopted CNNs for HAR. As an extension of these works, the Joint Distance Map (JDM) [205] produces view-invariant color-texture images encoding pair-wise distances of skeleton joints. Ke et al. [206] transformed each skeleton sequence into three "video clips", which were then fed to a pre-trained CNN to produce a compact representation, followed by a multi-task learning network for HAR. Kim and Reiter [202] utilized the Temporal CNN (TCN) [207] explicitly providing a type of interpretable spatio-temporal representations. An end-to-end framework to learn co-occurrence features with a hierarchical methodology was proposed in [208]. Specifically, they learned the point-level features of each joint independently, and then utilized these features as a channel of the convolutional layer to learn hierarchical co-occurrence features, and a two-stream framework was adopted to fuse the motion features. Caetano et al. introduced SkeleMotion [209] and the Tree Structure Reference Joints Image (TSRJI) [210] as representations of skeleton sequences for HAR. In [211], Skepxels were utilized as basic building blocks to construct skeletal images mainly encoding the spatio-temporal information of human joint locations and velocities. Besides, Skepxels were used to hierarchically capture the microtemporal relations between the joints in the frames, and Fourier Temporal Pyramids [9] were utilized to exploit the macro-temporal relations.
Plenty of researches focused on addressing certain specific problems. For example, the works of [205], [212], [213] focused on handling the viewpoint variation issue. Since features learned from skeleton data are not always translation, scale, and rotation invariant, several methods [214], [215] have also been designed to handle these issues. CNN architectures are often complex, resulting in high computational costs. Thus Yang et al. [216] proposed a Double-feature Double-motion Network (DD-Net) to make the CNN-based recognition models run faster. Additionally, Tang et al. [217] proposed a self-supervised learning framework under the unsupervised domain adaptation setting, which segments and permutes the time segments or body parts to reduce domain-shift and improve the generalization ability of the model.

GNN or GCN-Based Methods
Due to the expressive power of graph structures, analyzing graphs with learning models have received great attention recently [219], [220]. As shown in Fig. 2(c), skeleton data is naturally in the form of graphs. Hence, simply representing skeleton data as a vector sequence processed by RNNs, or 2D/3D maps processed by CNNs, cannot fully model the complex spatio-temporal configurations and correlations of the body joints. This indicates that topological graph representations can be more suitable for representing the skeleton data. As a result, many GNN and GCN-based HAR methods [186], [221] have been proposed to treat the skeleton data as graph structures of edges and nodes.
GNN is a connection model capturing the dependence within a graph through message passing between nodes. Si et al. [221] proposed a spatial reasoning network, followed by RNNs, to capture the high-level spatial structural and temporal dynamics of skeleton data. Shi et al. [222] represented the skeleton as a directed acyclic graph to effectively (a) RNN-based method.
incorporate both the joint and bone information, and utilized a GNN to perform HAR. More recently, GCN-based HAR has become a hot research direction [223], [224], [225], [226], [227]. Yan et al. [186] exploited GCNs for skeleton-based HAR by introducing Spatial-Temporal GCNs (ST-GCNs) that can automatically learn both the spatial and temporal patterns from skeleton data, as shown in Fig. 2(c). More specifically, the pose information was estimated from the input videos and then passed through the spatio-temporal graphs to achieve action representations with strong generalization capabilities for HAR. Since implicit joint correlations have been ignored by previous works [186], Li et al. [228] further proposed an Actional-Structural GCN (AS-GCN), combining actional links and structural links into a generalized skeleton graph. The actional links were used to capture action-specific latent dependencies and the structural links were used to represent higher-order dependencies. To better explore the implicit joint correlations, Peng et al. [229] determined their GCN architecture via a neural architecture search scheme. Specifically, they enriched the search space to implicitly capture the joint correlations based on multiple dynamic graph sub-structures and higher-order connections with a Chebyshev polynomial approximation. Besides, context information integration was used to effectively model long-range dependencies in the work of [230].
Shi et al. [231] proposed a two-stream Adaptive GCN (2s-AGCN), where the topology of graphs can be either uniformly or individually learned by the back-propagation algorithm, instead of setting it manually. The 2s-AGCN explicitly combined the second-order information (lengths and directions of human bones) of the skeleton with the first-order information (coordinates of joints). Wu et al. [232] introduced a cross-domain spatial residual layer to capture the spatio-temporal information, and a dense connection block to learn global information based on ST-GCN. Li et al. [233] fed frame-wise skeleton and node trajectories of a skeleton sequence to a spatial graph router and a temporal graph router to generate new skeleton-joint-connectivity graphs, followed by a ST-GCN for classification. Liu et al. [234] integrated a disentangled multi-scale aggregation scheme and a spatio-temporal graph convolutional operator named G3D to achieve a powerful feature extractor. Highlevel semantics of joints were introduced for HAR in [235]. Attention mechanisms were applied for extracting discriminative information and global dependencies in [236], [237]. Besides, to reduce computational costs of GCNs, a Shift-GCN was designed by Cheng et al. [238], which adopts shift graph operations and lightweight point-wise convolutions, instead of using heavy regular graph convolutions. Following this research line, Song et al. [239] proposed a multi-stream GCN model, which fuses the input branches including joint positions, motion velocities, and bone features at early stage, and utilized separable convolutional layers and a compound scaling strategy to extremely reduce the redundant trainable parameters while increasing the capacity of model. Different from the above mentioned methods, Li et al. [240] proposed symbiotic GCNs to handle both action recognition and motion prediction tasks simultaneously. The proposed Sym-GNN consists of a multibranch multi-scale GCN followed by an action recognition and a motion prediction heads, to jointly conduct action recognition and motion prediction tasks. This allows the two tasks to enhance each other.
Recently, Chen et al. [241] proposed a Channel-wise Topology Refinement Graph Convolution (CTR-GC) for dynamic topology and multi-channel feature modeling. In particular, CTR-GC takes a shared topology matrix as the generic prior for channels, then refines it with inferring channel-specific correlation to acquire channel-wise topologies. Chi et al. [242] proposed InfoGCN, which comprises an information bottleneck objective to learn maximally informative action representations, and an attention-based graph convolution to infer the context-related skeleton topology. Li et al. [243] proposed an Elastic Semantic Network (Else-Net), which consists of a GCN backbone model followed by multiple layers of elastic units for Continual HAR. Particularly, each elastic unit contains several learning blocks to learn diversified knowledge from different human actions over time, and a switch block to select the most relevant block with respect to the new incoming action.

Transformer-Based Methods
As discussed in Section 2.1.4, Transformers have shown great potential in sequential data processing. Thus, plenty of methods [244], [245] applied Transformers on skeleton sequences, with an emphasis on spatio-temporal modeling. Zhang et al. [244] proposed a Spatial-Temporal Specialized Transformer (STST), consisting of a spatial Transformer block to model pose information at frame level and a directional temporal Transformer module to capture long dynamics in the temporal dimension. A multitask self-supervision module was also designed to enhance the robustness of the model to noise. Plizzari et al. [245] introduced a Spatial-Temporal Transformer network (ST-TR), where the spatial and temporal self-attention modules learn intra-frame joint interactions and inter-frame motion dynamics, respectively. Qiu et al. [246] presented a Spatio-Temporal Tuples Transformer (STTFormer) architecture for HAR. Specifically, STTFormer first divides a skeleton sequence into several non-overlapping clips, and then utilizes the spatio-temporal self-attention module to capture multi-joint dependencies between adjacent frames. Finally, an inter-frame feature aggregation module is employed to aggregate sub-actions. Shi et al. [218] proposed a Sparse Transformer for Action Recognition (STAR), capable of processing varying-length skeleton input without extra preprocessing. STAR mainly relies on two modules including a sparse self-attention module to learn spatial relationships via sparse matrix multiplications, and a segmented linear self-attention module to model joint correlations in the temporal dimension.
Cheng et al. [247] presented a hierarchical Transformer for unsupervised skeleton-based HAR, along with a motion prediction pre-training task between adjacent frames to learn discriminative representations. Liu et al. [248] proposed a Kernel Attention Adaptive Graph Transformer Network (KA-AGTN), which is mainly composed of a skeleton graph transformer block to effectively capture the varying degrees of higher-order dependencies among joints, a temporal kernel attention module, and an adaptive graph strategy. To decrease the huge computation and memory cost, Wang et al. [249] proposed IIP-Transformer that exploits part-level skeleton data encoding for HAR. Specifically, IIP-Transformer regards body joints as five parts, and leverages their relationships to capture intra-and inter-part level dependencies.
In summary, the skeleton modality provides the body structure information, which is simple, efficient, and informative for representing human behaviors. Nevertheless, HAR using skeleton data still faces challenges, due to its very sparse representation, the noisy skeleton information, and the lack of shape information that can be important when handling human-object interactions. Hence, some of the existing works on HAR also focused on using depth maps, as discussed in the following section, since depth maps not only provide the 3D geometric structural information but also conserve the shape information.

DEPTH MODALITY
Depth maps refer to images where the pixel values represent the distance information from a given viewpoint to the points in the scene. The depth modality, which is often robust to variations of color and texture, provides reliable 3D structural and geometric shape information of human subjects, and thus can be used for HAR. The essence of constructing a depth map is to convert the 3D data into a 2D image, and different types of devices have been developed to obtain depth images, which include active sensors (e.g., Time-of-Flight and structured-light-based cameras) and passive sensors (e.g., stereo cameras) [19]. Active sensors emit radiation in the direction towards the objects in the scene, and then measure the reflected energy from the objects to acquire the depth information. In contrast, passive sensors measure the natural energy that is emitted or reflected by the objects in the scene. For example, as a type of passive sensor, stereo cameras usually have two or more lenses to simulate the binocular vision of human eyes to acquire depth information, and depth maps are recovered by seeking image point correspondences between stereo pairs [261]. Compared to active sensors (e.g., Kinect and RealSense3D), passive depth map generation is usually computationally expensive, and moreover, depth maps obtained with passive sensors can be ineffective in texture-less regions or highly textured regions with repetitive patterns.
In this section, we review methods that used depth maps for HAR. Note that only a few works [262], [263] used depth maps captured by stereo cameras for HAR, while most of the other methods [9], [56], [264] focused on using depth videos captured by active sensors to recognize actions, due to the availability of low-cost and reliable active sensors like Kinect. Consequently, here we focus on reviewing the methods using depth videos captured with active sensors for HAR. Table 4 shows the results of some depth-based HAR methods on several benchmark datasets.
Compared to hand-crafted feature-based methods [265], [266], deep learning models [9], [56] have shown to be more powerful and achieved better performance for HAR from depth maps. Inspired by the hand-crafted Depth Motion Map (DMM) features [267], a deep learning framework utilizing weighted hierarchical DMMs was proposed in [268]. As DMMs are not able to capture detailed temporal information, Wang et al. [56] proposed to represent depth sequences with three pairs of structured dynamic images [55] at the body, body part, and joint levels, which were then fed to CNNs followed by a score fusion module for fine-grained HAR.
The performance of this method was further improved in [264] by introducing three representations of depth maps, including dynamic depth images, dynamic depth normal images, and dynamic depth motion normal images. Rahmani et al. [9] transferred the human data obtained from different views to a view-invariant high-level space to address the view-invariant HAR problem. Specifically, they utilized a CNN model to learn the view-invariant human pose model and Fourier Temporal Pyramids to model the temporal action variations. To obtain more multi-view training data, synthetic data was generated by fitting synthetic 3D human models to real motion capture data, and rendering the human data from various viewpoints. In [57], multi-view dynamic images were extracted through multiview projections from depth videos for action recognition. In order to effectively capture spatio-temporal information in depth videos, Sanchez et al. [269] proposed a 3D fully CNN architecture for HAR. In their subsequent work [270], a variant of LSTM unit was introduced to address the problem of memory limitation during video processing, which can be used to perform HAR from long and complex videos.
Note that besides using depth maps obtained with active sensors or stereo cameras for HAR, there has been another approach designed for depth-based HAR from RGB videos. Specifically, Zhu and Newsam [271] estimated the depth maps from RGB videos using existing depth estimation techniques [272], [273], which were then passed through a deep learning architecture for action classification.
In general, the depth modality provides geometric shape information that is useful for HAR. However, the depth data is often not used alone, due to the lack of appearance information that can also be helpful for HAR in some scenarios. Thus, many works focused on fusing depth information with other data modalities for enhanced HAR, and more details can be found in Section 3.

INFRARED MODALITY
Generally, infrared sensors do not need to rely on external ambient light, and thus are particularly suitable for HAR at night. Infrared sensing technologies can be divided into active and passive ones. Some infrared sensors, such as Kinect, rely on active infrared technology, which emit infrared rays and utilize target reflection rays to perceive objects in the scene. In contrast, thermal sensors relying on passive infrared technology do not emit infrared rays. Instead, they work by detecting rays (i.e., heat energy) emitted from targets.
Some deep learning methods [30], [274] have been proposed for HAR from infrared data. In [275], the extremely low-resolution thermal images were first cropped by the gravity center of human regions. The cropped sequences and the frame differences were then passed through a CNN followed by an LSTM layer to model spatio-temporal information for HAR. To simultaneously learn both the spatial and temporal features from thermal videos, Shah et al. [276] utilized a 3D CNN and achieved real-time HAR. Instead of using raw thermal images, Meglouli et al. [277] passed the optical flow information computed from thermal sequences into a 3D CNN for HAR.
Inspired by the two-stream CNN model [44], several multi-stream architectures [10], [30], [278] have also been proposed. Gao et al. [30] passed optical flow and optical flow motion history images (OF-MHIs) [279] through a twostream CNN for HAR. In [10], both infrared data and optical flow volumes were fed to a two-stream 3D CNN framework to effectively capture the complementary information on appearance from still thermal frames and motion between frames. The two-stream model was trained using a combination of cross-entropy and label consistency regularization loss in [280]. To better represent the global temporal information, Optical flow motion history images (OF-MHIs) [279], optical flow and a stacked difference of optical flow images, were fed to a three-stream CNN for HAR [278]. Imran and Raman [281] proposed a four-stream architecture, where each stream consists of a CNN followed by an LSTM. The local/global stacked dense flow difference images [282] and local/global Stacked saliency difference images were fed to these four streams to capture both the local and global spatio-temporal information in videos. Mehta et al. [283] presented an adversarial framework consisting of a two-stream 3D convolutional auto-encoder as the generator and two 3D CNNs as the joint discriminator. The generator network took thermal data and optical flow as inputs, and the joint discriminator tried to discriminate the real thermal data and optical flow from the reconstructed ones.
In all, infrared data has been used for HAR, especially for night HAR. However, infrared images may suffer from a relatively low contrast and a low signal-to-noise ratio, making it challenging for robust HAR in some scenarios.

POINT CLOUD MODALITY
Point cloud data is composed of a numerous collection of points that represent the spatial distribution and surface characteristics of the target under a spatial reference system. There are two main ways to obtain 3D point cloud data, namely, (1) using 3D sensors, such as LiDAR and Kinect, or (2) using image-based 3D reconstruction. As a 3D data modality, point cloud has great power to represent the spatial silhouettes and 3D geometric shapes of the subjects, and hence can be used for HAR.
Early methods [286], [303] focused on extracting handcrafted spatio-temporal descriptors from the point cloud sequences for HAR, while the current mainstream research focuses on deep learning architectures [294], [295] that generally achieve better performance. Wang et al. [11] transformed the raw point cloud sequence into regular voxel TABLE 4 Performance of depth, infrared, point cloud, and event stream-based deep learning HAR methods. The datasets are MSRDailyActivity3D (M) [284], Northwestern-UCLA (N-UCLA) [285], UWA3D Multiview II (UWA3D II) [286], NTU RGB+D (N) [195], InfAR [30], MSR-Action3D [287], DvsGesture [288], and DHP19 [32]. sets. Then temporal rank pooling [55] was applied on all the voxel sets to encode 3D action information into one single voxel set. Finally, the voxel representation was abstracted and passed through the PointNet++ model [304] for 3D HAR. However, converting point clouds into voxel representations causes quantization errors and inefficient processing performance. Instead of quantizing the point clouds into voxels, Liu et al. [290] proposed MeteorNet, which directly stacks multi-frame point clouds and calculates local features by aggregating information from spatio-temporal neighboring points. Unlike MeteorNet [290], which learns spatio-temporal information by appending 1D temporal dimension to 3D points, PSTNet [295] disentangles the space and time to reduce the impacts of the spatial irregularity of points on temporal modeling. In the work of [293], a self-supervised learning framework was introduced to learn 4D spatio-temporal information from the point cloud sequence by predicting the temporal orders of 4D clips within the sequence. A 4D CNN model [290], [305] followed by an LSTM was utilized to predict the temporal order. Finally, the 4D CNN+LSTM network was fine-tuned on the existing HAR dataset to evaluate its performance. Wang et al. [292] proposed an anchor-based spatio-temporal attention convolution model to capture the dynamics of 3D point cloud sequences. However, these methods are not able to sufficiently capture the long-term relationships within point cloud sequences. To address this issue, Min et al. [291] introduced a modified version of LSTM unit named PointLSTM to update state information for neighbor point pairs to perform HAR.
Transformers have also recently gained increasing attention in point cloud-based HAR [289], [294], [296], [297]. Yang et al. [289] proposed a Point Attention Transformer (PAT), which mainly relies on a lightweight Group Shuffle Attention (GSA) module to learn relationships among points and a permutation-invariant, task-agnostic and differentiable sampling method, named Gumbel Subset Sampling (GSS), to sample a representative subset of input points. Fan et al. [294] introduced a point 4D convolution, followed by a Transformer to capture the global appearance and motion information across the entire point cloud video for HAR. Wei et al. [296] proposed a Point Spatial-Temporal Transformer (PST 2 ) for processing point cloud sequences. A self-attention-based module, called Spatio-Temporal Self-Attention (STSA), was introduced to capture the spatialtemporal context information, which can be leveraged for action recognition in 3D point clouds.
In all, point clouds can effectively capture the 3D shapes and silhouettes of subjects, and can be used for HAR. Generally, 3D point cloud-based HAR methods can be insensitive to viewpoint variations, since viewpoint normalization can be conveniently performed by rotating the point cloud in the 3D space. However, point clouds often suffer from the presence of noise and high non-uniformity distribution of points, making it challenging for robust HAR. In addition, processing all the points within the point cloud sequence is often computationally expensive.

EVENT STREAM MODALITY
Event cameras, also known as neuromorphic cameras or dynamic-vision sensors, which can capture illumination changes and produce asynchronous events independently for each pixel [301], have received lots of attention recently. Different from conventional video cameras capturing the entire image arrays, event cameras only respond to changes in the visual scene. Taking a high-speed object as an example, traditional RGB cameras may not be able to capture enough information of the object, due to the low frame rate and motion blur. However, this issue can be significantly mitigated when using event cameras that operate at extremely high frequencies, generating events at a µs temporal scale [301]. Besides, event cameras have some other characteristics, such as high dynamic range, low latency, low power consumption, and no motion blur, which make them suitable for HAR. Particularly, event cameras are able to effectively filter out background information and keep foreground movement only, avoiding considerable redundancies in the visual information. However, the information obtained with event cameras is generally spatio-temporally sparse, and asynchronous. Common event cameras include the Dynamic Vision Sensor (DVS) [306] and the Dynamic and Active-pixel Vision Sensor (DAVIS) [307].
The output data of event cameras is highly different from that of conventional RGB cameras, as shown in Table  1. Thus, some of the existing methods [12], [301] mainly focused on designing event aggregation strategies converting the asynchronous output of the event camera into synchronous visual frames, which can then be processed with conventional computer vision techniques. While handcrafted feature-based methods [308], [309] have been proposed, deep learning methods have been more popularly used recently. Given a fixed time interval ∆t, Innocenti et al. [301] first built a sequence of binary representations from the raw event data by checking the presence or absence of an event for each pixel during ∆t. These intermediate binary representations were then stacked together to form a single frame via a binary to decimal conversion. The sequence of such frames extracted from the whole event stream was finally fed to a CNN+LSTM for HAR. Huang et al. [299] utilized time-stamp image encoding to transform the event data sequence into frame-based representations, which were then fed to a CNN for HAR. More precisely, the value of each pixel in the generated time-stamp image encodes the amount of event taking place within a time-window. However, since the size of the frame to be processed is basically larger than the original Neuromorphic Vision Stream (NVS), the advantages of event cameras are diluted.
Therefore, different from the aforementioned methods, some other deep learning approaches [12], [298], [300], [310], [311] have been proposed, which directly take the event data as input of deep neural networks. Ghosh et al. [12] learned a set of 3D spatio-temporal convolutional filters in an unsupervised manner to generate a structured matrix form of the raw event data, which was then fed to a 3D CNN for HAR. George et al. [310] utilized Spiking Neural Networks (SNNs) for event stream-based HAR. The work in [298] treated the event stream as a 3D point cloud, which was then fed to a PointNet [304] for gesture recognition. Recently, Bi et al. [311] represented events as graphs and used a GCN network for an end-to-end feature learning directly from the raw event data. Besides, Plizzari et al. [302] proposed E 2 (GO) leveraging traditional CNNs by employing Squeeze And Excitation modules [312] to exploit micro-movements in event data. The E 2 (GO)MO has also been introduced to distill motion information from optical flow to event data.
In general, the event stream modality is an emerging modality for HAR, and has received great attention in the past few years. Processing event data is computationally cheap, and the captured frames do not usually contain background information, which can be helpful for action understanding. However, the event stream data cannot generally be effectively and directly processable using conventional video analysis techniques, and thus effectively and directly utilizing the asynchronous event streams for HAR is still a challenging research problem.

AUDIO MODALITY
Audio signal is often available with videos for HAR. Due to the synchronization between the visual and audio streams, the audio data can be used to locate actions to reduce human labeling efforts and decrease computational costs. Some deep learning-based methods [13], [313] have been proposed to perform general activity recognition from audio signals. However, in recent years, only a few deep learning methods were proposed for HAR from audio signal alone. For example, Liang and Thomaz [13] used a pre-trained VGGish model [314] as the feature extractor followed by a deep classification network to perform HAR.
Using audio modality alone is not a very popular scheme for HAR, compared to other data modalities, since audio signal does not contain enough information for accurate HAR. Nevertheless, the audio modality can serve as complementary information for more reliable and efficient HAR, and thus most of the audio-based HAR methods [25], [315], [316], [317] focused on taking advantage of multi-modal deep learning techniques, which are discussed in Section 3.

ACCELERATION MODALITY
Acceleration signals obtained from accelerometers have been used for HAR [318], due to their robustness against occlusion, viewpoint, lighting, and background variations, etc. Specifically, a tri-axial accelerometer can return an estimation of acceleration along the x, y, and z axes, that can be used to perform human activity analysis [319]. As for the feasibility of using acceleration signal for HAR, although the size and proportion of the human body vary from person to person, people generally have similar qualitative ways to perform an action, so the acceleration signal usually does not have obvious intra-class variations for the same action. HAR using acceleration signal can generally achieve a high accuracy, and thus has been adopted for remote monitoring systems [320], [321] while taking care of privacy issues.
Recently, deep learning networks have been widely used for acceleration-based HAR. In [14], [322], [323], [324], the tri-axial acceleration data was fed to different CNN architectures for HAR. Wang et al. [325] proposed a framework consisting of CNN and Bi-LSTM networks to extract spatial and temporal features from the raw acceleration data. Unlike the above-mentioned works, Lu et al. [326] utilized a modified Recurrence Plot (RP) [327] to transform the raw triaxial acceleration data into color images, which were then fed to a ResNet for HAR. Besides, some acceleration-based methods [328] focused on the fall detection task.
In general, the acceleration modality can be utilized for fine-grained HAR, and has been used for action monitoring, especially for elderly care due to its privacy-protecting characteristic. However, the subject needs to carry wearable sensors that are often cumbersome and disturbing. In addition, the position of the sensors on the human body can also affect the HAR performance.

RADAR MODALITY
Radar is an active sensing technology that transmits electromagnetic waves and receives returned waves from targets. Continuous-wave radars, such as Doppler [329] and Frequency Modulated Continuous Wave (FMCW) radars [330], are most often chosen for HAR. Specifically, Doppler radars detect the radial velocity of the body parts, and the frequency changes according to the distance, which is known as the Doppler shift. The micro-Doppler signatures generated by the micro-motion of the radar contain the motion and structure information of the target, which thus can be used for HAR. As for the FMCW radars, they can measure the distances of the targets as well. There are some advantages of using spectrograms obtained from radars for HAR, which include the robustness to variations in illumination and weather conditions, privacy protection, and the capability of through-wall HAR [22].
Several deep learning architectures have been proposed recently. The works in [15], [331], [332], [333], [334], [335] fed the micro-Doppler spectrogram images into CNNs for predicting action classes in different scenarios, such as aquatic activities [331], [333], different geometrical locations [335], and simulated environments [332]. Unlike the aforementioned methods, the work in [336] directly took the raw radar range data as input. The raw data was passed through an auto-correlation function, followed by a CNN to extract action-related features. Finally, a Random Forest classifier was used to predict the action class.
The two-stream architecture has also been investigated. Hernangómez et al. [337] designed a two-stream CNN taking both micro-Doppler and range spectrograms representing the structural signatures of the target as inputs. In the work of [338], micro-Doppler spectrograms and echos of the radar were fed to a two-stream CNN model for HAR.
By interpreting micro-Doppler spectrograms as temporal sequences rather than images, several RNN-based architectures [339], [340] have also been proposed recently. For example, Yang et al. [340] and Wang et al. [339] utilized an LSTM model and a stacked RNN model, respectively to predict the action classes.
In general, the characteristics and advantages of the radar modality make it suitable to be used for HAR in some scenarios, but radars are relatively expensive. Though HAR using radar data has achieved satisfactory results on some datasets, there is still plenty of room for the development of radar-based methods, and the work of [22] also pointed out some future directions in this area, such as handling more complex actions in real-world scenarios with radar data.

WIFI MODALITY
WiFi is considered to be one of the most common indoor wireless signal types nowadays [341]. Since human bodies are good reflectors of wireless signal, WiFi signal can be utilized for HAR, and sometimes even for through-wall HAR [342]. There are some advantages of using the WiFi modality for action analysis, mainly due to the convenience, simplicity, and privacy protection of WiFi signal, and also the low cost of WiFi devices. Specifically, most of the existing WiFi-based HAR methods [343], [344] focused on using the Channel State Information (CSI) to conduct the HAR task. CSI is the fine-grained information computed from the raw WiFi signal, and the WiFi signal reflected by a person who performs an action, usually generates unique variations in the CSI on the WiFi receiver.
Deep learning-based HAR from the CSI signal has received attention recently. Wang et al. [345] proposed a deep sparse auto-encoder to learn discriminative features from the CSI streams. In the work of [16], the WiFi-based samplelevel HAR model, named Temporal Unet, was proposed, which consists of several temporal convolutional, deconvolutional, and max pooling layers. This method classified every WiFi distortion sample in the series into one action for fine-grained HAR. LSTM networks have also been used for HAR using CSI signal [346], [347], [348]. Huang et al. [346] passed raw CSI measurements through a noise and redundancy removal module followed by a clip reconstruction module to segment the processed CSI signal into multiple clips. These clips were then fed to a multi-stream CNN+LSTM model to perform HAR. In the work of [347], the spatial features of the CSI signal were first extracted from a fully connected layer of a pre-trained CNN. These features were then fed to a Bi-LSTM to capture the temporal information for HAR. Chen et al. [348] directly passed the raw CSI signal through an attention-based Bi-LSTM to predict the action class. Different from the above-mentioned works, Gao et al. [349] transformed the CSI signal into radio images, which were then fed to a deep sparse auto-encoder to learn discriminative features for HAR.
In general, because of the advantages (e.g., convenience), the WiFi modality can be used for HAR in some scenarios. However, there are still some challenges which need to be further addressed, such as how to more effectively use the CSI phase and amplitude information, and to improve the robustness when handling dynamic environments.

MULTI-MODALITY
In real life, humans often perceive the environment in a multi-modal cognitive way. Similarly, multi-modal machine learning is a modeling approach aiming to process and relate the sensory information from multiple modalities [358]. By aggregating the advantages and capabilities of various data modalities, multi-modal machine learning can often provide more robust and accurate HAR. There are two main types of multi-modality learning methods, namely, fusion and co-learning. Fusion refers to the integration of information from two or more modalities for training and inference, while co-learning refers to the transfer of knowledge between different data modalities.

FUSION
As discussed in Section 2, different modalities can have different strengths. Thus it becomes a natural choice to take advantage of the complementary strengths of different data modalities via fusion, so as to achieve enhanced HAR performance. There are two widely used multi-modality fusion schemes in HAR, namely, score fusion and feature fusion. Generally, the score fusion [362] integrates the decisions that are separately made based on different modalities (e.g., by weighted averaging [382] or by learning a score fusion model [361]) to produce the final classification results. On the other hand, the feature fusion [379] generally combines the features from different modalities to yield aggregated features that are often very discriminative and powerful for HAR. Note that data fusion, i.e., fusing the multimodality input data before feature extraction [404], has also been exploited. Since the input data can be treated as the original raw features, here we simply categorize the data fusion approaches under feature fusion. Table 5 gives the results of multi-modality fusion-based HAR methods on the MSRDailyActivity3D [284], UTD-MHAD [355], and NTU RGB+D [195] benchmark datasets.  [23] 2018 Feature -94.6 --Khaire et al. [385] 2018 Score -95.1 --Khaire et al. [386] 2018 Score -95.4 --Ye et al. [387] 2015 RGB,S,D,PC Score ----Ardianto et al. [388] 2018 RGB,D,IR Score ----FUSION-CPA [389] 2020 S,IR Feature --91.6 94.5 ActAR [390] 2022 S,IR Feature ----Wang et al. [391] 2016

Fusion of Visual Modalities
With the emergence of low-cost RGB-D cameras, many multi-modality datasets [195], [284] have been created by the community, and consequently, some multi-modality fusionbased HAR methods have been proposed. Most of these methods focused on the fusion of visual modalities which are reviewed below. Fusion of RGB and Depth Modalities. The RGB and depth videos respectively capture rich appearance and 3D shape information, that are complementary and can be used for HAR. Early methods [411], [412] focused on extracting hand-crafted features which capture spatio-temporal structural relationships from the RGB and depth modalities for more reliable HAR. Since the current mainstream of research focuses on deep learning architectures, we review, in the following, deep learning methods which fuse RGB and depth data modalities. In [359], a four-stream deep CNN was introduced to extract features from different representations of depth data (i.e., three Depth Motion Maps [267] from three different viewpoints) and RGB data (i.e., a Motion History Image [413]). The output scores of these four streams were fused to perform action classification. By considering the depth and RGB modalities as a single entity, Wang et al. [360] extracted scene flow features from the spatially aligned and temporally synchronized RGB and depth frames. The bidirectional rank pooling [55] was utilized to generate two dynamic images from the sequence of scene flow features. The dynamic images were then fed to two different CNNs, and finally, their classification scores were fused to perform HAR. Wang et al. [58] represented the RGB and depth data as two pairs of RGB and depth dynamic images, which were then passed through a cooperatively trained CNN (c-ConvNet). The c-ConvNet consists of two streams that exploit features for action classification by jointly optimizing a ranking loss and a softmax loss. In [363], a hybrid network that consists of multi-stream CNNs and 3D ConvLSTMs [414] was introduced to extract features from RGB and depth videos. These features were then fused via canonical correlation analysis to perform action classification. Wang et al. [361] proposed a generative framework to explore the feature distribution across the RGB and depth modalities. The fusion was performed by constructing a cross-modality discovery matrix, which was then fed to a modality correlation discovery network for the final prediction. Dhiman et al. [362] designed a twostream network composed of a motion stream and a Shape Temporal Dynamic (STD) stream to encode features from RGB and depth videos, respectively. Particularly, the motion stream takes a dynamic image from the RGB data as input and outputs classification scores. The STD network consists of the Human Pose Model in [9] as its backbone, followed by several LSTMs and a softmax layer. The final classification scores were obtained by aggregating the scores of these two streams. Transformers have also become popular for multi-modal fusion [364], [366]. To handle the problem of tightly coupled spatio-temporal representation modeling, Zhou et al. [366] proposed a decoupling and recoupling spatio-temporal representation approach, which disentangles learning spatio-temporal representations. Specifically, decoupled spatial and temporal networks are introduced to learn spatial and temporal independent discriminative features, respectively. A self-distillation-based re-coupling spatio-temporal network consisting of a multi-scale layer and Transformer blocks is then employed to model spatiotemporal dependencies. Finally, a cross-modal adaptive posterior fusion method is utilized to perform information aggregation. Li et al. [364] introduced a Transformer framework for egocentric HAR, where RGB and its corresponding depth data are processed by an inter-frame attention encoder and then fused by a mutual attention block. Fusion of RGB and Skeleton Modalities. The appearance information provided by the RGB data, and the body posture and joint motion information provided by the skeleton sequences, are complementary and useful for activity analysis. Thus, several works have investigated deep learning architectures to fuse RGB and skeleton data for HAR. Zhao et al. [369] introduced a two-stream deep network, which consists of an RNN and a CNN to process skeleton and RGB data, respectively. Both the feature fusion and the score fusion were evaluated and the former achieved better performance. The two-stream architecture was also investigated in [368], [370], where the classification scores from a CNN model [370] or an RNN model [368] trained on skeleton data and a skeleton conditioned spatio-temporal attention network trained on RGB data were fused for final classification. Zolfaghari [367] designed a three-stream 3D-CNN to process pose, motion, and the raw RGB images. The three streams were fused via a Markov chain model for action classification. Liu et al. [8] proposed a spatiotemporal LSTM network, which is able to effectively fuse the RGB and skeleton features within the LSTM unit. In order to effectively capture the subtle variations in action performing, Song et al. [371] proposed a learning framework containing two streams of skeleton-guided deep CNN to extract features from RGB and optical flow. Particularly, the local image patches around human body parts were first extracted from both RGB and flow sequences, and then fed into two individual CNNs followed by a part-aggregated pooling layer to generate two fixed-length feature vectors corresponding to RGB and flow frames. The skeleton data and the two sequences of RGB and flow feature vectors were then passed through a three-stream LSTM model followed by a fusion layer to obtain the final classification scores. Das et al. [372] introduced a pose guided spatio-temporal attention network on top of a 3D CNN model taking RGB videos as input to perform HAR. In their subsequent work [374], they extended their work in [372] by paying attention to the topology of the human body while computing the spatio-temporal attention maps. Li et al. [373] proposed a two-stream network, which consists of three main components namely the ST-GCN network [186] to extract the skeleton features, R(2+1)D network [109] to extract RGB features, and a guided block which takes these features to enhance the action-related information in RGB videos. Finally, the score fusion approach was utilized to perform classification. Cai et al. [376] introduced a two-stream GCN network, respectively taking skeleton data and joint-aligned flow patches obtained from RGB videos as the inputs for HAR. RGBPose-Conv3D [378] represents skeleton data as a 3D heatmap volume and passes the RGB video and the heatmap volume through a two-stream 3D CNN framework whose architecture is inspired by [112]. Recently, Jing and Wang [377] proposed a Two-Pathway Vision Transformer (TP-ViT) to fuse RGB and skeleton data, which consists of a high-resolution path focusing on spatial information and a high-framerate path emphasizing the temporal information. Fusion of Skeleton and Depth Modalities. The skeleton data has been shown to be a succinct yet informative representation for human behavior analysis [415], which, however, is a very sparse representation without the encoding of the shape information of the human body and the interacted objects. Besides, skeleton data is often noisy, limiting the performance of HAR when it is used alone [8].
Meanwhile, depth maps provide discriminative 3D shape and silhouette information that can be helpful for HAR. Therefore, early methods [416], [417] have attempted to fuse hand-crafted features extracted from both the skeleton and the depth sequences for more accurate HAR. Recently, deep learning-based approaches have become the mainstream for fusing the skeleton and depth modalities. In order to learn the relationships between the human body and the objects, as well as the relations between different human body parts, Rahmani et al. [379] proposed an end-to-end learning framework, which consists of a CNN followed by the bilinear compact pooling and fully-connected layers for action classification. In particular, the model takes the relative geometry between every body part and others as the skeleton features and depth image patches around different body parts as the appearance features to encode body partobject and body part-body part relations for reliable HAR. Kamel et al. [380] utilized a three-stream CNN to extract action-related features from Depth Motion Images (DMIs), Moving Joint Descriptors (MJDs), and their combination. The DMI and MJD encode the depth and skeleton sequences as images. The output scores of these three CNNs were fused to obtain the final classification scores. Zhao et al. [381] utilized two 3D CNN streams taking raw depth data and Depth Motion Maps [267] as inputs, and a manifold representation stream taking 3D skeleton as input, for feature extraction from depth and skeleton sequences. The classification scores from these three networks were fused via score multiplications. Rani et al. [382] proposed a threestream 2D CNN to perform classification on three different hand-crafted features extracted from the depth and skeleton sequences, followed by a score fusion module to obtain the final classification result. Fusion of RGB, Skeleton, and Depth Modalities. Several hand-crafted feature-based methods [412], [418] have explored the fusion of RGB, skeleton, and depth modalities to further enhance the robustness of HAR. Meanwhile, deep learning-based methods [23], [383], [384], [385], [386] have received greater attention due to their superior performance. Shahroudy et al. [383] focused on studying the correlation between modalities, and factorizing them into their correlated and independent components. Then a structured sparsity-based classifier was utilized for HAR. Hu et al. [384] learned the time-varying information across RGB, skeleton, and depth modalities by extracting temporal feature maps from each modality, and then concatenating them along the modality dimension. These multi-modal temporal features were then fed to a stack of bilinear blocks to exploit the mutual information from both modality and temporal directions. Khaire et al. [385] proposed a fivestream CNN network, which takes Motion History Images [413], Depth Motion Maps [267] (i.e., from three different viewpoints), and skeleton images generated respectively from RGB, depth, and skeleton sequences as inputs. Each CNN was trained individually, and the output scores of these five streams were fused with a weighted product model [419] to obtain the final classification scores. In their subsequent work [386], three fusion methods were explored for combining skeletal, RGB, and depth modalities. Cardenas and Chavez [23] fed three different optical spectra channels from skeleton data [203] and dynamic images from both RGB and depth videos to a pre-trained CNN to extract multi-modal features. Finally, a feature aggregation module was used to perform classification. Other Fusion Methods. Besides, some other multi-modality fusion methods were proposed [388], [389]. In the work of [388], a multi-modal fusion model was built on top of the TSN framework [50]. The RGB, depth, infrared, and optical flow sequences were passed through TSNs to perform initial classification, followed by a fusion network applied on the initial classification scores to obtain the final classification scores. Main and Noumeir [389] utilized a 3D CNN and a 2D CNN to extract features from infrared and skeleton data, respectively. The feature vectors were then concatenated and exploited by a multi-layer perceptron for HAR.

Fusion of Visual and Non-visual Modalities
Visual and non-visual modalities can also be fused to leverage their complementary discriminative capabilities for more accurate and robust HAR models.

Fusion of Audio and Visual Modalities.
Audio data provides complementary information to the appearance and motion information in the visual data. Several deep learning-based methods have been proposed to fuse these two types of modalities for HAR. Wang et al. [391] introduced a three-stream CNN to extract multi-modal features from audio signal, RGB frames, and optical flows. Both feature fusion and score fusion were evaluated, and the former achieved better performance. Owens and Efros [315] trained a two-stream CNN in a self-supervised manner to identify any misalignment between the audio and visual sequences. The learned model was then fine-tuned on the HAR datasets for audio-visual HAR. Inspired by TSN [50], Kazakos et al. [25] introduced a Temporal Binding Network (TBN), which takes audio, RGB, and optical flow as inputs for egocentric HAR. The TBN network contains a threestream CNN to fuse the multi-modal input sequences within each Temporal Binding Window, followed by a temporal aggregation module for classification. Their results showed that the TBN outperformed the Temporal Segment Network (TSN) [50] for the audio-visual HAR task. Gao et al. [316] utilized audio signal to reduce temporal redundancies in videos. Particularly, this method distills the knowledge from a teacher network trained on video clips to a student network trained on image-audio pairs for efficient HAR. The student network is a two-stream deep model which takes the starting frame and audio signal of the video clip as input followed by a feature fusion, fully-connected, and softmax layers for audio-visual HAR. Inspired by the work of [112], Xiao et al. [317] introduced a hierarchically integrated audiovisual representation framework containing slow and fast visual pathways that are deeply integrated with a faster audio pathway at multiple layers. Two training strategies, including randomly dropping the audio pathway and hierarchical audio-visual synchronization, were also introduced to enable joint training of audio and video modalities.
Recently, Alfasly et al. [394] adopted a transformer model, i,e., BERT [420], to obtain the sentence-based semantic embeddings of each textual label in audio and video datasets. A Semantic Audio-Video Label Dictionary (SAVLD) is built in which each video label is mapped into k audio labels of which they all are considered semantically similar. Then, an Irrelevant Modality Dropout (IMD) module was proposed to select the most relevant audio modality embedding for each video. Finally, both video modality embedding and the selected audio modality embedding are fused for action classification. Zhang et al. [396] proposed an audio-adaptive model leveraging the rich audio information to adjust the visual representation, along with an audioinfused recognizer to maintain domain-irrelevant features. Likewise, Planamente et al. [393] proposed a novel Relative Norm Alignment (RNA) loss to align audio and visual modalities for egocentric HAR. Besides, Chen and Ho [395] presented a Multi-Modal Video Transformer named MM-ViT for compressed video action recognition by factorizing the self-attention mechanism of ViT [175] over the space, time and modality dimensions. Fusion of Acceleration and Visual Modalities. Some handcrafted feature-based methods [421], [422] have exploited the fusion of acceleration and visual modalities for HAR. Besides, many deep learning methods [397], [398], [399], [400], [401], [402] have also been proposed recently. Dawar et al. [398] represented inertial signal as an image, and utilized two CNNs to fuse the depth images and inertial signals using score fusion. Due to the limited depth-inertial training data, Dawar et al. [397] proposed a data augmentation framework based on depth and inertial modalities, which were separately fed to a CNN and a CNN+LSTM respectively. The scores of the two models were fused during testing for better classification. Wei et al. [399] respectively fed the 3D video frames and 2D inertial images to a 3D CNN and a 2D CNN for HAR, and the score fusion achieved better performance than feature fusion. Several depth-inertial fusion techniques have also been investigated in [400], [401] by designing different two-stream CNN architectures, where the inertial signal was transformed into images using the technique in [404]. Recently, Ijaz et al. [403] proposed a multi-modal Transformer for nursing action recognition, where the correlations between skeleton and acceleration data are modeled by transformers. Other Fusion Methods. Memmesheimer et al. [407] transformed the skeleton, inertial, and WiFi data to a color image, which was fed to a CNN to perform HAR. Imran and Raman [282] designed a three-stream architecture, where a 1D CNN, a 2D CNN, and an RNN were used for gyroscopic, RGB, and skeleton data, respectively. Several fusion methods were adopted to predict the final class label, while fusing features using canonical correlation analysis achieved the best performance. Zou et al. [341] designed a twostream architecture, named WiVi, which takes the RGB and WiFi frames as inputs of the C3D and CNN streams. A multi-modal score fusion module was built on top of the network to perform HAR. Recently, Doughty and Snoek [410] leveraged adverb pseudo-labels in RGB videos for semi-supervised fine-grained HAR. Shi et al. [408] proposed Multi-modal GCNs to explore modality-aware multi-action relations from RGB, audio, and text data. Akbari et al. [409] presented a Video-Audio-Text Transformer (VATT) taking the linear projection of video, audio, and text data as the input to extract multi-modal representations. VATT quantifies the granularity of the different modalities, and is trained by adopting the Noise Contrastive Estimation (NCE) that aligns video-audio pairs as well as video-text pairs.
Although the aforementioned multi-modality fusionbased HAR methods have achieved promising results on some benchmark datasets, the task of effective modality fusion is still largely open. Specifically, most of the existing multi-modality methods have complicated architec-tures which require high computational costs. Thus efficient multi-modality HAR also needs to be addressed.

CO-LEARNING
Co-learning explores how knowledge learned from auxiliary modalities can be used to assist the learning of a model on another modality [358]. Transferring knowledge among different modalities can overcome the shortcomings of a single data modality and enhance its performance. Unlike fusion methods, in co-learning methods, the data of the auxiliary modalities is only required during training rather than testing. This is particularly beneficial in cases where some modalities are missing during testing. Co-learning can also benefit the learning of a certain modality with fewer samples, by leveraging other correlated modalities with richer samples for assisting model training.

Co-Learning with Visual Modalities
Most of the co-learning-based HAR methods focused on colearning with visual modalities, such as RGB with depth modalities, and RGB with skeleton modalities. Co-Learning with RGB and Depth Modalities. Knowledge transfer [358] between the RGB modality which captures appearance information, and depth data which encodes 3D shape information, has been shown to be useful for improving the representation capability of each modality for HAR, especially when one of these modalities has a limited amount of annotated data for training. There have been some methods that used hand-crafted features for co-learning [411], [412]. Recently, Garcia et al. [423], [424], [425] proposed several deep learning-based HAR methods based on cross-modality knowledge distillation. In [423], a knowledge distillation framework was proposed to distill knowledge from a teacher network taking depth videos as input to a RGB-based student network. The knowledge distillation was achieved by forcing the feature maps and prediction scores of the student network to be similar to the teacher network. While in [425], an adversarial learningbased knowledge distillation strategy was proposed to learn the student stream. In their subsequent work [424], a threestream network taking RGB, depth, and optical flow data as inputs, was trained using a cooperative learning strategy, i.e., the predicted labels generated by the modality stream with the minimum classification loss were used as an additional supervision for the training of other streams. Co-Learning with RGB and Skeleton Modalities. Colearning between the RGB and skeleton modalities has also been investigated in existing works [24], [26], [426]. Mahasseni and Todorovic [24] utilized a CNN+LSTM network to perform classification based on RGB videos, and an LSTM model trained on skeleton data to act as a regularizer by forcing the output features of the two models (i.e., CNN+LSTM and LSTM) to be similar. Thoker and Gall [426] utilized TSN [50] as the teacher network taking RGB videos, and an ensemble of student networks (e.g., ST-GCN [186] and HCN [208]) handling skeleton data, to perform knowledge distillation for cross-modality HAR. Specifically, each student network was trained from the supervisory information provided by the teacher network as well as other student networks. In [26], a supervision signal from an auxiliary LSTM network taking the skeleton data as input was fed to a two-stream CNN+LSTM architecture taking RGB and optical flow sequences as inputs. The auxiliary supervision signal forces the model to align the distributions of the auxiliary (skeleton) and source (RGB and optical flow) modalities, which thus enhances the representation of the RGB and optical flow modalities. Hong et al. [427] exploited the Video Pose Distillation (VPD) strategy in a teacher-student architecture, where the teacher network, i.e., an off-the-shelf pose estimator, provides a weak supervision for training the student network extracting robust pose representation from RGB videos. Other Co-Learning Methods. Besides using RGB with depth or RGB with skeleton data for co-learning, other visual modalities have also been investigated. For example, Wang et al. [428] learned a transferable generative model which takes an infrared video as input and generates a fake feature representation of its corresponding RGB video. The discriminator consists of two sub-networks, i.e., a deep binary classifier which attempts to differentiate between the generated fake features and the real RGB feature representation, and a predictor which takes both the infrared video representation and the generated features as inputs to perform action classification. Chadha et al. [429] embedded the PIX2NVS emulator [430] (i.e., converting pixel domain video frames to neuromorphic events) into a teacher-student framework, which transfers knowledge from a pre-trained optical flow teacher network to a neuromorphic event student network.

Co-Learning with Visual and Non-visual Modalities
There are also several works on co-learning between visual and non-visual modalities [356], [431], [432]. Kong et al. [356] and Liu et al. [431] trained multiple teacher networks on non-visual modalities, such as acceleration, gyroscope, and orientation signal, to teach an RGB video-based student network through knowledge distillation with an attention mechanism. In [356], the knowledge was transferred by first integrating the classification scores of the teachers with attention weights learned by feeding the concatenated classification scores of the teachers to a feed-forward neural network. Then the attended-fusion scores were used as an additional supervision to train the student network. In [431], knowledge distillation was achieved by forcing the visual explanations (i.e., attention maps that highlight the important regions for the classification scores) of both the student and teacher networks to be similar to bridge the modality gap. In another work, Perez et al. [432] proposed a knowledge distillation framework to transfer knowledge from a teacher network trained on RGB videos to a student network taking the raw sound data as input. Specifically, an objective function [433], which encourages the student model to predict the true labels as well as matching the soft labels provided by the teacher network, was utilized. Recently, Yang et al. [434] proposed a Cross-modal Interactive Alignment (CIA) model for unsupervised domain adaptive video action recognition. Specifically, CIA consists of a Mutual Complementarity (MC) module to refine the transferability of each modality and a Spatial Consensus (SC) module to achieve cross-modal consensus.
Besides fusing multiple modalities and transferring knowledge between different modalities, there are also a few works that leverage the correlation between different modalities for self-supervised learning. For example, Alwassel et al. [435] leveraged unsupervised clustering in the audio/video modality as the supervisory signal for the video/audio modality, respectively. The audio-visual correspondence and temporal synchronization have also been used as self-supervised signals to learn discriminative audio and visual features [436], [437]. Several other works [438], [439] leverage narrations of videos as a weak supervisory signal to jointly learn video and text representations for action recognition and detection. Miech et al. [438] introduced a text-video embedding model, which is trained from the caption-clip pairs using the max-margin ranking loss for learning joint representations of video and language automatically. In their further study [439], a novel training loss derived from Multiple Instance Learning and Noise Contrastive Estimation, was introduced to address misalignments in narrated videos. Besides, Sun et al. [440] introduced a joint visual-linguistic model by extending BERT [420] to videos for zero-shot HAR. Likewise, Lin et al. [441] proposed a cross-modal Transformer architecture leveraging videos and text labels for zero-shot HAR. Specifically, a ResNet-Transformer integrates the learning of visual representations and visual-semantic associations into a unified architecture. At inference, the model only takes a new single-modality input (unseen action video) as input to produce its visual representation for zero-shot HAR.

DATASETS
A large number of datasets have been created to train and evaluate the HAR methods. Table 6 lists a series of benchmark datasets. In particular, the attributes of these datasets are also summarized.

DISCUSSION
In the previous sections, we review the methods and datasets for HAR with various data modalities. Below we discuss some of the potential and important directions that could need further investigation in this domain. Datasets. Large and comprehensive datasets generally have a vital importance for the development of HAR, especially for the deep learning-based HAR methods. There are many aspects which indicate the quality of a dataset, such as its size, diversity, applicability, and type of modality. Despite the large number of existing datasets that have advanced the HAR area greatly, to further facilitate the research on HAR, new benchmark datasets are still required. For example, most of the existing multi-modality datasets were collected in controlled environments, where actions were usually performed by volunteers. Thus collecting multi-modality data from uncontrolled environments to achieve large and challenging benchmarks, for further promoting multi-modality HAR in practical applications, can be important. Besides, the construction of large and challenging datasets for every person's action recognition in crowded environments can also be further investigated. Group action recognition [489], [490] and human-human interaction recognition [491], [492] tasks also demand diverse datasets.
Multi-modality Learning. As discussed in Section 3, several multi-modality learning methods, including multimodality fusion and cross-modality transfer learning, have been proposed for HAR. The fusion of multi-modality data which can often complement each other, results in improvements of HAR performance, while co-learning can be used to handle the issue of the lack of data of some modalities. However, as pointed out by [493], many existing multimodality methods are not as effective as expected owing to a series of challenges, such as over-fitting. This implies there remain opportunities to design more effective fusion and co-learning strategies for multi-modality HAR.
Efficient Action Analysis. The superior performance of many HAR methods is built on high computational complexity, while efficient HAR is also crucial for many real-life practical applications. Hence how to reduce the computational costs and resource consumption (e.g., CPU, GPU, and energy consumption), and achieve efficient and fast HAR, deserve further studies [134], [238].
Early Action Recognition. Early action recognition (action prediction) enables recognition when only a part of the action has been performed, i.e., recognizing an action before it has been fully performed [494], [495], [496], [497]. This is also an important problem due to its relevance in some applications, such as online human-robot interaction and early alarm in some real-life scenarios.
Few-shot Action Analysis. It can be difficult to collect a large amount of training data (especially multi-modality data) for all action classes. To handle this issue, one of the possible solutions is to take advantage of few-shot learning techniques [498], [499]. Though there have been some attempts for few-shot HAR [100], [185], considering the significance of handling the issues of scarcity of data in many practical scenarios, more advanced few-shot action analysis can still be further explored.
Unsupervised and Semi-supervised Learning. Supervised learning methods, especially deep learning-based ones, often require a large amount of data with expensive labels for model training. Meanwhile, unsupervised and semisupervised learning techniques [500], [501], [502], [503] often enable to leverage the availability of unlabelled data to train the models, which significantly reduces the requirement of large labeled datasets. Since unlabelled action samples are often easier to be collected than labeled ones, unsupervised and semi-supervised HAR are also an important research direction that is worthy of further development.

CONCLUSION
HAR is an important task that has attracted significant research attention in the past decades, and various data modalities with different characteristics have been used for this task. In this paper, we have given a comprehensive review of HAR methods using different data modalities. Besides, multi-modality recognition methods, including fusion and co-learning methods, have been also surveyed.
Benchmark datasets have also been reviewed, and some potential research directions have been discussed.