{'fraction_non_alphanumeric': 0.032424190114506014, 'fraction_numerical': 0.013723459347488347, 'mean_word_length': 4.900356475980309, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Action Transformer: A Self-Attention Model for Short-Time Pose-Based Human Action Recognition

Deep neural networks based purely on attention have been successful across several domains, relying on minimal architectural priors from the designer. In Human Action Recognition (HAR), attention mechanisms have been primarily adopted on top of standard convolutional or recurrent layers, improving the overall generalization capability. In this work, we introduce Action Transformer (AcT), a simple, fully self-attentional architecture that consistently outperforms more elaborated networks that mix convolutional, recurrent and attentive layers. In order to limit computational and energy requests, building on previous human action recognition research, the proposed approach exploits 2D pose representations over small temporal windows, providing a low latency solution for accurate and effective real-time performance. Moreover, we open-source MPOSE2021, a new large-scale dataset, as an attempt to build a formal training and evaluation benchmark for real-time, short-time HAR. The proposed methodology was extensively tested on MPOSE2021 and compared to several state-of-the-art architectures, proving the effectiveness of the AcT model and laying the foundations for future work on HAR.


I. INTRODUCTION
Human Action Recognition (HAR) is a problem in computer vision and pattern recognition that aims to detect and classify human actions. The ability to recognize people inside a scene and predict their behavior is fundamental for several applications, such as robotics [36], surveillance and security [46], [50], autonomous vehicles [33], [7], and automatic video captioning [55], [45], [49]. Most previous works dealing with HAR adopted datasets characterized by samples with a long temporal duration [25], [47], [32]. Thus, HAR has been mainly treated as a post-processing operation, classifying complex and long-lasting human actions by exploiting past and future information. Conversely, in this work, we focus on short-time HAR, which aims at continuously classifying actions within short past time steps (up to a second). This approach is fundamental to target real-time applications: in robotics, for example, HAR problem should be solved promptly to react to sudden behavioral changes, only relying on near past information.
In this paper, we propose a new model for HAR called the Action Transformer (AcT), schematized in Fig.1, inspired This work has been developed with the contribution of the Politecnico di Torino Interdepartmental Centre for Service Robotics PIC4SeR (https://pic4ser.polito.it) and SmartData@Polito (https://smartdata.polito.it). As for Vision Transformer models [18], [44], [15], a learnable positional embedding is added to each input token. Then, only the output class token is passed through a multi-layer perceptron head to obtain the final class prediction.
by the simple and prior-free architecture of the Vision Transformer [18]. The Transformer architecture [48] has been one of the most important deep learning advances of the last years in natural language processing (NLP). Moreover, multi-head self-attention has proven to be effective for a wide range of tasks besides NLP, e.g. image classification [18], [44], [15], image super-resolution [54], [56], and speech recognition [17]. Furthermore, optimized versions of the Transformer have been developed for real-time and embedded applications [8], proving that this architecture is also suitable for Edge AI purposes. Recently, many models for Human Action Recognition have proposed the integration of attention mechanisms with convolutional and recurrent blocks to improve the accuracy of models. However, solutions that rely exclusively on self-attention blocks have not been investigated for this task yet.
With AcT, we apply a pure Transformer encoder derived architecture to action recognition obtaining an accurate and low-latency model for real-time applications. We study the model at different scales to investigate the impact of the number of parameters and attention heads. We use the new single-person, short-time action recognition dataset MPOSE2021 as a benchmark and exploit 2D human pose representations provided by two existing detectors: OpenPose [9] and PoseNet [34]. Moreover, we compare AcT with other state-of-the-art baselines to highlight the advantages of the proposed approach. To highlight the effectiveness of self-attention, we conduct a model introspection providing visual insights of the results and study how a reduction of input temporal sequence length affects accuracy. We also conduct extensive experimentation on model latency on lowpower devices to verify the suitability of AcT for real-time applications.
The main contributions of this work can be summarized as follows: • We study the application of the Transformer encoder to 2D pose-based HAR and propose the novel AcT model, proving that fully self-attentional architectures can outperform existing convolutional and recurrent models for pose-based HAR. • We introduce MPOSE2021, a dataset for real-time short-time HAR, suitable for both pose-based and RGBbased methodologies. It includes 15429 sequences from 100 actors and different scenarios, with limited frames per scene (between 20 and 30). In contrast to other publicly available datasets, the peculiarity of having a constrained number of time steps stimulates the development of real-time methodologies that perform HAR with low latency and high throughput. The rest of the paper is organized as follows. Section II briefly reports the relevant research on HAR and selfattention deep learning methodologies. Section III presents the MPOSE2021 dataset, highlighting features and elements of novelty with respect to other existing datasets. Then, Section IV describes the architecture of the proposed AcT model, shortly recalling the main features of the Transformer encoder. Section V summarizes the experimentation conducted to verify the effectiveness of the proposed approach, gives a visual insight into the model functioning, studies its behavior under temporal information reduction, and measures its latency for real-time applications. Section VI draws some conclusions for this work and indicates future research directions.

II. RELATED WORKS
Human Action Recognition algorithms aim at detecting and classifying human behaviors based on a different source of information. Works in this field can be mainly subdivided into two broad categories: video-based and depthbased methodologies [42]. In both, the input consists of a sequence of points representing body joints, extracted from the corresponding RGB frame [28], in the case of video-based methods, or a point cloud, in the case of depth-based ones. In the latter, body joints are provided as 3D coordinates (skeletal data), such as those captured by Kinect sensors [38], [29], that can be possibly projected onto the 2D space [11], [53]. On the opposite, in this work, we focus on a videobased analysis, where body joints are already provided as 2D coordinates (pose data) by pose detection algorithms such as OpenPose [9] and PoseNet [34]. This characteristic makes 2D HAR methodologies applicable in a great range of applications using a simple RGB camera. Conversely, skeletal data require particular sensors to be acquired, such as Kinect or other stereo-cameras. That raises substantial limitations, such as availability, cost, limited working range (up to 5-6 meters in the case of Kinect [27]), and performance degradation in outdoor environments.
Angelini et al. first collected the MPOSE dataset for videobased short-time HAR [2], obtaining 2D poses by processing several popular HAR video datasets with OpenPose. Moreover, the authors proposed ActionXPose, a methodology that increases model robustness to occlusions. The sequences were classified using MLSTM-FCN [26], which exploits a combination of 1D convolutions, LSTM [23], and Squeeze-and-excitation attention [24]. The same authors successively applied their approach to anomaly detection [4], [3] and expanded MPOSE with the novel ISLD and ISLD-Additional-Sequences datasets. Conversely, Yan et al. [53] first applied OpenPose to extract 2D poses from the Kinetics-400 RGB dataset [11] and used graph convolutions to capture spatial and temporal information. Similarly, Shi et al. [39], [40] applied graph convolutions to pose information using two streams that extract information from both joints and bones to represent the skeleton. Hao et al. [21] introduced knowledge distillation and dense-connectivity to explore the spatiotemporal interactions between appearance and motion streams along different hierarchies. At the same time, Si   al. [41] put together a hierarchical spatial reasoning network and temporal stack learning network to extract discriminative spatial and temporal features. On the other hand, Liu et al. [30] proposed an ensemble of two independent bonebased and joint-based models using a unified spatial-temporal graph convolutional operator. Conversely, Cho et al. [14] first applied self-attention [48] to the skeletal-based HAR problem. More recently, Plizzari et al. [35], inspired by Bello et al. [6], employed self-attention to overcome the locality of the convolutions, again adopting a two-stream ensemble method, where self-attention is applied on both temporal and spatial information.
Unlike previous methodologies, this paper presents an architecture for HAR entirely based on the Transformer encoder, without any convolutional or recurrent layer. Moreover, we focus on verifying the suitability of our method for low-latency and real-time applications. For this reason, we introduce a new 2D pose-based dataset specifically designed for short-time HAR. Traditional datasets used by previous works, such as 3D NTU RGB+D [38], [29], or Kinetics-Skeleton [11], [53], include long temporal sequences which must be entirely scanned to make the correct classification. In contrast, the proposed MPOSE2021 dataset includes samples with a temporal duration of at most 30 frames. That makes it a new and more suitable benchmark to test the short-time and low-latency performance of HAR models.

III. THE MPOSE2021 DATASET
In this section, MPOSE2021 is presented as an RGBbased dataset designed for short-time, pose-based HAR. As in [2], [4], [3], video data have been previously collected from popular HAR datasets (precursors), i.e. Weizmann [20], i3DPost [19], IXMAS [51], KTH [37], UTKinetic-Action3D (RGB only) [52], UTD-MHAD (RGB only) [12], ISLD, and ISLD-Additional-Sequences [2]. Due to the heterogeneity of actions across different datasets, labels are remapped to a list of 20 common classes. Actions that cannot be remapped accordingly are discarded. Therefore, precursor videos are divided into non-overlapping samples (clips) of 30 frames each whenever possible and retaining tail samples with more than 20 frames. A visual representation of the number of frames per sample for each sub-dataset of MPOSE2021 is shown in Fig. 4. The peculiarity of a reduced number of time steps contrasts with other publicly available datasets and stimulates the development of methodologies that require low latency to perform a prediction. That would largely benefit many real-world applications requiring real-time perception of the actions performed by humans nearby.
Subsequently, clips not containing a single action are discarded. Moreover, ambiguous clips are relabelled whenever possible or discarded otherwise. This process leads to 15429 samples, where each sample represents a single actor performing a single action. The total number of distinct actors in MPOSE2021 is 100, and the number of samples for each action is reported in Fig.2, which also shows the distribution of precursor datasets.
OpenPose [9] and PoseNet [34] are used to extract landmarks from MPOSE2021 samples. The average confidence is computed for each sample as the mean across landmarks and frames. It turns out that the two detectors achieve different average confidences based on the considered precursor dataset. The box plot of Fig.3 describes the comparison statistics.
Due to the significant sample heterogeneity and the high number of actors, three different training/testing splits are defined for MPOSE2021, i.e. Split1, Split2, and Split3, by randomly selecting 21 actors for testing and using the rest of them for training. This division makes the proposed dataset a challenging benchmark to effectively assess and compare the accuracy and robustness of different methodologies. Moreover, the suggested evaluation procedure requires testing a target model on each split using ten validation folds and averaging the obtained results. That makes it possible to produce statistics and reduces the possibility of overfitting the split testing set with an accurate choice of hyperparameters.
With MPOSE2021, we aim at providing an end-to-end and easy-to-use benchmark to robustly compare state-of-the-art methodologies for the short-time human action recognition task. We thus release a code repository 1 to access the different levels of the dataset (video, RGB frames, 2D poses).
Moreover, we open source a practical Python package to access, visualize and preprocess the poses with standard functions. The Python package can be easily installed with the command pip install mpose.

IV. ACTION TRANSFORMER
In this section, we describe the architecture of the AcT network ( Fig.1) briefly recalling some preliminary concepts associated with the Transformer model [48].

A. AcT Architecture
A video input sequence with T frames of dimension H × W and C channels X rgb ∈ R T ×H×W ×C is pre-processed by a multi-person 2D pose estimation network that extracts 2D poses of dimension N × T × P, where N is the number of human subjects present in the frame and P is the number of keypoints predicted by the network. The Transformer architecture receives as input a 1D sequence of token embeddings, so each of the N sequences of pose matrices X 2Dpose ∈ R T ×P is separately processed by the AcT network. Nevertheless, at inference time, all detected poses in the video frame can be batch-processed by the AcT model, simultaneously producing a prediction for all N subjects. Firstly, the T poses are mapped to a higher dimension D model using a linear projection map W l 0 ∈ R P×D model . As in BERT [16] and Vision Transformers [18], [44], [8], [10], a trainable vector of dimension D model is added to the input T sequence. This class token [CLS] forces the self-attention to aggregate information into a compact high-dimensional representation that separates the different action classes. Moreover, positional information is provided to the sequence with a learnable positional embedding matrix X pos ∈ R (T +1)×D model added to all tokens. The linearly projected tokens and [CLS] are fed to a standard Transformer encoder F Enc of L layers with a postnorm layer normalization [48], [13], obtaining where X L ∈ R (T +1)×D model is the overall representation produced by the Transformer encoder at its last layer. Finally, only the [CLS] token x cls is fed into a linear classification head MLP head that performs the final class prediction whereẑ is the output logit vector of the model. At training time, the supervision signal comes only from the [CLS] token, while all remaining T tokens are the only input of the model. It is important to notice how the nature of the network makes it possible to accept a reduced number of frames as input even if trained with a fixed T . That gives an additional degree of freedom at inference time, making AcT more adaptive than other existing models.
The resulting network is a lightweight solution capable of predicting actions for multiple people in a video stream with high accuracy. The advantage of building on 2D pose estimations enables effective real-time performance with low latency and energy consumption.

B. Transformer Architecture
The Transformer encoder [48] is made of L layers with alternating H multi-head self-attention and feed-forward blocks. Dropout [43], Layernorm [5], and residual connections are applied after every block. The overall sequence of blocks of a Transformer encoder is summarized on the left of Fig.5.
Each feed-forward block is a multi-layer perceptron with two layers and GeLu [22] non-linearity. The first layer expands the dimension from D model to D ml p = 4 · D model and applies the activation function. On the other hand, the second layer reduces the dimension back from D ml p to D model . Instead, the multi-head QKV self-attention mechanism (MSA) is based on a trainable associative memory with keyvalue vector pairs. For the l-th layer of the Transformer encoder and the h-th head, queries (Q), keys (K) and values

Model
H D model D ml p L Parameters AcT-µ  1  64  256  4  227k  AcT-S  2  128  256  5  1,040k  AcT-M  3  192  256  6  2,740k  AcT-L  4  256  512  6 4,902k (V) are computed as Q = XW Q , K = XW K and V = XW V respectively, where W Q , W K and W V belong to R D model ×D h , being D h the dimension of the attention head. So, for each self-attention head (SA) and element in the input sequence X ∈ R T ×D model we compute a weighted sum over all values V. The attention weights A i j are derived from the pairwise similarity between two elements of the sequence and their respective query Q i and key K j representations. Therefore, it is possible to compute the attention weights A ∈ R T ×T for the l-th layer as and the final weighted sum SA as We perform this SA operation for all the H heads of the lth layer. We concatenate the results and linearly project the output tensor to the original dimension model as where W MSA ∈ R H·D h ×D model . All operations are schematized on the right side of Fig. 5. The input tensor X is firstly projected to T ×H·D h , and then a SA operation is performed to all H splits of the resulting projected tensor. Finally, all head outputs are concatenated and linearly projected back to D model . So, the attention mechanism takes place in the timedomain, allowing a global class embedding representation by correlating different time windows.
In order to reduce the number of hyperparameters and linearly scale the dimension of AcT, we fix D model /H = 64, varying H, D ml p , and L to obtain different versions of the network. A simple grid search using train and validation sets is performed to determine lower and upper bounds for the four parameters. In particular, in Table I, we summarize the four AcT versions with their respective number of parameters. The four models (micro, small, medium, and large) differ for their increasing number of heads and layers, substantially impacting the number of trainable parameters.

V. EXPERIMENTS
This section describes the main experiments conducted to study the advantages of using a fully self-attentional model for 2D pose-based HAR. First, the four variants of AcT described in Section IV are compared to existing state-of-theart methodologies and baselines on MPOSE2021. Only for , we use additional ensemble versions of our model named AcT-µ (x n), n being the number of ensembled instances. Then, we further analyze the behavior of the network in order to get a visual insight of the attention mechanism and study the performance under a reduction of temporal information. Finally, we study model latency for all the designed architectures with two different CPU types, proving that AcT can easily be used for real-time applications.

A. Experimental Settings
In the following experiments, we employ both the Open-Pose and PoseNet versions of the MPOSE2021 dataset. Either set of data has T = 30 and P = 52 or 68 features, respectively. In particular, for OpenPose we follow the same preprocessing of [2] obtaining 13 keypoints with four parameters each (position x, y and velocities v x , v y ), while PoseNet samples contain 17 keypoints with the same information. The training set is composed of 9421 samples and 2867 for testing. The remaining 3141 instances are used as validation to find the most promising hyperparameters with a grid search analysis. All results, training, and testing code for the AcT model are open source and publicly available 2 .
In Table I are summarized all the hyperparameters for the four versions of the AcT architecture and in Table II all settings related to the training procedure. The AdamW optimization algorithm [31] is employed for all training with the same scheduling proposed in [48], but with a step drop of the learning rate λ to 1e-4 at a fixed percentage (80%) of the total number of epochs. We employ the TensorFlow 2 3 framework to train the proposed network on a PC with 32-GB RAM, an Intel i7-9700K CPU, and an Nvidia 2080 Super GP-GPU. Following the previously defined benchmark strategy, the total training procedure for the four versions takes approximately 32 hours over the three different splits. We exploit publicly available code for what concerns other state-of-the-art models and use the same hyperparameters and optimizer settings described by the authors in almost all the cases. The only exception is made for learning rate, epochs number, and batch size to adapt the methodologies to our dataset and obtain better learning curves.

B. Action Recognition on MPOSE2021
We extensively experiment on MPOSE2021 considering some baselines, common HAR architectures, and our pro- posed AcT models. We report the mean and standard deviation of 10 models trained using different validation splits to obtain statistically relevant results. The validation splits are constant for all the models and correspond to 10% of the train set, maintaining the same class distribution. The benchmark is executed for both OpenPose and PoseNet data and repeated for all the three train/test splits provided by MPOSE2021. The baselines chosen for the benchmark are a Multilayer Perceptron (MLP), a fully convolutional model (Conv1D), and REMNet, which is a more sophisticated convolutional network with attention and residual blocks proposed in [1] for time series feature extraction. In particular, the MLP is designed as a stack of three fully connected (FC) layers with 512 neurons each, followed by a dropout layer and a final FC layer with as many output nodes as classes. Instead, the Conv1D model is built concatenating five 1D convolutional layers with 512 filters alternated with batch normalization stages and followed by a global average pooling operator, a dropout layer, and an FC output stage as in the MLP. Finally, the configuration used for REMNet consists of two Residual Reduction Modules (RRM) with a filter number of 512, followed by dropout and the same FC output layer as in the other baselines.
Regarding state-of-the-art comparisons, four popular models used for multivariate time series classification, and in particular HAR, are reproduced and tested. Among those, MLSTM-FCN [26] combines convolutions, spacial attention, and an LSTM block, and its improved version ActionXPose [2] uses additional preprocessing, leading the model to exploit more correlations in data and, hence, be more robust against noisy or missing pose detections. On the other hand, MS-G3D [30] uses spatial-temporal graph convolutions to make the model aware of spatial relations between skeleton keypoints, while ST-TR [35] joins graph convolutions with Transformer-based self-attention applied to both space and time. As the last two solutions also propose a model ensemble, these results are further compared to AcT ensembles made of 2, 5, and 10 single-shot models. We also report the achieved balanced accuracy for each model and use it as the primary evaluation metric to account for the uneven distribution of classes.
The results of the experimentation for OpenPose are reported in Table III and, in synthesis, in Fig.6. The fully convolutional baseline strongly outperforms the MLP, while REMNet proves that introducing attention and residual blocks further increases accuracy. As regards MLSTM-FCN and ActionXPose, it is evident that explicitly modeling both spatial and temporal correlations, made possible by the two separate branches, slightly improves the understanding of actions with respect to models like REMNet. MS-G3D, in its joint-only (J) version, brings further accuracy improvement by exploiting graph convolutions and giving the network information on the spatial relationship between keypoints. On the other hand, ST-TR shows performance comparable to all other single-shot models, despite, as MS-G3D, taking advantage of graph information.
The proposed AcT model demonstrates the potential of pure Transformer-based architectures, as all four versions outperform other methodologies while also showing smaller standard deviations. Moreover, even the smallest AcT-µ (227k parameters) is able to extract general and robust features from temporal correlations in sequences. Increasing the number of parameters, a constant improvement in balanced accuracy can be observed for splits 3, while split 1 and 2 present oscillations. The difference between splits reflects how much information is conveyed by training sets and how much the model can learn from that. So, it is evident that AcT scales best on split three because it presents complex correlations that a bigger model learns more easily. On the contrary, it seems AcT-µ is able to extract almost all the information relevant for generalization from split 2, as the accuracy only slightly increases going towards more complex models.
As explained in Section II, ST-TR exploits an ensemble of two networks modeling spatial and temporal sequence correlations, respectively. Moreover, MS-G3D leverages further information such as skeleton graph connections and the position of bones in one of the ensembled networks. Ensembles are very effective in reducing model variance and enhancing performance by exploiting independent representations learned by each network, so it is unfair to compare them with single architectures. For this reason, we create three ensemble versions of AcT-µ to have an even confront, with 2, 5, and 10 instances, respectively. To compute ensemble predictions, we average the output logits of the network instances before passing through a softmax function. The results reported at the bottom of Table III show that AcT-µ (x2) outperforms MS-G3D (J+B) in all the benchmarks except for balanced accuracy in split 3, despite having less than one-tenth of its parameters. Finally, the ensembles AcT-µ (x5) and AcT-µ (x10), made of 5 and 10 instances respectively, achieve even higher accuracy on all the splits with only around 1 to 2 million parameters. That proves how the balancing effect of the ensemble enhances model predictions even without feeding the network additional information.
As PoseNet data is mainly dedicated to real-time and Edge AI applications, only the models designed for this purpose have been considered in the benchmark, excluding MS-G3D, ST-TR, and AcT-L. In general, the results give similar insights. The tested models are the same as the previous case, with all the necessary modifications given by the different input formats. In the MLP case, however, performance seriously degrades as networks strongly tend to overfit input data after a small number of epochs, so the results are not included in Table IV. That is caused by the fact that PoseNet is a lighter methodology developed for Edge AI, and hence noisy and even missing keypoint detections are more frequent. That results in less informative data and emphasizes the difference between sequences belonging to different sub-datasets, confusing the model and inducing it to learn very specific but unusable features. The MLP is too simple and particularly prone to this kind of problem. Naturally, all the models are affected by the same problem, and the balanced accuracy on PoseNet is generally lower. The same considerations made for OpenPose apply in this case, where AcT outperforms all the other architectures and demonstrates its ability to give an accurate and robust representation of temporal correlations. Also, it is interesting to notice that Conv1D performs better than REMNet, proving to be less prone to overfitting, and that standard deviations are more significant than in the OpenPose case.

C. Model Introspection
In order to have an insight into the frames of the sequence the AcT model attends to, we extract the selfattention weights at different stages of the network. In Fig.7, MPOSE2021 test samples are propagated through the AcT-M model, and attention weights A of the three distinct heads are presented. It can be seen that the model pays attention to specific frames of the sequence when a specific gesture defines the action. On the other hand, attention is much more spread across the different frames for more distributed actions such as walking and running. Moreover, it is clear how the three heads mostly focus on diverse frames of the sequence. Finally, the rightmost columns show three attention maps of failed predictions. In these last cases, Fig. 8. Self-attention of the [CLS] token, computed as the normalized sum of the last layer of the different heads. Scores give a direct insight into the frames exploited by AcT to produce the classification output. The example clearly shows how bending positions are more insightful for the network to predict the jumping-in-place action. In the image, the attention score defines the skeleton alpha channel. attention weights are less coherent, and the model cannot extract a valid global representation of the scene.
Instead, in Fig.8, the last-layer self-attention scores of the [CLS] token are shown together with the RGB and skeleton representations of the scene. The scores are computed as the normalized sum of the three attention heads and give a direct insight into the frames exploited by AcT to produce the classification output. It can be seen that bent poses are much more informative for the model to predict the jumpingin-place action.
Moreover, we analyze the behavior of the network under a progressive reduction of temporal information. That can be easily done without retraining due to the intrinsic nature of AcT. In Fig.9, we present how the test-set balanced accuracy is affected by frame dropping. The two curves show a reduction starting from the beginning and the end Fig. 9. AcT-M balanced accuracy with an incremental reduction of temporal information. Due to the intrinsic nature of the network, it is possible to reduce the number of temporal steps without a retraining or any kind of explicit adaptation. of the temporal sequence, respectively. It is interesting to notice how the performance of AcT degrades with an almost linear trend. That highlights the robustness of the proposed methodology and demonstrates the possibility of adapting the model to applications with different temporal constraints.
Finally, we also study the positional embeddings of the AcT-M model by analyzing their cosine similarity, as shown in Fig.10. Very nearby position embeddings demonstrate a high level of similarity, and distant ones are orthogonal or in the opposite direction. This pattern is constant for all T frames of the sequence, highlighting how actions are not particularly localized and that relative positions are essential for all the frames.

D. Latency Measurements
We test the performance of all the considered models for real-time applications. To do so, we use the TFLite Benchmark 4 tool, which allows running TensorFlow Lite models on different computing systems and collecting statistical results on latency and memory usage. In our case, two CPUs are employed to measure model speed both on a PC and a mobile phone: an Intel i7-9700K for the former and the ARMbased HiSilicon Kirin 970 for the latter. In both experiments, the benchmark executes 10 warm-up runs followed by 100 consecutive forward passes, using 8 threads.
The results of both tests are reported in Fig. 11, where only the MLP has been ignored because, despite being the fastest-running model, its accuracy results are much lower than its competitors. The graph shows the great computational efficiency of Transformer-based architectures, whereas convolutional and recurrent networks result in heavier CPU usage. Indeed, in the Intel i7 case, REMNet achieves almost the same speed as AcT-S, but its accuracy is 2% lower. Moreover, AcT-µ is able to outperform REMNet, running at over four times its speed. MLSTM-FCN and ActionXPose, being smaller models, achieve lower latencies than the baselines:  Study of the latency of different tested models on a highperformance Intel CPU and on a mobile phone equipped with an ARMbased CPU. the former stays between AcT-µ and Act-S, while the latter performs similarly to AcT-S. Those results are remarkable but still outperformed by AcT-µ both on accuracy and speed.
The difference with the baselines is even more evident on the ARM-based chip, as convolutional architectures seem to perform poorly on this kind of hardware. Indeed, REMNet and Conv1D run as fast as AcT-M with significantly lower accuracies, and AcT-µ is ten times quicker. Nothing changes for what concerns MLSTM-FCN and ActionXPose, less accurate and three times slower than AcT-µ.

VI. CONCLUSION
In this paper, we explored the direct application of a purely Transformer-based network to human action recognition. We introduced the AcT network, which significantly outperforms commonly adopted models for HAR with a simple and fully self-attentional architecture. In order to limit computational and power requests, building on previous HAR and pose estimation research, the proposed methodology exploits 2D skeletal representations of short time sequences, providing an accurate and low latency solution for real-time applications. Moreover, we introduced MPOSE2021, a large-scale opensource dataset for short-time human action recognition, as an attempt to build a formal benchmark for future research on the topic. Extensive experimentation with our methodology clearly demonstrates the effectiveness of AcT and poses the basis for meaningful impact on many practical computer vision applications. In particular, the remarkable efficiency of AcT could be exploited for Edge AI, achieving good performance even on computationally limited devices. Future work may further investigate the AcT architecture with 3D skeletons and long sequence inputs. Moreover, relational information between points of the estimated pose are currently scarcely exploited by our methodology. So, future investigations may attempt to include skeleton graphs as prior knowledge in the positional embedding of the network without impacting latency.