{'fraction_non_alphanumeric': 0.03827057587069184, 'fraction_numerical': 0.014170260573333817, 'mean_word_length': 4.836294416243655, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
TSA-Net: Tube Self-Attention Network for Action Quality Assessment

In recent years, assessing action quality from videos has attracted growing attention in computer vision community and human computer interaction. Most existing approaches usually tackle this problem by directly migrating the model from action recognition tasks, which ignores the intrinsic differences within the feature map such as foreground and background information. To address this issue, we propose a Tube Self-Attention Network (TSA-Net) for action quality assessment (AQA). Specifically, we introduce a single object tracker into AQA and propose the Tube Self-Attention Module (TSA), which can efficiently generate rich spatio-temporal contextual information by adopting sparse feature interactions. The TSA module is embedded in existing video networks to form TSA-Net. Overall, our TSA-Net is with the following merits: 1) High computational efficiency, 2) High flexibility, and 3) The state-of-the art performance. Extensive experiments are conducted on popular action quality assessment datasets including AQA-7 and MTL-AQA. Besides, a dataset named Fall Recognition in Figure Skating (FR-FS) is proposed to explore the basic action assessment in the figure skating scene.

, Non-local module [39] performs dense correlation for all features (in green). (b) For each position in the spatio-temporal tube (e.g. blue), TSA module only performs sparse correlation for all features located in the tube. After TSA operation, features in the tube can capture rich contextual information from athletes' series of movements in raw videos. Residual connections are ignored for clear display.

INTRODUCTION
In addition to identifying human action categories in videos, it is also crucial to evaluate the quality of specific actions, which means that the machine needs to understand not only what has been performed but also how well a particular action is performed. Action quality assessment (AQA) aims to evaluate how well a specific action is performed, which has become an emerging and attractive research topic in computer vision community. Assessing the quality of actions has great potential value for various real-world applications such as analysis of sports skills [22,25,27,33,40], surgical maneuver training [20,45,46] and many others [4,5].
In recent years, many methods [22,25,33,40] directly applied the network of human action recognition (HAR) such as C3D [34] and I3D [1] to AQA tasks. Although these methods have achieved considerable performance, they still face many challenges, and their performances and efficiency are indeed limited. Firstly, the huge gap between HAR and AQA should be emphasized. Models in HAR require distinguishing subtle differences between different actions, while models in AQA require evaluating a specific action's advantages and disadvantages. Therefore, the performances of existing methods are inherently limited because of the undifferentiated feature extraction of video content, which leads to the pollution of body features. It is not appropriate to apply the framework in HAR directly to AQA without any modification. Secondly, existing methods cannot perform feature aggregation efficiently. The receptive field of convolution operation is limited, resulting in the loss of long-range dependencies. RNN has the inherent property of storing hidden states, which makes it challenging to be paralleled. An effective and efficient feature aggregation mechanism is desired in AQA tasks.
To solve all challenges above, we propose Tube Self-Attention (TSA) module, an efficient feature aggregation strategy based on tube mechanism and self-attention mechanism, shown in Figure  1. The basic idea of the TSA module is straightforward and intuitive: considering that AQA models require rich temporal contextual information and do not require irrelevant spatial contextual information, we combine the tube mechanism and self-attention mechanism to aggregate action features sparsely to achieve better performance with minimum computational cost. For example, during a diving competition, the athletes' postures are supposed to raise most attentions, instead of distractors such as the audience and advertisements in the background. The merits of the TSA module are three-fold: (1)High efficiency, the tube mechanism makes the network only focus on a subset of the feature map, reducing a large amount of computational complexity compared with Non-local module. (2)Effectiveness, the self-attention mechanism is adopted in TSA module to aggregate the features in the spatio-temporal tube (ST-Tube), which preserves the contextual information in the time dimension and weakens the influence of redundant spatial information. (3)Flexibility, consistent with Non-local module, TSA module can be used in a plug-and-play fashion, which can be embedded in any video network with various input sizes.
Based on TSA module, we proposed Tube Self-Attention Network (TSA-Net) for AQA. Existing visual object tracking (VOT) framework is firstly adopted to generate tracking boxes. Then the ST-Tube is obtained through feature selection. The self-attention mechanism is performed in ST-Tube for efficient feature aggregation. Our method is tested on the existing AQA-7 [24] and MTL-AQA [25] datasets. Sufficient experimental exploration, including performance analysis and computational cost analysis, is also conducted. In addition, a dataset named Fall Recognition in Figure Skating (FR-FS) is proposed to recognize falls in figure skating. Experimental results show that our proposed TSA-Net can achieve state-of-theart results in three datasets. Extensive comparative results verify the efficiency and effectiveness of TSA-Net.
The main contributions of our work are as follows: • We exploit a simple but efficient sparse feature aggregation strategy named Tube Self-Attention (TSA) module to generate representations with rich contextual information for action based on tracking results generated by the VOT tracker. • We propose an effective and efficient action quality assessment framework named TSA-Net based on TSA module, with adding little computational cost compared with Non-local module. • Our approach outperforms state-of-the-arts on the challenging MTL-AQA and AQA-7 datasets and a new proposed dataset named FR-FS. Extensive experiments show that our method has the ability to capture long-range contextual information, which may not be performed by previous methods.

RELATED WORKS
Action Quality Assessment. Most of the existing AQA methods focus on two fields: sports video analysis [22,25,27,33,40] and surgical maneuver assessment [20,45,46]. AQA works focus on sports can be roughly divided into two categories: pose-based methods and non-pose methods. Pose-based methods [22,27,37] take pose estimation results as input to extract features and generate the final scores. Because of the atypical body posture in motion scene, the performance of pose-based methods are suboptimal. Non-pose methods exploit DNNs such as C3D and I3D to extract features directly from the raw video and then predict the final score. For example, Self-Attentive LSTM [40], MUSDL [33], C3D-AVG-MTL [25], and C3D-LSTM [23] share similar network structures, but their difference lies in the feature extraction and feature aggregation method. Although these methods have achieved significant results, the enormous computational cost of feature extraction and aggregation module limits AQA models' development. Different from the aforementioned AQA methods, our proposed TSA module can perform feature extraction and aggregation efficiently. Self-Attention Mechanism. Self-attention mechanism [36] was firstly applied on the machine translation task in neural language processing (NLP) as the key part of Transformer. After that, researchers put forward a series of transformer-based models including BERT [3], GPT [28], and GPT-2 [29]. These models tremendously impacted various NLP tasks such as machine translation, question answering system, and text generation. Owing to the excellent performance, some researchers introduce self-attention mechanism into many CV tasks including image classification [10,12,21], semantic segmentation [2,42,44] and object detection [6,11,43]. Specifically, inspired by Non-local [39] module, Huang et al. [13] proposed criss-cross attention module and CCNet to avoid dense contextual information in semantic segmentation task. Inspired by these methods, our proposed TSA-Net adopts self-attention mechanism for feature aggregation. Video Action Recognition Video action recognition is a fundamental task in computer vision. With the rise of deep convolutional neural networks (CNNs) in object recognition and detection, some researchers have designed many deep neural networks for video tasks. Two-stream networks [1,9,32] take static images and dynamic optical flow as input and fuse the information of appearance and short-term motions. 3D convolutional networks [14,15,34] utilize 3D kernels to extract features form raw videos directly. In order to meet the needs in real applications, many works [8,18,35]  focus on the efficient designing of networks recently. The proposed TSA-Net take I3D [1] network as the backbone.

APPROACH 3.1 Overview
The network architecture is given in Figure 2. Given an input video with frames = { } =1 , SiamMask [38] is used as the single object tracker to obtain the tracking results represents the tracking box of the -th frame. In feature extraction stage, is firstly divided into clips where each clip contains M consecutive frames. All clips are further sent into the first stage of Inflated 3D ConvNets (I3D) [1], resulting in features as In feature aggregation stage, the TSA module takes tracking boxes and video feature X as input to perform feature aggregation, resulting in video feature X ′ = {x ′ } =1 with rich spatiotemporal contextual information. Since the TSA module does not change the size of the input feature map, x and x ′ have the same size, i.e., x ′ ∈ R × × × . This property enables TSA modules to be stacked in multiple layers to generate features with richer contextual information. The aggregated feature X ′ is further sent to the second stage of I3D to complete feature extraction, resulting in H = {h } =1 . H is the representation of the whole video or athlete's performance.
In prediction stage (i.e., network head), average pooling operation is adopted to fuse H along clip dimension, i.e., h = 1 =1 ℎ , h ∈ R × × × . h is further fed into the MLP_Block and finally used for the prediction of different tasks according to different datasets.

Tube Self-Attention Module
The fundamental difference between TSA module and Non-local module is that TSA module can filter the features of participating in self-attention operation in time and space according to the tracking boxes information. The TSA mechanism has the ability to ignore noisy background information which will interfere with the final result of action quality assessment. This operation makes the network pay more attention to the features containing athletes' information and eliminate irrelevant background information interference.
The tube self-attention mechanism can also be called "local Nonlocal". The first "local" refers to the ST-Tube, while "Non-local" refers to the response between features calculated by self-attention operation. So the TSA module is able to achieve more effective feature aggregation on the premise of saving computing resources. TSA module consists of two steps: (1) spatio-temporal tube generation, and (2) tube self-attention operation.
Step 1: spatio-temporal tube generation. Intuitively, after obtaining tracking information and feature map X of the whole video, all features in the ST-Tube can be selected directly. Unfortunately, owing to the existence of two temporal pooling operations in I3D-stage1, the corresponding relationship between tracking boxes and feature maps is not 1 : 1 but : 1. Besides, all tracking boxes generated by SiamMask are skew, which complicates the generation of ST-Tube.
To solve these problems, we propose an alignment method which is shown in Figure 3. Since I3D-Stage1 contains two temporal pooling operations, the corresponding relationship between bounding boxes and feature map is 4:1, i.e., { , +1 , +2 , +3 } is correspond to x , . All tracking boxes should be converted into mask first, and then used to generate ST-Tube. We denote the mask of correspond to x , as , ∈ {0, 1} × . The generation process of , is as follows: Where (·, ·) function calculates the proportion of the feature grid at ( , ) covered by . If the proportion is higher than threshold , the feature located at ( , ) will be selected, otherwise it will be discarded. The proportion of each feature grid covered by box ranges from 0 to 1, so we directly took the intermediate value of = 0.5 in all experiments of this paper. Four masks are further assembled into This mask contains all location information of the features participating in self-attention operation. For the convenience of the following description, is transformed into the position set of all selected features: Where Ω , is the basic component of ST-Tube and Ω , denotes the number of selected features of x , .
Step 2: tube self-attention operation After obtaining X and Ω , , the self-attention mechanism is performed to aggregate all features located in ST-Tube, as shown in Figure 4. The formation of the TSA mechanism adopted in this paper is consistent with [39]: Where denotes the index of an output position whose response is to be computed. ( , , , ) is the input index that enumerates all positions in ST-Tube. Output feature map y and input feature map Figure 4: Calculation process of the TSA module. "⊕" denotes matrix multiplication, and "⊗" denotes element-wise sum.
Owing to the existence of tube mechanism, only the features inside the ST-Tube can be selected and participate in the calculation of self-attention.
x have the same size. (·, ·) denotes the pairwise function, and (·) denotes the unary function. The response is normalized by (x) = Ω , . To reduce the computational complexity, the dot product similarity function is adopted: Where both (·) and (·) are channel reduction transformations. Finally, the residual link is added to obtain the final X ′ : Where y denotes an embedding of y . Note that x ′ has the same size with x , so TSA module can be inserted into any position in deep convolutional neural networks. For the trade-off between computational cost and performance, all TSA modules are placed after Mixed_4e. Thus, = 4 and = = 14. Compared with the Non-local operation, the TSA module greatly reduces the computational complexity in time and space from to Note that the computational cost of TSA can only be measured after forwarding propagation because Ω , is generated from .

Network Head and Training
To verify the effectiveness of the TSA module, we extend the network head to support multiple tasks, including classification, regression, and score distribution prediction. All tasks can be achieved by changing the output size of MLP_block and the definition of the loss function. The implementation details of these three tasks are as follows:  Classification. When dealing with classification tasks, the output dimension of MLP_block is determined by the number of categories. Binary Cross-Entropy loss (BCELoss) is adopted.
Regression. When dealing with regression tasks, the output dimension of MLP_block is set to 1. Mean Square Error loss (MSELoss) is adopted.
Score distribution prediction. Tang et al. [33] proposed an uncertainty-aware score distribution learning (USDL) approach and its multi-path version MUSDL for AQA tasks. Although experiment results in [33] proved the superiority of MUSDL compared with USDL, a multi-path strategy will lead to a significant increase in computational cost. However, the TSA module can generate features with rich contextual information by adopting a self-attention mechanism in ST-Tube with less computational complexity.
To verify the effectiveness of the TSA module, we embed the TSA module into the USDL model. The loss function is defined as Kullback-Leibler (KL) divergence of predicted score distribution and ground-truth (GT) score distribution: Where is generated by MLP_block, and is generated by GT score. Note that for dataset with difficulty degree (DD), = × is used as the final predicted score.

EXPERIMENTS
We carry out comprehensive experiments on AQA-7 [24], MTL-AQA [25], and FR-FS datasets to evaluate the proposed method. Experimental results demonstrate that TSA-Net achieves state-ofthe-art performance on these datasets. In the following subsections, we first introduce two public datasets and a new dataset named Fall Detection in Figure Skating

Datasets and Evaluation Metrics
AQA-7 [24]. The AQA-7 dataset comprising samples from seven actions. It contains 1189 videos, in which 803 videos are used for training and 303 videos used for testing. To ensure the comparability with other models, we delete trampoline category because of its long time. MTL-AQA [25]. The MTL-AQA dataset is currently the largest dataset for AQA tasks. There are 1412 diving samples collected from 16 different events in MTL-AQA. Furthermore, MTL-AQA provides detailed scoring of each referee, diving difficulty degree, and live commentary. We followed the evaluation protocol suggested in [25], so that there are 1059 samples used for training and 353 used for testing. Figure Skating). Although some methods have been proposed [27,40] to evaluate figure skating skills, they are only based on long-term videos which last nearly 3 minutes. These coarse-grained methods will lead to the inundation of detailed information in a long time scale. However, these details are crucial and indispensable for AQA tasks. To address this issue, we propose a dataset named FR-FS to recognize falls in figure skating sports. We plan to start from the most basic fault recognition and gradually build a more delicate granularity figure skating AQA system. The FR-FS dataset contains 417 videos collected from FIV [40] and Pingchang 2018 Winter Olympic Games. FR-FS contains the critical movements of the athlete's take-off, rotation, and landing. Among them, 276 are smooth landing videos, and 141 are fall videos. To test the generalization performance of our proposed model, we randomly select 50% of the videos from the fall and landing videos as the training set and the testing set.   Evaluation Protocols. Spearman's rank correlation is adopted as the performance metric to measure the divergence between the GT score and the predicted score. The Spearman's rank correlation is defined as follows:

FR-FS (Fall Recognition in
Where and represent the ranking of GT and predicted score series, respectively. Fisher's z-value [23] is used to measure the average performance across multiple actions.

Implementation Details
Our proposed methods were built on the Pytorch toolbox [26] and implemented on a system with the Intel (R) Xeon (R) CPU E5-2698 V4 @ 2.20GHz. All models are trained on a single NVIDIA Tesla V100 GPU. Faster-RCNN [30] pretrained on MS-COCO [19] is adopted to detect the athletes in all initial frames. All videos are normalized to = 103 frames. For all experiments, the I3D[1] pretrained on Kinetics [16] is utilized as the feature extractor. All videos are select from high-quality sports broadcast videos, and Table 4: Comparison with state-of-the-arts on MTL-AQA.
Considering the complexity differences between datasets, we adopt different experimental settings. In AQA-7 and MTL-AQA datasets, all videos are divided into 10 clips consistent with [33]. Random horizontal flipping and timing offset are performed on videos in training phase. Training epoch is set to 100. All video score normalization are consistent with USDL [33]. In FR-FS dataset, all videos are divided into 7 segments to prevent overfitting. Training epoch is set to 20.

Results on AQA-7 Dataset
The TSA module and the Non-local module are embedded after Mixed_4e of I3D to create TSA-Net and NL-Net. Experimental results in Table 1 show that TSA-Net achieves 0.8476 on Avg. Corr. , which is higher than 0.8102 of USDL. TSA-Net outperforms USDL in all categories except the snowboard. This is mainly caused by  the size issue: the small size of the target leads to the small size of the ST-Tube, resulting in invalid feature enhancement (AQA-7 snow. #056 in Figure 5). Note that the TSA module is used in a plugand-play fashion, comparative experiments in Table 1 can also be regarded as ablation studies. Therefore, we didn't set up a separate part of ablation in this paper.
The effect of different number of TSA module. Inspired by the multi-layer attention mechanism in Transformer [36], we stack multiple TSA modules and test these variants on AQA-7. Experimental results in Table 2 show that TSA-Net achieves the best performance when = 2. Benefit from the feature aggregation operations conducted by two subsequent TSA modules, the network can capture richer contextual features compared with USDL. When = 3, the performance of the model becomes worse, which may be caused by overfitting. Computational cost analysis. Computational cost comparison results are shown in Table 3. Note that only the calculation of TSA module or Non-local module is counted, not the whole network. Compared with NL-Net, TSA-Net can reduce the computation by 68.7% on average and bring 0.0058 AVG. Corr. improvement. This is attributed to the tube mechanism adopted in TSA module, which can avoid dense attention calculation and improve performance simultaneously. Among all categories in AQA-7, the TSA module saves up to 87% of the computational complexity on skiing and snowboard. Such a large reduction is caused by the small size of the ST-Tube. However, the small ST-Tube will hinder the network from completing effective feature aggregation and ultimately affect the final performance. This conclusion is consistent with the analysis of the results in Table 1.

Results on MTL-AQA Dataset
As shown in Table 4, the TSA-Net and NL-Net is compared with existing methods. Regression network head and MSELoss are adopted in two networks. Experimental results show that both TSA-Net and NL-Net can achieve state-of-the-art performance and the NL-Net is better. The performance fluctuation of TSA-Net is mainly caused by different data distribution between two datasets. Videos in MTL-AQA have higher resolution (640x360 to 320x240) and broader field  is achieved when = 2 (i.e., TSAx2-Net), while TSA-Net with only one TSA module achieves minimum MSE and computational cost simultaneously. This phenomenon is mainly caused by low computational complexity of TSA-Net. The sparse feature interaction characteristics of TSA module achieve more efficient feature enhancement and have the ability to avoid overfitting. Although the performance of TSA-Net can be improved by increasing the number of TSA modules, it will increase computational cost. To achieve the balance between computational cost and performance, we only take = 1 in all subsequent experiments.

Results on FR-FS Dataset
In FR-FS dataset, we focus on the performance improvement that the TSA module can achieve. Therefore, Plain-Net and TSA-Net are implemented, respectively. The former does not adopt any feature enhancement mechanism, while the latter is equipped with a TSA module. As shown in Table 6, TSA-Net outperforms Plain-Net by 4.33%, which proves the effectiveness of TSA module. Visualization of Temporal Evolution. A case study is also conducted to further explore the performance of TSA-Net. Two representative videos are selected and the prediction results of each video clip are visualized in Figure 7, . All clip scores are obtained by deleting the temporal pooling operation in Plain-Net and TSA-Net.
In the failure case #308-1, both Plain-Net and TSA-Net can detect that the athlete falls in the fourth chip which highlighted in red, but only TSA-Net gets the correct result in the end (0.9673 for Plain-Net and 0.2523 for TSA-Net). The TSA mechanism forces the features in ST-Tube interact with each other in the way of self-attention,  Figure 7: Case study with qualitative results on FR-FS. The failure case #308-1 is above the timeline, while the successful case #241-3 is below the timeline.
which makes TSA-Net regard the standing up and adjusting actions after falling as errors in clip 5 to 7. It seems that TSA-Net is too strict in fall recognition, but the analysis in the successful case #241-3 has overturned this view. Two models get similar results, except for the second clip (colored in blue), which contains the take-off and rotation phase. Plain-Net has great uncertainty for the stationarity of take-off phase, while TSA-Net can get high confidence results. Based on visual analysis and quantitative analysis, it can be concluded that the TSA module is able to perform feature aggregation effectively and obtain more reasonable and stable prediction results.

Analysis and Visualization
Reasons for choosing tracking boxes over pose estimation. In sports scenes, high-speed movements of the human body will lead to a series of challenges such as motion blur and self-occlusion, which eventually result in failure cases in pose estimation. Results in Figure 6 show that Alphapose [7] cannot handle these situations properly. The missing of human posture and background audience posture interference will seriously affect the evaluation results. Previous studies in FineGym [31] have come to the same results as ours. Based on these observations, we conclude that methods based on pose estimation are not suitable for AQA in sports scenes. Missing boxes and wrong poses will significantly limit the performance of the AQA model. Therefore, we naturally introduce the VOT tracker into AQA tasks. The proposed TSA-Net achieves significant improvement in AQA-7 and MTL-AQA compared to pose-based methods such as Pose+DCT [27] and ST-GCN [41] as shown in Table 1 and 4. These comparisons show that the TSA mechanism is superior to the posture-based mechanism in capturing key dynamic characteristics of human motion. Visualization on MTL-AQA and AQA-7. Four cases are visualized in Figure 5. The tracking results generated by SiamMask are very stable and accurate. The final predicted scores are very close to the GT score since the TSA module is adopted. Interestingly, as shown in Figure 5, the VOT tracker can handle various complex situations, such as the disappearance of athletes (#02-32), drastic changes in scale (#056) and synchronous diving (#082). These results show that the tracking strategy perfectly meets the requirements of AQA tasks and verify the effectiveness of the TSA module.

CONCLUSION
In this paper, we present a Tube Self-Attention Network (TSA-Net) for action quality assessment, which is able to capture rich spatiotemporal contextual information in human motion. Specifically, a tube self-attention module is introduced to efficiently aggregate contextual information located in ST-Tube generated by SiamMask. Sufficient experiments are performed on three datasets: AQA-7, MTL-AQA, and a dataset proposed by us named FR-FS. Experimental results demonstrate that TSA-Net can capture long-range contextual information and achieve high performance with less computational cost. In the future, an adaptive mechanism of ST-Tube will be explored to avoid the sensitivity of the TSA-Net to the size issue.