{'fraction_non_alphanumeric': 0.0321459918080749, 'fraction_numerical': 0.019821533060269163, 'mean_word_length': 4.830490405117271, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Improving Action Quality Assessment using ResNets and Weighted Aggregation

Action quality assessment (AQA) aims at automatically judging human action based on a video of the said action and assigning a performance score to it. The majority of works in the existing literature on AQA transform RGB videos to higher-level representations using C3D networks. These higher-level representations are used to perform action quality assessment. Due to the relatively shallow nature of C3D, the quality of extracted features is lower than what could be extracted using a deeper convolutional neural network. In this paper, we experiment with deeper convolutional neural networks with residual connections for learning representations for action quality assessment. We assess the effects of the depth and the input clip size of the convolutional neural network on the quality of action score predictions. We also look at the effect of using (2+1)D convolutions instead of 3D convolutions for feature extraction. We find that the current clip level feature representation aggregation technique of averaging is insufficient to capture the relative importance of features. To overcome this, we propose a learningbased weighted-averaging technique that can perform better. We achieve a new state-of-the-art Spearman’s rank correlation of 0.9315 (an increase of 0.45%) on the MTL-AQA dataset using a 34 layer (2+1)D convolutional neural network with the capability of processing 32 frame clips, using our proposed aggregation technique.


I. INTRODUCTION
Action quality assessment (AQA) addresses the problem of developing a system that can automatically judge the quality of an action performed by a human. This is done by processing a video of the performance and assigning a score to it. The motivation to develop such a system stems from its potential use in applications such as health care [1], sports video analysis [2], skill discrimination for a specific task [3], assessing the skill of trainees in professions such as surgery [4]. Such a system might also have use in video retrieval systems based on action quality based ranking or rehabilitation assisting programs. Despite this, there have been relatively few works in the literature investigating this problem. However, in recent years, some works [3], [5]- [8] have given attention on this problem. Most of the currently available datasets dedicated for action quality assessment contain videos of athletes performing some action (diving, snowboarding, etc.) and corresponding scores assigned by expert human judges.
Almost all the works in this domain have treated the problem of assigning a score to a video of human action performance as a regression problem [3], [5]- [7]. Most approaches boil down to extracting higher-level features from a set of frames belonging to an RGB video of the action and finally training a linear-regressor to predict a score based on these features. Most of them [5]- [7] utilize a convolutional neural network [9] to extract complex higher-level features to train the linear-regressor on. More specifically, the best performing models [5], [6], [10] make use of the C3D network [11].
In the image classification domain, the introduction of ResNets translated into better feature extraction and higher performance not only in image classification but also in related image-based tasks such as semantic segmentation, object recognition [12]. 3D versions of ResNets capable of processing videos have been proposed and shown to achieve state-of-the-art results in the task of action recognition [13]. A slight variation of this, called (2+1)D ResNets, also has been proposed by Tran et al. [14] which decouples the spatial and temporal dimensions of convolution. The authors demonstrated superior performance on action recognition datasets with this architecture.
We hypothesize that utilizing a residual connection based deep neural network would extract features from video clips that are richer in quality compared to the ones extracted using the C3D network. The reasoning behind our hypothesis is the fact that ResNets have more convolutional layers than C3D, which can extract higher-level features than C3D. This is supported by the findings reported in Hara et al. [13].

Feature Extractor
Weight Decider Aggregator Predicted Score Regressor Fig. 1. Overview of our work: Generally, the input video is divided into clips. A feature extractor extracts features from these clips. These features are then aggregated into a video level feature vector. A linear-regressor predicts action quality scores based on this feature vector. We improve the feature extractor by using a ResNet instead of the commonly used C3D. The newly introduced Weight-Decider proposes weights based on the clip level features for better aggregation.
The authors demonstrate that in the task of action recognition when trained on big-scale datasets (such as Kinetics [15] and Sports-1M [16]), C3D based architectures are outperformed by ResNet based architectures. Also, it is customary to divide a large video into small clips, process each with feature extractor CNNs, and then aggregate the clip level features through averaging. For AQA, processing each and every frame in the video is very important for producing an accurate score prediction. Hence, we think, processing bigger clips should further enrich the quality of clip level features. Furthermore, using a sophisticated method to aggregate the information extracted from the clips should increase the accuracy of the score.
However, the question remains: does the current big scale datasets dedicated to AQA contain enough data to effectively train deep ResNets? Pretraining on large scale actionrecognition datasets [15], [16] might help the ResNets learn to extract good quality features from currently available AQA datasets. But the question remains at which depth does the network start overfitting on the data. That is, is there an optimum depth for the ResNets? We further wonder whether using (2+1)D convolution has any advantage over 3D convolution in action quality assessment.
To answer the posed questions and to validate our hypothesis, we test various 3D and (2+1)D ResNet feature extractors to find answers to these questions. MTL-AQA dataset [10] with 1412 samples of diving is the biggest dataset published to date focusing on action quality assessment. We find that 3D and (2+1)D ResNets of depth 34 and 50 with pretraining on bigscale action recognition datasets have performance comparable to the state of the arts on the MTL-AQA dataset. We see that (2+1)D and 3D convolutions perform fairly similarly. However, for 34 layer (2+1)D ResNets, we experiment with 3 different versions that can process 8, 16, or 32 frame clips at once and find the 32 frame clip version to clearly outperform the rest. It even outperforms deeper ResNets. Our results further suggest that processing longer clips is more beneficial than going deeper with convolutions. We propose a novel aggregation scheme where the clip level extracted feature vectors are multiplied with weight vectors proposed by a shallow neural network, which we call Weight-Decider (WD), before they are summed together. The WD proposes the weight vector based on the corresponding feature vector. We find that this scheme generally boosts performance. The 34 layers (2+1)D ResNet with WD processing 32 frame clips achieves a Spearman's rank correlation of 0.9315 on the MTL-AQA dataset, achieving a new state-of-the-art. Contributions: • To the best of our knowledge, this is the first work to do a comparative analysis of the effect of the depth, convolution type, and input clip size of the ResNet feature extractor on the final quality of the scores predicted in AQA. • A learning-based aggregation technique novel in the field of AQA. • One of our approaches outperform all the previous works in this domain on the MTL-AQA dataset.
II. RELATED WORK AQA: Pirsiavash et al. [5] were the first to work in the AQA domain. The authors proposed a novel dataset containing videos of Diving and Figure-skating. The dataset was annotated with action quality scores by expert human judges. The authors used Discrete Cosine Transform (DCT) and human pose extracted from the videos as high-level features and used this as input to a Support Vector Regressor, which predicted the score. Venkataraman et al. [17] used approximate entropy of the pose and found the results to improve. The primary limitation of these techniques is their dependence on the extracted human pose, which is difficult to do accurately for videos containing complex body motions or athletics, and their lack of capability of picking up important visual cues from surroundings.
Looking at more recent results, we can see a trend of using the Convolutional 3D (C3D) network [11] as a feature extractor. This is not surprising as C3D has been shown to be very successful at capturing salient motion cues and appearance through 3D spatio-temporal convolutions on the related task of action recognition. Parmar and Morris [6] proposed three architectures, C3D-SVR, C3D-LSTM, C3D-LSTM-SVR, which all used features extracted from short video clips using C3D network, and later aggregated them for predicting an action score either using Support Vector Regressor (SVR), or Long Short-Term Memory (LSTM), or a combination of both. Xiang et al. [7] used Pseudo-3D (P3D) network [18] as feature extractor as well as broke the video into action specific segments and fused the extracted features later. Li et al. [19] used 9 different C3D networks to process 9 different clips corresponding to different stages of diving. These features are aggregated and processed through convolutional and fully-connected layers to produce a final AQA score. Parmar and Morris [3] proposed a new dataset containing samples from 7 different scores to see if knowledge transfer is possible in AQA. The authors used the C3D-LSTM model and trained it to predict scores across 6 different actions. In a later work, Parmar and Morris [10] took a multitask approach towards action quality assessment. Here they released a novel AQA dataset called MTL-AQA. Their proposed multi-task learning based C3D-AVG-MTL framework outperformed all previous works on the MTL-AQA dataset. Their approach was to extract features using a C3D network and aggregate these through averaging, training 3 task dependant heads containing linear-regressor, Softmax function, and Gated Recurrence Unit [20] to do score prediction, action classification, and to generate captions in that order. Tang et al. [8] took a probabilistic approach to address the inherently uncertain nature of predicting action quality score. They used I3D [21] architecture to extract clip level features, averaged as aggregation, and finally predicted parameters of a probabilistic distribution form which the final score prediction was sampled.
Our proposed approach differs from these works in that we plan to use 3D and (2+1)D ResNets as feature extractor and we aggregate these features using the WD network, which is a light-weight and learning-based feature aggregation scheme.
Spatiotemporal models: Processing video data to encode them into higher-level representations is a very important part of computer vision. The specific problem of action-recognition from videos has received significant attention in this case. Initially, this domain was dominated by the tracking of spatiotemporal points [22]- [25]. However, with the emergence of deep-learning and the introduction of large-scale datasets, deep convolutional neural networks [9] have become the default choice when it came to extracting video representations. Here, the most popular architectures are 2D CNNs (2 stream approach using RGB frame and optical flow [26], [27]), Temporal-Segment-Networks [28] and 3D CNNs (C3D [11], I3D [21], P3D [18], (2+1)D CNN [14]).
3D convolutions are becoming the most popular choice with the improvement of processing power and GPUs. C3D [11] network can process video clips of 16 frames to find higherlevel representations. 3D ResNets [13] try to use residual connections to go deeper with convolutions without degrading performance. The I3D network [21] has also achieved competitive performance. All these networks are trained on big scale action recognition datasets such as Kinetics [15] and Sports-1M [16]. More recently, (2+1)D convolution blocks have been shown to optimize faster and achieve better results in action recognition by Tran et al. [14]. They propose (2+1)D ResNets that utilize (2+1)D convolution blocks. Open-source implementation and pretrained weights on large-scale action recognition datasets (Kinetics [15], Sports-1M [16], IG-65M [29]) exist for various depths and input clip size. This makes these networks ideal for transfer learning to a related task such as Action Quality Assessment from videos.
Diba et al. [30] use a method called "STC Block" which is similar to our proposed aggregation method for action recognition. However, they utilize this on spatial and temporal features separately after each convolution layer for action recognition, whereas our method is applied to the output of the CNN to aggregate clip level spatiotemporal features for performing AQA.

III. OUR APPROACH
A. General Pipeline Overview . Each of the N clips is then processed by a feature extractor. This feature extractor takes in a clip C i and outputs a feature vector f i . For the feature extractor, we utilize ResNets [12]. We experiment with 3D ResNets [12] and (2+1)D ResNets [14] with varying depth and input clip size. We take the final average-pooling layer features from the ResNet and pass them through a number of additional fully-connected layers. The final output is a 128-dimensional feature vector. Next, we aggregate these clip level features to obtain a global video level representation. Finally, a linear-regressor is trained to predict the score from the video level feature representation. Following the majority of previous works [3], [5], [6], [10], [18], we model the problem as linear regression. This makes sense as the action quality score is a real number as opposed to one from a set of discrete values. If the dataset includes the difficulty degree of the action, then we multiply the final score prediction with difficulty degree just like real world judges do.
To experiment with the relation of the ResNet feature extractor's depth with the AQA pipeline's ability to learn, we experiment with 3 different depths: 1) 34 layer: We experiment with both 34 layer 3D ResNets and (2+1)D ResNets. The only difference being 3D ResNet uses 3 × 3 × 3 convolution kernels, on the other hand (2+1)D ResNet uses a 1 × 3 × 3 convolution followed by 3 × 1 × 1 convolution. We take the final average-pool layer outputs, which is a feature vector of size 512, and pass it through 2 back-to-back fully-connected layers having 256 and 128 units. The final 128 dimensional feature vector is defined as the output of the feature extractor. The 3D ResNet takes input 16 frame clips, making the input size 16×3×112×112. Due to the availability of pre-trained weights, we additionally experiment with 3 different variations of (2+1)D 34-layer ResNet, each processing different sized clips.   All of our proposed frameworks work with videos of 96 frames. It divides the video into non-overlapping and contiguous clips. In the case of 16 frame feature extractors, the original video is divided into 6 clips, for 32 frame feature extractors, the video is divided into 3 clips, and for 8 frame feature extractors, the video is divided into 12 clips. Each clip is processed sequentially using the same feature extractor, later aggregating the 128-dimensional feature vectors to obtain one video level feature descriptor to train the linear-regressor on.

B. Feature Aggregation
Most of the previous works dealing with AQA, process the entire input video by first dividing it into multiple smaller clips of equal size, due to memory and computational budget. Most CNNs are designed to process 8, 16, or 32 frames at once. Then the features extracted by the CNN are aggregated to form a video level feature description and further regression is done on this feature vector.
The best performing works aggregated the clips by simply averaging them [6], [8], [10]. One work titled C3D-MSCADC architecture by Parmar and Morris [10] tried to use Multiscale Context Aggregation with Dilated Convolutions [31] to aggregate the clipwise features, but this was outperformed by C3D-AVG-MTL architecture in the same work where simple averaging was done to aggregate. Some other works [3], [6] aggregated using LSTMs [32]. However, LSTM networks, which make sense in theory because of their ability to handle time sequences, perform worse due to the lack of big-enough datasets dedicated to AQA.
We propose that simply averaging the clip-wise features is an ineffective measure. It should not be able to preserve the temporal information available in the data. This follows from the fact we could change the order of the clip level features and we will still get the same average and hence the same score prediction. Furthermore, if we look at real-world judging schemes, we will see those expert judges focus more on mistakes and deviations and these have a bigger impact on the score. Hence we think, a weighted averaging technique might be more suitable, as the linear-regressor in the final layer will be able to base its decision on features more important from each clip.  More concretely, if the feature vector extracted from clip C i is f i , we propose the video level feature vector as where w i is a weight vector corresponding to the feature vector f i and represents Hadamard Product or elementwise multiplication. The weights corresponding to W i is of the same dimensions as f i and learned using a small neural network of 4 layers. This smaller neural network takes as input 128-dimensional feature vector f i and runs it through fully connected layers containing 64, 32, 64, and 128 layers. All but the final layers employ a ReLU activation function. The architecture is explained in figure 3. Finally, to ensure the weights corresponding to the same element of different weight vectors sum up to one, a softmax is applied along with the corresponding elements of all the weight vectors. We call this shallow neural network Weight-Decider(WD).
(3) Finally, the linear-regressor can predict the score using the feature vector f video as proposed by equation (1).

A. Datasets and Evaluation metrics:
MTL-AQA Dataset [10]: This is the biggest dataset dedicated to AQA. It contains 1412 video samples. All the samples are of Olympic dive collected from 16 events. The videos contain 103 frames. They have varying perspectives and camera angles. The dataset contains samples of both male and female athletes, individual and synchronous diving, 3m and 10m platform diving, the final action quality scores from the Olympic judges, task difficulty level, commentary from the broadcast of the event, and fine-grained action labels. The most popular split used by the contemporary works divides the dataset into a 1059 sample training set and 353 sample testing set as proposed by Parmar and Morris [10]. We use the same split in our experiments.
Evaluation Metric: In line with previously published literature, we use Spearman's rank correlation as the evaluation metric. Spearman's rank correlation ranges from -1 to +1, and it measures the correlation between two sequences containing ordinal or numerical data. No underlying distribution of the data is assumed other than that there is at least an ordinal relation in the data. It is calculated using the equation ρ = Spearman's rank correlation d i = The difference between the ranks of corresponding variables n = Number of observations B. Implementation Details: We implemented our proposed methods using PyTorch [33]. We experimented with various types of 3D ResNets [13] and (2+1)D ResNets [14] as feature extractors. All the 3D ResNets and (2+1)D ResNets processing 16 frame clips and were pre-trained on kinetic-700 [15] action recognition dataset 1 . The (2+1)D Resnets processing 8 frame clips and 32 frame clips were pre-trained on IG-65M dataset [29] and fine tuned on Kinetic-400 [15] action recognition dataset 2 . For each ResNet feature extractor, we separately experimented using both averaging as feature aggregation and WD as feature aggregation.
We did temporal augmentation by randomly picking an ending frame from the last 6 and chose the preceding 96 frames for processing. The video frames are resized to 171 × 128 and a center crop of 112 × 112 was taken. We performed spatial augmentation by random horizontal flipping. The 96 frames were divided into 6 clips in case of 16 frame clips, 12 clips in case of 8 frame clips and 3 clips in case of 32 frame clips. All the clips were non-overlapping and contained frames in the original order present in the video. Batch-normalization [34] was used in the convolutional layers for regularization.
We defined the loss function as a sum of L2 and L1 loss between the predicted score and ground-truth score as Parmar and Morris [10] suggested to and then we optimized this loss. For each experiment, the entire network was trained endto-end using the ADAM optimizer [35] for 50 epochs. We found through trial and error that a learning rate of 0.0001 for modules with randomly initialized weights and 0.00001 for modules with pretrained weights to produce the best results. We used training batches of size 2 and test batches of size 5.

C. Results on MTL-AQA Dataset:
Because the MTL-AQA dataset contains difficulty-degree of the action, and real-world judges multiply their score with difficulty-degree to produce a final score, we choose to multiply the output score of the linear-regressor with difficultydegree.   In Table I, we present the experiment results of varying the depth of the ResNet feature extractor as well as varying the aggregation scheme. All the ResNets, in this case, are initialized with Kinetics-700 [15] Action Recognition dataset pretrained weights. We can see that 34 layer ResNet with WD as aggregation performs the best with a Spearman's correlation of 0.8990. This leads us to conclude that when initialized with pretrained weights on a related task like action recognition, the MTL-AQA dataset has enough data to train at least 34 layer deep ResNets without overfitting. Interestingly, increasing the depth to 50 layers somewhat decreases the Spearman's correlation. However, the results are still competitive. With 101 layer deep ResNets, even when initialized with pretrained weights from Kinetics Action Recognition dataset [15], overfitting occurs fairly quickly. The overfitting is also evident from the train/test curves presented in figure 4.The likely reason behind this is the increased number of parameters leading to overfitting. This leads us to establish that the current biggest AQA dataset has enough data to train a 34-layer and 50layer ResNet feature extractor with generalization, however it overfits 101-layer ResNet feature extractor. Because of how (2+1)D ResNets are designed, they have a similar parameter count to their 3D counterparts [14]. Because the overfitting is occurring due to the high parameter count, we do not repeat the experiment with a (2+1)D ResNet-101 feature extractor.
We can see from Table I that 3D and (2+1)D convolutionbased architectures with the same depth have similar performances. We can also see using WD as aggregation can give the performance a little boost in some cases. However, in case of ResNet-101, performance decreases by quite a big margin when WD aggregation is used. We think the reason might be that introducing the WD increases parameter count leading to more overfitting, which translates to worse performance.  Table I, this was the best performing model.
We present our results in Table II. We can see clearly that as the number of frames in each clip increases, the performance of the pipeline does too. We hypothesize that longer clips allow the ResNet to look for bigger patterns in the temporal dimension, which in turn enables the feature descriptors extracted by the ResNets to be more informative. This enables the linear-regressor to better discriminate between similarlooking examples with fine-grained action quality differences. It is further observable, no matter the clip size, using WD over simple averaging as aggregation gives a boost in performance. However, this performance boost is quite significant in case of 8 frame clips. We think the reasons are: • Using 8 frame clips, the 96 frame video is divided into 12 clips, which means finally 12 clip level feature descriptors need to be aggregated. On the other hand, using 32 frame clips means finally 3 clip level feature descriptors are being aggregated. Thus, whatever detrimental effect the averaging might have, it will be more prominent when the number of objects being averaged is larger, and less when this number is smaller. Hence using a 32 frame clip, the performance gained by using WD aggregation over averaging is only 0.0026 (0.28%), while in case of 8 frame clips, the performance gain is 0.0283 (3.30%). • CNNs with bigger clips as input can look at more frames, this in effect increases their temporal horizon. It follows that the feature vectors extracted would have a better encoding of action patterns across time, to begin with. Thus they perform well enough even with averaging as aggregation. But using WD increases performance nevertheless. For qualitative results, refer to Table III. Due to space constraints, we show every 16 th frame processed starting from frame 0. The blue scores correspond to score prediction produced using WD as aggregation, while the black scores correspond to score prediction produced using average as aggregation. The 8, 16, and 32 correspond to input clip sizes.
Comparison with the state of the art: In Table IV, we compare our best performing models of each depth with previous state-of-the-art works on the MTL-AQA dataset. We can see that our ResNet34(2+1)D processing 32 frame clips with WD as aggregation scheme outperforms all previous works in the literature. This shows the effectiveness of our approach.
If we further look at the results in Table I and compare with  Table IV, we see that 34 layer ResNets based pipelines achieve performance comparable to the SOTA when trained on the AQA-MTL dataset. The 50 layer ResNets perform somewhat

V. CONCLUSION
In this work, we proposed a ResNet based regression oriented pipeline for action quality assessment. We demonstrated experimentally that the MTL-AQA dataset has enough data to train 34 and 50 layer ResNet based pipelines when initialized with pretrained weights from a related task (like action recognition). Our experiments suggest processing longer clips is more effective than using deeper ResNets. We also propose a sophisticated learning-based aggregation technique called WD to replace simple averaging. Experiments show our methods to be more effective than previous works. In the future, we want to investigate with CNNs that can process longer clips (64 or higher) to see if this translates to better performance.