{'fraction_non_alphanumeric': 0.0290945573112616, 'fraction_numerical': 0.013920240782543265, 'mean_word_length': 4.909225639125602, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A Spatio-Temporal Attention Convolution Block for Action Recognition

We propose a simple and effective 3D neural network module (STAT) embedded in spatiotemporal attention for action recognition. For a given intermediate feature map, our module sequentially infers the distribution of attention along the two dimensions of space and time, and multiplies it with the current feature map in the form of residual to achieve adaptive generation of the next stage feature map. STAT is a 3D convolution general module combined with attention. It is compatible with any 3D convolution network and can easily replace the 3D convolution kernel. The additional overhead it generates is negligible, and it can be trained end-to-end together with ordinary 3DCNN. By comparing the performance of the currently popular 3D networks on the UCF101 and HMDB51 datasets, experiments show that STAT has certain improvements on most 3D networks, which proves that STAT has a certain universality.


Introduction
The emergence of the convolutional neural network represented by AlexNet [1] indicates that deep learning brings a revolution to complex AI tasks. Among them, human action recognition is challenging and very interesting work. Although the network ResNet-152 [2] has been successful in 2D images, it's hard to deal with data with temporal relationships, such as videos. But the time series on video is the key issue for video analysis. In the video, the target character is unique in space and time, even in the same space and time period. It's natural for people to regard video as a time series of spatial problems, simultaneously modeling of time and space using 3D CNN, and an effective method for action recognition is obtained [3][4] [5]. Some work like (I3D [4]) has achieved better results in motion recognition, compared with the traditional method (IDT [6]), its improvement is not as good as that of image recognition. According to the network architecture proposed by C3D [7], stacking the 3D convolution layer is more and more common. Although 3DCNN Networks can model video spatiotemporal better than the traditional methods, its time complexity and flexibility are cursed. Video usually has a complex background and is accompanied by the interaction between people and other objects, which has a lot of redundancy in space-time, which will affect the differential representation of the two states by the model. The human visual system [8] is rather sensitive to changes in time and space, and this gift allows humans to observe other people's movements at certain times. Therefore, it is a wise choice to introduce the attention mechanism into the model of video action recognition.
In this paper, we introduce a Spatial-Temporal Attention (STAT) module to apply the attention machine to the video action recognition model, the test case is shown in Figure 1. STAT models the Figure 1. Time and space focus on the pole vault. From this picture we can see that spatial attention has been tracking the movements of the task, while temporal attention is shown in the middle two rows with a smaller range of movements, thus giving less attention than the other two rows.
video in space and time respectively. STAT only appears in a part of 3D convolution, it can replace some 3D convolution functions or insert into the 3DCNN model, which can not only capture time and motion information but also increase the flexibility of the model without increasing the complexity of the model. Finally, we improve the performance of most current 3D models with fewer parameters and less time.

Related work
With the development of computer science, human beings have accumulated a large amount of data in life and production. After great success in two-dimensional images, people have shifted their focus to video analysis. The core of video analysis is human action recognition. The accuracy and rapidity of action recognition will directly affect the follow-up results of the video analysis system, and most of the research on video analysis focuses on spatial and temporal modeling. Before the rise of deep learning, most video classifications were based on features designed by hand and typical machine learning methods. Some proposed video representations include SIFT-3D [9], spatiotemporal interest points (STIPs) [10] and Cuboids [11]. However, the most typical one is the improved Dense Trajectory (IDT), which is widely regarded as the most advanced [6]. It shows a powerful effect in video classification. After coding, it is input into the classification function to get the final classification category. The emergence of deep learning pushes video classification to a new stage. Convolution neural network has achieved great success in image processing and has become a popular method for image processing [12][13][14]. Ji et al.[15]proposed to extract the temporal and spatial characteristics of video data through 3D convolution core, capture the motion information of the video stream, and finally combine the generated channel information to obtain the final feature description. Karpathy et al. [16] Studied the fusion of time information and spatial information and hoped that the time characteristic could enhance the connectivity between convolutional layers and calculate the activation amount through spatiotemporal convolution. However, the result was that the network performance of processing a single frame was similar to that of the whole spatiotemporal video volume. In the field of image classification, the training model obtained by ImageNet can be well applied to other fields and tasks. This has inspired J. Carreira et al. [4] to make a large-scale video data set, on which the trained models can perform better on a small set (HMD51 [17], UCF101 [18], etc.).After analyzing the most advanced dual-stream network and 2D convolution+LSTM network at that time [19][20][21][22], a dual-stream model based on 3D convolution was designed. However, after pre-training on Kinetics data sets and migration to HMDB51 and UCF101, the effect was unexpectedly wonderful. However, it seems that the pooling or RNN operation added on the 2D network cannot well extract the connection between the low-level information of the video. Therefore, some R2+1D [23][24][25]  nonlinearity. In this work, we pay more attention to the flexibility and lightweight of the network, using (R2 + 1D) as our backbone network, even if it can be applied in other action recognition models.

Figure 2. The structure of Spatial(up) and Temporal(down) Attention Block
With the development of attention mechanisms, the class of deep learning has been promoted to a higher level. The combination of video and attention is based on RNN structure [26][27][28], using only spatial or temporal attention structures [29,30],or Spatio-Temporal attention [22,31]. Inspired by the classical non-local means in computer vision, Xiaolong Wang et al. [32] proposed a non-local operation to capture the long-distance dependence and establish the connection between two pixels with a certain distance on the image and the connection between two frames in the video. It has successfully applied self-attention mechanisms to data in video structures and can be inserted into many computer vision structures, which is undoubtedly a successful start. Compared with the high complexity of non-local, Woo S, et al [33] proposed a lightweight universal space-channel attention mechanism. With the addition of the CBAM module, the model has better performance and interpretation and pays more attention to the target object itself.
Our work focuses on embedding the spatial-temporal attention mechanism into a flexible R2+1D network. On the one hand, it aims to model space and time more accurately and extract more useful information. The other is to increase the flexibility of the network, reduce the complexity of the 3D network, and reduce the network operation time.

Spatial-Temporal Attention block
Suppose there is an intermediate feature map , where C,T,H,W denote the channel number , the frame number of the input video clip, the height and width of the frame-level feature map respectively. Take F as the input to SATA, compared with R2 + 1D structure, STAT contains a 2D spatial attention map and a 1D temporal attention mechanism , as illustrated in Fig.1. It should be noted that for the previous 3D or dual-stream networks, the feature map of each channel is obtained through 3D convolution kernel or 2D convolution incorporating time perception. Therefore, it captures the combined information between the target object and the environment. CBAM [33] focuses on the channel and spatial information and ignores the time information. The objective of STAT is to integrate attention into the network module to refine the feature map of different stages, as Figure 2 shows. In terms of space, the model is expected to focus more on the target and the interaction between the target and the environment, and for the time, it is expected to focus more on the moment when the behavior occurs. The manifestation of attention function is as follows: Where G denotes attention machine, and specifies the element-wise multiplication operation.
To integrate attention into convolution operation and promote effective and efficient learning, we consider a convolution module embedded in the attention mechanism, which integrates the separated convolution and attention mechanism. Therefore, the process of redefining the above feature attendance into a two-step sequence is as follows: Where S F and T F denote the spatial and temporal attention, respectively. The function f represents the spatial and temporal pooled connection operation. The details of each attention module are described below.

Spatial attention block
Spatial attention pays more attention to the target objects and the interaction between them in the video. It tries to locate them over time and generate the spatial attention feature map by using the spatial relationship between features. To calculate spatial attention, we first apply the average pooling and maximum pooling operations on the channel axis, denoted as 1 s_avg ,which can not obtain all the information on the feature map. Inspired by the residual block, we introduce the information of the original feature map and connect them to generate an effective feature descriptor. In this way, the information area can be highlighted effectively, but the features ignored by attention will not be lost. Then concatenate the three maps, deploy a spatial attention network where σ denotes the sigmoid function and the kernel size can be set to 3×3×3.

Temporal attention block
Temporal attention focuses on the "when" aspect of video attention. Specifically, it measures the motion pattern of the target object in a particular way over time in a video sequence. Each channel of the feature map is an important feature detector [33] and contains the meaning of a given image. In the entire convolution process, the number of channels of the intermediate feature map is always increased to obtain more abundant semantics. Although increasing the number of channels can extract image information more finely, the model often produces an overfitting phenomenon. A high number of channels may lead to a large amount of redundant information, which affects the extraction of favorable information from the model.
To facilitate computation, we compress the spatial dimension W H × to design a compact temproal . Although the average pooling has a strong ability to aggregate the global space, we still add the maximum pooling to prevent missing smaller or occluded targets. Max pool pays more attention to important local features, while average pooling pays more attention to global features. We where m, n are parameter factors. Generally, we will take m=n.This way can enrich the feature layer and is a pooling strategy extended for the advantages of the two pooling methods. Besides, we also introduce overlapping pooling to pool the information of adjacent pixels in the process of pooling output features, to obtain the key feature information lost by non-overlapping again. Firstly, we aggregate the spatial information of feature map by using average pooling,max-pooling operations to generate two different spatial context descriptors:

Attention module settings
For a specific input image, spatial and temporal attention from two perspectives complement each other to focus on "what" and "when". Because it can replace some units in the network, the two modules can be used as the minimum unit or embedded in the residual block. We find that the algorithm is embedded in the residual block and the result is better when the channel changes in the convolution process. We will discuss the details in the next section.

Experiments
We utilized two popular fine-grained action recognition benchmarks: HMDB51 and UCF101. HMDB51 contains 6766 videos with 51 categories. Each action contains at least 51 videos with a resolution of 320 * 240. It mainly includes general facial movements, body movements, and human-object interaction actions. UCF101 contains 101 types of actions, each of which is performed by 25 people. There are 13320 videos with a resolution of 320 * 240. It mainly includes five kinds of actions: human-object interaction, body movement, human to human interaction, playing music equipment, and various sports. A common practice we follow is as follows [23]. We divided the data into a training set, validation set, and test set by 7:1:2 or use cross-validation. Each clip center in the video is cropped to a size of 112x112. To be fair, all models use the same settings. We take Top1 / 5, time consumption, and model size as performance evaluation. The purpose of our experiment is to verify whether STAT can improve the fitting ability and generalization ability of the model itself. The experiment was carried out in a three-dimensional network. We will discuss STAT as the smallest unit, insert remaining modules and compare with baseline, and discuss the role of adding spatial and temporal attention separately. We choose some pure 3D networks [7][23] [2] [4] [34], which are very popular in recent years to verify.
For [7] and [23], we do not use the pre-training model, and divide the data into a training set, verification set, and test set, and replace the 3D convolution module with STAT in the residual block. Compared with (2 + 1)d replacing STAT(unit) and R2plus1D(base). Of course, we also set up two sets of experiments: spatial attention and temporal attention. The results are shown in Table 1.    Table 1 and figure 2, we can see that the introduction of temporal attention (2+1) D_Tempo and spatial attention (2+1) D_Spatial can promote the results, but they have different performance on two different data sets. They perform worse than introducing them together, which may be that replacing them in the form of (2+1)D performs better than in the residual block. Under the same conditions, replacing the convolutional block in the residual attention module with STAT has better performance on top1 and top5, and there is only a slight overhead in model size and time.
For [2], [4], [34], we remove the last convolution block of each model and replace it with stat. at the same time, we cut the center of each frame in the video by 224x224, and use triple cross-validation to train. The results are shown in Table 2. Table 2 shows us the effectiveness of the introduction of STAT. These complex 3D networks have strong information extraction and characterization capabilities. However, the design of the network itself is very complicated and prone to overfitting. STAT has introduced different pooling operations many times, which reduces the complexity of the model to a certain extent and also ensures the relative integrity of the information, which proves that STAT is effective to a certain extent.