{'fraction_non_alphanumeric': 0.043845609855957125, 'fraction_numerical': 0.022443890274314215, 'mean_word_length': 4.715379706445437, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
End-to-end Video-level Representation Learning for Action Recognition

From the frame/clip-level feature learning to the video-level representation building, deep learning methods in action recognition have developed rapidly in recent years. However, current methods suffer from the confusion caused by partial observation training, or without end-to-end learning, or restricted to single temporal scale modeling and so on. In this paper, we build upon two-stream ConvNets and propose Deep networks with Temporal Pyramid Pooling (DTPP), an end-to-end video-level representation learning approach, to address these problems. Specifically, at first, RGB images and optical flow stacks are sparsely sampled across the whole video. Then a temporal pyramid pooling layer is used to aggregate the frame-level features which consist of spatial and temporal cues. Lastly, the trained model has compact video-level representation with multiple temporal scales, which is both global and sequence-aware. Experimental results show that DTPP achieves the state-of-the-art performance on two challenging video action datasets: UCF101 and HMDB51, either by ImageNet pre-training or Kinetics pre-training.


Introduction
In recent years, human action recognition has received increasing attention due to potential applications in humanrobot interaction, behaviour analysis and surveillance. From traditional hand-crafted features based methods [1,2,3,15] to deep learning methods [4,5,6,19,20,24,25], from small scale video datasets [11,12] to large scale video datasets [28,32], considerable progresses have been made in the community of human action recognition.
However, for the video representation building based on deep learning, there still remains some challenges to be solved and one of them is the long-range temporal structure modeling [6,19,20,24]. As an image doesn't evolve over time like a video, it could be safely represented by its crop with the image-level label as the supervision [14]. Given a video, early approaches tried to learn the frame-  level feature [4] or clip-level feature [17]. At test time the frame-level predictions are averaged to get the videolevel prediction [4]. While the clip-level features are aggregated into a video representation for SVM classification [17]. Recently, it is found that the classifier trained by single frame or short clip of a video 1 would confuse itself when it meets the look-alike frames/clips in the videos of different classes [13,20], as shown in Figure 1. To overcome the confusion caused by partial observation training, several works have been trying to learn the video-level representation [13,10,16,19,20,24,50]. They primarily follow two different pipelines: (1) aggregating the framelevel features of two-stream architectures into a fixed-size video-level representation [13,19,20,24,50], which employs complex encoding methods [22,23,52,53]; (2) applying 3D convolutions on long-term input frames to learn a spatio-temporal representation [10,16]. However, their representations are either built with insufficient (e.g. 3) frames/stacks sampled per video [19], or without end-toend training [20,24], or restricted to single temporal scale modeling [20,50], thus may not be discriminative enough and be sub-optimal. The representations of some methods are very high dimensional [13,24], which make them hard to train and deploy. Moreover, compared to 2D convolutions, long-term 3D convolutions [16] have to reduce the spatial resolution of input frames to relieve memory consumption, from which finer cues are easily lost.
To address these problems, a new video-level representation learning method is proposed in this paper, called Deep networks with Temporal Pyramid Pooling (DTPP) as illustrated in Figure 2. Our method aims to build the video-level representation in an end-to-end manner using enough frames sparsely sampled across a video. The trained model has the video-level representation and could make the video-level prediction both in training and testing instead of aggregating the frame-level predictions like some former two-stream style methods [4,5,6,25]. A temporal pyramid pooling layer is used to encode the frame-level features into the fixed-size video-level representation with multiple temporal scales, where a coarse-to-fine architecture is employed to capture the temporal structure of the human actions in videos.
Our method is built upon the two-stream style methods [4,5,6,19,25], where the spatial and temporal streams take as input RGB images and optical flow stacks respectively. The major difference is that our method aims to learn the video-level representation rather than the framelevel feature. By mapping between the video-level representation and prediction, it is expected to avoid the mistakes that the two-stream style models easily make due to partial observation training. Our method is also similar to some local encoding methods in action recognition [20,26,27]. The difference is that their methods are not trained end-toend [20,26,27], or restricted to single temporal scale learning [20].
In summary, our contributions include: (a) We propose an end-to-end video-level representation learning method for action recognition, dubbed Deep networks with Temporal Pyramid Pooling (DTPP); (b) A temporal pyramid pooling layer is used to aggregate the frame-level features into the fixed-size video-level representation, which captures the temporal structure of videos in a multi-scale fashion; (c) DTPP achieves the state-of-the-art performance both on UCF101 [12] and HMDB51 [11], either by ImageNet [48] pre-training or Kinetics [28] pre-training.
In the remainder of this paper, related works are discussed in Section 2. Section 3 describes our proposed approach. In Section 4, our method is evaluated on the datasets. Finally, conclusions are given in Section 5.

Related works
Hand-crafted video representation. Before the surge of deep learning, the hand-crafted features are dominant in action recognition, such as Space Time Interest Points [2], Cuboids [29], Trajectories [3,30]. Among them, improved Dense Trajectory (iDT) [3] is currently the basis of state-of-the-art handcrafted methods, which explicitly considers the motion features by pooling rich descriptors along dense trajectories and compensates camera motions. Then by encoding methods like BoW, Fisher vector, the descriptors are aggregated into the video-level representation. Some classification pipelines based on CNN are still built upon iDT, e.g. trajectory-pooled deep-convolutional Descriptor (TDD) [31].
Frame-level feature by deep learning. Simonyan et al. [4] designed two-stream ConvNets, which applies Im-ageNet pre-training for the spatial stream and optical flow for the input of the temporal stream. During training, it samples one RGB image or optical flow stack from each video. Temporal Segment Networks (TSN) [5,6] is an extension of the two-stream ConvNets by using deeper CNN and cross-modality pre-training. It aggregates several (3 or 7) frame-level predictions into a global video prediction during training and finally outperforms iDT [3] by a large margin. C3D [17] applies 3D convolutions on a short video clip with consecutive frames (typically 16). However, all these approaches build the frame/clip-level features instead of the video-level representations, thus suffer from the confusion caused by partial observation training. Motivated by the recent success of LSTM in sequence modeling [33,34,35,36,37], there are also attempts that employ LSTM in video action recognition [38,39,40,41]. Ng et al. [39] trained LSTM with the CNN feature as input and aggregated predictions of all the time steps by linear weighting into the video-level prediction. LSTM has also been used to learn the sequence features of videos in unsupervised settings [38] and to learn the attention model [40]. However, due to its additional parameters as well as the differences between video frame and speech, text [41], LSTM has not shown its capacity in action recognition yet.
Video-level representation by deep learning. Researchers have found the false label assignment problem [5,6,19,20] caused by training from single frame or short clips of a video. Based on deep learning, several video-level representation learning methods have emerged recently. Varol et al. [16] explored the long-term temporal convolution to learn the video-level representation. However, because of the large amount of parameters of 3D convolution [16,17], they had to downsample video frames to reduce memory consumption, thus be sub-optimal. Diba et al. [19] proposed Deep Temporal Linear Encoding (TLE), which achieves high performance by bilinear encoding [23,22]. However, the video-level representation of TLE contains insufficient information of a given video because it only samples 3 frames or flow stacks during training. Our method samples more (25) frames/stacks during training. There are also extensions [13,24] that insert traditional encoding methods, like Fisher vector [52] and VLAD [53] into deep CNN. They are designed to aggregate the local spatio-temporal features of CNN across the whole video into several clusters. The representations of these methods are either very high dimensional, e.g. 32,768-D for Ac-tionVLAD [13], 131,072-D for Deep Quantization [24], or without end-to-end training, e.g. Deep Quantization [24]. While the representation dimension of our method is lower than their methods, i.e. 7168-D and it is also trained endto-end. Moreover, our method explicitly considers multiple temporal scale video representation building, which is lacked in the single temporal scale methods, e.g. Deep lOcal Video Features (DOVF) [20] and Dynamic Image Networks [50].
3. End-to-end video-level representation learning 3.1. Overall architecture Figure 2 shows the architecture of Deep networks with Temporal Pyramid Pooling (DTPP). Given a video X consisting of t frames, each frame is denoted as x i and the whole video as X = {x 1 , x 2 , . . . , x t }. Considering the redundancy of video frames and memory limitation, not all video frames are used to represent a video. Instead, each video is firstly divided into T segments of equal durations and then a frame is sampled from each segment. These frames are feeded to a ConvNet and the ConvNet computes the features for each frame independently. The obtained features are vectors {S 1 , S 2 , . . . , S T } where S i ∈ R d , i = 1, 2, . . . , T , d is the frame feature dimension. To aggregate several frame-level features into a video-level representation, a temporal pyramid pooling (TPP) layer is placed on top of them. By end-to-end training, a fixed-size representation with multiple temporal scales is obtained.
Network architecture. The BN-Inception network [8] is applied as the backbone of DTPP. The BN-Inception has a depth of 33 layer, including 1 fully connected layer. As mentioned above, forward propagation is performed for each video frame independently. We extract the feature of the last convolutional layer after a global average pooling operation for each frame, which is a 1024-D vector. A temporal pyramid pooling layer is used to aggregate these frame-level features into a video-level representation, as shown in Figure 3. The number of pyramid levels of the temporal pooling layer is K and the total number of bins is M = K i=1 2 i−1 . In each temporal bin, the responses of each filter is pooled by a function G: {S i , S i+1 , . . . , S j }→Q i→j . Different functions G are investigated for the temporal pooling layer • Average pooling: • Max pooling: Formally, the output of the temporal pyramid pooling layer P is M ×d dimensional, which is a video-level representation with K pyramid levels where P ∈R M ×d , P i is the pyramid level i video representation with dimension 2 i−1 ×d. Note that the coarsest pyramid level P 1 covers the entire video. Each of the higher pyramid levels P i , i = 2, . . . , K consists of the pooled bins aligned by the temporal order. Thus, the obtained representation is globally sequence-aware. In this way, DTPP is equipped with a degree of sequence regularization, which is lacked in some two-stream style methods, e.g. TLE [19], TSN [5,6], Deep Quantization [24]. The video-level representation P followed by a dropout layer is the input to the fully-connected layer, which outputs a video-level prediction given video X.
Two-stream ConvNets. Each input frame is rescaled by keeping the aspect ratio and its smaller side is resized into 256 pixels. RGB images and optical flow stacks are used as the inputs of two separate models respectively, i.e. spatial stream and temporal stream. For spatial stream, each input frame size is 224×224×3. For temporal stream, each input frame size is 224×224×10, which is the stack of 5 consecutive optical flow fields [5].

Training
Consider a dataset of N videos with n categories . . , n} is the label. Our goal is to optimize the parameters of ConvNets to build the video-level representation P by end-to-end training. Formally, the video-level prediction could be obtained directly where ϕ is a softmax operation, Y ∈ R n . W c and b c are the parameters of the fully connected layer. In the training stage, combining with cross-entropy loss, the final loss function is  where Y (y i ) is the value of y i th dimension of Y . The backpropagation for the joint optimization of the T segments can be derived as: Model pre-training. Two kinds of pre-training are used: ImageNet [48] pre-training and Kinetics [28] pre-training. For the first one, the spatial stream is initialized by the model pre-trained on ImageNet and adapted by fine-tuning. To speed up convergence, we borrow the temporal stream of TSN [5] trained on video action datasets to initialize our temporal model. For the second one, a large scale video  (5) and (6) 4. i = i + 1 until i = Iter dataset, Kinetics dataset [28] is used to pretrain models. Kinetics dataset [28] is built to explore the effect of transfer learning from large scale video dataset to small scale video dataset. The pre-trained TSN [5,6] models provided by Xiong [44] are used, which are firstly adapted to our DTPP framework and further fine-tuned on small video action datasets [11,12]. The convolutional parameters of two streams are transferred from the pre-trained models and the fully connected layers are randomly initialized. The learning procedure is summarized in Algorithm 1.
Training detail discussions. T = 25 RGB images and optical flow stacks are sampled from each video by default. In later ablation studies, different number of frames/stacks are also sampled from each video to explore the effect of number of training frames.

Inference
For each video during testing, 25 RGB images and optical flow stacks are sampled. Meanwhile, the crops of 4 corner and 1 center, and their horizontal flippings are ob-tained from the sampled frames. From one of these 10 crops, 1 video-level prediction is obtained. Corresponding 10 video-level predictions are averaged to get the final prediction. All predictions are obtained without the Softmax normalization.
Model fusion For the fusion of the predictions from two streams, weighted averaging scheme is used, where the fusion weight for each stream is 0.5 by default. The scores are fused without the Softmax normalization.

Experiments
In this section, the evaluation datasets and the implementation details of our approach are firstly introduced. Then, quantitative and qualitative results on these datasets are reported.

Datasets and implementation details
Datasets. Experiments are conducted on two challenging video action datasets: UCF101 [12] and HMDB51 [11]. The UCF101 dataset contains 13,320 videos with 101 action classes. The HMDB51 dataset contains 6,766 videos with 51 action classes. For both datasets, standard 3 training/testing splits are used for evaluation. The mean average accuracy over three testing splits is finally reported for the comparison with other approaches.
Implementation details. The mini-batch stochastic gradient descent algorithm is used to learn the model parameters, where the batch size is set to 128 2 videos. The L2 norm of gradients is clipped at 40 and momentum term is set to 0.9. Instead of decreasing the learning rate according to a fixed schedule, the learning rate is lowered by a factor of 10 after validation error saturates. Specifically, the learning rate for spatial stream is initialized as 0.01, ended at 0.00001, while the learning rate for temporal stream is initialized as 0.001, ended at 0.00001. The dropout ratio of the dropout layer for the video-level representation of both streams is set to 0.8. The same data augmentation techniques are used for all frames/stacks sampled from one video, including location jittering, horizontal flipping, corner cropping and scale jittering [5]. The optical flows are computed using the method of [18]. All experiments are implemented with Caffe [7]. One NVIDIA TITAN X GPU is used for training the spatial stream and two for the temporal stream. OpenMPI 3 is used for the data parallel of the 2 Note that the batch size for updating gradients is different from the batch size for each forward and backward pass in train.prototxt. The batch size in train.prototxt for both streams is 4. For the spatial stream, iter size in solver.prototxt is 32, 1 GPU is used, so the batch size (for updating gradients) = batch size × iter size × n gpu = 128. For the temporal stream, iter size in solver.prototxt is 16, 2 GPUs are used, so the batch size (for updating gradients) = batch size × iter size × n gpu = 128. 3  temporal stream. Codes will be available online.

Ablation studies
In this section, different aspects of learning the videolevel representation with our approach are investigated by experiments. These experiments are all conducted on the split 1 of the UCF101 dataset.
Temporal pyramid pooling. The number of pyramid levels and kernel of temporal pooling are selected according to the performance of the network. Max pooling is used as the temporal pooling kernel by default. As shown in Table 1, with 1 pyramid level, i.e. global max pooling over 25 frame-level features, the two-stream accuracy of DTPP has achieved 94.3%. With further finer scale pooling, the performance is enhanced and it finally plateaus at 4 pyramid levels. Note that the representation of 4 pyramid levels has nearly twice the dimension of that of 3 pyramid levels. A single pyramid level with 3 segments is also tested, whose accuracy is 0.3% lower than that of 2 pyramid levels with the same dimension, which demonstrates the merits of multi-scale temporal pooling. The temporal pyramid pooling layer with average pooling kernel and 3 pyramid levels is also verified, which gains lower accuracy than that with max pooling kernel. Thus, the temporal pooling layer with 3 pyramid levels and max pooling kernel is selected both for the spatial and temporal streams by default.
Effect of end-to-end training. The effect of end-to-end training on the video-level representation is explored. For direct comparison, TSN [5] with BN-Inception, a framelevel feature learning method is chosen to do experiments. We also design a variant of TSN, "TSN+TPP", which is adapted from TSN by adding a TPP layer with 3 pyramid levels and max pooling kernel. This variant is only fine-tuned in the final fully connected layer, while the parameters of the convolutional layers are initialized by the trained reference models on UCF101 (split 1) and kept fixed. As shown in Table 2, though without end-to-end training, TSN+TPP improves the accuracy of TSN by 0.9%, showing the importance of building the video-level representation. Our method not only exceeds TSN, but also outperforms TSN+TPP by 0.5%, demonstrating the merits of end-to-end training in building the video-level representa-Spatial Temporal Two-stream TSN [5] 85

tion.
With different number of frames for training and testing. Due to the nature of temporal pyramid pooling, when the temporal dimension of input video frames varies, the model could still generate a fixed-size representation. Hence in testing, it could generalize to frame number which is different from training frame number per video. This is different from some video-level representation learning methods, e.g. TLE [19] and DOVF [20], which need fixed-size input in testing. While the frame-level feature learning approaches, such as two-stream ConvNets [4] and TSN [5,6], suffer from the sub-optimal mapping caused by the late fusion of the prediction scores. Our method based on temporal pyramid pooling could generalize to less or more frames during testing in representation level, instead of prediction level. In Figure 4, we show the performance of DTPP under different number of training frames per video, e.g. "DTPPseg4" is trained by sampling 4 frames from each video. "TSNseg3" is the TSN [5] trained by sampling 3 frames from each video. Under different number of testing frames, the accuracy of DTPP is generally stable and consistently higher than that of TSN. Specifically, "DTPPseg4" outperforms "TSNseg3" under all the testing frames cases. It clearly suggests that building the video-level representation could avoid the mistakes that the frame-level methods easily make, even using less frames for training. It could also be observed that with more frames each video for training, the performance of DTPP could be improved, showing the significance of using enough frames to build the videolevel representation.

Comparison with the state-of-the-art
After the above analysis of DTPP, final experiments on all the three testing splits of UCF101 and HMDB51 are implemented with our proposed methods. Specifically, during training, 25 frames/short clips are sampled from each video and a temporal pooling layer of 3 levels and max pooling kernel is employed. During testing, 25 frames are selected from each video. For the late fusion of stream scores, the weights for each stream are equal, unless otherwise specified.
The performances on UCF101 and HMDB51 are reported respectively in Table 3 and 4. As shown in Table 3, the two-stream accuracy of DTPP increases from the split 1 to split 3 of UCF101, largely because of the increasing  Figure 4. Exploration of different number of testing frames for TSN [5], and different number of training and testing frames for DTPP on UCF101 (split 1), accuracy(%). amounts of training videos, i.e. 9,537, 9,586 and 9,624 respectively. The two-stream late fusion could greatly improve the accuracy over single stream. We also empirically find that with our video-level representation learning method, a equal weight (with weight 0.5) for the two-stream late fusion ensures the highest accuracy, instead of a higher weight for the temporal stream in former work, e.g. 0.6 in TSN [5] and 2/3 in two-stream ConvNets [4]. By using more RGB images, our video-level representation learning could narrow the performance gap between the spatial and temporal stream. Further late fusion (with weight 0.5) with the traditional methods, MIFS [1] and iDT [3] is helpful. Partly because of lacking training videos, the performance of DTPP on HMDB51 shown in Table 4 is less satisfied than that on UCF101. One more reason is that the videos of HMDB51 are more difficult than that of UCF101. In late fusion, a more bigger weight (0.6) for the temporal stream than the spatial one gains higher accuracy than a equal weight, showing the fact that the motion information of HMDB51 is more important than that of UCF101.
frame-level feature learning method TSN [5,6], the videolevel representation learning method TLE [19] with only 3 frames, the video feature learning method without endto-end training and without multi-scale temporal modeling, DOVF [20]. More surprisingly, DTPP even exceeds Dynamic Image Networks [50] either by single model or fusion with iDT [3], which employs the ensemble of four models, all the frames of testing videos and a more advanced network architecture, i.e. ResNeXt [51]. Dynamic Image Networks [50] is also a video-level representation learning method. However, it is limited in single temporal scale modeling and its encoded dynamic inputs may lose the cues of original frames [50]. DTPP is also compared with other methods pre-trained on Kinetics dataset [28]. As shown in Table 5, on the UCF101 dataset, DTPP pre-trained on Kinetics exceeds TSN with BN-Inception as backbone by 1.0% and TSN with Inception v3 [49] by 0.7%. DTPP is on par with Twostream I3D [28] on UCF101, but exceeds it on HMDB51 by 1.4%, thereby achieving the state-of-the-art performance when pre-trained on Kinetics.

Visualization
Because of the large improvements of DTPP over previous methods on HMDB51, visualizations are done on the testing set (split 1) of this dataset. Corresponding results of TSN [5] are also displayed for clear illustration. Figure 5 shows the accuracy of each action. For most actions, the accuracy of our DTPP is higher than that of the TSN. For example, for "sit", "cartwheel", and "swing baseball", 'draw sword", the accuracy of DTPP is more than 15% higher than that of TSN. Moreover, these classification results are shown in Figure 6. It is noticed that DTPP performs better than TSN in the videos with sequence pattern, e.g. "sit" and "stand", "cartwheel" and "handstand". One possible reason is that the video-level representation of DTPP is sequence-aware, which is lacked in TSN. DTPP also behaves well when the human object interaction pat- tern exists in videos, e.g. "swing baseball", "draw sword". Though we don't have specific designs on detecting detailed objects, DTPP seems to capture the fine-grained objects by using contextual information during building the videolevel representation.

Conclusion
In this paper, we propose Deep networks with Temporal Pyramid Pooling (DTPP), an end-to-end video-level representation learning approach. DTPP uses a temporal pyramid pooling layer to aggregate the frame-level features of videos into a multi-scale video-level representation, being both global and sequence-aware. A set of problems are studied, including the number of pyramid levels and pooling kernel, the effect of end-to-end training as well as the effect of training frame number and testing frame number per video. Finally, DTPP achieves the state-of-the-art performance on UCF101 and HMDB51, either by ImageNet pre-training or Kinetics pre-training. Though the videolevel representation is obtained by end-to-end training in this work, a more through extension is to calculate the optical flow in a network fashion [45,46], where the flownet could also be fine-tuned. A unified spatial-temporal videolevel representation could also be obtained by the fusion [10,43] of the feature maps from both streams.