{'fraction_non_alphanumeric': 0.0362955566650025, 'fraction_numerical': 0.012606090863704443, 'mean_word_length': 4.863729508196721, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Action Recognition with Deep Multiple Aggregation Networks

Most of the current action recognition algorithms are based on deep networks which stack multiple convolutional, pooling and fully connected layers. While convolutional and fully connected operations have been widely studied in the literature, the design of pooling operations that handle action recognition, with different sources of temporal granularity in action categories, has comparatively received less attention, and existing solutions rely mainly on max or averaging operations. The latter are clearly powerless to fully exhibit the actual temporal granularity of action categories and thereby constitute a bottleneck in classification performances. In this paper, we introduce a novel hierarchical pooling design that captures different levels of temporal granularity in action recognition. Our design principle is coarse-to-fine and achieved using a tree-structured network; as we traverse this network top-down, pooling operations are getting less invariant but timely more resolute and well localized. Learning the combination of operations in this network -- which best fits a given ground-truth -- is obtained by solving a constrained minimization problem whose solution corresponds to the distribution of weights that capture the contribution of each level (and thereby temporal granularity) in the global hierarchical pooling process. Besides being principled and well grounded, the proposed hierarchical pooling is also video-length and resolution agnostic. Extensive experiments conducted on the challenging UCF-101, HMDB-51 and JHMDB-21 databases corroborate all these statements.

Among the ML solutions -for action recognition -those based on deep networks are currently witnessing a major interest [4], [33], [34], [64], [65], [75], [79] but their success is tributary to the availability of large amount of labeled training data and also the appropriate choice of their architectures including convolutional and recurrent ones [1], [12]- [14], [33]. In particular, convolutional networks are designed by stacking multiple convolutional, pooling and fully connected layers; successful architectures for action recognition include two-stream 2D/3D convolutional neural networks (CNNs) operating on appearance and motion flows, and CNNs combined with Long Short-Term Memory (LSTM) networks [102] that capture coarse temporal structure of actions as well as 3D CNNs [1] which capture fine (local) temporal structures. However, and beside issues related to scarcity of labeled data and the large number of training parameters (especially in 3D CNN models), the effort in the design of deep networks, that capture the relevant motion information in videos, has been focused essentially on optimizing their convolutional and fully connected layers 1 while comparatively the design of optimized pooling layers received less attention especially on non-vectorial data including video sequences. The difficulty in designing architectures with suitable pooling (a.k.a aggregation) operators, particularly on video sequences stems from the eclectic properties of videos (namely their duration, temporal resolution and velocity of moving objects as well as the granularity of their action categories) and this makes pooling design very challenging. This challenge is further exacerbated by the lack of labeled video data (covering all the variability) compared to other neighboring problems such as image classification that benefit from labeled sets which are at least an order of magnitude larger compared to the current action recognition datasets while the task is intrinsically far more challenging; as a result, these action recognition models are more subject to overfitting.
In order to attenuate such effect, pooling methods [89]- [91] have been designed, and most of them are based on global measures including max and averaging operators. Pooling plays a key role in reducing the dimensionality of convolutional feature maps and thereby the number of training parameters and enhances the resilience, of the learned CNN representations, to the lack of training data and to the acquisition conditions. However, it comes at the detriment of some relative loss in the discrimination power especially when video data belong to fine-grained action categories. Indeed, pooling contributes in diluting (averaging) convolutional features which are highly important in discriminating fine-grained categories and these averaging operators are rather more appropriate for coarse-grained actions (see Fig. 1). Alternative and more recent solutions [1], [12], [13] rely on sampling and stacking CNN features in order to build spectrogramlike fixed length representations that also preserve the granularity of video actions. Nonetheless, both methods suffer from several drawbacks; on the one hand, pooling methods based on global statistical measures are time/duration agnostic (and hence invariant) but less discriminating while spectrogram-like methods are discriminating but time/duration aware (less invariant) and highly sensitive to the aforementioned video acquisition conditions and may result into a loss of informations, especially when videos are not well resolute.
A more suitable pooling should gather the advantages of these two families of methods while discarding their inconvenients. Following this goal, we consider in our work a hierarchical aggregation scheme that describes moving scenes at multiple temporal granularities while also being resilient to their highly variable acquisition conditions. Top levels in this hierarchical aggregation provide orderless (invariant) but less discriminating motion and appearance representations which capture coarse-grained action categories (as global averaging techniques [33]) while bottom levels correspond to fine-grained, timely resolute and order-sensitive video representations (as spectrogram based techniques [33]). The design principle of our proposed solution is coarse-tofine and allows us to capture a gradual change of invariance and granularity; as we traverse the hierarchy top-down, our video representations are getting less invariant but timely more Examples of fine and coarse-grained actions. The first row shows three action categories from the MLB-YouTube dataset [98]: "No swing", "Swing" and "Bunting" which are difficult to distinguish as they have very small differences. The second row shows two instrument playing actions from the UCF-101 dataset [19]: "cello" and "violin" which are also difficult to distinguish as their arm/hand locations and directions are similar. In contrast, the third row shows "Pat on back", "Butt kick" and "Shaking hand" actions (taken from NTU RGB+D dataset [101]) which are relatively easier to distinguish. resolute and fine-grained. However, knowing a priori which levels in this hierarchy are the most appropriate in order to capture the actual granularity of our video data is challenging and also combinatorial; hence, learning this combination "end-to-end" and in a differentiable manner is rather more appropriate.
Considering this line of research, other related works [24], [26], [27], [29], [31] try to model granularity of actions in videos by incorporating specific modules into CNNs. The method in [24] samples, from each video, frames as well as their associated optical flow components and adds a spatio-temporal module to CNN in order to capture hierarchical relationships between appearance and motion features. The method in [26] stacks a hierarchical temporal pooling layer on the top of motion and appearance CNN streams in order to build fixed-length video representations. In [27], authors sample a set of frames by first splitting videos into segments and taking frames from each segment, and build a spatial pyramid to extract multi-scale appearance features from different convolutional layers. These features are then concatenated and fed to a three level temporal hierarchy. The work in [29] samples video frames at different temporal resolutions, and feeds them to a 3D CNN in order to extract their respective features followed by a temporal hierarchy which down-samples and concatenates the resulting features. Finally, the method in [31] achieves frame sampling followed by a temporal pooling in order to build features at different pyramidal levels; the resulting features are afterwards fed to a temporal relational layer that groups these features at different scales. While all these methods rely on a hierarchical temporal aggregation scheme, none of them considers the issue of learning the best combination of levels in these temporal aggregation hierarchies, and this turns out to be highly effective as shown in the following sections.
In this paper, we introduce a novel scheme for action recognition based on Deep Multiple Aggregation Networks. Given a hierarchy of aggregation operations, the goal is to learn a combination of these operations that best fits a given action recognition ground-truth. We solve this problem by minimizing a constrained objective function whose parameters correspond to the distribution of weights through multiple aggregation levels; each weight captures the granularity of its level and its contribution in the global learned video representation. Besides handling aggregation at different levels, the particularity of our solution resides in its ability to handle variable length videos (without any up or down-sampling) and thereby makes it possible to fully benefit from the whole frames in videos.
The rest of this paper is organized as follows. First, we describe in Section 2 our motion and appearance streams used to build frame-wise representations. Then, we introduce in Section 3 our main contributions; two methods -based on linear/nonlinear kernel combination and "end-to-end" two stream CNN training -that aggregate and combine the obtained frame-level representations into temporal pyramids in order to achieve action recognition. Finally, we show the validity of these contributions through extensive experiments using standard and challenging video datasets including UCF-101, HMDB-51 and JHMDB-21.

FRAME-WISE TWO-STREAM VIDEO DESCRIPTION AT A GLANCE
We consider a collection of videos S = {V i } n i=1 with each one being a sequence of frames t=1 and a set of action categories (a.k.a classes or categories) denoted as C = {1, . . . , C}. In order to describe the visual content of a given video V i , we rely on a two-stream process (see Fig. 2); the latter provides a complete description of appearance and motion that characterizes the spatio-temporal aspects of moving objects and their interactions. The output of the appearance stream (denoted as {φ a (f i,t )} T i t=1 ⊂ R 2048 ) is based on the deep residual network (ResNet-101) trained on ImageNet [23] and fine-tuned on UCF-101 [19] while the output of the motion stream (denoted as {φ m (f i,t )} T i t=1 ⊂ R 2048 ) is also based on the ResNet 101 network but trained on optical flow image pairs [16], [94]; these pairs correspond to the horizontal and the vertical displacement fields which are linearly transformed in order to make their ranges between 0 and 255.
Following the line in [16] and in order to adapt the pretrained ResNet-101 to optical flow data, we slightly update the input layer of the original ResNet 2 . Indeed, the number of channels is reset to 20 instead of 3 in the original ResNet; the initial weights of these 20 channels are obtained by averaging the 3 original (appearance) channel weights and by replicating their values through the 20 new motion channels. During training, closely related methods (namely [16]) split each video into N continuous segments, and for each segment, a frame f is randomly selected to feed an appearance stream ResNet and a stack of optical flow is also taken (starting from f ) as an input to the motion stream. In the setting of [16], scores obtained from the softmax layers of motion and appearance streams are fused through different frames using a segmental consensus function in order to make class prediction at the video level; in other words, for each test video, 19 frames 3 are uniformly sampled and passed through appearance and motion streams and their scores are combined as votes for all the action categories. As shown subsequently, and in contrast to [16], our proposed method relies on a different aggregation scheme that models coarse as well as fine grained action categories; besides, our method does not require any frame (re)sampling -which 2. Already available/pretrained on ImageNet to capture the appearance. 3. The reason for choosing 19 frames is explained by the fact that the minimum number of video frames in UCF-101 is 28, hence 19 is the maximum number from which a stack of 10 optical flow frames can be taken.  may degrade performances (as also shown later in experiments) -indeed, our method effectively leverages the entire set of video frames.

MULTIPLE AGGREGATION LEARNING
Given a video V i with T i frames, we define N as a tree-structured network with depth up to D levels and width up to 2 D−1 . Let N = ∪ k,l N k,l with N k,l being the k th node of the l th level of N ; all nodes belonging to the l th level of N define a partition of the temporal domain [0, T i ] into 2 l−1 equally-sized subdomains. A given node N k,l in this hierarchy aggregates the frames that belong to its underlying temporal interval. Each node N k,l also defines an appearance and a motion representation respectively denoted as ψ a k, Fig. 4. Depending on the level in N , each representation captures a particular temporal granularity of motion and appearance into a given scene; it is clear that top-level representations capture coarse visual characteristics of actions while bottomlevels (including leaves) are dedicated to fine-grained and timely-resolute sub-actions. Knowing a priori which levels (and nodes in these levels) capture the best -a given action category -is not trivial. In the remainder of this section, we introduce a novel learning framework which achieves multiple aggregation design and finds the best combination of levels and nodes in these levels that fits different temporal granularities of action categories.
Considering the motion stream, we define -for each node N k,l -a set of variables β m = {β m k,l } k,l (with β m k,l ∈ [0, 1] and k,l β m k,l = 1) which measure the importance (and hence the contribution) of As shown above, the variant in (*) corresponds to a concatenation scheme while (**) corresponds to averaging. Similarly to motion, we define the aggregations and the set of variables β a = {β a k,l } k,l associated to appearance stream. In the remainder of this section, and unless explicitly mentioned, the symbols m, a are omitted in the notation and all the subsequent formulation is applicable to motion as well as appearance streams.

Shallow multiple aggregation learning
In this section, we consider all the representations {ψ k,l (.)} k,l fixed on all the video frames, and only the mixing parameters in β are allowed to vary. Given the set of action categories C = {1, . . . , C}; we train multiple classifiers (denoted {g c } c∈C ) on top of these level-wise representations. In practice, we use maximum margin classifiers whose kernels correspond to combinations of elementary kernels dedicated to {N k,l } k,l . These classifiers are suitable choices as they allow us to weight the impact of nodes in the hierarchy N and put more emphasis on the most relevant granularity of the learned representations. Hence, depending on the granularity of action categories, these classifiers will prefer top or bottom levels of N .
Considering a training set of videos {(V i , y ic )} i associated to an action category c, with y ic = +1 if V i belongs to the category c and y ic = −1 otherwise, the max margin classifier associated to this action category c is given by is a set of positive parameters and K is a positive semi-definite (p.s.d) kernel [104]. In order to combine different nodes in the hierarchy N and hence design appropriate aggregation, we consider multiple representation learning that generalizes [17] both to linear and nonlinear combinations (see also [6], [7], [85]). Its main idea consists in finding a kernel K as a combination of p.s.d elementary kernels {κ(., .)} associated to {N k,l } k,l . Considering the two map variants in Eq. (1), we define these kernels as As β k,l ∈ [0, 1], the kernel K is p.s.d resulting from the closure of the p.s.d of κ w.r.t the sum and the product. Using K, we train the max margin classifiers {g c } c∈C whose kernels (in Eq. 2) correspond to level-wise linear (resp. cross-wise nonlinear) combinations of elementary kernels dedicated to {N k,l } k,l . Hence, using a maximum margin formulation, we find the parameters β = {β k,l } k,l and As the problem in Eq. 3 is not convex w.r.t β, {α c i } taken jointly and convex when taken separately, an EM-like iterative optimization procedure can be used: first, parameters in β are fixed and the above problem is solved w.r.t {α c i } using quadratic programming (QP), then {α c i } are fixed and the resulting problem is solved w.r.t β using either linear programming for (*) and QP for (**). This iterative process stops when the values of all these parameters remain unchanged or when it reaches a maximum number of iterations.

Deep multiple aggregation learning
In this section, we consider an end-to-end framework that learns the parameters β m and β a together with (i) the ResNet parameters (denoted as α a , α m ) 4 , (ii) the MLP+softmax parameters (denoted as γ a , γ m ) as well as (iii) the mixing parameters (referred to as w a and w m ) which respectively capture the importance of appearance and motion streams in action recognition. Considering E as the regularized cross-entropy loss 5 associated to our complete network (in Fig.  2), we find the optimal α = {α m , α a }, β = {β m , β a }, γ = {γ m , γ a } and w = {w m , w a } by solving the following constrained minimization problem In spite of having many differences w.r.t usual losses used in deep learning, this objective function can still be solved using gradient descent and backpropagation. However, many differences exist and should be carefully tackled; indeed, whereas the forward step can be achieved, gradient backpropagation (through our multiple aggregation layer) should be achieved while considering videos with a varying number of frames. Besides, constraints on β s should also be handled. In what follows, we discuss all these updates in the optimization process. 4. In the rest of this paper, the notation α refers to the ResNet parameters and not the max margin classifiers anymore. 5. Regularization is achieved using 2 weight decay.
Optimization. Considering ρ() as the output of the final layer of our deep network and considering ∂E ∂ρ available, the gradients ∂E ∂w , ∂E ∂γ (w.r.t the preceding mixing and MLP layers) could easily be obtained using a straightforward application of the chain rule (as already available in the used PyTorsh tool). However, ∂E ∂β , ∂E ∂α cannot be obtained straightforwardly; on the one hand, any step following the gradient ∂E ∂β should preserve equality and inequality constraints in Eq. (4) while a direct application of the chain rule provides us with a surrogate gradient which ignores these constraints. On the other hand, the variable number of frames for different training videos requires a careful update of ∂E ∂α as shown subsequently.

Constraint implementation.
In order to implement the equality and inequality constraints during the optimization of the objective function (4), we consider a re-parametrization as β m k,l = h(β m k,l )/ k ,l h(β m k ,l ) for some {β m k,l } k,l with h being strictly monotonic real-valued (positive) function and this allows free settings of the parameters {β m k,l } k,l during optimization while guaranteeing β m k,l ∈ [0, 1] and k,l β m k,l = 1. During back-propagation, the gradient of the loss E (now w.r.tβ's) is updated using the chain rule as .(δ p,q,k,l − β m p,q ), which simply reduces to β m k,l (δ p,q,k,l − β m p,q ) p,q,k,l when h(.) = exp(.). Similarly, we implement the constraints associated to the appearance stream.
ResNet update. As discussed earlier, motion and appearance ResNets are recurrently (iteratively) applied frame-wise prior to pool the underlying feature maps using multiple aggregation. It is clear that the number of frames intervening in this aggregation is video-dependent, and thereby the number of terms in these aggregations (and the number of ResNet branches/instances) is also varying. Hence, a straightforward application of the chain rule in the whole architecture -in order to update ∂E ∂α -becomes possible only when this architecture is unfolded, and this requires fixing the maximum number of frames (denoted as T ) and sampling temporally all the videos in order to make T i constant and equal to T . Note that beside requiring all the ResNet instances to share the same parameters (as in Siamese nets), this results into a cumbersome architecture even for reasonable T values. Furthermore, frame sampling requires interpolation techniques which are highly dependent on quality, duration and temporal resolution of videos and this may result into spurious motion/appearance details (especially on short videos; even when timely well resolute) which ultimately leads to a significant drop in action recognition performances.
In order to avoid these drawbacks and to fully benefit from the available number (and also temporal resolution) of frames -without using multiple instances of "Siamese-like" ResNets and without resampling -we consider an alternative gradient estimation. The latter relies on a membership measure µ which assigns each frame f i,t to nodes in the temporal pyramid as µ k,l i,t = 1 {t∈N k,l } . Using this membership measure together with the chain rule, the gradient of the loss E w.r.t the parameters of the ResNet α can be updated as Similarly, we evaluate the gradient for the appearance stream. From the above equation, it is clear that when k = l = 1, all the frames {f i,t } contribute in the estimation of the gradient, while for other nodes, only a subsets of frames (belonging to these nodes) are used. Nonetheless, all the frames contribute evenly through all the nodes and hence in gradient estimate, without any sampling. Note also that this formulation implicitly implements weight sharing as the above gradient can equivalently be written as the sum of gradients, shared through multiple streams of an unfolded architecture, with each stream being dedicated to one frame. However, the advantage of the above formulation resides again in its computational efficiency and also its ability to leverage all (possibly variable numbers of) frames in videos while an unfolded architecture requires sampling a fixed number of frames and handling multiple ResNet branches which may clearly lead to intractable training.

EXPERIMENTS
In this section, we evaluate the impact of our multiple aggregation design on the performance of action recognition and we compare it against other aggregation strategies as well as the related work using three standard datasets: UCF-101, HMDB-51 and JHMDB-21 [19], [93]. UCF-101used to comprehensively study the different settings of our model -is the largest and most challenging; it includes 13,320 video shots belonging to 101 categories with variable duration, poor frame resolution, viewpoint and illumination changes, occlusion, cluttered background and eclectic content ranging from multiple and highly interacting individuals to single and completely passive ones. We also consider HMDB-51 and JHMDB-21 for further comparisons; the latter include 6766 (resp. 928) videos belonging to 51 (resp. 21) action categories. In all these experiments, we process all the videos using ResNet-101 (as a backbone network) in order to extract all the underlying appearance and motion representations framewise. Then, we apply different aggregation schemes prior to assign those videos to classes. We use the same evaluation protocols as the ones suggested in [18,67,69] (i.e., train/test splits) and we report the average accuracy over all the categories of actions.
We train our complete temporal pyramid-based networks (in Fig. 2) for respectively 130, 100 and 65 iterations on UCF-101, HMDB-51 and JHMDB-21 using the PyTorch SGD optimizer. For appearance stream, we set the learning rate to 0.001 and reduce it by a factor of 10 every 25, 20, 10 iterations for resp. UCF-101, HMDB-51 and JHMDB-21. For motion stream, we set the learning rate to 0.005 and we reduce it by the same factor after "80 and 110", "60 and 80", "50 and 60" iterations on the three sets respectively. Experiments on individual streams are run using 4 Titan X Pascal GPUs (with 12 Gb) and last 72h for UCF101, 36h for HMDB-51 and 15h for JHMDB-21 (on the appearance stream) and 96h for UCF101, 48h for HMDB-51 and 24h for JHMDB-21 (on the motion stream) while on the joint stream experiments are run using 4 Tesla P100 GPUs (with 16 Gb) and last 100h, 55h and 30h on the three sets respectively.
data is limited compared to other neighboring tasks (such as image classification) on which these convolutional networks were initially trained. Hence, in order to select the most appropriate convnet among a collection of existing ones (namely [34], [95], [96]), we measure the performance of our temporal pyramid based on the design in [33]. The results in Table 1 show that the deeper the network, the better are the performances. However, in our experiments, we consider ResNet-101, which provides descent action recognition performances while being relatively less memory and time demanding compared to the other networks and particularly ResNet-152 (see again Table  1).

Settings and Performances
Firstly, we show a comparison of action recognition performances using different settings. Extensive experiments, reported in Tables. (2) and (3), show that our hierarchical aggregation design makes it possible to select the best configuration (combination) of level representations in order to improve the performance of classification; indeed, the results show a clear gain as the depth of the hierarchy increases and compared to global average pooling (level 1). This gain results from the match between the temporal granularity of the learned level-wise representations in the hierarchy and the actual granularity of action categories. Note that in all these performances, multi-level node concatenation provides a clear gain compared to averaging, especially on deeper levels of the temporal pyramid, both on motion and appearance streams. The rational is that multi-level node concatenation preserves better the temporal granularity of actions compared to averaging. Hence, in the remainder of these experiments, we keep concatenation when learning "end-toend" joint combination of appearance and motion streams.
Secondly, we compare the performance of the two settings (shallow and deep) of our multiple aggregation design using both motion and appearance streams taken individually and combined; as already discussed, the parameters w a , w m of this fusion are optimized as a part of the endto-end learning process. Results reported in Table 4 show the complementary aspects of the two streams in all the settings as their fusion brings a clear gain in performance. Moreover, we observe that the contribution of the motion stream is strictly increasing (and a contrario strictly decreasing for appearance stream) as the level of the temporal pyramid increases (see the distribution of w in Table 4). This clearly corroborates the highest impact of motion (compared to appearance) when modeling the temporal granularity of action categories (see later Fig. 5). We also observe a higher positive impact on performances as the depth of our temporal pyramids increases; again, these results are obtained using "concatenation" instead of "averaging", as the former already globally overtakes the latter on motion and appearance streams when taken individually (see

TABLE 2
This table shows level-wise performances using the motion stream both for shallow and deep models. These performances are reported both for "averaging" and "concatenation". In these initial experiments -in order to compare the performances of shallow and deep designs under comparable conditions -we fine-tune only the last fully connected layer of ResNet-101 along with the parameters of the temporal pyramid (TP).  TABLE 3  This table shows level-wise performances using the appearance stream both for shallow and deep models. These performances are reported both for "averaging" and "concatenation". In these initial experiments -in order to compare the performances of shallow and deep designs under comparable conditions -we fine-tune only the last fully connected layer of ResNet-101 along with the parameters of the temporal pyramid.

Fusion
Shallow

TABLE 4
This table shows level-wise performances of joint (2-stream) fusion for both shallow and deep methods. These results are shown only for "concatenation" as the underlying baseline performances reported in tables. 2 and 3 are better than "averaging". In contrast to tables (2) and (3), all the parameters of the whole network (including ResNet) are allowed to vary.
We further investigate the potential of our method using multiple instances of temporal pyramids both for motion and appearance streams as well as their joint fusion. The rationalfrom this setting -resides in the heterogeneity of action categories and their dynamics which may affect the accuracy; indeed, the apex of some actions appears early in video clips while for others later or spread through all the video duration. Hence, instead of learning a single monolithic temporal pyramid per stream, we stack multiple instances of temporal pyramids with different weights β, each one dedicated to a subclass of actions whose dynamics (not category)

TABLE 5
This table shows the evolution of the performances w.r.t different # of temporal pyramids per stream. In order to combine the outputs of these multiple pyramids (when using concatenation), we add a succession of FC+ReLU+BatchNorm to reduce the dimensionality from "63 (number of nodes in TP of 6 levels) × 128 (node dimension) × # TPs" to "128". All these results correspond to temporal pyramids of 6 levels.
are similar 6 . We learn the parameters of these pyramids "end-to-end" as discussed earlier for single pyramids. Table 5 shows the performances w.r.t the number of pyramids. In spite of an increase of the number of training parameters in these multiple pyramids (without any increase of training data), we observe an improvement; we believe that adding extra training data will bring a further and clearer gain in performances.  RGB and OF stand for the number of input RGB frames and the number of optical flow frames used in the appearance and the motion streams respectively. These performances are obtained using a temporal pyramid of six levels.

Sampling, surrogate gradient and efficiency
Table. 6 shows the impact of our method -with and without frame sampling -on the performance of action recognition. These results are obtained using a single pyramid. From these results, it is easy to see that performances get better as the number of sampled frames increases reaching asymptotically the best performances when all the frames are used. This behavior is similar both on motion and appearance streams. However, we notice that motion stream which is based on optical flow data is more sensitive to sampling than appearance stream so the accuracy of the former is clearly proportional to the number of frames. Put differently, motion stream builds a better representation and hence becomes more important for the overall action classification when it is fed with more optical flow data as shown again in Table 6 (settings #6 and #7). However, taking all the frames during backpropagation, comes at the expense of a substantial increase of computation; when considering all the 2.5 millions frames of our videos on UCF-101, training costs 72h (resp. 96h) for appearance (resp. motion) stream using 4 Titan X GPUs (with 12 Gb) and 6. These subclasses of actions are not explicitly defined in a supervised manner but implicitly by allowing enough flexibility in the multiple instances of temporal pyramids in order to capture different (unknown) subclasses of action dynamics.   Table 5). (b-c) Weight distribution of multiple temporal pyramids of motion and appearance streams (corresponding to the fourth row in the same table). Warmer colors correspond to higher weights while cooler colors to lower ones.
100h on the joint stream using 4 Tesla P100 GPUs (with 16 Gb). This high cost results from the large number of visited frames when (re)estimating the gradient, in Eq. 6 w.r.t the parameters of the ResNet, through the epochs of backpropagation. In order to make the evaluation of Eq. 6 (and hence training) more tractable (with a controlled loss in classification performances), we consider a surrogate gradient defined as here P i r stands for a subset of selected frame time-stamps, in a given video V i , that contribute to gradient estimation at the r th epoch. We consider a periodic selection mechanism which guarantees that all the frames are evenly used through epochs; in practice, P i r = {t ∈ [0, T i ], t ≡ r (mod K)} with 1/K being the fraction of frames used per epoch. With this mechanism, gradient evaluation still relies on the entire set of frames in the training set, but their use is distributed through epochs and this makes the evaluation and training process far more efficient while maintaining close performances (see Table. 7). For instance, when K = 24, training is 24× faster compared to the most accurate setting (strategy #8 in Table 6) as only 8 frames are used (on average "per epoch-per video") in Eq. 7 instead of 185; furthermore, since all the frames contribute equally through all the epochs, the loss in accuracy is contained. These performances are obtained on individual and joint streams using the same aforementioned hardware resources.

Comparison against related work
Finally, we compare the performance and the complementary aspects of our method against related state of the art action recognition methods [1], [14], [33], [34], [94] on UCF-101, HMDB-51 and JHMDB-21. The closely related method in [33] is based on deep framewise representations which are aggregated and classified using a hierarchy of multiple temporal granularities. However, the method in [33] differs from the one proposed in this paper in different aspects: first, framewise representations are extracted using ResNet-152 pretrained only on ImageNet and not fine-tuned on UCF-101. Besides, the method in [33] is based only on appearance stream and more importantly, the design principle of our proposed method is deep and consists in weighting the contribution of each level in the temporal pyramid as a part of an "end-to-end" learning process while in [33] this weighting scheme is relatively shallow and excludes the ResNet from training. All these differences explain the significant under-performances of [33] compared to our method as shown in Table. 8. Extra comparisons in Table. 8 also include global averaging techniques as well as spectrogramlike representations. The former produces a global representation that averages all the frame representations while the latter keeps all the frame representations and concatenate them prior to their classifications. Note that these two settings are related to the two extreme cases of our hierarchy, i.e., the root and the leaves. In particular, the spectrogram of a video V with T frames is obtained when the number of leaf nodes, in the hierarchy, is exactly equal to T . Global averaging techniques (shown in Table. 8) include [14]; the latter is based on colorized   , "col-heatM" stands for colorized heatmaps, "Spect" for spectrograms, "A" for appearance, "M" for motion, "2S" for two-streams, "GAP" for global averge pooling and "OF" for optical flow. In our experiments, (i) ResNet-152 is pretrained on ImageNet, (ii) ResNet-101 is pretrained on ImageNet and fine-tuned on UCF-101 (for both appearance and motion) and (iii) ResNet18 is pretrained on ImageNet and fine-tuned on UCF-101 (again for appearance and motion). In these results, the symbol "" stands for "a method does not apply or was not applied (results not available)" in the underlying works.
heatmaps and corresponds to timely-stamped and averaged framewise probability distributions of human keypoints. These colorized heatmaps are fed to a 2D CNN for classification; note that colorized heatmaps provide video-level representations which capture globally the dynamics of video actions without any scheme to emphasize the most important temporal granularities of these actions and this results into low accuracy as again displayed in Table. 8.
The last category of methods (shown in Table. 8) include convolutional networks based on 2D and 3D spatio-temporal filters [1], [94]. These methods are based either on one or two streams; one for motion and another one for appearance followed by a global average pooling. Both methods are similar to ours; they combine motion and appearance streams and their design is end-to-end but clearly differ in their pooling mechanisms and the way frames are exploited. Indeed, these related techniques rely on sampling strategies that vectorize video sequences into fixed length inputs while our method keeps all the frames in order to build temporal pyramids. Another major difference w.r.t our method resides in the huge set used in order to train these related architectures. Nevertheless, while these streams are highly effective their combination with our hierarchical aggregation, through a late fusion 7 , brings a noticeable gain in performances. We also observe the same behavior on all the combinations of our two stream model with other baselines and other related methods (including two stream 3D CNNs [1] and spectrograms [33]); indeed, from the results shown in Table. 8, our hierarchical method brings a clear gain w.r.t most of these methods. Note that some of these models rely on extra datasets (including Kinetics) in order to pretrain their CNNs while our method is trained only on the original datasets.

CONCLUSION
We introduce in this paper a temporal pyramid approach for video action recognition. The strength of the proposed method resides in its ability to learn hierarchical pooling operations that capture different levels of temporal granularity in action recognition. This is translated into learning the distribution of weights in the temporal pyramid, that capture these granularities, by solving constrained minimization problems. Two settings are considered: shallow and deep. The former relies on solving a constrained quadratic programming problem while the latter on optimizing the parameters of a deep network including a temporal pyramid module both on motion and appearance streams as well as their combination. We also consider variants of the deep learning framework that designs multiple instances of temporal pyramids each one dedicated to a particular subcategory of action granularities and also a procedure that allows us to efficiently train the network at the detriment of a slight decrease of its classification accuracy. The advantages of these contributions are established, against different baselines as well as the related work, through extensive experiments on challenging action recognition benchmarks including UCF-101, HMDB-51 and JHMDB-21 datasets. As a future work, we are currently investigating the issue of learning other aggregation schemes, besides hierarchical averaging and concatenation, and also the extension of this method to other benchmarks and other visual recognition tasks.