{'fraction_non_alphanumeric': 0.034758511480601745, 'fraction_numerical': 0.009263657957244655, 'mean_word_length': 4.809797608095677, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
CTRN: Class-Temporal Relational Network for Action Detection

Action detection is an essential and challenging task, especially for densely labelled datasets of untrimmed videos. There are many real-world challenges in those datasets, such as composite action, co-occurring action, and high temporal variation of instance duration. For handling these challenges, we propose to explore both the class and temporal relations of detected actions. In this work, we introduce an end-to-end network: Class-Temporal Relational Network (CTRN). It contains three key components: (1) The Representation Transform Module filters the class-specific features from the mixed representations to build graph-structured data. (2) The Class-Temporal Module models the class and temporal relations in a sequential manner. (3) G-classifier leverages the privileged knowledge of the snippet-wise co-occurring action pairs to further improve the co-occurring action detection. We evaluate CTRN on three challenging densely labelled datasets and achieve state-of-the-art performance, reflecting the effectiveness and robustness of our method.


Introduction
Action detection is a challenging computer vision problem which targets at finding precise temporal boundaries of actions occurring in an untrimmed video. Many studies on action detection focus on videos with sparse and well-separated instances of action [4,23,25]. For instance, action detection algorithms on popular datasets like THUMOS [12] and Activi-tyNet [2] generally learn representations for single actions in a video. However, in daily life, human actions are continuous and can be very dense. Every minute is filled with potential actions to be detected and labelled. The methods designed for sparsely labelled datasets are hard to generalize to such real-world scenarios.
Towards this research direction, several methods [9,17,18] have been proposed to model complex temporal relationships and to process datasets like Charades [19], TSU [7] and Mul-tiTHUMOS [24]. Those datasets encompassing real-world challenges share the following characteristics: Firstly, the actions are densely labelled and background instances are rare in these videos compared to sparsely labelled datasets. Secondly, the video has rich temporal structure and a set of actions occurring together often follows a well defined temporal pattern. For example, drinking from bottle always happens after taking a bottle and reading a book also related to opening a book in Fig 1. Finally, humans are great at multitasking, multiple actions can co-occur at the same time. For example, reading book while drinking water.
Existing methods have mostly focused on modelling the variation of visual cues across time locally [15] or globally [17] within a video. However, these methods take into account the temporal information without any further semantics. Real-world videos contain many complex actions with inherent relationships between action classes at the same time steps or across distant time steps (see Fig. 1). Modelling such class-temporal relationships can be extremely useful for locating actions in those videos.
To this end, we introduce Class-Temporal Relational Network (CTRN) to harness the relationships among the action classes in a video. To explore such relations, CTRN first filters the class-specific representation from the input features at each time step in a video. Then the transformed per-class representation is utilized for modelling the inter-class relations. (1) Across different time-steps, a graph-based layer is proposed to learn the dependencies between different action classes of the video. This learned relation map is shared among all the time-steps to refine the action features from the related actions (e.g. open the book and read book). Then a temporal layer is used to aggregate features from the same class over time to enable the graph-based layer to explore both short-term and long-term class dependencies.
(2) At the same time step, a graph-based classifier is proposed to leverage the privileged co-occurring action probabilities to improve co-occurring action detection.
To summarize, the main contributions of this work are: 1) A graph-based module to explore action class relations across different time-steps. 2) A graph-based classifier to tackle the co-occurring action challenge. 3) We evaluate our model on three challenging densely labelled datasets for the action detection task. Our method outperforms state-of-the-art results using fewer parameters and FLOPs.

Related Work
Action detection has received a lot of interest in recent years [6,8,11,17]. In this work, we focus on densely labelled action detection for handling videos with additional temporal relationships between different action classes [7,19,24]. Different from the sparsely labelled detection methods which output a sparse set of action snippets [2,12], densely labelled de-tection methods need to predict what action is occurring at every snippet in a video. There are two principle frameworks for densely labelled action detection: the anchor-based [4,22] and the Seq2Seq-based [6,15] frameworks. The anchor-based frameworks are often slow and suffer from over-generated proposals and rigid boundaries. Seq2Seq-based [18] frameworks apply temporal filters over snippet-wise features with a snippet-level classification, therefore, it interprets the sequence of images to a sequence of predictions. Compared to anchor-based methods, the Seq2Seq methods achieve better performance for action detection on datasets with densely distributed actions. This is mainly because of the combinatorial explosion of action proposals generated by the former methods. Thus, the recent methods are following the Seq2Seq framework: Lea et al. [15] introduced temporal convolutional network (TCN) for the action detection task. This method increases the temporal reception field by using dilated convolutions to model long temporal patterns. Similarly, Piergiovanni et al. [18] introduced Temporal Gaussian Mixture (TGM) layers. In contrast to standard convolution layer, TGM computes the filter weights based on Gaussian distributions, which enables TGM to learn longer temporal structures with a limited number of parameters. However, both methods focus only on the temporal modelling while overlooking the action class relations in the untrimmed videos. Although Super-event [17] models the latent contextual representation among actions, the modelled Super-event represents only the latent correlation among the time steps and not among the action classes. Most related to our research direction, Tirupattur et al. [21] introduced MLAD that can explore the class-temporal relations with a set of self-attention layers: an inter-class attention map for every time-step and an inter-time attention map for every action class. However, the large number of attention maps leads to huge computational costs for long untrimmed videos and hence, limits the model to learn the discriminative relations among the action classes. To tackle this, we propose CTRN, which is a graph-based model. Different from MLAD, CTRN explores the action class relation shared by all the time steps but in different temporal scales. This design enables CTRN to effectively handle both short-term and long-term action relations simultaneously. Moreover, we propose a G-Classifier that focuses on the co-occurring actions taking into account the inter-dependencies of the actions in the training distribution. The proposed method is introduced in details in the following section.

Proposed Method
In this section, we formulate the proposed end-to-end model Class-Temporal Relational Network (CTRN) for action detection. As shown in Fig. 2, our model is composed of four major components. The Visual Encoder encodes the video into a sequence of snippet-level spatio-temporal representation. This representation is fed to a Class-temporal Relational Network (CTRN) that predicts the action labels at each time instant. The sub-components in CTRN consist of the following: Firstly, a Representation Transform Module, which transforms the mixed visual representation into a class-wise representation. Secondly, a Class-Temporal Module explores the action class relations across different time-steps and at different temporal resolutions. Finally, a G-Classifier which classifies the class-temporal features into action categories. Unlike previous binary classifiers [17,18] that overlook the dependencies between the action classes, G-Classifier leverages the privilege class dependencies within the training data, thus improving the co-occurring action detection performance. In the following, we introduce these modules in details.

Visual Encoder
Similar to most action detection models [15,17,18], our model processes the features on top of video snippet representations extracted from 2D/3D CNNs. In this work, we use spatio-temporal features extracted from RGB and Optical Flow (OF) I3D networks [3] to encode appearance and motion information respectively. Then, a video is divided into T non-overlapping snippets, each snippet consisting of 16 frames. The inputs to the RGB and Flow deep networks are either the color images or the corresponding OF frames of a snippet. We stack the snippet-level features along the temporal axis to form a T × D 1 dimensional video representation, denoted as X. The action instances in X are always longer than a snippet and their visual representation mixes information of all action classes. As a result, X is not discriminative enough, and needs both temporal and class modelling. To this end, we develop the class-temporal relationship from the input representation X within CTRN which is described in the following. Note that the model architecture remains the same for both RGB and OF streams.

Representation Transform Module
The input X is first fed into the Representation Transform Module (RTM). The goal of this module is to transform the input to a class-specific representation and to lightweight the channel size to facilitate the following computation. In practice, RTM duplicates the input features C times into a new dimension representing the action classes followed by a channelmixer MLP with non-linear activation and dropout.MLP is the linear transformation layer [1] to do the linear projection. The equation can be formulated as: where X ∈ R T ×C×D 2 is the output representation of RTM. D 2 = D 1 β in which β is larger than 1 to shallow the channel size. In order to learn class-specific representation, we embed an auxiliary branch with a G-classifier that maps X to the action labels (see Fig. 2). This transformed feature representation is further exploited to explore the class and temporal relations in the subsequent modules of the network.

Class-Temporal Modeling
The Class-Temporal Module (CTM) is the key component of CTRN that exploits the classtemporal relations of its input feature. Inspired by the recent success of Graph Convolutional Network (GCN) in relational reasoning [11,14,23,25], we build this module with GCN. The objective of this component is to update the feature representations by propagating the information across different classes and across different time steps. For modelling the action class relations, we introduce a Class-GCN (C-GCN) layer while the traditional Temporal Convolutional Network (TCN) layer [15] is utilized to aggregate the temporal information.
The combination of C-GCN and TCN enables CTM to capture the class semantic information along different temporal hierarchies. Thanks to the learnable graph structure, C-GCN is adaptive with the temporal scale set by TCN.
In the following, we first introduce how we map the feature representation to the graph structure and then we introduce the CTM components.

Representation-to-Graph Mapping
For GCN to process the action relations, the data is to be converted into a graphical structure. As we have transformed the representation into the class-specific format, thus each vertex of the graph represents an action class at a time step with an embedding vector belonging to R D 2 . In total, the graph consists of C × T vertices whose topology is defined by an adjacency matrix (A C ). This matrix determines whether there are connections (i.e. relations) and its weights determine the intensity of the connections.

Class-GCN (C-GCN)
Class-GCN aims at performing the cross-class reasoning over the constructed graph representation. The relations between the many action instances are complex and are different across videos. Besides, multiple C-GCNs are stacked in CTM through which C-GCNs capture different levels of semantic information. Consequently, the graph adjacency A C learns from the data itself for it to be adaptive across different temporal scales.
In practice, A C ∈ R C×C is parameterized and is optimized together with other parameters in the training process. Moreover, to differentiate the class relations owing to different videos, the adjacency matrix A C learns the inter-dependencies among the classes using a self-attention mechanism. For this, the input feature X C in ∈ R D 2 ×T ×C is first embedded using bottleneck convolutional layer (i.e. 1 × 1). After that, the output feature maps are rearranged into R D 2 T ×C and R C×D 2 T followed by a matrix multiplication. The value of the resultant matrix is then normalized by a softmax activation. Now, the superimposed adjacency matrix A C can be formulated as: where X C in is the input of the C-GCN, and W 1 and W 2 are the weights of the bottleneck convolutions. Each value in this matrix can be seen as a soft edge between two vertices. The learned graph is shared across different time-steps but unique for different layers and videos. This design choice can capture the inter-class dependencies in a video and makes C-GCN scalable across different temporal scales. Finally, we perform the graph convolutional operation with the formulation in [14]: where W 3 ∈ R D 2 ×D 2 is the learnable weight matrix. The operation with A C and with W 3 represents the message passing and vertex feature updating, respectively. Finally, X C out is rearranged to R D 2 ×T ×C .

CTM Block
As shown in Fig. 2, there are L blocks in CTM, each block is composed of a C-GCN and a TCN layer along with batch normalization and non-linear activations. To stabilize the training, two residual connections are added in each block. As mentioned earlier, TCN [15] aggregates the features across the temporal dimension while increasing the size of the temporal receptive field. In this work, we set a fixed kernel size K for all the TCNs. Thanks to the hierarchical structure of CTM, C-GCN can focus on short-term action-dependencies in lower blocks and long-term action dependencies in higher blocks. The refined feature representation from the last block is fed into G-Classifier for the snippet-level classification.

G-Classifier (G-Clf)
Finally, we introduce a graph-based G-Classifier to perform the final snippet-level classification. In action detection, multiple actions could happen simultaneously; thus, prior knowledge of inter-dependencies among different action classes can benefit in making precise predictions. Inspired from [5] in image recognition, we introduce a GCN-based classifier in our task, which has different working mechanisms. In the previous work [5], the input of GCN is the word embedding of the label and the output is the classifier weights between the feature representation and the prediction. In contrast to that, our input and output to the GCN are the video representation and the prediction scores respectively. As our input is directly the video representation, thus the action occurrence information within each snippet are captured, which benefits the information propagation between relevant classes through graph connections. Compared to the standard binary classifier [17], G-Classifier has an additional message passing step between the potential co-occurring action pairs, thus improving the co-occurring action detection performance. Different to C-GCN, G-Classifier models only the snippet-level features and focuses only on the actions that occur simultaneously.
In practice, firstly, we compute the co-occurrence probabilities of all the action pairs in the training snippets. M i j indicates the concurring times for action class C i and C j . Then, the conditional probability matrix P i j = P(C j |C i ) is given by: 5) where N i indicates the occurrence times of C i in training set, and P i j ∈ R C×C indicates the probability of class C j given that C i occurs at the same time. In fine-grained action datasets, some rare co-occurrences may add noise for detecting other common actions, and the number of co-occurrences from training and test set may not be completely consistent. In this work, we perform a thresholding operation to binarize the conditional probability matrix to filter the noisy edges and make the classifier more robust to inconsistent action classes. If P i j ≥ θ , A S i j is assigned 1, otherwise 0, where θ is the threshold. The computed co-occurrence matrix A S is a binary correlation matrix which in turn defines the adjacency matrix of the graph for G-Classifier. The feature of a node is computed by the weighted sum of its own features and the adjacent nodes' features. However, the binary correlation matrix may change the feature   scale [14] and make the node feature over-smoothed [16]. To alleviate this problem, we normalized the A S following the re-weighted scheme in [5]. Different to the learnable adjacency matrices in C-GCN, A S is fixed during training. The formulation of this G-Classifier is given below: where S is the prediction score, σ is the sigmoid activation. X L is the output feature from the last block of the Class-Temporal Module, and W S ∈ R 1×D 2 are the learnable weights of the G-Classifier. To learn the parameters, we optimize the multi-label binary cross-entropy loss with the prediction results from the RTM and CTM. The total objective is formulate as: where α is a weighting factor. Thus, by jointly optimizing both the entropy losses, the model learns the relevant action labels per segment along with learning the class-specific semantics across the Representation transform module.

Experimental Settings
Datasets. We evaluate our method on three densely labelled action detection datasets: Charades [20], TSU [7] and MultiTHUMOS [24]. These datasets contain videos of different types: (1) sports and daily living videos, (2) short and long videos. We follow the original settings of these datasets for action detection. All these datasets are evaluated by the perframe mAP.  Table 2: Evaluation on the Charades dataset using the action-conditional metric [21]. P AC -Action-Conditional Precision, R AC -Action-Conditional Recall, F1 AC -Action-Conditional F1-Score, mAP AC -Action-Conditional Mean Average Precision. τ indicates the temporal window size. τ = 0 corresponds to the actions occuring at the same time.
Implementation. To have a fair comparison with previous works [17,18,21], our network is built on top of I3D, where D 1 is 1024 and D 2 is 64. Dropout probability is 0.3. For CTM, we choose a 5-block (L) structure. For C-GCN, the adjacency matrix is initialized by 1 and normalized by columns. In TCN, the kernel size K is 9 and padding rate is 4. For G-Classifier, θ is set to 0.05. While learning the parameters, the weighting factor α is 1.2 and the random seed is fixed. We use Adam optimizer [13] with an initial learning rate of 0.001, and we scale it by a factor of 0.3 with a patience of 10 epochs. The network is trained on a 4-GPU machine for 300 epochs. For two-stream network, a mean pooling is performed between the prediction logits of the RGB and Flow streams.

Comparison with State-of-the-Art Methods
The proposed CTRN is compared with previous state-of-the-art methods on the Charades, TSU and MultiTHUMOS datasets in Table 1. Our proposed method outperforms current state-of-the-art methods on all three datasets. For example, +6.9% (relatively +37.5%) w.r.t. MLAD [21] on Charades while using only RGB. We then show the ability of CTRN capturing action co-occurrence, we evaluate with the action-conditional metric [21] in Table 2.
Compared with state-of-the-art methods, our method achieves higher performance on all action-conditional metrics showing that CTRN effectively models action dependencies both within a time-step (i.e. co-occurring action, τ = 0) and throughout time (τ > 0). To confirm the advancement of our method, we present further comparisons with MLAD. Firstly, we compare the model efficiency and complexity in Fig. 3. MLAD is about 2 times larger in parameters and 3.5 times larger in FLOPs than CTRN while processing the same batch of videos. Hence, our method is more lightweight and computationally efficient than MLAD. Secondly, we visualize the detection results of an example video in Fig. 4 for both MLAD and CTRN. Qualitatively, we find that CTRN can detect the actions more precisely than MLAD.

Ablation studies
Firstly, in Table 3, we study the complementation of the components in the proposed network on the Charades dataset. We first discuss how RTM leads to a better feature representation of the input spatio-temporal feature map from I3D. RTM is an essential pre-step before class-temporal modelling. Thanks to RTM that filters the class-specific feature, the model can slightly improve the detection performance (+0.5%). We then explore how the different components in CTM affects the action detection performance. We find that both C-GCN    Secondly, we analyse if G-Classifier can better detect the co-occurring actions. We compared our method with the standard binary classifier [17], Chen et al. [5] by computing the mAP with the snippets containing more than one action. All three classifiers are build upon the same backbone (CTRN). We find that the standard binary classifier, Chen et al. and G-Classifier achieve 24.1%, 25.3% and 27.6% respectively on Charades, reflecting that G-Classifier can better detect the co-occurring actions.
To further explore the relationship between inter-classes and temporal modules, we perform ablations with C-GCN and TCN having different numbers of blocks. In Table 4, we find that while having more blocks: C-GCN improves marginally since it captures action relations in a single temporal scale; TCN improves the performance with more blocks due to increased temporal receptive field; C-GCN+TCN achieves the best performance than TCN since C-GCN can capture action relations in more temporal scales when combined with TCN (see the improvements for 5 blocks).
Finally, we explore how hard thresholding affects the performance of G-Classifier. In Table 5, we find that G-Classifier performs better with the threshold since hard threshold filters the noisy edges.

Qualitative Analysis
In Fig. 5, we show the adjacency matrix of G-Classifier in Charades (157 classes), which provides the information of all the co-occurring action pairs with high probabilities. For example, holding a vacuum & tiding something on the floor and fixing hair & watching Figure 6: Visualization of the learned C-GCN adjacency matrix A C for different layers. Here, we visualize the 1 st , 3 rd and 5 th block's adjacency matrices. For simplicity, we provide only the relevant action classes in the example video.
in a mirror are the actions always occur at the same time. Prior access to such privilege knowledge is crucial for detecting the co-occurring actions in the densely labelled videos.
In CTM, TCN is used to aggregate the temporal information which enables C-GCN to explore action relations at different temporal scales. To validate the usage of these layers, in Fig. 6, we visualize the learned adjacency matrix of C-GCN from three different blocks. We find that in Block 1, C-GCN focuses on capturing the contextual information pertaining to locally related action classes. For example, eat sandwich & hold sandwich and drink water & hold cup are always occurring closely in the video. Then we find that, Block 3 has increased the temporal receptive field, thus, C-GCN can capture the long-term dependencies between distant action classes. For example, Pour water and Drink water. Finally, Block 5 possess the largest receptive field where each local snippet feature contains the whole video information. Therefore, C-GCN in this block models all the potential action relations in the video, resulting in many activated links in the adjacency matrix.

Conclusion
In this work, we propose a novel class-temporal relation network for action detection. This network can handle both action class relations at the same time-step, and across different time-steps by using its three key components, namely Representation Transform Module, Class-Temporal Module, and G-Classifier. The network is evaluated on three challenging densely labelled datasets and achieves state-of-the-art performance on all of them. Furthermore, this network has less computation cost and parameters than the representative baseline method. For future perspectives, we will investigate combining C-GCN and TCN into a single layer.