{'fraction_non_alphanumeric': 0.04461610655885644, 'fraction_numerical': 0.018120781142836516, 'mean_word_length': 4.706282183316169, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Interactive Fusion of Multi-level Features for Compositional Activity Recognition

To understand a complex action, multiple sources of information, including appearance, positional, and semantic features, need to be integrated. However, these features are difficult to be fused since they often differ significantly in modality and dimensionality. In this paper, we present a novel framework that accomplishes this goal by interactive fusion, namely, projecting features across different spaces and guiding it using an auxiliary prediction task. Specifically, we implement the framework in three steps, namely, positional-to-appearance feature extraction, semantic feature interaction, and semantic-to-positional prediction. We evaluate our approach on two action recognition datasets, Something-Something and Charades. Interactive fusion achieves consistent accuracy gain beyond off-the-shelf action recognition algorithms. In particular, on Something-Else, the compositional setting of Something-Something, interactive fusion reports a remarkable gain of 2.9% in terms of top-1 accuracy.


Introduction
Why can humans easily understand a complex action (e.g., "Moving sth. and sth. closer to each other" as shown in Figure 1(c)) even when it is performed with different objects in various environments? Their ability to perform compositional reasoning between entities in the natural world is probably the most plausible explanation [29]. For example, humans often first decompose the scene of Figure 1 (c) into some semantic instances (i.e., two boxes and two hands), and then recognize the activity by observing the relative change of the distance between instances. This ability helps humans to learn some knowledge that is easy to be generalized to the novel environment with unseen combinations. Inspired by this, we hope to make machines show a similar capability in action recognition, namely, compositional action recognition [20,13].
In the era of deep learning, most methods [2,18,38] extract appearance representations by strong backbones for action recognition. Due to the complexity of actions and the limited number of samples, the appearance representation will unavoidably bring inductive bias [32] (e.g., the appearance-based model may confuse the actions in Figure 1 (c) and (d) which have almost the same appearance). Compositional action recognition just requires the model to be partly insensitive to the appearance of objects and scenes. Therefore, it is necessary for this task to extract features from other inputs (e.g., positional information of instances that can be used to easily distinguish Figure 1 (c) from (d)) rather than the only image. Obviously, the positional feature does not work on some actions with the changes in the property of objects, e.g., the paper is torn by hands in Figure 1 (b). Therefore, fusing multi-level features becomes the main issue of compositional action recognition.
However, the features from different levels can vary significantly in modality and dimensionality. In general, appearance features extracted from images have thousands of dimensions, but positional features can even be described by vectors of 4 dimensions. Therefore, it is not appropriate to roughly merge multi-level features in a late fusion way, e.g., [20] directly concatenates appearance and nonappearance features without any interaction between them.
In this work, we integrate multi-level features in an interactive way, i.e., projecting features across different spaces and guiding this process by an auxiliary task that does not require additional supervision. Our approach includes the following three steps. i), Positional-to-Appearance Feature Extraction: builds instance-centric appearance representation from images according to tracklets and combine them with non-appearance features [20] into instancecentric joint representations; ii), Semantic Feature Interaction: builds category-aware pairwise relationships among these joint representations in the latent space to generate semantic features for each instance; iii), Semantic-to-Positional Prediction: projects the semantic features back to positional space by an auxiliary task. Predicting the future state of the instances is a good choice which not only does not require additional annotations but also helps the model understand the temporal changes of instances. Thus, we predict the location and offset of each instance to promote the interactive fusion of multi-level features.
The proposed approach is evaluated on two challenging datasets including the Something-Else dataset [20] and a fine-grained version of Charades [28] annotated by [13]. Experimental results show that our model significantly out-performs state-of-the-art methods [20,34,33] on compositional action recognition. In particular, on the Something-Else dataset, our approach achieves 2.9% and 3.3% boost on top-1 and top-5 accuracy, respectively, compared with the end-to-end model proposed in [20]. Besides, our model shows good generalization ability in the few-shot setting, implying that a better way of feature fusion is helpful to resist over-fitting.

Related Work
Compositionality in Activity Recognition. Human activities are composed of a set of subactions in the temporal domain [27,26], and various subjects and objects in the spatial domain [13,1]. Recent works proposed the compositional annotations or settings based on some popular video-based datasets [7,28]. For example, Materzynska et al. [20] provided object bounding box annotations for the Something-Something dataset [7] and presented a compositional setting in which there is no overlap between the verb-noun combinations in the training and testing set. Besides, built upon Charades [28], Action Genome [13] decomposed activities into spatiotemporal scene graphs as the intermediate representation for understanding them. To control the scene and object bias, Girdhar et al. [5] created a synthetic video dataset, CATER, in which the events are broken up into several atomic actions in the spatial and temporal domain. In this work, we focus on generalizing compositional action to novel environments by integrating different levels of features in an interactive way. Instance-centric Video Representation. For video understanding, previous works [33,18,2,14,4] focused on designing deep and powerful backbones to extract appearance features from each frame, however, in which rich relationships [13,17,19,34] between different instances (objects or hands or persons, etc.) are hard to be mined in the spatial and temporal domain. To this end, some recent researches [1,34,19,37,31] focused towards extracting instance-centric representations from videos and building the spatial and temporal relationships among them. In this paper, we construct instance-centric representations from both low-level and high-level information for compositional action recognition. Prediction in Video. Predicting small image patches [24,25,30] or full frames [3,21,22] in videos has received increasing attention in recent years. A line of recent works [11,16,12,39,36] focused on disentangling each frame into several instances and predict the state of them as mass, location, velocity, etc. Sequence prediction mechanism has also been used in proxy tasks [8,9,23] to learn self-supervised representation for video. In this work, we predict the location and offset of instances from instancecentric semantic representations for promoting the fusion of multi-level features.

Problem Statement
Formally, given a video with T frames, N instances (e.g., objects and hands/persons), and the associated tracklets, we denote the RGB input of this video as I ∈ R T ×H×W ×3 , the tracklets of instances as B ∈ R T ×N ×4 , the category information of instances as C ∈ R T ×N ×1 (only indicates "hand/person" and "object"), and the activity label of this video as l. Compositional activity recognition aims at understanding the unseen combination of action (performed by subjects including hands and persons) and objects in each video.
Based on the above inputs, there are multiple levels of features can be used to understand a complex activity. Such as low-level appearance features extracted from images I that contain the attributes of instances or the environment. Beyond that, we can also obtain high-level positional features that describe the trajectories (i.e., motions) of instances from tracklets B, and the high-level category feature used to identify each instance from C. In general, appearance features take on thousands of dimensions, but non-appearance (e.g., positional and category) features can be represented by vectors of several tens of dimensions. [20] has validated that both low-level and high-level features are useful for understanding a complex compositional activity. But, the features extracted from B, C, and I were simply fused in a late stage. This pipeline ignored that multi-level features often significantly differ in modality and dimensionality.

Interactive Fusion of Multi-level Features
To this end, we propose a simple yet effective interactive fusion framework that leverages high-level information to assist the feature extraction from low-level information and predict high-level future information with an auxiliary loss. The general formulation of this framework can be abstracted as: where A denotes the low-level appearance feature extracted from images I; B and C represent the high-level positional and category information, respectively; l is the activity label of each video. The total loss of this framework is split into the main recognition loss L reg and the auxiliary loss L aux , and λ is used to balance them. The framework has three major components. First, we project B to fit A in a high-dimension feature space and extracts the basic instance-centric joint features by F(B, A); Second, we further explore the semantic interaction among them by G(·); Third, we employ H(·) to project semantic representations from the latent space back to the low-dimensional space of B supervised by an auxiliary loss L aux . Within such a generic framework, we complete the feature fusion through performing multiple interactions between different levels of features. Note that F(·), G(·), and L aux can be designed into different forms with the requirement that F(·) is built on instances rather than frames, G(·) needs further leverage more existing supervision to promote feature fusion in the latent space, and L aux can be free from the additional annotation.

Positional-to-Appearance Feature Extraction
Function F(·) aims at mixing positional and appearance information during the feature extraction stage by projecting a set of boxes on an image. In this work, we adopt a simple way: extracting appearance representation for each instance from the image according to the high-level positional information. This method makes full use of positional supervision to eliminate the scene bias in representations.
Specifically, we sample T frames from each video and employ I3D [2] to extract the spatiotemporal appearance representation from images I. The output of the last convolutional layer is a feature map with the dimensions of where d fea is the channel number. To get N instance-centric features, we apply a RoIAlign [6] on the top of the last convolutional layer to crop and rescale the appearance feature for each instance according to the associated tracklets B. Finally, we denote the appearance features as F app ∈ R T 2 ×N ×dapp . Furthermore, we combine the appearance features with non-appearance features used in [20]. We can easily achieve positional features F bb ∈ R T 2 ×N ×d bb from a quadruple of p t i = c x , c y , w, h (where c x and c y are the center coordinate, and w and h represent its width and height) and category features F cate ∈ R T 2 ×N ×dcate from category information C by word embedding. Then, F bb and F cate are concatenated and embedded into non-appearance representation F non app ∈ R T 2 ×N ×dnon app via an MLP. Finally, we concatenate F app and F non app at the last dimension as basic instance-centric joint representations F ∈ R T 2 ×N ×djoint for each video, where d joint = (d app + d non app ).

Semantic Feature Interaction
As the core elements of our framework, basic instancecentric joint representations extracted from each instance are isolated now, which limits the fusion of different levels of information. To this end, we instantiate function G(·) by constructing pairwise relationships between each instance with the consideration of category information. Details are presented as follows. Compositional reasoning. Different from previous relational modules [20,33,37] that construct relationships between a specific feature node and its neighbors indistinguishably, we propose a compositional reasoning to model the different types of pairwise relationships between each instance with the consideration of their category information. Formally, we represent each video by a set of instancecentric joint features as F ∈ R T 2 ×N ×djoint and perform compositional reasoning among them as: where f t i and g t i denote the basic joint and semantic features for i-th instance at t-th time step. [·, ·] represents a concatenation operation that used to compose a pair of instance-centric features, i.e., f t i and f t j where j = i. E ss , E so , and E oo represent different instance-pair sets (i.e., subject-subject, subject-object, and object-object interactions, where subject indicates hand or person) that can be defined according to category information C. Functions φ ss (·), φ so (·), and φ oo (·) designed to encode different pairwise interactions and ψ S (·) used to fuse them with basic joint features, can be implemented by MLPs. This module outputs instance-centric semantic features G ∈ R T 2 ×N ×dsem . Temporal flow. Given semantic features G, we further aggregate them for each instance over time. To be flexible, we apply RNN [35] architecture to capture the temporal dependencies as: where Z i is the spatiotemporal representation of the i-th instance, θ τ denotes the learnable parameters used in RNN, Concat(·) is used to concatenate all T 2 outputs from RNN, and ψ T (·) is an MLP used to encode the concatenated features. Z = {Z 0 , Z 1 , Z 2 , · · · , Z N } is average pooled as the final video-level representation used to predict the compositional activity.

Semantic-to-Positional Prediction
To promote the fusion between appearance and nonappearance features, we implement H(·) and L aux by predicting the future positional information of each instance from the observed semantic features. Instead of only predicting the accurate coordinates of each instance, we also estimate the offsets of it. Details are described as follows.
Predictor. We first estimate the future state of each instance by the observed features. Formally, given the previous t observed semantic features of i-th instance as {g 0 i , g 1 i , · · · , g t i }, we predict the (t + 1)-th state as: Here, we denote T obs as the number of frames observed by the predictor in each step, thus t ∈ [T obs : T 2 − 1]. RNN is used to modeling the temporal structure hidden in sequence data and the last output is viewed as the predicted stateĝ t+1 i . More importantly, recurrent architecture do not sensitive to the length of input data, which is flexible for modeling the temporal information on sequences of variable length. Therefore, this predictor can share the parameters θ τ with the recurrent architecture used in Eq. 3. Decoder. Given the predicted stateĝ t+1 i ∈ R dsem for the i-th instance in time t + 1, we first estimate its 4dimension positional information via a simple linear layer asp t+1 . Besides, we also estimate the offsets (i.e., the difference between the center coordinates < c x , c y > of the same instance in two consecutive time) of each instance asô t+1 Auxiliary loss. For simplicity, we measure the error between prediction and ground truth by euclidean distance and sum up them over space and time as:

Implementation Details
Input: Our approach takes as input T frames uniformly sampled from each video, and the associated tracklets and category information for N (= 4) semantic instances (hands/persons and objects, no more than four) in the scene. Each frame is resized into the resolution of 224 × 224. Besides, the MLP used to fuse non-appearance features and ψ T (·) used in the temporal flow, are composed of two linear layers with the output dimension of 512, thus d non app = 512 and d joint = 1024. φ ss (·), φ so (·), φ oo (·) and ψ S (·) defined in Eq. 2 are implemented by MLPs each of which is single linear layer with the output dimension of 1024. The RNN used in Eq. 3 and Eq. 4 is implemented by a LSTM [10] with two layers and the dimension of hidden states is equal to 1024. T obs used in prediction is set to T 4 . L reg is a simple cross entropy loss.

Setup
Dataset. We evaluate the proposed approach on two datasets: i), Something-Else [20] (built on [7] with the compositional setting forcing the combinations of action and objects cannot overlap between training and testing sets). Something-Else contains 174 categories of activities but only 112, 795 videos (54, 919 for training and 57, 876 for testing) with the compositional setting; ii), Charades [28] with only 9,601 videos annotated by Action Genome [13]. We still follow the official train-validation split provided in [28] and each video may contain multiple actions out of 157 classes.
In this work, we do not evaluate our approach on the detected bounding boxes mentioned in [20] as only the ground-truth of them is released in [13,20]. Certainly, it is another challenge for this task to be free from timeconsuming ground-truth annotations, which is worth exploring in the future. Methods and baselines. To figure out the effectiveness of different components designed in our approach, we compare the following methods: • I3D: A frame-level baseline that apply a 3D Con-vNet [2,34] built on ResNet-50 to extract spatiotemporal representations.
• Ours-base: A basic version of our model that average pools the basic instance-centric joint representations in the spatial domain as frame-level representations and concatenate them over time followed by an MLP.
• Ours-SFI: One variant of our approach that further explore the latent interactions among the basic joint representations by the block of Semantic Feature Interaction (SFI).
• Ours-SFI + Pred: The complete version of our approach that further apply semantic-to-positional prediction to promote the fusion between different types of features, based on the previous variant.
Moreover, we also compare our approach with recent state-of-the-art methods on object-based video understanding, such as NL [33], STRG [34], and STIN [20]. Ours-NLs, Ours-STRG, and Ours-STIN represent that we build these modules on basic instance-centric joint representations under our framework. Notably, (I3D, Ours-SFI+Pred) and (I3d, STIN + NL) represent that combine the framelevel I3D model with the existing model ensemble but train them separately, all other methods are trained in an end-toend fashion. Training details. We employ SGD with fixed hyperparameters (0.9 momentum and 10 −4 weight decay) to optimize all our models with the initial learning rate 10 −2 . The I3D model used in this work is pre-trained on Kinetics-400 [15].

Results on Something-Else
Similar to [20], we train non-appearance based models (that do not use any appearance information) in 50 epochs and reduce the learning rate to 1/10 of the previous one at epoch 35 and 45. However, we train appearancebased models in 30 epochs and decay the learning rate by the factor of 10 for every 5 epochs, since appearance representation can help the model converge faster. All our models are fed with 8 frames and trained in a batch-size of 72 on this dataset. Ablation study. We first conduct some ablation experiments to evaluate each component of our approach using the ground-truth bounding boxes, and the results are reported in Table 1. All these methods are the variants of our approach, except the I3D.
As we can see that I3D performs poorly when using only 8 frames, though its performance is reported as 46.8% in terms of top-1 accuracy in [20] with 16 frames. With the same setting, our basic interactive fusion framework, i.e., Ours-base (appearance), improves the top-1 and top-5 accuracy by 16.1% and 14.1%, compared with the frame-based model, i.e., I3D (T = 8). Ours-base (non-appearance) can easily surpass the I3D model, indicating the effectiveness of non-appearance features on this task. As expected, Oursbase achieves significant gains in accuracy benefited by fusing both non-appearance and non-appearance features.
Equipped with the interactions among semantic representations, Ours-SFI brings 2.2% and 1.4% improvements for the basic framework on top-1 and top-5, respectively. After that, Ours-SFI + Pred improves the top-1 and top-5 by 1.3% and 1.0% again by introducing auxiliary positional prediction. Finally, we combine Ours-SFI + Pred and I3D ensemble and achieve the best results on this dataset against other variants. Comparisons with the state-of-the-art method. Besides, we also compare our approach with the most related work [20] and some reasoning modules [33,34,20]  be easily plugged into our approach, as shown in Table 2.
Overall, Ours-SFI + Pred is superior to all existing methods and (I3D, Ours-SFI + Pred) achieves state-of-the-art performance on both top-1 and top-5 accuracy.
In particular, Ours-SFI + Pred significantly improves the top-1 and top-5 accuracy by 5.0% and 6.4%, respectively, compared with I3D + STIN + NL. In an ensemble way, (I3D, Ours-SFI + Pred) still gains 2.9% and 3.3% compared with [20]. Compared with NL [33], STRG [34] and STIN [20], Ours-SFI + Pred has remarkable improvements in accuracy benefited from the auxiliary prediction task which promotes the fusion of multi-level features. Few-shot setting. To further evaluate the generalization capability of our approach, we conduct some experiments on the few-shot setting which has 88 base and 86 novel categories proposed in [20]. For a fair comparison, we follow the same settings and training strategies used in [20]. We first train our approach on base categories and then directly finetune it on all the novel categories. More details can be found in [20]. In this work, only 5-shot and 10-shot results are reported in Table 3.
As expected, our approach shows strong generalization on the few-shot setting against other methods. In particular, Ours-SFI achieves the best results on 5/10-shots, compared Method base few shot top-1 top-5 5-S 10-S I3D (T = 8) [2,34] 66.0 89. 8   with previous all joint-training methods [20,33,34] and the frame-based method I3D [34]. With the help of the auxiliary prediction branch, Ours-SFI + Pred further boosts the performance with 1.1% and 1.3% on 5-Shot and 10-Shot, respectively. Ours-SFI + Pred outperforms I3D † + STIN + NL whose I3D is pre-trained with 16 frames by 2.6% in both 5-shot and 10-shot setting, though trail it on the base categories. Finally, we combine our approach and the I3D model in an ensemble way, and achieve state-of-the-art results.

Results on Charades
There are multiple actions in each video of this dataset, the positional prediction of each instance is meaningless because no one knows which action will appear in the next frame. Therefore, we drop the auxiliary prediction branch from our approach and train all models in 50 epochs and decay the learning rate to 1/10 of the previous one at epoch 35 and 45. Similar to previous works [33,34], we combine a sigmoid layer and the binary cross-entropy loss to handle the multi-label classification problem and report the mean average precision (mAP) results in Table 4.
To avoid inaccurate detection, we take only T = 32 sampled from annotated frames provided by [13] as the input of all methods on this dataset. Therefore, with just a part of the original sequence is observed, the baseline method I3D can only achieve 17.9% mAP that is lower than its performance on the original dataset, but it does not affect the following comparisons. We can see that both I3D + STIN and I3D+ STIN + NL trail the I3D model on mAP but Ours-base improves it by 1.7%, suggesting that naive late fusion may harm the joint representation when some levels of features are useless. Equipped with our interactive fusion framework, Ours-STIN achieves a remarkable gain of 6.8%, compared with I3D + STIN [20]. As Method mAP (%) I3D [2,34] 17.4 I3D + STIN [20,34] 14.4 I3D + STIN + NL [20,34] 15.6 Ours-base 19.1 Ours + NLs [33] 19.1 Ours + STRG [34] 21.7 Ours + STIN [20] 21.2 Ours-SFI 25. 2   Table 4. Action recognition on the Charades dataset.
expected, Ours-SFI is superior to recent relational methods (i.e., Ours-NLs/STRG/STIN), benefited by the compositional reasoning which considers the different types of interactions among instances.

Diagnostic Studies
Effect of λ in objective function. The hyper-parameter λ is designed to balance the contribution from the auxiliary prediction task and the main recognition task. As shown in Fig 3, Ours-SFI + Pred can achieve stable benefits (above the dashed blue line) from the auxiliary prediction task when 3 ≤ λ ≤ 9, compared with Ours-SFI. Thus, we simply set λ to 5 for the proposed approach in experiments.  Category analysis. Overall, our approach surpasses I3D + STIN + NL [20] on 84.5% categories and the top categories are shown in Figure 4 (a). In particular, our approach is far beyond [20] on some categories, such as "pouring sth. out of sth.", "spinning sth. so it continues spinning", etc., which involve not only relative motions but also object properties. It is understandable that our method is worse than I3D + STIN + NL on "showing a photo of sth. to the camera" (in Figure 4 (e)) in which only a static photo in the video. Visualization. We also visualize some predictions on the Something-Else dataset in Figure 4. We can see that I3D + STIN + NL is easy to learn the bad appearance or positional  bias due to the rough late fusion of different features. For instance, in Figure 4 (b), I3D + STIN + NL prefers to capturing the concept of "sth. is inside sth." when sees a container containing certain objects similar to the I3D model, but ignores the negligible yet very important positional selfchange of objects. Besides, I3D + STIN + NL ignores the concept of "pretend" in Figure 4 (c) due to the worse bias from positional features. However, our approach generates predictions by fusing different levels of features in an interactive way, rather than prefer to one of them.

Conclusion
In this work, we first point out that main challenge of compositional action recognition is to integrate features that often differ significantly in modality and dimensionality. To alleviate this problem, we propose a novel interactive fusion framework to project these features across different spaces and promote the fusion process by an auxiliary prediction task. Our framework is composed of three steps, namely, positional-to-appearance feature extraction, semantic feature interaction and semantic-to-positional prediction. Experimental results on two action recognition datasets show the robust generalization ability of our interactive fusion framework on compositional action recognition. This work delivers that the understanding of human action is actually a process of gradually fusing features from multiple sources with diverse strategies. We hope that this work can inspire future research on compositional activity understanding.