{'fraction_non_alphanumeric': 0.03489003529731197, 'fraction_numerical': 0.00984251968503937, 'mean_word_length': 5.328608247422681, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Human-Centered Prior-Guided and Task-Dependent Multi-Task Representation Learning for Action Recognition Pre-Training

Recently, much progress has been made for self-supervised action recognition. Most existing approaches emphasize the contrastive relations among videos, including appearance and motion consistency. However, two main issues remain for existing pre-training methods: 1) the learned representation is neutral and not informative for a specific task; 2) multi-task learning-based pre-training sometimes leads to sub-optimal solutions due to inconsistent domains of different tasks. To address the above issues, we propose a novel action recognition pre-training framework, which exploits human-centered prior knowledge that generates more informative representation, and avoids the conflict between multiple tasks by using task-dependent representations. Specifically, we distill knowledge from a human parsing model to enrich the semantic capability of representation. In addition, we combine knowledge distillation with contrastive learning to constitute a task-dependent multi-task framework. We achieve state-of-the-art performance on two popular benchmarks for action recognition task, i.e., UCF101 and HMDB51, verifying the effectiveness of our method.


INTRODUCTION
Action recognition is a hot topic in the computer vision community. It has many practical applications such as intelligent surveillance, human-computer interaction and behavior analysis [1]. The critical challenge of video-based action recognition is to model the complex spatial-temporal information in videos, which is more difficult than understanding static images [2]. Early works usually follow a two-stream paradigm [3,4,5,6] or exploit 3D convolutional neural network (3D CNN) [7,8,9,10] to explore the visual appearances and temporal dynamics. However, these supervised methods need great annotation effort [11]. Recently, many works focus on action recognition pre-training with self-supervised learning frameworks without using categorical labels. These ap- proaches usually follow the design of pretext tasks for representation learning, such as temporal shuffle [12], future frame prediction [13], video-based space-time cubic puzzle completion [14], and contrastive learning-based tasks [15,16,2].
Although much great effort has been made for video representation learning, two main issues remain for existing pretraining methods. The commonly used contrastive learning pre-training paradigm emphasizes the instance-level similarity [17], which can hardly capture abundant semantic information in videos, resulting in neutral and less informative representation. Therefore, it further causes severe performance degradation on many specific downstream tasks with very different objectives compared with the pre-training stage. Secondly, many multi-task learning-based pre-training frameworks fail to consider the potential conflicts among different and inconsistent objectives from multiple tasks, leading to sub-optimal solutions [18,19].
To address the above issues, in this paper, we propose a novel prior-guided and task-dependent multi-task representation learning framework for video-based action recognition pre-training. First, we incorporate the human-centered prior by distilling informative knowledge from a human parsing teacher model with an encoder-decoder network. This is based on the intuition that the informative human parsing knowledge can reflect human actions, which is well-aligned with the downstream action recognition objectives. An example is shown in Fig. 1 for a better demonstration. In addition, we also combine the contrastive learning with both appearance and motion consistency into a multi-task learning framework. To avoid the potential conflict from multiple tasks, task-dependent models are employed. The generated task-dependent representations are further combined for downstream tasks. The framework of our proposed method is demonstrated in Fig. 2. We conduct extensive experiments for action recognition on UCF101 and HMDB51 datasets and achieve state-of-the-art (SOTA) performance, verifying the effectiveness of our proposed method.
The main contributions of this paper are summarized as follows: 1) we present a novel framework that incorporates the human-centered prior for representation learning by knowledge distillation (KD) from the human parsing teacher model; 2) we employ a multi-task learning framework with the task-dependent representation learning strategy; 3) we conduct experiments on action recognition task, demonstrating the effectiveness of the proposed multi-task learning framework for video representation learning.

RELATED WORK
Self-Supervised Learning for Visual Representation. Selfsupervised learning aims at learning discriminative representation by leveraging information from unlabeled data. Most works have explored the self-supervised visual representation learning based on the design of pretext tasks, such as image inpainting [20], permutation [21], predicting jigsaw puzzles [22], and contrastive learning strategies. Recently, the extension from image to video representation learning has become increasingly popular due to the richer temporal information of videos. Like representation learning for images, the selfsupervised video representation learning also focuses on the design of pretext tasks yet with the extension of considering temporal consistency in video clips. For example, [23,12,24] attempt to shuffle the frame and clip order along the temporal dimension; [15] proposes a pretext task to predict future frames; [14] learns video features through designing spacetime cubic puzzles, and [25] proposes the SpeedNet to predict the motion speed in videos. In addition, contrastive learning is also widely used in the design of pretext tasks. Specifically, [15] proposes a dense predictive coding method with contrastive loss for video frames; [16] exploits contrastive multi-view video coding with the inspiration by [26] for image coding; [2] proposes a relative contrastive speed perception task to learn the motion information of videos. However, the representation by self-supervised training may be not well-aligned with the objectives of downstream tasks, leading to severe performance degradation.
Video-Based Action Recognition. Some works of action recognition follow a two-stream architecture to model ap-pearance and temporal information separately. For example, [3] exploits both spatial 2D CNN and temporal CNN to extract the spatial information and motion information on RGB frames and optical flows. [27] utilizes a shift module for better capturing temporal information. [28] proposes a slow-fast network to extract spatial semantics and temporal motion information in the video clips. However, such 2D CNN-based methods adopted in the two-stream architecture have limited ability to capture the dynamics of visual tempos [29]. To address such issues, some recent works of action recognition is based on 3D CNN and its variants, that extract both appearance and temporal information jointly. Specifically, [8] firstly employs 3D convolutions on adjacent frames to model the spatial and temporal information of video. [9] inflates the pre-trained 2D convolutions to 3D convolutions kernels. [10] decomposes 3D convolution kernels into a 2D+1D paradigm to improve the performances of 3D CNNs. [30] proposes a self-attention mechanism to model long-range temporal dynamics of videos. Despite the great progress has been made in recent works, how to better generate more informative representations with self-supervised pre-training framework for action recognition is still under explored and needs further study.

PROPOSED METHOD
In this section, we present our prior-guided and taskdependent multi-task representation learning framework for video-based action recognition pre-training. We firstly describe how to incorporate the human-centered prior information by distilling informative knowledge from a human parsing teacher model with an encoder-decoder network, to enrich the semantic capability of the representation. Then, we combine knowledge distillation (KD) with video contrastive learning into a multi-task learning framework to boost the discrimination with fused task-dependent representations.

Human-Centered Prior Knowledge Distillation
Since most action recognition videos are related to human actions, we incorporate the human-centered prior into representation learning with a pre-trained human parsing guided teacher network. Denote f l (·) and h h (·) as the low-level encoder and high-level encoder of the human prior representation module, respectively, where the low-level encoder f l (·) shares the weights across multiple tasks. Given the input video clip X, the human prior representation z h can be obtained by To guide the learning of the human prior representation, we employ a pre-trained human parsing network f t (·) as the teacher model. Given the middle frame of input video clip Fig. 2. Overview of our proposed framework. Video clips are firstly augmented and then sent to a shared low-level encoder to generate common low-level feature maps across multiple tasks. Then the low-level feature maps are sent to two distinct tasks, namely the human-centered prior knowledge distillation task and video contrastive learning task, following a multi-task learning framework. The knowledge distillation branch is supervised by a pre-trained human parsing teacher model, while the contrastive learning branch combines both motion and appearance contrastive relationships. Task-dependent embeddings are learned from two distinct high-level encoders and concatenated as the final representation.
X m , the teacher model can generate the human parsing segmentation feature map F t as follows, In the meanwhile, the prior representation is fed into a decoder g h (·) to generate a parsing segmentation as follows, We assume both F t and F h are normalized feature maps after the softmax operation. F h is supervised by the teacher model with the KL-divergence loss L KL as follow, where i, j represent the spatial index, c represents the segmentation class index, C is the number of segmentation classes, and N is the number of spatial dimensions. With human parsing guided learning, the prior representation z h should contain rich semantic information related to human-centered actions.

Task-Dependent Multi-Task Learning
To learn discriminative representations among videos, inspired by [2], we also combine the video contrastive learning module into a multi-task learning framework. From the output of the low-level encoder f l (·), we stack another highlevel encoder h c (·), which is different to h h (·) used in the human prior module, to generate contrastive representations z c as follows, z c = h c (f l (X)).
Two more projection heads, i.e., g m (·) and g a (·), are further employed to learn motion contrastive and appearance contrastive with the following margin ranking loss and In-foNCE loss [31], where ) represents the distance between the anchor motion embedding to the embedding with the same/different playback speed [2], and q +/− = exp(d(g a (z c ), g a (z +/− c ))/τ ) represents the similarity between anchor appearance embedding and the embedding from the same/different video clips.
With the human prior knowledge distillation module and video contrastive module, we form a multi-task learning framework with the following total loss, where λ k , λ m and λ a are the weights of individual loss. Instead of widely adopted multi-task learning approaches that different tasks share the common representation, we use taskdependent and uncorrelated representations z h and z c to learn different semantic information from videos, which can better avoid the conflict across different tasks. We use the concatenated representation [z h ||z c ] as the final representation of each video clip.

Implementation Details
We adopt Kinetics-400 [9] dataset for self-supervised pretraining. We employ the SOTA human parsing model SCHP [32] as the pre-trained teacher model for human-centered prior knowledge distillation. After the pre-training process, we finetune the model on UCF101 [33] and HMDB51 [34] datasets for downstream action recognition task. We evaluate the performance based on top-1 and top-5 accuracy (Acc@1, Acc@5, respectively). More implementation details are demonstrated in the Supplementary Material.

Evaluation on Video Action Recognition Task
Comparison with SOTA Methods. We compare our method with other SOTA methods on UCF101 and HMDB51 datasets. We report top-1 and top-5 accuracy with C3D [8] and R(2+1)D [10] architectures in Table 1. RSPNet * denotes Ablation Study. We conduct ablation studies on the effectiveness of each component of our proposed method on UCF101 and HMDB51 datasets with the results shown in Table 2. Denote the removal of the knowledge distillation module as "w/o KD", and the use of task-independent representation with the shared representation across multiple tasks as "TI". It can be found that there is a significant degradation for the corresponding variants compared with the full model, further demonstrating the effectiveness of our proposed method on individual components.
Visualization. To better reveal the effectiveness of our framework, we show some visualization examples of the region of interest by using class-activation map (CAM) technique [40] in Fig. 3. The produced heatmap is added on the original video frames for each example in the figure. The first row shows the results from RSPNet [2], while the second row represents the results of our proposed method. It is shown that our method concentrates more on the human action, while RSPNet is distracted by uncorrelated background regions. The visualization results demonstrate the validity of our motivation that incorporates human-centered prior knowledge for multi-task pre-training to learn discriminative representations.
More visualization examples are included in Supplementary Material.

CONCLUSION
In this paper, we present a novel prior-guided and taskdependent multi-task representation learning framework for video-based action recognition pre-training. First, we incorporate the human-centered prior information by knowledge distillation from the human parsing teacher model to enrich the learned representation. Moreover, we follow the multitask learning framework with the task-dependent representation learning strategy to solve the conflict of multi-task training paradigm. Experimental results on the action recognition task demonstrate the effectiveness of the proposed multi-task learning framework for video representation learning.