{'fraction_non_alphanumeric': 0.03191430622273756, 'fraction_numerical': 0.013914195792612225, 'mean_word_length': 4.987438016528926, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Temporal Action Detection by Joint Identification-Verification

Temporal action detection aims at not only recognizing action category but also detecting start time and end time for each action instance in an untrimmed video. The key challenge of this task is to accurately classify the action and determine the temporal boundaries of each action instance. In temporal action detection benchmark: THUMOS 2014, large variations exist in the same action category while many similarities exist in different action categories, which always limit the performance of temporal action detection. To address this problem, we propose to use joint Identification-Verification network to reduce the intra-action variations and enlarge inter-action differences. The joint Identification-Verification network is a siamese network based on 3D ConvNets, which can simultaneously predict the action categories and the similarity scores for the input pairs of video proposal segments. Extensive experimental results on the challenging THUMOS 2014 dataset demonstrate the effectiveness of our proposed method compared to the existing state-of-art methods for temporal action detection in untrimmed videos.


I. INTRODUCTION
Temporal action detection has risen much attention in recent years because of the continuously booming of videos in the Internet. This task is a very challenging problem, given a long untrimmed video, action detection aims to predict the action categories and also localize the start and end time of actions of interest. However, actions of the same label could look much different in different poses, duration, background and so on. At the same time, actions of different classes may show many similarities. Such variations within the same action class and similarities within different action classes could make action detection more challenging, especially in untrimmed videos. There are multiple activities per video and many similarities between different action classes in the large-scale video dataset THUMOS 2014 [8], like CliffDiving and Diving as shown in Fig. 1 (a) and (b). There are also large variations in pose of the same action class, like CricketBowling as shown in Fig.  1 (c) and (d). Therefore, it is an essential topic in temporal action detection task to reduce the intra-action variations while enlarge the inter-action differences.
Current methods [13], [14], [25], [31] for temporal action detection always adopt the proposal-classification framework, which has been very successful in object detection. This proposal-classification framework consists of three steps. Firstly, action proposals are generated as candidates by temporal sliding windows. Secondly, the candidate segments are used to train a classifier for action recognition. Finally, some postprocessing and non-maximum suppression (NMS) procedures are conducted to refine temporal boundaries from proposal segments to precisely localize boundaries of action instances. Instead of sliding window, [28] used a method called TAG [24] to generate proposals. SSAD [11] directly detect action instances in untrimmed videos by proposing a novel Single Shot Action Detector network to skip the proposal generation step. However, these methods ignore the problem of low classification accuracy which is caused by intra-action variations and inter-action differences.
In this work, we also adopt the proposal-classification framework to perform temporal action detection task. However, our work only focuses on action classification. For temporal action classification, we argue that it is effective to use identification and verification simultaneously for reducing intra-action variations while enlarging inter-action differences. Given a set of video proposal segments, identification network can output the category of each video proposal segments, which can be treated as a multi-class recognition task [20], [23], [26]. While verification network takes a pair of video proposal segments as input and determine whether they belong to the same action or not, which can be treated as a binaryclass classification task. The verification model ignores the relationship between the input video proposal pairs with other proposals, and the identification model also dosen't take similarity between video proposal pairs into consideration. To address this problem, our work proposes to combine identification and verification model into a siamese network that can not only predict action categories but also estimate the similarities of the input video proposal pairs.
Our contributions are two-fold: (1) To the best of our knowledge, our work is the first to incorporate the identification and verification model into temporal action detection framework. We propose a Identification-Verification siamese (IVS) network that predicts action categories and similarity scores at the same time, thus improving temporal action detection accuracy.
(2) Our proposed method significantly outperforms the state-of-art methods on the challenging action detection dataset: THUMOS 2014.
The paper is organized as follows. We first review some related works in Section II. In Section III, we describe our framework of Identification-Verification. Section IV provides the experimental results on the large-scale action detection dataset. At last, we conclude our works in Section V.

II. RELATED WORK
In this section, we will briefly review some related works, including action recognition, object detection, temporal action detection and person Re-identification. Action recognition. Action recognition has been made great progress in the past few years with the wide adoption of convolutional neural networks (CNNs). Action recognition aims to determine the category of an action instance, so the models of action recognition can be used in temporal action detection to extract features. The previous works [15], [21], [22] try to capture both appearance and motion information, both of which are essential to action recognition. However, most of these methods are based on the short trimmed videos, in which the only action is performed during the whole video time. Hense, there is no need to consider the temporal boundaries of each action instance. Typical datasets used in action recognition such as UCF101 [16], HMDB51 [10] and Sports-1M [9] include amount of trimmed videos. Object detection. Earlier methods of object detection are mostly based on hand-crafted low-level features. In recent years, inspired by the success of deep learning, object detection has achieved state-of-art performance. Most approaches are based on the pipeline of proposal-classification. R-CNN [4] uses SelectiveSearch [19] for proposal generation, CNN framework for feature extraction, SVM for proposal classification, and bounding box for regression. Fast R-CNN [3] uses RoI pooling layer to make features update all network layers. Faster R-CNN [12] uses RPN network to generate region proposals which reduces the time consumption. In our work, we also adopt the proposal-classification pipeline for temporal action detection task. Temporal action detection. Temporal action detection framework gets closely related to object detection framework. This task focuses on learning how to detect action instances in untrimmed videos where the boundaries and categories of action instances have been annotated. Most works focus on the framework of detection which are based on proposal and classification. For example, SCNN [14] uses sliding windows to generate proposals, and then classifies proposal segments of different categories. To get temporal boundaries of each action instance, SCNN uses localization network to boost temporal localization accuracy. SSN [28] tries to determine the completeness of each action instance via a structured temporal pyramid, and this can effectively improve the accuracy of classification and localization. However SSAD [11] dosen't use the proposal generation step, which directly detects action instances and predicts confidence scores of action instances in untrimmed videos. Person re-identification. Person re-identification (re-ID) has become increasingly popular due to its application and research significance [29]. It is usually regarded as an image retrieval problem and aimed at determining whether the person has been appeared in another camera. Many works [2], [17], [30] used verification models to person re-identification. Verification models usually take a pair of images as input and output a similarity score by calculating the distance between features. Our works are different from these. Firstly, we jointly use identification and verification in temporal action detection task to improve the accuracy of classification. Secondly, our model is pretrained on 3D ConvNets (C3D) [6], [18] which has shown great performance in video analysis [25]. Thirdly, our network is trained to minimize the joint of identification and verification losses.

III. OUR APPROACH
In this section, we will introduce our Identification-Verification siamese (IVS) approach in details. The framework of our approach is shown in Fig. 2. We apply a siamese network which jointly trains identification and verification models. We use 3D ConvNets as our basic network architecture and simultaneously consider identification loss for action category prediction and verification loss for similarity estimation.

A. Siamese network
Our siamese network combines identification model and verification model. Given an input pair of action proposals V 1 and V 2 , and their action labels, the two inputs share weights on 3D ConvNets, and we get two high-level features f 1 , f 2 after two fully connected layers. Then each of them is fed to our identification model for action prediction. Simultaneously, f 1 and f 2 are also fed to our verification model for similarity estimation. 3D ConvNets. Our siamese network adopts 3D ConvNets as the basic architecture for extracting high-level video action features. The 3D ConvNets consist of five layers. Each layer contains one or two 3D Convolutional Layer and one Pooling Layer. We use the notations C (number of filter) for the 3D Convolutional Layer and P (temporal kernel size, temporal stride) for the 3D Pooling Layer. The layout of these two types of layer in our architecture are shown in Fig. 3 is as   After that, we add a fully connected layer FC2 and the output is fed to a 2-way softmax which produces a distribution over the 2 class (same / different).

B. Loss Function
In our siamese network, we not only determine the action labels of the input pair-wise video proposal segments, but also estimate the similarity which we called the verification signal. The network is trained to minimize the three losses, two identification losses and one verification loss. Identification loss. For identification model, we predict the action label for each input video segment, through a sof tmax operation to predict the probability distribution p = [p 1 ,p 2 , · · · ,p n ] over the n action classes, where f is the high-level action feature, α I denotes the parameters of FC1, and * denotes the convolutional operation. Our identification model calculates the loss between the predicted probabilityp and the truth probability p, where the truth probability p = [p 1 , p 2 , · · · , p n ] is defined, Verification loss. To get high classification accuracy, we introduce the thought of verification to encourage the features extracted from videos of the same action class to be similar and enlarge the variations of features extracted from the different actions at the same time. We add a verification signal (similarity score) s = [s 1 , s 2 ] = [0, 1], where 1 means the two video proposal segments are from the same action category, while 0 denotes different action categories. Firstly, we compute the Euclidean distance f E of the two high-level features f 1 and f 2 through the E layer, Then the sof tmax operation is adopted to predict the similarity score p = [ p 1 , p 2 ] over 2 classes (different / same), where α V denotes the parameters of FC2.  Finally, following the computation of identification loss, our verification loss is defined between the verification signal s and the predicted similarity score p, Overall loss. In our siamese network, given an input pair of action proposals V 1 and V 2 , the overall loss is formed by combining identification losses and verification loss, where L I1 and L I2 are the identification loss corresponding to the action proposals V 1 and V 2 respectively. L V is the verification loss. λ ≥ 0 is a tradeoff parameter to balance the contribution of the two parts, and through empirical validation, we find that λ = 1 works well in practice.

IV. EXPERIMENTS
In this section, we evaluated our proposed method on standard benchmark dataset: THUMOS 2014. We first introduce the evaluation dataset and the experimental settings, then compare our proposed Identification-Verification siamese (IVS) network with some state-of-art approaches.

A. Dataset
The temporal action detection THUMOS 2014 dataset is challenging and widely used. The dataset contains two tasks: action recognition and temporal action detection. For action detection, the dataset contains 1010 videos for validation and 1574 videos for testing, both of which are untrimmed videos. For action recognition, the training set is UCF101 of short trimmed videos with no temporal annotations. Following the protocol in [14], [25], we used 200 validation set videos (including 3007 action instances) and 213 test set videos (including 3358 action instances) with temporal annotation to train and evaluate our Identification-Verification network.

B. Evaluation metrics
The THUMOS 2014 dataset has its own convention of reporting performance metrics, which evaluates Average Precision (AP) for all action categories and calculate mean Average Precision (mAP) for evaluation. We just follow the convention and calculate the mAP at different IoU thresholds. A prediction is correct only when it has the correct category prediction and its temporal IoU with ground truth instance is larger than the overlap threshold. The IoU thresholds used to compare the performances in our experiment are 0.1, 0.2, 0.3, 0.4, 0.5.

C. Implementation details
We implemented our network based on Caffe [7]. For temporal action detection, we used 200 annotated untrimmed video in THUMOS 2014 validation set as our training set. Firstly, we directly used ActivityNet dataset [1] to finetune the R-C3D model [25] to generate the training and testing proposals. Secondly, we used C3D model [6] parameters to initialize our Identification-Verification network, and trained it with the generated training proposals. We used SGD to learn CNN parameters in our framework, with batch size 5 and momentum 0.9. The initial learning rates were set to 0.001 for our siamese network.

D. Experimental results
We conducted extensive experiments to demonstrate the effectiveness of our proposed Identification-Verification siamese network for temporal action detection, including the following aspects.
Comparison with state-of-art methods. We compared our IVS method with six state-of-art methods, including PSDF [27], SCNN [14], CDC [13], SSAD [11], R-C3D [25] and SSN [28]. The results are shown in  overlap IoU in detection task. We also show two prediction results of our proposed IVS method for the THUMOS 2014 test set in Fig. 4. It shows the excellent performance of our IVS method compared to the ground truth. The effect of proposals. For Identification-Verification siamese network, we adopted the SCNN [14] and R-C3D [25] as our proposals generation. For SCNN, we directly put the results of SCNN as proposals to our IVS network, which is termed as SCNN + IVS. For R-C3D, we used ActivityNet [1] dataset to finetune the model which is different from the original paper that was finetuned with UCF101 [16] dataset. We term the method of finetuned with ActivityNet dataset as R-C3D * . Then the results of R-C3D * , as proposals, were directly fed to ours IVS network, which is termed as R-C3D * + IVS. As shown in  of SCNN, R-C3D * and our method with overlap IoU threshold 0.5 are shown in Fig. 5. Parameter λ. λ is a tradeoff parameter to balance the contribution of identification and verification. To explore the effects of identification and verification, we conducted experiments with different values λ = {0, 0.5, 1, 2}. As shown in TABLE III, comparing λ = 0 with λ = 0.5, we can see that verification loss bring improvement for temporal action detection. We can get the best mAP results when λ = 1. The results show that our joint Identification-Verification achieves good performance for action detection. Verification loss vs. Contrastive loss [5]. We adopted the verification loss to measure the similarity of the input video proposal pairs. To study the effect of our verification loss, we conducted experiment by replacing it with the contrastive loss which is a commonly used pair-similarity measure in siamese network. Since the entire model was trained end-to-end with jointly optimize identification loss and verification loss, we only replaced the verification loss by contrastive loss in our overall loss function. As shown in TABLE IV, we find that using contrastive loss leads to lower mAP results than our verification loss, which demonstrates the effectiveness of our verification loss.

V. CONCLUSION
In this paper, we propose a Identification-Verification siamese network to get high classification accuracy by reducing intra-action differences and enlarging inter-action variances. The proposed model simultaneously considers identification for action classification and verification for similarity estimation. Experimental results on THUMOS 2014 dataset demonstrate the effectiveness of our proposed siamese network for temporal action detection. Our joint Identification-Verification model also can be easily applied on different proposal generation networks, and achieve significantly improvements compared to other methods in the temporal action detection task.