{'fraction_non_alphanumeric': 0.03607425545242298, 'fraction_numerical': 0.014150674790729458, 'mean_word_length': 5.209865629420085, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Equivalent Classification Mapping for Weakly Supervised Temporal Action Localization

Weakly supervised temporal action localization is a newly emerging yet widely studied topic in recent years. The existing methods can be categorized into two localization-by-classification pipelines, i.e., the pre-classification pipeline and the post-classification pipeline. The pre-classification pipeline first performs classification on each video snippet and then aggregate the snippet-level classification scores to obtain the video-level classification score, while the post-classification pipeline aggregates the snippet-level features first and then predicts the video-level classification score based on the aggregated feature. Although the classifiers in these two pipelines are used in different ways, the role they play is exactly the same---to classify the given features to identify the corresponding action categories. To this end, an ideal classifier can make both pipelines work. This inspires us to simultaneously learn these two pipelines in a unified framework to obtain an effective classifier. Specifically, in the proposed learning framework, we implement two parallel network streams to model the two localization-by-classification pipelines simultaneously and make the two network streams share the same classifier, thus achieving the novel Equivalent Classification Mapping (ECM) mechanism. Considering that an ideal classifier would make the classification results of the two network streams be identical and make the frame-level classification scores obtained from the pre-classification pipeline and the feature aggregation weights in the post-classification pipeline be consistent, we further introduce an equivalent classification loss and an equivalent weight transition module to endow the proposed learning framework with such properties. Comprehensive experiments are carried on three benchmarks and the proposed ECM achieves superior performance over other state-of-the-art methods.


Introduction
Temporal action localization aims to localize action instances from the given untrimmed video by determining the start temporal points, end temporal points, and the corresponding action categories. In the studied weakly supervised setting, the action localizers are learned directly from video-level labels, without requiring the fine segment-level annotations. Thus, weakly supervised temporal action localization can alleviate the burdensome and expensive human annotation and further realize the potential learning process on web-scale unlabeled videos.
In order to eliminate the ambiguity brought by the weak supervision, most previous works adopt the localization-by-classification pipelines to estimate the video-level classification score, which can be categorized into two main pipelines, i.e., the pre-classification pipeline (see Fig. 1 (a)) and the post-classification pipeline (see Fig. 1 (b)). Among the existing methods, UntrimmedNets [1] makes the pioneering exploration for the pre-classification pipeline. It first performs classification at each temporal point to obtain the class activation sequence, which is then aggregated to predict the video-level classification score. Based on this work, CMCS [2] proposes a diversity loss to model the completeness of actions. Meanwhile 3C-Net [3] introduces action count cues to distinguish adjacent action sequences. In the aforementioned pre-classification pipelines, the classification mapping functions are learned based on each temporal point and the corresponding local neighbors. Such a mechanism would be beneficial to capturing local contrast information but tends to be less effective in perceiving long-term connections within each given video. There are also works, such as [4,5,6], which adopt the post-classification pipelines. These works first aggregate features from all temporal points to form the video-level feature representation, then they predict the video-level classification scores by classifying the aggregated feature representation. The advantage of these methods is that the aggregated feature can represent long-term relationships. However, when performing classification at each temporal point in the evaluation phase, the class activation sequences generated by these methods show insufficient discriminability to localize the actions that belong to different categories.
From the above discussion, we can observe that both the pre-classification pipeline and the postclassification pipeline aim at learning the effective classification mapping functions to predict the classification scores from the input features. The difference is that the pre-classification pipeline uses the classifier to perform classification on the features of each temporal point, while the postclassification pipeline uses the classifier to perform classification on the aggregated features of the whole video sequence. Note that the post-classification stream uses wieghted sum to aggregate snippet features. By such a linear operation, the aggregated feature remains in the original feature space. When an ideal classification mapping function is given, both pipelines would obtain accurate classification results. This inspires us to simultaneously learn these two pipelines under a newly proposed Equivalent Classification Mapping (ECM) mechanism to obtain the desired classifier and then use the classifier to identify the presence of the actions on each temporal point.
To implement such a new learning mechanism, the proposed method (see Fig. 1 (c)) has the following three main characteristics: • Compiling the basic equivalent classification mapping spirit, we implement two parallel network streams to model the pre-classification pipeline and the post-classification pipeline, respectively, and make the two network streams share the same classifier.
• Besides the basic classification loss of each network stream, we introduce a new equivalent classification loss, which contains a classification-to-classification consistency term and an aggregation-to-classification consistency term to penalize the inconsistency of the classification scores from two network streams and the inconsistency of the classification score and the aggregation weights, respectively.
• Unlike the traditional post-classification pipelines [4,5,6] which infer the feature aggregation weights in self-attention-like manners, the post-classification stream in the proposed learning framework is equipped with a weight-transition module that transits the frame-level classification scores obtained from the pre-classification stream to generate the feature aggregation weights for the post-classification pipeline.
In summary, ECM contributes to the revealing of the equivalence mechanism, together with designing equivalence-based components. The equivalence mechanism indicates that both the classification for snippet features in the pre-classification pipeline and the classification for aggregated features in the post-classification pipeline pursue the same ideal classifier. Equivalence mechanism is overlooked by previous methods but plays an essential role, as verified in Section 4. Although conceptually simple, an intuitive implementation via sharing classifier only achieves 21.6. Thus, adequately mining the equivalence is critical for good results. Specifically, we design equivalence-based components, i.e., weight-transaction module and two consistency losses (L c2c and L a2c ). Empirically, ECM starts from two simple and widely used baselines (with performance 19.6 and 17.1), employs equivalence-based components, achieves 29.1 and builds new SOAT without bells and whistles. Moreover, ECM can serve as a clean and solid baseline for future studies on weakly supervised action localization.

Related work
Supervised action localization Early explorations adopt a detection-by-classification pipeline. S-CNN [7] classifies sliding-window proposals to localize action instances. CDC [8] and Lin et al. [9,10] predict the actionness score for each temporal point. Meanwhile advances in object detection bring inspiration to action localization. Some methods follow Faster R-CNN [11] and perform two-stage action localization, e.g., R-C3D [12], CBR [13], TURN TAP [14] and TAL [15]. Similarly, some methods follow one-stage object detection methods [16,17] and perform action localization, e.g., SSAD [18] and GTAN [19]. In addition, the recurrent memory module is used to capture long-term dependencies, such as SS-TAD [20], SST [21] and the work from Yuan et al. [22]. Apart from the above explorations, some noticeable works also include modeling temporal structure [23,24], modeling context [15,25,26], modeling relationships among action proposals [27], localizing action instances from a part of videos [28], etc. In summary, the supervised methods can explicitly learn from segment-level annotations and achieve accurate localization. However, they are limited by the expensive annotations as well, which can be alleviated by the studied weakly supervised method.
Weakly supervised action localization UntrimmedNets [1] proposes a pre-classification pipeline with a soft aggregation mechanism. Later, W-TALC [29] considers the co-activity similarity to model inter-video similarities and differences. Furthermore, CMCS [2] proposes to jointly learn multiple classification networks and require them to generate diverse responses. Focused on the quality of the class activation sequence, there are some other promising works. CleanNet [30] learns regression to adjust the action segments. TSM [31] models the action structure via a multi-phase process. 3C-Net [3] introduces action count cues. BaSNet [32] tries to suppress the background response. In general, the foundation of pre-classification methods is the class activation sequence, generated by the classification mapping function.
Apart from the above methods, there is another pipeline, namely the post-classification pipeline. Inspired by the success of CAM [33], STPN [4] proposes to first aggregate video features then perform classification. A similar intuition can be found in ST-GradCAM [34]. Later, Nguyen et al. [5] extend STPN [4] by introducing background modeling. Meanwhile, MAAN [6] develops the post-classification pipeline with a marginalized average aggregation module. In general, the post-classification pipeline is aware of the complete video features, but its response to each category may be not discriminative enough.
Video recognition and spatio-temporal action detection Video recognition researches can provide high-level feature representations and bring inspiration for the action localization task. The widely used video recognition models include C3D [35], two-stream network [36], I3D [37], etc.
Recently proposed video recognition models also include TrajectoryNet [38], SlowFast [39] and the work of Fan et al. [40]. Spatio-temporal action detection aims to detect action with the bounding box at each frame. Recent progress includes learning to track [41], VideoCapsuleNet [42], SlowFast networks [39], etc. In addition, some works consider more practical scenarios. For example, Choi et al. [43] propose to mitigate the sense bias when recognizing video. Chéron et al. [44] perform spatio-temporal action detection with varying types of weak supervision.

Method
Given a dataset containing C action categories and each video in the dataset has the classification label y = [y 1 , y 2 , ..., y C ], where y c ∈ {0, 1}, c ∈ [1, C] indicates whether there is an action instance belonging to category c. Each video is firstly divided into T snippets and then a feature extractor is used to extract the feature representation f t ∈ R D , t ∈ [1, T ] from each video snippet. Under the weakly supervised learning scenario, the proposed model learns the classifier Φ = [Φ 1 , Φ 2 , ..., Φ C ] from the video-level classification label y to discover the temporal locations of the desired action instances from each video. When the training process is complete, we forward each input video through the learned classifier and obtain the class activation sequences. Then, following [32], we adopt a post-process to reject the categories whose classification scores are smaller than a threshold τ . Next, the remaining class activation sequences are min-max normalized along the temporal dimension, which is followed by the temporal actionness grouping operation [23] to localize action instances. Finally, redundant action instances are filtered out via Non-Maximum Suppression. Output layer (class activation sequence) Figure 2: Architecture of the classifier used in our learning framework.
As shown in Fig. 1 (c), the body network consists of two streams: the pre-classification stream and the postclassification stream. They share the same classification mapping function but make predictions in different manners. To endow the classifier Φ with the helpful temporal reception field, we use three temporal convolutional layers to build it (see Fig. 2), where the kernel sizes of the first two layers are 3 while it of the last layer is 1. Here, the classifier is used to predict the classification scores for each temporal point, i.e., each video snippet. Specifically, in the pre-classification stream, the extracted video features are directly passed through the classifier. Then, obtain classification scores along the temporal points and generate the class activation sequence for the whole video. After that, we adopt the top-k mean strategy to aggregate the classification scores for each category c and obtain the video-level classification score s e = [s e 1 , s e 2 , ..., s e C ].
... ...  In the post-classification stream, the inputs consist of both the video features F and the categoryspecific aggregation weights. Here, we explore both the action and background weights W a ∈ R C×T , W b ∈ R C×T to generate the category-specific action features . Specifically, the category-specific feature for either the action or the background is obtained by operating the weighted sum over the video features F: where w a c,t and w b c,t are the elements in W a and W b , respectively. Then, each sub-classifier Φ c is applied on the corresponding category-specific action feature f a c to obtain the final classification score denotes the network forward operation.
Class Activation Sequence Figure 4: Equivalent weight-transition module. We predict the foreground and background aggregation weights from the class activation sequence. c indicates temporal convolution, indicates sigmoid activation.
Since the classifier uses the same structure as the one used in the pre-classification stream, we evenly divide the raw video features as well as the corresponding aggregation weights into three folds by sampling every three snippets, and implement the feature aggregation process within each fold (see Fig. 3). Comparing to the strategy that directly extracting the whole video features in one fold, the aggregated video-level features extracted by our strategy can better fit the input structure of the classifier, thus facilitating a more effective equivalent learning scheme.

Equivalent weight-transition module
The post-classification stream requires category-specific aggregation weights W a and W b to aggregate features. Considering that the class activation sequence obtained from the pre-classification stream can reveal the probability that a temporal point belongs to an action instance, we propose to obtain the aggregation weights by transiting the class activation sequence, as shown in Fig. 4. We use a convolutional layer with kernel size 1 to predict the importance values for each temporal point. After that, we apply the sigmoid activation and obtain the action aggregation weights W a , while the background aggregation weights can be obtained via W b = 1 − W a . Compared with the conventional methods [4,5,6,32], the proposed equivalent weight-transition module shows two characteristics. Firstly, existing works adopt an extra network to predict the aggregation weights, while the proposed module can directly obtain aggregation weights from the pre-classification stream. Besides, existing methods predict category-agnostic weights and use them to obtain one aggregated feature to represent the input video. In contrast, we learn category-specific weights and use them to obtain the aggregated features for each different action categories. Such category-specific features would bring richer representation and assist ECM to clearly distinguish different action categories, as verified in Section 4.3.

Equivalent classification loss
The equivalent classification loss L equ consists of a classification-to-classification consistency term L c2c and an aggregation-to-classification consistency term L a2c , i.e., L equ = L c2c + αL a2c , where α is the coefficient. Given an input video, the pre-classification stream predicts the classification score s e , while the post-classification stream predicts the classification score s o with action aggregation weights W a . Based on the intuition that the classification scores predicted by the two network streams should be identical for the same input video, we introduce the classification-to-classification consistency term L c2c to the learning loss function: The classification-to-classification consistency term L c2c plays a role in making one stream perceive the predictions from the other stream. Despite the simplicity, it obviously facilitates the learning of the classification mapping function, as shown in Section 4.3.
Besides classification-to-classification consistency, we further explore a new training strategy, namely aggregation-to-classification consistency training. The motivation is that the video-level classification score should be consistent with the aggregation weights. Specifically, to meet the conventional classification loss, the attention weights may only highlight the most discriminative snippets within an action. In contrast, the proposed aggregation-to-classification loss L a2c not only requires the action attention weights W a to highlight snippets within an action, but also requires the background attentions W b = 1 − W a to exclude all action snippets. This is not explored by classification loss.
Then, for an input video that contains k action categories, we select classification scores for present action categories from s o and s o according to classification label y. After that, , which contains k positive labels and k negative labels. Finally, we perform sigmoid activation over prediction s and define the aggregation-to-classification consistency term L a2c as: The proposed aggregation-to-classification consistency training strategy can guide ECM to better distinguish action segments and backgrounds. Besides, the existing background modeling strategy [5] and background suppression strategy [32] can be regarded as the specific cases of this strategy, as they only consider the background category when adjusting attention weights.
Finally, we also calculate the basic classification loss L cls = L cls,e + L cls,o , where L cls,e and L cls,o are the classification loss from the pre-classification stream and the post-classification stream, respectively. Formally, this problem can be formulated as a multi-label classification problem. We perform L1−normalization on the original classification label y and calculate the cross-entropy loss L cls,e and L cls,o . In summary, the overall loss function required to be optimized is L = L cls + βL equ , where β is the balance coefficient. Feature extraction To extract features, each video is evenly divided into T snippets and we uniformly sample 16 frames from each snippet, similar as previous works [4,5]. The optical flow is calculated via the TV-L1 [47] algorithm. I3D model [37] pre-trained on the Kinetics-400 dataset is used to extract video features, without finetuning on THUMOS14 [45] or ActivityNet v1.3 [46]. We extract both the appearance features and the motion features, which are concatenated together to represent the video sequence. The concatenated feature dimension is 2048. Finally, the snippet number for THUMOS14, ActivityNet v1.2 and ActivityNet v1.3 are 750, 100 and 100, respectively.
Training and evaluation details ECM is implemented using PyTorch [48]. Adam [49] solver is used to optimize the network. For all experiments, we set batch size as 16, the learning rate as 2 × 10 −4 and the weight decay as 5 × 10 −4 . We train ECM for 150 epochs, 40 epochs and 40 epochs for THUMOS14, ActivityNet v1.2 and ActivityNet v1.3, respectively. The parameters are empirically determined via gird search. Specifically, we set k = 1 8 for the top-k mean aggregation. The balance coefficients are α = 100 and β = 0.05. In evaluation, the threshold to reject absent categories is τ = 0.25. The implementation code is in the supplementary materials.

Ablation studies
We perform ablation studies on the THUMOS14 dataset and the results are reported in Tab. 4. Under threshold 0.5, the pre-classification stream and post-classification stream achieve 19.6 and 17.1, respectively. Moreover, we share the classifier between the pre-classification stream and the post-classification stream, transit the aggregation weights from the class activation sequence in the pre-classification stream, and the performance reaches 22.8. Based on this, the proposed aggregationto-classification consistency training strategy can lift the performance to 24.2, demonstrating it is rational to constrain the consistency between the aggregation weights and the action presence. Finally, the complete ECM achieves 29.1, under the cooperation of sharing the classifier, the equivalent classification loss and the weight-transition module. To verify the effectiveness of classifier sharing, we remove classifier sharing from ECM and keep all other components unchanged. Under this setting, the model obtains 23.4, which is much inferior to 29.1 of full ECM model. This demonstrates classifier sharing (i.e., the equivalence mechanism) is the foundation of ECM, which makes the proposed equivalence-based components work effectively.
In addition, we study different strategies to integrate the pre-classification stream and the postclassification stream together. A simple strategy is to directly fuse the localization results from these two streams. We elaborately try different fusing methods: (1) First localizing action instances from two class activation sequences (CASs), then merging the localization results. (2) First fusing two CASs via weighted sum (we try different weights), then localizing action instances from the fused CAS 1 . Actually, the above experiments are ensemble methods, where we train two individual classifiers and ensemble the localization results. The best performance is 21.4. Based on the equivalence mechanism, we share the classifier between the pre-classification stream and the postclassification stream, the performance is 21.6. Moreover, if the attention weight is transited from the pre-classification stream, the performance would be 22.8. Above experiments demonstrate the rationality of sharing classifier and weight transition.
Beyond that, we verify the effectiveness of class-specific feature representation. We follow existing methods [4,5,6] and learn class-agnostic aggregation weights for the post-classification stream, such a variant gets 27.6. In comparison, the 1.5 performance improvements of ECM is mainly caused by rich feature representation. To make a thorough analysis about ECM, we use the diagnosing tools [51] analyze the localization results. The results are shown in Tab. We qualitatively visualize the class activation sequences in Fig. 5. When only using the preclassification stream or the post-classification stream, the generated class activation sequences cannot distinguish two adjacent action instances (see the left case), or shows high responses for a part of background points (see the right case). This will lead to error action localizations. In contrast, the proposed ECM method precisely shows high responses for the action segments and confidently suppresses the response of backgrounds, leading to accurate localization results.

Conclusion
In this paper, we introduce the equivalent classification mapping mechanism to weakly supervised temporal action localization task, which starts from both the pre-classification stream and the postclassification stream to simultaneously learn one classification mapping function. Assisted with the classification equivalent loss and the weight-transition module, ECM achieves state-of-the-art performance on three benchmarks. Considering ECM is simple to implement and achieves good performance, subsequent researches are potential to observe further improvements when applying the ECM mechanism to the pre-classification methods [2,3,32] and post-classification methods [4,5,6]. Besides, we plan to verify the performance of ECM on instructional videos [52] or large scale action localization datasets [53,54]. Furthermore, it is a promising direction that extending ECM to other similar research areas, for example, weakly supervised object localization [33] and detection [55].

Broader Impact
We present a novel method to discover actions from untrimmed videos. As three benchmark datasets show large varieties, the proposed method does not leverage biases in the data, which is further verified by the performance improvements on all three datasets. Because it only requires video-level labels to train, the proposed method is potential to be applied to internet video platforms, e.g., YouTube, to tackle the large-scale, fast-growing untrimmed videos. Besides, the proposed method can help to discover informative video segments, and filter out most of the meaningless backgrounds. For example, the proposed method can help to build a smart video surveillance system, which can be used for monitoring the patients, monitoring the operation of the production line, etc.
On the contrary, it should also be noticed that employees working on video monitoring may be put at disadvantage from this research. The automatic video processing and understanding system would reduce existing jobs, while it may create new jobs that can alleviate this scenario. Because the proposed method directly learns from the video-level category label, the failure cases mainly come from incomplete or missing localizations. Currently, the localization quality is not comparable to fully supervised methods, but it can be gradually improved when more effective algorithms are proposed, and large-scale videos and powerful computation ability is available to train the model.

Supplymentary Material
In this supplementary material, Section "Implementation of the post-classification stream" provides details about the implementation of the post-classification stream. Section "Details of the ablation studies" elaborates on the setups and results for the ablation studies. Next, we report frame accuracy in Section "Frame accuracy", while the detailed comparisons on three benchmarks are shown in Section "Detailed comparisons". Finally, more qualitative results are shown in Section "Qualitative results".
Implementation of the post-classification stream The post-classification stream shares the same classifier as the pre-classification stream, which consists of three convolutional layers. We show the implementation details of each operation as well as the size for the corresponding output tensor in Tab. 7. Specifically, each categoryspecific feature contains three feature vectors. The first two temporal convolutional layers adopt the ordinary temporal convolution to learn feature representation from the category-specific features. Then, the last layer adopts the group temporal convolution to predict the categoryspecific classification score. Finally, we perform average pooling along the temporal dimension to generate the video-level classification score.

Details of the ablation studies
Performance of the post-classification stream To validate the performance of the post-classification stream, we need an extra network to predict the category-specific aggregation weights. Based on previous works [4,32], we try different network architectures to determine a proper one. In Tab. 8, the network architecture is represented in the form of "(input)-middle layer-[output]". For example, "(2048)-512- [20]" indicates that the input is a 2048-D feature, the middle layer is a convolutional layer with 512 filters, and the output is a 20-D classification score.
We find that a proper architecture to infer the aggregation weights might be (2048)-2048-2048- [20], where there are two middle layers with 2048 filters. This setting achieves mAP=17.1 under threshold 0.5 and average mAP=18.6 under threshold [0.1:0.7]. When we use more parameters, e.g., using convolutional layers with kernel size 3, the performance drops. Thus, we report 17.1 for the performance of the post-classification stream.

Fuse localization results from two separate streams
In ablation studies, when the pre-classification stream and the post-classification stream do not share the classifier (indicated as " " in Tab. 4 of the paper), the evaluation process would obtain two different class activation sequences. Then, we study two ways to fuse the class activation sequences, where the first one is "concatenating localization results" (Concat) while the second one is "weighted sum class activation sequence" (WeightedSum).
Given a video, we feed it through both the pre-classification stream and the post-classification stream, obtain two class activation sequences. As for Concat, we individually obtain two groups of action localization results via thresholding on two class activation sequences. Then we concatenate the two results together and apply Non-Maximum Suppression to generate the final localization results. As for WeightedSum, this strategy is proposed by previous works [4,5]. It fuses two class activation sequences via weighted sum and then localizes actions based on the obtained class action sequence.  The experimental results are reported in Tab. 9. It can be found that all fusion methods exceed the individual pre-classification stream or post-classification stream. WeightedSum achieves 20.8 and Concat achieves 21.4. We report the performance 21.4 in our paper.  All of the pre-classification method, the post-classification method and the proposed ECM method directly learn classifiers to predict the classification score for each temporal point. Following the frame accuracy metric adopted by the action segment researches [56,57], we measure the frame accuracy of the generated class activation sequences, so as to validate the quality of the point-level classification. Similar to previous works [56,57], we do not consider the background category in order to prevent the case that a method predicts all frames as background but still achieves high performance.
Given the obtained class activation sequences, we first predict the video-level classification scores and reject categories whose classification scores are lower than threshold τ = 0.25. Then, for the remaining responses, we perform max pooling at the temporal dimension and obtain the point-wise classification predictions. Finally, the frame-wise classification accuracy is reported in Tab. 10. It can be found that the pre-classification method performs somewhat better than the post-classification method, with a margin of 3.7%. The proposed ECM brings obvious improvement, with a margin of 11.6%. The results demonstrate that the proposed equivalent classification mapping mechanism can learn a high-quality classifier and can discover more accurate action snippets from backgrounds.

Feature representations
As the feature representation plays an important role in most temporal action localization methods, we discuss the features used by each work for fair comparisons in Tab. 11, Tab. 12 and Tab. 13. "I3D" indicates the Inflated 3D features [37]. "TS" indicates the two-stream [36] features. Xiong et al. [58] design a two-stream network architecture for the video classification task, which is used to extract two-stream features by latter works. Specially, the spatial network uses the ResNet architecture [59] and the temporal network uses the BN-Inception architecture [60]. Both the I3D model and the two-stream model are trained on the Kinetics-400 dataset. "P3D" indicates pseudo-3D [61] features. "C3D" indicates 3D convolutional [62] features. "UNT" indicates UntrimmedNet [1] features, with the model architecture illustrated in TSN [63]. "Res" indicates that LTSR [64] uses ResNet [59] to extract the feature for each video frame. Apart from extracting features, there are some works directly learn from video frames, e.g, R-C3D [12], CDC [8] and S-CNN [7]. As most of the weakly supervised methods use the I3D feature, we implement our experiments with the I3D feature for fair comparisons.

Comparison experiments on THUMOS14
In Tab. 11, we elaborately compare ECM with recent temporal action localization methods, including both supervised methods and weakly-supervised methods. As the official evaluation metric of the THUMOS14 dataset is the mAP under threshold 0.5, we sort all the methods according to this. In Tab. 11, "Weak †" indicates the methods rely on extra annotations apart from video classification labels. Both STAR [34] and 3C-Net [3] rely on the action count information to localize action instances.
It can be found that ECM performs superior among the weakly supervised works under threshold 0.5.
It is promising to notice that ECM can exceed some supervised works published in the early years, e.g., ICCV 2017 and CVPR 2017.
Among the weakly supervised works, we notice STAR [34] and ASSG [65] achieve good performance under low thresholds. STAR [34] integrates the temporal gradient-weighted CAM mechanism and the attention mechanism to localize action instances. Apart from the commonly used video-level classification labels, it also relies on the action frequency annotation to train the model. With the metric of average mAP under threshold [0.1:0.5], the reported performance is 44.0 without the use of action frequency. In contrast, under the same metric, ECM achieves 46.3. ASSG [65] first initializes seed regions from the class activation sequences, which are generated by a well-performed temporal action localization method. Then, it progressively extends the seed regions and learns the classification model. However, the performance of ASSG [65] heavily relies on the quality of initialization. Consequently, when initializing it by STPN [4], it only achieves 20.9 under threshold 0.5. Furthermore, the heavy dependence of initialization would limit its application to practical scenarios.

Qualitative results
We show qualitative results of the class activation sequences and the corresponding ground truth for THUMOS14, ActivityNet v1.2 and ActivityNet v1.3 in Fig. 6, Fig. 7 and Fig. 8, respectively.