{'fraction_non_alphanumeric': 0.03454384410983171, 'fraction_numerical': 0.013604051690854171, 'mean_word_length': 4.809737432378942, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 2, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Progressive Localization Networks for Language-based Moment Localization

This paper targets the task of language-based video moment localization. The language-based setting of this task allows for an open set of target activities, resulting in a large variation of the temporal lengths of video moments. Most existing methods prefer to first sample sufficient candidate moments with various temporal lengths, and then match them with the given query to determine the target moment. However, candidate moments generated with a fixed temporal granularity may be suboptimal to handle the large variation in moment lengths. To this end, we propose a novel multi-stage Progressive Localization Network (PLN) which progressively localizes the target moment in a coarse-to-fine manner. Specifically, each stage of PLN has a localization branch, and focuses on candidate moments that are generated with a specific temporal granularity. The temporal granularities of candidate moments are different across the stages. Moreover, we devise a conditional feature manipulation module and an upsampling connection to bridge the multiple localization branches. In this fashion, the later stages are able to absorb the previously learned information, thus facilitating the more fine-grained localization. Extensive experiments on three public datasets demonstrate the effectiveness of our proposed PLN for language-based moment localization, especially for localizing short moments in long videos.


INTRODUCTION
Localizing actions/activities in videos is an increasingly important but challenging research task for video understanding, which usually can be grouped into two sub-fields: i.e., temporal action localization [7,15,51] and language-based moment localization [13,33,74]. For temporal action localization, it aims to temporally localize segments whose action labels are within a pre-defined list of actions. Due to the pre-defined scheme and the limited action labels, it fails to effectively handle unseen activities in the real world. Therefore, a new task, language-based moment localization, is introduced by Hendricks et al. [18] and Gao et al. [12]. As shown in Figure 1(a), given an untrimmed video and a natural language sentence query, the language-based moment localization task aims to temporally localize a specific video segment/moment which semantically matches the given query. This task is more flexible than temporal action localization, and recently attracts widespread interests in multiple research communities, such as computer vision [39,78] and multimedia [3,83]. In this paper, we target the task of language-based moment localization due to its potential applications in intelligent video surveillance, robotics, etc.
In the language-based moment localization task, the language-based queries are usually free-form and can describe diverse video content, which allows for an open set of video activities, e.g., a short activity of person closes the door, or a relatively more complex activity of the person walks into the house puts down the food. Such an open setting leads to a large variation in the temporal lengths of the target moments, as shown in Fig. 1(b). Considering such a large variation, how to generate high-quality candidate moments (a.k.a. proposals) is one of the key questions to tackle this challenging task.
Most existing works prefer to first sample sufficient candidate moments with various temporal lengths, and then match them with the given query to determine the target moment by their crossmodal similarity [12,16,18,20,30,31,34,73,82]. The popular solution of generating candidate moments mainly consists of two steps. The first step is to split the input video into an ordered sequence of video clips with a fixed interval. The second step is to enumerate [18,82] or sample [18] contiguous set of clips generated from the first step for obtaining sufficient candidate moments which have various temporal lengths. In this popular strategy, the setting of the split interval for sampling video clips is quite important to the localization performance. A large split interval usually results in a sparse set of video clips where each clip has a relatively large temporal granularity. Such video clips would construct a set of coarse-grained candidate moments that may not be able to effectively handle the target moments of short temporal lengths. In contrast, a small split interval usually results in a dense set of clips of small temporal granularity, which will accordingly generate a set of redundant candidate moments. Such redundant candidates would facilitate the localization of short-length target moments but also inevitably hinder the optimization of the learning objective.
To alleviate above dilemma, in this paper we explore the video clips of multiple diverse temporal granularities for better handling the moments with large variations in temporal lengths. We propose to progressively localize the target moment in a coarse-to-fine manner, which is inspired by humans' decision process. When humans perform language-based moment localization, they may first roughly search the target moment and then refine it. For example, given a query of a person opens a cabinet as illustrated in Fig. 1(a), one may first search the segments where a person appears then further localize the action of opening a cabinet. To this end, we propose a Progressive Localization Network (PLN), as shown in Fig. 2, which progressively localizes the target moment via multiple stages and each stage has a localization branch with a specific temporal granularity.
Different from existing works [18,31,82] that utilize a fixed interval to produce video clips, we propose to use different intervals across multiple stages resulting in clips of diverse temporal granularities. Specifically, we use a large interval in the early stage and a small interval in the late stage. By this way, the localization branch in the early stage focuses on the moments generated with the clips of large temporal granularity, while the later localization branch focuses more on the moments generated with the clips of a small granularity. Such a coarse-to-fine manner contributes to handling the moments with large variations in temporal lengths. Moreover, all branches of the model are not independent as we introduce a conditional feature manipulation module and an upsampling connection to bridge them. Although the model has multiple stages, it can be trained jointly in an end-to-end style. Our main contributions are roughly summarized as follows: • We propose a Progressive Localization Network (PLN) which progressively localizes the target moment in a coarse-to-fine manner. Through the progressive localization via multiple stages with diverse temporal granularities, PLN shows superiority in localizing short moments in long videos. To the best of our knowledge, this paper is the first work to localize the target moment progressively. • We propose a conditional feature manipulation that adaptively manipulates the video clip features by referring to the learned knowledge of the previous stage, making the clip features more suitable for the more fine-grained localization in the later stage. Besides, we also introduce an upsampling connection to inject the learned coarse relevance of the previous stage into the later stage. All these components are beneficial to multi-stage localization. • We conduct extensive experiments on three public datasets: TACoS [45], ActivityNet Captions [22] and Charades-STA [12]. Our PLN achieves a new state-of-the-art performance on TACoS, and compares favorably to state-of-the-art methods on ActivityNet Captions and Charades-STA. Besides, PLN is better at localizing short moments in long videos than recent one-stage models.

Temporal Action Localization
Given an untrimmed video, the task of temporal action localization is asked to localize segments where a pre-defined list of actions happen and predict the action label of localized segments.
Owning to the success of deep learning in video understanding area, significant progress has been made in this task recently [1,4,7,25,28,32,50,51,69,71,77]. Existing efforts on temporal action localization could be roughly categorized into two groups: multi-stage approaches [4,77] and one-stage approaches [1,24]. For multi-stage approaches, they split the localization task into multiple steps mainly involving proposal generation, classifying whether actions of interest happen in proposals, and proposal boundary refinement. For instance, Shou et al. [51] propose a multistage CNN model, which generates candidate segments, recognizes actions, and localizes temporal boundaries in three stages, respectively. By contrast, one-stage approaches integrate multiple steps together, showing much more efficient inference. For instance, Liu et al. [35] directly predict starting, ending and center of action proposal. It is worth noting that our multi-stage PLN is essentially different from the above multi-stage models for temporal action localization. In [4,77], each stage plays a specific functional role and all stages have to be cascaded to generate the final localization results. By contrast, our model is able to perform localization in each stage, and the different stages focus on the distinct temporal granularities of input videos.

Language-based Moment Localization
Language-based moment localization task is more challenging than temporal action localization. The task not only requires understanding the video content [37,72] but also performs cross-modal reasoning [41,67] to predict the target moment. As the pioneers for this task, Hendricks et al. [18] and Gao et al. [12] first generate candidate moments by sliding windows, and then determine the target moment according to the given natural language query. In particular, Hendricks et al. [18] propose to embed candidate moments with its context and queries into a shared space, where the candidate moment showing the highest relevance score with the given query is selected as the target moment. Gao et al. [12] first fuse the candidate moment and query, and further employ a temporal regression network to produce their alignment scores and location offsets. After that, this task has received increasing attention [6,19,30,31,83,84]. Following the research line of [12], a number of works [9,16] try to exploit the semantic concepts to improve the performance. Among them, instead of utilizing sliding windows to generate candidate moments, Chen et al. [9] employ extracted semantic concepts from the input video to generate candidate moments with high probabilities. Similarly, Xu et al. [66] first weight the video frames by their similarity to the given query, and select the frames that are more relevant to the query to build candidate moments. Considering the above methods that ignore the temporal dependency of candidate moments, Zhang et al. [82] construct the whole candidate moments into a 2D temporal feature map and then use a convolutional network to model the temporal dependencies of adjacent moments. As such temporal dependency of candidate moments is helpful, we also inherit it into our proposed multi-stage model. As a follow-up work of [82], Liu et al. [29] consider more candidate moments, and model all pairs of start and end indices within the video simultaneously with a biaffine mechanism. In [19], Hu et al. introduce hashing techniques into the language-based moment localization task, which improves its efficiency and scalability in practice.
We also observe that some works utilize temporal anchors to localize the target moment without generating candidate moment in advance [5,27,40,60,74,75,79,84]. These methods inherit the anchor ideas of object detection [44,46], utilizing multiple pre-defined temporal anchors of different lengths. For instance, Chen et al. [5] sequentially exploit the fine-grained frame-by-word interactions between video and sentence, and feed the fused features into binary classifiers to predict relevance scores of multiple pre-defined temporal anchors at each time step. In a follow-up work, Wang et al. [60] additionally employ a boundary module to further explore the boundaryaware information, which leads to more precise localization. Despite the good performance, the anchors need to be carefully designed for a specific dataset and the performance is sensitive to the sizes and number of anchors [57].
Recently, we notice a solution of formulating the localization task as an end-to-end regression problem [6,36,39,47,78,81]. For instance, Rodriguez et al. [47], Mun et al. [39] and Liu et al. [33] directly regress the starting and the ending of the target moment based on the fused video-query feature by an end-to-end model. In [78], Zeng et al. regress the distances from each frame to the starting and ending of the target moment described by the query. With the similar idea of [78], Chen et al. [6] also regress the distance, while utilizes a frame feature pyramid to capture multi-level semantics. Additionally, reinforcement learning which is well known to be effective for playing electronic games, is also applied to localize activity in videos [17,54,61,63]. These works formulate the localization task as a sequence decision problem. As the most recent work using reinforcement learning, Sun et al. [54] propose to use two agents respectively adjust the start location and end location to move towards the target moment. Additionally, we also observe that Cao et al. [3] apply adversarial learning paradigm in this task, which jointly optimize the performance of both video moment ranking and video moment localization.
As the task involves both videos and text, there are works that focus on devising more effective cross-modal interaction modules [10,20,30,31,66,76,78]. For example, Liu et al. [31] propose a word attention based on the temporal context information in videos. Yuan et al. [76] design an elegant co-attention module to further exploit both video and query content. Zeng et al. [78] devise a multi-level interaction module to fuse video and text using hierarchical feature maps. Similarly, in [27], Lin et al. propose a multi-stage cross-modal interaction, aiming to explore the intrinsic relations between video and query. Our model is orthogonal to the improvement in cross-modal interaction, allowing us to flexibly embrace state-of-the-art cross-modal interaction modules.
Recently, we notice an increasing effort on solving the localizing task in a weakly-supervised manner [26,38,55,62,80] or an unsupervised manner [14]. For instance, instead of utilizing annotated video-sentence pairs for model training, Gao et al. [14] resort to the existing visual concept detectors and a pre-trained image-sentence embedding model, and transfer the knowledge from the image domain to the video domain. Though the weakly-supervised or unsupervised method are promising, their performance are still much worse than supervised methods. In this work, we mainly focus on the supervised methods. Additionally, different from the above one-stage works, our paper is the first work for progressively localizing the target moment by multiple stages in a coarse-to-fine manner. It is worth noting that our multi-stage PLN is different from previously proposed two-stage model [65]. In [65], Xiao et al. divide localization process into two stages, where they generate candidate moments in the first stage and then match the generated moments with the given sentence query in the second stage. By contrast, in our work, we regard the one whole localization process as one stage. Besides, multiple stages progressively perform localization with distinct temporal granularities, and are fused to predict the final result.

PROGRESSIVE LOCALIZATION NETWORK
Given an untrimmed video and a natural language sentence query , the language-based moment localization task is required to localize a segment/moment which is semantically relevant to the given sentence query. As the task requires cross-modal reasoning and videos often contain intricate activities, it is not easy to directly localize the target moment in videos, especially for short moments in long videos. Therefore, we propose to progressively localize the target moment in a coarse-to-fine manner, and devise a model named as Progressive Localization Network (PLN). PLN has multiple stages and each stage has a localization branch ( Fig. 2 illustrates the structure of two-stage PLN). In each stage (= 1, ..., ), a localization branch predicts a new localization result P t according to the given video , the sentence query and the learned information H −1 of the previous stage: where H −1 indicates a feature map from the localization branch of the stage − 1, which contains the learned coarse relevance of candidate moments with the given query. Such information benefits the later stage for more fine-grained localization and also connects the multiple localization branches in the whole model. It is worth noting that the previous information in the first stage is unavailable, so we ignore the H t−1 for 1 and only use the video and sentence as the inputs. By combining the localization branches 1 to together, our proposed model localizes the target moment progressively and more accurately. In what follows, we first depict the input representation, following by the description of localization branch, conditional feature manipulation, training and inference details.

Input Representation
Video Representation. Following the common practice [6,12,56,68], we use the pre-trained CNN models to represent videos. Concretely, given a video, we first split it into a fixed number  of basic video units, where each unit consists of multiple consecutive frames. For each unit, we extract the deep features using a CNN model pre-trained on ImageNet per frame, and mean pooling is performed to aggregate the features. In order to make the feature more compact, we further employ a fully connected (FC) layer with a ReLU activation. Consequently, the video is described by a sequence of unit feature vectors V = {u } =1 , where u i ∈ R is the feature vector of the -th unit, denotes the dimensionality of the unit feature vectors and is the number of the video units. It is worth noting that 3D CNNs, such as C3D [58], can also be used for feature extraction when treating consecutive frames as individual items.
Sentence Query Representation. To represent sentence queries, we employ recurrent neural network which is known to be effective for modeling long-term word dependency in natural language text. Specifically, given a sentence query consisting of words, we first embed each word of the given sentence into a word vector space by a pre-trained GloVe model [42], which results in a sequence of word embedding vectors { 1 , 2 , ... }. The word embedding vectors are then sequentially fed into a three-layer LSTM network, and the last hidden state vector of the last LSTM is taken as the representation of the sentence query, i.e., f ∈ R , where is the hidden state size of the LSTM.

Localization Branch
In each localization branch, there is a component localizer which aims to localize relevant video moment under the guidance of the given natural language sentence query and learned coarse information of the previous stages. Here, we choose 2D-TAN [82] as our component localizer, considering its state-of-the-art performance for language-based moment localization task. It is worth noting that theoretically other localization methods can also be used in our proposed progressive localization network. As 2D-TAN is not specifically designed for the multi-stage localization manner, it can not be directly employed. Hence, we adapt it to our proposed progressive multi-stage localization manner by additionally introducing a conditional feature manipulation and an upsampling connection layer. In what follows, we detail the localization branch in terms of candidate moment construction, relevance score prediction, and conditional feature manipulation. Candidate Moment Construction. Given a video, this module generates candidate moments from the input video and then represent them into a 2D temporal feature map for further reasoning. Concretely, in the -th stage, given a video represented with a sequence of video unit feature vectors V = {u } =1 , we split the video with a specific interval thus obtaining video clips {c } = , denoted as C , where c indicates the representation of -th generated video clip obtained by mean pooling over the corresponding feature vectors. We perform the clip generation with different intervals in different stages resulting in video clips of the distinct temporal granularities, and make it satisfying −1 < . The larger means the smaller temporal granularity of generated clips. In this way, the model structure conforms to the coarse-to-fine manner and is also helpful for locating relevant moment progressively. Additionally, in the stage ( > 1), we also devise a Conditional Feature Manipulation (CFM) module to modulate the video clip features under the guidance of the previous stage, expecting to refine the clip features and make them more suitable for the more fine-grained localization in the later stage. The structure of CFM is illustrated in Fig. 4 and its detail will be introduced in the following section.
Based on the above generated video clips, we enumerate the contiguous clips to build candidate moments. Theoretically, given a sequence of generated video clips, we can totally generate =1 different candidate moments with diverse lengths. In order to facilitate the exploration of the temporal dependency between candidate moments, following the previous work [82], we restructure the whole generated candidate moments to a 2D temporal feature map M ∈ R × × , where the first two axes indicate the start and end clip indexes at the stage respectively, and the third axis represents the moment features. For example, M [ , , :] = m , indicates the candidate moment that starting with the -th video clip and ending with the -th video clip, which is obtained by employing max pooling over the corresponding clip features, as exemplified in Fig.3. However, using all candidate moments will bring much more redundant information. To alleviate it, we use a sparse sampling strategy [82] that removes the redundant candidates containing large overlaps with the selected candidates. Briefly, we densely sample moments of short duration, and gradually increase the sampling interval when the moment duration becomes long. The restructured 2D temporal feature map M is further used for the relevance score prediction.
Relevance Score Prediction. Given a list of candidate moments represented as a feature map M and a query represented as f , this module aims to predict the relevance score map P t ∈ R × , where P t [ , ] indicates the relevance of the candidate moment starting with the -th video clip and ending with the -th video clip with the given query.
Concretely, we first project M and f into a -dim unified space by a Fully Connected (FC) layer respectively, and fuse them through an element-wise multiplication, resulting in the fused feature map F . Besides, we also consider the learned information in the previous stage crucial for the final prediction, thus employ a connection to inject the coarse relevance information of the previous stage into the follow-up stage. Notice that there is no previous stage in the first stage. Formally, after the injection, we obtain the fused feature that absorbed the information of the previous stage: where ⊛ denotes the element-wise max pooling, indicates our devised upsampling connection, and H −1 is a feature map of the last convolutional layer in the previous stage. Here we can not directly employ a skip connection, as the sizes of F and H −1 are different due to the different temporal granularities in different stages. Therefore, we devise a new upsampling connection which is implemented by stacked upsampling blocks, and each block has an upsampling layer with a factor of 2 and two stacked convolutional layers with 3 × 3 kernels, as shown in the upper right of Fig. 2. Then a convolutional network is further utilized to exploit the candidate dependency based on the fused feature map G . The convolutional network is comprised of 2 convolutional layers with 5 × 5 kernels, and each layer is followed by a ReLU activation. The output of the convolutional network in the -th stage is denoted as H t . Finally, we feed the output H t into a FC layer to predict the final relevance score map P ∈ R × : where indicates an element-wise sigmoid activation, W t denotes the affine matrix and b t is the bias term. In our implementation, the FC layer is implemented with a 1x1 convolutional layer.
Conditional Feature Manipulation. As shown in Fig. 4, given a sequence of video clip feature vectors C = {c } = , CFM modulates the given features under the guidance of the previous stage, obtaining the manipulated features as C = {c i } =1 .
Specifically, for each clip feature vector c in C , we modulate it to c with respect to the feature map H −1 . First of all, a max pooling layer is employed to aggregate H −1 to a feature vector h −1 . Afterwards, we fuse h −1 and c through element-wise multiplication and then a FC layer with the activation is employed to generate a scaling vector a ∈ R : where W and b parameterize the FC layer, ⊙ indicates the Hadamard product. With the above generated scaling vector a , the clip feature vector v is manipulated by scaling as: Our CFM is inspired by FiLM [43] which manipulates a neural network's intermediate features for visual reasoning. In our PLN, the motivation for CFM is two-fold: (i) we wish to adaptively recalibrate the clip features, thus making the clip features more suitable for the more fine-grained localization in the later stage. (ii) we wish to make clip features diverse across the stages thus promoting the localization branches more complementary. After performing the conditional feature manipulation, the clips of the given video can be represented as C = {c i } =1 , which is used in the later stages ( > 1) to replace the original video clip features C .  Fig. 4. The structure of conditional feature manipulation, which adaptively modulates the clip features under the guidance of the previous stage.

Training and Inference
Training. To train the proposed PLN, we add a classification loss per localization branch, and multiple losses of all branches are jointly considered. Specifically, in the -th stage, a cross-entropy loss is employed, which is defined as: where ∈ P is the predicted relevance score of the -th candidate moment and denotes the total number of valid candidates in the -th stage. For supervision labels, we employ soft labels based on candicate moments' temporal Intersection over Union (IoU) with the ground-truth moment instead of hard binary labels. Specifically, the label in the -th stage is defined as: where denotes IoU of the corresponding candidate moment with the ground-truth moment, indicates the IoU threshold. Recall that candidate moments are generated with clips of different intervals in different stages, so and are different across the stages. Finally, the joint loss of our proposed model with stages is defined as: where indicates trade-off coefficient for the -th stage. The PLN is trained by minimizing the Eq.8 in an end-to-end manner except the CNN used for extracting video features and the GloVe for word embedding are pre-trained and fixed.
Inference. After the model being trained, each localization branch is able to predict a relevance score map. Depending on which relevance score map is used for final localization, here we consider two strategies: Strategy 1. In this strategy, we select one score map among all the predicted score maps, while ignoring the others. According to the selected score map, all the corresponding candidate moments will be ranked and Non Maximum Suppression (NMS) strategy is further used to remove the redundant predicted moments.
Strategy 2. Considering all the predicted relevance score maps might be valuable for localization, we ensemble all the predicted score maps to obtain the final fused score map. Specifically, if a candidate moment is associated with multiple score maps, we average their corresponding scores in all stages as the final fused relevance score. Similarly, all the corresponding candidate moments are ranked in terms of their fused score and the NMS strategy is also performed.

EVALUATION
In this section, we first introduce our experimental settings, then compare our proposed PLN with its best setup against the state-of-the-art on three datasets. Furthermore, we conduct extensive ablation studies to investigate the impact of major design choices in the proposed model. Finally, we analyse the results on different types of moments.

Datasets.
We conduct extensive experiments on three commonly used language-based moment localization datasets: TACoS, ActivityNet Captions and Charades-STA. Table 1 summarizes the brief statistics of these datasets, and Fig. 5 shows the distributions of target moment lengths on the three datasets.
TACoS [45] is built based on the MPII Cooking Composite Activities video dataset [48], consisting of 127 videos about activities that happened in the kitchen. The average length of videos is about 286 seconds, and the average length of target moments is about 27 seconds. Among them, lots of queries are about short fine-grained actions, such as takes out the knife. Such short spanned moments show the challenging nature of this dataset. We follow the setup in 2D-TAN [82], with 9790, 4436 and 4001 sentence-moment pairs for training, validation and testing, respectively.
ActivityNet Captions [22] is the largest dataset for the task of moment localization with natural language, which is built on ActivityNet v1.3 dataset [2]. The dataset is originally developed for video captioning, and now popular for the task of moment localization with natural language as these two tasks are reversible. Compared with the above two datasets, the videos are open domain and more diverse in content. It totally has 19811 videos, and the average length is about 117 seconds. For the fair comparison, we follow the dataset split used in [82,84] using 37417, 17505, and 17031 sentence-moment pairs for training, validation, and testing, respectively.
Charades-STA [12] is built on the top of the Charades dataset [52] which is introduced for action recognition and localization. The Charades dataset has 9, 848 videos about daily indoor activities and each video is annotated with the video-level descriptions. The videos are 30 seconds long on average. Gao et al. [12] extend this by including the sentence-level temporal annotation to create Charades-STA which has 12408 sentence-moment pairs for training and 3720 pairs for testing.

Evaluation Metric.
Following the previous work [5], we report the performance in terms of the metrics @ , = ( ∈ {1, 5}, ∈ {0.1, 0.3, 0.5, 0.7}) and . @ , = is defined as the percentage of test queries for which at least one relevant moment with higher IoU than is found among the top-retrieved results.
is the mean IoU of top-1 prediction result with the ground truth moment over all test queries. All results are reported in percentage (%).

Implementation Details.
Here we detail common implementations of our proposed model in terms of model structure, training and inference. For the sentence encoding, the size of hidden states in the LSTM is 512, so = 512. For the video features, we utilize the same visual features provided by [82], i.e., 4096-dim C3D feature [58] on TACoS, 500-dim C3D feature on ActivityNet Captions, and 4096-dim VGG feature [53] on Charades-STA. Besides, we notice a trend of using a stronger I3D feature on Charades-STA, so we also use the 1024-dim I3D feature provided by [47]. The parameters of the convolution networks (ConvNet) in all stages are shared. For the upsampling connection, we use 2 upsampling blocks for two-stage models(i.e., =2), and 1 upsampling blocks for three-stage and four-stage models(i.e., =1). We use two-stage PLN to compare the state-of-the-art models, and the number of sampled clips is set to 32 and 128 on TACoS, 16 and 64 on ActivityNet Captions and Charades-STA. On ActivityNet Captions, we additionally add a sinusoidal positional encoding [59] to a sequence unit features, thus boost the order of the sequence. For the model training, we empirically set the loss weights to 1.0 and 1.5 for the two-stage model, 1.0, 1.3 and 1.5 for the three-stage model, and 1.0, 1.2, 1.5 and 2.0 for the four-stage model. The threshold with ground-truth moment is set to be 0.5. Additionally, we use Adam optimizer [21] to train the model. The initial learning rate and the batch size are empirically set to be 0.0001 and 32 respectively. The maximum number of epochs is 50. For model inference, we use non maximum suppression (NMS) strategy to remove the redundant predicted moments. Following the previous work [82], the threshold of NMS is set to be 0.4 on TACoS, 0.5 on ActivityNet Captions and 0.45 on the Charades-STA dataset.

Quantitative Comparison.
In this section, we compare our best model with the state-ofthe-art methods. The analysis on our model with different setting are summarized in Section 4.3. Table 2 shows the performance comparison on both TACoS and ActivityNet Captions, where all the models use the same C3D feature. Besides, except for our proposed PLN, all other models are one-stage. Our proposed multi-stage PLN consistently achieves the best performance on TaCoS, and compares favorably to existing methods on ActivityNet Captions. Moreover, our multi-stage PLN in a coarse-to-fine manner outperforms 2D-TAN [82] with a clear margin. 2D-TAN can be deemed as the degraded version of our proposed model, which only has one localization branch with clips of a fixed temporal granularity as input. The results again justify the viability of localizing the target moment via multiple stages with diverse temporal granularities. Additionally, we also notice a phenomenon that the performance improvement of our PLN over 2D-TAN is more significant on TACoS than that on ActivityNet Captions. As the moment length proportion of the entire video on TACoS is much smaller than that on ActivityNet Captions, we conclude that our PLN with multiple stages is more beneficial for localizing relatively short moments in long videos.
The performance comparison on Charades-STA is shown in Table 3. With the same VGG feature, our PLN performs the best except in terms of Rank@5 with IoU of 0.5. The result again confirms the effectiveness of PLN. Additionally, with the same I3D feature, PLN is worse than LGI [39] on Table 2. Performance comparison on TACoS and ActivityNet Captions, where all the models use the same C3D feature. As the scores of LGI on TACos are unavailable on the original paper, we have re-trained it using its public code with the same video feature. Other scores are directly cited from the original papers. The best and second-best results are in bold and underlined, respectively.  Table 3. Performance comparison with the same VGG or I3D feature on Charades-STA. The best and secondbest results are in bold and underlined, respectively. Charades-STA. We conjecture it to that LGI explores local-global video-text interactions via temporal attention for video-text fusion, while PLN utilizes a relatively simple element-wise multiplication. Moreover, LGI is a regression-based method, which is good at fitting data of small variations. Hence,

Method
LGI performs better on Charades-STA where the temporal length variation of target moments is small (see Fig. 5). However, on TACoS and ActivityNet Captions where the temporal length variations of target moments are much larger, our proposed PLN consistently performs better.
The results allow us to conclude that our proposed PLN is better to handle moments with large variations in temporal lengths. We attribute it to that PLN has multiple stages for localization and different stages are responsible for the candidate moments generated with the clips of distinct temporal granularities.

Qualitative
Comparison. Fig. 6 shows the qualitative comparison of our proposed PLN with 2D-TAN [82] and LGI [39]. . Additionally, we found that our PLN model generally performs better on videos where the target moment is visually distinguished with the around frames (see the third example) than that on videos where the target moment is visually very similar to the around frames (see the last example). Among the PLN variants, PLN 2,1 indicates our proposed two-stage PLN using the predicted score map of stage 1 for prediction, while PLN 2,2 is the same two-stage model but uses the output of stage 2. In general, PLN 2,1 (the early stage) gives relative coarse predictions, while LN 2,2 (the later stage) gives relative more accurate predictions. For example, in the first example, PLN 2,1 only gives an IoU score of 0.77, PLN 2,2 achieves a higher IoU score of 0.96. It is worth noting that PLN 2,2 fuses both coarse and fine-grained information of all stages for localization. The results show the importance of the coarse-to-fine information for language-based moment localization. Additionally, we also illustrate the predicted score maps of PLN 2,1 and PLN 2,2 in Fig. 7. In a specific score map, each score indicates its predicted relevance of a specific candidate moment with respect to the given query. Besides, candidate moments that are closer to each other in the   Fig. 6. Qualitative comparison of our proposed PLN with 2D-TAN [82] and LGI [39]. GT denotes the groundtruth result. PLN 2,1 indicates our proposed two-stage PLN using the output of stage 1 for prediction, while PLN 2,2 is the same two-stage model but uses the output of stage 2. score map typically have larger IoU between each other. As shown in the predicted score maps of our PLN 2,2 , moments near the target moment give high scores, while moments far from the target moment have relatively low scores. The results show that our model is able to distinguish which moment is more relevant to the given query.

Ablation Studies.
In this section, we first investigate the impact of major design choices, i.e., "Which prediction strategy?", "How many stages?" to better understand our proposed PLN. Additionally, we study the contribution of upsampling connection (UC) and conditional feature manipulation (CFM) in PLN.

Which Prediction Strategy?
In this experiment, we use the two-stage and three-stage PLN as our base models to investigate the two different prediction strategies. Table 4 summarizes the performance on TACoS. For Strategy 1, models using the score map of the later stage achieve better performance. For example, for the three-stage model, its score gradually improves from 22.75, 25.79, to 26.89 as the stage increases. Additionally, in general, Strategy 2 is slightly worse than Strategy 1 which utilizes the prediction of the last stage. Recall that Strategy 2 fuses the score maps of all stages for prediction. This result is inconsistent with the conclusion that fusion typically gains some performance improvement in other multimedia-related tasks [11]. We attribute it to that the later stage in PLN has already absorbed the information of the previous stage, thus further fusing the results of previous stages has less impact. Therefore, we use Strategy 1 based on the output of the last stage as our default inference strategy.  Table 5 summarizes the performance of PLN variants with the different number of stages on TACoS. In terms of mIoU, the models with multiple stages consistently beat the one-stage counterparts by a clear margin. The result shows the effectiveness of localizing moments by multiple stages. For multi-stage models, the two-stage one turns out to be better than the three-stage and four-stage models. We conjecture this is due to relatively limited number of training video on TACoS, only 75 training vidoes. As the stage of the model increases, the training difficulty of the model also increases, accordingly requiring more data for training. Therefore, we conjecture that the training videos on TACoS are not enough for training the three-stage and four-stage models. On ActivityNet Captions and Charades-STA, the three-stage and four-stage models are comparable to the two-stage model (See the supplementary materials). Considering the two-stage PLN achieves the best overall performance, we use the two-stage PLN as the default setting.

Coarse-to-Fine Manner?
To verify the viability of our devised coarse-to-fine manner for PLN, we compare it with the PLN using the same temporal granularity of video clips in all stages ( is the same for all stages). As shown in Table 6, our PLN with the coarse-to-fine temporal granularities (line 4 in each table block) outperforms the model with the same temporal granularity across stages (line 2, 3), showing the superiority of the coarse-to-fine manner for the multi-stage PLN. Additionally, we also report the results of one-stage model (line 1), and find it better than the  two-stage model without the coarse-to-fine temporal granularities (line 3). The result reveals the necessity of the coarse-to-fine manner for multi-stage localization.

The Effectiveness of UC and CFM.
In order to investigate the contribution of upsampling connection (UC) and conditional feature manipulation (CFM) in the proposed PLN, we conduct ablation studies and summarize the results in Table 7. On Charades-STA, we observe that our full model performs the best in terms of all metrics. Removing each component from PLN, i.e., UC or CFM, would result in performance degeneration. The result demonstrates the effectiveness of each component in our PLN.

Analysis on Different Types of Moments
To investigate how our multi-stage PLN performs on different groups of moments, we group target moments according to their length proportion of the entire video length in the test set. We compare PLN with the 2D-TAN model that to some extent can be seemed as a one-stage PLN. As shown in Fig. 8, we observe a phenomenon that the both models perform the worst on the first group where moments have the shortest average temporal duration. It to some extent demonstrates that localizing the short moments is more challenging. In almost groups, our PLN consistently outperforms 2D-TAN, which again indicates the effectiveness of our PLN for language-based moment localization. Especially, PLN achieves the highest relative performance improvement of 48.5% in the first group on TACoS and 12.6% on ActivityNet Captions. It demonstrates that our proposed multi-stage localization mechanism with the coarse-to-fine manner can better handle the relatively short target moments in long videos than one-stage 2D-TAN.

CONCLUSION
This paper shows the viability of resolving language-based moment localization in a progressive coarse-to-fine manner. We contribute a novel multi-stage Progressive Localization Network which is capable of localizing the target moment progressively via multiple localization branches. The localization branches are connected via a conditional feature manipulation module and an upsampling connection, making the later stage absorb the previously learned coarse information, thus facilitate the more fine-grained localization. Besides, we also show the potential of our multi-stage PLN for localizing short moments in long videos, which has been ignored by previous works. We believe that this simple and effective multi-stage progressive architecture can be of interest to many language-based moment localization research efforts. In future work, we will apply the idea of coarse-to-fine localization to other related tasks such as video object grounding [64,70], and video relation detection [23,49].