{'fraction_non_alphanumeric': 0.033163318850618136, 'fraction_numerical': 0.017211342694624605, 'mean_word_length': 4.897264512934769, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Dynamic Attention guided Multi-Trajectory Analysis for Single Object Tracking

Most of the existing single object trackers track the target in a unitary local search window, making them particularly vulnerable to challenging factors such as heavy occlusions and out-of-view movements. Despite the attempts to further incorporate global search, prevailing mechanisms that cooperate local and global search are relatively static, thus are still sub-optimal for improving tracking performance. By further studying the local and global search results, we raise a question: can we allow more dynamics for cooperating both results? In this paper, we propose to introduce more dynamics by devising a dynamic attention-guided multi-trajectory tracking strategy. In particular, we construct dynamic appearance model that contains multiple target templates, each of which provides its own attention for locating the target in the new frame. Guided by different attention, we maintain diversified tracking results for the target to build multi-trajectory tracking history, allowing more candidates to represent the true target trajectory. After spanning the whole sequence, we introduce a multi-trajectory selection network to find the best trajectory that delivers improved tracking performance. Extensive experimental results show that our proposed tracking strategy achieves compelling performance on various large-scale tracking benchmarks. The project page of this paper can be found at https://sites.google.com/view/mt-track/.

. The comparison between single trajectory tracking (the first row) and our dynamic target-aware attention guided multi-trajectory tracking framework (the bottom row). Existing trackers usually estimate one location only for each frame, which may prone to model drift in challenging scenarios even the global search scheme adopted. Our proposed multi-trajectory tracking framework maintain multiple locations guided by dynamic target-aware attention for each frame and significantly increase the dynamics of baseline trackers. which drives us to devise a novel and more advanced tracking strategy to tackle these challenging factors.
According to our observation, many of the existing single object trackers that follow the tracking-by-detection framework generally attempt to track the target object, which is initialized in the first frame, based on a unitary local search window. However, without modeling global information appropriately, such a local search strategy can be sensitive to the aforementioned challenging factors, thus visual trackers can be prone to drifting away. To handle these issues, some researchers propose to incorporate global search strategy into visual tracking [3] [4] [5] [6] [7]. Despite that current global search approaches can handle the issues of the local search tracking strategy to some extents, they still suffer from the dilemma of selection between local and global search for each frame. Moreover, it is also difficult for current approaches that introduce global information to accurately locate the potential regions that contain the target, especially when multiple similar objects appeared in the image simultaneously.
With deeper investigation, we find that current approaches, even those that cooperate local and global search results, only model target appearance and estimate target status in a relatively static way, i.e., the tracking paradigms for each frame is fixed and static to a single bounding box. In other words, current visual trackers can be easily distracted by similar objects or noisy backgrounds when challenging factors like occlusions and out-of-view movements occur in the sequence if the target appearance modeled by the tracker is corrupted in arXiv:2103.16086v1 [cs.CV] 30 Mar 2021 the estimated bounding box. Furthermore, in existing trackers, the noise of an unreliable tracking result at a frame could also be continuously accumulated in later frames when the estimated target bounding box at each frame drift away from true position. This easily leads to more severe drifting. Even though some studies [8] [9] [10] introduce re-detect mechanism to correct the inaccurate tracking results once the target object is found to be lost, the already failed tracking results before re-detection will not be corrected anyway, and these erroneous tracking results still degrade the overall tracking performance.
To tackle these problems, we instead seek to explore a novel search strategy that can allow more dynamics for single object tracking.
We are inspired by the recent progress of image captioning [11] [12] task, in which given images are generally embedded into deep representations, and a sequential model is applied to deliver corresponding language descriptions based on the embedded representations. In particular, by maintaining multiple words at each time step, the decoding stage provides more adequate language descriptions by applying beam search algorithm to allow more dynamics when analyzing the maintained words. We believe that the visual trackers can also take advantage of such a strategy that searches better results through different candidates, if multiple different tracking states and trajectories are maintained during tracking.
In this study, we propose a novel dynamic attention-guided multi-trajectory tracking framework to introduce more dynamics for tackling various challenging tracking issues. As shown in Fig. 2, our tracker consists of three main modules, i.e., a novel dynamic target-aware attention network to guide the global search for the target, a baseline tracker for local search about the target, and a multi-trajectory selection network. Different from existing works [4] [5] [6], our dynamic targetaware attention network maintains multiple target templates and fully utilizes hierarchical semantic feature representations obtained from different convolutional layers to provide more robust appearance model for the target and more accurate candidate regions when performing global search. Meanwhile, in each frame, the baseline tracker is employed to conduct local search for the target. By performing a joint local and global search for visual tracking in a parallel manner for each frame, we obtain and maintain multiple candidate states for the target. After spanning the whole video sequence, we employ a multi-trajectory selection network to find the best tracking trajectory based on different maintained tracking results for the sequence.
To sum up, the contributions of this paper can be summarized in the following three aspects: − We propose a novel dynamic attention-guided multitrajectory based tracking framework for single object tracking. In particular, based on multiple tracking trajectories maintained for the target, our proposed approach introduces more dynamics for cooperating local and global tracking results to help tackle the challenging tracking issues effectively.
− We propose a dynamic target-aware attention network to build a more robust and dynamic appearance model for global search based tracking. We also devise a novel multitrajectory selection network to estimate the best tracking results according to maintained multi-trajectory information.
− Our proposed tracking strategy achieves compelling performance on several large-scale tracking benchmarks, validating the effectiveness of our proposed method.

II. RELATED WORK
Siamese Network based Trackers. Many trackers are developed based on Siamese network due to its high accuracy and efficiency. Tao et al. [13] propose to use the Siamese network to learn a fixed matching function and tracking the target object without any update. Wang et al. [14] further developed this framework by introducing adversarial samples for robust visual tracking. Bertinetto et al.
[?] use a fully convolutional Siamese network for tracking by measuring the region-wise feature similarity between the target object and candidates (named SiamFC). GOTURN is proposed by Held et al. [15] which uses a motion prediction model developed based on Siamese network for high-speed tracking. Recently, many researchers focus on designing a more powerful network for Siamese based tracker, such as SiamDW [1], SiamRPN++ [2]. Some works improved the Siamese network based tracker by cascaded region proposal networks [16], relation reasoning [17], dynamic target template update [18] [19], re-detection [20], meta-learning [21], [22], and others [23]. Although these works all improved the Siamese network based tracker from different perspectives, however, they still adopt the greedy search based strategy for visual tracking. Therefore, these trackers may be sensitive to the challenging factors we mentioned above. In this paper, we propose a novel multi-trajectory tracking framework for visual tracking which can address issues caused by greedy search to some extent.
Long-term Tracking. Existing long-term tracking algorithms usually introduce re-detection mechanism in the tracking procedure. Kalal et al. [24] propose a tracking-learningdetection framework for long-term tracking. They utilize optical-flow based matcher for local search and also adopt the ensemble of weak classifiers for re-detection. Ma et al.
[8] propose a long-term correlation filter which use KCF for local tracking and external random ferns classifier for long-term detector. Valmadre et al. [25] develop a long-term tracker named SiamFC+R, which also integrates a simple redetection scheme with SiamFC and conduct re-detection when the SiamFC's response is lower than a pre-defined threshold. Some works attempt to address the tracking task with key point matching [26] [10] or global proposal mechanism [7] [4] [5] [6]. However, as noted in [3], the keypoint extractors and descriptors are not reliable in complex environments, which may limit their overall performance. Zhu et al. [7] propose the EBT tracking based on EdgeBox [27] and improve the baseline method significantly. Yan et al. [3] propose a 'Skimming-Perusal' tracking framework for real-time and long-term tracking. They use a switching mechanism to decide local or global proposals should be used in each frame and achieve better results on OxUvA dataset. Fan et al. [28] propose a parallel tracking and verifying framework (PTAV) which can achieve better results on UAV20L dataset. Wang et al. [4] [5] [6] develop the target-aware attention mechanism for global proposal generation and integrate with MDNet for robust tracking. GlobalTrack is proposed in [29] which only employ global search on whole video frames also achieves good performance on long-term datasets. Dai et al. [30] propose offline-trained meta-updater to address the update issue in long-term tracking.
Trajectory-based Tracking. Although very few, there are still some related works that exploit trajectory information to achieve tracking. For example, MTA [31] is proposed to conduct tracking by trajectory selection using tracking results obtained from STRUCK [32]. In practice, the MTA is subject to poor hand-crafted features and empirical multitrajectory analysis that can not be optimized. In addition to MTA, researchers also develop MHT [33] to conduct multi-object tracking based on different trajectories. To track multiple objects, the MHT designs a track tree construction and updating scheme to allow more dynamics. However, there are many differences between our proposed method and MHT. First, MHT relies on hand-crafted features and requires detected bounding boxes, while our framework is built upon more powerful deep convolutional features and does not require the detection model to provide bounding box results. Second, MHT uses tree structure to derive trajectories, while our method maintain independent tracking results and derive trajectories based on beam search strategy which can allow more trajectories to describe a single target. Also, MHT relies on detection models to encode target appearances, while our method can dynamically encode appearances. Lastly, rather than MHT that performs empirical tree construction algorithm to deliver trajectory results, our proposed multi-trajectory selection network can be optimized to select the best trajectory result.

III. OUR PROPOSED APPROACH
In this section, we will first give an introduction to the baseline tracker THOR used in this work. Then, we will give an overview of our proposed modules. After that, we will dive into the details on dynamic attention model, trajectory selection network. We mainly focus on the motivation, detailed network architecture, and advantages of our proposed modules for object tracking.

A. Preliminary: THOR
THOR is a Siamese network based visual tracker proposed by Axel Sauer et al. [19]. They design a simple but effective target template update mechanism for tracking procedure. Specifically, they utilize two modules, i.e., the short-term module (STM) and long-term module (LTM) to store the tracked results. The authors define a novel diversity measure in the space of Siamese features to select the most diverse templates.
For the long-term module, they try to maximize the volume Γ(f 1 , ..., f n ) of the parallelotope formed by the feature vectors f i of the template T i . They use convolutional operation to compute the similarity between different templates in memory and obtain a Gram matrix: where G is a square n × n matrix. Therefore, the Gram determinant (i.e., the determinant of G) can be written as: The template can be incorporated into the memory, if the Gram determinant can be increased when replacing one of the allocated templates. The short-term module is introduced to handle abrupt movements and partial occlusion. They update the memory slots of STM in a first-in, first-out manner. Different from LTM, they compute the diversity measure γ of STM as follows: The authors integrate the LTM and STM module with three Siamese network based trackers (SiamFC [34], SiamRPN [35] and SiamMask [36]), and achieve better tracking performance than the baseline methods. More details of THOR can be found in [19]. THOR works well on short-term and small datasets, the performance on large-scale and long-term tracking datasets are still unknown. This tracker also adopts a greedy search in a local search window which makes their performance unsatisfied on challenging benchmarks. In this paper, we introduce a novel dynamic target-aware attention mechanism and integrate with SiamRPN based THOR for robust tracking in the multi-trajectory manner.

B. Overview
In general, our proposed method consists of three key components, including the dynamic target-aware attention network, a local search-based baseline tracker, and the multitrajectory selection network. Given a testing video sequence, we first perform local tracking based on a Siamese tracker (e.g. THOR [19]) to search the target according to the response map of a local region. In the meantime, we perform global search to locate the target within the whole frame. To tackle the challenging factors, such as heavy occlusions, fast motions, and out-of-view movements, we introduce a novel dynamic target-aware attention network to help construct a more robust and dynamic appearance model, facilitating the global tracking to better locate the target. Moreover, rather than cooperating local and global search results in a relatively static way, we propose to maintain multiple different tracking results obtained from local and global search in a parallel manner to achieve multi-trajectory tracking with more dynamics. At the end of testing video, with the help of a carefully designed trajectory selection network, we select the trajectory with the highest confidence score to deliver a further refined final tracking results. C. Dynamic Target-Aware Attention Network 1) Our Approach: We proposed the dynamic target-aware attention network to construct dynamic appearance model by maintaining multiple target templates. In particular, we first extract feature maps from three different convolutional modules to deliver hierarchical and more powerful representations. Then, we treat target features as convolutional filters. We employ these filters to perform convolution on the feature maps of the whole image. The output features will be later fed into a gate layer which is used to control the information flow. Afterwards, the gated features will be gradually fed into a decoder network and output corresponding attention map. Inspired by [37], we incorporate the point proposals to make the attention map focus more on the target regions encoded by the coordinates of the bounding box corners to handle the issues caused by similar target objects. In our proposed method, we encode point proposals into horizontal and vertical feature maps with relative CoordConv [37] and regard it as prior information to estimate the current attention map. Also, we apply an RoI pooling layer to extract the point feature and feed it into an adaptive instance layer [38] to achieve instance-level attention estimation. Moreover, we employ the local search-based tracker to help collect and maintain a set of short-term and long-term target templates (i.e., the dynamic template pool in Fig. 4) to provide dynamic target feature representations during the tracking procedure. In this way, we can utilize the updated target templates to conduct dynamic target-aware attention estimation. In general, the proposed dynamic target-aware attention network can provide more accurate global search regions than the static counterpart methods [4] [5] [6]. An overview of our attention network is given in Fig. 3.
2) Network Architecture: Given a video frame I and a target template T , we first resize them into 300 × 300 and 100 × 100, respectively. Then, we put them into an encoder network with two branches. Each branch is a residual network (ResNet-18 used in this paper) which is pre-trained on ImageNet classification task. To obtain a more effective feature representation, we extract the hierarchical semantics from feature maps of both inputs from three convolutional blocks whose dimension are [ where denotes the convolutional operator. Then, F T I i is fed into a gate layer to achieve controllable information flow which is widely used in recurrent neural network (such as forget gate, input gate in LSTM [39]). This process can be written as:F where σ is a sigmoid gate layer and the denote dot product between two matrix. These features will be fed into a decoder network gradually using skip connections to predict corresponding target-aware attention map. Global search-based tracking that only utilizes static target templates could suffer from the issue of the existence of multiple similar target objects in the image. Since the global search module will provide multiple search regions for different similar target objects, it can be very difficult for the tracker to recognize the true target. In this paper, we propose to utilize point proposal (x, y) to help target-aware attention estimation for a more robust global search. The center location of the previous tracking result is considered as the point proposal. Specifically, the point proposal is used in two ways: the location prior of target object and "characteristic" embedding. For the location prior, existing works [40] [37] [41] all shown that the pixel coordinates are important to disambiguate different object instances. Liu et al. [41] proposed a simple CoordConv layer to encode the pixel coordinates into feature maps by creating a tensor of the same size as input that contains pixel coordinates normalized to [-1, 1]. The authors of [37] further proposed a Relative CoordConv block which can utilize standard backbone networks for their task, such as pre-trained ResNet. Following [37], we also construct two coordinate feature maps, one for x-coordinates and one for y-coordinates. The values of constructed x-map vary from −1 to +1, where −1 and +1 corresponding to x − R and x + R, respectively. The y-map has similar attributes and the R is a hyper-parameter used to denote the radius of Relative CoordConv. The constructed prior map is concatenated with feature maps produced by the regular convolutional network. Besides, we also attain the "characteristic" embedding based on the point proposal. More specifically, given the point proposal (x, y), we extract a 1 × 1 × 512 feature Q(x, y) with RoI pooling on the feature maps generated from the third convolutional layer of decoder network. This feature vector is then fed into two fully connected layers and the AdaIN layer [38] to achieve instance selection. The output feature will be integrated into the decoder network.
Using the template representation collected and maintained by the local search-based baseline tracker to achieve Siamese tracking, our attention estimation can be implemented in a dynamic manner, while previous works are all static targetaware attention. In other word, we can borrow the updated target representation in the tracking procedure for more accurate attention estimation. Adaptively switching between local and global search is one intuitive approach for robust tracking as many previous trackers do [3]- [6]. To maximize the benefits of dynamic appearance model for the whole video sequence, our tracker runs in a batch manner.
3) Advantages of Our Approach: Compared with original target-aware attention based trackers, the highlights of our proposed dynamic attention model can be listed as follows: Firstly, the original method exploits the features of the target object and global image based on simple concatenation operation without considering the relationships and interactions between the two types of features. This may limit the representational power of the model. Secondly, they only utilize the feature map from the last convolutional layer of their encoder, which can not fully utilize the hierarchical semantic information for better attention prediction. Thirdly, they only utilize the feature map of the target object initialized in the first frame. However, since the appearance of the target is generally continuously changing in a tracking sequence, such a fixed appearance model is almost infeasible to track the target that undergoes significant appearance variations in the corresponding video sequence. Last but not the least, their attention model is also too primitive to handle multiple similar target objects. In practice, there would be multiple high response regions which may easily distract the tracker on the similar but non-target objects.
As a result, the aforementioned issues inspire us to design an advanced new attention scheme for tracking in challenging scenarios, especially for long-term object tracking. In particular, we propose the dynamic target-aware attention mechanism, as shown in Fig. 3, to introduce more dynamics in the global search-based tracking procedure. By further addressing the issues of the original target-aware attention methods [4] [5] [6] in this study, our method constructs a more robust and more dynamic target-aware attention model for global searchbased tracking.

D. Multi-Trajectory Selection Network
Based on aforementioned dynamic attention model, we explore the beam search strategy to replace greedy search policy for visual tracking. Specifically, we keep multiple different tracking results during tracking and find the best trajectory to approach robust tracking. In particular, in each frame, instead of simply selecting the location/proposal with the highest confidence score as the final tracking result, we keep record of multiple candidate tracking results in a beam search manner: first, we obtain the candidate regions from the global search module, i.e., the dynamic target-aware attention network. Then, we locate the target from these regions with the local search-based tracker and select the most reliable topk results as the current search results, obtaining k potential trajectories, as shown in Fig. 4. Similar operations are executed for the subsequent video frames until the end of the video. After this procedure is completed, we measure the quality of tracking results in each frame with the trajectory selection network. A new trajectory with a maximum selection score will be chosen as the final tracking result of the current test video. For example, suppose that a video has T frames and each frame has 3 candidates, we always maintain 3 trajectories for tracking, since introducing more candidates will cost excessively large computational time. As illustrated in Fig. 5, two threshold parameters δ 1 and δ 2 are used to measure the quality of current tracking results for 3 trajectories. We denote Traj-1 as the local search result. We also denote r as the confidence score of the local search based tracker. For each frame, if r ≤ δ 1 , the search region will be switched into global attention region with best similarity (i.e., the dark orange points in Fig.  5, different colors means various confidence), therefore we have a new trajectory Traj-2. Meanwhile, if r ≤ δ 2 , we search the target object from another global attention region if existed, therefore, we can attain the Traj-3. The Traj-1, 2, 3 then form the 3-trajectory search. This procedure shares a similar idea with beam search which is a heuristic search algorithm that explores a graph by expanding the most promising node in a limited set and widely used in image caption [42]. Our proposed multi-trajectory selection policy and beam search all attempt to maintain multiple candidates for final selection. It is easy to find that our tracker can conduct joint local and global search in an adaptive manner for robust tracking and our model can still work well on long-term videos. After performing multi-trajectory based tracking through the whole video, as shown in Fig. 6, we take all the available information as the input of the multi-trajectory selection network to accurately predict the quality of each trajectory. The encoded information includes input video frames, initial target template, predicted attention map, the cropped tracking result and corresponding target-aware attention patch, the specific value of predicted bounding box, and its similarity score predicted by the Siamese tracker. For all the input patches, we resize them into 300 × 300 and then feed them into a residual network. By aggregating these features via concatenation and reshaping, we send the aggregated features into a fully connected layer for dimension reduction. For bounding boxes and scores, we utilize two fully connected layers to encode them into corresponding feature vectors, respectively. Then, these feature vectors will be concatenated and fed into two fully connected layers for regression. The ground truth for this regression task is the IoU (Intersection over Union) between the predicted trajectory and ground truth annotations. In our implementation, we adopt GIoU [43] which is an improved evaluation metric between two bounding boxes.
OTB-2015 [47] contains 100 video sequences and also defines 9 attributes such as Illumination Variation, Scale Variation, Occlusion, Deformation. It is one of the most widely used benchmark datasets for visual tracking since its release in 2015.
GOT-10k [46] is constructed based on the backbone of WordNet structure [51]. It populates the majority of over 560 classes of moving objects and 87 motion patterns. It contains 10,000 videos totally, with more than 1.5 million manually labeled bounding boxes. The authors select 280 videos as the test subset and the rest of videos are used for training.
OxUvA [25] is developed for the training and evaluation of long-term trackers. It comprises 366 sequences spanning 14 hours of video which can be categorized into 22 classes. This dataset is divided into train and testing subset, which contains 200 and 166 videos respectively.
LaSOT [48] is the currently the largest long-term tracking dataset which contains 1400 video sequences with more than 3.5M frames in total. The average video length is more than 2,500 frames and each video contains challenging factors deriving from the wild, e.g., out-of-view, scale variation. It provides both natural language and bounding box annotations which can be used for the explorations of integrating visual and natural language features for robust tracking. For the evaluation of LaSOT dataset, Protocol I employs all 1400 sequences for evaluation and Protocol II uses the testing subset of 280 videos.
VOT2018-LT [49] is a long-term dataset which contains 35 videos with a total length of 146817 frames. It is calculated that the target object will disappear for 12 times and each lasting on average 40 frames for each video.
UAV123 [50] and UAV20L [50] is an aerial video dataset which designed for low altitude UAV target tracking. It is consisted of 123 videos comprising more than 110K frames. They also provide a high-fidelity real-time visual tracking simulator for evaluation. The authors also merge these subsequences and pick the 20 longest sequences for long-term evaluation, also termed UAV20L.
For OTB-2015 [47], GOT-10k [46], LaSOT [48], UAV123 [50] and UAV20L [50], Precision Plots and Success Plots are adopted for the evaluation (also termed PR and SR). The first evaluation metric illustrates the percentage of frames where the center location error between the object location and ground truth is smaller than a pre-defined threshold (20-pixel threshold are usually adopted). The second one demonstrates the percentage of frames the Intersection over Union (IoU) of the predicted and the ground truth bounding boxes is higher than a given ratio. It is worthy to note that the AO is also adopted for the evaluation of GOT-10k [46] dataset. The AO denotes the average of overlaps between all ground truth and estimated bounding boxes. The VOT2018-LT [49] dataset adopts Precision, Recall and F1-score for the evaluation. Specifically, the definition of these metrics are: where TP, FP and FN are used to denote the True Positive, False Positive and False Negative, respectively. For the Ox-UvA [25] long-term tracking benchmark, the TPR, TNR and MaxGM are adopted for the evaluation. Specifically, the TPR gives the fraction of present objects that are reported present and correctly located, while TNR gives the fraction of absent objects that are reported absent. The MaxGM is a single measure of tracking performance which can be formulated as:

B. Implementation Details
In this paper, we utilize binary cross-entropy loss to train the dynamic target-aware attention network. The ground truth mask is obtained from existing tracking datasets by whiting the target object and black the background regions as previous works do [4] [5] [6]. The batch size is 20, learning rate is 1e − 4. For the training of the trajectory selection network, we first run our target-aware attention guided THOR to collect the predicted trajectories and attention maps on selected 3k video sequences from the training subset of the GOT-10k dataset. After that, we train this network for 50 epochs on the collected dataset. The initial learning rate is 0.001, the batch size is 10 and the Adagrad [52] optimizer is selected to optimize these two networks. For each time step, we input 12 frames for the TSN network for the trajectory evaluation due to the limitation of our GPU. δ 1 and δ 2 are experimentally set as 0.5 and 0.6 in all the experiments. Our code is developed based on PyTorch and the experiments are conducted on a server with Ubuntu 16.04.3 LTS, Intel(R) Xeon(R) CPU E5-2620 v4@2.10GHz, GeForce RTX 2080. The source code is available at https: //github.com/wangxiao5791509/DeepMTA_PyTorch.

C. Comparison on Public Benchmarks
In this section, we will report the tracking results of our method and other trackers on LaSOT, GOT-10k, UAV20L, OTB-2015, UAV123 and OxUvA datasets, respectively. It is worthy to note that we utilize our THOR+BS-3T version (termed DeepMTA) to compare with other trackers, although higher performance can be obtained with more trajectories.
LaSOT [48]: As shown in Fig. 7 (a), our tracker achieve 0.411 and 0.444 on precision plots and success plots based on the P rotocolI, which are significantly better than the baseline THOR (0.374/0.411) and other trackers. For the P rotocolII, our results are also better than these trackers as illustrated in Fig. 7 (b). This fully demonstrates the effectiveness of our proposed tracker.
GOT-10k [46]: As we can see from Table I, our tracker achieves better results than the baseline method THOR and also most of the other recent and popular trackers, such as MDNet and ECO 1 . Specifically, the baseline tracker THOR achieve 0.447, 0.538 on the AO and SR 0.50 , respectively, while our tracker which utilize the dynamic target-aware attention guided multi-trajectory tracking obtained better results (0.462 and 0.556 on the evaluation metric).
OTB-2015 [47]: As shown in Table II, THOR achieves 0.787/0.641 on the OTB-2015 dataset which are already better than most of the compared trackers, such as CFNet, SiamFC and Staple. Thanks to the dynamic target-aware attention, which can provide us global search regions for more robust tracking, we achieve 0.799/0.650 on this dataset. Although our overall performance is not better than some trackers (ECO: 0.910/0.691) on this dataset, however, our results are significantly better than these trackers on large-scale tracking benchmarks, such as LaSOT. We think this may be caused by overfitting of their tracker on this small and short-term dataset. Tracking results on LaSOT benchmarks. DeepMTA is our proposed tracker in this paper. Table II, the baseline tracker THOR achieves 0.758/0.697 on PR and SR, which is comparable with existing trackers like RT-MDNet, SiamRPN and DaSiamRPN. Our tracker attains the best performance on this benchmark compared with these trackers, which are 0.814/0.746 on PR/SR, respectively. These experiments also fully demonstrate the effectiveness of our proposed dynamic target-aware attention guided multi-trajectory tracking framework.

UAV123 [50]: As shown in
UAV20L [50]: As shown in Table II, our tracker achieves 0.715/0.570 on the PR and SR, respectively, which are significantly better than the baseline tracker THOR and also some recent strong trackers like PTAV, RT-MDNet and SRDCF. This experiment fully validated the effectiveness of our proposed multi-trajectory tracking framework.
OxUvA [25]: As shown in Table III, we report the results on the test dataset of OxUvA which contains 166 long-term videos. The baseline method THOR achieves 0.320, 0.410 on MaxGM and TPR, while our tracker obtains 0.340 and 0.463.

D. Ablation Study
In this section, the following notations are needed to be watchful to better understand our model. Specifically, the MDC is short for the modules of convolutional operators between the feature maps of target template and global images. PP denotes the point proposal module, GS is short for global search mechanism, and BS means the beam search scheme for robust tracking, i.e., the multi-trajectory analysis module.
Analysis on Dynamic Target-Aware Attention. To check the effectiveness of each component of our target-aware attention model, we implement the following component analysis: 1). ResNet+Concat: naive version for target-aware attention estimation. We directly concatenate the feature map of the target object and global image as previous work does [4].  2). ResNet+MDC: we conduct convolutional operations on multiple hierarchical feature maps, to check the effectiveness of interactions between target template and global images.
3). ResNet+MDC+PP: we integrate the point proposal to check the influence of spatial coordinates.
In this section, we utilize MAE (Mean Absolute Error), which is a widely used evaluation metric in the salient object detection community [63], [64], to measure the quality of predicted attention maps of each model. As shown in Table IV, the MAE of ResNet+Concat is 4.27, while the ResNet+MDC and ResNet+MDC+PP are 3.99 and 2.23, respectively. It is easy to find that the introduced convolutional operation and spatial coordinates are all contributed to the dynamic targetaware attention estimation. Some attention maps predicted with these models can be found in Fig. 8, and more results are provided in Fig. 12. Analysis on Multi-Trajectory Tracking. To check the effectiveness of our proposed multi-trajectory inference strategy, we conduct the following component analysis: i). THOR: the baseline method used in this paper; ii). THOR+GS: we introduce target-aware attention for global search and integrate with THOR to check the effectiveness of target-aware attention; iii). THOR+BS-2T/3T: we utilize double/triplet-trajectory search strategy for component-ii to check the influence of multi-trajectory tracking; iv). THOR+BS-3T-TSN: we use the trajectory selection network (TSN) for final trajectory selection to check the effectiveness of this module. Table I, the baseline tracker THOR achieves 0.447/0.538 on AR and SR, respectively. When integrating our dynamic target-aware attention module into THOR, the tracking performance improved to 0.453/0.545, which fully demonstrates the effectiveness of dynamic target-aware attention for global search. We also conduct tracking based on our proposed multi-trajectory tracking framework and further improve the tracking results. For example, the THOR+BS-2T/3T improves the tracking results from 0.453/0.545 to 0.458/0.550 and 0.461/0.554, respectively. These experiments fully validated the effectiveness of our multi-trajectory tracking framework. It is also worthy to note that the aforementioned trackers are all select the final trajectory based on their response score only. However, the response score can not reflect the true results in some complex scenarios. The performance can be further improved by using our proposed trajectory selection network, which not only considers the response score from tracker but also the target-aware attention and consistency of the tracked target object. The results of THOR+BS-3T-TSN fully validated the effectiveness of this module.

As shown in
From a methodological point of view, more trajectories mean more dynamics we can capture with our model. As shown in Table I, we can find that the tracking results can be further improved by introducing more trajectories. Specifically, the tracking results of 1, 2 and 3 trajectories on the GOT-10k dataset are 0.453/0.545, 0.458/0.550 and 0.461/0.554 which are consistent with our views. In addition, we also evaluate Analysis on the Generalization. In our experiments, we select the THOR as our baseline tracker for tracking. It is worthy to note that our proposed algorithm can also be integrated with other tracking algorithms due to it is a generic module. In this section, we test our module by integrating them with SiamFC++ [65], SiamMask [36], DiMP [66], MDNet [56], SiamRPN++ [2], and ATOM [58] on the GOT-10k and VOT2018LT dataset to check the generalization.
As shown in Table V, we can find that our module can improve all the baseline approaches on both datasets. Specifically, the SiamFC++ achieves 0.604/0.734 on the AO/SR respectively, while our method attain 0.609/0.738 on the GOT-10k dataset. We also improve the SiamMask from 0.451/0.541 to 0.461/0.555, the MDNet from 0.299/0.303 to 0.392/0.433. The improvement of SiamFC++ and SiamMask on the GOT-10k dataset is relatively little, due to the videos are short, therefore, there is little room for our module to improve final results. For the long-term benchmark VOT2018LT which contains 35 videos, we can find that the improvements are significant. More detail, the SiamFC++ achieves 0.689, 0.457, 0.549 on precision/recall/F1-score respectively, meanwhile, we improve these metrics to 0.684, 0.544, 0.606. For the minor decrease of SiamFC++ on the Precision, we think this maybe caused by the tradeoff between local search and global search mechanism. The experimental results based on MDNet, SiamMask, DiMP, ATOM, and SiamRPN++ trackers also demonstrate that our proposed modules are effective for tracking task, especially on the long-term video sequences.
Efficiency Analysis. The baseline tracker THOR running at 112 FPS reported in their original paper (named as THOR-I in Table VI); while it running at 79.73 FPS on a Laptop with CPU Intel I7 and GPU NVIDIA RTX 2070 (i.e., the THOR-II in Table VI). This speed is tested on the whole OTB-2015 dataset. Our tracker can run at 12.10 and 12.07 FPS respectively when 2 and 3 trajectories are adopted. It is also worthy to note that this running time includes both the multi-trajectory tracking and multi-trajectory selection. We believe that our tracker can obtain better running efficiency, if better GPU is utilized, such as NVIDIA RTX 2080.
Influence of Threshold Parameters. In this work, two parameters δ 1 and δ 2 are very important for final tracking. We report the tracking results on OTB-2015 dataset with different settings in Table VII. Specifically speaking, we first fix δ 2 as 0.6, and test the performance with various values of δ 1 , i.e., from 0.3 to 0.9. We can find that the results will be better when the δ 1 is 0.4. Then, we fix the δ 1 as 0.4, and check the results with various values of δ 2 . Finally, it is easy to find that we can attain the best performance when δ 1 and δ 2 are set as 0.4 and 0.6. Meanwhile, we can also find that our tracker is not so sensitive to these two parameters. Because the results are relatively stable when the value changing from 0.3 to 0.7.
Analysis on Different Number of Frames for TSN Module. In our experiments, the TSN network is used for trajectory evaluation by processing the whole trajectory in a batch manner, i.e., we set the batch size as 12, due to the limited memory of our GPU. Much larger frame number can also be used, if the GPU meets the demand. To analyse the influence of this parameter, we also tested other values on GOT-10k dataset. From Fig. 9, we find that the overall results are relatively stable when different values are used, i.e., 6, 8, 10, 12. Attribute Analysis. In this section, we report the results of our tracker and some state-of-the-art trackers on each attribute as shown in Fig. 10 (Success plots) and Fig. 11 (Precision plots). It is easy to find that our tracker achieves the best performance on most of attributes, including out-of-view,  Visualization. In this section, we give some visualization of our attention maps and tracking results respectively.
Attention Maps: As we can see from Fig. 12, our dynamic target-aware attention network can locate the attention regions which most related to the target object we want to track for each video. Our model show good robustness to clutter background, heavy occlusion, motion blur and view rotation, etc. Specifically speaking, our attention model can still locate the target object accurately in the clutter background, such as the shark and person in the first and second row, respectively. This fully validated the effectiveness of our convolutional operation based target-aware attention prediction. It is also worthy to note that our attention can reflect the occluded target object to some extent, such as the shark in the 0026 and 1054 frame at the first row, the dog in the 1030 at the fourth row. These amazing attention results are brought by our dynamic point proposal, which can provide spatial coordinate information for a more accurate target object location. Besides, the fox and car in the third and fifth row demonstrate that our dynamic attention model is robust to the view variation of the target object.
According to the aforementioned analysis, we can conclude that our dynamic attention model shows good robustness to challenging factors, such as motion blur, heavy occlusion, out-of-view, clutter background. The tracking results on each attribute on the LaSOT dataset also proved the robustness of our model, as shown in Section IV-D.
Tracking Results: As shown in Fig. 13, we give some visualizations of our tracker and other state-of-the-art trackers on the LaSOT dataset. It is intuitive to find that our tracker is robust to challenging factors such as out-of-view, clutter background, scale variation. For example, the flying kite in the first row will become out of the view, when it occurred back, our tracker can still capture its location due to the utilize of joint local and global search scheme. However, many other trackers failed to locate the target object due to only the local search mechanism used in their procedure. This also demonstrates the importance of accurate prediction of dynamic target-aware attention maps. For the second and third row, we can find that our tracker (red bounding box) can locate the target object more accurately than the baseline tracker THOR (green bounding box). This fully validates the effectiveness of our tracking algorithm. For the fourth row, we can find that our tracker can still work well in challenging scenarios, while many other trackers (including the baseline tracker) are easily influenced by the clutter background.

E. Discussion
The baseline tracker THOR only use local search under tracking-by-detection framework; the target-aware attention model provides the global attention map which can be used for global search for baseline tracker. On the other hand, the baseline tracker can provides tracking results to dynamically modeling the target object in the target-aware attention module. Therefore, these two modules are complementary to each other. Tracking by switching between local and global search is an intuitive way for robust object tracking, however, this may still confuse trackers when challenging factors occurred. Therefore, the multi-trajectory analysis module is introduced to collect multi-trajectories and conduct tracking by multitrajectory selection. This procedure is mainly implemented by the trajectory selection network. It is also worthy to note that our algorithm tracking the target object in an batch manner which will be beneficial for specific applications, such as video analysis in sport and surveillance.

V. CONCLUSION AND FUTURE WORK
In this paper, we propose a novel multi-trajectory tracking framework, which significantly increased the dynamics of visual tracking. Specifically, we maintain multiple tracking results for each frame based on joint local and global search. To conduct a more accurate global search, we design a novel dynamic target-aware attention module which receives dynamic target templates and coordinates as condition and estimate target location from global views. After all the video frames are processed, we select the best trajectory with our proposed trajectory selection network, which considers multiple information, such as attention maps, response scores, and coordinates of BBox. Extensive experiments are conducted on multiple tracking dataset, including short-term and longterm tracking datasets.
In our implementation, we simply select the best-scored trajectory as our tracking result, but different trajectories may have their own good tracking result clips. How to design an efficient and effective trajectory fusion scheme to achieve better tracking performance is a worthy study problem. The efficiency of our tracker can also be improved by adaptively choosing the number of trajectories. In other words, limited trajectories are needed for simple videos and more trajectories can be employed for challenging videos. We will focus on these two issues in our future works.