{'fraction_non_alphanumeric': 0.037855418454935626, 'fraction_numerical': 0.013747317596566524, 'mean_word_length': 4.73447413958854, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Multi-Drone based Single Object Tracking with Agent Sharing Network

Drone equipped with cameras can dynamically track the target in the air from a broader view compared with static cameras or moving sensors over the ground. However, it is still challenging to accurately track the target using a single drone due to several factors such as appearance variations and severe occlusions. In this paper, we collect a new Multi-Drone single Object Tracking (MDOT) dataset that consists of 92 groups of video clips with 113,918 high resolution frames taken by two drones and 63 groups of video clips with 145,875 high resolution frames taken by three drones. Besides, two evaluation metrics are specially designed for multi-drone single object tracking, i.e. automatic fusion score (AFS) and ideal fusion score (IFS). Moreover, an agent sharing network (ASNet) is proposed by self-supervised template sharing and view-aware fusion of the target from multiple drones, which can improve the tracking accuracy significantly compared with single drone tracking. Extensive experiments on MDOT show that our ASNet significantly outperforms recent state-of-the-art trackers.


I. INTRODUCTION
Drone or general unmanned aerial vehicle (UAV) is widely used in our daily life. Specifically, the application scenarios of drone based visual tracking cover live broadcast, military battlefield, criminal investigation, sports and entertainment [1], [2], [3]. Compared with static cameras and handheld mobile devices, drones can dynamically move and cover a wide ground area, which is very suitable to track fast-moving targets.
To perform robust tracking on drones, large-scale datasets with high-quality annotations play a critical role to promote the development of algorithms. Recently, several benchmark datasets have been collected with a single drone, including UAV123 [1], Campus [4], VisDrone-2018 [2], and UAVDT [3]. As shown in Figure 1, compared with static cameras that only collect data in a certain area, the drone can also dynamically track the target in the air from a broad view. However, it also brings additional challenges to visual tracking, including tiny targets, camera motion, high density distribution of targets, etc.
To solve the above issue, incorporating multiple drones is an effective solution to improve the performance and robustness of object tracking to occlusion and appearance ambiguities. Thus, several algorithms focus on long-term tracking and reidentification based on multiple cameras in video surveillance [5], [6], [7], [8], [9], [10], [11], [12], [13]. In the past few years, a few multi-camera benchmark datasets have been constructed with overlapping or non-overlapping views of cameras [14], [15], [11]. Some datasets with fully overlapped views are constrained to short time intervals and controlled conditions [14]. These multi-camera datasets are specially collected for multi-object tracking or person re-identification across cameras.
Although many datasets are provided for visual tracking, they are built for either single drone tracking or multi-camera tracking. However, there are few benchmark datasets for multidrone visual tracking. In this paper, to combine the advantages of both drone based tracking and multi-camera tracking, we present a multi-drone single object tracking (MDOT) dataset. MDOT consists of 92 groups of video clips with 113, 918 high resolution frames taken by two drones and 63 groups of video clips with 145, 875 high resolution frames taken by three drones. In each group of video clips, the same target is tracked by multiple drones. Moreover, we annotate 10 different types of attributes, including daylight, night, camera motion, partial occlusion, full occlusion, out of view, similar object, viewpoint change, illumination variation, and low resolution. To evaluate tracking algorithms in our dataset, we propose two new evaluation metrics, i.e., adaptive fusion score (AFS) and ideal fusion score (IFS). Specifically, AFS measures the performance of multi-drone tracker that fuses the tracking results using the online fusing strategy, while IFS is the ideal fusion performance when we assume that the multi-drone system can accurately select the tracking results of the drone with better performance. On the other hand, to use multidrone complementarity, we propose an agent sharing network (ASNet), which shares templates across drones in a selfsupervised manner and fuses the tracking results automatically for robust and accurate visual tracking. A re-detection strategy on drone based tracker is proposed to deal with drift of targets by enlarging the size of search region when the target is judged to satisfy the defined condition. Experiments on MDOT demonstrate that ASNet greatly outperforms 20 recent stateof-the-art tracking algorithms.
The contribution of this paper can be summarized as follows.
• To the best of our knowledge, we propose the first multidrone single object tracking dataset (MDOT) which consists of total 155 groups of video clips with 259, 793 high resolution frames and rich annotations. • Two new evaluation metrics are designed for multidrone single object tracking, i.e., automatic fusion score (AFS) and ideal fusion score (IFS). AFS evaluates the performance of multi-drone tracker that fuses the tracking results using the learned weights while IFS is the theoretically optimal fusion performance (upper bound) of a multi-drone tracker. IFS is specially proposed to inspire researchers to design superior multi-drone tracker with more useful fusion strategies. • The agent sharing network (ASNet) is proposed for perform multi-drone visual tracking, which effectively exploits multi-drone shared information in a view-aware manner. ASNet can be considered as a baseline tracker for the multi-drone based single object tracking task.

II. RELATED WORK
A. Single Object Tracking.
In the field of visual tracking, single object tracking has achieved massive attention. Generally, single object trackers can be categorized into generative models and discriminative ones. Generative models search for the most similar area to the template of the previous frames, e..g, Kalman filtering and mean-shift [16]. Discriminative models treat visual tracking as a binary classification task, which distinguish the target from the background, e.g., Struck [17] and TLD [18].
Inspired by the success of deep learning in image classification and object detection, deep trackers have achieved superior performances. MDNet learns an end-to-end deep tracker upon convolutional neural network by a video-specific fully connected layer [19]. Siam-FC exploits a siamese network by learning the feature maps of both the target and the search region, and using a convolutional operation to obtain the response map [20]. Recently, some successful techniques in object detection, e.g., region proposal network (RPN), are embedded into object tracking models. SiamRPN and its variants (DaSiamRPN [21], SiamRPN++ [22] and C-RPN [23]), are proposed by using more powerful network structure as backbone and attractive blocks to learn feature maps with better representation abilities. Discriminative correlation filter (DCF) can be learned very efficiently in the frequency domain via fast Fourier transform and therefore achieves very impressive tracking efficiency [24], [25], [26], [27], [28], [29], [30], [31]. To cope with the severe variations, discriminative feature representation, nonlinear kernel, scale estimation, spatial regularization, continuous convolution, spatial-temporal regularization are introduced to pursue a balance between accuracy and speed for correlation filter based trackers. The performance of single object tracking is easily affected by severe appearance variations, occlusions, and out-of view cases, which could be solved by using multiple cameras.

B. Multi-Camera Tracking.
Multi-camera tracking uses information of different views by estimating a common axis or subspace, or fusing multiview information, to improve the robustness of trackers to occlusion, drift and other variations. Most existing works focus on multi-object tracking, especially multi-person detection and tracking in overlapping views or across non-overlapping views [6], [9], [11], [7], [13]. The single-camera trajectories can be previously given or obtained by pedestrian detection and tracking. As multiple cameras are static, the spatial relations between cameras are either explicitly mapped in 3D, learned by tracking known identities, or obtained by comparing entry/exit rates across pairs of cameras. The multicamera information can be fused in different stages. Singlecamera tracking is first performed in each camera to create trajectories of multiple targets, and then inter-camera tracking is carried out to associate the tracks [13]. Trackers exploit optimization to maximize the coherence of observations for predicted identities. The spatial, temporal and appearance information of trajectories is used to construct an affinity  matrix. The nodes are then partitioned into different identities by bipartite matching or maximal internal weights.

C. Datasets
The recent progresses in visual tracking rely on scarcelyavailable large scale tracking datasets with high-quality annotations to a great extent. There have been significant growth of the volume and diversity of benchmark datasets for visual tracking, e.g., TC-128 [34], OTB-2015 [33], NUS-PRO [40], VOT2016 [35], LaSOT [41], and TrackingNet [42]). Most datasets focus on single camera and single object tracking. For drone based visual tracking, several benchmark datasets have been collected with a single drone, including UAV123 [1], Campus [4], VisDrone-2018 [2], and UAVDT [3]. UAV123 dataset contains a total of 123 video sequences and more than 110K frames for single object tracking [1]. Campus dataset includes 929.5k frames which contains various types of objects [4]. VisDrone-2018 dataset consists of 263 video clips formed by 179,264 frames and 10,209 static images from 14 different cities for object detection and tracking in both images and videos [2]. UAVDT dataset consists of 80,000 representative frames with bounding boxes as well as up to 14 kinds of attributes from 10 hours raw videos for object detection and tracking in videos [3]. To track and identify multiple objects across different cameras, a few multi-camera benchmark datasets are collected for multi-object tracking and person re-identification. NLPR-MCT consists of four subsets with at most 255 IDs with 3 to 5 non-overlapping cameras [39]. DukeMTMC is the largest multi-camera multi-object tracking dataset that consists of videos of 2834 IDs and 8 cameras in the outdoor scene with both overlapping views and blind spots [11]. All trajectories are manually annotated and identities are associated across cameras. The existing datasets are collected either for single object tracking or multicamera tracking using static sensors. In this work, we collect a benchmark dataset using multiple drones, which is an effective supplement to the existing datasets.

III. MULTI-DRONE SINGLE OBJECT TRACKING DATASET
In this section, we present the collected benchmark dataset (MDOT) and evaluation metrics for multi-drone single object tracking.

A. Data Collection
Our dataset is collected by multiple DJI PHANTOM 4Pro drones. Specifically, the drones are controlled by several professional human operators from different altitudes in various outdoor scenes (e.g., park, campus, square, and street), as illustrated in Figure 2. In order to increase the targets' diversity of appearances and scales, the same target is tracked by multiple drones from different view-angles and different altitudes, ranging from 20m to 100m. The dataset has 155 groups of video clips with 259, 793 high resolution frames in two sub-datasets. The two-drone based dataset (Two-MDOT) consists of 92 groups of video clips with 113, 918 high resolution frames taken by two drones, while the three-drone based dataset (Three-MDOT) contains 63 groups of video clips with 145, 875 high resolution frames taken by three drones. Two-MDOT was collected in 2018 while Three-MDOT was collected in 2019. Hence, there is no overlap between Two-MDOT and Three-MDOT. Besides, the dataset is divided into train set (37 groups in Two-MDOT and 28 groups of in Three-MDOT) and the test set (55 groups in Two-MDOT and 35 groups of in Three-MDOT).
As presented in Table I, most of the previous datasets are collected by one camera where the appearance of targets is not abundant. Although NLPR MCT and DukeMTMC are used for evaluating multi-target tracking and person reidentification, they are collected by static cameras. In comparison, MDOT can dynamically track the targets with moving drones (see Figure 3). Note that we do not collect the lidar data because several drones with lidar are much expensive compared with visible light camera equipped drones and the accurate sensing distance of lidar is about 200m, which does not show obvious advantage compared with visible light cameras.

B. Annotation
For annotation, we collect the images with the size of 1280×720 and use the commonly used annotation tool VATIC [43] to annotate the location, occlusion and out of view information of targets. After that, LabelMe [44] is used to refine and double-check the annotations frame-by-frame. Moreover, the targets in 155 sequences are divided into 9 categories, i.e., pedestrian, car, carriage, motor, bicycle, tricycle, truck, dog, bus, and the targets in each category are also diverse. Moreover, as shown in Table II, all the sequences are  The statistics with respect to 10 attributes are summarized in Figure 4. Notice that CM, IV and LR occur in most sequences, which may significantly degrade the performance of trackers. Similar to the setting of the classic single object tracking task, we manually annotate the tracking target in the first frame across different drones with respect to the same object.

C. Evaluation Metrics
Single object tracking is usually evaluated by success and precision plots [33]. However, for multi-drone based tracking, the results of the algorithms should be evaluated upon the multi-drone fusion results. To this end, we propose two new metrics for multi-drone single object tracking, i.e., automatic fusion score (AFS) and ideal fusion score (IFS).
• Automatic fusion score evaluates the performance of multi-drone tracker that fuses the tracking results using the online fusing strategy. Definition 1: Let h v i and y v i be the tracking result (i.e., location, width and height of the box) and ground truth of the i-th frame and v-th drone. AFS is defined as where s(·, ·) is a evaluation metric for single object tracking (i.e., success and precision scores) and w v is the weight for the v-th drone. n and V is the number of frames in a video clip and the number of drones, respectively. The value of w v should be zero or one. w v is automatically learned and online updated for each frame during the tracking process. • Ideal fusion score is the ideal fusion performance when we assume that the multi-drone system can accurately select the tracking results of the drone with better performance. It is defined to evaluate the extreme performance of a multi-drone tracking system, which can guide the design of a superior multi-drone tracker. Definition 2: Let h v i and y v i be the tracking result and ground truth of the i-th frame and v-th drone. IFS is defined as During the evaluation stage, OPE metrics (Success/Precision) are the traditional metrics for single object tracker. Based on the OPE, AFS and IFS are proposed for multi-drone tracker. Compared with AFS, IFS shows that there is still a gap from the upper bound of multi-drone  fusion, which inspires us to design superior multi-drone tracker with more useful fusion strategies.

IV. AGENT SHARING NETWORK
The key challenge of multi-drone tracking is how to share the inter-drone information and adaptively fuse the tracking results. To deal with this challenge, each drone is considered as an agent and we propose an agent sharing network (ASNet) for multi-drone tracking, which can effectively exploit the interagent complementary information, see Figure 5.

A. Network Architecture
Dynamic siamese network (DSiam) [45] can enable effective online learning of target appearance variation and background suppression from previous frames. Therefore, we choose DSiam as the base tracker and develop the corresponding multi-drone tracker. A common tracker is trained for all drones in that all drones track the same target in the same scene. Therefore, there is no bias toward different drones. We focus on the online tracking process and design the agent sharing network from three aspects, i.e., template sharing, view-aware fusion and target re-detection.
Let O v 1 and Z v t denote the templates of the first frame and search regions of the t-th frame with respect to the v-th drone, respectively. By an embedding block f , e.g., convolutional neural network (CNN), deep features can be extracted for both the templates and search regions, i.e., f (O v 1 ), and f (Z v t ). The key components of Dsiam are the target appearance variation transformation and background suppression transformation [45]. For ASNet, we need to determine the transformation for all drones. The target appearance variation transformation M v t−1 with respect to the v-th drone is learned by where is the tracked target of the t − 1-th frame for the v-th drone. ⊗ denotes the circular convolution, which can computed rapidly in the frequency domain [45]. M v t−1 can capture the target variation under temporal smoothness assumption and therefore contributes greatly to online learning. Similarly, the background suppression transformation W v t−1 can be learned.
where G v t−1 is the region centering at the target with the same size of Z v t−1 .G v t−1 is obtained by multiplying G v t−1 with a Gaussian weight map. W t−1 can suppress the background information and therefore induces superior tracking performance. More details about the solution of M v t−1 and W t−1 can be found in [45].
Compared with visual tracking using single drone, ASNet shares the templates of all drones, and obtains the response maps corresponding to the templates of multiple drones. As the reliability of templates of multiple drones is different, we adaptively fuse the response maps of multiple-templates in a self-supervised manner. Finally, the tracking results of multiple drones can be adaptively fused by tracking scores.

B. Self-supervised Template Sharing
As the appearance of the target may vary greatly, the templates of all drones can be shared to improve the tracker robustness of single drone. The response map S kv t of the tracker on the k-th drone, corresponding to the template of the v-th drone is calculated as where corr (A, B) is the correlation operation, which can be considered as a convolution operation on B with A as the convolution filter. For the k-th drone, we obtain a set of response maps, S k1 t , S k2 t , · · · , S kV t . To fuse the response maps, we propose a self-supervised fusion strategy. Specifically, we use the tracking results of the previous t − 1 frames as the supervised information to guide the weights learning of V templates. Let O kv t denote the tracked target of the tracker on the k-th drone using the template of the v-th drone. The fusion weights u k t can be learned by where The weights reflect the correlation between tracked target and target template of the (t − 1)-th frame. Given the learned weights and response maps, we can obtain the fused response map for the tracker on the k-th drone, i.e., For a multi-drone tracking system with V drones, we can obtain V fused response maps in total, i.e., S 1 t , S 2 t , · · · , S V t .

C. View-aware Fusion
To generate the final results on the multi-drone tracking, we use the auto view-aware fusion scheme when we obtain the V fused response maps. For the k-th drone's response map, we search the maximum value g(s k t ) in the response map, and obtain its respective location p k t . g(s k t ) is defined as tracking score with respect to the t-th frame on the k-th tracker. Then, we can obtain the index of the best response map b t by b t = arg max g(s 1 t ), g(s 2 t ), · · · , g(s v t ) .
The respective location p bt t in the b t -th drone is the position of the target. The weight w v of the drone tracker with the best response map obtained from Eq. 8 is set as one and the rest as zero.

D. Target Re-detection
As camera motion often occurs in the drone based tracking, the target location may vary dramatically in successive frames. To solve the problem, we use the target re-detection strategy based on the past and current frames. For the t-th frame, let l q t denote a set of scores for the past q frames. µ n t and σ q t denote mean and standard deviation of l q t . Inspired by the peak to sidelobe ration in MOSSE [46], the threshold for target redetection is defined as where λ is a pre-set parameter. The target may be lost when the score is less than ω q t or the tracking score is the threshold T score . If so, we use the local-to-global strategy to expand the search region step by step to re-detect the target [21]. After using the proposed re-detection strategy, the tracking performance is greatly improved.

V. EXPERIMENTS
We evaluate our method compared with 20 recent state-ofthe-art single object tracking algorithms on the machine with a E5-2620 v3 CPU and a NVIDIA TITAN Xp GPU. Note that the existing multi-camera tracking methods, are specially designed for multi-object tracking and therefore cannot apply to multi-drone single object tracking directly. The source codes for other algorithms are from the authors.

A. Overall Performance
We report the success and precision scores, tracking speed and the references of each algorithm in Table III and Table  IV. The tracking performance on each drone are reported using the baseline single object trackers. Besides, the overall performance of the baseline trackers is given by calculating success and precision scores of all drones together. Note that for the proposed ASNet, we report success and precision plots using AFS in Definition 1 in terms of multi-drone tracking. As shown in Table III and Table IV, GFSDCF achieves the best precision score of 59.5 on Two-MDOT and 65.6 on Three-MDOT. Following the GFSDCF tracker, other correlation filter trackers also obtain the great performance in precision score and success score, e.g., ECO, STRCF, CSRDCF. Besides, due to extensive offline training, siamese tracking approaches DSiam, SiamRPN++ and SiameseFC show the top performance as well. Our proposed ASNet significantly outperforms the baseline trackers on all sub-datasets. Compared with the best baseline tracker, the precision scores are improved by 14.8 on Two-MDOT and 12.6 on Three-MDOT, respectively. The results show that compared with tracking using only one drone, multi-drone tracking using our proposed ASNet greatly boost the tracking performance, which validates the necessity and effectiveness of multi-drone tracking. The significant improvement comes from the fusion of complementary information across multi-agents in case of great appearance variations.
To further analyze the performance, we report the overall performance of the proposed ASNet and 20 compared state-ofthe-art trackers in Figure 6. Notably, the success and precision plots of ASNet are drawn based on the AFS metric defined in our paper. ASNet chieves the best performance on the proposed dataset, i.e., 48.2 success score on the Two-MDOT subset and 53.3 success score on the Three-MDOT subset. This is because of the fusion of complementary information across multi-agents in case of great appearance variations  in our framework. We can conclude that SiamRPN++ [22] and GFSDCF [31] rank the second and third place compared with other methods in terms of success score, respectively. Following the above three trackers, the siamese network based tracker DSiam [45] and correlation filters based tracker ECO [47] obtain slight inferior performance in both success and precision scores.

B. Attribute-based Performance
To further analyze the performance, we report the success scores of algorithms over 10 attributes in Figure 8. It can be concluded that the performances on attributes CM, FOC, VC and IV are inferior than that on other attributes. This is maybe because the target appearance is heavily changed in these situations. We can observe that the performance of DSiam on attributes CM, SO, VC are far ahead of other methods. For other attributes, DSiam achieves the best performance on the test set, while the gap between DSiam and the best competitor ECO on the other UAV based datasets is small. Moreover, our proposed tracker ASNet achieves the best performance on all attributes. There is a significant gap between ASNet and other compared trackers, which owes to the discriminative appearance information from template sharing and re-detection strategy. Figure 9 shows the precision plots of compared tracking algorithms over 10 challenging visual attributes. It can be concluded that the performances on attributes CM, FOC, VC and IV are inferior than that on other attributes. This is maybe because the target appearance is heavily changed in these situations. Moreover, our proposed tracker ASNet achieves the best performance on all attributes. There is a significant gap between ASNet and other compared trackers, which is attributed to the discriminative appearance representation from template sharing and re-detection strategy in our method. Following our method, DSiam, SiamRPN++ and GFSDCF achieve good performance in most attributes, much better than the remain compared tracking methods.

C. Ideal Fusion Score
To investigate the ideal performance of a multi-drone tracking system, as shown in Figure 7, we report the success and precision plots of the existing trackers with IFS metric on MDOT test set. Note that there is a big difference in the ideal fusion performance of different trackers. Similar to ensemble learning techniques, IFS is up to the performance of the base tracker on each drone. If the tracking results of the baseline tracker, e.g., DSiam, can be ideally fused with the precision of 79.5 on Two-MDOT and 83.2 on Three-MDOT, respectively. For any single object tracker, IFS can be used to guide the design of a multi-drone tracker based on the base tracker. As the tracking mechanism of the base trackers are different, more generalized or base tracker-specific fusion strategies are expected to be designed by the research community to boost the performance of multi-drone tracking.

D. Ablation Study
As shown in Table V, we analyze the importance of each component in ASNet on our MDOT dataset, i.e., re-detection, template sharing and view-aware fusion. Specifically, we use DSiam as the base tracker of ASNet and add re-detection, template sharing, view-aware fusion. Re-detection. We first add the re-detection component in the DSiam tracker. In this module, the hyperparameter λ is set to 2 in Two-MDOT and 1.25 in Three-MDOT. As shown in Table  V (2), the re-detection module can improve precision score of 2.8/2.1 and success score of 1.5/1.2 on Two-MDOT and Three-MDOT respectively. It indicates that the re-detection module can decrease the possibility of target drifting especially for long-term tracking. Template Sharing.   score (0.6/0.6). When we combine target re-detection and template sharing, the precision score and success score are further improved. View-aware Fusion. As presented in Table V (5), if we only apply the view-aware fusion in the DSiam tracker, the performance is greatly improved. Then we take the view-aware fusion strategy into account, consistent improvements can be achieved on both drones, as shown in Table V (6, 7). Finally, we add the re-detection module, template sharing design and view-aware fusion scheme to the DSiam tracker (ASNet). Thus we can observe considerable improvement in precision score (14.9/14.1) and success score (10.6/10.4).

E. Discussions
As there exists exchange of information across drones, we need to take the impact of synchronization and latency into account. Similar to the setting of multi-camera tasks, we assume that videos of multiple drones are synchronous by starting tracking across drones simultaneously. Re-detection is conducted on each drone separately while view-aware fusion only needs a tracking score value per drone. Hence, the only component of ASNet, i.e., template Sharing, is affected by communication latency. For ASNet, even if the template sharing strategy is not adopted, we can still get a superior performance, as shown in Table V (7). In real-world applications, with the development of communication network technology, if the communication latency can be ignored, we can exploit the sharing of vision information across drones more effectively.
VI. CONCLUSIONS In this paper, we present a new multi-drone single-object tracking (MDOT) benchmark dataset for the object tracking community. MDOT is an unique platform for developing drone based tracking algorithms and multi-drone tracking systems. Moreover, Two evaluation metrics, i.e., adaptive fusion score (AFS) and ideal fusion score (IFS) are proposed for multidrone single object tracking. To exploit the complementary information across drones, an agent sharing network (ASNet) is proposed by sharing inter-drone templates, fusing multidrone tracking results and re-detecting the targets. Extensive experiments on MDOT show that ASNet outperforms the state-of-the-art single object trackers, which validates the effectiveness of multi-drone tracking.