{'fraction_non_alphanumeric': 0.03672599649034846, 'fraction_numerical': 0.013474555026322386, 'mean_word_length': 4.9231625835189305, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Keyfilter-Aware Real-Time UAV Object Tracking

Correlation filter-based tracking has been widely applied in unmanned aerial vehicle (UAV) with high efficiency. However, it has two imperfections, i.e., boundary effect and filter corruption. Several methods enlarging the search area can mitigate boundary effect, yet introducing undesired background distraction. Existing frame-by-frame context learning strategies for repressing background distraction nevertheless lower the tracking speed. Inspired by keyframe-based simultaneous localization and mapping, keyfilter is proposed in visual tracking for the first time, in order to handle the above issues efficiently and effectively. Keyfilters generated by periodically selected keyframes learn the context intermittently and are used to restrain the learning of filters, so that 1) context awareness can be transmitted to all the filters via keyfilter restriction, and 2) filter corruption can be repressed. Compared to the state-of-the-art results, our tracker performs better on two challenging benchmarks, with enough speed for UAV real-time applications.


I. INTRODUCTION
Combined with extensibility, autonomy, and maneuverability of unmanned aerial vehicle (UAV), visual object tracking has considerable applications in UAV, e.g., person tracing [1], autonomous landing [2], aerial photography [3], and aircraft tracking [4]. Notwithstanding some progress, UAV tracking remains onerous because of the complex background, frequent appearance variation caused by UAV motion, full/partial occlusion, deformation, as well as illumination changes. Besides, computationally intractable trackers are not deployable onboard UAVs because of the harsh calculation resources and limited power capacity.
Recently, the framework of discriminative correlation filter (DCF) [5], aiming to discriminate the foreground from the background via a correlation filter (CF), is widely adopted in UAV object tracking. The speed is hugely raised because of its utilization of the circulant matrices' property to carry out the otherwise cumbersome calculation in the frequency domain rather than spatial one. Yet the circulant artificial samples used to train the filter hamper the filter's discriminative ability. This problem is called boundary effect because the artificial non-real samples have periodical splicing at the boundary. Several approaches [6]- [13] expand the search area for alleviating boundary effects, but the enlargement Red frames are served as keyframes generating keyfilters. Keyfilters carry out context learning intermittently and influence the current filter training for mitigating filter corruption. Feature of current frame is correlated with the filter trained in the last frame, producing a response map. Red and black rectangles denote respectively the results from KAOT and baseline.
has introduced more context noise, distracting the detection phase especially in situations of similar objects around.
In literature, the context-aware framework [14] is proposed to reduce the context distraction through response repression of the context patches. However, the frame-by-frame context learning is extremely redundant, because the capture frequency of drone camera is generally smaller than the frequency of context variation, e.g., the interval time between two consecutive time in a 30 frame per second (FPS) video is 0.03 second, but generally the context appearance in aerial view remains unchanged for a certain time far more than 0.03 second. In addition, the learned single filter without restriction is prone to corruption due to the omnipresent appearance variations in the aerial scenarios.
In this work, inspired by keyframe-based simultaneous localization and mapping (SLAM) [15], the keyframe technique is used to raise the tracking performance efficiently and effectively. The contributions of this work are two-fold: • A novel application of the keyfilter in UAV visual object tracking is presented. Keyfilters generated at a certain frequency learn the context intermittently and enforce temporal restriction. Through the restriction, the filter corruption in the time span is alleviated and context noise is efficiently suppressed. • Extensive experiments on 193 challenging UAV image sequences have shown that the keyfilter-aware object tracker, i.e., KAOT, has competent performance compared with the state-of-the-art tracking approaches based on DCF and deep neural network (DNN).

II. RELATED WORKS A. Discriminative correlation filter
In recent years, the framework of discriminative correlation filter (DCF) [5] has broadly aroused research interest due to its remarkable efficiency. Yet classic CF-based trackers [16]- [18] have limited performance due to the lack of negative samples, i.e., the circulant artificial samples created to train the CF hugely reduce its discriminative power. One solution to this problem is spatial penalization to punish the filter value at the boundary [6]- [10]. Another solution is cropping both the background and target to use negative samples in the real word instead of synthetic samples [11]- [13]. However, the aforementioned approaches are prone to introduce context distraction because of enlarging search area, especially in the scenarios of similar object around.

B. Prior work to context noise and filter corruption
In literature, M. Mueller et al. [14] proposed to repress the response of context patches, i.e., the features extracted from surrounding context are directly fed into classic DCF framework and their desired responses are suppressed as zero. The context distraction is thus effectively repressed, consequently the discriminative ability of the filter is enhanced. Nevertheless, the frame-by-frame context learning is effective but not efficient, and its redundancy can be significantly reduced. Another problem of classic DCF trackers is that the learned single filter is commonly subjected to corruption because of the frequent appearance variation. Online passive-aggressive learning is incorporated into the DCF framework [19] to mitigate the corruption. Compared to [19], the presented keyfilter performs better in both precision and speed.

C. Tracking by deep neural network
Recently, deep neural network has contributed a lot to the development of computer vision. For visual tracking, some deep trackers [20]- [22] fine-tuning the deep network online for high precision yet run too slow (around 1 fps on a high-end GPU) to use in practice. Other methods like deep reinforcement learning [23], unsupervised learning [24], continues operator [8], end-to-end learning [25] and deep feature representation [26] have also increased the tracking accuracy. Among them, incorporating lightweight deep features into online learned DCF framework has exhibited competitive performance both in precision and efficiency.

D. Tracking for unmanned aerial vehicle
Mechanical vibration, motion blur, limited computation capacity and rapid movement have made UAV tracking an extremely demanding task. In literature, the presented UAVtailored tracking methods generally have lower robustness and accuracy [4], [27]- [29]. In light of offline training on the large-scale image datasets, deep feature for robust representation can improve performance significantly, yet the speed of existing deep-feature based trackers mostly run slow even on a high-end GPU [9]. This work aims to improve the speed and accuracy for the deep feature-based DCF framework for real-time UAV applications.

III. REVIEW OF BACKGROUND-AWARE CORRELATION FILTER
The objective function of background-aware correlation filters (BACF) [12] is as follows: where y ∈ R M , x d ∈ R N and w d ∈ R M denote the desired response, the dth one of D feature channels and correlation filter respectively. λ is a regularization parameter and E(w) refers to an error between the desired response y and the actual one. is the spatial correlation operator. The main idea of BACF is to utilize a cropping matrix B ∈ R M ×N to extract real negative samples. However, more background distraction is introduced because of the enlargement.

IV. KEYFILTER-AWARE OBJECT TRACKER
Inspired by the keyframe technique used in SLAM, the keyfilter is firstly proposed in visual tracking to boost accuracy and efficiency, as illustrated in Fig. 2. The objective function of KAOT tracker is written as follows: where the third term is response repression of context patches (their desired responses are zero), and S p is the score of pth patch to measure the necessity of penalization (introduced in IV-B). w d ∈ R M andw d ∈ R M are the current filter and keyfilter, respectively. γ is the penalty parameter of the gap between w d andw d . To improve the calculation speed, Eq.
(2) is calculated in the frequency domain: where ⊗ is the Kronecker product and I D ∈ R D×D is an identity matrix.ˆdenotes the discrete Fourier transform with orthogonal matrix F.X T = X 0 ,Ŝ 1 X 1 , · · · , S pXP ,
Lagrangian parameter is updated as follows: andŵ * j+1 is obtained through the following formula: subscript j denotes the the value at last iteration and subscript j + 1 denotes the value at current iteration.

B. Context patches scoring scheme
This work adopts a simple but effective scheme for measuring the score of context patches through Euclidean distance. Specifically, the size of omni-directional patches located around the object is the same as that of the object. The score of patch p is calculated as follows: where |OO p | denotes the Euclidean distance between the object and context patch p (p = 1, 2, ..., P ) (between center points) and s is the base score which is a constant number. w, h are respectively the width and height of the object rectangle. Through Eq. (11) , the patch which is closer to object, obtains a higher score for stronger penalization.

C. Keyfilter updating strategy
Starting from the first frame, the keyfilter is generated at a certain frequency using keyframes and current keyfilter refers to the latest trained keyfilter, as shown in Fig. 2. Current filter is restricted by current keyfilter through the punishment introduced by the gap between current filter and keyfilter. In other words, current keyfilter is updated every c frames (c = 8 in this work). When the (n + 1)th keyframe arrives (frame k = c × n + 1), the filter of current frame (keyfilter (n + 1)) is trained under influence from the keyfilter n. As for the non-keyframes after keyfilter (n + 1), the filters of them are learned with the restriction of current keyfilter (keyfilter (n+ 1)). The detailed work-flow of KAOT tracker is presented in Algorithm 1.

A. Implementation details
KAOT adopts both the hand-crafted and deep features, i.e., histogram oriented gradient (HOG) [39], color name (CN) [40] and conv3 layer from VGG-M network [41]. The value of γ is set as 10, and the base score s is set as 0.28. ADMM iteration is set to 2 for raising efficiency. All trackers are implemented in MATLAB R2017a and all the experiments are conducted on the same computer with an i7-8700K processor (3.7GHz), 48GB RAM and NVIDIA GTX 2080 GPU. It is noted that the original codes without any modification are employed in this work for fair comparison.     Figure 3 demonstrates the overall performance of KAOT with other state-of-the-art realtime trackers on DTB70 and UAV123@10fps. On DTB70 dataset, KAOT (0.757) has an advantage of 4.4% and 9.1% over the second and third best tracker ECO (0.722), ARCF (0.694) respectively in precision, along with a gain of 0.2% and 6.6% over the second (ECO, 0.502) and third best tracker (ARCF, 0.472) respectively in AUC. On UAV123@10fps dataset, KAOT (0.686, 0.479) ranks second place followed by the third place UDT+ (0.675, 0.478). ECO is the only tracker performing better than KAOT. Nevertheless, it utilizes continuous operator to fuse the feature maps elaborately, while KAOT just uses the simple BACF as baseline. Notice that ECO can further enhance its performance with our framework. Average precision on the two datasets and speed (evaluated on DTB70) are reported in Table I. KAOT is 27% faster than ECO when achieving higher precision. Discussions: DTB70 [32] dataset is recorded on a drone with more frequent and drastic displacements compared to UAV123@10fps [33], thus increasing the tracking difficulties. Our method exhibits relatively big advantages on DTB70, proving the robustness of our method in the scenarios of strong motion.
2) Attribute-based performance: Precision plots of six challenging attributes are demonstrated in Figure 4. In the cases of background clutter, KAOT improves the ECO by 9.0% in light of the intermittent context learning which can suppress the background distraction effectively. In situations of in-plane rotation and deformation, KAOT has a superiority of 10.2% and 23.0% respectively compared to ECO. This is attributed to the keyfilter restriction, which can prevent the filter from aberrant variation. In addition, KAOT exhibits excellent performance in the scenario of fast camera motion and motion blur, which is desirable in aerial tracking.
3) Qualitative evaluation: Qualitative tracking results on five difficult UAV image sequences are shown in Figure 5. Besides, the respective center location error (CLE) variations of five sequences are visualized in Figure 6. Specifically, in ChasingDrones sequence where tracking is bothered by strong UAV motion, KAOT has effectively repressed the distraction of the context, so it can perform well despite the large movement in a certain complex context. Only the pre-trained UDT+ tracks successfully in addition to KAOT. Motion blur occurs in sequences RcCar6 and Gull1 (severe example is shown at frame 16 in RcCar6). In this situation, KAOT has kept tracking owing to the mitigated filter corruption. As for the last two sequences, keyfilter restriction and intermittent context learning have collaboratively contributed to successful tracking.

C. Comparison with non-real-time trackers
KAOT is also compared with five non-real-time trackers using deep neural network, as shown in Table II. To sum up, KAOT has the best performance in terms of both precision and speed on two benchmarks. In addition, compared to DeepSTRCF (using the same features as KAOT), our tracker has more robust performance in precision on both two datasets and is around 2.4 times faster than it. Therefore, the efficiency and accuracy of KAOT tracker can be proven.

D. Limitations and future works
Keyframe selection: This work only adopts a simple periodic keyframe selection mechanism, which is possible to introduce distraction when the tracking on the keyframes is not reliable. More elaborated strategy can be employed to adaptively choose the keyframe and further enhance the robustness. Re-detection and rotation: Though KAOT performs favorably in the situations of drastic appearance change like blur,  deformation, etc., it is still limited when the object disappear for a long time. Also, KAOT can not handle the rotation situations. Thus the re-detection and rotation-aware modules can be added to raise the performance. Speed: The speed of KAOT is around 15 fps with a GPU and can be used in real-time applications. However, KAOT tracker is implemented on MATLAB platform and the code is not optimized, so the speed can be further improved.

VI. CONCLUSIONS
This work proposes keyfilter-aware object tracker to repress the filter corruption and lower the redundancy of context learning. Extensive experiments on two authoritative datasets have validated our tracker performs favorably in precision, with enough speed for real-time applications. This keyfilter-aware framework and intermittent context learning strategy can also be used in other trackers like C-COT [7] and STRCF [19] to further boost their performance. We strongly believe that our method can be used in practice and promote the development of UAV tracking applications.