{'fraction_non_alphanumeric': 0.03914824285487995, 'fraction_numerical': 0.01574981331040257, 'mean_word_length': 4.84395662523142, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Learning Spatial-Frequency Transformer for Visual Object Tracking

Recent trackers adopt the Transformer to combine or replace the widely used ResNet as their new backbone network. Although their trackers work well in regular scenarios, however, they simply flatten the 2D features into a sequence to better match the Transformer. We believe these operations ignore the spatial prior of the target object which may lead to sub-optimal results only. In addition, many works demonstrate that self-attention is actually a low-pass filter, which is independent of input features or key/queries. That is to say, it may suppress the high-frequency component of the input features and preserve or even amplify the low-frequency information. To handle these issues, in this paper, we propose a unified Spatial-Frequency Transformer that models the Gaussian spatial Prior and High-frequency emphasis Attention (GPHA) simultaneously. To be specific, Gaussian spatial prior is generated using dual Multi-Layer Perceptrons (MLPs) and injected into the similarity matrix produced by multiplying Query and Key features in self-attention. The output will be fed into a Softmax layer and then decomposed into two components, i.e., the direct signal and high-frequency signal. The low- and high-pass branches are rescaled and combined to achieve all-pass, therefore, the high-frequency features will be protected well in stacked self-attention layers. We further integrate the Spatial-Frequency Transformer into the Siamese tracking framework and propose a novel tracking algorithm, termed SFTransT. The cross-scale fusion based SwinTransformer is adopted as the backbone, and also a multi-head cross-attention module is used to boost the interaction between search and template features. The output will be fed into the tracking head for target localization. Extensive experiments on both short-term and long-term tracking benchmarks all demonstrate the effectiveness of our proposed framework.

. Comparison of the activation maps estimated by our proposed Spatial-Frequency Transformer and existing state-of-the-art Transformer-based visual tracker TransT [16]. Note that, our model pays more attention to the edge position and specific target components (e.g. head and tail), rather than the less discriminative structures (e.g. trunk). the position of the target object in the continuous video frames based on initialized bounding box in the first frame. It has been widely used in many practical applications, such as unmanned aerial vehicles tracking [1], [2], multimodel tracking [3], [4], military field, etc. With the help of deep learning techniques [5], [6], many end-to-end visual trackers are proposed, such as the multi-domain based tracker [7] and Siamese matching-based tracking [8]- [10]. Although these trackers achieve high performance on existing benchmark datasets, their performance is still limited when facing challenging factors like dramatic deformation, heavy or full occlusion, fast motion, etc.
To address the aforementioned issues, current researchers attempt to design powerful neural networks for robust feature representation learning. The early works like the SiamFC [9] and SiamRPN [11] pave new ways for Siamese matchingbased tracking. The key step is calculating the similarity between the template and the search branch. Usually, the ResNet [12] are adopted as the backbone for feature extraction and correlation computation. More and more Siamese trackers [10], [11], [13] are proposed under this framework, but their tracking performance is still poor in challenging scenarios due to the utilization of convolutional layers. Many works demonstrate that the limited receptive field of convolutional kernels may bring sub-optimal results only [14], [15].
Recently, the Transformer [16] is becoming a hot research arXiv:2208.08829v1 [cs.CV] 18 Aug 2022 topic in both Natural Language Processing (NLP) [17] and Computer Vision (CV) [14]. Its core module is termed selfattention, which models the long-range relations well. Some researchers already have simply exploited the combination of Transformer and CNN networks, such as TransT [16], TrDIMP [18], STARK [19], and obtained better results than full CNN-based trackers. These works mainly focus on fusing the template and search regions using Transformer layers. They simply flatten the two-dimensional feature maps into one-dimensional feature sequences and calculate the attention scores between each patch. We think such 2D to 1D transformation may lose the spatial prior information of the target object. Therefore, how to retain the spatial prior of the target object when learning the long-range relations using Transformer is an urgent problem to be solved. On the other hand, more and more works [20], [21] demonstrate that the self-attention mechanism is actually a low-pass filter, which is independent of input features or key/queries. In other words, the Transformer may suppress the high-frequency component of the input features and preserve or even amplifies the lowfrequency information. It is natural to raise the following question: how can we complement the spatial prior and the high-frequency component of the target sequence when using the deeply stacked multi-head self-attention modules? With this question in mind, in this paper, we design a novel spatial-frequency Transformer framework for visual object tracking, termed SFTransT. Generally speaking, our proposed SFTransT follows the Siamese tracking framework, and it contains four main parts, including the cross-scale fusion-based SwinTransformer, Multi-Head Cross-Attention (MHCA) module, Spatial-Frequency Transformer (SFFormer), and tracking head. Specifically, the SwinTransformer is used as the backbone network to extract the feature representations of template and search regions. Note that, cross-scale features are encoded and fused for more accurate representation. Then, we feed the backbone features into the MHCA module for interactive learning of template and search features. The output will be fed into our newly proposed spatial-frequency Transformer, which considers both Gaussian prior and low-/high-frequency feature information, and averts the unbalanced response in different frequency signals. Finally, the tracking head is adopted to directly transform the enhanced features into target localization, without the widely used correlation operators. An overview of our proposed SFTransT is illustrated in Fig. 2.
To sum up, we draw the contributions of this paper in the following three aspects: • We analyze the limitations of current Transformer based trackers and propose a novel spatial-frequency Transformer for visual object tracking. It simultaneously models the spatial prior and frequency information for more robust feature representation learning. • We propose a cross-scale fusion-based SwinTransformer as the shared backbone network for tracking, which makes our model a fully Transformer-based visual tracking framework. • Extensive experiments are conducted on nine largescale and popular tracking benchmark datasets, includ-ing LaSOT, TNL2K, TrackingNet, GOT-10K, etc. These experimental results fully validate the effectiveness of our proposed module for tracking, and also demonstrate that our tracker achieves state-of-the-art performance, for example, it obtains an SR of 69.1% on LaSOT [22] and 72.7% on GOT-10K [23].

II. RELATED WORK
In this section, we will give a brief review of the Siamesebased trackers, Transformer based trackers, and Gaussian prior utilized in deep learning. More related works can be found in the following survey papers [24], [25] and paper list 1 . Siamese based Tracking. With the release of SiamFC [9] which formulated the visual tracking as a similarity matching problem between the template and search region, many subsequent trackers have been developed to gain performance improvement. SiamRPN [11] introduced RPN [26] structure to predict object scale with the variation of aspect ratio. SiamRPN++ [8] utilized the deeper network ResNet-50 [12] as the parameter-shared backbone and took the representation ability of tracking framework to new levels. STMtrack [27] followed the Siamese framework but did not share the backbone of the template and search branch at all. Wang et al. proposed the adversarial positive instance generation for visual tracking [28]. With the object detection and segmentation development, some trackers [10], [13], [29]- [31] introduce anchor-free prediction head to cut down the heuristic parameters. SiamR-CNN [32] joined Faster R-CNN [26] with Siamese framework, realizing satisfying robustness in scale change and potential distractor object. Pi et al. [33] introduced an instance-based feature pyramid to achieve adaptive feature fusion. Zhang et al. [34] designed a target-distractor-aware model with discriminative enhancement learning loss to learn target representation.
The above Siamese trackers have achieved gratifying performance in attributions of motion blur, deformation, etc, but are still easily affected by similar distractors and heavy occlusion which suppress the robustness of trackers. To solve these issues, some trackers introduced online learning approaches. Martin et al. [35]- [37] collected the historical hard negative samples to accelerate the discriminate feature learning process and verified the effectiveness of discriminate online tracking. The authors of KeepTrack [38] introduced a learned association network to discriminate distractors for continuing robustness tracking. UpdateNet [39] broke the linear template update strategies and proposed the update network to learn the nonlinear fusion template. TANet [40] adopted a joint local-global search scheme for robust tracking which works well in long-term videos. Inspired by these trackers, our proposed SFTransT also follows the Siamese tracking framework. Nevertheless, our model adopts cross-scale fusionbased SwinTransformer [15] as the backbone and obtains more robust features for visual object tracking. Transformer based Tracking. Vision Transformer (ViT) firstly introduced Transformer [17] into image classification [14] and achieved competitive performance with CNNs.
It divided the input images into local patches and flattened feature maps into patch sequences. Motivated by the development of vision Transformers, traditional tracking approaches combined the CNN networks with Transformer components. TransT [16] first introduced the Transformer structure into the tracking field and utilized the self-attention and cross-attention of DETR [41] to enhance the ego-context and cross-feature. TrDiMP [18] considered the encoder and decoder network into parallel branches to explore the temporal contexts across the two branches. STARK [19] followed the encoder-decoder framework and combined the template with search branch into one branch after feature extraction. In this way, STARK removed the cross-fusion operations in Siamese trackers and kept the tracking framework similar to detection. DTT [42] similarly kept encoder-decoder Transformer architecture and dense prediction to build a discriminative tracker. ToMP [43] proposed a Transformer-based model prediction module to replace the traditional optimization-based predictor and capture global relations between frames. Cross-modality Transformer (CMT) was proposed in work [44] to achieve RGB-Event fusion-based tracking. Nie et al. [45] generate the fine-grained prior knowledge with explicit representation for template target.
However, these Transformer based trackers are all limited by the context-representation defect of CNN backbone [12], [46]. With the release of Swin-Tranformer [15], SwinTrack [47] evolved the backbone from CNN [12] into Transformer to learn much superior feature extraction and representation capability. CSWinTT [48] further evolved the attention from pixel-to-pixel to cyclic shifting window level. Moreover, SBT [49] proposed the single branch network independent of the Siamese framework, allowing the template and search features to interact deeply at each stage of feature extraction. MixFormer [50] further proposed the one-stage one-branch structure and utilized the CVT [51] for feature extraction and information integration. Although existing Transformer driven trackers achieve impressive performance, they may obtain sub-optimal results only due to the ignorance of spatial prior and low-/high-frequency feature learning. In contrast, our proposed SFTransT tracker considers the two aspects in a unified tracking framework, which bring us a higher performance on multiple benchmark datasets. Gaussian Prior in Deep Learning. Gaussian distribution, also called normal distribution, is widely used in machine learning. Most machine learning models are based on the assumption that data obey Gaussian distribution, such as naive Bayesian classification and GMM [52]. That is to say, the probability of being located near centre is always higher than the position probability of edge position. For natural language generation, D2GPo [53] proposed to add the 1D datadependent Gaussian prior objective into maximum likelihood estimation of context, considering the Gaussian prior distribution as the similarity between prediction words. Guo et al. [54] introduced 1D Gaussian prior to self-attention mechanism for language inference, modelling the local structure of sentences. Additionally, it proves that the 1D Gaussian prior probability can correct the distance to the central word.
For the computer vision tasks, NWD [55] utilized the spatial characteristic and modelled the 2D Gaussian distribution as the bounding box prior knowledge to compute the feature similarity by their corresponding Gaussian distributions. With similarity motivation, Yang et al. [56] converted the rotated bounding box into 2D Gaussian distribution and proposed the Gaussian Wasserstein Distance (GWD) to approximate the rotational bounding box. Above detection and language generate works all consider the Gaussian prior as the probability density function distribution. Unlike the aforementioned language and detection works, DRAW [57] applied a Gaussian filter to yield smoothly varying image patches for image generation. All these works inspired us to design novel Gaussian prior for robust visual object tracking.

III. METHODOLOGY
In this section, we will first give an overview of our proposed tracker in Section III-A. Then, we delve into the details of our spatial-frequency Transformer tracking framework, which contains four main modules: feature extraction module, interactive learning module, spatial-frequency Transformer module, and tracking head. After that, we will talk about the details of training and testing of our tracker in Section III-E.

A. Overview
In this work, we propose a novel visual tracker based on Siamese matching framework, termed SFTransT, as shown in Fig. 2. Given the template and search frame, we design a powerful backbone network for the feature extraction based on cross-scale fusion-based SwinTransformer [15]. After obtaining the backbone features, we introduce MHCA (Multi-Head Cross-Attention) modules to boost the interactive learning between the search and template features. Then, the enhanced features will be fed into the spatial-frequency Transformer which considers the Gaussian spatial prior and low-/highfrequency feature information simultaneously. Finally, we utilize a three-layer MLP head to predict the target object position and classification confidence scores. Our network can be trained in an end-to-end manner by jointly optimizing the BCE, CIOU, and L1 loss. In the following subsections, we will give a detailed introduction to these modules respectively.

B. Feature Extraction using Cross-Scale Fusion based Swin-Transformer
Given the first video frame T and the i th search images I i , we first resize them into T ∈ R 3×H T ×W T and I i ∈ R 3×H I ×W I respectively. Then, these images or patches are fed into a shared backbone network for feature embedding, such as the widely used ResNet [12]. Although good performance can be obtained, more and more works demonstrate that the feature representation may not be optimal due to the limited receptive field caused by local convolutional filters. Inspired by the success of Swin Transformer [15] in the image classification task, in this work, we adopt the Swin-Tiny as our backbone network because it achieves a good balance between performance and efficiency. More in detail, the fourth block of Fig. 2. An overview of our proposed SFTransT visual object tracking algorithm. It follows the Siamese matching framework which takes the template and search frame as input. The Swin-Tiny network is adopted as the backbone and the cross-scale features are fused as embedded features. Then, a Multi-Head Cross-Attention (MHCA) module is used to boost the interactions between the dual features. The output will be fed into our core component Spatial-Frequency Transformer, which models the Gaussian spatial prior and low-/high-frequency feature information simultaneously. More in detail, the GGN is adopted to predict the Gaussian spatial attention which will be added to the self-attention matrix, then, the GPHA is designed to decompose them into low-and high-pass branches to achieve all-pass information propagation. Finally, the enhanced features will be fed into the classification and regression head for target object tracking. C , R , S represent the Concatenate, Rearrange, and Shift-and-Sum operators respectively.
Swin-Tiny is removed to gain the effective strides as 8× and reduce parameters from 28.8M to 12.6M. The shift window size is set as 8 to accommodate our tracking task.
To fully utilize the hierarchical features of the Swin-Tiny network, we propose a novel Cross-Scale Fusion (CSF) module to adaptively fuse them for a more robust representation. As shown in Fig. 2, when we input an image whose dimension is 3 × H × W into the Swin-Tiny network, the dimension of features output from second and third block are respectively. Therefore, the fused features can be obtained via: where [ , ] denotes the concatenate operation along the channel dimension. Rearrange is used to increase the resolution of feature maps without scale information missed. M LP is consisted of two fully connected (FC) layers to project the feature dimension from 3C to D. Therefore, we can get the search and template feature F I ∈ R D× H I In our experiments, C and D are set as 96 and 256, respectively.

C. Interactive Feature Learning using MHCA
After we obtained the backbone features F I ∈ R D× H I for the search and template branch, we feed them into the Multi-Head Cross-Attention (MHCA) module to boost the interactive feature learning. Specifically, the feature maps are firstly reshaped into patch sequences, . Then, the input template features are transformed into query Q T , key K T , and values V T using non-shared fully connected (FC) layers, i.e., The cross-attention from template to search feature used in n th head H n I can be mathematically formulated as: where F F N denotes the feed-forward network, which targets to magnify the fitting ability of cross-attention. LN is short for layer normalization. N denotes the number of heads used in cross-attention. Similarly, the cross-attention from search to template features can be obtained by: It is worth noting that the MHCA mentioned above or other Transformer modules all adopt the sinusoidal position embedding [41], which is added to the input sequences.

D. Spatial-Frequency Transformer
Once we get the output from the MHCA module, we can directly feed them into the tracking head for target localization like many trackers do [16], [18], [42]. However, we can find that the aforementioned feature interaction between search and template features is based on 1D feature vectors to make full use of the Transformer layers. Although good performance can already be obtained, the feature enhancement based on feature vectors cannot capture the spatial prior which is also very important for visual object tracking. In addition, many works demonstrate that the Transformer layer is actually a low-frequency passer and may suppress the high-frequency features. To address these issues, we propose a novel Spatial-Frequency Transformer (SFFormer) which will be introduced in the following subsection. Gaussian Spatial Prior Attention. As shown in Fig. 3, the first key component of our proposed unified Spatial-Frequency Transformer is the Gaussian spatial prior generation. As we all known, the center position (x c , y c ) and standard deviation (σ w , σ h ) are two important parameters for the Gaussian-prior distribution generation. In this work, we adopt a simple but effective Gaussian Generation Network (GGN) to generate these parameters and variables adaptively. Specifically, the GGN contains three parallel MLP branches M LP 1 , M LP 2 , M LP 3 separately. The M LP 1 is the center prediction network, which is solely used in the first attention layer of Spatial-Frequency Transformer (SFT). The M LP 2 is used to predict the center position bias, which is deployed on the subsequent layers of SFT structure. Note that the center bias will fine-tune the initial center position from M LP 1 to adopt the specific situation of the current layer. The M LP 3 branch generates the variance of weight and height of Gaussian weight, also called the standard deviation of G(x, y).
Given the search features F i , we can get the Gaussian map using GGN and the detailed procedure can be written as: where α is the hyper-parameter to control the scaling factor. As a deformation of standard normal distribution, Gaussian distribution has the same characteristic, i.e., assign the patches with different weight factors according to the distance with the center position. Therefore, Gaussian distribution has innate preference weight with patch positions. After we obtained the Gaussian prior maps, the second question is how can we integrate them into the attention mechanism? Inspired by the fact that Gaussian maps are also 2D prior images, therefore we treat them as bias terms and fuse them into the similarity matrix of the self-attention mechanism. This procedure can be formulated as: where log denotes the logarithm operation for smooth attention decay of Gaussian map, W Attn represents the weight of query attention, and d k is the dimension of key feature vectors.
To better fit the Gaussian prior maps with multi-head attention mechanism, we extend the aforementioned Gaussian maps into multi-head fields. Specifically, we broaden the output of single-head Gaussian map generation network M LP 1 , M LP 2 , M LP 3 dimension from 2, 2, 2 into 2 × N, 2 × N, 2×N , where N is the head number of multi-head attention. Then, the proposed multi-head Gaussian prior attention weight can be rewritten as: where n ∈ [1, . . . , N ]. Based on the Gaussian-weighted attention weight, our Gaussian Prior Attention (GPA) module can be formulated as: where the F F N consists of three FC layers.

High-Frequency Emphasis Attention.
Another key component of our SFFormer is the High-Frequency Emphasis Attention which enhances the low-/high-frequency feature learning significantly. It is inspired by the recent findings that the attention mechanism inherently amounts to a low-pass filter [20], [58]. Therefore, the stack of multi-head attention layers in Transformer may suppress the high-frequency component of input features severely and generate over-smoothing output feature sequences. This may lead to the degeneration towards Direct-Current (DC) component 2 . To protect the high-frequency features meanwhile keep the DC component, we decompose the co-attention weight into DC and highfrequency components. As shown in Fig. 3, given the Gaussian-weighted similarity matrix W n Attn ∈ R N ×S I ×S T , we first decompose it into dual branches: the average DC bias matrix branch and highfrequency components branch. More in detail, this procedure can be formulated as: where 1 ∈ R N ×S I ×S T . As illustrated in Eq. 10, for each query patch, the sum of similarity scores, i.e., the attention weights, between a query patch and value sequence is always one due to the use of Sof tmax operator. Therefore, the DC component of the similarity matrix W n Attn is averaged by the value sequence length. Correspondingly, the high-frequency signal can be obtained by subtracting DC bias from the given attention weight matrix. To emphasize the high-frequency attention weight, we introduce a learnable parameter vector β ∈ R 1×N into the high-frequency branch. β n is initialized with zero and assigned as the specific and adaptive magnification of the n th separated attention head.
Through the high-frequency emphasis attention module, we evolve the attention from the low-pass filter into an all-pass filter to avoid the degradation of the high-frequency feature. Note that, emphasizing the attention weight component will not affect the basic attention flow structure and merely introduce N × L extra parameters. N and L denote the number of attention heads and layers used in SFFormer. Therefore, the final formulation of our newly proposed Gaussian Prior aware High-frequency emphasis Attention (GPHA) can be written as: In our practical implementation, this module is stacked for L layers for building SFFormer and our visual tracker. The outcome of the last GPHA block will be directly fed into classification and regression heads for target localization. To simplify and keep a clean framework, we adopt dual simple and classical three FC layers as the tracking heads.

E. Training and Tracking Phase
Training Phase. Given two frames in a random sampling within a 200-frame window video as template and search frame, the training framework will resize the template and search frame into 128 × 128, 256 × 256 respectively. The proposed tracker is optimized in a two-branch end-to-end fashion with three loss functions. For the classification prediction, we adopt the binary cross-entropy (BCE) loss which can be formulated as: where s is the s th patch in the predicted probability sequence. y s , y s separately denote the ground truth label and predicted class. Different from existing works which adopt the rectangle classification, in this work, we employ the ellipse ground-truth label which can be written as: where (g xc , g yc , g w , g h ) denote the center coordinate, width and height of ground-truth boxes, (p x , p y ) is the classified point. When the point falls in the ellipse label, we will regard it as the foreground class. Otherwise, outside the ellipse is the background. Inspired by the Hungarian algorithm in DETR [41], we combine the CIOU loss [59] and L1 loss for the position prediction. The regression loss can be written as: where b s ,b s shows the ground-truth box and prediction box coordinate. λ 1 and λ 2 are the weight hyper-parameters to balance the CIOU and L1 loss. Therefore, the overall loss function can be written as: We simply set the trade-off parameter λ 1 = 5, λ 2 = 2, λ 3 = 10 without any hyper-parametric searching. Tracking Phase. During inference, we crop the template size from the first frame in the video sequence and feed it into our backbone network to get the template feature. The template feature will be cached and employed in the following tracking phase. For the second and following frames, we firstly crop them into 256 × 256 search region. Then, the template and search feature sequences are parallelly fed into the end-to-end framework to gain classification and regression maps. To achieve robust long-term visual tracking, in this work, we adopt a simple but effective local-global search strategy. It is inspired by [40] but much simplified to avoid extra parameters and computational cost. The major difference between local and global search is shown below: • Local search strategy: employing the center and four times area of the previous frame bounding box as the search region of search frames. • Global search strategy: using the whole image as the search region of search frames. Because our tracker crops the search region into 256 × 256 immediately after selecting the search region, the global strategy will keep the same computational cost as local and have not any hyper-parameters introduced. In our tracker, the switching from local to global search will be executed when the classification confidence is less than the threshold τ c for τ f consecutive frames. Inspired by the local-global switch strategy proposed in TANet [40], we simply set τ f = 5. To achieve a good balance between robustness and highperformance tracking, we simply set the failure threshold as τ c = 0.98. Note that, these two parameters can be optimized by a hyper-parametric searching scheme. We leave this at our future work.
In our experiments, multiple evaluation metrics are adopted, including Success Rate, Precision Rate, Normalized Precision Rate, and Average Overlaps. Specifically, Success Rate (SR) is the metric to measure the IOU overlap between ground-truth and prediction boxes. Precision Rate (PR) measures the center's precise degree between the predicted point and ground-truth center. Usually, only when the distance is lower than 20 pixels will be treated as a success track. Normalized Precision Rate (NPR) extra considers the scale size of the ground truth box and normalizes the precision rate. SR and PR are the all-purpose measure metrics in OTB100, NFS, UAV123, LaSOT, TrackingNet, LaSOT_ext, WebUAV-3M and TNL2K, while NPR is usually used in TrackingNet, LaSOT, LaSOT_ext, WebUAV-3M, and TNL2K. Additionally, the Average Overlaps (AO), SR 0.5 and SR 0.75 are specific metrics for GOT-10K. AO calculates the average overlaps between all ground-truth and predicted bounding boxes. SR 0.5 and SR 0.75 laid down the different thresholds in IOU overlap of SR.

B. Implementation Details
The used backbone Swin-Tiny is initialized using pretrained weights on the ImageNet-1K dataset. For all of the attention modules in our framework, we set the dropout as 0.1. Sampled image pairs are cropped with brightness jittering and horizontal flip. The total training epoch, iteration, and mini-batch are 500, 1k, and 48, respectively. We set the initial learning rate as 1e-5 for optimizing the cross-scale fusionbased Swin-Tiny and 1e-4 for other components without pretrained model. The optimizer is AdamW [68] with a weight decay of 1e-4. The learning rate has decayed a magnitude at the 400 th epoch. The SFFormer consist of six GPHA blocks. Our code is implemented based on Python 3.8 and Pytorch 1.8.0 on a server with 4× RTX 3090 GPUs.

C. Comparison with Other State-of-the-art Trackers
In this section, we report our tracking performance on nine widely used benchmark datasets and compare it with other state-of-the-art (SOTA) trackers. Results on OTB100 [62] and NFS [63]: OTB100 is a short-term and approaching saturation benchmark used in visual tracking. The performance comparison can be found in Table III, most trackers achieve SR results around 70%. Our SFTransT obtains a slightly better result compared to ToMP101 with 70.3%/91.6% on SR/PR. For the NFS [63] dataset, the 30 FPS (Frame Per Second) version is adopted for evaluation which is full of fast motions and challenging distractors. As shown in Table III, our tracker achieves a performance of 66.0%/80.3% on SR/PR, which is slightly better than TransT [16] and comparable with ToMP101. Results on GOT-10K [23]: GOT-10K is a recently released large-scale short-term benchmark in visual tracking. Following the one-shot training protocols of GOT-10K, our tracker merely use the training set of GOT-10K for model learning. We evaluate our testing results on the benchmark official online server. As shown in Table I [49]), we overstep 2.3% in AO. SwinTrack-T is a tracker that employs the same backbone network (Swin-Tiny [15]) as ours. Our tracker outperforms it with a remarkable gap with our spatial-frequency design. Furthermore, compared with TransT [16], we also gain a significant 5.6% performance improvement. Results on TrackingNet [61]: TrackingNet contains more than 30K training videos with a 30-frame interval labelling strategy. It covers an average of 1300 frames per sequence. As shown in Table II, our tracker achieves an advanced performance on this large-scale benchmark dataset, i.e., 82.9%, 87.3% 81.3% on the SR/NPR/PR metrics, respectively. More in detail, our results are better than recent strong Transformer based trackers, including ToMP101 [43] (81.5%, 86.4%, 78.9%), TransT [16] (81.4%, 86.7%, 80.3%). Results on UAV123 [64]: UAV123 is a dataset collected by the low-attribute intelligent unmanned aerial vehicle (UAV). It consists of 123 sequences with many long-term videos and small target objects inside. As shown in Table IV, our tracker achieves a new state-of-the-art result, especially 71.3% on SR, outperforming MixFormer-1k by +2.6%. Results on LaSOT [22]: As a long-term tracking benchmark, LaSOT contains over an average of 2500 frames for the 280 testing videos. The experimental results are illustrated in Table V, and many SOTA trackers are compared, e.g., MixFormer [50], ToMP [43], SBT [49], CSWinTT [48], Keep-Track [38], SwinTrack [47]. We can find that our proposed tracker achieves 69.0%/78.1% on SR/NPR, which is better than recent SOTA trackers SBT-large, ToMP101, SwinTrack-T, and Mixformer-1k. We also give a comparison of the SR scores on different attributes in Fig. 4. Note that, our proposed tracker SFTransT outperforms the TransT by more than +4% on all attributes. Results on LaSOT_ext [67]: LaSOT_ext is an extension of LaSOT benchmark dataset. It introduces 150 new video sequences with 15 new classes. It is also a long-term tracking dataset that contains over 2500 frames on average for each sequence. As shown in Table VI, our tracker achieves 46.4% on SR and beats the previous SOTA tracker ToMP101. Results on TNL2K [65]: TNL2K is a new high-quality benchmark consisting of 700 high-diversity testing videos with natural language specifications. We follow the rule with bounding box guided tracking and achieve the best performance on TNL2K. As shown in Fig. 5, we receive state-of-the-art results with 54.6%, 60.6%, and 55.5% on SR/NPR/PR respectively. This experiment demonstrates that our tracker obtains significant improvements, even compared with SOTA trackers   such as SiamR-CNN [32], SuperDiMP [36], LTMU [70], AutoMatch [69], KYS [71], etc.
Results on WebUAV-3M [66]: WebUAV-3M is the newest and largest-scale UAV tracking benchmark to date, containing over 3.3 million frames across 4,500 videos with million-scale dense annotations. The test subset of WebUAV-3M has 780 videos and 120 target classes. We evaluate our SFTransT with the guidance of visual box annotations. As shown in Fig. 6, SF-TransT gets a new state-of-the-art result and surpassed Keep-Track (+3.4%), TransT(+12.2%) and AutoMatch(+12.5%) on SR.
All in all, we can find that our proposed SFTransT achieves better performance on all these benchmarks than most of the compared trackers. We also set new state-of-the-art results on UAV123 and GOT-10K datasets, which fully validated the effectiveness and advantages of our tracker.

D. Ablation Study
In this section, extensive ablation studies are conducted to validate the effectiveness of our core component. Analysis on Feature Extraction, Fusion, and Augmentation. In this subsection, we implement four kinds of feature extraction and fusion networks to validate the effectiveness of our feature extraction network, i.e., #1, #2, #3, #4, as shown in Table VII. More in detail, the Layer-wise Aggregation (LWA) module was first proposed in SiamRPN++, which linearly combines three output layers with learnable weights [8], [10]. #4 represents our proposed Cross-Scale Fusion (CSF) based Swin Transformer feature extract network. ResNet50 is kept the same processing with SiamRPN++ [8] and SiamBAN [10]. As we can see from Table VII, when we replace the backbone with ResNet50, the performances dropped around 1.1%-1.8% compared with Swin-Tiny based version. By comparing algorithm #2 and #1, algorithm #4 and #3, we can find that the tracking results can be improved by more than 0.6% in SR and 0.8% in AO, with the help of Cross-Scale Fusion network. These experiments fully validated the effectiveness of our proposed Swin-Tiny base on Cross-Scale Fusion network.
In addition, we can also find that our feature interactive learning module MHCA helps for our final tracking results, by comparing the algorithm #4 and #5. Specifically, this module improves the SR|PR, AO from 68.1|73.3 to 69.0|73.9, 71.3 to 72.7 on the LaSOT and GOT-10K dataset, respectively. Analysis on Spatial-Frequency Transformer. To verify the effectiveness of our proposed SFFormer, multiple components are tested in this section, as shown in Table VIII. Specifically, the GPA means the Gaussian-prior attention, HEA denotes the high-frequency emphasis attention module, GPHA is our proposed Gaussian-prior aware high-frequency emphasis attention. w/o multi-G means we only use the same Gaussian maps for different attention heads in a GPA block.
The algorithm #1 listed in Table VIII is the baseline tracker which employs two-layer MHCA only, without any extra spatial and frequency attention layers. By comparing algorithms #1 and #2, it is easy to find that even simple Gaussianprior attention can boost the tracking results on both datasets. Specifically, the baseline results are improved by +0.6|0.6, +0.4|1.4|0.6 on the LaSOT (SR | PR) and GOT-10K (AO | SR 0.5 | SR 0.75 ) dataset. When the multi-G is adopted (i.e., the tracker #3), the overall performance can be improved from      By comparing the algorithm #4 with #1, we can find that the overall performance can also be improved on both datasets. This demonstrates that High-frequency Emphasis attention (HEA) is effective in our framework. When we combine the HEA with multi-head Gaussian-prior weighted attention, i.e., our proposed GPHA layers, the overall results can be improved significantly, as we can see by comparing the algorithm #5 and #1. These results fully demonstrate the necessity and importance of spatial prior and frequency information for Transformer based tracking. Analysis on Loss Functions and Classification Labels. In this paper, we adopt different loss functions and classification labels for the training of our tracker. As shown in Table IX, we report and compare the tracking results with different settings on the training, validation, and testing subset of GOT-10K dataset. It is easy to find that the CIOU loss improves our model from 71.5 to 72.0 on the AO metric by comparing tracker #1 and #2. From the comparison of tracker #1 and #3, #2 and #4, we can find that the ellipse label works better than the regular rectangle labels. According to all these experimental results and analyses, we can draw the conclusion that the carefully designed loss functions and classification labels can further improve the final tracking results without using external training samples.

E. Parameter Analysis
In this section, we will give a detailed introduction to the parameter analysis, including the number of GPHA layers and model parameters. Analysis of Different Number of GPHA Layers. To further verify the effectiveness of our Spatial-Frequency Transformer, in this section, we test our model using different layers of GPHA blocks on the GOT-10K dataset. As shown in Table X, the best tracking results can be obtained when 6 GPHA layers are used, i.e., the AO is 0.727 on the GOT-10K dataset. When only 2 layers are adopted, the tracking efficiency is the best, i.e., 35 FPS.

F. Attribute Analysis
In this section, we report the experimental results under challenging factors defined in the LaSOT and TNL2K dataset,

Annotation
TransT MixFormer-1k OURS SiamR-CNN TrDiMP STARK including aspect ration change, adversarial samples, low resolution, partial occlusion, rotation, scale variation, etc. As shown in Fig. 4, our tracker achieves better results even compared with recent SOTA tracking models, such as the MixFormer-1k and TransT, on the LaSOT dataset. The results reported in Fig. 7 also validated the effectiveness of our tracker and beat many SOTA trackers, including SiamR-CNN, SuperDiMP, and AutoMatch.

G. Visualization
In addition to the aforementioned quantitative analysis, in this section, we also give a visualization of spatial-frequency attention and tracking results as the qualitative analysis, to help readers better understand our tracker. Spatial-Frequency Attention. For a more specific analysis of spatial-frequency attention, we give a visualization of headspecific attention weight and classification activation map, as shown in Fig. 10. The first row is the video frame with spatial-prior. It is easy to find that our proposed SFFormer can focus on the center of search regions. The last two rows denote the specific head information representation without Fig. 10. Visualization of the attention weight of multi-head spatial-frequency attention mechanism. Best viewed in color and zoom in. and through our SFFormer network. N represents the head index of attention blocks and the dimension of each subfigure is S I × S T . We can easily find that the former row contains much more DC components without our SFFormer, especially the 2 th , 5 th , 6 th , 7 th , 8 th nearly pure DC bias information. Thanks to the proposed SFFormer which enhances the highfrequency features effectively, we can not only protect the DC bias (6 th ) but also emphasize the high-frequency feature in some heads. These visualized attention maps fully validated that our SFFormer emphasizes high-frequency branch information and avoids the degeneration of target object attention effectively.
In addition, we also visualize and compare the activation maps with the TransT, as illustrated in Fig. 9. We can find that our activation maps focus more on the edge position and specific target components. Therefore, our tracker is more accurate than the SOTA tracker TransT. Tracking Results. As shown in Fig. 8, we give a visual comparison of the tracking results of our tracker and other five SOTA trackers, including MixFormer-1k, STARK, TrDiMP, SiamR-CNN, and TransT. It is intuitive to observe that our tracker can be more robust when facing occlusion, deforma-tion, similar distractor, scale variation, fast motion, etc.

H. Failure Cases
We believe the aforementioned analysis can prove the robustness and accuracy of our proposed spatial-frequency Transformer tracker. At the same time, our tracker does not explicitly model distractors for dense and small-scale targets, therefore, it is still easily affected by these factors. We give some visualization of the failed cases in Fig. 11. In future works, to address this issue, we will consider adopting graph feature matching to model the relations between the target object and background objects.

V. CONCLUSION
In this paper, we propose a novel Spatial-Frequency Transformer (SFFormer) network for the visual object tracking task. Precisely, our proposed tracking framework is termed SF-TransT and it contains four main modules, including the crossscale fusion-based Swin-Transformer, MHCA, SFFormer, and tracking head. The Swin-Transformer is adopted as our backbone network for feature extraction. The MHCA is used to boost the interactive feature learning between the search and template branches. The SFFormer is designed to jointly models Gaussian spatial prior and low-/high-frequency information. The enhanced features will be fed into the tracking head for target object localization. Extensive experiments on multiple benchmark datasets all validated the stable improvement of our proposed tracker. In our future works, we will consider designing a Transformer network to build graph matching model that explicitly models the target object's appearance with discriminative learning in dense scenes to suppress the influence of distractors and background clutter.