{'fraction_non_alphanumeric': 0.03649014302280634, 'fraction_numerical': 0.01453420950908388, 'mean_word_length': 4.8162470023980815, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Revisiting Color-Event based Tracking: A Unified Network, Dataset, and Metric

Combining the Color and Event cameras (also called Dynamic Vision Sensors, DVS) for robust object tracking is a newly emerging research topic in recent years. Existing color-event tracking framework usually contains multiple scattered modules which may lead to low efficiency and high computational complexity, including feature extraction, fusion, matching, interactive learning, etc. In this paper, we propose a single-stage backbone network for Color-Event Unified Tracking (CEUTrack), which achieves the above functions simultaneously. Given the event points and RGB frames, we first transform the points into voxels and crop the template and search regions for both modalities, respectively. Then, these regions are projected into tokens and parallelly fed into the unified Transformer backbone network. The output features will be fed into a tracking head for target object localization. Our proposed CEUTrack is simple, effective, and efficient, which achieves over 75 FPS and new SOTA performance. To better validate the effectiveness of our model and address the data deficiency of this task, we also propose a generic and large-scale benchmark dataset for color-event tracking, termed COESOT, which contains 90 categories and 1354 video sequences. Additionally, a new evaluation metric named BOC is proposed in our evaluation toolkit to evaluate the prominence with respect to the baseline methods. We hope the newly proposed method, dataset, and evaluation metric provide a better platform for color-event-based tracking. The dataset, toolkit, and source code will be released on: \url{https://github.com/Event-AHU/COESOT}.


Introduction
Visual object tracking targets at locating the initialized target object using a series of bounding boxes in the RGB videos. It is one of the most important tasks in computer vision and provides a good basis for other vision tasks, such as activity recognition, person re-identification, and trajectory analysis. Many representative trackers are proposed like MDNet [35], SiamFC [2], SiamRPN [27], DiMP [3], TransT [11]. Although these RGB-based trackers work well in simple scenarios, their performance is still poor under challenging scenarios like low illumination, over-exposure, fast motion, heavy occlusion, and background clutter.
To address the issues caused by single-modal defects, some researchers resort to fusing RGB and Event cameras for reliable object tracking. As the event camera is a bioinspired sensor, which records binary events (i.e., the ON and OFF event) in each pixel when the variation of light intensity exceeds the given threshold. It shows significant advantages over RGB cameras in the high dynamic range (HDR), low energy consumption, and low latency. The dense temporal resolution makes it almost free from the interference of motion blur, and HDR ensures its imaging performance in low-illumination and over-exposure scenarios. Meanwhile, the RGB cameras provide color and texture information which are also very important for tracking. The integration of RGB and event cameras provides a new way for practical tracking.
Although very few, several works have explored the fusion scheme for color-event-based tracking [21,24,44,51,52]. To be specific, Zhang et al. [51] design the cross-domain feature integrator to fuse the RGB and event data. Huang et al. [24] reconstruct grey frames from event streams to enhance the feature learning for their SVM-based tracker. These algorithms divide the tracking into multiple stages, including backbone network, feature fusion, interactive head, and tracking head, as shown in Fig. 1 (a)-(b). Note that, Siamese matching based color-event trackers [44,51] involve the template and search feature extraction and matching, which further complicates the tracking framework. Their model achieves about 20 and 14 FPS only, which is hard to employ in practical applications. This inspired us to think about designing a unified and simple multi-modal tracking framework to achieve efficient and accurate tracking. The recently proposed Transformer [41] has shown its natural advantages and effects in simplifying network structure, such as object detection [6], segmentation [53], and tracking [13,28]. However, there is still no work to design a unified tracking framework using Transformer for color-event object tracking task.
In this paper, we propose a simple and unified singlestage tracking framework to accomplish color-event tracking, termed CEUTrack. As shown in Fig. 1, it contains three main modules, including the projection embedding layer, unified backbone network, and tracking head. Given the event stream, we first transform the event points into voxels through voxelization for efficient representation. Then, the template and search inputs are cropped from both RGB frames and event voxels. After embedding layers, these inputs are projected into multi-modal tokens and fed into the unified Transformer network. Finally, the tracking head projects the output tokens to predict the trajectory of the target object. Different from the popular Siamese framework (dual-stream for unimodal, four-stream for bimodal data), our proposed CEUTrack is a single-stream tracking framework that concatenates all of the tokens as one input. Therefore, it simplifies the color-event tracking significantly and achieves a higher running efficiency, i.e., 75 FPS on color-event data. Extensive experiments demonstrate that our model achieves comparable or even new SOTA (State-Of-The-Art) performance on existing colorevent tracking datasets.
To better validate the effectiveness of our model and facilitate the community, we also propose a large-scale benchmark dataset for Color-Event based Single Object Tracking, termed COESOT. It contains 1354 aligned video sequences captured using a DAVIS346 event camera, and the training and testing subset contain 827 and 527 videos, respectively. The target objects cover a wider range of categories than existing color-event tracking datasets [44,51], including vehicles, pedestrian, card, toy, bird, monkey, tiger, elephant, zebra, crocodile, etc. These videos fully reflect the key challenging factors in visual and event tracking. 17 attributes are annotated to help detailly evaluate the performance of the trackers. More detailed introductions and comparisons can be found in Section 4 and Table 1. In addition, we also propose a new evaluation metric named BreakOut Capability score (BOC) for visual object tracking. Different from existing metric which treats each video equally to get a precision score, such as PR (Precision Rate), NPR (Normalized Precision Rate), and SR (Success Rate), the BOC score focus more on the prominence of the evaluated tracking algorithm compared with existing ones. In another word, a higher weight proportion will be given for a challenging video, while an ordinary proportion will be assigned for a simple video. The difficulty of each video is comprehensively measured by the accuracy of existing trackers.
To sum up, the main contributions of this paper can be summarized as the following four aspects: • We propose a simple and unified tracking framework based on Transformer network, termed CEUTrack, which is the first simplified one-stage backbone for color-event tracking that achieves feature extraction, fusion, and interactive learning simultaneously.
• We propose a large-scale and general benchmark dataset for color-event tracking, termed COESOT. The categories of target objects and video numbers in COESOT are significantly richer than existing datasets. We also benchmark multiple modern baseline methods for future works to compare.
• We propose a new evaluation metric for the tracking task, termed BOC score. It can better reflect the outstanding ability of the tracking algorithm compared with the existing baseline methods on difficult videos.
• Extensive experiments on multiple benchmark datasets (COESOT, VisEvent, FE108) validated the effectiveness and efficiency of our tracker. It sets new SOTA performance on existing datasets meanwhile runs at about 75 FPS.

Related Work
Event-based Tracking. Event-based tracking attracts more and more attention in recent years. To be specific, based on the DVS camera, ATSLTD [10] proposes an event-frame combined algorithm to construct the event to frames and warp the spatio-temporal information of asynchronous retinal events. Chen et al. [9] further feed the sequence of TSLTD frames to a retinal motion regression network to perform an end-to-end 5-degree-of-freedom (5-DoF) object motion regression. Ramesh et al. [37] use the discriminative representation for the object with online learning for long-term event-based tracking. Chamorro et al. [8] present an ultra-fast tracking algorithm able to estimate 6-DoF motion and a robust mechanism for the matching of events with projected line segments with very fast outlier rejection. Ignacio et al. [1] propose an asynchronous patch-feature tracker that relies solely on events and processes each event individually as soon as it gets generated. To solve the HDR situation, Jiang et al. [25] combined the offline-trained detector with an online-trained tracker in a specific Kalman fusion scheme. SiamEvent [7] correlates the embedded events at two timestamps to compute the target edge similarity and use the Siamese network via edge-aware similarity learning. STNet [50] employs the Transformer network and spiking neural network (SNN) to extract temporal and global spatial information. However, with the above development, eventbased tracking still has low tracking ability because of the limited event-modality representation and lack of visual information (e.g., texture, color, and shape).
Color-Event based Tracking. Tracking by combining color frames and event streams is an interesting and reliable way to achieve high-performance target object localization. For instance, Daniel et al. [21,22] firstly leverage the complementarity of event and visible cameras to realize low latency feature tracking. DashNet [48] jointly processes the synchronous activation from artificial neural networks and SNN spikes signal resources based on the complementary filter and attention module. ESVM [24] is a color-event tracker proposed by Huang et al., which targets tracking objects by adaptive search region mining and online SVM (Support Vector Machine) updating. Liu et al. [30] propose a three-step tracking algorithm by fusing RGB frames and event streams simultaneously. Zhang et al. [51] propose self-and cross-domain attention with an adaptive weighting scheme to fuse visual and event images. Wang et al. [44] build the CMT which is a cross-modality transformer module that performs well for color-event tracking. Although good performance can be achieved in some scenarios, these trackers involve multiple components, such as fusion strategies, multi-step structures, augmentation modules, and dual-stream (Siamese) frameworks to locate the target object. Therefore, these algorithms suffer from heavily limited tracking efficiency with superfluous module design. In this paper, we propose a simple and unified colorevent tracking framework that can realize more efficient and accurate tracking.

Overview
As mentioned in previous paragraphs, most existing multi-modal visual trackers are developed based on classification, correlation filters, or Siamese tracking frameworks. Researchers focus on designing new fusion modules for bimodal-based tracking tasks, like the color-event data studied in this work. Therefore, the obtained framework may be too complicated to employ in practical applications. In this work, we propose a simplified color-event tracking framework, termed CEUTrack, as shown in Fig. 2. Given the event streams, we first transform them into voxel representation via voxelization operation. Then, we crop the template patch and voxel according to the initialized location from the color frame and event voxel respectively. The search patch and voxel are also extracted from search regions. A project layer is adopted to transform the four inputs into token representations. The input embeddings and position encoding features are concatenated as a unified representation of our tracker. Then, we propose a unified Transformer network as the backbone to learn the spatialtemporal feature representation. Finally, the obtained tokens are fed into the tracking head for target object localization. More details about each module will be introduced in the following paragraphs respectively.

Input Representation
With the bimodal input, color frames and event streams can be mathematically defined as: where I r is the r-th color frame, x i and y j are the spatial coordinates, i ∈ {1, ..., W }, j ∈ {1, ..., H}. W, H are the width and height of the color frame. E r consists of multiple event points e s , s ∈ {1, ..., S}. S is the total number of points tensor. (x s , y s ) are spatial coordinates of the event point, t s is the timestamp of each event point and p s is the polarity, i.e., ON or OFF event which corresponds to the red or blue point in Fig. 2.
To make full use of the CNN network, previous researchers usually transform the asynchronous event streams into an image-like representation. The transition from asynchronous to synchronous will lose a lot of temporal information. In this paper, we transform the event points into voxels which preserves the temporal information well, and the voxels top-selection mechanism also fully exploits the sparse representation of event data which reduces the computational complexity. Specially, we split the whole event stream E into event tensor E r based on the time interval (i.e., the time stamps) of color frames. Then, we divide the largescale irregular event points in each event tensor into volu- and corresponding features f opq ∈ R Q×16 , therefore, L = 19. The voxelization process generates the voxels based on the voxel volume. With the input event stream fragment R W ×H×T along the space (W, H) and time T respectively, the single voxel volume is w × h × t accordingly. Inspired by the 3D object detection benchmark SECOND [47], we calculate the mean of voxel features with the 3D coordinates and polarity in a certain voxel as the voxel-wise representation. Therefore, the generated total voxels number With the dense events recapitulated to a sparsity voxel set, the complexity of the following network can be significantly dropped.
Given the first color frame and event voxel set, we first crop the template and search region which is 2 and 4 times large than the initialized target location. Then, we resize the template and search patches for the color frame modality into z f ∈ R Hz×Wz×3 and x f ∈ R Hx×Wx×3 . For the event modality, we remove the voxel grids whose coordinates are not in the search or template region and then select the top M x = 4096 or M z = 1024 voxel grids in the sorted voxels. These voxels are sorted by the density of event points. Therefore, we can get the template voxels z v ∈ R Mz×L and search voxels x v ∈ R Mx×L . Once we obtained all four inputs, we feed them into the proposed unified network, as shown in Fig. 2.

Unified Backbone Network
Before feeding the inputs into backbone network, we propose a projection layer to transform them into token rep-resentations. The projection layer consists of four parallel and split Conv BN Relu blocks. For the color frame, the corresponding block project the template z f and search patches x f into feature embeddings F f z ∈ R Nz×C and F f x ∈ R Nx×C using two non-shared 16 × 16 Conv operators. Note that, the N z and N x is the number of template and search tokens. For the event voxel, we flatten the search x v and template voxel z v into F v x ∈ R Nx×C and F v z ∈ R Nz×C based on the non-shared 4 × 4 Conv. In addition, we take the position embedding into consideration and add them with feature embeddings. More in detail, the position embedding is shared between search regions of the color frame and event voxel domain, i.e., P x ∈ R Nx×1 . Similar shared position embedding can be obtained for template regions, i.e., P z ∈ R Nz×1 .
After the four projected tokens are obtained, we concatenate them into a unified token sequence U s ∈ R 2(Nz+Nx)×C and feed into the stacked and cascade twelve Transformer layers. As shown in Fig. 2, each Transformer layer consists of two layernorm (LN), multihead self-attention (MSA), multi-layer perception (MLP), and residual connections. The detailed computation process can be formulated as: We adopt the vanilla base version of ViT [19] as our backbone network, which provides a clean architecture and publicly available pre-trained weights. It avoids the tedious pro- To obtain the location of target object, we employ a standard tracking head to directly estimate the center position and scale of the bounding box. Specifically, the tracking head contains three branches that predict the classification score, center offset, and the size of bounding box, respectively. Each branch consists of four 3 × 3 Conv BN Relu and a 1 × 1 Conv layer respectively.

Training and Testing Phase
To train our proposed CEUTrack effectively, the following three loss functions are adopted, including the focal loss [29] for classification, L1 loss and GIOU loss [39] for bounding box regression. The overall loss function can be written as: where the trade-off parameters λ 1 , λ 2 , and λ 3 are set as 1, 1, and 14, respectively. Note that these parameters are manually set to balance the magnitude of loss functions without hyperparametric searching.
In the tracking procedure, we first crop the template patch from the first frame based on the initialized bounding box. The template patch is resized into a fixed scale, i.e., 128×128×3, in our experiment. For the event voxel, we get its template representation via the following two rules: 1) the coordinate of the voxel grid is within the cropped template region; 2) top-1024 voxel grids are computed based on the density of event points. For the template with voxels less than 1024, zero voxels will be used for padding. The preprocessed template voxel and image will be saved into the cache for subsequent tracking. For the following frames, we crop the search patch for the color frame domain and resize it into 256 × 256 × 3. We select top-4096 voxels by following the aforementioned rules as the search voxel representation. Then, we combine the search and template representations of both modalities for tracking by feeding them into the backbone network and tracking head for target object localization. Similar operations are executed for other color frames and event voxels until the end of the testing video sequence.

COESOT Benchmark Dataset
Data Collection. Existing color-event tracking datasets are small-scale [23,30,33], frames lack color information [51], or timestamp misaligned of partial data [44]. Also, the category of the target object is limited as illustrated in Table 1. In this work, we propose a general dataset for Color-Event camera based Single Object Tracking, termed COESOT. It contains 1354 color-event videos with 478,721 RGB frames. We split them into a training and testing subset, which contains 827 and 527 videos, respectively. The videos are collected from both outdoor and indoor scenarios (such as the street, zoo, and home) using the DAVIS346 event camera with a zoom lens. Therefore, our videos can reflect the variation in the distance at depth, but other datasets are failed to. Different from existing benchmarks which contain limited categories, our proposed COESOT covers a wider range of object categories (90 classes), as shown in Fig. 3 (a). It mainly reflects four groups, including persons, animals, electronics, and other goods.
Video Annotation. The ground truth of the proposed COESOT dataset is densely annotated, i.e., in a frame-by- frame way. The absent label is also provided to help researchers design their trackers. Inspired by VisEvent [44], we annotate each testing video sequence with 17 attributes to help researchers evaluate their trackers in specific challenging environments, e.g., full occlusion (FOC), deformation (DEF), rotation (ROT), fast motion (FM), partially occlusion (POC), low illumination (LI), scale variation (SV), background object motion (BOM), motion blur (MB), overexposure (OE), etc. The distribution of videos in each attribute is shown in Fig. 3 (b). The statistical distribution of the ground truth center position is shown in Fig 3 (c). More details can be found in our supplementary materials. Evaluation Metric. For the evaluation metric of CO-ESOT, we adopt four metrics to evaluate trackers' performance, including SR, PR, NPR, and our newly proposed BOC. For the motivation of BOC metric, existing evaluation metrics treat each video in a dataset equally and report an overall result. We perceive the difficulty of each video is different, and the newly proposed high-quality visual tracker should be given a higher reward if it succeeds in tracking an object while previous methods failed. With this in mind, we first compute the difficulty of each video in the COESOT dataset based on the SR score of the tested baseline trackers. Then, we weigh the tracking result of each video of the evaluated tracker using the obtained dif- ficulty value. Thus, our proposed BOC (BreakOut Capability) score effectively validates the real effectiveness and conspicuousness ability compared of exist trackers. If there are N testing videos and T compared base trackers, the formulation of BOC can be written as: where SR(evalT i ) denotes the success rate score of the evaluated tracker on video i, SR(baseT i t ) denotes the SR score of compared base tracker t on the video i.

Experiment
Three frame-event datasets are used in our experiments, including FE108, VisEvent, and the newly proposed CO-ESOT. More details of datasets can be found in our supplementary material.

Comparison with Other SOTA Trackers
Results on COESOT. As shown in Table 2, we report the results of 28 popular and recent trackers. It is easy to find that our proposed CEUTrack achieves the best tracking performance and outperforms other strong trackers by a large margin. To be specific, we obtain 62.0, 70.5, 69.0, and 20.77 on the SR, PR, NPR, and BOC metrics, respectively. Compared with single-stream Transformer based tracker OSTrack [49] which directly takes the fused color-event images as input and gets 59.0, 66.6, 65.4, 18.63 on these metrics, our method obtains improvement up to +3%, +3.9%, +3.6%, +2.14%. We also beat the strong tracker SiamR-CNN [42] which ranks second place on the COESOT benchmark by +1.1%/+3.0%/+2.7%/+1.69%. From the perspective of BOC score, as shown in Fig. 4, our  proposed CEUTrack, SiamR-CNN [42], and AiATrack [20] rank the 1st, 2nd, and 3rd, respectively, which fully demonstrates the breakthrough performance of our tracker. Results on VisEvent. Due to some videos in the Vi-sEvent dataset missing the source file of event streams (i.e., the aedat4 file), we removed these videos in this experiment and get a test subset with 172 sequences. As shown in Table 4, it is easy to find that our method beats other SOTA trackers significantly, including PrDiMP [17], STNet [50], TransT [11].
Results on FE108. This dataset contains only 32 test sequences with grey frames and event stream pairs. As shown in Table 3, our model achieves 55.58, 84.46 on the AUC and PR metric, which is significantly better than the compared strong trackers like the DiMP [4], PrDiMP [17], CMT-ATOM [44]. This fully demonstrates the effectiveness of our proposed unified single-stream color-event tracking framework.
Tracking Speed. With the simplified and unified framework, our CEUTrack can not only realize SOTA performance on three frame-event tracking datasets but also achieve a high tracking efficiency. As shown in Table 2, our tracker runs at 75 FPS which ranks 3rd place on the COESOT dataset and is significantly faster than many other high-performance trackers, such as SiamR-CNN (5 FPS), LTMU [14] (13 FPS), CMT-ATOM (14 FPS). Note that, SiamRPN [27] achieves the fastest tracking speed, i.e., 160 FPS, however, the accuracy is worse than ours significantly.

Ablation Study
Analysis of Modality Complementary. To validate the effectiveness of color frame and event stream fusion for visual tracking, we report the tracking results on COESOT with unimodal and bimodal data. As shown in Table 5, when only color frames are used, we can get 58.9, 66.6, 65.4, 18.13 on SR, PR, NPR, BOC, respectively.
The overall tracking results can be improved significantly when the event streams are added, i.e., 62.0, 70.5, 69.0, 20.77. If we utilize event streams only for tracking, the overall results are not satisfactory. It is easy to find that the event frames-/voxels-based trackers are all inferior to the tracker-4. This may be caused by the fact that the event streams are sparse in spatial view and also hard to discriminate compared with background. These results demonstrate the importance of color frames and the effectiveness of event streams, further validating the complementary of color and event modality in visual tracking.
Analysis on Different Event Representations. In this paper, we transform the event streams into voxel representation and achieve high-performance tracking (62.0, 70.5, 69.0 on SR, PR, NPR metrics), as the selected voxels consider the spatial sparsity and preserve the temporal information. Actually, the event streams are popularly represented as frame formats, like event frames, timesurface, and reconstructed grey images. In this part, we train and test the tracking using the mentioned three representations and report the results in Table 6. It is easy to find that when the color and event frames are added with a ratio of 0.2:1, we can get 59.0, 66.9, 65.9 on SR, PR, NPR metrics,  respectively. When we combine the color and event frames in a feature-level fuse way, we can obtain better results, i.e., 60.4, 68.6, 67.3. In this experiment, we also generate the timesurface representation for event stream using the timeoriented approach [26] and get 60.9, 69.1, 67.7. The grey image can be reconstructed from event streams using recurrent reconstruction model [38], termed Event RecIMG in Table 6. It achieves 60.5, 68.8, 67.4 on the three metrics which are similar to other image-like event representations. From all the experimental results and comparisons, we can find that the combination of color frames and event voxels achieves the best tracking results on the COESOT dataset. Analysis on Voxel Sampling. Voxelization plays a very important role in our tracking framework, in this subsection, we test different sets of parameters for voxel sampling to analyze its influence. As shown in Table 7, with the total voxel number m × n × τ , we fix the spatial sampling to get m = W w = 34, n = H h = 26 and search the appropriate timestamps sampling τ = T t and the voxels topselection. From the different sets of these parameters, we can find that much better results and equilibrium speed can be obtained when τ = 20 and search/template voxels select top 4096/1024.

Attribute Analysis
In this paper, 17 attributes are annotated for our CO-ESOT dataset to help analyze the influence of challenging  factors. As shown in Fig. 5, we report the results under 10 regular attributes, such as MB, FOC, ROT, LI, etc. We can find that our proposed CEUTrack achieves better results than the compared SOTA trackers when facing these challenging factors. A more comprehensive visualization and analysis can be found in our supplementary material.

Visualization
We provide detailed quantitative analysis in previous sub-sections, in this part, we give some visualization about the feature maps and tracking results for better understanding and qualitative analysis of our tracker. As shown in Fig. 6, our tracker (the red bounding box) can robustly and accurately track the object when facing scale change, dense objects, rotation, and occlusion compared with current SOTA trackers. We also visualize the response scores of candidate center positions of the target objects, as shown in Fig. 7. The response maps and color frames are added as one sample, and the corresponding event images are also provided for comparison and understanding only. It is easy to find that our tracker can locate the target object accurately.

Conclusion
In this paper, we present a unified network, dataset, and metric for color-event visual tracking. Our network CEU-Track novelly votes the event stream into voxels to keep the spatial-temporal information as well and unifies the multimodel tracking field with a single backbone network, canceling the complexity designing of modern multi-model tracking (e.g., modality fusion strategies, multi-backbone, and Siamese matching modules). Extensive experiments on three datasets verify the effectiveness and efficiency of our tracker. To help further research and solve the deficiency of current benchmarks, we propose a large-scale and highquality dataset COESOT for the color-event tracking community, which presents color frames, event frames, event voxels, source files, and dense annotations, etc. We also provide a toolkit for tracking evaluation and comparison with 28 baseline tracking results on COESOT. The evaluation process newly introduced a metric BOC for validating the real effectiveness and conspicuousness ability compared of exist trackers. In our future works, we will consider pretraining our backbone using paired frame and event streams in a self-supervised way to achieve higher tracking performance.

Appendix
In this supplementary material, we provide further details on various aspects of our tracking pipeline and benchmark. First, we provide additional detailed information about the proposed COESOT and other datasets in Section A. Then, more details about the implementation and difference of our CEUTrack are described in Section B. Afterward, we report more visualization and performance evaluation in Section C.

A. Details of COESOT
In this section, we provide more details about the data collection followed by a detailed description of the baseline trackers employed and finally we provide further details about the existing datasets.

A.1. Hardware for Dataset Collection
The data of COESOT are all collected by the color DAVIS346 camera with a zoom lens, as shown in Fig. 8. Therefore, our dataset contains many target objects at a far distance, while previous datasets like the FE108 [51] and VisEvent [44] datasets are not. As shown in Fig. 9, our COESOT contains more than 1300 video sequences, which is the most large-scale and modal-aligned well dataset for the current color-event community and outperforms existing datasets with a significant margin.

A.3. Datasets Used in Our Experiment
In addition to our proposed COESOT dataset, we also report our experimental results on the FE108 and VisEvent datasets. The details can be found below: FE108 Dataset. FE108 is a tracking dataset collected by a gray-scale DAVIS346 camera. Because of the characteristic of the camera, the visual frames captured are grayscale and the timestamps of events are all aligned with gray frames. Note that the videos in the FE108 dataset are all collected in indoor scenarios without the zoom lens. FE108 contains 21 object target categories, which consist of three groups: animals, vehicles, and daily goods. It annotates the target ground-truth with four attributes: low-light, high dynamic range, fast motion with and without motion blur on visual frames. Compared with our proposed COESOT, FE108 contains much fewer target categories, attributes, Figure 10. Overall performance comparison on the proposed COESOT dataset. Best viewed by zooming in. and frames. In addition, our COESOT also provides the absent labels while the FE108 dataset does not.
VisEvent Dataset. VisEvent is a tracking dataset collected by a color DAVIS346 camera without a zoom lens. The training and testing subsets contain 500/320 sequences respectively. However, with the characteristic of VisEvent, some video sequences are misalignment and some sequences lack event source files (i.e. *.aedat4). In our evaluation experiment, due to the voxelization operation, the source files are essential. Therefore, we select the 205 training sequences and 171 testing videos to evaluate the tracker's performance. For a fair comparison, the evaluation performances reported in our paper are all evaluated with the same testing video sequences. Compared with our proposed COESOT, VisEvent didn't count the target object categories and contains much fewer frames and video sequences especially filtered useless videos. Therefore, we suppose our COESOT is the largest-scale and most reliable dataset to date.

B.1. Implementation Details
For our COESOT and FE108, we trained on their own training subsets and testing on testing subsets respectively. For VisEvent, we remove the videos which only have RGB domain inputs, lack event source files, or contain misaligned timestamps, and left the training and test 205/172 videos respectively. Without any specific fine-tuning operation, the three datasets are trained with the same following setting. In the training phase, we randomly sample the template and search images in a 100-frame window. The model is trained with an AdamW optimizer with a weight decay of 1e-4 and a batch size of 32. The initial learning rate is set as 1e-4 and decays a magnitude at the 40-th epoch. The training epochs are set as 50 with sampling 60k frame pairs every epoch. The dropout rates of Transformer layers in CEUTrack are all set as zero. Note that, different from currently popular training strategies, our training process does not employ any data augmentation strategy (e.g., horizontal flip, brightness jitter, image blur, relative shift). Code is implemented on Python 3.8 and Pytorch 1.8 with an RTX-3090 GPU.

B.2. The Difference of Our Framework
As shown in Fig. 1, the mainstream framework in multimodel tracking includes the early fusion and middle fusion pipelines. In the detail of Fig. 1 (a), the early fusion tracking will first merge the RGB and event frames into one sample and then follow the classical Siamese framework to feed the fused search and template into the shared backbone network and other modules.
At the same time, the middle fusion tracking ( Fig. 1 (b)) combines the Siamese framework with the modal fusion. It divides the backbone into the non-shared parallel network and extracts four features, separately from the RGB template, event template, RGB search, and event search. Then, the feature of RGB template and event template will merge into the fused template feature based on the fusion module. Similarly, the fused search feature is also generated based on the RGB search feature and event search feature. Further, there will following some correlation/matching modules based on the fused template and search features.
Different from above all, the proposed CEUTrack drop all of the fusion design and gets a unified backbone network for RGB-Event tracking which bridge the gap between multi-modal tracking and general RGB tracking, as shown in Fig. 1 (c). The unified network merges the four multimodal inputs into one input and generates the single-stream and single-stage backbone for multi-modal tracking. Fur-  thermore, due to the similarity principle and general character, we believe our unified framework can be extended to other multi-modal tracking pipelines, (i.e., RGB-Thermal, and RGB-Depth tracking). We leave this as our future works.

C. Additional Evaluation Details
In this part, we provide more visual results and comparisons of our tracker. First, we show the attribute performance results compared with baseline trackers. Secondly, we visualize more frames of different sequences, including our accuracy and robustness, breakout ability, and failure cases. Finally, we visualize different event representation formulations for intuitive comparison.

C.1. Overall and Attributes Performance
With the detailed attributes annotation of COESOT, we build the evaluation toolkit for plotting the diagram of curves about the overall performance and attributes performance comparison. As shown in Fig. 10, the overall performance comparison of COESOT is reported about all of the baseline trackers and our CEUTrack obtains a considerable gain with all of them with respect to SR/PR/NPR in 28 baseline trackers. SR measures the rate of successfully tracked frames whose overlaps exceed the given threshold, i.e., 0.5. PR represents the percentage of frames where the center position error between the predicted bounding box and ground truth is less than 20 pixels. Since PR is sensitive to target size and image resolution, NPR normalizes the precision as in [34]. As shown in Fig. 11, our toolkit evaluate the performance of each attribute and plot the curve figures of success plot and precision plot. Our CEUTrack achieves the best accuracy performance on almost all of these challenges, like motion blur, rotation, fast motion, full occlusion, scale variation, etc.

C.2. Event Representation Visualization
As the ablation study of event representation, the event can be represented by various methods, which can also be employed as the reference input for future works. Fig. 12 shows three different event-building algorithms. The first line is the RGB frames and the second one is the most popular event count frames which add the event through the corresponding timestamps [44]. The third line is the event reconstruction frames, which build event frames from a stream of events based on a recurrent network [38]. The last one is the time-surfaces event presentation formulation [26], which describes event-based spatio-temporal features. In our proposed COESOT dataset, for all video sequences, we have provided RGB frames, the above three event formats, and our event voxel representation format. Corresponding conversion scripts are also provided for an easy start. Further, we provide the original source file with .aedat4 extension for all video sequences to facilitate future researchers, e.g. convert to new event representations.

C.3. Limitations
In real tracking scenarios, when the target remains stationary or moves very slowly under dark/overexposed, our method may be violated due to too sparse dynamic event voxels and limited color view. Moreover, fast camera movement will generate too dense events and the voxel topselection strategy may extract unsuitable search voxels, indicating that there is still unexplored room for color-event based object tracking.