{'fraction_non_alphanumeric': 0.035715557454687887, 'fraction_numerical': 0.01954919346223694, 'mean_word_length': 5.042168674698795, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
CRACT: Cascaded Regression-Align-Classification for Robust Visual Tracking

High quality object proposals are crucial in visual tracking algorithms that utilize region proposal network (RPN). Refinement of these proposals, typically by box regression and classification in parallel, has been popularly adopted to boost tracking performance. However, it still meets problems when dealing with complex and dynamic background. Thus motivated, in this paper we introduce an improved proposal refinement module, Cascaded Regression-Align-Classification (CRAC), which yields new state-of-the-art performances on many benchmarks. First, having observed that the offsets from box regression can serve as guidance for proposal feature refinement, we design CRAC as a cascade of box regression, feature alignment and box classification. The key is to bridge box regression and classification via an alignment step, which leads to more accurate features for proposal classification with improved robustness. To address the variation in object appearance, we introduce an identification-discrimination component for box classification, which leverages offline reliable fine-grained template and online rich background information to distinguish the target from background. Moreover, we present pyramid RoIAlign that benefits CRAC by exploiting both the local and global cues of proposals. During inference, tracking proceeds by ranking all refined proposals and selecting the best one. In experiments on seven benchmarks including OTB-2015, UAV123, NfS, VOT-2018, TrackingNet, GOT-10k and LaSOT, our CRACT exhibits very promising results in comparison with state-of-the-art competitors and runs in real-time.


Introduction
As one of the important problems in computer vision, visual tracking has many applications including video surveillance, intelligent vehicles, human-machine interaction, etc. Despite considerable progress made in recent years, robust tracking remains challenging because of many factors such as occlusion, distractor, scale changes, deformation, motion blur and so on [13].   [49,12]) in image (a) and our cascaded regression-align-classification (CRAC) in image (b). Best viewed in color and by zooming in.
In this paper we focus on model-free single object tracking. Specifically, given the target in initial frame, a tracker aims at locating it in all subsequent frames by determining its position and scale. Inspired by the Siamese tracking algorithm [1] and the region proposal network (RPN) [43], SiamRPN [31,30] formulates tracking as an one-shot inference problem and has attracted great attention owing to its excellent performance in both accuracy and speed. It simultaneously predicts classification results and regression offsets for a set of pre-defined anchors to generate proposals. Encouraged by the success of SiamRPN, improvement has been proposed (e.g., [49,12]) with an additional refinement process, which further regresses and classifies in parallel each proposal (see Figure 1(a)). Particularly, regression is used to adjust the locations and sizes of proposals for better accuracy, and classification to distinguish the target object from background in proposals for better robustness.
Despite improvements achieved, trackers with the above proposal refinement still fail in presence of complex background because of degenerated classification, caused by two problems: (1) In classification task, the features of proposals are directly extracted based on their locations. The inaccuracy in these locations (e.g., due to large scale changes) may contaminate the proposal features (e.g., due to irrelevant background information) and consequently degrades classification results. (2) Background appearance information, which may vary over time and plays a crucial role in distinguishing target from similar objects, is ignored in classification and may hence cause drift to distractors in background.

Contribution
Motivated by aforementioned observations, in this paper we design a new proposal refinement module to improve the robustness of visual tracking.
First, we introduce a novel simple yet effective cascade of regression-align-classification (CRAC) for proposal refinement, which is different than the parallel regression and classification utilized in existing approaches (Figure 1 (a)). This design is motivated by the fact that the offsets from box regression can serve as guidance to sample more accurate proposal features. CRAC consists of three sequential steps, i.e., box regression, feature alignment and box classification, as shown in Figure 1 (b). Specifically, box regression aims at further adjusting scales of proposals for better accuracy; feature alignment leverages offsets from box regression to better align proposals for improving feature quality; and box classification produces refined classification scores for aligned proposals. The key design in CRAC is to connect box regression and classification via an alignment step, instead of separating these two tasks. Such design enables more accurate features of aligned proposals, improving robustness of classification in refinement.
Then, to improve the robustness against background distractors, we develop an identification-discrimination component in the box classification step of CRAC. Specifically, the identifier learns offline a distance measurement and utilizes reliable fine-grained target template to select the proposal most similar to target. The discriminator, drawing the inspiration from success in discriminative regression tracking [7,8,35], learns online a discrete-sampling-based classification model using background and temporal appearance information to suppress similar objects in the proposals. By collaboration of identifier and discriminator, CRAC effectively inhibits distractors in the box classification step.
Furthermore, to enhance representation of proposals, we introduce a pyramid RoIAlign (PRoIAlign) module for proposal feature extraction. PRoIAlign is capable of exploiting both local and global cues of proposals, and hence allows CRAC to deal with target deformation and rotation.
We integrate CRAC in the Siamese tracking framework to develop a new tracking algorithm named CRACT (CRAC Tracker). CRACT first extracts a few coarse proposals via a Siamese-style network and then refines each proposal using CRAC. Then the proposal with the highest classification score is selected to be target. In thorough experiments on seven benchmarks including OTB-2015 [52], UAV123 [38], NfS [26], VOT-2018 [29], TrackingNet [39], GOT-10k [22] and LaSOT [10], our CRACT achieves new state-of-the-art results and significantly outperforms its Siamese baselines, while running in real-time. The implementation and results will be released upon publication of this work.
In summary, we make the following contributions. 1) A new cascaded regression-align-classification (CRAC) module is developed for proposal refinement to improve the accuracy and robustness in tracking.
2) A novel identification-discrimination component is introduced to leverage offline and online learning of target and background information for handling distractors. 3) A pyramid RoIAlign strategy is designed to exploit both local and global cues of proposals for further improving robustness of CRAC. 4) A new tracker dubbed CRACT is developed based on the CRAC module, and achieves new state of-the-art results on numerous benchmarks.

Related Work
Visual object tracking has been extensively researched in recent decades. In this section, we discuss the most relevant work and refer readers to [44,33,32,37] for comprehensive surveys.
Siamese Tracking. Treating tracking as searching for a region most similar to the initial target template, Siamese network has attracted great attention in tracking. The approach of [47] utilizes a Siamese network to learn a matching function from videos, and then uses it to search for the target object. Despite promising result, this approach runs slowly due to heavy computation. The work of [1] proposes a fully convolutional Siamese network (SiamFC) which efficiently computes the similarity scores of candidate regions. Owing to balanced accuracy and speed, SiamFC has been improved in many follow-ups [54,50,17,55,16,31,30]. Among them, the work of [31] introduces the SiamRPN by combining Siamese network and region proposal network [43] for tracking, achieving more accurate results with faster speed. To improve SiamRPN in dealing with distractors, the work of [57] leverages more negative training samples for learning a distractor-aware classifier. The approaches of [49,12] cascade multiple stages to gradually improve the discrimination power of classification. In addition, for more accurate result, the approaches of [51,53] integrate an additional segmentation branch into SiamRPN. More recently, anchorfree Siamese trackers [56,5,15] are proposed by predicting object bounding box offsets from a single pixel.
Cascade Structure in Tracking. Cascade architecture has been a popular framework for vision tasks, and our CRACT also shares this idea for tracking. The work of [21] regards tracking as a proposal selection task and introduces a twostep tracker in which object proposals are first extracted and then classified with an online model. The approach of [49], based on SiamRPN [31], presents a two-stage framework in which the proposals generated in the first stage are further identified and refined to choose the best one as the tracking result. The algorithm in [12] suggests a multi-stage framework that cascades multiple RPNs to improve performance of Siamese tracking.   [4,20,28] that formulate tracking as a rigid regression problem. Because of fast solution using fast Fourier transformation, this type of trackers usually run fast. Recently, motivated by powerful representation, deep feature has been applied in discriminative regression tracking [36,8,6], significantly boosting performance. To further exploit the advantages of deep features, existing methods [45,35,7,2] propose to learn a convolutional regression model with the deep learning framework, which effectively improves performance. Notably, the work of [7] introduces a novel scale estimation approach by IoU-Net [24], leading to more accurate result.
Our Approach. In this paper, we regard tracking as a proposal selection task. Our approach is related to but different from SiamRPN [31] which treats tracking as one-shot proposal selection and may suffer from large scale changes and distractors. In contrast, we propose a novel CRAC refinement module to improve proposal selection and achieve better performance. Our method is also relevant to [49,12] by sharing similar idea of refining proposals. However, unlike in [49,12] that separately performs regression and classification for refinement, our method takes a cascade structure for refinement. Furthermore, different from [49] using only local cues for proposal, we present pyramid RoIAlign to enhance proposals with both local and global information.

Tracking with Cascaded Regression-Align-Classification
In this section, we formulate object tracking as selecting the best proposal and introduce a novel simple yet effective cascaded regression-align-classification (CRAC) module to refine proposals for such purpose.
As shown in Figure 2, our method contains proposal extraction and proposal refinement. In specific, we first use a Siamese region proposal network to filter out most low confident regions and keep only a few initial proposals. Then, each proposal is fed to the CRAC module for refinement of scale and classification results. During tracking, we rank all refined proposals using the initial and refined classification results, and the proposal with highest score is selected to be the final target. To maintain strong discriminative ability of our tracker, the discriminator in box classification of CRAC is online updated using intermediate results.

Proposal Extraction
The goal of proposal extraction is to filter out most negative candidates and retain a few initial proposals similar to target object. This procedure is crucial as one of the proposals from this stage determines the final tracking result. Therefore, it is required to be robust enough to include targets of interest into proposals and to avoid contamination from background. In addition, high efficiency is desired in the proposal extraction. Taking the above reasons into consideration, we leverage Siamese region proposal network, as in [31,30,49,12], for proposal extraction.
The architecture of Siamese RPN contains two branches for target template z and search region x, respectively. As illustrated in Figure 2, using ResNet [19] as backbone, we first extract the features φ 4 (z) and φ 4 (x) after block 4 for z and x. Notice that, the feature extraction backbones for z and x share the same parameters. Then, φ 4 (z) and φ 4 (x) are fed to RPN, which simultaneously performs classification and regression for predefined anchors on search region (Please see architecture of RPN in the supplementary ma-  terial). With the classification scores and regression offsets of anchors, we generate N proposals using Non-maximum Suppression (NMS). We represent N proposals as {p i } N i=1 , and classification result of p i is denoted as c i . The loss rpn to train Siamese RPN comprises two parts including a cross entropy loss for classification and a smooth L 1 loss [14] for regression. We refer readers to [31,14] for more details.

CRAC for Proposal Refinement
Because the proposals may contain distractors and/or not be good enough to handle large object scale variations, we develop a cascaded regression-align-classification (CRAC) module that refines each coarse proposal by cascading three steps, i.e., box regression, feature alignment and box classification, for better selection. Figure 3 illustrates the architecture of CRAC. We show the detailed parameters of each component of CRAC in the supplementary material due to limited space.

Box Regression
Since only one-step regression of coarse proposals may not be sufficient to handle object scale changes, we employ an additional box regression in CRAC to further adjust locations and sizes of proposals. In specific, as shown in Figure 3, we first use pyramid RoIAlign (PRoIAligh) module (Section 3.3) to extract the feature of each proposal. In order to improve regression accuracy, we employ features from multiple layers. Particularly, we concatenate the features φ 4 (x) and φ 3 (x) after blocks 4 and 3 and use a conv layer to obtain fused feature maps φ 34 (x) (Figure 2). Afterwards, the feature f i of proposal p i is obtained through PRoIAlign as follows, As a high-level task, we aim at learning a generic box regression model. Similar to the Siamese tracking [1,31], we incorporate the target in the first frame as prior information. Likewise, we use multi-level features and obtain the initial target feature as follows,  where φ 34 (z) is fused feature maps for target ( Figure 2) and b 1 denotes initial object box. Then, the box regression offset r i of p i is obtained via where the box regression model R first concatenates f i and f init , and then applies a conv layer and three consecutive fc layers to output a 4-dimension vector r i = (r x i , r y i , r w i , r h i ). The loss reg to train the box regression model is smooth L 1 loss [14].

Feature Alignment
Proposal classification is important, as it greatly affects final proposal selection. Existing refinement method (e.g., [49]) directly extracts proposal features for classification. However, if the locations of proposals are inaccurate, their classification results may be degraded. Thanks to cascade structure of CRAC, we can alleviate this issue by aligning each proposal using offsets from box regression step. By doing so, more accurate proposal features can be used for classification.
In particular, with regression offsets r i from Eq. ( 3), we adjust location and size of p i as follows, and adjusted center coordinates of proposal p i and its width and height, respectively. Withx i ,ỹ i ,w i ,h i , we can obtain the refined proposalp i for p i , and extract more accurate feature usingp i viaf wheref i represents the aligned feature for p i . In comparison with f i , the alignedf i is more accurate (see Figure 4), which leads to better classification result. In addition, more accurate features can also benefit the training of box classification.

Box Classification
Since the proposals contain various distractors, a more discriminative classification module is desired in CRAC. Existing methods (e.g., [49,12]) learn an additional matching sub-network to further classify the proposals for better selection. Owing to more balanced training samples, the classification model in refinement is more discriminative than that for proposal extraction. Despite this, these approaches still fail in presence of hard distractors due to ignorance of background information, which is crucial for distinguishing target from similar objects.
In this work, a joint identification-discrimination module is introduced in the box classification step of CRAC. Specifically, the identifier matches offline each proposal with reliable target template to find the most similar one. Different from the identifier, the discriminator learns online a classification model by exploiting background appearance information to suppress similar objects in proposals. By collaboration of these two components, our method enjoys both reliability of target template to select most similar proposal and the strong discriminative ability to suppress the difficult distractors, leading to robust classification.
Identification. The identifier aims to compute the similarities between proposals and target template. To this end, we leverage a relation network [46] to learn offline a distance measurement between the template and a proposal owing to simplicity and efficiency, similar to [49]. Since the identifier is learned to be generic, no update is required. As an advantage, the identifier will not be contaminated by background, and thus can resist accumulated errors in discrimination part caused by model update. We compute the identification scoreν i for refined proposalp i as follows, where the identification model I first concatenatesf i and f init , and then uses a conv layer and three fc layers to obtain a 2-dimension vectorν i , as shown in Figure 3. The loss ide to train the identification is cross entropy loss.
Discrimination. Different from the identifier, the discriminator focuses on suppressing similar distractors by exploiting background appearance information. For this purpose, we develop an online discrete-sampling-based classifier D with a light network architecture of one conv and two fc layers, as illustrated in Figure 3. We compute the discrimination scoreτ i forp i as follows, where w denotes the parameters of the discrimination network.
To train discriminator, drawing inspiration from the success of discriminative regression tracking [7,8,35,20], we use the L 2 loss to learn w as follows, where X j represents the feature of a training sample, Y j is a discrete (binary) label, and λ is a regularization parameter. Notice that, unlike identifier trained on image pairs, we generate a set of discrete samples for training discriminator. We utilize the conjugate gradient method in [7] to optimize discrimination network owing to its efficiency. We refer readers to [7] for more details.
It is worth noting that, despite being relevant to discriminative regression tracking [7,8,35,20], our discriminator is different in several aspects: (1) instead of performing classification on a large search region, our method only classifies a few discrete candidate proposals, which is more efficient; (2) the labels of training samples in our method are discrete (binary), which avoids boundary effects by using soft Gaussian labels as in [7,8,35,20]; and (3) because the training samples are discrete, we can easily implement the hard negative mining by focusing more on similar object regions in background.
With Eq. (6) and Eq. (7), we compute the box classification scores i for refined proposalp i viã where α is a trade-off parameter andν + i denotes the positive classification score inν i .

Pyramid RoIAlign
Existing refinement approaches like [49] adopt RoIAlign [18] to extract proposal features. Specifically, the features of proposals are usually pooled to a fixed size (e.g., 6×6). Despite simplicity, such features may be constrained to local target information and therefore sensitive to rotation and deformation. To alleviate this problem, we introduce a pyramid RoIAlign (PRoIAlign) module, which utilizes multiple RoIAlign operations to extract proposal features at different pooling sizes. For example, for size 1×1, the proposal features contain global target information. To leverage both local and global cues, pooled features with different sizes are concatenated for fusion to derive more robust local-global proposal features. Figure 5 illustrates the architecture of our PRoIAlign module. In our implementation, the PRoIAlign module is designed to have three levels, i.e., 6×6, 3×3 and 1×1, for proposal feature extraction.

Training and Tracking
Training. The training of CRACT comprises two parts: (1) offline training of Siamese RPN, box regression and identifier, and (2) online training of discriminator in box classification. The first part is trained using image pairs, and the   Feature alignment to obtain {fi} N i=1 using Eq. (5); 10 Box classification to obtain {si} N i=1 using Eq. (9); 11 Select the best proposal to determine the target box bt using Eq. (10) ; 12 Collect training samples based on bt and update the discriminator when necessary; total training loss L = rpn + reg + ide . Similar to [31,49], the ratios of anchors are set to [0.33, 0.5, 1, 2, 3] in RPN. The intersection over union (IoU) thresholds to determine anchors as positive (greater than threshold) or negative (less than threshold) are 0.6 and 0.3. We generate up to 64 samples from one image pair for RPN training. We choose at most 16 and 32 proposals for box regression and identifier training, respectively. The IoU thresholds to determine the proposals at positive (greater than threshold) or negative (less than threshold) are both 0.5. The second part is online trained during tracking. In particular, we draw 200 positive and 1000 negative samples in the first frame for initial training. The optimization strategy for training and update follows [7] except training samples are discrete.
To improve robustness, we use hard negative mining by increasing the number of similar distractors in negative samples. Algorithm 1 summarizes the tracking with CRACT.

Experiments
Implementation. We implement CRACT in python using PyTorch [41] on a single GTX 1080 GPU with 8GB memory. We utilize ResNet-18 [19] as backbone and borrow its parameters trained on ImageNet [9]. The number N of proposals during tracking is empirically set to 10. The trade-off parameters α and β are 0.4 and 0.8, respectively. The update interval K for the discriminator is 10. n + and n − are set to 50 and 200, respectively. The learning rate of the offline training part is 10 −2 with a decay of 10 −4 . It is trained end-to-end with SGD by 50 epochs. We apply LaSOT [10], TrackingNet [39], GOT-10k [22] and COCO [34] for offline training, excluding the one under testing. The online training and update of the discriminator utilizes the strategy in [7]. Hard negative mining is used in update. Our tracker runs at 28 frames per second (fps).
NfS [26]. NfS consists of 100 sequences for evaluation on high frame rate videos. We evaluate our approach on 30 fps version. Table 3 demonstrates our result and comparison to 11 trackers. Our CRACT achieves the best result with 0.625 SUC score, which outperforms the second best DiMP-50 with 0.619 SUC score by 0.6% and the third best SiamBAN with 0.594 by 3.1%.
VOT-2018 [29]. VOT-2018 contains 60 videos for tracking. We compare CRACT with 11 trackers and Table 4  TrackingNet [39]. TrackingNet offers 511 videos for evaluation. Table 5 shows comparison results of CRACT with 7 state-of-the-art trackers. Our method achieves the best results of 0.724, 0.824 and 0.754 on PRE, NPRE and SUC LaSOT [10]. LaSOT is a recent long-term tracking benchmark. We evaluate our approach under protocol II in which 280 videos are provided for testing. Figure 6 shows our results and comparison with 9 state-of-the-arts. CRACT achieves the second best results with 0.628 normalized PRE score and 0.549 SUC score, slightly lower than the 0.642 normalized PRE score and 0.560 SUC score by DiMP-50. Compared with ATOM and SiamRPN++ with 0.499 and 0.495 SUC scores, CRACT shows clear performance gains of 5.0% and 5.4%. GOT-10k [22]. GOT-10k offers 180 challenging videos for short-term tracking evaluation. We compare CRACT to 5 trackers as displayed in Table 6. CRACT performs the best with 0.620 AO score, outperforming the second best DiMP-50 with 0.611 AO score. Besides, CRACT obtains a significant performance gain of 10.7% compared to SPM. Due to limited space, we demonstrate qualitative tracking results and comparisons in supplementary material.

Ablation Study
To verify each component in CRACT, we conduct ablative experiments on OTB-2015 [52] and NfS [26]. Cascade structure. In this paper, we introduce a novel proposal refinement module with cascade structure. We verify its effectiveness by designing a refinement module with parallel structure by removing feature alignment (see detailed architecture in supplementary material). Table 7 shows results of parallel and cascade refinement. We observe that CRACT with parallel refinement achieves SUC scores of 0.713 and 0.609 on OTB-2015 and NfS. By utilizing cascaded proposal refinement, the results are significantly improved to 0.726 (1.3% gain) and 0.625 (1.6% gain), which clearly evidences the advantage of using more accurately regressed proposals for proposal selection. Identification-discrimination. We propose a joint module of discrimination and discrimination in CRAC for proposal classification. In fact, either the identifier or discriminator can be used individually for proposal classification. However, each has advantages and disadvantages. The identifier can easily recognize the target from non-semantic distrac- Nonetheless, it has a risk of model contamination caused by update. By collaboration of identifier and discriminator, they can complement each other for better robust proposal selection. We verify the effects of individual and joint use of identifier and discriminator. Table 8 shows the comparison. Using identifier only and discriminator only achieves SUC scores of 0.715 and 0.712 on OTB-215. with joint consideration of them, the performance is significantly boosted to 0.726. Likewise, the best result of 0.625 SUC score is obtained when combining identifier and discriminator. Pyramid RoIAlign. Different from current tracker [49] using RoIAlign [18] for proposal extraction, we present a simple yet effective PRoIAlign to exploit global and local cues. Table 9 shows the results with RoIAlign and our PRoIAlign. We observe that PRoIAlign improves the SUC scores from 0.719 to 0.716 on OTB-2015 and from 0.615 to 0.625 on NfS, respectively, showing the advantage of exploring various cues in performance improvement.

Conclusion
In this paper, we propose a novel tracker dubbed CRACT for accurate and robust tracking. CRACT first extracts a few coarse proposals and then refines each proposal using the proposed cascaded regression-align-classification module. During inference, the best proposal determined by both coarse and refined classification scores is selected to be the final target. Experiments on seven benchmarks demonstrate its superior performance. In the future, we plan to improve the performance of CRACT by integrating mask segmentation into our cascade refinement.