{'fraction_non_alphanumeric': 0.03631505663643548, 'fraction_numerical': 0.012117562939826139, 'mean_word_length': 4.97974797479748, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
The Devil is in the Details: Self-Supervised Attention for Vehicle Re-Identification

In recent years, the research community has approached the problem of vehicle re-identification (re-id) with attention-based models, specifically focusing on regions of a vehicle containing discriminative information. These re-id methods rely on expensive key-point labels, part annotations, and additional attributes including vehicle make, model, and color. Given the large number of vehicle re-id datasets with various levels of annotations, strongly-supervised methods are unable to scale across different domains. In this paper, we present Self-supervised Attention for Vehicle Re-identification (SAVER), a novel approach to effectively learn vehicle-specific discriminative features. Through extensive experimentation, we show that SAVER improves upon the state-of-the-art on challenging vehicle re-id benchmarks including Veri-776, VehicleID, Vehicle-1M and Veri-Wild. SAVER demonstrates how proper regularization techniques significantly constrain the vehicle re-id task and help generate robust deep features.


Introduction
Re-identification (re-id), the task of identifying all images of a specific object ID in a gallery, has been recently revolutionized with the advancement of Deep Convolutional Neural Networks (DCNNs). In the field of Computer Vision, this revolution is most notable in the area of person re-id. Lou et al. [28] recently developed a strong baseline method that supersedes state-of-the-art person reid methods by a large margin, using an empirically derived "Bag of Tricks" to improve the discriminative capacity of DCNNs. This advancement has created a unique opportunity for the research community to develop innovative yet simple methods to push the boundaries of object re-id.
Specifically, vehicle re-id has great potential in surveillance and intelligent transportation applications. However, the task of vehicle re-id is particularly challenging since vehicles with different identities can be of the same make, model  and color. Moreover, the appearance of a vehicle varies significantly across different viewpoints. Therefore, recent DCNN-based re-id methods focus attention on regions containing discriminative information to improve model invariance to orientation and occlusion. To this end, many high performing re-id approaches rely on additional annotations for local regions that have been shown to carry identity-dependent information, i.e. key-points [41,16,17] and parts bounding boxes [11,46] in addition to the actual IDs of the objects of interest. These extra annotations help DCNN models jointly learn improved global and local feature representations and significantly boost performance [16,48] at the cost of increased model complexity. Despite providing considerable benefit, gathering costly annotations such as key-point and part locations cannot be scaled to the growing size of vehicle re-id datasets. As manufacturers change the design of their vehicles, the research community has the burdensome task of annotating new vehicle models.
In an effort to re-design the vehicle re-id pipeline without the need for expensive annotations, we propose SAVER to automatically highlight salient regions in a vehicle image. These vehicle-specific salient regions carry critical details that are essential for distinguishing two visually similar vehicles. Specifically, we design a Variational Auto-Encoder (VAE) [19] to generate a vehicle image template that is free from manufacturer logos, windshield stickers, wheel patterns, and grill, bumper and headlight designs. By obtaining this coarse reconstruction and its pixel-wise difference from the original image, we construct the residual image. This residual contains crucial details required for re-identification, and acts as a pseudo-saliency or pseudo-attention map that highlights discriminative regions in an image. Figure 1 demonstrates how residuals can highlight valuable fine-grained details needed for re-identification between two visually similar vehicles. We compute the convex combination of the original image and its corresponding residual as the input to the deep feature extractor.
The rest of the paper is organized as follows. In section 2, we briefly review recent relevant vehicle re-id works. The detailed architecture of each step in the proposed approach is discussed in section 3. Through extensive experimentation in section 4, we show the effectiveness of our approach on multiple challenging vehicle re-id benchmarks [43,22,9,27,24], obtaining state-of-the-art results. Finally, in section 5 we justify our design choices.

Related Works
Learning robust and discriminative vehicle representations that adapt to large viewpoint variations across multiple cameras, illumination and occlusion is essential for re-id. Due to a large volume of literature, we briefly review several recent relevant works on vehicle re-identification.
With the recent breakthroughs of deep learning, we can easily learn discriminative embeddings for vehicles by feeding images from large-scale vehicle datasets, such as Vehicle-ID [22], VeRi-776 [24], Veri-Wild [27] Vehicle-1M [9], PKU-VD1 & VD2 [43], CompCars [44], and CityFlow [40], to train a DCNN that is later used as the feature extractor for re-id. However, for the vehicles of the same make, model, and color, this global deep representation usually fails to discriminate between two similar-looking vehicles. To address this issue, several auxiliary features and strategies are proposed to enhance the learned global appearance representation. Cui et al. [4] fuse features from various DCNNs trained with different objectives. Suprem et al. [36] proposes the use of an ensemble of re-id models for vehicle identity and attributes for robust matching. [41,23,46,11,16] propose learning enhanced representation by fusing global features with auxiliary local representations learned from prominent vehicle parts and regions, e.g., headlights, mirrors. Furthermore, Peng et al. [31] leverages an image-to-image translation model to reduce cross-camera bias for vehicle images from different cameras before learning auxiliary local representation. Zhou et al. [50] learns a viewpoint-aware representation for vehicle re-id through learning and a viewpoint-aware attention. Similarly, [48,32] leverage attention guided by vehicle attribute classification, e.g., color and vehicle type, to learn attributebased auxiliary features to assist the global representation. Metric learning is another popular approach to make representations more discriminative. [47,2,3,21] propose various triplet losses to carefully select hard triplets across different viewpoints and vehicles to learn an improved appearance-robust representation.
Alternatively, [42,45,39] use generative adversarial network (GAN) to synthesize vehicle images with diverse orientation, appearance variations, and other attributes to augment training data for more robust training. [25,26,34,38,14,29,15] propose methods for improving the matching performance by also making use of spatio-temporal and multi-modal information, such as visual features, license plates, inter-camera vehicle trajectories, camera locations, and time stamps.
In contrast with prior methods, SAVER benefits from self-supervised attention generation and does not assume any access to extra annotations, attributes, spatio-temporal and multi-modal information.  Proposed SAVER pipeline. The input image is passed through the VAE-based reconstruction module to remove vehicle-specific details. Next, the reconstruction is subtracted from the input image to form the residual image containing vehicle-specific details. Later, the convex combination (with trainable parameter α) of the input and residual is calculated and passed to the re-id backbone for deep feature extraction. The entire pipeline is trained via triplet and cross entropy losses, separated via a batch normalization layer (BN Neck) proposed in [28].

Self-Supervised Attention for Vehicle Re-identification
Our proposed pipeline is composed of two modules, namely, Self-Supervised Residual Generation, and Deep Feature Extraction. Figure 2 presents the proposed end-to-end pipeline. The self-supervised reconstruction network is responsible for creating a per-image reconstruction template that generates the overall shape and structure of a vehicle while obfuscating discriminative details. This enables us to highlight salient regions and remove background distractors by subtracting the reconstructed template from the input image. Next, we feed the convex combination (with trainable parameter α) of the residual and original input images to ResNet-50 [12] model to generate robust discriminative features. To train our deep feature extraction module, we use techniques proposed in "Bagof-Tricks" [28] and adapt them for vehicle re-identification. "Bag-of-Tricks" offers a strong baseline that improves upon complex state-of-the-art methods.

Self-Supervised Residual Generation
In order to generate the crude shape and structure of a vehicle while removing small-scale discriminative information, we leverage prior work in image segmentation [1] and image generation [19]. Specifically, we construct a novel VAE  Fig. 3. Self-Supervised image reconstruction required for subsequent residual generation. The input image goes through the convolutional encoder and is mapped to 3-dimensional latent variable. Using the VAE re-parameterization trick, a sample from the standard multivariate Gaussian is drawn and scaled via mean µ and co-variance Σ of the latent variable. Lastly, θ is up-sampled with a convolutional decoder to generate the input image template with most fine grained details removed.
architecture that down-samples the input image of spatial size H × W through max-pooling into a latent space of spatial size H 16 × W 16 . Afterwards, we apply the re-parameterization trick introduced in [19] to the latent features via their mean and covariance. Next, we up-sample the latent feature map as proposed by [30] to prevent checkerboard artifacts. This step generates the reconstructed image of size H × W . Fig. 3 illustrates the proposed self-supervised reconstruction network.
Formally, we pre-train our reconstruction model using the mean squared error (MSE) and Kullback-Leibler (KL) divergence such that where and In Eq. 1, λ sets the balance between the MSE and KL objective functions. Also, I o and I g in Eq. 2 refer to the original and generated images respectively. Finally, in Eq. 3 M is the dimensionality of the latent features θ ∈ R M with mean µ = [µ 1 , . . . , µ M ] and covariance matrix Σ = diag(σ 1 2 , . . . , σ M 2 ), that are reparameterized via sampling from standard multivariate Gaussian ∼ N (0, We pre-train this model on the large-scale Vehicle Universe dataset, introduced in section 4.2.1, prior to training our end-to-end pipeline, as described in Section 4. This pre-training allows the reconstruction model to generalize to vehicle images with a larger variety of make, model, color, orientation, and image quality. The reconstruction model is able to capture domain invariant features that can later be fined-tuned for a particular dataset. Additionally, pretraining helps improve the rate of convergence for the end-to-end pipeline. It is important to note that unlike traditional VAE implementations, we use threedimensional latent feature maps, i.e., channel, height and width dimensions, rather than one-dimensional latent vectors, i.e., only channel dimension, for the sake of computational efficiency and reconstruction quality as we preserve more spatial information. Additionally, we scale the KL divergence coefficient when calculating the reconstruction function and back-propagating gradients to improve reconstruction quality. We further explore the effect of the KL divergence scaling factor λ in Section 5. Once the self-supervised image reconstruction network generates the input image template, we subtract the generated image from original input to obtain the residual image, i.e. I r = I o − I g .

Deep Feature Extraction
Since vehicle images reside on a high-dimensional manifold, we employ a DCNN to project the images onto a lower-dimensional vector space while preserving features that can effectively characterize a unique vehicle identity. To this end, we use a single-branch ResNet-50 architecture as our DCNN. To train this model, we use hyper-parameters suggested in [28], which are originally tailored for person re-id. These "Bag of Tricks" are shown to help a DCNN traverse the optimization landscape using gradient-based optimization methods more efficiently and result in high quality discriminative features. In particular, we observed that the following techniques significantly contribute to the performance of the vehicle re-id baseline model: 1 -Learning Rate Warm-Up: [6] has suggested increasing the learning rate linearly in initial epochs of training to obtain improved weight initialization. This is a significant contributing factor to the enhanced performance of our baseline model. 2 -Random Erasing Augmentation (REA): To better handle the issue of occlusion, [13] introduced REA with the goal of encouraging a network to learn robust representation.

-Label Smoothing:
In order to alleviate the issue of over-fitting to the training data, [37] proposed smoothing the ground-truth labels. 4 -Batch Normalization (BN) Neck: To effectively apply both classification and triplet losses to the extracted features, [28] proposed passing features from a BN layer after the triplet loss. This also significantly improves vehicle re-id performance.
The ResNet-50 feature extractor model is trained to optimize for triplet and cross entropy classification losses which are calculated as follows: and In Eq. 4, B, b i , a, γ, P(a) and N (a) are the total number of batches, i th batch, anchor sample, distance margin threshold, positive and negative sample sets corresponding to a given anchor respectively. Moreover, x a , x p , x n represent the ResNet-50 extracted features associated with anchor, positive and negative samples. In addition, function d(., .) calculates the Euclidean distance of the two extracted features. Note that in Eq. 4, we used the batch hard triplet loss [13] to overcome the computational complexity of calculating the distance of all unique triplets of data points. Here we construct batches so that they have exactly K instances of each ID used in a particular batch, i.e. B is a multiple of K. In Eq. 5,x i refers to the extracted feature for an image belonging to class i after passing through the BN Neck layer. Furthermore, c(x i ), b i are the weight vector and bias associated with class i in the final classification layer respectively; N and C represent the total number of samples and classes in the training process.

End-To-End Training
After pre-training the self-supervised residual generation module, we jointly train the VAE and deep feature extractor. We compute the convex combination of input images and their respective residuals using a learnable parameter α, i.e. I c = α × I o + (1 − α) × I r , allowing the feature extractor network to weight the importance of each input source. Moreover, the end-to-end training helps the entire pipeline adapt the residual generation such that it is suited for the re-id task. In summary, the loss function for end-to-end training is the following: In Eq. 6, the scaling factor η is empirically set to 100.

Experiments
In this section, we first present the different datasets on which we evaluate the proposed approach and describe how vehicle re-identification systems are evaluated in general. Next, we present implementation details for the proposed selfsupervised residual generation, deep feature extraction and end-to-end training. Finally, we report experimental results of the proposed approach.

Vehicle Re-Identification Datasets
We evaluate SAVER on six popular vehicle re-id benchmarks, including VeRi-776, Vehicle-ID, VeRi-Wild, Vehicle-1M and PKU-VD1&VD2. Table 1 presents the statistics of these datasets in terms of the number of unique identities, images and cameras. Additionally, we highlight four additional datasets of unconstrained vehicle images, including CityFlow, CompCars, BoxCars116K [35], and StanfordCars [20], used in the pre-training our self-supervised reconstruction network.  Re-id systems are commonly evaluated using the Cumulative Match Curve (CMC) and Mean Average Precision (mAP). A fixed gallery set is ranked with respect to the similarity score, e.g., L 2 distance, of each image with a given query image. CMC@K measures the probability of having a vehicle with the same ID as the query image within the top K elements of the ranked gallery. It is a common practice to report CMC@1 and CMC@5. Similarly, mAP measures the average precision over all images in a query set.

Implementation Details
Here we discuss the implementation of both the self-supervised residual generation and deep feature extraction modules. In general, we resize all the images to (256, 256) and normalize them by a mean and standard deviation of 0.5 across RGB channels before passing them through the respective networks. Moreover, similar to [17], we pre-process all images with the Detectron object detector [7] to minimize background noise. All experiments use a fixed random number generator seed to ensure that all results are reproducible.

Self-Supervised Residual Generation
To pre-train the self-supervised residual generation module, we construct a large-scale dataset, namely Vehicle Universe. We specifically consider vehicles from a variety of dataset distributions to improve the robustness of our model. We utilize data from several sources, including CompCars, CityFlow, PKU-VD1&VD2, Vehicle-1M, Vehicle-ID, VeRi-776, VeRi-Wild, CompCars, StanfordCars, and BoxCars116K. In total Vehicle Universe has 3706670, 1103404 and 11146 images in the train, test and validation sets respectively.

Deep Feature Extraction
As mentioned in section 3.2, we use ResNet-50 for feature extraction, employing the 'Bag of Tricks' [28]. In all of our experiments the learning rate starts from 3.5e − 5 and is linearly increased with the slope of 3.1e − 5 in the first 10 epochs for the purpose of warm-up. Afterwards, it is decayed by a factor of 10 every 30 th epoch. In total, the end-to-end pipeline is trained for 150 epochs via Adam [18] optimizer. Furthermore, we use an initial value of α = 0.5 for convex combination and γ = 0.3 for the triplet loss in Eq. 4.

Experimental Evaluation
In this section, we present evaluation results of the global appearance model (baseline) and global appearance model augmented with self-supervised attention (SAVER) on different re-id benchmarks discussed in section 4.1. Table 2 reports the evaluation results on Veri-776, a popular dataset for vehicle re-id. SAVER improves upon the strong baseline model. Most notably, SAVER gives +1.5% improvement on the mAP metric. We note that α in the convex combination of the input and residual saturates at 0.96, which means the model relies on 96% percent of the original image and 4% of the residual to construct more robust features.  Table 3 presents results of baseline and SAVER on VehicleID test sets of varying sizes, i.e. small (S), medium (M) and large (L). Performance improvement of +1.0% in CMC@1 over the baseline model can be observed for all the test splits. Note that since VehicleID does not include a deterministic test set, we randomly generate test sets of varying sizes and keep them fixed across all experiments for the sake of consistency. To better demonstrate the discriminating capability of the proposed model, we visualize the attention map of both baseline and the proposed SAVER models on an image of this dataset using Gradient Class Activation Mapping (Grad-CAM) [33]. In Figure 4, it is clear that SAVER is able to effectively construct attention on regions containing discriminative information such as headlights, hood and windshield stickers.

Veri-Wild
Evaluation results on the VeRi-Wild dataset are presented in Table 4. Notably, our proposed residual generation model is improved upon    Table 5 reports the results of baseline and the proposed methods. Similar to VehicleID dataset, Vehicle 1M does not include fixed evaluation sets, therefore we randomly construct the evaluation splits and keep them fixed throughout the experiments. With the value of α = 0.98 the proposed selfsupervised residual generation module improves upon the baseline model in all metrics across all evaluation sets. Table 6 highlights evaluation results on both PKU-VD datasets. Similar to most re-id datasets, VD1&2 also have S/M/L evaluation sets. However, due to the extreme size of these data splits, as shown in Table 1, we are only able to report numbers on the small evaluation set. The performance of SAVER is comparable to our baseline model. Moreover, the final value of α = 0.99 indicates that baseline models is already very strong, and has almost no room for improvement. We can conclude that our performance on these data sets are saturated. Qualitatively, in Figure 5 we show two failure cases of SAVER on these datasets. Note that how extremely similar these images are and it is nearly impossible to differentiate them based on only visual information.

State-of-the-Art Comparison
In this section, we present the latest state-of-the-art vehicle re-id methods and highlight the performance of the proposed SAVER model. Table 7 reports the state-of-the-art on re-id benchmarks. It can be seen that our proposed model despite its simplicity, surpasses the most recent state-of-the-art vehicle re-id works without relying on any extra annotations or attributes. For the case of Veri-776 and Veri-Wild datasets, we also try the method of re-ranking suggested in [49] and achieved considerable mAP scores of 82.0 and 87.0 respectively.

Ablation Studies
In this section, we design a set of experiments to study the impact of different neural network architectures on the quality of reconstructed images, and also understand the impact of key hyper-parameters. In addition, we are interested in understanding how we can maximally exploit the reconstructed images in deep feature extraction. The experimental results of the reconstruction network are evaluated on the Vehicle Universe dataset, and experiments regarding the deep feature extraction module are evaluated on VeRi-776 and Vehicle-ID datasets.

Effect of Different Reconstruction Architectures
Here, we study the reconstruction quality of Auto-Encoder (AE) [1], VAE [19], and GAN [8] methods. Moreover, we study the use of Bilateral Filtering (BF) as a baseline for texture smoothing, subsequent residual generation and vehicle re-id. Figure  6 qualitatively illustrates the reconstruction of each method for a given vehicle identity. We notice that both the AE and GAN models attempt to recreate finegrained details, but often introduce additional distortions. Specifically, the GAN model generates new textures, modifies the logo and distorts the overall shape of the vehicle. As a result, GANs produce sharper images with various artifacts that diminish the quality of the residual image required by the re-id network. Also note that although bilateral filter attempts to smooth images, it is unable to remove the critical details needed in residuals and vehicle re-id. The VAE is able to reconstruct the image by removing minute details and smoothing out textures. As a result, the VAE is able to generate the detailed residual maps needed for our proposed re-id method. Table 8 presents evaluation metrics on Veri-766 and VehicleID for each of the generative models and bilateral filtering.  Figure 7 demonstrates how larger values of λ result in a more blurry reconstruction. Intuitively, this parameter offers a natural level for balancing the reconstruction quality of fine-grained discriminative features. As λ approaches 0, our VAE model approximates the reconstruction quality of a traditional Auto-Encoder. Empirically, we found that λ = 1e−3 produces high quality vehicle templates, while removing discriminative information across all datasets.

Incorporating Residual Information
To effectively exploit complimentary information provided by the residuals, we designed a set of four additional experiments to evaluate the performance of these models on the Veri-776 and Vehicle-ID datasets. These experiments are conducted as discussed bellow: A. We only feed the VAE reconstruction as input to the re-id network. The purpose of this experiment is to understand how much critical information can be inferred from the VAE reconstruction. B. We only feed the residual image into the re-id pipeline. In this experiment we are interested to find out how much identity-dependent information can be extracted from a residual map. C. We use the residual maps to excite the actual image of the vehicle through point-wise matrix multiplication. D. We concatenate the residual image with the actual input image. Therefore, in this experiment we feed a six-channel image to the feature extraction module. Table 9 presents the results of experiments A to D and highlights their performance against the baseline and SAVER models. In experiment A, the deep feature extractor is trained using the reconstructed image from the VAE. Intuitively, this method provides the lowest performance since all discriminating details are obfuscated. Interestingly, experiment B, training a deep feature extractor using only residual images, is able to perform nearly as well as our standard baseline. This reaffirms the idea that local information is essential for vehicle re-id. Experiment C performs considerably worse than the baseline model, indicating that point-wise multiplication with the sparse residual removes key information. Lastly, experiment D performs lower than our baseline. This can be attributed to the ImageNet [5] weight initialization, which is not well suited for six-channel images.

Conclusion
In this paper we have shown the benefits of using simple, highly-scalable network architectures and training procedures to generate robust deep features for the task of vehicle re-identification. Our model highlights the importance of attending to discriminative regions without additional annotations, and outperforms existing state-of-the-art methods on benchmark datasets including VeRi-776, Vehicle-ID, Vehicle-1M, and VeRi-Wild.