{'fraction_non_alphanumeric': 0.025254470426409905, 'fraction_numerical': 0.007317744154057772, 'mean_word_length': 5.109579831932773, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Self-aligned Spatial Feature Extraction Network for UAV Vehicle Re-identification

Compared with existing vehicle re-identification (ReID) tasks conducted with datasets collected by fixed surveillance cameras, vehicle ReID for unmanned aerial vehicle (UAV) is still under-explored and could be more challenging. Vehicles with the same color and type show extremely similar appearance from the UAV's perspective so that mining fine-grained characteristics becomes necessary. Recent works tend to extract distinguishing information by regional features and component features. The former requires input images to be aligned and the latter entails detailed annotations, both of which are difficult to meet in UAV application. In order to extract efficient fine-grained features and avoid tedious annotating work, this letter develops an unsupervised self-aligned network consisting of three branches. The network introduced a self-alignment module to convert the input images with variable orientations to a uniform orientation, which is implemented under the constraint of triple loss function designed with spatial features. On this basis, spatial features, obtained by vertical and horizontal segmentation methods, and global features are integrated to improve the representation ability in embedded space. Extensive experiments are conducted on UAV-VeID dataset, and our method achieves the best performance compared with recent ReID works.


I. INTRODUCTION
ENEFITING from the rapid development of microelectronics, navigation and communication technologies, UAV technology has made great progress, and are widely used in both military and civil fields, including air reconnaissance, patrol security and urban planning. The high-quality optical images obtained by UAV are used for computational analysis to realize object detection and scene classification [1]- [3], which is of great significance in the field of intelligent security. Vehicle re-identification (ReID), aiming at locating query vehicles from gallery set accurately, works as a key technology for social public security and smart city construction.
Existing ReID works mainly focus on datasets collected by surveillance cameras, which possess fixed locations and limited viewpoints. It is the key to mine distinguishing characteristics for vehicle ReID. Many fine-grained feature extraction methods have emerged consequently, which can be generally divided into three categories: attention mechanism-based [4], regional feature-based [5] and component feature-based [6] [7]. Attention mechanism can help models highlight valid features and has been widely used for image processing. Considering the imbalance of discriminative features in different spatial locations and different channels, spatial and channel attention network (SCAN) was proposed [4]. What's more, as local region conveys more distinctive visual cues, Region-Aware deep Model (RAM) also extracts features from a series of regions [5], to encourage the deep model to learn more discriminative features. Besides, some of works utilize detailed annotation to find out key-parts positions to highlight effective component characteristics [6], including window, light and brand.
Vehicle ReID for UAV remote sensing images is still in the exploration stage in terms of methods and datasets. Teng contributed a novel dataset called UAV-VeID as shown in Fig.1 [8], proposed a viewpoint adversarial training strategy and a multi-scale consensus loss to promote the robustness and discriminative power of learned deep features simultaneously. From Fig. 1, we can see that vehicle images sharing the same color and type demonstrate excessive similarity but variable vehicle orientation, which raises a high demand for efficient feature extraction method. Whereas regional feature-based methods and component feature-based methods have the limitations of high dependence on input alignment and detailed annotations respectively, which are not suitable for the ReID in UAV application. According to the above analysis, this letter proposes a selfaligned spatial feature extraction network (SANet), which can make deep network focus on feature extraction among a certain region and improve its representation ability. The primary contributions of our research are summarized as: on the one hand, we introduce a network with three branches to integrate global and spatial features, for sake of extracting fine-grained features. On the other hand, we introduce a self-alignment module and design loss function to align input images, which makes it feasible to segment images according to a certain standard without viewpoint annotations.

II. METHODOLOGY
As mentioned, this letter aims to extract effective finegrained features without relying on annotations. Therefore, we design the whole structure and the corresponding loss function to meet it.

A. Overall Framework
Our deep network architecture consists of three branches: global branch, upper/lower branch, and left/right branch. Obviously, the former is used to extract global features, and the latter two are used to extract spatial features in different directions. Input images have different deep features after three parallel network branches, and we concatenate them to obtain the final representation vector. The overall architecture is illustrated in Fig. 2.
Global Branch: It is worth mentioning that the first four blocks in Resnet-50 are selected as shared shallow networks. In addition, the global branch is composed of block_5 in Resnet-50, pooling layer and 1×1 convolution layer, which is used to reduce the dimension of embedded features.
Top/down Branch: After shared shallow networks, there is a self-alignment module in this branch to convert the input images to a uniform orientation. What's more, the step size of spatial feature extraction is set to 1 for the sake of improving the resolution of feature maps. Then, we get the top and down spatial features in the way of horizontal segmentation.
Left/right Branch: It possesses the same architecture as another spatial branch and shares the parameters of the selfalignment module with it. In particular, we use vertical segmentation to obtain the left and right spatial features.

B. Self-alignment Module
Considering the gigantic diversity in vehicle orientations, it is fundamental to align input images for extracting spatial features. This part is introduced to implement spatial transform [9], which can convert the input to a uniform orientation without relying on the viewpoint annotations. Consequently, it is feasible to divide vehicle areas in a fixed manner in the subsequent network.
The self-alignment module takes the output feature maps of shallow shared neural networks as input, and outputs the selfaligned feature maps. The calculation process could be depicted as follows. Firstly, a customized localisation network utilizes the input feature maps U to generate the adaptive transform parameters  : , where loc f is the localisation network function and loc W is corresponding parameters. In this letter, the localisation network consists of two convolutional layers and two fully connected layers as shown in Fig. 2. Then the grid generator creates a regular sampling grid denotes the pixels on the output feature maps V . Therefore, the spatial transform between feature maps can be formulated as: ( ) 11 12 13 where ( ) xy are the coordinates in the input feature maps U and output feature maps V respectively. The main demand of spatial transform from UAV's perspective is to achieve self-alignment by rotating vehicle targets, therefore affine transformation is selected in this letter to meet this requirement. Finally, V could be obtained by means of differential image sampling. It is obvious that annotations are not necessary during aforementioned computational process. The aligned feature maps are more suitable for UAV vehicle ReID tasks when extracting spatial features.

C. Objective Function
Firstly, we regard the ReID task as a multi-classification problem, and carry out classification loss constraints on the features extracted from the three branches respectively: Secondly, it is impossible to achieve self-alignment of input images only by triplet loss constraint on global features, which can be proved by related ReID methods who employ both triplet loss and classification loss. Therefore, in addition to the metric loss g tri L for global features, this letter also sets corresponding metric loss for top, down, left and right spatial features respectively, which can be formulated as: The other three losses d tri L , l tri L , r tri L and gs tri L + have a similar form, which encourages the distance of positive pairs to be smaller than negative sample pairs by a threshold m . Using triplet loss functions to make the segmented spatial features gather in the embedded space, the self-aligned module can generate the adaptive parameters according to images with variable orientations, so as to guarantee the consistency of spatial features. In other words, the triplet loss functions designed for spatial features replace the viewpoint annotations to some extent.

D. Implementation Details
As mentioned above, pre-trained Resnet-50 is selected as backbone network. Input images are resized to 256 256  , and two methods of random erasing and color jitter are adopted. The margin of the triplet loss is set to 0.3, and the batch size is set to 32 8 4 = , indicating that 4 images of 8 vehicle ids are randomly selected in each batch. Adam optimizer with an initial learning rate of 0.0005 is adopted, which is adjusted according to cosine annealing strategy dynamically. Since the convergence result of self-aligned module in the network has a great impact on network performance, the learning rate of the spatial transform module of spatial feature extraction branch is set as 0.0005×0.05 to avoid the failure of convergence due to excessive initial learning rate. In the process of training, considering that Resnet-50 loads pre-training parameters, we only optimize the parameters of spatial transform module, 1×1 dimensional reduction convolution layer and the full connection layer during the first 10 epochs. Experiments are implemented based on NVIDIA 2080Ti GPU with 12GB RAM.

III. EXPERIMENTS
In this section, abundant experiments have been carried out on UAV vehicles dataset to testify the effectiveness of our method. Firstly, we briefly introduce necessary information about UAV-VeID [8] dataset and primary evaluation metric employed in this letter. Secondly, we analyze carefully the influence of the spatial features by conducting empirical studies, and the results of spatial transformation by visualization as well. At last, comparison experiments are conducted on UAV-VeID dataset and compared with recent works.

A. Datasets and Evaluation Metrics
Datasets. We firstly review necessary information about UAV vehicle dataset used in this letter, which is collected by UAV-mounted cameras.
UAV-VeID is constructed from video sequences captured by drones in different locations, backgrounds and lighting conditions, such as highway intersections, road crossings, parking lots, etc. The flight altitude of UAV is between 15 and 60 meters, and the vertical angle of the camera is between 40 and 80 degrees, resulting in the variable sizes and perspectives of vehicle targets. UAV-VeID dataset contains 41,917 images of 4,601 vehicles, of which the corresponding images of 1,797 vehicles, 596 vehicles and 2208 vehicles are used as training set, verification set and test set, respectively.
Evaluation Metrics. During the test, the Euclidean distance between the embedded features of query set and gallery set is firstly calculated and sorted in ascending order, and then the distance metric matrix is returned. Because there is only one ground truth match for a given query in UAV-VeID test set, we use only CMC-k to evaluate method performances, showing the probability of correct matching in the top-k ranked retrieved results. CMC-k calculates the mean top-k accuracy of all queries, which can be calculated as: Since it only considers the first match in evaluation process, it is accurate for datasets where there is only one ground truth match for a given query, such as UAV-VeID dataset.

B. Model Analysis
In this subsection, we analyze the number of spatial feature blocks and then visualize the results of self-alignment.
The empirical study on spatial feature blocks of our SANet is implemented on UAV-VeID test set and the results are exhibited in Table Ⅰ, where M denotes the number of blocks obtained during segmentation in the spatial branch of SANet, dim represents the dimension of integrating global and spatial features. It can be seen that there is little difference in ReID performance under the three kinds of spatial feature blocks, which can be interpreted as the high-dimensional features may contain more redundant information. Considering that in the process of vehicle ReID, CMC-1 evaluation metric has more practical application value, and the lower feature dimension is conducive to realizing vehicle ReID between larger databases. Therefore, we select SANet when M is 2 as the final network structure.
To testify whether the introduction of the self-alignment module achieved the expected effect, we randomly select several vehicle images of UAV-VeID test set and record the spatial transform parameters regressed by localisation network. Then the parameters are directly applied to the input image to verify the result of self-alignment, as illustrated in Fig. 3. As we can see, for input images with changeful views, the method proposed in this letter basically realizes unsupervised selfalignment, and can uniformly convert the input images to the "upper right" orientation. This result can provide guarantee for subsequent spatial feature extraction. Fig. 3. Visualization of self-alignment results. Each row shows two groups of vehicle images before and after the spatial transformation.

C. Comparison with Recent Works
As mentioned above, this letter uses CMC-k index to evaluate the performance of ReIDs algorithm on UAV-VeID test set, and compares it with several typical algorithms, as shown in table Ⅱ. Among them, baseline model is trained by classification and triplet loss with ResNet-50 as backbone. Obviously, our method (SANet) achieves the best CMC-1 accuracy of 74.94%, outperforming other compared methods. It is interesting to observe that, CN-Nets [10], designed for finegrained image retrieval with a coarse-to-fine framework, shows a better performance compared with RAM [5] and SCAN [4]. Among them, RAM obtains regional features through horizontal segmentation directly, which is not suitable for the situation where vehicle orientations cannot be determined from the perspective of UAV. SCAN takes advantage of attentional mechanism to extract regional features but performs poorly. This can be interpreted as algorithms designed for vehicle ReID tasks on traditional surveillance videos do not consider the viewpoint variety in UAV-VeID dataset. VSCR [8] and AM+WTL [11] utilize perspective annotations and color prior information respectively. The former uses adversarial learning to generate features that are robust to perspective changes, while the latter emphasizes the target and reduces the background through attention mask. However, both of them are not as effective as method proposed in this letter, which indicates that correct fine-grained feature extraction is more conducive to distinguishing similar vehicles. In general, SANet relies on no detail annotation and achieves the optimal performance of UAV vehicle ReID. More intuitively, Fig. 4 shows the CMC curves of each algorithm on the UAV-VeID test set, which shows the probability of correct results in the top k images in the ReID results. Therefore, the closer the CMC curve is to the upper left corner, the stronger the performance of the corresponding algorithm is. The figure shows that the effect of SANet algorithm is generally superior to other comparison algorithms. Specifically, the hit rate of SANet Vehicle Association before rank-10 is higher than VSCR, and the performance of the two algorithms after rank-10 is similar. validity of our methods. The more vehicle features with the same ID in the embedding space gather, the more distinctive and robust the features learned by this network are. In this letter, vehicle images of 15 IDs in the UAV-VeID test set are randomly selected. As shown in Fig. 5, t-SNE [12] dimension reduction method is adopted to reduce the obtained 2560dimensional representation vector to 2-dimensional vector. It can be seen that algorithm proposed in this letter can make multiple images with the same ID more compact in the embedded space, which is more conducive to distinguish similar vehicle targets. At last, we demonstrate some vehicle ReID results on UAV-VeID in Fig. 6, where the baseline and the SANet are compared. As we can see, our proposed method has better capability to distinguish the match results from similar false positives in gallery set compared with our baseline model. It demonstrate that our model learns more efficient and robust vehicle features.

IV. CONCLUSION
In this letter, we analyzed the characteristics of vehicle images from the perspective of UAV firstly. Aiming at the variability of vehicle target orientation, we introduced self-alignment module and designed triplet loss with spatial features to realize the alignment of input images without annotations. On this basis, we formulated a vertical and horizontal segmentation method to extract spatial features, and integrate global features and spatial features simultaneously to improve the ability of representation in the embedded space. Extensive experiments verified that our method achieved the best performance on UAV-VeID dataset compared with recent works. In general, our model proposed in this letter does not rely on any additional annotation but achieves optimal performance in terms of evaluation metrics, which is more potential to be applied to real-world scenarios.
V. REFERENCE