{'fraction_non_alphanumeric': 0.0332343018278866, 'fraction_numerical': 0.012236265672994613, 'mean_word_length': 4.990950226244344, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Multi-Level Attention for Unsupervised Person Re-Identification

The attention mechanism is widely used in deep learning because of its excellent performance in neural networks without introducing additional information. However, in unsupervised person re-identification, the attention module represented by multi-headed self-attention suffers from attention spreading in the condition of non-ground truth. To solve this problem, we design pixel-level attention module to provide constraints for multi-headed self-attention. Meanwhile, for the trait that the identification targets of person re-identification data are all pedestrians in the samples, we design domain-level attention module to provide more comprehensive pedestrian features. We combine head-level, pixel-level and domain-level attention to propose multi-level attention block and validate its performance on for large person re-identification datasets (Market-1501, DukeMTMC-reID and MSMT17 and PersonX).


Introduction
Person re-identification faces a heavy label dependency problem like many other deep learning problems. The high annotation cost also limits the development of existing person re-identification methods. Therefore, semisupervised/unsupervised deep learning methods are attracted by more and more researchers as they do not rely too much on data annotation. A common semi-supervised approach is to pre-train the model supervised on a labeled dataset, then enhance the generalization of the network in some ways, and transfer the network to a dataset without labels for unsupervised learning Zhong et al., 2019]. Usually, these approaches are also called unsupervised domain adaptation (UDA) for person re-identification. Another fully unsupervised learning (FUL) approaches [Ge et al., 2020b;Lin et al., 2019; means that they do not use any annotated information, and the features extracted by the neural network are used to obtain pseudo-labels by clustering algorithms (such as K-means [Ji et al., 2020] and hierarchical clustering algorithm [Lin et al., 2019]), then the pseudo-labels guide the network to mine more accurate pedestrian features. In general, the UDA methods perform better than the FUL methods due to the introduced supervised information on the source datasets. However, the UDA approaches are still limited by complex training processes and unignorable differences between the source and target domains, which prevent them from being a universal solution.
As a sub-problem of image retrieval, the essence of person re-identification is how accurately pedestrian features are matched. Therefore, how to get a neural network that can extract more accurate pedestrian features is one of the keys to deep learning-based person re-identification methods. In supervised learning, relying on the constraint of pedestrian labels, researchers have designed methods such as part features [Guo et al., 2019] and attention mechanism [Zhang and Yang, 2021;He et al., 2021] to extract more accurate pedestrian features. Since the multi-headed self-attentive models represented by transformer [Vaswani et al., 2017] and VIT [Dosovitskiy et al., 2020] have been proposed, the self-attentive mechanism has been accepted by more and more researchers as an effective method for extracting more discriminative features without introducing manual partition and additional annotated information. However, due to the specificity of person re-identification data, pedestrian images captured by same camera may often contain similar backgrounds. Without the label constraint, it leads the unsupervised model with attention module to treat the background information as classification basis incorrectly, which affects the differentiation of pedestrian features and seriously reduces the accuracy of re-identification.
To address the above problems, we design a Multi-Level Attention (MLA) structure based on cluster contrast for unsupervised person re-identification. Specifically, we design an attention structure that combines three different scales: pixel-level, head-level, and domain-level. Head-Level Attention (HLA), which is known as Multi-Headed Self-Attention (MHSA), they are usually used to process large scale images. And on the small-sized pedestrian images, the normal position encoding cannot well divide the image, meanwhile, because of the specificity of the pedestrian images, HLA cannot get the effective attention region. Therefore, before the features input into HLA, we use pixel-level attention (PLA) to obtain weights of each pixel with its surrounding pixels and use them to constrain HLA to limit the attention region on the pedestrians themselves. Considering the specificity of pedestrian data, where all targets are pedestrians, we use domainlevel attention (DLA) to fuse pedestrian features across identities to better distinguish between foreground (pedestrian) and background of images, which supplements image-level (both pixel-level and header-level) attention.
Overall, the MLA can optimize the extracted features and improve the clustering results; on the other hand, better clustering results can also guide the model to extract more accurate pedestrian features. Eventually, our method achieves competitive results compared to other state-of-the-art unsupervised person re-identification models. The following contributions are included in our paper.
• We constrain the multi-head self-attention module by adding a Pixel-Level Attention (PLA) module to make the Head-Level Attention (HLA) module applicable to person re-identification data. • We design and add a Domain-Level Attention (DLA) module for the whole dataset to improve the effectiveness of PLA and HLA by using the specificity of person re-identification data. • With a combination of pixel-level, head-level, and domain-level attention, we successfully applied the attention module to unsupervised pedestrian re-identification task and achieved competitive results.  [Fan et al., 2018] used the K-means algorithm for clustering, and they raised the confidence limit of the clustering similarity considering that the features extracted by the model were coarse at the early stage of training. Lin et al. [Lin et al., 2019] use hierarchical clustering, alternating the training and clustering steps, combining a fixed proportion of images into clusters at each iteration and constraining the distribution of clusters in some way. Dai et al. [Dai et al., 2021] also used an iterative approach, but used the DBScan algorithm for clustering and improved the update strategy of the memory dictionary during loss calculation.

Transformer and attention mechanism
Before the transformer structure [Vaswani et al., 2017] was proposed, many person re-identification methods consciously used attention mechanisms to obtain more discriminative features. Zheng et al.  trained an additional affine transformation module to actively crop the pedestrian region of interest. Zhou et al. [Yong et al., 2020] proposed an interpretable attention-based part model, learning the correlation among pedestrian parts. While ViT [Dosovitskiy et al., 2020] enables transformer to handle image input, ResT [Zhang and Yang, 2021] frees transformer from the limitation of input image size and makes it more flexible to work with arbitrary size images. The proposal of external attention [Guo et al., 2021] provides the idea of using the prior knowledge of the data. TransReID [He et al., 2021] applies the complete transformer structure to person re-identification networks for the first time and achieves state-of-the-art performance through the explicit position encoding. Although the transformer and attention mechanism have achieved encouraging results in person re-identification tasks, most of the person re-identification networks using the attention module are based on supervised training data. It means that data labels are necessary for the model training. Currently 2020] have designed an attention module that combines spatial attention and channel attention and achieved state-of-theart person re-identification performance with attention mechanism under unsupervised conditions.

Method
Attention-based methods rely on the constraint of labels to guide the neural network to pay more attention to regions of the image that have distinguishing features. While unsupervised learning happens to lack the necessary label information, therefore to embed the self-attention module under the unsupervised learning condition requires the high performance of the baseline method itself. We added the attention module based on transformer structure to the hierarchical clustering-based model with low accuracy, and the results are shown in Table 1, where the performance decreases rather than increases on these datasets. We illustrate the training strategy proposed in the paper as Figure 1. First, as the orange arrow indicates, the unlabeled images are passed through the neural network to obtain the unlabeled features. Then the pseudo-labels are obtained by clustering using the DBScan algorithm, as shown by the blue arrows, and the loss function is calculated by the constraints of the pseudo-labels to optimize the parameters of the neural network. In particular, we modify the last ResBlock of the ordinary ResNet by using three attention modules with different scales instead of the original convolutional layers, and their specific structures are detailed in Sections 3.1, 3.2 and 3.3.
The DBScan algorithm [Ester et al., 1996] requires two manually set parameters: the clustering radius and the minimum cluster size M inP ts. In our experiments, we set to 0.4 and M inP ts to 4 to achieve the best results. In the training phase, referring to the method proposed in the paper [Dai et al., 2021], the classification weight matrix is replaced by a memory dictionary, which is initialized by randomly selected sample features from each cluster and updated by the strategy of batch hard during training. Finally, we use the following ClusterNCE Loss [Dai et al., 2021] as the loss function.
where the feature to be classified is x and the temperature parameter τ is used to control the interval between clusters. The smallest loss value is obtained if x is most similar to the i-th cluster feature C i in the memory dictionary among the total K identities.

Head-level attention
While the transformer module has proved its great performance on various machine learning tasks, it is also widely known for its huge memory cost, especially in computer vision. Generally, in the case of insufficient memory space, the batch size of training data needs to be sacrificed to accommodate large and complex models. However, due to the storage and update requirements of the dynamic memory dictionary, the batch size in training must guarantee a certain size to select effective batch hard samples. BoTNet [Srinivas et al., 2021] provides us with an effective solution. Srinivas et al. argue that replacing the convolutional layer with a transformer structure on only the last residual block close to the semantic features can improve the performance of the whole model without significantly increasing the network parameters.
As shown in Figure 2, we use three 1 × 1 convolutional layers as W q , W k and W v to obtain q (query), k (key) and v (value) respectively. In which, q is matrix multiplied with k to obtain the weights corresponding to v. Meanwhile, in VIT [Dosovitskiy et al., 2020], position embedding is manually assigned to each patch, similarly, we multiply position encoding with q to obtain this similar correspondence. To complete the embedding of the position encoding, we sum this correspondence with the weight information (q k) elementwisely. After softmax, the weight information containing the position encoding is matrix multiplied with v to output the final head-level attention features.

Pixel-level attention
In BoTNet [Srinivas et al., 2021], the position encoding pos is obtained by summing two matrices obeying standard normal distribution element-wisely. This position encoding achieves global (all2all) self-attention on the 2-D feature map, but it is not suitable for the person re-identification. Compared with multi-target classification, pedestrian images usually contain only one target to be classified and the size of the image is small, and all the backgrounds belong to distracting information. Meanwhile, pedestrian images captured by the same camera contain similar backgrounds, so under unsupervised conditions, all2all attention will make the network pay more attention to image regions unrelated to pedestrians and use this distracting information as the clustering basis, leading to worse clustering results. Therefore, we believe that there must be an initial weight assignment between the foreground and background of the pedestrian image in the features before the position encoding information is embedded into the features. Considering the baseline model already has a good performance, we consider that the original features already contain the initial distinction between foreground and background. Therefore, by assigning weights pixel-by-pixel to the image features before inputting them into the head-level attention module, we generate a new feature map that stores attention information with pixel-level precision, which we call pixel-level attention (PLA).
First, let us review the ViT [Dosovitskiy et al., 2020] model, where the input image is divided into N tokens, and a variable is added to each token to encode the position. When the input token is x and the position is encoded as E pos , the input with position encodingx can be expressed as: (2) where x ∈ N and E pos ∈ N . However, these position labels are built on the division of the whole input image, and the size and number of divisions are fixed. For pedestrian images, this division obviously has great limitations. Therefore, we divide the image by pixels, each token x represents one feature pixel and replace the position encoding with weight, then the above equation can also be seen as pixel-wise attention which is encoded by weights. In general, we want to get the weight of a pixel on a 2D feature map, but at the same time we do not want to get the global weights, but only the weight of a small area around this pixel. Combining the above requirements, it is obvious that convolution is a very useful and effective method. Specifically, as shown in Figure 3, for a pixel x on the 2-D feature map, we can obtain the pixel-wise weights by using a 3×3 convolution layer with zero-padding and use sigmoid (σ(·)) for scaling the weights. Then the weights are no longer added to the pixel x but multiplied with it. Therefore we can represent the pixel attention PLA as follows:

Domain-level attention
The person re-identification task differs from the conventional classification task in that even pedestrians with different identities share the general features of the target pedestrian over the whole dataset. Therefore, we believe that training an attention module over the entire dataset can combine the pedestrian information in all images. On the one hand, the learned attention enables the network to better distinguish between the foreground and background of pedestrian images and to be more robust to the complex shift of angles in pedestrian data; on the other hand, it enables the network to focus on more complete pedestrian information that is generalized across the entire dataset, rather than just a particular part of the pedestrian.
As shown in Figure 4, for the input feature F h×w×c , where h and w are the length and width of the feature, respectively, and the number of channels is c. Similar to the way in HLA, we obtain q through a convolutional layer with the kernel of 1×1. After that, we design two 1×1 convolution layers without bias as storage units, representing the data set level k 1×1 D and v 1×1 D , for replacing k T and v in self-attention. where the number of (input channel, output channel) of k 1×1 D and v 1×1 D are (c, c k ) and (c k , c), respectively, and their initial weight parameters are transposed to each other. In addition, we add an extra step of normalization after softmax, which can reduce the effect of outlier features (such as features of excessively obscured pedestrian samples) and prevent attention from failing. Finally, the whole process of domain-level attention can be described as: where N orm represents the normalization operation.

Datasets and implementation details
We validate our proposed method on three well-known large-scale real person re-identification datasets (Market-1501[Zheng et al., 2015, DukeMTMC-reID [Ristani et al., 2016] and MSMT17 [Wei et al., 2018]). We use ResNet-50 [He et al., 2016] which is pre-trained on ImageNet [Deng et al., 2009] as the backbone model and use the DBScan algorithm [Ester et al., 1996] as the clustering method as mentioned before.
Specifically, we tested our approach in an experimental environment with two 1080ti GPUs and 32G RAM. The size of the input images is set to 256×128, and each batch consists of 64 images, including 8 pseudo-identities each of which contains 8 images. As mentioned before, the two parameters and M inP ts required by DBScan were set to 0.4 and 4, respectively, and the rest of the settings were the same as in [Ge et al., 2020b]. The number of clustering iterations is 50, the learning rate is initialized with 1.6e-4, and decreases by 90% for every 20 epochs.

Comparison with state-of-the-art methods
In Table 2, 3 and 4, we compare the results of our method with other FUL and UDA methods. On three datasets, our method achieved the state-of-the-art results. With the same experimental equipment, besides top-n accuracy, which is higher than CCR [Dai et al., 2021] on all datasets, our method achieves at least 2.3% improvement in mAP over CCR especially on the more complex MSMT17 dataset, which reaches 4.2%.

Ablation studies
We validate the impact between our proposed three levels of attention modules on the Market-1501 and DukeMTMC-reID     Table 6, HLA heavily reduces the discriminability of pedestrian features in unsupervised learning without constraints of identity labels. From Figure 5(c), it can be seen that, as we analyzed before, HLA cannot correctly distinguish the foreground and background of the image, except for some parts of the ground.
Comparing Figure 5(b) and Figure 5(d), besides expanding the range of attention slightly, PLA improves the attention to the body parts of the pedestrians. After combining PLA and HLA, the attention in Figure 5(e) is more fitted to the pedestrians compared to Figure 5(c) and 5(d).
The results in Table 6 show that not only the whole MLA   structure can obtain better performance than the baseline method, but also adding DLA individually can significantly improve the model performance. Meanwhile, it is clear from Figure 5(f) that the DLA pays attention to the pedestrian's feet in addition to the pedestrian's body. And unlike Figures  5(b), 5(d) and 5(e), the small portion of the background above the image is excluded. Eventually, the pedestrian description in MLA features is more comprehensive and also better distinguishes the background and foreground.

Conclusion
In this paper, we design a multi-scale attention module containing pixel-level, head-level, and domain-level attention.
Constraining multi-headed attention by inter-pixel relationships and foreground features of the dataset without introducing label information and achieving state-of-the-art performance. In the next work, we hope to further optimize the structure of multi-level attention by extending it to the overall neural network rather than limiting it to the last residual block, referring to the pyramidal structure for a more comprehensive pedestrian feature description.