{'fraction_non_alphanumeric': 0.042239736259207745, 'fraction_numerical': 0.015917168907433162, 'mean_word_length': 5.070669168230144, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
DeepChange: A Large Long-Term Person Re-Identification Benchmark with Clothes Change

Existing person re-identification (re-id) works mostly consider short-term application scenarios without clothes change. In real-world, however, we often dress differently across space and time. To solve this contrast, a few recent attempts have been made on long-term re-id with clothes change. Currently, one of the most significant limitations in this field is the lack of a large realistic benchmark. In this work, we contribute a large, realistic long-term person re-identification benchmark, named as DeepChange. It has several unique characteristics: (1) Realistic and rich personal appearance (e.g., clothes and hair style) and variations: Highly diverse clothes change and styles, with varying reappearing gaps in time from minutes to seasons, different weather conditions (e.g., sunny, cloudy, windy, rainy, snowy, extremely cold) and events (e.g., working, leisure, daily activities). (2) Rich camera setups: Raw videos were recorded by 17 outdoor varying resolution cameras operating in a real-world surveillance system. (3) The currently largest number of (17) cameras, (1, 121) identities, and (178, 407) bounding boxes, over the longest time span (12 months). Further, we investigate multimodal fusion strategies for tackling the clothes change challenge. Extensive experiments show that our fusion models outperform a wide variety of state-of-the-art models on DeepChange. Our dataset and documents are available at https://github.com/PengBoXiangShang/deepchange.


Introduction
Person re-identification (re-id) aims to match the person identity classes of bounding box images extracted from non-overlapping camera views [15,55,62]. Extensive re-id models were developed in the past decade [2-4,6,11,16,19, 27-30, 33, 38, 43, 53, 56-60, 63-67], thanks to the availability of reasonably sized benchmarks [32,49,52,61]. The majority of these methods consider the short-term search scenario with a strong assumption that the appearance (e.g., clothes, hair style) of each person is stationary. We name this conventional setting as short-term re-id in the follows. Unfortunately, this assumption would be easily broken once the search time span is enlarged to long periods (such as days, weeks, or even months) as an average person often changes the outfit during different day time and across different weathers, daily activities and social events. As shown in Figure 1, a specific person was dressed the same only in a short time (e.g., minutes or hours) but with different clothes/hairs and associations over a long time and across seasons and weathers. Relying heavily on the clothes appearance, the previous short-term re-id models are unsuitable and ineffective in dealing with unconstrained clothes changes over space and time.
Recently there have been a few studies for tackling the long-term re-id situations focusing on clothes change [23,34,39,48,50,51,54]. Since there are no appropriate datasets publicly available, to enable research these works introduced several small-scale long-term re-id datasets by using web celebrity images [24], synthesizing pseudo person identities [34,48], collecting person images under simulated surveillance settings [39,48,50,54]. While these dataset construction efforts are useful in initiating the research, it is obvious that a real, large-scale long-term re-id benchmark is missing and highly demanded. The impact and significance of such a benchmark in driving the research progress has been repeatedly demonstrated in the short-term re-id case [32,51,52,61]. However, it is much more challenging to establish a large re-id dataset with clothes change as compared to the short-term counterparts. This is due to that: (1) More video data need to be collected and processed over long time periods; (2) Labeling person identity becomes much more difficult when a person is dressed up with different outfit from time to time. Regardless, we believe that it is worthwhile to overcome all these challenges.
To facilitate the research towards the applications of long-term person search in reality, we contribute the first large-scale person re-id dataset with native/natural appearance (mainly clothes) changes, termed as DeepChange. Different from existing datasets, DeepChange has several Conventional re-id settings usually assume stationary clothes per person and are hence valid only for the short-term application scenarios. For long-term re-id cases, we must consider the unconstrained clothes change challenge. Note, for clarity only part of true (red arrow) and false (blue arrow) matches are plotted. Best viewed in color. unique characteristics: (1) The raw videos are collected in a real-world surveillance camera network deployed at a residential block where rich scenes and realistic background are presented over time and space. (2) The videos cover a period of 12 months with a wide variety of different weathers and contexts. To the best of our knowledge, this is the longest time duration among all re-id datasets, presenting natural/native personal appearance (e.g., clothes and hair style) variations with people from all walks of life. (3) Compared to existing long-term re-id datasets, it contains the largest number of (17) cameras, (1121) identities, and (178K) bounding boxes. Overall, DeepChange is the only realistic, largest long-term person re-id dataset, created using the real-world surveillance videos.
We make the following contributions: (1) A large scale long-term person re-id dataset, called DeepChange, is introduced. Compared with existing alternative datasets, this dataset offers more realistic and more challenging person re-id tasks over long time with native appearance changes. With a much larger quantity of person images for model training, validation, and testing, DeepChange provides a more reliable and indicative test bed for future research. (2) We conduct extensive experiments on the DeepChange dataset, including seminal CNNs [18,22,41,45], Transformers [8,47], and state-of-the-art long-term reid models [24]. (3) To tackle the clothes change challenge, we investigate multimodal fusion strategies (e.g., gray images, edge maps [68], key points [1]) and achieve new stateof-the-art results on DeepChange.

DeepChange Benchmark
Venue and climate Our raw videos were collected from a real-world surveillance system for a wide (14 hectares) and dense block, with the middle temperate continental monsoon climate. This venue has various scenes, including crowded streets, shops, restaurants, construction sites, residential buildings, physical exercise areas, car parking, sparsely populated corner, etc. Thus it has two major advantages: (i) Identity Diversity: Persons cross a wide range  of ages and professions, e.g., lactating baby, very older person, office lady, elementary student, high school student, worker, deliveryman, religious. Some identity examples are illustrated in Figure 2. (ii) Weather Diversity: During our collecting, we have observed a temperature variation ranging from −30 • C (in winter) to 35 • C (in summer). Therefore, we have collected persons appearing in various weathers, e.g., sunny, cloudy, windy, rainy, snowy, extremely cold. Some image samples with snow are shown in Figure 3, where the snow can cause noticeable appearance changes on both clothes or background. Altogether, the identity and weather diversities will be embodied in drastic appearance changes, enabling realistic long-term reid benchmarking with our DeepChange dataset. Figure2 demonstrates some bounding boxes of an identity randomly selected from our dataset, where we can observe dramatic appearance changes across weathers, seasons, etc.
Security camera Our raw videos were recorded by 17 security cameras with a speed of 25 FPS (frames per sec-ond) and different resolutions (1920 × 1080 spear camera ×14, 1280 × 960 spherical camera ×3). These 17 cameras are part of a large-scale video surveillance system. In particular, these cameras are mounted on the exterior walls of buildings or on lamp posts, and their height is approximately 3 to 6 meters. These cameras are monitoring various views including crowded streets, construction sites, physical exercise areas, car parking, sparsely populated corner, etc. Therefore, these cameras provide diverse scenes, identities, behaviors, events, etc.
Video collecting Our collecting is a very long course over 12 months across two different calendar years. Every month, we randomly collected raw videos on 7 to 10 randomly selected days to cover as much weather as possible. On each selected day, we collected video from dawn to night to record comprehensive light variations. A huge amount of videos can provide highly diverse clothes changes and dressing styles, with the reappearing gap in time ranging from minutes, hours, and days to weeks,  months, seasons, and years. The permission of using these videos was granted by the owner/authority for research purposes only.
Annotation It is much more difficult to recognize persons who have changed clothes or hair styles in the surveillance videos, even by human eyeballing. To minimize mistakes and errors, before labeling, we need to pay vast efforts to watch thousands of hours of videos to familiarize the persons recorded in the videos. Further, it is extremely challenging when persons wear masks, hats, or heavy winter clothes. During labeling, we inspected the labeled persons frequently to avoid duplicate identities assigned to the same person. As a consequence, the quality of our annotations is satisfactory.
Pedestrian detection After labeling, we used Faster RCNN [40] to detect bounding boxes. For each selected bounding box, we annotated person ID, camera ID, tracklet ID, and time stamp. Finally, we detected and blurred the face area for privacy protection. Data statistics All person identities were captured by at least two cameras with most seen by 2 ∼ 6 cameras (as shown in Figure 4c). Figure 4e indicates that the labeled bounding boxes are distributed from 6 am to 9 pm. As illustrated in Figure 4f, the bounding box ratios of persons wearing spring&autumn, summer, and winter clothes are 59.97%, 32.99%, and 7.03%, respectively. More detailed statistics can be found in Figure 4. Data splitting We shuffled all our collected identities, and then orderly picked 450, 150, and 521 IDs for training, validation, and test, respectively. In validation and test sets, given a tracklet, we randomly chose ∼ 5 bounding boxes as queries/probes, and the remaining boxes were split into the gallery. Details were summarized in Table 1. Diversity and challenge As aforementioned, this wide (14 hectares) and dense block provides various identities (as shown in Figure 2), and middle temperate continental monsoon climate causes diverse clothes changes (as demonstrated in Figure 2). Our long-term video collection makes full use of these characteristics of this venue and the climate. We observed that obvious hair-style changes often happened in long-term surveillance videos. In Figure 5, we present some random cases with simultaneous clothes and hair style changes. It is interesting to see that hair style changes should also be considered in long-term re-id, as this

Comparison with Existing Datasets
We compare our DeepChange with existing re-id datasets with and without clothes change. We only discuss the publicly available and non-synthetic datasets. As summarized in Table 2, compared with existing long-term datasets, DeepChange has the largest number of identities, cameras, and bounding boxes, with the longest time span. Besides, it is the only dataset offering four seasonal dressing styles. As seen in Table 3, DeepChange still has the largest number of cameras and bounding boxes even in comparison to traditional short-term re-id datasets.

Multimodal Fusion for Tackling the Clothes Change Challenge
Conventional re-id methods mostly use the color information [15,52,62]. At presence of clothes change, this may be insufficient given that different clothes could vary arbi- 1 More illustrations are provided in the appendix. trarily in color. To tackle this challenge, we consider a simultaneous use of multiple modalities so that additional information irrelevant to the clothes' appearance can be leveraged concurrently. In particular, we consider three more modalities, grayscale images, edge maps, and skeleton key points, for providing the potentially useful knowledge about body contour and part proportion. For cost-effective multimodal fusion, we explore two strategies: (i) CNN-based late-fusion (Figure 6a): Multiple input modalities are separately encoded by a multi-branch network, followed by feature concatenation. (ii) Transformer-based early-fusion (Figure 6b): The patch tokens from multiple modalities are fused by weighted sum in a position-wise manner, followed by self-attention representation learning.

Experiments
We evaluated common deep CNN models and state-ofthe-art re-id methods on DeepChange.

Protocols and metrics
In the traditional short-term person re-id, it is assumed that the appearance of a specific person does not change across time and space. However, this assumption often does not hold for the long-term setting. Hence, in our benchmark we allow the true matches coming from the same camera as the probe/query image but from a different time and trajectory. Following [9,12,13,42,52,61], we used both Cumulated Matching Characteristics (CMC) and mean average precision (mAP) as accuracy metrics.
Implementation details For a fair comparison, all experiments were implemented in the same software and hardware platform. In training only random horizontal flipping was used for data augmentation. For reproducibility, we conducted a unified early stop strategy for all experiments. We empirically adopted mAP on evaluation set as early stop metric and set patience as 30 epochs, thus the checkpoints with the highest validation performance were chosen to report test performance. As all the backbones were initialized by the weights pretrained on ImageNet [7], we empirically used an initial learning rate of 1e −4 , multiplied by a decay factor 0.1 every 20 epochs. ReIDCaps [24] and BN-Neck re-id [36] use the customized loss functions, while the other models were trained by minimizing the softmax based cross-entropy loss. During evaluating and testing, we extracted the features of bounding boxes from the penultimate layer to conduct re-id matching. Edge Box [68] and Open Pose [1] toolboxes were used to extract edge maps and detect body keypoints (15 keypoints, 54D vector), respectively. For the multimodal methods, body keypoint vectors were passed into a module of two fully-connected layers with Batch Normalization [26] and ReLU [14] activations, while CNNs were used to encode RGB images, grayscale images, and edge maps.
(ii) As re-id specific models, BNNeck re-id and ReIDCaps achieve good performances. Similarly, both benefit from deeper networks, as seen from #40 to #46, #49 to #48. (iii) ViT obtains the best mAP score among all the unimodal methods (#52), indicating that self-attention representation is superior in dealing with clothes change challenge over CNN models. (iv) With different modalities (#5 RGB, #6 grayscale, and #7 edge map) for ResNet50, it is observed that RGB is the best. Plausible reasons for lower performance with grayimages include: Although being more tolerant with clothes change, they offer less information than colorful ones. Besides, CNNs are pretrained on colorful ImageNet images. Note, we did not test keypoint modality in isolation, as it fails on 13.45% (23K out of 178K) of person images.
(v) From the multimodal results, we see that both edge maps and grayscale images are useful for performance by feature late-fusion (#5 vs. #56, #57, and #58). This validates our hypothesis that jointly using multiple modalities with specific appearance indeed provides extra information for tackling the clothes change challenge. It is expected that higher-quality edge maps would contribute more to the performance. Again, we find that using stronger networks (e.g., #59, #60, #61) can further improve the results in the multimodal case. On the other hand, multimodal fusion also  helps state-of-the-art re-id models (e.g., #42 vs. #62). This indicates good complementary effect between multimodal fusion and model refinement. With the proposed fusion method on RGB and grayscale images, interestingly the standard ViT model achieves the best mAP scores, surpassing a wide variety of CNNs and CNN based re-id methods by a clear margin. This indicates that Transformers might be a stronger architecture for long-term person re-id, with promising investigation potentials for the future.
(vi) Unlike short-term person re-id, we see that in long-term setting mAP scores are significantly lower than rank@1 rates, a consistent finding as in [24,54]. This is the most fundamental challenge with clothes change, as there is little correlation between different clothes a specific person would wear over space and time. It is also not an easy task for human beings. Due to high clothes diversity in our DeepChange dataset, this challenge becomes more severe. For example, given a random query image (Figure 7a), the true matches with varying clothes often form a long tail distribution in rank (Figure 7b). This contributes to a low mAP score.

Conclusions
In this paper, we have introduced the only realistic, largescale long-term person re-id datasets with natural clothes change, named DeepChange. This aims for facilitating the research towards more realistic person re-id applications over space and time. Compared with existing alternative datasets, it is established uniquely using real-world video sources without any artificial simulation. Constructed with a huge amount of annotation efforts, DeepChange contains the largest number of cameras, identities, and bounding boxes, and covers the longest time period and native appearance change. We conduct extensive experiments using a wide variety of CNN and Transformer models and state-of-the-art re-id methods, offering a set of rich baselines for future research works. Further, we investigate multimodal fusion strategies for overcoming the clothes change challenge, and achieve new state-of-the-art results on our DeepChange dataset.

Future work
In the future efforts, we would try to extend this work mainly in following aspects: (i) continually collecting video to provide longer term test bed, e.g., three years, (ii) annotating more person identities to enlarge training/validation/testing subsets and provide more complex appearance changes, (iii) accommodating more cameras to provide more view variety, (iv) creating a video-based longterm re-id dataset.

Supplementary figures
We provide more example images to further demonstrate the diversity of this dataset in person's appearance. Figure 8 shows more samples with snow and rain, where we can see that snow and rain can cause noticeable appearance changes on both clothes and background. Figure 9 provides other random cases with simultaneous clothes and hair style changes. Figure 10 demonstrates multiple bounding boxes of an identity randomly selected from our DeepChange dataset, where we can observe dramatic appearance changes across weathers, seasons, etc.