{'fraction_non_alphanumeric': 0.0270677825529669, 'fraction_numerical': 0.014476362813843022, 'mean_word_length': 4.58816094375395, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Person re-identification across different datasets with multi-task learning

This paper presents an approach to tackle the re-identification problem. This is a challenging problem due to the large variation of pose, illumination or camera view. More and more datasets are available to train machine learning models for person re-identification. These datasets vary in conditions: cameras numbers, camera positions, location, season, in size, i.e. number of images, number of different identities. Finally in labeling: there are datasets annotated with attributes while others are not. To deal with this variety of datasets we present in this paper an approach to take information from different datasets to build a system which performs well on all of them. Our model is based on a Convolutional Neural Network (CNN) and trained using multitask learning. Several losses are used to extract the different information available in the different datasets. Our main task is learned with a classification loss. To reduce the intra-class variation we experiment with the center loss. Our paper ends with a performance evaluation in which we discuss the influence of the different losses on the global re-identification performance. We show that with our method, we are able to build a system that performs well on different datasets and simultaneously extracts attributes. We also show that our system outperforms recent re-identification works on two datasets.


Introduction
In many domains, such as surveillance or digital signage, being able to automatically recognize a person across different, non-overlapping cameras, without the help of a human operator is very valuable. This task is known as person re-identification and can be extremely challenging since great variations can occur between the different cameras. Figure 1 shows two images, taken from two different cameras from three academic datasets: VIPeR [2], CUHK01 [3] and CUHK03 [4]. Variation can be large between two pictures belonging to the same dataset such as body pose, luminosity, view angle or background.
In many works, person re-identification is based on a similarity score between a pair of images. If the two images represent the same person, the similarity score is high. Two aspects are usually studied. The first one consists in extracting robust invariant features to represent the appearance of a person [5,6,7]. The second is metric learning [8,9]: it consists in learning the best possible metric to discriminate between positive and negative samples.
Recently, convolutional neural networks demonstrated very high efficiency in several computer vision problems such as image segmentation [10] or object recognition [11]. Many research projects have proved that deep neural networks are also extremely efficient for re-identification [12,13,14,15].
To train such a deep neural network, large datasets are mandatory. Recently, re-identification datasets large enough to train deep models have emerged [13,4,16]. In many works [17,15], a neural network is trained on a large dataset and then fine tuned on a smaller one. Consequently, for the performance evaluation, a specific fine tuned model is used to evaluate its corresponding dataset. For an industrial purpose, having a single model able to perform well on many datasets is extremely important. It means that the model can handle different situations, which enables deploying the same model on cameras installed in different environments.
Re-identification with CNNs is usually performed using features extracted by the neural network from identities during the training phase. Attributes, that are more high-level features, like gender, clothes length, handbag may be extremely valuable for re-identification since such features are truly robust to view-angle and cameras change. Schumann et al. [15] demonstrated that using only attributes leads to low performance compared to the features learned by a CNN from the identities. A good approach is therefore to use a combination of attributes and features extracted from identities.
To have a system able to make use of attributes, an access to a large dataset annotated with attributes is required. Nevertheless, it is difficult to acquire large training data for a set of attributes since manual annotations is extremely expensive. Thus, only a subset of re-identification datasets is annotated with attributes and many of them will remain attributeless. It is therefore a problem to build a general system that performs well on several datasets and make use of at-tributes. To deal with the variation of size and annotation of re-identification datasets, we present in this paper a multi-task learning approach which learns the re-identification task from a combination of several datasets. Furthermore, our system is able to take advantage of attribute information in dataset annotated with.
Two main strategies are used in the re-identification community for training deep neural networks. We will describe them in more detail in the next section. The first one [18,19,20] is based on siamese networks, contractive or triplet losses. The second one [21,14], used in our work, is based on classification losses. Since the last layer is a linear classifier, classification methods ensure that features are linearly separable. Consequently, the distance between features belonging to two different classes increases. Nevertheless, with this approach, the intraclass variation is not controlled. Intuitively, reducing the intra-class variations can make the features more discriminant and then increase the re-identification performance. In this work, we add one task in our multi-task learning objective: a task designed to force the features of same identities to be the closest as possible. For the implementation, we employ a method described in [1] which proposes a loss called center loss. We then evaluate the interest of this center loss for our re-identification system.
The contributions of our work are three folds: -We build a model that learns a generic representation of the person using several datasets for re-identification (CHUK01 [3], CHUK03 [4], MARS [16], ViPER [2], Market1501 with attributes [22]) to build a system that performs well on all of these datasets without performing a specific fine tuning. -We take advantage of attributes available in some re-identification datasets such as hair length, top/bottom color, clothes length. We have a multi-task learning objective: the re-identification task, learned from all the datasets and the attribute classification tasks, learned from a subset of the available datasets. -We evaluate an auxiliary task designed to control the intraclass variation of the re-identification features. This task is based on the center loss described in [1].

Related work
Person re-identification Many studies are lead on re-identification, and today CNN and deep learning approaches are well studied and show very good performance on many datasets [23,21,19,24]. Two types of approaches are usually chosen for recent re-identification works with CNN. First one is based on siamese networks [25,18], triplet loss networks [26,19], quadruplet loss [20] to learn a representation based on different and identical couple/triplet. The other approach is based on identity losses [21,16], in which each identity is seen as a class. A classification loss function, such as softmax cross entropy is usually employed.
Training is different whether one chooses the first or the second approach. Triplet loss networks can be difficult to train since one needs to preprocess data to find triplets and hard positive and negative samples [26]. Compared to the number of samples in the dataset, the couple/triplet needs for training dramatically increase and can lead to slow convergence.
Ahmed et al. [23] takes two images for both train and test time to be able to decide whether the two images represent the same person or not. This approach needs to perform an inference each time we need to compare two images, which requires lot of computing power during a search.
To deal with several datasets Xiao et al. [21] developed a guided dropout strategy to learn person representation across different dataset. Other approaches specialize a trained network to a particular dataset. For example, because a large CNN needs many samples, many works fine-tune a network trained on a large re-identification dataset to a smaller one to not overfit [17,15,23].
Multi-task learning for person re-identification Multi-task learning [27] has been applied to re-identification. For example, in [28] the authors use a siamese network, the different tasks are attributes classification tasks. In [29], re-identifications from multiple cameras are regarded as related tasks. Some approaches use a network with two branches [30,15], with jointly optimized losses.
Attributes for person re-identification Attributes have been extensively studied in re-identification [31,32,33]. Attributes can preserve robust information of a person across different point of views or conditions and then it is natural to use them for re-identification. More recently, attributes have been used with deep learning approaches. These architectures can be trained with relatively large re-identification datasets annotated with attributes [34,13]. Some researchers use architectures with two branches, one for the re-identification and one for the attribute extraction and combine them [19,12]. Su et al [35] use three stages of fine tuning and a triplet based loss. Matsukawa et. al [12] only use a combination of losses based on attributes to create a representation able to perform re-identification.

Proposed approach
In this paper we present a deep learning approach for the problem of person reidentification. Given an image with a person, the network outputs a global representation of this person. This representation should be independent from the person pose. We call it the signature of the person. Furthermore, in our architecture, the same network also outputs a list of attributes. The complete attributes list we support is detailed in Table 1. To decide if two pictures represent the same person, we compute the cosine distance between the two signatures. The smallest the distance is, the more likely it is for the two signatures to represent the same person.

Model design
The network architecture used in our work is represented Figure 2. It is based on the resnet50 [36] model followed by a dropout (DP) and two fully connected layers, F C1 and F C2.
FC1 This layer is the global person representation (the signature). Therefore, it is used to perform the re-identification. In our architecture, its size is set to 4096. Two losses are used to train this layer. One identity loss and the center loss [1]. The objective of the identity loss is to make the network able to classify each identity into the correct class. It is a multi-class classification, therefore the cross-entropy softmax loss is used. The identity loss forces the deep features to be separable. To reduce the intra class variation, we use the center loss, introduced by Wen et al. [1] that organizes the deep features around a center for each classes. During the training, the centers are learned and the distance between the deep features and their corresponding center are minimized. Wen et al shown that the center loss is differentiable: our deep neural network can therefore be trained with a standard algorithm based on gradient descent. As stated in [1], the center loss is given by (1) The balance between the identity loss and the center loss is done by a α factor. FC2 Our system is trained with attributes. We support several attributes and, for each of them, we train a classifier. Two types of classifiers are used for the two types of attributes. For the binary attribute (e.g. male/female) binary classifiers are used and trained with a sigmoid cross entropy. For the multiclass attributes (e.g. top/bottom colors) a softmax cross entropy loss is used.
To avoid corrupting the global representation of FC1, we choose to connect the attributes classifiers on FC2, which is a 100 dimensions layer. The weight of the attributes losses is controlled by a λ parameter.
Multi-task learning The learning objectives are controlled by several losses, two for FC1 and n for FC2, with n the number of attributes. As showed in Figure 3, the learning is performed by the combination of all these losses. Let's L id the identity loss, L cs the center loss and L att the sum of the attribute losses, our total loss is given by (2) Re-identification process The re-identification process is based on the global representation FC1 extracted from pictures. The similarity between pictures is computed with the cosine distance.
In which a and b are two 4096 dimension vectors extracted from two pictures by the system. And < ., . > represent the euclidean dot product.  The CNN architecture for the re-identification. A resnet50 [36] network is followed by two fully-connected layers. The FC1 layer is used to compute the similarity distance and it is trained with the center loss and an identity loss. FC2 is trained with the attribute losses. A dropout layer (DP) is added before FC1 and FC2.

Learning strategy
The aim of our work is to build a single system able to perform well simultaneously on all the datasets listed on Table 2 and to recognize pedestrian attributes. We describe in this section the way we chose to reach this goal.
Joint datasets learning One of the main objectives of our system is to show good performances on diverse datasets. Thus to build our training set, we proceed by mixing the training sets of these datasets. This approach is valid since there is no identity overlap between the different datasets. Let's consider we have n datasets, the number of identity of the i th is denoted by d i . The total number of identities of our global dataset is The identities in the datasets are not represented by the same number of images. Thus one can not simply merge the datasets since the information contained in the smallest datasets would then be negligible compared to that contained in the large ones. To tackle this issue we employe a weighted cross entropy for the identity loss.
Let nc k i represent the number of images in the k th class of the i th dataset. q k i represents the value of the corresponding logit and p k i the associated ground truth. The identity loss l id is thus given by This ensure a high weight for the classes under represented and an lower weigh for the most frequent ones. This loss is also appropriate for an optimization in batch mode, in which we compute the weighted loss for each element of the batch and we compute the mean over all the cross entropies. Let N b be the size of the batch we denote the loss corresponding to the g th element of the batch by l (g) id defined as in (4). One can then write the final loss L id : The training is done with the Adam optimizer [37]. The initial learning rate is set to 0.0001. We start the training of our system with a resnet50 [36] network pretrained on imagenet.
Attribute recogntion task To be able to have only one network able to output both re-identification and attributes, multi-task learning is used. We use three losses: the ones defined in (4) and in (1) for the re-identification task and another for the attributes extraction task. As for the identity loss, the attribute loss can be written as a modified cross entropy. Let suppose there are K annotated classes in our dataset, with N att attributes. The loss corresponding to the l th attribute of the j th class is written l (j,l) a and is: Where d l att represents the dimension of the logit layer of the l th attribute, p d the output of the d th logit and q d the corresponding ground truth. One can then write the loss relative to the attributes: During training, we randomly sample batches of images. While all the images are annotated for the re-identification task, only some of them are annotated with attributes. Therefore the re-identification loss is computed with all the batch samples, the attributes loss is updated on a subset of the batch. More details can be found on Figure 3, in which each sample of a batch has an identity used for the re-identification task. Some of them also have attributes annotations. The identity loss is thus always computed on all the samples of the batch. On this example, the identity loss is computed on the 9 batch samples. The attribute loss is only computed with the samples having attributes annotation. In the batch given in example, there are only 3 samples annotated.

Lets define
With N b representing the batch size as in (5). The loss L defined in the figure  3 can the be written as follows: However in the dataset annotated with attributes, appearance frequencies of each attribute are not equal. For example, the blue pants class is more represented than the pink pants in the dataset. To deal with this unbalanced dataset
We first present these datasets and the protocol followed to compute the performance metrics. Then we show the results of our approach and compare them to the state of the art.

Datasets
CUHK01 Dataset with 3,884 images of 972 pedestrians, each identity is observed by two cameras view. Each person has two images from the first camera and two images from the second camera. All pedestrian images are manually cropped CUHK03 dataset contains 14,097 cropped images of 1,467 identities. Each identity is observed by two camera views and contains 4.8 images in average for each view. There are two types of bounding boxes: the manually labeled pedestrian bounding boxes and the automaticaly detected bounding boxes obtained by a pedestrian detector. VIPeR dataset is a very challenging dataset, since it contains 632 pedestrian image pairs taken from arbitrary viewpoints with various illumination conditions or poses. Market 1501 This dataset contains 32668 images annotated using a DPM (Deformable Part Model) giving 1501 identities, split in a training set of 751 identities and a test set of 750 identities. Each identity is captured by at most 6 and at least 2 so that cross view search can be performed. Furthermore one can focus on the search from one viewpoint in each other. The dataset has been annotated with 27 attributes per ID.
The statistics of this four datasets are summarized on Table 2.

Metric and protocol
Metric To evaluate our model we employed the Cumulative Matching Characteristic (CMC) [39] which is the most used metric on re-identification works [16,2,3,19,28]. The CMC curve represents the probability of correct re-identification on the y-axis against the number of candidates returned on the x-axis. The CMC rank1 is very important since it measures the ability of the system to truly identify a person.
For the different datasets we use the protocol described in [21] which is based on [40]. For CUHK01 and VIPeR we divide the identities of the dataset in two equal parts, i.e 485 and 316, for the test set and the training set. For CUHK03, we use the commonly used split: 1467 identities for the training set and 100 identities for the test set.
Our gallery sets and probe sets are constructed as follows. For VIPeR, which has two camera views, we randomly select an image from the first camera as probe image. The gallery image is the same identity taken from the other camera. For CUHK03 and CUHK01 we use a similar protocol, we ensure that images of the gallery sets and probe sets are not the same camera. As stated in [21,40] both the manually and automatically cropped images were used in our experimentation.
Training protocol We train several networks to compare the influence of the center loss and the attributes loss.
For the first stage, we train networks without the attributes losses. The resnet50 network we use is pretrained on imagenet. The weights are the one distributed by the Tensorflow community. The name of the checkpoint is resnet v2 50 2017 04 14.tar.gz and can be downloaded from the Tensorflow GitHub repository 1 . The learning is performed using the Adam [37] optimizer set with an initial learning rate at 10 −4 for this stage. We train 4 networks, each network is trained with a particular center loss value (0.0, 0.05, 0.06 and 0.1).
For the second stage we load the weight previously computed and we launch a train with the Adam optimizer with learning rate sat to 10 −6 . We generate two types of models, the first type are models still trained without the attributes losses and the second type are models trained with the attributes losses. At the end of these two stages we therefore have generated 10 models, for the 5 center loss values we have a model with attributes and a model without attributes.
For the training, the hyperparameters we used are available int Table 3. Table 3: Hyperparameters used in our model. CS α is the learning rate for the center loss as described in [1]. We discuss in this section both the influence of the center loss and the attributes. We first discuss only the center loss. To show its influence, the models are trained without attributes. Then we show the influence of the attributes. We therefore use our models trained with the attributes and the center loss.

Influence of the center loss
On this section we vary the α parameter which controls the center loss weight on the global loss.
We focus in this section on performance without the attributes loss. The results are the ones in the No Attributes columns from the Tables 4 . To interpret the results we draw on Figures 4a (resp. Figure 4b) the value of the rank-1 CMC during the training (resp. the value of the center loss during the training). The rank-1 CMC during training is computed with all the datasets combined. We thus compute the CMC against all the identities of all the datasets. We control the regularization of our network to have a rank-1 CMC close to 70. Higher values lead to a rapid overfitting as we will show with a center loss value (α) set to 0.1.
The different values of the center loss lead to different performance. After 0.1 we observe a drop of performance on all the three datasets. During training, the value of the center loss decreases according to the λ parameter. This effect can be seen figure 4b. With the value of 0.1 the center loss decreases dramatically. This has a strong influence to the CMC computed on the training datasets showed on Figure 4a: the rank-1 CMC reaches a value near to 1.0 which leads to overfitting.
This shows that the center loss has a strong effect during the training, putting the features from a same identify close to a same center increases the capacity of the neural network. With our set of hyperparameters the ideal set of value of the center loss is lower than 0.1. The center loss has not the same impact on all the datasets. With VIPeR the value increase of 4.5 points. For CUHK01 and CUHK03 the value increase of 0.1 and 4.5 points. It shows that the center loss really helps the network to output general representation of pedestrian. Indeed in the ViPER dataset the variation between the two images of a same class is higher than the ones in the other datasets. Thus the network has to be able of great generalization to perform well on ViPER.

Influence of the attributes
We now focus on the attributes. Our objectives are to understand how the attributes help the re-identification score and how our model performs on the attributes extraction task. To compare the influence of the attributes we train our network with the different values of center loss with the attribute system activated. To activate the attribute we add the factor 100 on all the attributes losses for our experiments(the λ factor). This factor has been empirically found and produces the best results. As shown in Table 4 the attributes losses make the network more efficient on all the datasets.

Comparison with the state of the art
We compare the performance of our system with the state of the art on CUHK01, CUHK03 and VIPeR. Results are shown on Table 5. We take our best model in this Table, i.e our model with the attributes losses enabled and the α value set to 0.06. Our system shows better performances on CUHK01 and CUHK03 than the state of the art while being lower on VIPeR. It shows that our approach indeed enables a network to be performant on a large variety of datasets without needing a special retraining for each of them.

Attributes performances
In this section, the performances of our network will be presented. The tests are run over the test set of the dataset Market-1501. Since some of the classes occur too rarely in the test set to be representative (for instance the bottom yellow and purple) they are removed from the tests. The average precision of the network on the attributes recognition are presented in Table 6. The possible values of the attribute are available on Table 1. For both bottom color and top color attributes, we compute the mean of the average precision of each color. These mean average precision are shown on Table 6 for the column bot.col and top.col. This shows that our system is able to learn to recognize attributes and to perform re-identification at the same time. The attributes classifiers show very good performance on attributes such as gender, hair length, or backpack. Some attributes such as hand bag have a low average precision. This is probably because these classes are under represented on the train dataset. Even if we manage the attribute unbalance during training, when the number of samples for a given class is too limited the network cannot properly generalize.

Conclusions
In this paper we have shown a CNN architecture for person re-identification. This architecture is trained with multi-task learning in order to have a system able to be trained from different datasets with different labels. We have demonstrated our approach on 4 datasets: two datasets are relatively small (VIPeR and CUHK01), an another one is larger (CUHK03) and the last one is annotated with attributes (Market 1501). We have evaluated the influence of the different tasks on the global performance (center loss and attributes losses). We proved that combining the different losses leads to better performances. The center loss has a strong influence on performance. We have shown that our system performs well on CUHK01, CUHK03 and VIPeR, and outperforms recent re-identification works on CUHK01 and CUHK03.