{'fraction_non_alphanumeric': 0.034553290963547374, 'fraction_numerical': 0.014391116955219519, 'mean_word_length': 4.836495416755489, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Adaptive Graph Representation Learning for Video Person Re-identification

Recent years have witnessed a great development of deep learning based video person re-identification (Re-ID). A key factor for video person Re-ID is how to effectively construct discriminative video feature representations for the robustness to many complicated situations like occlusions. Recent part-based approaches employ spatial and temporal attention to extract the representative local features. While the correlations between the parts are ignored in the previous methods, to leverage the relations of different parts, we propose an innovative adaptive graph representation learning scheme for video person Re-ID, which enables the contextual interactions between the relevant regional features. Specifically, we exploit pose alignment connection and feature affinity connection to construct an adaptive structure-aware adjacency graph, which models the intrinsic relations between graph nodes. We perform feature propagation on the adjacency graph to refine the original regional features iteratively, the neighbor nodes information is taken into account for part feature representation. To learn the compact and discriminative representations, we further propose a novel temporal resolution-aware regularization, which enforces the consistency among different temporal resolutions for the same identities. We conduct extensive evaluations on four benchmarks, i.e. iLIDS-VID, PRID2011, MARS, and DukeMTMC-VideoReID, the experimental results achieve the competitive performance which demonstrates the effectiveness of our proposed method.


I. INTRODUCTION
As an important and challenging problem in computer vision, person re-identification (Re-ID), aims at precisely retrieving the same identities from the gallery with a person of interest as a query given, has a wide range of applications in intelligent surveillance and video analysis [1]. Typically, person Re-ID is carried out in the domain of individual images without capturing the temporal coherence information. More recently, a number of video person Re-ID approaches [2]- [18] emerge to directly perform the person context modeling at the video level, which is more fit for practical use with more visual cues exploited for coping with complicated circumstances.
In the literature, most existing methods for video person Re-ID first extract the feature vector frame by frame and generate the video-level feature representation by temporal Fig. 1. Overview of graph construction in our proposed method. 1) The feature map from each frame is processed by a pyramid pooling module, the extracted regional features are treated as the graph nodes. 2) The pose alignment adjacency graph A p (colorful solid line) is constructed by connecting the regions containing the same human part. 3) The feature affinity adjacency graph A f (yellow dotted line) is constructed by measuring the affinity of regional features. 4) The adaptive structure-aware adjacency graph is built by combining two graphs. Best viewed in color, some graph edges are omitted for clarity. aggregation, and subsequently compare them in a particular metric space. Although recent deep learning based methods have made notable progress, it remains challenging due to occlusion, viewpoints, illumination, and pose variation in the video. To address these issues, recent studies [4]- [6], [19], [20] concentrate on aggregating the effective images' regions with attention mechanisms. However, under the circumstances of complicated situations (e.g. occlusion and pose variations), these approaches are often incapable of effectively utilizing the intrinsic context relations between person parts across frames, which play an important role in learning robust video representations. For instance, if the body parts are occluded in the first frame, the appearance cues and contextual information arXiv:1909.02240v1 [cs.CV] 5 Sep 2019 from the other frames are complementary. Hence, how to adaptively perform relation modeling and contextual information propagation among spatial regions is a key issue to solve in video person Re-ID.
Motivated by the aforementioned observations, we propose an adaptive graph learning scheme to model the contextual relations and propagate complementary information simultaneously. As shown in Figure I, we construct two kinds of relations named pose alignment connection and feature affinity connection between the spatiotemporal regions. Specifically, 1) pose alignment connection: the regions containing the same part are connected to align the spatial regions across the frames. With the pose alignment connection, we are capable of capturing the relations between human body parts; and 2) feature affinity connection: we define the edges by measuring the visual correlations of extracted regional features. With the feature affinity connection, we can model the visually semantic relationships between regional features accurately.
By combining these two complementary relation connections, we obtain an adaptive structure-aware adjacency graph. Then we capture the contextual interactions on the graph structure via graph neural network (GNN). As a result, the effective and discriminative messages aggregated from neighbors are used to refine the original regional features. With feature propagation, the discriminative power of the informative regional features is enhanced while the noisy parts are weakened.
Based on the observations in [21], the visual cues in the video are rich yet possibly redundant, and some keyframes are sufficient to represent the long-range video. As a consequence, we propose a novel regularization, which enforces the consistency among different temporal resolutions, to learn the temporal resolution invariant representation. Specifically, the frame-level features are randomly selected as the subsequences, and input into the attention module, then the output video representations are enforced to be close to each other in the metric space.
Overall, the main contributions of this work are summarized as follows: • We propose an adaptive structure-aware spatiotemporal graph representation based on two types of graph connections for relation modeling: pose alignment connection and feature affinity connection. By combining these two relation connections, the adaptive structure-aware graph representation is capable of well capturing the semantic relations between regions across frames. • We propose a novel regularization to learn the temporal resolution invariant representation, which is compact and captures the discriminative information in the sequence. We conduct extensive experiments on four widely used benchmarks (i.e. iLIDS-VID, PRID2011, MARS, and DukeMTMC-VideoReID), and the experimental results demonstrate the effectiveness of our proposed method.

II. RELATED WORK
Person Re-ID. Person Re-ID in still images is widely explored [22]- [32]. Currently, the researchers start to focus on video-based person Re-ID [2], [33]. Facilitated by deep learning technique, impressive progress has been observed with video person Re-ID recently. McLaughlin et al. [2] and Yan et al. [33] employ RNN to model the inter-sequence dependency and aggregate the features extracted from the video frames with average pooling or max pooling. Zhou et al. [3] separately model the spatial and temporal coherence with two RNN networks: the temporal model (TAM) focuses on discriminative frames and spatial model (SRM) integrates the contexture at different locations for better similarity evaluation. Wu et al. [20] extend GRU with attention mechanism to selectively propagate relevant features and memorize their spatial dependencies through the network. Dai et al. [9] propose a S 2 T N network to address the pose alignment and combine the bi-directional LSTM with residual learning to perform temporal residual learning.
Recently, the attention networks are widely studied for temporal feature fusion. In [4], [5], the discriminative frames are selected with attention temporal pooling, where each frame is assigned with a quality score and then fused to a final video representation. Similarly, Zhang et al. [21] employ reinforcement learning to train an agent to verify whether the pair of images are same or different, and the Q value is a good indicator of the difficulty of image pairs. In [6], [13], [15], [19], the authors extend the temporal attention to spatiotemporal attention to select informative regions and achieve the impressive improvements. Chen et al. [12] leverage the body joints to attend to the saliency parts of the person in the video to extract the discriminative local features in a siamese network. And in [7], [11], the video representation is generated by considering not only intra-sequence influence but also inter-sequence mutual information. Different from the previous 2D CNN based methods, 3D convolution neural network (3D CNN) is also adopted to address the video person Re-ID [10], [14], [16]. Wu et al. [10] adopt 3D CNN and 3D pooling to aggregate the spatial and temporal cues simultaneously. Li et al. [14] propose a variant of ResNet by inserting multi-scale 3D (M3D) layer and residual attention layer (RAL) into the ResNet. Similarly, Liu et al. [16] incorporate nonlocal modules with ResNet50 as the Non-local Video Attention Network (NVAN), and propose a spatially and temporally efficient variant. Moreover, attributions are utilized to generate the confidence as the weight for sub-features extracted from video frames in [34]. The generative models are adopted to address the occlusion and pose variant in [17], [18].
Graph Models. Graph models are utilized in several computer vision tasks, and Graph Neural Networks (GNN) is introduced in [35] to model the relations between graph nodes, and a large number of the variants [36]- [41] are proposed. In recent, Re-ID methods [42]- [46] combined with graph models are also explored. Cheng et al. [42] formulate the structured distance relationships into the graph Laplacian form to take advantages of the relationships among training samples. In [43], an algorithm that maps the ranking process to a problem in graph theory is proposed. Shen et al. [44] leverage the similarities between different probe-gallery pairs for updating the features extracted from images. Chen et al. [45] involves multiple images to model the relationships among the local and global similarities in a unified CRF. Yan The overall architecture of our proposed method. 1) T frames are sampled from a long-range video with a restricted random sampling method. 2) In graph branch, for the output of each image, pyramid pooling is used to extract the N × d-dimension feature, where N represents the number of regions, the feature vector for each region has d dimensions.
3) The extracted feature vectors are treated as the graph nodes, we then employ GNN to perform feature propagation on the graph iteratively in graph feature propagation module. 4) We carry out the attention module to generate the discriminative video representation, the subsequences are randomly selected and forward into the attention module to learn a consistent video representation. 5) Feature vectors from graph branch and global branch are concatenated for testing. et al. [46] formulate the person search as a graph matching problem, and solve it by considering the context information in the probe-gallery pairs. To address the unsupervised Re-ID problem, Ye et al. [47] involves the graph matching into an iteratively updating procedure for a robust label estimation. In a nutshell, the graph model based methods in Re-ID usually build up a graph to represent the relationships among training samples, where the graph nodes are images or videos. While in our proposed method, the graph is dynamically learned with prior knowledge to model the intrinsic contextual relationships among the regions in an image sequence, the local, global and structure information are propagated among the different regional features to learn the discriminative video feature representation.

A. Overview
Video person Re-ID aims to retrieve the identities from the gallery with the given queries. The overall architecture of our proposed method is illustrated in Figure 2. Given a longrange video for the specific identity, T frames are randomly sampled with restricted sampling method [6], [48], and then grouped as an image sequence {I t } t=1,...,T . We first feed them into the ResNet50-based [49] feature extractor module, in which the stride of the first residual block in conv5 is set to 1. In the global branch, 3D global average pooling is used for the feature maps and produces a video representation x gap ∈ R d . In the graph branch, we obtain regional features [24], where the feature maps are vertically partitioned into 1, 2, and 4 regions 1 in our experiments, and N = 7 is the number of regions for an individual frame. Then, we utilize the pose information and feature affinity to construct an adaptive structure-aware adjacency graph, which captures the intrinsic relations between these regional features. In the graph feature propagation module, 1 In this paper, regions and nodes are interchangeably for the same meaning. the regional features are updated iteratively by aggregating the contextual information from neighbors on the graph. Next, we utilize the attention module to yield the video representations. The network is supervised by identification loss and triplet ranking loss together. We will discuss these modules in the following sections.

B. Adaptive Graph Feature Propagation
As discussed in Section I, the relations between human parts are beneficial for mitigating the impact of complex situations such as occlusion and clutter background. So, how to describe the relationships between different human parts and propagate contextual messages is critical for learning the discriminative video representations. The graph is commonly used to model this kind of relations, and we adopt GNN to leverage the information from the neighborhood.
Adaptive Structure-Aware Adjacency Graph. To depict the relations of human parts, we employ the pose information and feature affinity to construct an adaptive structure-aware is the vertex set containing T · N nodes, where each node v i corresponds to a spatial region in the frame. To define the edge A ∈ R (T ·N )×(T ·N ) on the graph, we introduce two types of relations: pose alignment connection and feature affinity connection.
The pose alignment connection is defined by leveraging the human body joints: two regions (nodes) are connected if they contain the same human parts. Formally, we define a set S i for each region v i where S i ⊆ {head, body, leg}. The pose alignment adjacency graph A p for the two nodes v i and v j is then calculated as follows: where | · | means the cardinality of a set. We obtain S i with the following procedure, first, we locate the joints of the human body by making use of a human pose estimation algorithm, this is illustrated in Figure 3. Then we separate these estimated keypoints into three parts: head part (consist of nose, neck, eyes, and ears), body part (consist of shoulders, bows, and wrists), and leg part (consist of hips, knees, and ankles). For the i-th spatial region, S i is constitutive of the parts located in this spatial region. We present an example in Figure 3, the feature maps are vertically partitioned into 4 regions B 1 , B 2 , B 3 , B 4 . In image I 1 and I 2 , the head part is in B 1 and B 2 respectively, so the pose alignment connection between these two nodes is set to 1. Then, we can create the pose alignment adjacency graph A p . Pose alignment adjacency connection reflects only the coarse relations between different spatial regions and the recent method [50] shows the dynamic graph could learn better graph representations compared to the fixed graph structure. To describe the fine relations between the regions, we propose to learn an adaptive feature affinity adjacency graph A f , which aims to capture the affinity between the regions. For two nodes v i and v j , the node features are x i and x j respectively, then the entry of adjacency graph A f is formulated as follows: We calculate the edge weight matrix A by combining the pose alignment adjacency matrix and feature affinity matrix: where γ is the weight parameter to balance the pose alignment adjacency matrix and the feature affinity matrix, and γ is set to 1 in our all experiments.
Graph Feature Propagation Module. After obtaining the graph, we perform contextual message propagation to update original spatial regional features iteratively. As shown in Figure 4, we employ Graph Neural Network (GNN) [35] to aggregate the information from neighbors for each node. In the graph feature propagation module, we stack L graph feature propagation layers, in the l-th layer, the aggregation and updating operations are defined as follows: where i ∈ {1, 2, . . . , T · N }, l ∈ {1, 2, . . . , L}, and x (l) i stands for the refined regional feature output from l-th feature propagation layer and x (0) i = x i is the original node feature, F (l) (·) is the combination of an FC-layer and batch normalization layer to encode the contextual messages from neighbors, A (l) refers to the adaptive structure-aware adjacency graph, and α is used to balance the aggregated feature and original feature, which is set as 0.1 in our experiments. The output from graph feature propagation module is denoted asX = [x 1 ,x 2 , . . . ,x T ·N ], wherex i ∈ R d is the updated regional feature vector. Temporal Attention Module Given the updated regional featuresX, we perform a simple yet effective spatio-temporal attention [15] to obtain the video representation, which is calculated as: As discussed in Section I, in order to model the consistency of subsequences, we randomly select T − i frames from the image sequence, where i = 1, 2, . . . , T s . And then feed the feature vectors of these subsequence frames into the temporal attention module to obtain video representations x graph,1 , . . . , x graph,Ts . To keep the consistency of different subsequences, we enforce these video representations to be close to each other in the metric space.

C. Loss Functions
We employ two kinds of losses to jointly supervise the training of parameters: cross entropy loss and soft hard triplet loss [51], the losses are formulated as follows: where P and K are respectively the number of identities and sampled images of each identity. So there are P · K images in a mini-batch, x i,a , x i,p and x j,n are the features extracted from the anchor, positive and negative samples respectively, D(·) is the L2-norm distance for two feature vectors. For the output from global branch, we have two losses l global xent and l global htri : for the output from graph branch, we have two losses l graph xent and l graph htri similarily: where BN (·) is the BNNeck introduced in [52], [·] means concatenation. The total loss is the summation of the four losses: IV. EXPERIMENTS

A. Datasets
PRID2011 [53] dataset consists of person videos from two camera views, containing 385 and 749 identities, respectively. Only the first 200 people appear in both cameras. The length of image sequence varies from 5 to 675 frames, but we use only the sequences whose frame number is larger than 21.
iLIDS-VID [54] dataset consists of 600 image sequences of 300 persons. For each person, there are two videos with the sequence length ranging from 23 to 192 frames with an average duration of 73 frames.
MARS dataset [55] is the largest video-based person reidentification benchmark with 1,261 identities and around 20,000 video sequences generated by DPM detector and GMMCP tracker. The dataset is captured by six cameras, each identity is captured by at least 2 cameras and has 13.2 sequences on average. There are 3,248 distracter sequences in the dataset, it increases the difficulty of Re-ID.
DukeMTMC-VideoReID dataset is another large scale benchmark dataset for video-based person Re-ID, which is derived from the DukeMTMC dataset [56] and re-organized by Wu et al. [57]. The DukeMTMC-VideoReID dataset contains totally 4,832 tracklets and 1,812 identities, it is separated into 702, 702 and 408 identities for training, testing and distraction. In total, it has 369,656 frames of 2,196 tracklets for training, and 445,764 frames of 2,636 tracklets for testing and distraction. Each tracklet has 168 frames on average, and the bounding boxes are annotated manually.

B. Evaluation Metrics
For evaluation, we employ the standard metrics used in person Re-ID literature: cumulative matching characteristic (CMC) curve and mean average precision (mAP). CMC curve judges the ranking capabilities of the Re-ID model, mAP reflects the true ranking results while multiple ground-truth sequences exist. For PRID2011 and iLIDS-VID datasets, we follow the evaluation protocol used in [53]. Each dataset is divided into two parts for training and testing, the final accuracy is the average of "10-fold cross validation", only CMC accuracy is reported in PRID2011 and iILIDS-VID because of the equivalence of CMC and mAP on these two datasets. For MARS and DukeMTMC-VideoReID dataset, both CMC and mAP are reported.

C. Implementation Details
Settings. Our experiments are implemented with Pytorch and four TITAN X GPUs. ResNet50 [49] is first pre-trained on ImageNet, and the input images are all resized to 256 × 128. In the training stage, we employ restricted random sampling strategy [6] to randomly sample T = 8 frames from every video and group them into a tracklet. We update the parameters by employing ADAM [58] with a learning rate of 1×10 −4 and weight decay of 5×10 −4 . We train the network for 300 epochs, the learning rate decays to 1 10 every 100 epochs. For batch hard triplet loss, we set P = 4 and K = 4 in our experiments. In the temporal attention module, T s is set as 3. In the testing stage, cosine distance between the representations is calculated for ranking, a video containing T v frames is split into T chunks firstly, then we make use of two kinds of strategies: 1) the first image is collected as an image sequence to represent this video; 2) In each chunk, i-th frames are grouped as an image sequence, we can obtain Tv T image sequences, and the video representations are averaged as a single video representation. In our experiments, the first strategy is fast and the second strategy is more accurate.
Pose Estimation. To generate the human body joints, we adopt AlphaPose [59] as our pose estimation algorithm, which achieves state-of-the-art performance in human pose estimation. There are total of 18 joints detected for each person image, which costs 20ms on average. And we show some examples in Figure 5.
Results on MARS and DukeMTMC-VideoReID From Table I and Table II, it is not difficult to find that our proposed method outperforms the existing approaches. On MARS dataset, our proposed method surpasses the previous best approach STE-NVAN [16] by 0.6% and 0.7% in terms of Rank-1 and mAP. And on DukeMTMC-VideoReID, our method achieves the best performance with 97.0% and 95.4% at Rank-1 and mAP accuracy respectively. The results outperform the state-of-the-art STA [15] by a large margin. These experimental results confirm the effectiveness and superiority of our proposed method.
Results on iLIDS-VID and PRID2011 As shown in Table III, results demonstrate the advantages of the proposed method over existing state-of-the-art approaches on iLIDS-VID and PRID2011. Specifically, our proposed method achieves the Rank-1 accuracy of 84.5% and 94.6% on these two datasets, and surpass all the previous approaches without incorporating optical flow. On these two small-scale datasets, by comparing Snnipet and Snippet+OF, we can observe that the motion information provides more reliable features than appearance cues. While even compared to those methods utilizing optical flow, the results of our proposed method is also compatitive.

E. Ablation Study
To analyze the effectiveness of components in our proposed method, we conduct several experiments on MARS dataset. The experimental results are summarized in Table V and  Table IV.
Analysis on feature propagation module. We carry out experiments to investigate the effect of varying the number of feature propagation layers. We evaluate the results of stacking 1, 2, and 3 propagation layers based on the model combined with adaptive pose alignment adjacency graph and consistent loss, i.e. +A p + A f + consistent in Table V. As shown in Table IV, we find out that the performance is consistent in these settings, where Rank-1 accuracies are all above 89.0% and mAP accuracies are all above 80.5%. In addition, the performance with K = 2 surpasses the other settings, so we stack 2 propagation layers in our experiments.
Analysis on components. In Table V, Baseline contains only the ResNet backbone and 3D global average pooling, and is supervised by l gap xent and l gap htri , the Rank-1 and mAP accuracy of baseline approach is 87.8% and 78.0% respectively. +Attention refers to adopt temporal attention module in  Qualitative results. We visualize the qualitative results in Figure 6 with Grad-CAM [61], which is popularly used in computer vision problem for a visual explanation. The class activation maps (CAMs) generated by the baseline model (second row) and our proposed method (third row) are provided. For the video sequence fragment, in the third and fourth image, the baseline model is incapable of capturing the informative person cues, and our proposed method is more robust to the clutter background.
Retrieval Results As illustrated in Figure 7, we provide the  images w/o graph w/ graph Fig. 6. Qualitative results for the baseline model (w/o graph) and our proposed method (w/ graph). We employ Grad-CAM to visualize the regions focused by the Re-ID model. The first row is the image sequences sampled from videos. In the second and third row, the class activation maps for the baseline model and our proposed method are provided. Compared to the model without graph learning, we find out that our proposed method is robust to occlusion and clutter background. The results for different sequences are separated by the line.
retrieval results on MARS dataset for the baseline model and our proposed method. The illustration presents the improvement of our proposed method.

V. CONCLUSION
This paper proposes an innovative graph representation learning approach for video person Re-ID. The proposed method can learn an adaptive structure-aware adjacency graph over the spatial person regions. By aggregating the contextual messages from neighbors for each node, the intrinsic affinity structure information among person feature nodes is captured adaptively, and the complementary contextual information is further propagated to enrich the person feature representations. Furthermore, we propose a novel regularization to enforce the consistency among different temporal resolutions, and it is beneficial for learning the compact and discriminative representations. The experimental results on four standard benchmarks demonstrate the effectiveness of the proposed scheme, and extensive ablation studies validate the feasibility of components in the network.