{'fraction_non_alphanumeric': 0.03874451569355383, 'fraction_numerical': 0.015963550455619306, 'mean_word_length': 4.778276131045242, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Gait Recognition via Effective Global-Local Feature Representation and Local Temporal Aggregation

Gait recognition is one of the most important biometric technologies and has been applied in many fields. Recent gait recognition frameworks represent each gait frame by descriptors extracted from either global appearances or local regions of humans. However, the representations based on global information often neglect the details of the gait frame, while local region based descriptors cannot capture the relations among neighboring regions, thus reducing their discriminativeness. In this paper, we propose a novel feature extraction and fusion framework to achieve discriminative feature representations for gait recognition. Towards this goal, we take advantage of both global visual information and local region details and develop a Global and Local Feature Extractor (GLFE). Specifically, our GLFE module is composed of our newly designed multiple global and local convolutional layers (GLConv) to ensemble global and local features in a principle manner. Furthermore, we present a novel operation, namely Local Temporal Aggregation (LTA), to further preserve the spatial information by reducing the temporal resolution to obtain higher spatial resolution. With the help of our GLFE and LTA, our method significantly improves the discriminativeness of our visual features, thus improving the gait recognition performance. Extensive experiments demonstrate that our proposed method outperforms state-of-the-art gait recognition methods on two popular datasets.


Introduction
Gait recognition is a biometric technology depending on the uniqueness of human walking posture. Since human gait can be captured in long-distance conditions and the recognition process does not need the subject's cooperation, gait recognition can be widely applied in many fields, such as video surveillance, intelligent transportation, etc. However, * Shunli Zhang is the corresponding author. the performance of gait recognition suffers from many conditions, e.g. changing clothing, carrying conditions, crossview, speed change, and resolution [4,14,24]. Therefore, improving the performance of gait recognition in a complex external environment is still highly desirable.
Recently, many existing gait recognition methods employ convolutional neural networks (CNNs) to generate gait feature representations and achieve better recognition performance than the traditional approaches. In general, the feature representations can be divided into two categories: global and local feature based representation. Global feature based representation methods extract gait features from whole gait frames. Shiraga et al. [18] use 2D CNNs to extract global gait features from Gait Energy Image (GEI). Chao et al. [3] also use 2D CNNs to extract global features at the frame-level. Local feature based representation methods extract and combine local gait features from local gait parts. Zhang et al. [27] partition human gaits into different local parts and use multiple separate 2D CNNs to extract local features. Fan et al. [5] design a focal convolution layer to further extract local features from feature maps.
However, the aforementioned methods only utilize either global or local feature for representation, thus limiting the recognition performance. In particular, the global feature representations may not pay enough attention to the details of the gait, while the local feature representations may lose the global context information of the gait and neglect the relations among local regions. Moreover, Wolf et al. [20] introduce 3D CNNs to extract the robust spatial-temporal gait features. However, traditional 3D CNNs requires fixedlength gait sequences for classification and thus are not able to address different length of videos directly.
To address the above issues, in this paper, we propose a novel cross-view gait recognition framework by learning effective representations from global and local features. Specifically, we build a new feature extraction module, called Global and Local Feature Extractor(GLFE), in the 3D CNNs framework to attain discriminative representations from both global and local information of gait frames.
In the GLFE module, we design a new Global and Local Convolutional layer (GLConv) to extract both global and local features in a principle way. The global feature extractor focuses on the entire visual gait appearance, while the local one pay attention to the gait details. Then, the GLFE module is composed of multiple GLConv layers. By combining global and local gait feature maps, the GLFE module is able to attain more discriminative feature representation.
Since existing 2D CNNs based methods [5,3] usually use a spatial pooling layer to downsample feature resolution, the spatial information will be lost gradually. To fully exploit spatial information, we develop a novel Local Temporal Aggregation (LTA) operation to replace the traditional spatial pooling layer and aggregate temporal information in local clips. In this fashion, we leverage the temporal resolution to attain higher spatial resolution.
Since the proposed method employs 3D convolutions, the temporal convolution is applied to aggregating temporal information.
The main contributions of this paper are three-fold. 1) We present a novel gait recognition framework to obtain a discriminative gait representation. In this framework, we introduce a new Global and Local Feature Extraction (GLFE) module with our Global and Local Convolutional layers (GLConv).
2) We propose a novel Local Temporal Aggregation (LTA) operation to aggregate local temporal information while preserving the spatial information.
3) The proposed method has been evaluated on public datasets CASIA-B and OUMVLP. The experimental results demonstrate that it can achieve state-of-the-art performance, especially in complex conditions.

Related Work
One of the typical gait recognition methods is to model 3D human [1,2,28]. The 3D human model often uses multiple cameras to capture gait data from different view angles and reconstruct the human 3D model, and then is utilized to generate 3D gait features for recognition. Although the 3D human model can carry more information and has strong robustness in cross-view condition, it is difficult to build a 3D human gait model in real scenes. Meanwhile, building human 3D model requires complex computation. Hence, most recent researchers conduct human gait recognition based on 2D gait data, which makes the cross-view become the challenging factor.
Recently, in the works based on 2D gait data, gait sequences are usually synthesized into a gait template for recognition. To degrade the effect of cross-view, existing gait recognition methods in this fashion can be divided into two types. The first type makes use of hand-crafted viewinvariant feature [6,16,11] which normalizes different view angles to a specific view angle. Although normalizing views can address the problem of cross-view, it causes the gait information loss during the transformation process. The second type utilizes the View Transformation Model (VTM) [12,22] to build the correlation between two different view angles. Then, one gait view can be transformed to another for recognition during the test stage. However, it is difficult to fully collect all view angles of the human gait in reality.
Inspired by the great achievements of CNNs in face recognition and action recognition, some researchers leverage CNNs for gait recognition. Although the cross-view is not addressed explicitly, CNNs can extract more robust gait features, improving the feature representation ability. Recently, designing gait recognition methods based on 2D or 3D convolutional networks becomes popular. Shiraga et al. [18] propose the GEInet structure based on 2D convolution. They utilize two convolutional layers to extract features from GEI. Zhang et al. [25] introduce a Siamese framework with 2D convolution which also works on GEI. Wu et al. [21] propose the LB and MT 2D CNNs method. However, the temporal information of gait sequences may be lost during the synthetic process of GEI. Therefore, some researchers first extract gait features at the frame-level gait image instead of GEI and then conduct operations to generate gait templates [3,27,5]. Chao et al. [3] use a statistical function to build gait templates after frame-level feature extraction. However, these methods only extract the feature from each frame, which uses spatial information but abandons temporal relation between the frames. Zhang et al. [27] introduce a temporal attention mechanism to learn the attention scores of each frame, which are used to adaptively adjust the weights of specific frames. Fan et al. [5] propose a novel framework named GaitPart, to extract spatial features from gait sequences and then models the temporal dependencies of the extracted 2D features. To make use of both the spatial and temporal information, 3D convolution is introduced [20,15]. For example, Wolf et al. [20] utilize 3D convolution for feature extraction in gait recognition. However, traditional 3D CNNs need to take a fixed-length gait sequence as an input, which cannot deal with different length videos adaptively. In this paper, the proposed method not only uses 3D convolution but also introduces statistics function to aggregate temporal information for representation.
To further improve the feature representation ability, some researchers [27,5] utilize local gait features to replace global features. Zhang et al. [26] partition human gait image into four different parts as local human gaits. Then, they use multiple 2D CNNs to extract local gait feature of each local part. Fan et al. [5] propose a novel convolution layer, called Focal Convolution Layer, to extract local gait features from local feature maps. Although it can learn more details, local gait features neglect the relation of different local regions. Therefore, in this paper, we propose a novel GLFE module to extract more comprehensive features, which contain both global and local information.

Proposed Method
In this section, we first overview the framework of the proposed method. Then we describe the key components of the proposed method, including Local Temporal Aggregation (LTA), Global and Local Feature Extractor (GLFE) and Generalized-Mean (GeM) pooling layer [17]. Finally, the details of training and testing are presented.

Overview
The overview of the proposed method is shown in Fig. 1, which aims to extract more comprehensive feature representation for gait recognition, including three key components. First, we use a convolution to extract the shallow features from the original input sequence. Next, the Local Temporal Aggregation (LTA) operation is designed to aggregate the temporal information and preserve more spatial information for trade off. After that, the Global and Local Feature Extractor (GLFE) is implemented to extract the combined feature ensembling both global and local information. Then, we leverage temporal pooling and GeM pooling layer to implement feature mapping. Finally, we choose the triplet loss [3,5] and cross entropy loss to train the proposed model.

Local Temporal Aggregation
Previous works [3,5] use a specific pattern "CL-SP-CL-SP-CL" to extract features, where CL means convolutional layer and SP denotes the spatial pooling layer. However, the spatial information may be lost due to the SP downsampling operation. Considering that temporal information in a gait sequence is periodic, we present the LTA operation to replace the first spatial pooling layer, which can integrate temporal information of local clips and maintain more spatial information.
Assume that X in ∈ R C1×T1×H1×W1 is the input of local temporal aggregation, where C 1 is the number of channels, T 1 is the length of the gait sequence and (H 1 ,W 1 ) is the image size of each frame. The process can be formulated as follows where f b×1×1 a×a×a (·) denotes the 3D convolution operation with kernel size a and temporal stride b. X LT A ∈ R C2×T2×H1×W1 is the output of LTA operation.

Global and Local Feature Extractor
Except the global gait features [3,18], some recent researchers propose different gait recognition frameworks which utilize local gait information, as shown in Fig.2(a)(b) [27,5]. For example, Zhang et al. [27] propose the ACL framework to extract local gait features by using multiple separate 2D CNNs. Fan et al. [5] develop a focal convolutional layer to extract the local features and then combine them. Although it contains more detailed information than the global gait features, the local gait features do not pay attention to the relations among local regions. Hereby, we propose a novel GLFE module to extract features, which can take advantage of both global and local information. The GLFE module is implemented by the GLConv layer, which contains global and local feature extractors. The global feature extractor can extract the whole gait information, while the local feature extractor is used to extract more details from local feature maps. The GLConv has two different structures due to different combinations, e.g. GLConvA and GLConvB. The GLFE module includes four layers, "GLConvA-SP-GLConvA-GLConvB" as shown in Fig.1.
The GLConv layer is shown in Fig.2(c). Assume that its input is X global ∈ R c1×t×h×w , where c 1 is the number of channels, t is the length of feature maps and (h,w) is the image size of each frame. We first partition the global feature map into n-parts as local feature maps X local = X i local |i = 1, ..., n , where n is the number of partitions Figure 2. Architectures of different gait feature extractors. ⊕ means element-wise addition, "Concat" means concatenating the feature maps of different parts horizontally, "2DCNN" denotes a 2D convolution neural network, "conv2d" denotes a 2D convolution operation and "conv3d" means a 3D convolution operation. and X i local ∈ R c1×t× h n ×w corresponds to the i-th local gait part. Then, we use 3D convolutions to extract global and local gait features, respectively. Note that all local feature maps share the same convolutional weights. There are two ways to combine the global and local feature maps, i.e. by element-wise addition (GLconvA) or by concatenation (GLconvB). The GLconvA and GLconvB layers can be representated as and where cat means concatenating operation. Y global and Y local can be represented as and where f 3×3×3 (·) and f 3×3×3 (·) denote 3D convolutions with kernel size 3. Based on the above two forms of GLGonv layer, the GLFE module can be built to extract gait features after LTA operation. Experimentally, GLConvA is used to implement the first few GLGonv blocks and GLConvB is ulitized to realize the last one in the GLFE module.

Feature Mapping
Since the length of the input gait sequences may be different, we introduce temporal pooling to aggregate the temporal information of the whole sequence [3,5]. Assume that X GLF E ∈ R C3×T2×H2×W2 is the output of GLFE module, where C 3 is the number of input channels, T 2 is the length of feature maps and (H 2 ,W 2 ) is the spatial size of the feature in each frame. Because of the spatial pooling layer in the GLFE module, the spatial size becomes (H 2 ,W 2 ), while the length of feature maps remains unchanged. The temporal pooling T P (·) can be realized by where F T2×1×1 M ax (·) means Max-pooling layer. Y T P ∈ R C3×1×H2×W2 is the output of temporal pooling.
To improve feature representation ability, researchers develop the spatial feature mapping operation with weighted sum [3,5]. After temporal pooling, gait feature maps are split into strips and two statistical functions, max and average, are used to aggregate each strip's information. The spatial feature mapping can be represented as where Y M A ∈ R C3×1×H2×1 is the output of spatial feature mapping. However, the weighted sum strategy is inflexible because trade-off parameters are predefined manully. Hereby, we introduce the Generalized-Mean pooling (GeM) to integrate the spatial information adaptively. The GeM pooling layer GeM (·) can be represented as where Y GeM ∈ R C3×1×H2×1 is the output of GeM operation. p is the parameter which can be learned by network training. Specifically, if p = 1, Y GeM is equal to Avg 1×1×W2 (Y T P ), and if p → ∞, Y GeM is equal to M ax 1×1×W2 (Y T P ). Then, we use multiple separate fully connected layers to further aggregate the information from channels of Y GeM . The feature mapping can be defined as Y out is the output of feature mapping with H 2 horizontal features, each of which has C 4 channels. F sf c (·) denotes the multiple separate fully connected layers.

Loss Function
To effectively train the proposed gait recognition model, we simultaneously utilize the triplet loss [8] and cross entropy loss .
The triplet loss can improve the inter-class distance and reduce the intra-class distance, which can help the cross entropy loss to identify Human IDs. In the training stage, each horizontal feature of Y out is fed into the combined loss function to calculate the loss independently. The combined loss function L combined can be defined as where L tri and L cse mean the triplet loss and cross entropy loss, respectively. L tri can be defined as where i and j are the samples from the same class A, while k represents the sample from class B. F (·) denotes the feature extraction and mapping operation of the proposed method. D(d 1 , d 2 ) is the Euclidean distance between d 1 and d 2 . m is the margin of the triplet loss. The operation [γ] + is equal to max(γ, 0).

Training Details and Test
Training Stage. During the training stage, we first feed the input gait sequences into the proposed network to generate gait feature representation Y out . Then, the combined loss function is used to compute the loss and Batch ALL (BA) is adopted as the sampling strategy, which is the same as [8,3,5]. Specifically, each batch contains P subject IDs, and K samples are selected from each subject ID. Correspondingly, the batch size is P × K. During the training stage, considering the memory limit, we set the length of input sequence to T. Test Stage. During the test stage, the whole gait sequences are put into the proposed network to obtain gait features Y out . Then, the Y out ∈ R C4×1×H2×1 can be flattened to a feature vector with dimension C 4 × H 2 and then taken as a sample. To calculate Rank-1 accuracy, the test dataset is divided into two sets, i.e. the gallery set and the probe set. The gallery set is regarded as the standard view to be retrieved, while the feature vectors of the probe are used to match the feature vectors from the gallery view. Multiple metric strategies, such as Euclidean distance and cosine distance, can be used to calculate similarity between the samples from gallery and probe set. Specifically, Euclidean distance is selected as the metric strategy.

Datasets
We evaluate the performance of the proposed method on two commonly used datasets, i.e. CASIA-B [24] and OUMVLP [19] datasets. CASIA-B. The CASIA-B dataset [24] is the largest crossview gait database. It includes 124 subjects, each of which has 10 groups of videos. Among these groups, six of them are sampled in normal walking (NM), two groups are in walking with a bag (BG), and the rest are in walking in coats (CL). Each group contains 11 gait sequences from different angles ( 0 • -180 • and the sampling interval is 18 • ). Therefore, there are 124(subject) × 10(groups) × 11(view angle) = 13,640 gait sequences in CASIA-B. The gait sequence of each subject are divided into training set and test set. During the training stage, three training settings are configured according to the different training scales [3]: small-scale training (ST), medium-scale training (MT), and large-scale training (LT). For these three settings, 24, 62, and 74 subjects are chosen as the training set and the rest 100, 62, and 50 subjects are used for test, respectively. All gait data from the training set are used to train the proposed model during the training stage. In the test stage, the sequences NM#01-NM#04 are taken as the gallery set, while the sequences NM#05-NM#06, BG#01-BG#02, and CL#01-CL#02 are considered as the probe set to evaluate the performance. OUMVLP. The OUMVLP dataset [19] is one of the largest gait recognition databases, which contains 10,307 subjects in total. Each subject contains two groups of videos, Seq#00 and Seq#01. Each group of sequences are captured from 14 angles: 0 • -90 • and 180 • -270 • with sampling interval 15 • . We adopt the same protocol (5,153 subjects are taken as training data and 5,154 subjects are used as test data) as [3] and [5] to evaluate the performance of the proposed method. In the test stage, the sequences in Seq#01 are taken as the gallery set, while the sequences in Seq#00 are regarded as the probe set for evaluation.

Implementation Details
We adopt the same preprocessing approach as [3] to obtain gait silhouettes for CASIA-B and OUMVLP datasets. is first set to 1e-4, and reset to 1e-5 and 5e-6 after 150K and 200K iterations, respectively. In particular, since OUMVLP has 100 times more IDs than the CASIA dataset, we add the label smooth operation into the cross entropy loss [9]. The weight decay is first set to 0 and reset to 5e-4 after 200K for OUMVLP dataset.

Comparison with State-of-the-Art Methods
Evaluation on CASIA-B. We compare the proposed method with the latest gait recognition methods, including ViDP [10], CMCC [13], GaitSet [3], AE [23], MGAN [7], CNN-LB, CNN-3D, CNN-Ensemble [21], ACL [27] and GaitPart [5] on CASIA-B. The experimental results are shown in Table.2. It can be observed that the proposed method achieves the best recognition accuracy at almost all angles. We further analyze the comparison results in details by considering different conditions (NM, BG, and CL) and different dataset scales. We first explore the effect of different conditions (NM, BG, and CL). From Table. 2 we can observe that the accuracy has significantly decreased when the external environment changes. For example, the recognition accuracy of GaitPart in three conditions NM, BG, and CL is 96.2%, 91.5%, and 78.7% with the setting of LT. For the pro-posed method, the recognition accuracy in these conditions is 97.4%, 94.5%, and 83.6%, which outperforms GaitPart by 1.2%, 3.0%, and 4.9%, respectively. The experimental results show that the proposed method has significant advantages in the BG and CL conditions, indicating that the proposed model can extract more robust gait features. Even in the ST and MT setting, we can see the similar recognition results that the proposed method also obtains the best performance. On the other hand, human gait can be collected from arbitrary view angles and conditions in a real scenario. Hereby, we focus on the robustness of gait recognition under various external environment factors. Based on Table. 2, we further calculate the average accuracy of three conditions. Table.3 indicates that the comparison results of the proposed method with state-of-the-art gait recognition methods, including GaitSet [3] and GaitPart [5]. It can be observed that the average accuracy of the proposed method is 91.8%, which outperforms GaitSet and GaitPart by 7.6% and 3.0%, respectively.
We also discuss the comparison results with different setting of dataset scales. Chao et al. [3] provide an evaluation protocol, in which the performance of the method is evaluated on three different scales, i.e. ST, MT, and LT on CASIA-B. In this paper, we present the complete experimental results with these three settings. The experimental results are shown in Table. 2. We can observe that the recognition accuracy of the GaitSet [3] is 79.5%, 92.0%, and 95.0% for three settings under the condition of NM, respectively. For the proposed method, the corresponding recognition accuracy is 86.0%, 95.9%, and 97.4%, which increases by 6.5%, 3.9%, and 2.4%, respectively. Specifically, the proposed method can achieve larger improvement in the setting of small-scale dataset.
Evaluation on OUMVLP. We further evaluate the performance of the proposed method on the OUMVLP dataset. For fair comparison, we take the same training and test protocols as the GaitSet and GaitPart methods [3,5]. 10,307 subjects are divided into two groups, where 5,153 subjects are used for training and the rest subjects are used for test. During the test stage, Seq#00 is regardes as probe and  Seq#01 is taken as the gallery. Table 4 desplays the comparison results of our method and several famous algorithms, including GEINet [18], GaitSet [3], GaitPart [5] and GLN [9]. It can be seen that our method can achieve the best recognition performance in most cases.

Ablation Study
In this paper, the proposed recognition framework includes several key modules, e.g. Local Temporal Aggregation, Global and Local Feature Extractor and Generalized-Mean pooling layer. Hereby, we design different ablation studies to analyze the contribution of each key module. Analysis of GLFE module. Different from most gait recognition methods which only extract global or local gait features, we propose a novel global and local feature extractor to extract the gait feature, which contains more compre-hensive gait information. The GLFE module is composed of three global and local convolutional layers, i.e. the GLconv layers, each of which includes a global branch and a local branch. To explore the contribution of the global and local branches, we design the ablation study to explore the role of different branches. All experiments are conducted with the setting LT. The experimental results are shown in Table  5. We can observe that the accuracy of using either global or local branch is 97.1% and 96.2% in the condition of NM, while the accuracy of using both two branches is 97.4%, which increases by 0.3% and 1.2%, respectively. Meanwhile, the global and local convolution layers also improve the performance in the conditions of BG and CL. Compared with only global or local feature, the gait features extracted by GLconv can better represent the gait information. In addition, the local branch of GLconv can produce different partitions. To explore the optimal partitions in the GLConv layer, we further design the ablation study by setting different partitions. Table 5 indicates that the number of partitions has weak influence on accuracy. For example, the accuracy of the GLFE module with N =2, N =4, and N =8 is 97.3%, 97.4%, and 97.4% in the condition of NM. In the method, we finally choose the GLFE module with N =8 to implement the model. Analysis of spatial feature mapping. Traditional spatial feature mapping [3,5] uses F M ax (·) or F Avg (·) to aggregate spatial information, as shown in Equ.7. However, using only F M ax (·), F Avg (·), or their weighted sum, cannot realize the mapping adaptively. In this paper, we introduce the GeM pooling to implement spatial feature mapping. To verify the effectiveness of the GeM pooling layer, we design the comparison experiment by implementing methods with different spatial feature mapping strategies on the CASIA-B dataset. Note that all experiments are implemented with the LT settings. The experimental result is shown in Table 6. It can be observed that the accuracy of GeM (·) achieves the highest average accuracy under all conditions. Specifically, the average accuracy of F 1×1×W2

M ax
(·) and GeM (·) is 91.3% and 91.8%, respectively, meaning that the GeM pooling can obtain more robust feature representation. In addition, in the conditions of BG and CL settings, the GeM pooling performs better than all the competing pooling methods. For example, GeM (·) outperforms F 1×1×W2 M ax (·) by 0.5% and 1.1% on BG and CL settings, respectively. Analysis of local temporal aggregation. To maintain more spatial information, we introduce local temporal aggregation operation to replace the first spatial pooling layer used in traditional gait recognition methods [3,5]. To analyze the contribution of the LTA operation, we design the methods with different downsampling strategies. The experiments are conducted with the LT settings on the CASIA-B dataset. The experimental results are shown in Table 7. It can be observed that the accuracy of using two spatial pooling layers is 96.3%, while the accuracy of "LTA+SP" is 97.4%, indicating that using the LTA operation can achieve better performance than only using spatial pooling. Besides, we also build the model by replacing both spatial pooling with LTA operation. However, the accuracy in this setting has decreased heavily. The reason may be that two LTA operations make more temporal information lost. Therefore, we take the pattern "LTA+SP" as the final downsampling strategy in our method.

Conclusion
This paper proposes a novel gait recognition framework to generate discriminative feature representations under the 3D convolution formulation. First, to extract more comprehensive gait information, we present a global and local feature extractor to extract robust gait features for representation. Second, to make use of more information, we also explore the effect of different downsampling approaches and introduce local temporal aggregation to replace the traditional spatial pooling layer. In addition, we introduce the Generalized-Mean pooling layer to adaptively aggregate the spatial information, improving the feature mapping performance. The experimental results on public datasets verify the effectiveness of the proposed method.