{'fraction_non_alphanumeric': 0.03166443474880124, 'fraction_numerical': 0.02241997494492203, 'mean_word_length': 4.813661476644902, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A new framework for Person Re-identification: Integrated level feature pattern (ILEP)

The system for re-identifying persons is used to find and verify the persons crossing through different spots using various cameras. Much research has been done to re-identify the person by utilising features with deep-learned or hand-crafted information. Deep learning techniques segregate and analyse the features of their layers in various forms, and the output is complex feature vectors. This paper proposes a distinctive framework called Integrated Level Feature Pattern (ILFP) framework, which integrates local and global features. A new deep learning architecture named modified XceptionNet (m-XceptionNet) is also proposed in this work, which extracts the global features effectively with lesser complexity. The proposed framework gives better performance in Rank1 metric for Market1501 (96.15%), CUHK03 (82.29%) and the newly created NEC01 (96.66%) datasets than the existing works. The mean Average Precision (mAP) calculated using the proposed framework gives 92%, 85% and 98%, respectively, for the same datasets.


Related Works
Research works on person re-id methods were done in different directions. Earlier re-id methods used hand-crafted features [11][12][13][14][15] of the images for feature extraction. Different distance metrics [16][17][18][19][20][21] were used to compare these hand-crafted features to attain output. A few researchers [22][23][24] worked both on feature extraction and distance metrics. The hand-crafted features may be based on colour, texture, shape or spatial layout. The most commonly used feature of images is independent of the view, and the resolution of an image is colour [25,26]. The color information was grabbed in the forms of color edge [27], color texture [28], color histograms [29,30], color moments [31] and color correlograms [32]. Shape [33][34][35] is another feature that gets the intensity difference between the regions. Patterns in the image appearance and spatial definition are expressed as texture information [36][37][38][39]. Many researchers worked on similar measures after the extraction of the features. Some of the works related to metric learning for re-id are Local Fisher Discriminant Analysis [23], saliency weighted distances [40], Mahalanobis metric learning [19], Marginal Fisher Analysis [23], Locally Adaptive Decision Functions and attribute consistent matching [17].
Deep learning-based works gave another dimension in person reidentification. These works implemented person re-id with Convolutional Neural Network (CNN). CNN was used to retrieve more features from the images. CNN models were used in two different ways, either as classification models [41,42] or as Siamese models [43][44][45] using image pairs. The person re-id cannot merit a lot of training data since the applications are done in real-time. Even datasets like VIPeR [11] provide only two images for identifying a person. In the CNN model, the feature extraction uses three overlapping parts of the given image by sending it through two convolutional layers and one fully-connected layer. The outputs from these layers are fused for the feature vector. The feature vectors of two images are fed into cosine similarity metrics for finding the match. This work was emendated by Li et al. [41] by adding a patch-based matching layer used for multiplying the responses of convolution layers from incompatible horizontal stripes of two images. Ahmed et al. [46] worked on the Siamese model to improve its effectiveness by calculating the cross-input neighbourhood difference using one location of an image to the features in the neighbouring locations of another image. Deepening this CNN was done using PersonNet [47], a small convolutional filter. Long Short-Term Memory (LSTM) was proposed to incorporate the Siamese network to advance this method [48]. The image parts are processed sequentially to memorise the spatial connections. Varior et al. [49] proposed another advancement in CNN by adding a gating function after every convolutional layer, which gives better accuracy. Many state-of-art surveys were carried out on person reidentification methods [50][51][52][53]. Considering these surveys, the volume of works better comprehended in this research area and the effect of using local and global features are also understood in person re-id works. This knowledge motivated us to incorporate both local and global features into the framework. The main contributions of this paper are.

A novel framework named Integrated Level Feature Pattern (ILFP) is proposed
for person re-id. 2. Global features of the images are extracted using the newly proposed m-XceptionNet. 3. Local features are extracted using spatial-oriented gradient and texture. 4. A new dataset, NEC01, is created and used for performance analysis along with the state-of-art datasets.

Proposed ILEP work with m-XceptionNet
The proposed work implements person reidentification using the integrated-level framework with joint local and global features. The texture information in terms of HOG and LBP are used as local features. These features are optimised using Principle Component Analysis (PCA). The global features are extracted by using m-XceptionNet. Both local and global feature sets are combined in the fully connected layer called the dense layer, which further produces the classification output based on these features.

Spatial-oriented Gradient and Texture
The images of the person reidentification system may not employ many unique features because they may be taken from far apart and in different illumination and contrast. In order to make it accountable for the changes that happen in contrast and illumination, the strengths of gradient measures should be normalised locally. During this process, the cells are grouped to form bigger spatially connected blocks. HOG descriptor [38] is calculated by forming the vector of normalised cell histogram components by utilising the block regions. The calculation is done in an overlapping manner to acquire the contribution from each cell repeatedly.
Similarly, texture information also can contribute to the feature well by incorporating LBP [54].

Descriptor extraction using HOG
The images used for person reidentification may vary in direction, position, illumination, and contrast. These variations should be normalised in spatial domain and direction. They can be normalised by extracting the descriptors in scale space and by normalising the rotation by detecting the extrema of scale-space, assigning the orientation, and extracting the descriptor. Fig. 1 illustrates a patch with their corresponding gradient directions of the given input image.
. Gaussian smoothing is applied for testing the gradients by using several discrete derivative masks. Several smoothing scales had undergone testing. The One-dimensional vector [-1, 0, 1] works best as the mask for testing. If larger masks are used, performance gets decreased. When analysing the strengths of gradient vectors, it differs with local variations and contrast change.
Hence local contrast normalisation helps us to get a proper descriptor block.

Descriptor extraction using LBP
Local Binary Pattern (LBP) is proved to be an efficient local texture feature. The LBP descriptor helps us to retrieve the inter-relationships between the pixels. In-person re-id, this information helps a lot to get the unique feature because the major information that can be retrieved from the image is the relationship between the pixels. Fig. 2 gives the formation of the LBP feature vector. .

Fig. 2. LBP Feature formation of the image
The LBP calculation is done in a 3 x 3 window. For each neighbouring pixel, eight neighbouring pixels exist. The neighbouring pixels are substituted with 1 when that pixel value is bigger than the mid pixel and 0 otherwise. This process gives 8 binary numbers in the neighbouring positions. It is multiplied by where i is the corresponding position. The sum of these values is replaced in the mid pixel position, and the process is done throughout the image to get the LBP descriptor.

Global Feature extraction using m-XceptionNet
Different variations were proposed when the researchers used Convolutional Neural Network (CNN) according to the applications. The design of CNN started with LeNet [42], which has some convolution layer and max-pooling layer. Convolution layers were used to extract the features and sub-sampling in the spatial domain using the max-pooling layer. AlexNet architecture was proposed in 2012 analysed the change of adding many convolution layers between max-pooling layers [55]. Later, more deeper architectures like VGGNet [56] were proposed. During this time, a new architecture style was introduced by Szegedy et al. [57]. They proposed an inception module, as shown in Fig. 3. Stack of modules was used in different inception-based architectures, whereas convolution layers were stacked in the normal CNN architectures. The inception modules are called convolutional feature extractors that can learn richer features using fewer parameters, as depicted in Fig. 3.

Fig. 3. Inception module in InceptionV3
The inception module was initially used for deep learning using the 1 x 1 convolution layer and then different spatial correlations for different output channels. This kind of inception module was handled in GoogLeNet [57], Inception V3 [58] and Inception-ResNet [5].
The next advancement in the deep learning framework was separable convolution which learns deeper than the inception module. In this, spatial convolution is done for every input channel independently, following a 1x1 convolution. Separable convolution works better than the normal inception module due to non-linearity and change in the order of operation. In 2017, another framework called XceptionNet was proposed by Francois completely based on depthwise separable convolution layers. This framework decouples feature maps of CNNs, which contains the mapping of spatial and cross-channel correlations.
The Xception architecture comprises three flows: entry, middle and exit. The feature extraction phase of the Xception network has 36 convolutional layers modelled into 14 modules with linear residual connections except for the first and last modules. Based on this architecture, this paper proposes a new network called m-XceptionNet. The proposed m-XceptionNet is given in Fig. 4. The input image with 128x128 is fed into two convolution layers to get an intermediate output with 64 feature maps. This is fed into two separable convolutions layers and a max-pooling layer. The feature vector is added with the output of the convolution layer. Subsequently, the output of this layer is fed into three sepconv layers. The output image is fed into two sepconv layers and one max-pooling layer. These steps will generate the output with 1024 features maps and is fed into two sepconv layers and one global average pooling layer to produce 2048 feature vectors.
Initially, an image of size 128 x 128 with 3 channels (red, green, blue) is given as input to the network. The convolution operation is applied on the image with 32 filters, each with the filter size of 3 x 3, the activation function of RELU and the stride of 2 pixels. The resultant output dimension is 63 x 63 x 32. Then, another convolution is done with 64 filters, encompassing a filter size of 3 x 3 and a stride of 1 pixel. Later, batch normalisation is applied.
Following this, separable convolution of 128 filters is applied on the output twice with a filter size of 3 x 3, followed by max pooling. Further convolution of output after batch normalisation layer is performed with 128 filters. This output is connected to the output of max pooling using the residual connection. This forms a single block in the proposed network. This block is repeated three times and followed by a similar block, which yields 2048 features after global average pooling.

Fig. 4. Proposed m-XceptionNet
The proposed m-XceptionNet performs well because of depth-wise separable convolution levels used in this architecture. It extracts the information in spatial and depth dimensions. The depth-wise separable convolution can be broken down into two operations, Depth-wise convolution and Point-wise convolution. Fig. 5 shows Depth-wise and Point-wise convolution operations. Depth-wise convolution is the channel-wise n×n spatial convolution. If 3 channels are used in depth-wise convolution, then three n×n spatial convolutions are obtained. When Point-wise convolution is applied, 1x1 convolution is used to change the dimension. Here, refers to the number of pixels on the image, refers to resultant dimension after depthwise convolution is applied and refers to the size of the filter. In depth wise convolution operation, the convolution is applied to a single channel at a time. Here, the input size is , the filter size is , and the number of filters is M. This produces output with size with operations. In pointwise operation, a 1x1 convolution operation is applied on the M channels. The input size as filter size is and the number of filters is N. This produces output with size with operations. Thus, the total number of multiplications in depth-wise separable convolution is as given.

… (1)
With a reduced number of multiplications than the convolution operation, the depth-wise separable convolutions can greatly reduce the computation time involved in the process.

Principle Component Analysis (PCA)
PCA is a method that descends dimensionality and was done by understanding the correlation between the variables. Though the dimensionality reduction is done more or less, this algorithm tries to preserve the variations between the variables to a greater extent. The covariance matrix is calculated for the variables for which the eigenvectors are found. Thus, by completely transforming the type of variables to a different set, the variation is achieved. These eigen vectors are known as the principal components, which are orthogonal. Ordering of the principal components is done to showcase the variation of the variables. The first principal component will give maximum variation, which can be achieved using the original variables. This reduction in variables helps the algorithm to utilise the variables effectively with lesser time. Spatial-oriented texture and gradient using LBP and HOG extract the local features. The global feature of the image is also important since the local feature misses the correlation between the pixels of the entire image. This is implemented by deep learning the raw image. Thus the proposed work focuses on retrieving both local and global features and ascends the accuracy of this work.

ILFP Framework
The detailed proposed ILFP framework is described as shown in Fig. 6. The image is fed into m-XceptionNet, LBP framework and HOG algorithm. The m-XceptionNet produces 2048 features that give the global vicinity of the image.

Experimental Results and Discussion
Person reidentification has a wide scope in the computer vision society in the current decade with various applications like video surveillance, pedestrian search, multi-camera tracking and behaviour analysis. Our experiments are executed on Google Colab with Nvidia GPU. Our network is trained in batches with size 32. The total epochs are set to 200 for Market1501, CUHK03 and NEC01 datasets. Adam optimiser is used with the base learning rate, which is initialised to 0.001. The images are re-sized to 128 x 128 with normalisation, and it has 3 channels (RGB). This proposal has used two dropout layers with a dropout rate of 0.5 and 0.3, respectively.

Datasets
There are quite a few numbers of publicly available datasets that various person reidentification algorithms have tested. Out of all benchmark datasets, the most used CUHK03 and Market1501 datasets have been employed with our system to evaluate its performance. In addition, we have employed our dataset NEC01. Table 1 emphasises the details of datasets experimented on in the proposed work. CUHK03 is a specific setup for each person images with three different partitions for the gathered images in a well-known recent dataset was provided by the Chinese University of Hong Kong (CUHK). It provides the bounding boxes detected from deformable part models (DPM) with manual labelling. CUHK03 has 1, 467 identities which are collected from 5 pairs of cameras and holds 13, 164 images. The dataset is stored in .mat file contains three cells as detected, labelled and testset. Detected means the bounding boxes, which are estimated by the pedestrian detector. Labelled means the bounding boxes that are labelled as human, and testset contains the testing protocols. Fig. 7 shows the sample pair of images taken from CUHK03 dataset. The Market1501 dataset is a dataset for person reidentification collected in front of a supermarket in Tsinghua University. Five cameras with high-resolution and one low-resolution camera are used. Cameras are arranged in overlapping with different cameras. This dataset contains 32, 668 annotated bounding boxes of 1, 501 identities. Each annotated identity is present in at least two cameras so that a cross-camera search can be performed.
The Market1501 dataset is annotated using the following rules. For each detected bounding box to be annotated, a bounding box with ground truth is annotated containing the pedestrian. Then, the ratio of the overlapping area to the union area is calculated for the detected and hand-drawn bounding boxes. If the ratio is larger than 50%, the Deformable Part Model (DPM) bounding box is marked as good. If the ratio is lesser than 20%, the bounding box is marked as a distractor. If not, it is marked as junk, and the image has zero influence on the reidentification accuracy. Fig. 8 shows the pair of sample images from the Market1501 dataset. Most of the benchmarked dataset is available with images of persons who do not belong to a particular region. So, it is planned to test native images with the proposed system. This paper utilises the institution's surveillance cameras fixed at various locations, and the dataset collection is done between Jan 2020 and Sep 2020. Images are captured using 3 disjoint cameras. It consists of 8053 images of both students and faculty members. Fig. 9 gives the sample pair of images from NEC01dataset. . Fig. 9. Few Sample images of NEC01

Performance Evaluation Metrics
Evaluation of the proposed system with rank-i accuracy and mean average precision (mAP) is employed and tested with the dataset discussed in the earlier section. Here, rank-i accuracy estimates the similarity between the query image and the subsequent gallery images. The similarity values are sorted in ascending order and organised in a table. Top-k accuracy is estimated by checking the top-k gallery images which contain the matched query image. If the correctly matched image is found in the i th position of the retrieved list, then rank-i is set as true. Rank can be verified at rank-1, 5, 10 etc.
Performance analysis is done by using Mean average precision (mAP) and Rank-1 accuracy. First, the Precision-Recall curve is drawn. The area under this curve is known as Average precision for that query. The average of all these APs is known as mAP. This measure is meant to be precise because it combines both the precision and recall of an algorithm. Let the query image be matched with 7 images from the gallery, and matched images are positioned at 1 st , 4 th , 5 th and 6 th , as shown in Fig. 10. The precision is marked below each Besides, any classification process is evaluated with accuracy and loss of training and test dataset. These value for each dataset is plotted as a graph in Fig. 11, 12 and 13. The present investigation recorded the results of different datasets discussed on various features ranging from 50 to 200. Experimentations were done for various existing methods as well. Table 2 gives the comparison of different metrics with reference to the number of features on different datasets, and Table 3 gives the comparison of different methods on different datasets. The highest Rank-1 accuracy is achieved on Market 1501 with 150 features. The mAP values reached the highest for NEC01 with 100 features. There is a decline in metric values after reaching 100 features almost on all datasets.
The state-of-art methods adapted to the dataset dealt showed the highest value of Rank-1 accuracy and mAP for combined HOG, LBP, and PCA with 100 features on the NEC01 dataset. For all other datasets also, this combination won with the highest Rank-1 accuracy and mAP metric. The graphical representation of both the results is depicted in Fig. 14 and Fig. 15, respectively.
In Rank1 metric, HOG features with this network have produced 2.72%, 4.15%, and 4.75% than LBP for Market1501, CUHK03 and NEC01. The mAP metric also HOG performed with an increase of 2.3%, 6.17% and 1.11% than LBP.
HOG and LBP used the gradient around a pixel. So, we tried a combination of HOG and LBP features which obtained better results than HOG or LBP separately. 3.19%, 4.73% and 7.74 higher than LBP in terms of Rank1 and 1.16%, 0.56% and 2.85% increase in the HOG Rank1 metric on Market1501, CUHK03 and NEC01, respectively. Similarly, combined HOG and LGP results are better than LBP and HOG in the mAP metric also.
The proposed ILFP framework with 100 features yielded the highest result than all methods adapted earlier with our system on datasets. 1.47%, 0.96% and 2.67% higher than HOG and LBP feature combination in Rank1 metric. 1.1%, no change and 2.08% increase in mAP metric, respectively.

Conclusion
The present investigation proposed an Integrated Level Feature Pattern (ILFP) framework for person reidentification. The ILFP framework assimilates both local features and global features for re-identifying the persons. The spatial-oriented texture information in terms of local-level features is extracted by applying LBP and HOG. The fused features are optimised into 100 features using PCA. On the other hand, to obtain the global features, we propose a new CNN model as m-XceptionNet. This integrated framework extracts both pixel-level and overall features, which helps to get better performance over the other existing works. ILFP framework performed different features ranges of 50, 100, 150 and 200. In that, our ILFP with 100 features provides better Rank1 accuracy as 96.15%, 82.29% and 96.66% for Market1501, CUHK03, NEC01, respectively. The mean average precision (mAP) accuracy is highly appreciable as 92%, 85%, 98% for Market 1501, CUHK03, NEC01, respectively. Also, the proposed ILFP is compared with different methods, and the performance is plotted. As future research, local features using Scale Invariant Feature Transform (SIFT) and Speeded Up Robust Features (SURF), Binary Robust Invariant Scalable Keypoints (BRISK) feature detector and descriptor can be detected. The global features are extracted using m-XceptionNet for the ILFP framework to improve the effectiveness of the person re-id process.