{'fraction_non_alphanumeric': 0.029253969404770534, 'fraction_numerical': 0.014246356847676358, 'mean_word_length': 4.878329426805881, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Hand Side Recognition and Authentication System based on Deep Convolutional Neural Networks

: The human hand has been considered a promising component for biometric-based identification and authentication systems for many decades. In this paper, hand side recognition framework is proposed based on deep learning and biometric authentication using the hashing method. The proposed approach performs in three phases: (a) hand image segmentation and enhancement by morphological filtering, automatic thresholding, and active contour deformation, (b) hand side recognition based on deep Convolutional Neural Networks (CNN), and (c) biometric authentication based on the hashing method. The proposed framework is evaluated using a very large hand dataset, which consists of 11076 hand images, including left/ right and dorsal/ palm hand images for 190 persons. Finally, the experimental results show the efficiency of the proposed framework in both dorsal-palm and left-right recognition with an average accuracy of 96.24 and 98.26, respectively, using a completely automated computer program.


I. INTRODUCTION
A Biometric system is known as a system that allows the recognition of a person characteristic of using his/her behavioral or physical biometric traits. Unlike traditional methods as tokens and passwords, biometric traits cannot be copied, forgotten, or lost. Nowadays, biometrics system provides more efficient and reliable means of identity verification. There is a wide range of modern technologies for biometric authentication and identification, where various body features can be used to determine a person's identity, including hand shape, iris shape, face geometry, fingerprint and even speech patterns [10]. The human hand is one of the most used biometric in security systems all over the world. Human hand contains distinguishable features that are capable of authenticating the identity of an individual, through which you can gain access to laptops, tablets, and mobiles [1]. Hand geometry-based identification systems use the geometric features of the hand such as the width and length of the fingers, the perimeter, and the diameter of the palm. Currently, hand-based biometric authentication and identification technology play an important role in providing safety for real-time environments based on human interaction due to its low cost in acquiring information and its reliability in identifying the individuals [2]. Manuscript  In fact, it is found to be more efficient than other biometrics in terms of system speed and accuracy [3].
For authentication and identification purpose, hand recognition should be performed by systematically analyzing specific features that are common to everyone's hand, such as the fingerprint, the length of fingers, the distance between joints, and so on. These features are then combined in a single feature vector that uniquely identifies each person using a suitable classifier. To extract such features, it is necessary to recognize the hand side, where some of these features are extracted from the dorsal hand side, e.g., the distance and appearance of the finger joints, where other features are extracted from palm hand side, e.g., fingerprint. If we take our hands with its features into account, the dorsal hand-side can be used as a promising future for human authentication. To this end, this paper introduces a robust framework for hand side recognition and biometric authentication based on texture enhancement and deep learning.
Neural networks have seen an interest over the last years and successfully used across an unusual range of problems due to their high accuracy and speed [4]. Convolutional Neural Networks (CNN), or so-called ConvNet, is a well-known type of artificial neural network that is specifically designed to process and recognize large pixel data. CNN can take an image as input, define biases learnable and weights to different objects in the image and be able to differentiate one from the other. The major advantage of using CNN in classification and recognition is its reliability in learning distinct features immediately from the raw input image and predicting it. In addition, CNN overrules the need for a traditional hand-crafted feature extractor, which is computationally intense and requires previous information of the image types.
CNN is shown to be efficient for several tasks in image processing and computer vision, such as image transformation [5], image segmentation [6], texture synthesis [7], text recognition [8], image classification [9,11], face detection and recognition [12,13,14], handwriting character recognition [15], human eye tracking [16], hand vein recognition [17][18][19], and many other applications. For instance, a pattern recognition method for dynamic hand gesture recognition was presented in [20]. This model combined a CNN with a weighted fuzzy min-max (WFMM) neural network. Each module performed by feature extraction and analyses. The results showed that the proposed method can reduce the effect caused by the temporal variation and spatial of feature points.
CNN was also used in [21] to recognize hand gestures. The calibration of hand position and the skin model orientation were used to get the testing and training data for CNN. The experimental results of seven hand gestures showed the feasibility and reliability of this method with average accuracies of 95.96%.
The authors of [25] introduced a TensorFlow, which is an open-source deep learning system that has been recently developed to be performed at large scale and in heterogeneous environments [26]. TensorFlow has been successfully used for several tasks and applications, such as handwritten digit recognition [27], facial recognition [28], Mammogram classification [29], and estimation of GNSS single frequency ionospheric delay [30].
In [31], the Authors classify MNIST datasets using Tensorflow, and they have used Rectified Linear Unit (ReLu), softsign, oftplus, sigmoid and Hyperbolic Tangent (tanH) to contrast the effects of various activation functions on classification results. The result shows that the most accurate classification obtained using the relu activation function.
A CNN architecture contains three main parts [32], which are a) Input layer that get the input images, b) Hidden layers which are usually pooling and convolutional layers that break up the image into features and analyzes them, and c) A fully connected layer that takes the output of pooling/convolution layers and predicts the more suitable class describing the input image. However, these three parts are displayed in Fig.  1. As shown in the figure, CNN consists of many layers, i.e., (1) an input layer of raw image, (2) convolution/pooling layers, and (3) an output layer representing the image labels or categories.
The goal of the convolutional layer is filtering; to check patterns in the part of the image called the "feature extractor" layer for features of the image are extracted within this layer by applied a spectrum of convolution kernels to the input and crossing the result using a sigmoid function. Subsampling or Pooling layer partitions the input image into a set of non-overlapping rectangles and to reduce the resolution of the feature vectors; it is used between two convolution layers and as a result, it reduces the memory consumption, the required computation, and the number of parameters. Finally, a fully connected input layer takes the output of the Subsampling layers. Then, it flattens them and transforms them into a single vector to give the final probabilities for each label.

II. LITERATURE REVIEW
In this part, we display a summary review of the previous hand side recognition and biometric authentication methods.

A. Hand side recognition
A dorsal hand recognition and validation scheme was proposed in [56]. To capture the hand image, a camera mounted on the top of a laptop screen was used to capture the hands on the keyboard and identify the frames by the adaptive chromatic procedure. The recognition was then performed by extracting some distinct features, which were analyzed in the Levenberg-Marquardt using the neural controller.
In [57], a new system for person recognition and verification based on their right hand's images was introduced. This method assessed two feature sets, i.e., independent component features and Hausdorff distance of the hand contours of the hand silhouette images. The results showed that classification and verification performances found to be satisfying.
In [58], authors combine hand geometry features and palm print to produce high performance. They used more than one feature: hand geometry (consists of hand and fingers length, hand and fingers width), palm length and width, finger length, and palm ratio. After create a bounding box of each object and obtaining 20 of features from each image, the features are combined to create a feature vector. The experimental results shown that the proposed system obtains an accuracy of 95.5%

B. Biometric authentication
In [39], the authors present an authentication method for mobile devices equipped with multi-touch screens by applied especially designed multi-touch gestures with one swipe. Both behavioral characteristics and hand geometry are recorded to use in authentication. The performance of the proposed framework is evaluated using a multi-touch dataset collected from 161 people. The experimental results show that the proposed method achieves an EER of 5.84% and the performance is improved to an EER of 1.88% with training.
The authors of [40] present a methodology for a contact-free low-cost multimodal biometric system using principal palm lines and hand geometry based on geometric relations and image processing algorithms for key point detection and palm image extraction. The experimental results for this method show promising results.
A multi-biometric system that combines slap fingerprints, hand geometry and palm dorsal vein for personal authentication was proposed in [41]. The authors present an acquisition that can get slap images and IR hand dorsal images together to procure images. The Palm dorsal vein has been extracted from the palm dorsal area while the width and length of labeled fingers have been utilized to determine the IR hand geometry. An IR hand dorsal image has been segmented into the palm dorsal area and fingers. The experimental results show that incorporating the knowledge of IR hand images can be used for authentication.
In [42], the authors present a multifactor user authentication based on the geometry of hand and the motion signal of a piece of in-air-handwriting. The proposed method is applied on a dataset of 100 users, and the obtained results show a high performance with 0.6% EER.

III. PROPOSED METHOD
The proposed framework can be divided into three main stages. In the first stage, the hand images are segmented by morphological filtering, automatic thresholding, and active contour deformation, and then enhanced by refining the background of the hand images. The second stage classifies the hand side to left/right or palmer/dorsal using CNN trained by the previously enhanced hand images. The classification phase performs in two steps: testing and training steps. The final stage is biometric authentications using the classified hand images.

(A) Hand Segmentation and Enhancement
The purpose of this phase is enhancing the appearance of the region of interest and eliminating the noise in the hand images prior to training and testing procedures. To achieve this, the green channel of the colored hand image is extracted and used to determine the Region of Interest (ROI) of the hand image because its weight is bigger than the red and blue color channels in forming the most important component, i.e., luminance component, in the coloured image. Then, average filter is applied to eliminate the noise occurred during image acquisition.
Then, the Antistrophic Diffusion Filter (ADF) [33], with (k = 0.08) and three iterations, is applied to eliminate noises and needles details while preserving the edges in the image. Automatic Otsu's thresholding is then applied to get the initial segmentation of the ROI. To enhance the obtained segmentation, hand regularization is utilized by applying morphology filter, i.e., erosion, to refine the hand contour, and then, dilating the obtained object to retrieve the boundary pixels missed during the previous operation. Finally, the final segmentation of the ROI is obtained by performing active contour process to get more accurate segmentation. In this step, the Chan-Vese algorithm, which is region-based active contour deformation method proposed in [34], is performed due to its fast, flexible, and accurate implementation. After getting the accurate detection of the ROI, the background of the hand image, i.e., Region of Non-Interest (RONI) is refined to highlight the hand region. Fig. 2 below summarizes the hand segmentation and enhancement phase. Samples of hand images before and after the proposed enhancement phase are displayed in Fig. 3. As can be shown in Fig. 3, the applied segmentation and enhancement process eliminate the noise around the hand contour and highlight the hand appearance.

(B) Hand-Side recognition
To classify the hand image to left/right or dorsal/palm side, CNN is trained on the enhanced images obtained from the previous phase, and then used to classify any input image. To this end, a 7-layer CNN is designed for the hand side recognition problem.
The input layer contains (50x50=2500) neurons corresponding to the hand side images. However, the hidden layers contain a convolutional layer of 3 convolutional filters with a pixel kernel window utilized on the input patches. The next layer is a max-pooling layer with (2X2) subsampling ratios. However, the training and test steps of this phase are summarized in Fig. 4. To achieve this, two activation Functions have been used, one to the hidden layer and the other one to the output layer. Rectified Linear Unit (ReLu) activation Function [35] has been used in the Hidden layer.
ReLu is chosen to augmentation the nonlinearities in the network because it making the decision function more distinguishing [36]. In fact, the constant gradient of ReLu results in faster learning. In the output layer, a sigmoid function, which is a Logistic function, has been used and it means that whatsoever the input, the output ranging between 0 and 1.

(C) Biometric authentication
Biometric authentication is a security process comparing an input biometric data to the confirmed authentic data stored in a database to verify that a person is who is says he is. For authentication purpose, we utilize hash function on the previously classified hand images of 50 persons.
Hash algorithms various from cryptographic hashing algorithms where small changes in the image give fully different hashes since the hand images belonging to different users have some differences. The image hash algorithm (e.g., difference, perception, average, and wavelet) analyzes the image construction on luminance without color acquaintance. Image hash function work even if the images are compressed, resized, or with adjusted colors or contrast. In this paper, average hash algorithm is used to tell whether two images look nearly identical with a difference ratio less than or equal to 5%.

IV. RESULT AND DISCUSSION (A). Dataset Details
For the experiments, a publicly available hand dataset, i.e., 11K hand dataset [37] has been used. The dataset includes 11.076 hand images of size (1200x1600 Pixels). These images are belonging to 190 persons, between 18-75 years old, the person required to close and open his fingers of the right and left hands randomly, and each hand was captured from both palmar and dorsal sides. Table 1 below summarizes the detail of the 11K hand images dataset. Fig. 5 displays samples of the hand images from 11K hand dataset, categorized as left/right side images (first row), and dorsal/ palm side images (second row).
For image segmentation and enhancement in hand-side recognition and biometric authentication, the designed framework was implemented using non-optimized MATLAB. For the hand-side recognition, a TensorFlow machine learning package provided by Google at [38] were used. TensorFlow is Python-embedded domain-specific language for hardware-accelerated machine learning. In biometric authentication, we use the hash function implemented in python with a Similarity ratio of (0.95). Both phases and systems were performed on a PC Pentium 3.2GHz i5 and with 8 GB RAM.

(B). Evaluation parameters
In this paper, we use several evaluation standards to evaluate the proposed framework. The process of evaluating the model is one of the most important ways to measure the success of the model in the future and to obtain the best model for representing the data. Below is a detailed description of the metrics used in the evaluation process.
Accuracy: It is considered one of the most important methods used to judge the success of the model. It can be applied to calculate the accuracy of the proposed system of hand images, as follows: TN is the true negative states, (2) Recall: also known as sensitivity is the number of correct positive instances divided by the number of all relevant samples. It is measure by the following equation: … Eq.
(3) F1 Score: Calculated by the harmonic average of both accuracy and recall, the best value for the f1 score when it is closer to 1 and it is worse at 0.
... Eq. (4) Specificity: is the proportion of negative instances out of the total actual negative instances and it is calculated as following: ... Eq. (5) Correlation Coefficient: or the coefficient of correlation (CoC) measure is a method for establishing the probability that a statistical relationship exists between two variables. It is ranged between -1 and 1, and can be calculated as following: ... Eq. (6) False Positive Rate (FPR): a ratio of the false negatives to the sum of the false negatives and true positives, calculated as following: ... Eq. (7)

(C). Performance evaluation
First of all, for hand-side recognition, several experiments were performed on randomly selected images from the entire dataset to choose the most suitable parameters to get the best recognition performance. For instance, the structure element used for morphological filtering was assigned to 10. This value was selected based on the ROI size in the dataset, if a higher value was used, some shade regions would be included in the ROI mask, whereas using a smaller value would lead to miss some ROI pixels. For hand contour refinement, there were three parameters used by the Chan-Vese method; the smoothing parameter was set to 1 and the contraction-bias parameter was set by default to -1. The maximum number of iterations was 50 iterations for the 11K hand dataset used in the evaluation phase. The proposed framework evaluation in this paper has three-fold: first is the evaluation of left/right side recognition performance, second is the evaluation of the dorsal/palm side recognition performance, and third Biometric performance evaluation. The details of the experimental results are discussed in the following sections.

Left-Right Hand side recognition performance:
To train the proposed CNN-model for left/right recognition purpose, the dataset was split into two parts: training and validation datasets with 0.3. The total number of samples was 5608. However, the model was trained with 3925 samples and validated with 1683 training. The dataset was performed up to a maximum of 15 epochs using the CNN method with batch size of 32. The left/right side recognition performance was summarized in Table 1. As can be seen, the proposed framework yielded a high performance in left/right side recognition. It obtained an average accuracy of 98.26, precision of 99.24, recall of 99.96, and F1 score of 99.6. Moreover, the proposed framework achieved a high specificity value of 99.26 and a low false positive rate of 0.0074. These values proved the reliability of the designed framework in determining if the given hand images are for left or right hands. In addition, this will help researchers in partitioning the training images automatically instead of doing this manually. Fig. 6 displays the training and validation accuracy and loss with different epochs. As shown in the figure, the 15 epochs gives the best performance in the left-right hand side recognition. As we can see, the model yields the best results in training the given data with a high degree of flexibility because it gets close to all the points with a minimum value of validation and training loss. In addition, it gives a maximum value of training and testing accuracy. Depending on the number of correct and incorrect predictions, it is shown that the classifier performs well.

Dorsal-Palm Hand side recognition Performance
To evaluate the design framework in dorsal/palm recognition, the CNN model was trained with 7557 samples and validated with 3240 samples, with a total number of hand samples of 10797. The model was performed up to a maximum of 15 epochs and a batch size of 32. Table 3 shows the performance results for dorsal/palm recognition. The proposed framework successes in distinguishing the dorsal and palm hand images, with a high accuracy of 96.24. In addition, it yielded a high precision, recall, and F1 score of 99.98, 98.55, and 99.26, respectively. The obtained correlation coefficient was 0.9 and the false positive rate was very low, i.e., 0.001.This proves that the designed framework can be reliably used to classify the provided hand images into dorsal or palm images, so that, the appropriate features will be extracted from the given images. Fig. 8 shows the validation and training loss and accuracy obtained with different epochs by the model of left-right hand side recognition. The model fits the training data well after the 15 epochs and it does well in the testing set. The model gets close to all the points and it gives a maximum value of training and testing accuracy and minimum value of training and validation loss.
Furthermore, the performance of the dorsal-palm hand side recognition was evaluated by 2-Class Confusion Matrix as shown in Fig. 9, the confusion matrix shows that the proposed model obtained 5532 true negative instances, 81 false positive instances. The model had only one false negative instance and 5183 for true positive. According to these numerical values, the designed model performs well and gives a high number of correct predictions. Overall, the experimental results of the designed framework proved its reliability and efficiency in solving both left/right and dorsal/palm recognition problems. Our evaluation shows that both validation loss and accuracy were in concurrently with the training loss and accuracy in each model. It also shows that the model is not overfitting, where the validation loss was decreasing and there was not much gap between training and validation accuracy.
For further evaluation, the designed framework trained with 11K hand dataset was applied on different hand images from various sources, and its success in giving the correct prediction. However, there still have some failed cases due to the low quality of the captured hand images. During the experiments, it has been noticed that both models failed in recognizing the blurry hand images, the images of hand with many accessories. Such these situations make the hand features not clear and discriminative. As a result, giving a correct predication becomes more difficult.

Biometric Authentication performance
To evaluate the designed framework in biometric authentication, 100 hand samples of dorsal images and a 100 hand samples of palm images were used. The hash function get 50 images of hand as input and compare them with 50 other images belonging to the same people in both cases (Dorsal / Palm hand-side).

Biometric authentication using dorsal hand side
The biometric authentication performance using dorsal hand side was summarized in Table 3. As can be seen, the proposed framework yielded a high performance, where it obtained an accuracy of 95, precision of 94.11, F1 score of 95.04, recall of 96, and sensitivity of 96. Moreover, the proposed framework achieved a high specificity value of 92.5 and Negative prediction value of 95.9. These values proved the reliability of the designed framework in determining if the given hand images are for the authentic person or not.
The performance of the biometric authentication using dorsal hand side was displayed by 2-Class confusion matrix as shown in Fig. 10 (a). The confusion matrix shows that the proposed model obtained 47 true negative instances, 3 false positive instances, 48 of true positive and only two false negative instances. According to these numerical values, the designed model performs well and gives a high number of correct predictions.

Biometric authentication using palm hand side
Our biometric authentication using palm hand side obtained a high accuracy of 96.24. In addition, it yielded a high precision, recall, and F1 score of 99.98, 98.55, and 99.26, respectively. The obtained correlation coefficient was 0.9 and the false positive rate was very low, i.e., 0.001. Fig. 10 (b) represents the 2-Class confusion matrix to further describe the performance of the authentication using palm hand side model. As can be seen, our model obtained true negative instances of 48, only two false positive instance, false negative instances of 4 and true positive instances of 46. Fig. 10 (b) represents the 2-Class confusion matrix to further describe the performance of the authentication using palm hand side model. As can be seen, our model obtained true negative instances of 48, only two false positive instance, false negative instances of 4 and true positive instances of 46.
Overall, the experimental results of the designed framework proved its reliability and efficiency in solving both left/right and dorsal/palm recognition problems. Our evaluation shows that both validation loss and accuracy were in concurrently with the training loss and accuracy in each model. It also shows that the model is not overfitting, where the validation loss was decreasing and there was not much gap between training and validation accuracy.
However, there still have some failed cases due to the low quality of the captured hand images. During the experiments, it has been noticed that the authentication method failed to identify the images of a hand with accessories and If the two pictures are not in the same position. Such states make the hand characteristics not clear. As a result, giving a correct prediction is more complex in these cases.

V. CONCLUSIONS AND FUTURE WORK
This paper presents a fully automated hand side recognition and authentication framework based on deep learning and hashing method. The performance evaluation proves the efficiency of the proposed framework in both recognition and authentication problems. The high average accuracy of 96.24 is obtained for dorsal/palm recognition and 98.26 for left/right recognition. Moreover, the proposed method proves its efficiency in detecting authentic users. The experimental results show that the presented method obtained accuracy of 95 using dorsal hand side and 94 for palm hand side. For future work, the proposed framework can be extended to solve more complicated recognition problems, such as age and gender recognition problems using hand images.