{'fraction_non_alphanumeric': 0.028335564480142794, 'fraction_numerical': 0.015952699687639446, 'mean_word_length': 4.459500609013398, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Recognition of sign language hand gestures using leap motion sensor based on threshold and ANN models

Hand gesture recognition is a topic that is still investigated by many scientists for numerous useful aspects. This research investigated hand gestures for sign language number zero to nine. The hand gesture recognition was based on finger direction patterns. The finger directions were detected by a Leap Motion Controller. Finger direction pattern modeling was based on two methods: threshold and artificial neural network. Threshold model 1 contained 15 rules based on the range of finger directions on each axis. Threshold model 2 was developed from model 1 based on the behavior of finger movements when the subject performed hand gestures. The ANN model of the system was designed with four neurons at the output layer, 15 neurons at the input layer, seven neurons at the first hidden layer and 5 neurons at the second hidden layer. The artificial neural network used the logsig as the activation function. The result shows that the first threshold model has the lowest accuracy because the rule is too complicated and rigid. The threshold model 2 can improve the threshold model, but it still needs development to reach better accuracy. The ANN model gave the best result among the developed model with 98% accuracy. LMC produces useful biometric data for hand gesture recognition.


INTRODUCTION
Nowadays, the way human operates machine continues to evolve away from a physical button or relay to a virtual control method [1]. The interaction between human and machine becomes more flexible and mimic the communication between humans. Some methods were developed to control machines such as eye control [2] and muscle control [3]. Hand gesture is one of the methods developed by numerous researchers not only for sign language but also to build interaction between humans and machines and flex sensors [4][5][6][7]. A user interface that can be operated by mid-air hand gestures will become more popular shortly because of the expansion in the virtual technology and smart devices [8]. A hand gesture controlling application for computers such as on a website, VLC player and PowerPoint was developed by [9].
Some sensors are being developed such as flex sensors [10], gyroscopes [11] and cameras [12] to detect human activities. These sensors can be used together using a data glove for the hand gesture recognition system. A data glove is a special function glove integrated with hand gesture sensors [13]. The sensors are attached to the glove. The sensors measure physical variable of movements when the hands move [14].Camera sensor faced challenge to remove noise from background of the hand. The average accuracy of the system with plain background was 94%, but it was only 64% when the background was not plain [9].
Measurement identifiers for hand gestures were divided into seven groups in [15]. They were finger position, finger direction, finger velocity, palm position, palm velocity, palm normal plane, and hand direction. These variables could be detected well by the Leap Motion Controller (LMC).This device can turn the hands and fingers into input devices to control a computer as substitutes of a mouse and keyboard [16]. The LMC could detect hand gesture and be used for rehabilitation purposes [17]. The hand gesture was also able to control a robot through the LMC [18]. It proves that LMC has a good performance to get biometric data.
An anthropomorphic gripper was controlled by hand gestures based on an LMC. The performance of the LMC in the detection of hand gestures to control anthropomorphic hand could match the performance of a data glove [19]. The accuracy of the leap motion sensor and the data glove in reading the bending angle was also investigated by [14]. The average percentage error for the bending angle of LMC was larger than the data gloves. But the performance of LMC based on accuracy to control soft finger was better than that of the data glove.
The performance of the LMC to detect 10 gestures were studied. Its performance was also compared to Kinect devices [20]. The characteristics of fingers used in leap motion were fingertip distance, fingertip angles and fingertip elevations. On the other hand, Kinect had curvature and correlation. Support Vector Machine (SVM) method was implemented to recognize the hand gesture. The accuracy of the LMC was 80.86% and the Kinect was 89.71%. It shows that the accuracy of the LMC was better than the data glove but lower than the accuracy of the Kinect. But, LMC was cheaper, lighter and smaller than the Kinect device.
Object tracking [21], feature extraction and recognition [22] method are the main issue in a hand gesture recognition system. A Hidden Markov Model (HMM) was trained with the Baum-Welch algorithm using Left-Right Banded [23]. The average recognition rate of this system was 90%. Superpixel-Graph Mover's Distance (SP-EMD) method remained the problem of human habits to perform hand gestures. These habits formed variations of hand shapes. A canonical superpixel-graph was developed by [24]. In this research, hand gestures were recognized by a real time system. The finger positions were the input for the system. The characteristics of finger movements on the X, Y, and Z-axis of the leap motion were investigated to build the model of hand gesture. Two models were developed: threshold model and artificial neural network and their performances were studied.

RESEARCH METHOD
This is a real time hand gesture recognition system. The inputs are hand gestures and the sensor is an LMC. The user interface between human and machine are made in the Processing Software. Users can directly see the interpretation of the machine on their hand gestures.

Hand gesture and finger direction
There were ten hand gestures investigated in this research as shown in Figure 1. They were the signs for number 'zero' to 'nine'. The gestures were performed by the right hand of the operators. The side of hand seen by readers was the same side of the hand facing the leap motion infrared camera. The position of the hand was about 25 cm in front of the leap motion camera, as illustrated in Figure 2. The Y-axis of the sensor was toward the hand. The Z-axis was the opposite of the gravity direction and the X-axis pointed to the left side of the operators. The hand gesture in this research was recognized based on finger directions as the feature. The finger direction (σ) depended on its trigonometric function of angle (θ) about the X, Y and Z-axis. Thus, a finger had three directions. The direction ranged from 1 to -1.If the angle was 0° than the finger direction was -1. If the angle was 180° than the finger direction was 1.The relationship between the angle and the direction was investigated by experiments in this research.

Threshold method
The threshold model was developed based on the minimum and maximum values of finger directions. The finger directions of five subjects were recorded when they performed number zero to nine. They demonstrated each number 80 times. So, there were 400 data for each number from five subjects. For those data, minimum and maximum finger directions were calculated. These values were set for the threshold condition. A gesture was only recognized if the finger direction ranged from minimum and maximum value. So, if a finger worked in 3 different axes, there were 15 rules from five fingers for a sign number. These rules were investigated to be simplified.

Artificial neural network method
The outputs of the system represented number zero to nine with the inputs from the right-hand gestures. To represent the numbers, four-bit binaries were designed for the neural network outputs (o1 to o4). The binary number for zero was 0000 and the binary number for nine was 1001. The complete information for this representation is shown in Table 1.
The directions of the five fingers as the features of the right hand became the inputs of the artificial neural network. There were three directions for each finger: the direction for X, Y, and Z-axis. Thus, the system which recognized the right hand with five fingers needed fifteen inputs (i1 to i15). Two hidden layers were implemented between the input and output. The first hidden layer had 7 neurons (j1 to j7) and the second hidden layer had 5 neurons (k1 to k5). Two hidden layers and output layer had bias (bj, bk and bo), as shown in Figure 3. The activation function of the neuron was a logsig function. The artificial neural network was trained using Lavenberg-Marquardt and the performance of training was evaluated based on mean square error.

Research procedure
Five volunteers participated in this research to determine and train the model based on their hand gestures. The first experiment was to study the characteristic of hand gestures based on finger directions. Then, they performed gestures eight times for each number to establish finger directions for number zero to nine. So, there were 4000 gestures from five volunteers for the ten numbers. The finger directions were recorded for minimum, maximum and average points. Based on those statistical data, two threshold models were designed and the neural network model was trained. Three models of hand gestures were designed. Their performances were studied based on accuracy. The threshold model and neural network model were tested using 1000 data.

Characteristic of finger directions
The characteristics of finger directions were investigated based on two experiments. The first experiment is shown in Figure 4. The first experiment had two types of finger movements. Index finger, middle finger, ring finger and little finger were of the first type, as shown in Figure 4(a). The fingers started from the straight position (0°) than bent to 30°, 60°, 90°, 120°, 150°, and 180°positions. Thumb finger movement was of the second type, as shown in Figure 4(b). The directions of the thumb finger were recorded for 0°, 30°, 60°, 90°, 120° and 150°.  The result shows that the finger directions based on experiment 1 have patterns on X-axis, Y-axis and Z-axis, as shown in Figure 5.Based on the X-axis results, thumb finger and index fingerhave different patterns compared to the other three fingers. Whenthe fingers bent from 0° to maximum position, the direction of thumb finger increased when it decreased on the other fingers. When the thumb finger moved to the right side,the other three moved to the left of the hand.It causedan increment of direction for the thumb finger. The state was different for index finger on the X-axis. Its direction almost had no difference on all bending situation. It concluded the situation that the index finger stayedalmost at the same position on the X-axis.

477
The directions of fingers on the Y-axis have a clear pattern. Index, middle, ring and little finger have the same pattern. Their directions started at 0.0, reached the maximum point which was almost 1.0 and then decreased to 0.0. The 0.0 direction describes that the fingers are perpendicular to the Y-axis and the 1.0 direction means that the fingers are about 180° to the Y-axis.Especially for thumb finger, the direction started from 0.25 and reached the maximum position on 0.70.
The most explicit linear relationship between hand gesture and finger direction can be gained on Z-axis. The 0° position has -1 direction and the 180°position has 1 direction. However, in the case of the thumb finger, it has a small range of direction on the Z-axis because of the limitation of its movement. It is better to choose X-axis to detect the thumb finger movement because it has the largest linear area than other axes. The advantage of an extensive range linear area is to provide more accuracy to detect the finger angle. The second experiment investigated the directions of fingers when they moved only on XZ-plane. The subjects pulled the finger together for the normal position. Then the finger spread apart in three steps: small, medium and big spreads. The hand gesture for this experiment is shown in Figure 6. Based on the result shown by Figure 7, the direction of fingers changed for thumb,ring and little finger with a linear trend.But, the index finger and middle finger almost stayed in the same direction on X-axis. The directions of all fingers on the Y-axis tended to be similar among all hand gestures. There was no impact of the gesture to the finger directions. It changed randomly but in a small range direction. It correlates to the direction of the finger which is always the same as the Y-axis which is 90°. The direction of this angle is 0.0. The same situation happened on Z-axis. The directions of fingers were almost the same for all gestures. The normal position shows that all fingers were about 0° from the Z-axis. Then four hand gestures only gave a small change on it. The relationship between hand gesture and finger direction with the linear method on this axis is indistinguishable.

Accuracy 3.2.1. Threshold model 1
This is the basic model. Five subjects performed hand gestures 10 times for number zero to nine. Three variables were calculated based on those data: minimum, maximum and average of finger directions, as shown in Figure 8. The minimum and maximum of the directions were used for the threshold model. In Method I, all fingers considered all axes. The resulting accuracy of this model is still low. Only numbers zero and four were detected with 60% accuracy by this model. But on average, the accuracy is only 28%. The worst condition happened with numbers 1 and 2 where this model failed to detect any gesture for this number. The complete data are shown in Table 2.

Threshold model 2
This model was developed based on threshold model 1 by considering the patterns of directions based on the experiment 1 and 2. From those experiments, some fingers only had slight movement in specific axes. Thus, the rule was deleted for that situation. Table 3 was used to build threshold model 2. Only X-axis was considered to build the rule for the thumb finger. The little and ring finger had complete rules on the X, Y, and Z-axis. Nevertheless, the index and the middle finger only worked on Y and Z-axis. The accuracy of threshold model 2 is slightly better than the threshold model 1, as shown in Table 4. However, this method is not recommended for the hand gesture recognition system. This method had difficulty in anticipating the variety of human hand shapesdue to differences in habits of each person during hand movements [24].  Table 3. Active axes to build threshold model 2

Artificial neural network
An artificial neural network was designed to recognize the hand gesture. The number of input was 15. They were the directions of five fingers in the XYZ-coordinate system. There were two hidden layers. The first hidden layer had seven neurons and the second hidden layer had five neurons. There were 400 data for training and 100 data for testing. The result shows that the best validation performance was 2.48×10 -6 at epoch 38. The correlation between the output and the target for training is 0.99. There is three weighted phases for this neural network: between input and the first hidden layer,as shown in Table 5, between the first hidden layer and the second hidden layer,as shown in Table 6 and finally between the second hidden layer and output layer,as shown in Table 7.  The performance of ANN with 15 inputs and four binary outputs can solve the problem of accuracy in the threshold model. The ANN accuracy is 98% with 1000 hand gesture data for testing, as shown in Table 8. The only mistake occurred for gesture zero. The position of fingers of this gesture made the LMC fail to calculate the hand gesture. As shown in Figure 1, the position of the palm for zero is different from the other gestures. The palm is perpendicular to the Y-axis for zero gesture but it is 180° for the other nine gestures. The thumb finger located at the farthest location to the origin of LMC. Other fingers hide it. On the other hand, the accuracy of LMC to detect the angle of thumb is the lowest than the accuracy to detect other fingers [14]. Accuracy is the challenge of hand gesture recognition using LMC. Table 9 shows the performance of some hand gesture methods. The hand gesture recognition system using LMC was developed by [17]. This research applied the Support Vector Machine (SVM) and K-Nearest Neighbor (KNN) to recognize the hand gesture. The accuracy of the system was 97.3% and 97.9%. The hand gesture recognition for the Arabic number was studied by [25]. The hand features were finger position and the angle on X and Y-axis. It has 2-fold and 10-fold methods to recognize the hand gesture. The performance was 94.2% and 95.1%. In this research, AAN can recognize 98.0% hand gesture for numbers. The difference between [25] and this research is the hand gesture used to represent the numbers. EOG [2] and EMG [3] and flex sensor [4].

CONCLUSION
In this study, the gestures of the right hand are detected using LMC. LMC calculated the position of five fingers for sign language of number 0 to 9. Two methods for the recognition system were based on threshold and ANN. The threshold method was design based on the minimum and maximum finger directions. Recognizing hand gestures using the threshold model is very difficult. This method has difficulty accommodating the variety of hand shapes. The system needs more training data to improve the accuracy of this method. An ANN model can give better results with the number of data for training. An ANN with 15 inputs and four binary outputs can raise the accuracy of the system up to 98%. The problem with LMC is the capability to detect the thumb finger for number zero because the other fingers hide it. The implementation of ANN to the features of fingers driven by LMC is an alternative method for human and machine interaction.