{'fraction_non_alphanumeric': 0.02541598098372646, 'fraction_numerical': 0.014298774913146827, 'mean_word_length': 4.76433389544688, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Toward an immunity-based gait recognition on smart phone: a study of feature selection and walking state classification

Toward an immunity-based gait recognition to integrate the identification results from multiple smart phone sensors, in our previous study, we collected gait data using an iOS application when some subjects walked along a corridor carrying an iPhone in the following 3 holding states: 1) in the pocket, 2) holding the phone to the ear, and 3) looking on the screen. We extracted 43 features from the user-generated acceleration data of all the recorded gait data, and then identified subject using some machine learning algorithms. However, we have not yet carefully examined the necessity of the 43 features. We also have coped with only walking on a flat land. In this study, we first develop a similar gait record application on Android devices. Using the application, we collect gait data for 15 subjects in 5 walking states adding to going upstairs and downstairs. From the 3-axes accelerometer data, we extract 52 features adding 9 features of maximum, minimum, and energy for each axis to the 43 features, and then select the subset of the 52 features with small number and high accuracy. The result shows that the accuracies of going upstairs and downstairs are improved by the feature selection. © 2016 The Authors. Published by Elsevier B.V. Peer-review under responsibility of KES International.


Introduction
Mobile devices of smart phones, tablets, and wearable computers contain increasing valuable private information, so that user authentication is essential in order to prevent the leak of the private information from unauthorized or careless use. However, password authentication or biometrics authentication based on fingerprint is explicitly performed only on entry-point. Because the explicit authentication forces the user to input a specific action, for example, to type the password and to scan the fingerprints, it is user-unfriendly to annoy the user. A promising approach of implicit and continuous user authentication without frequent user involvement is behavior-based biometrics authentication. The traditional behavioral modalities are voice and sign. Keystroke authentication on personal computer has also been studied since the 1990s. As smart phone has been rapidly spread, various researches of behavior-based biometrics authentication on smart phone have progressed, for instance, keystroke-based authentication, gait or gesture authentication using accelerometer, touch screen-based biometrics, and authentication using multiple sensors. Although the behavior-based biometrics authentication is a hopeful implicit and continuous user authentication, the behavior-based authentication has general disadvantages that false alarms often occur depending on user's tasks or psychological conditions, and authentication accuracy gets worse as increasing the number of users to be identified compared with the physiologic-based biometrics, for example, fingerprint. The continuous behavior-based user authentication is a challenging issue.
We are also trying to develop a behavior-based user authentication/identification system to continuously monitor user behaviors on smart phone 1,2,3,4 . At first, we focused on operational behaviors on touch screen and reported the performance of touch screen-based biometrics authentication 1 . Next, we examined gait authentication or recognition during walk using 3 axes accelerometers installed in smart phone 2,3,4 . In addition, our latest study 4 has shown the application of the immunity-based diagnosis model proposed by Ishida 5 to multimodal biometrics to integrate the identification results from multiple sensors in smart phone as illustrated in Fig.1. In this figure, each node with its own credibility is corresponding to single sensor-based identification and final meta-identification. The outcome from sensor-based identification to meta-identification indicated by solid arrows is identification result. The credibility of meta-identification is the sum of the outcomes weighted by the credibility of single sensor-based identification. If the credibility of meta-identification becomes over a threshold, then it output identified user or state. The test outcome between single sensor nodes indicated by broken arrows will be decided by time, frequent, and accuracy. Toward the immunity-based gait recognition on smart phone, this paper is a follow-up report on the acceleration-based gait recognition.
In our previous studies 2, 3, 4 , we have developed an iOS application for smart phone to record the logged time, the gravity, the acceleration, the rotation of a phone, and the magnetic field around the 3 axes using the accelerometer, the gyroscope, and the electromagnetic compass. We collected gait data using the application when some subjects walked along a corridor carrying an iPhone in the following 3 holding states: 1) in the pocket, 2) holding the phone to the ear, and 3) looking on the screen. We focused on the user-generated acceleration data except gravity in the world coordinate system from all the recorded data set. And then we extracted 43 features from the user-generated acceleration data. We identified which user or which state using Weka which is a collection of machine learning algorithms for data mining tasks 6 . In the preliminary experiment 3 for only 4 subjects, the authentication results showed that 1.30% false acceptance rate (FAR) at 2.34% false rejection rate (FRR) was obtained when the phone was in the pocket while the FRR was extremely bad as looking on the screen. In the following study 4 , we carried out the additional experiments increasing the number of subjects and adding the data recorded on other day. We confirmed the correctly classified rate when the phone was in the pocket was not yet affected by the number of subjects while the performance in the cases of calling and looking was influenced by the small number of subjects.
Here, there are some questions: Are the 43 features sufficient? Which of the 43 features greatly contribute to the performance? In our dairy life, do we not only walk on a flat corridor but also ascend and descend stairs or slopes? What about Android device because the world market share of Android is much larger than that of iOS?
In this study, we first develop a similar gait record application on Android devices. Using an Android smart phone installed the application, we collect gait data for 15 subjects in 5 walking states which is to add the above 3 kinds of walking on a flat land to going upstairs and downstairs. From the accelerometer data of each axis, we extract 52 features adding 9 features of maximum, minimum, and energy for each axis to the previous 43 features. In the first experiment, we then select the subset of the 52 features with as small number but high accuracy as possible using the attribute selection of Weka. And we examine the accuracy using the subset of selected features for each walking state as comparing with all the 52 features. In the second experiment, we try to enter the 52 features from all the walking states of all the subjects to classification algorithm, that is, automatically identify not only subject but also walking state.

Related Work
Identifying people from gait pattern with accelerometers was first proposed by Ailisto et al. in 2005 7 and followed by Gafurov et al. study 8 . They used high-quality dedicated accelerometers to record the acceleration on walking. Some acceleration-based gait researches 9, 10, 11, 12 using smart phone have recently progressed because widespread smart phones are generally equipped with accelerometers. In most former studies, accelerometers were mounted on hip, arm, or ankle, or smart phones were in the pocket or the pouch in fixed manners. In other words, the acceleration data were collected in the same situation. However, when a user walks, smart phone is not only put in the pocket but also used calling or looking/touching on the screen. In these situations, the direction of the phone should be taken into account. Primo et al. 12 also pointed out the position of the phone and then compared the case of placing a Google Nexus in each of the right and left hand pockets with the case of holding two phones (one in each hand). Ngo et al. 13 addressed the practical sensor-orientation inconsistency and showed that the proposed matching algorithm was robust to any initial sensor-orientation. However, they employed dedicated 4 accelerometers fixed at different orientations and locations on the same plate inside a backpack, that is, they do not use smart phone.

Gait record application on Android device
In our previous studies 2, 3, 4 , we used our developed iOS gait record application to record not only the acceleration but also more data such as the rotation of the phone around the 3 axes using the gyroscope and the magnetometer on iPhone smart phones. Because the world market share of Android devices is much larger than that of iOS devices, for the first step in this paper, we try to develop a similar gait record application on Android devices. In the following, we explain two special notes: coordinate transformation and sampling for the development of the Android application.
In many former studies of acceleration-based gait recognition, smart phones were in the pocket or the pouch in the same fixed manner for each subject. However, smart phone is not only put freely in the pocket but also used calling or touching on the screen. In these situations, the orientation of the phone should be taken into account. The difference of acceleration on walk between a subject and the others can result from the different orientation of the phone in the pocket, for example, the phone facing inward or outward to body, rather than the difference of subjects gait patterns. Since the raw acceleration data of smart phone is usually recorded on the 3 axes of the device, that is, in the local coordinate system (Fig.2 left), the raw data is transformed into the data in the world coordinate system (Fig.2 right) by the orientation of the phone. The transformed 3 axes acceleration data is independent of the direction of the phone in the pocket or held by hand.
In our previous iOS application, the CMDeviceMotion has enum constants for indicating the reference frames when device motion starts. For example, enum constant CMAttitudeReferenceFrameXMagneticNorthZVertical describes a reference frame in which the Z axis is vertical and the X axis points toward magnetic north in the horizontal plane. In other words, we can easily get the transformed 3 axes acceleration data in the world coordinate system on iPhone smart phone. However, the Android does not have such convenient reference frames, we need to program a coordinate transformation by ourselves. According to SensorEvent reference for Android Developers 13 , we can fortunately transform the coordinate by getRotationMatrix and getOrientation methods of SensorManager class using sensor events values of TYPE_ACCELEROMETER and TYPE_MAGNETIC_FIELD. Fig.3

(a) and (b)
show the example of raw acceleration data and the transformed data, respectively. The transformed Z-axis acceleration corresponding to up and down movement of body periodically oscillates with large amplitude.
Another different point between our iOS and Android applications is the data sampling method and rate. Our iOS application can collect constant 100 data per second, which corresponds to 100Hz sampling rate. Our Android application can receive notifications from the SensorManager when sensor values are changed. The third argument of the registerListener method of the SensorManager is the rate at which sensor events are delivered. The rate in our android application is set to SENSOR_DELAY_NORMAL. The application can record average 166 data per second.
The following data set of not only the acceleration but also sensor data using the gyroscope and the magnetometer is logged by our android application. The unit is noted in parentheses.
Logged time (ms) Raw 3 axes acceleration data separated from gravity (m/s 2 ) Transformed 3 axes acceleration data separated from gravity (m/s 2 ) Gravity (m/s 2 ) 3 axes magnetic field by the magnetometer ( T) Raw 3 axes angular velocity by the gyroscope (rad/s) Transformed 3 axes angular velocity by the gyroscope (rad/s) 3 rotations: roll, pitch, and azimuth (rad)

Preprocessing for acceleration data
We focus on the transformed acceleration data except gravity from all the recorded data set. In each experiment, each subject makes one round trip along the corridor and goes up and down the stairs with the landing of the stairs. Fig.4 (a) and (b) show the examples of the transformed 3 axes acceleration data when a subject walks along the corridor and goes up stairs with a smart phone in the pocket. From the left graph, during about 5 seconds from the beginning of the plot, the subject is putting the phone into the pocket. The subject starts to walk after about 8 seconds. This not walking phase should be removed by a preprocessing before next feature extraction process. By the same reason, the preprocessing also deletes the data before and after turnaround. The not walking phase on the landing as going up and down the stairs should be also removed.

Feature extraction and feature selection
We divide the preprocessed acceleration data for each axis into non-overlapping windows. The number of data in each window is 300, the same as our previous studies. However, the time interval of each window is different because acceleration data are collected about 100 times per second by the previous iOS application while about 166 times per second by the Android application. In other words, each window in this study has about 2-second interval. The window size in the research by Kwapisz et al. 9 is 200 for acceleration data which are recorded about 20 times per second, so that the time interval of the window is 10 second. Such much time on gait recognition is undesirable, the succeeding studies such as Nickel et al. 10 and our previous works shorten the recognition time into 3 second. To our surprise, 1-second interval is taken by Gait-ID 11 . Primo et al. 12 employs overlapping windows, each containing 100 data and having an overlap of 50 data with the next window.
From the 300 accelerometer data of each axis in each window, we extract characteristic behavioral features suitable for user recognition. On the one hand, the existing studies 2, 3, 4, 9 including our studies extracted a total of 43 features which were variations of just 6 basic features. However, they did not examine which features were efficient. On the other hand, Primo et al. 12  Time between peaks for each axis (TbpX, TbpY, TbpZ): Time in milliseconds between peaks in the sinusoidal waves associated with most activities. In fact, the first peak is the maximum point among the 300 data, the next peak is the second maximum point, and then at least 3 peaks are found. The times between contiguous peaks are averaged. For example, for the 300 data shown in Fig. 5, the time t1 between the first peak and the third one and the time t2 between the third and the second are averaged. Binned distribution for each axis (Bd1, Bd2, Bd3, Bd4, Bd5, Bd6, Bd7, Bd8, Bd9, Bd10 for X, Y, and Z): We determine the range of data (maximum -minimum) in the 300 data, and then divide this range into 10 equal sized bins as illustrated in Fig. 5. We record the fraction of the 300 data that fall within each of the bins. Maximum for each axis (MaxX, MaxY, MaxZ) Minimum for each axis (MinX, MinY, MinZ) Energy for each axis (EneX, EneY, EneZ) Next we attempt to select the total 52 features into subset with as small size but high accuracy as possible. For the classification of subjects or walking states, this study uses Weka 6 with a collection of machine learning algorithms. Weka also has the function of attribute (feature) selection. The attribute selection of Weak possesses two approaches, namely, filter and wrapper approach with various evaluator and search algorithms. The filter approach calculates an estimated value for each attribute and then selects a subset of some attributes, each of which has the estimated value greater than a threshold. The wrapper approach actually apply a machine learning algorithm to training data with a sampled subset of attributes, repeat the application of the algorithm for the other attribute subsets, and select the best subset which has the highest classification accuracy. In this study, we handle the wrapper approach where the evaluator is set to WrapperSubsetEval, the search to BestFirst, and the machine learning algorithm to Random Forest because this algorithm achieves the highest correctly classified rate as mentioned in subsection 4.2.
In Nickel et al. 10 , Hidden Markov Model or Dynamic Time Warping have been directly applied to time series acceleration data without feature extraction. Deep learning 15 which can automatically acquire features from raw data is also a promising approach. However, these techniques probably need high computational resources for smart phone. We will try to apply the techniques to our data in future.

Gait recognition
To recognition a user and/or a walking state, we enter the features of all the subjects into a classification algorithm which is selected from Weka. Kwapisz et al. 9 have used only 2 classification algorithms: decision trees (J48) and Neural Network (NN) from Weka. In our primary study 2 which employed all of the 56 classification algorithms in Weka, we confirmed that Radial Basis Function (RBF) was the best of all the algorithms. In our following study 3 , Bayesian Network (BN) and Random Forest (RF) from Weka also performed better, but decision trees (J48) was the worst for all the case. Therefore, our latest research 4 employed 4 algorithms, that is, BN, NN, RBF, and RF. However, when RBF is applied to our present data, it takes an extremely long learning time compared with the other algorithms for some of data classification. In this study, we use Support Vector Machine (SVM) which is one of famous machine learning algorithms instead of RBF. In summary, BN, NN, RF, and SVM are applied in the present study.
For each algorithm, we use the default settings, that is, automatically optimization methods and the ten-fold cross validation. In the ten-fold cross validation, the data of all the windows are randomly partitioned into 10 equal size samples. 9 of the 10 samples are used as training data, and the remaining one is retained as the validation data. The cross-validation process is then repeated 10 times, with each of the 10 samples used exactly once as the validation data. The 10 results can then be averaged to produce a single evaluation. Though the ten-fold cross validation is a default test option of Weka, there are some validation with the different separation of training data and testing data.
To evaluate the recognition performance, we use one metrics: correctly classified rate, that is, accuracy. This study deals with not gait authentication but gait recognition. The gait recognition classifies who user using a classifier for all the users, while the gait authentication identifies whether user or not (self or non-self) using a classifier for each user. The final goal of this study is the gait authentication, but there was not enough time to train a classifier for each user. We will show the results of the gait authentication in the near feature. For the gait authentication, two metrics, that is, FRR (False Rejection Rate) and FAR (False Acceptance Rate) are widely used.

Data Collection
We collected walking data for 15 subjects using our developed Android gait record application when each subject carried an Android smart phone installed the application. The smart phone used in all the data collection was TORQUE SKT01 produced by Kyocera. Many studies on gait recognition excluding Kwapisz et al. 9 have collected gait data as walking on a flat land. In our dairy life, we not only walk on flat ground but also ascend and descend stairs or slopes. In this study, we collected gait data in 5 walking states as follows: State 1 (pocket): walking on a flat land as holding in the pocket which contains only the phone. State 2 (calling): walking on a flat land as holding the phone to the ear but not talking (pretending to call). State 3 (looking): walking on a flat land as just looking on the screen but not operating the phone. State 4 (up): going up stairs as holding the phone in the pocket. State 5 (down): going down stairs as holding the phone in the pocket. In state 1, 2, and 3, similar to our latest research using iPhone 5, each subject made one round trip along the corridor with about 50m distance for about one minute. In state 4 and 5, each subject went upstairs from 1st floor to 4th floor and moved downstairs from 4th floor to 1st floor with some landings of the stairs. After each subject finishes walking through all the states, the phone is returned to get all the recorded data into a personal computer through USB connection.

Results
In the first experiments, we apply each classification algorithm to all the 52 features from each of 5 walking states. In other words, we perform the user recognition after we manually separate the walking states. Fig. 6 depicts the correctly classified rate (%) of 15 subjects by 4 algorithms, that is, Bayesian Network (BN), Neural Network (NN), Random Forest (RF), and Support Vector Machine (SVM) for each walking state. From the result, the correctly classified rate of state 1 (pocket) is the highest of all the states while the accuracy of state 5 (down) is the lowest. In our previous studies using iPhone 3,4 , we also observed the deterioration of performance in state 2 (calling) and 3 (looking) compared with state 1 (pocket). The reason is probably that although the vibration of walking leg can directly transmit to the smart phone in the pocket, the hand carrying the phone suppresses the vibration of walking in state 2 and 3. The correctly classified rates of state 4 (up) and 5 (down) have the same tendency as Kwapisz et al. 9 where the accuracies of 3 states (pocket, up, and down) using neural network were 90.9%, 63.3%, and 54.5%, respectively. In terms of the comparison of the classification algorithms, we confirm the accuracies of RF for all the walking states is the best of the 4 algorithms. It is the reason that we select RF in the wrapper approach of feature selection. Here note that all the accuracies of gait recognition are too far from those of commercial products such as fingerprint authentication which have approximately 99.99% and 99.999% accuracy. It may result from the general disadvantages of the behavior-based authentication that false alarms often occur depending on the wide dispersion of user's behavior as mentioned in Section 1. According to the current accuracy of the gait behavior, the gait authentication can be used as the complementation of primary physiologic-based biometrics authentication such as fingerprint or face. If the continuous gait authentication detects an illegal user on a smart phone, then the smart phone can launch the primary authentication with higher accuracy.
Next, we pick out the subset of features with as small number but high accuracy as possible from all the 52 features for each walking state using the attribute selection of Weak, and classify user using the subset of selected features. Table 1 shows correctly classified rate by RF using selected features for each walking state, the number of selected features, and the selected features. From the result, even if the number of features is reduced by more than half of the 52 features, almost the same accuracies for all the states are retained. Interestingly, we observe that the classification rates using the filtered features for state 4 and 5 are improved from 66.67% to 70.67% in state 4, 56.0% to 60.67% in state 5. Although the reason is yet unclear until we will analyse all the features in detail, the 43 features proposed by Kwapisz et al. 9 and the 30 features computed by Primo et al. 12 seem to be redundant for gait recognition on smart phone with restricted computational resource.  In the second experiments, we enter the 52 features from all the walking states of all the subjects to classification algorithm, that is, automatically identify not only subject but also walking state. Fig. 7 is the result of accuracy of classifying 75 classes, 15 subjects, and 5 walking states by 4 algorithms: BN, NN, RF, and SVM. To classify 75 classes means that both 15 users and 5 walking states are simultaneously specified. The accuracies of 75 classes by 4 algorithms are slightly less than the average values over the 5 walking states manually separated in Fig. 6, that is, 63.79, 68.03, 72.91, and 69.33 by 4 algorithms. The classification of 15 subjects is to identify only subject whatever walking state. The variance of gate pattern of the same subject spread, so that the accuracies of 15 subjects decline except RF. The RF may have high generalization capability. As the number of subject increases, the deterioration of the correctly classified rate may be expected together with longer learning time. So we should first specify walking states by machine learning algorithm and then identify subject on the specified state because the classification of 5 states in Fig. 7 has high accuracy. Table 2 is the accuracy of classifying 75 classes, 15 subjects, and 5 walking states by RF using selected features. The result suggests that we do not necessarily use all the features like Table 1.

Conclusions
In this study, we first developed a similar gait record application on Android devices. Using the application, we collected gait data for 15 subjects in 5 walking states adding to going upstairs and downstairs. From the 3-axes accelerometer data, we extracted 52 features adding 9 features of maximum, minimum, and energy for each axis to the 43 features, and then selected the subset of the 52 features with small number and high accuracy. From the result, the random forest has high learning capability in this study. In addition, the accuracies of going upstairs and downstairs were improved by the feature selection. The result suggested we do not necessarily use all the features.
In future, we will achieve the immunity-based gait authentication to integrate the identification results from multiple smart phone sensors. When we will implement our immunity-based gait authentication to the real system, we should separate enrollment phase and authentication phase of gait pattern in order to detect the suspicious behavior of the user as shortly as possible. By the detailed analysis for more users, we also should examine possibilities that some of the smart phone user have a different way to carry the phone every time the use it. The research field of behavior-based continuous biometrics is rapidly progressing, for example, Project Abacus 16 which is the Google project for next-generation authentication for Android. We should clarify the difference and the superiority of the proposed immunity-based gait authentication in comparison with the latest researches.