{'fraction_non_alphanumeric': 0.03036193802408527, 'fraction_numerical': 0.010757648144182138, 'mean_word_length': 5.02202380952381, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Side-Channel Inference Attacks on Mobile Keypads using Smartwatches

Smartwatches enable many novel applications and are fast gaining popularity. However, the presence of a diverse set of on-board sensors provides an additional attack surface to malicious software and services on these devices. In this paper, we investigate the feasibility of key press inference attacks on handheld numeric touchpads by using smartwatch motion sensors as a side-channel. We consider different typing scenarios, and propose multiple attack approaches to exploit the characteristics of the observed wrist movements for inferring individual key presses. Experimental evaluation using commercial off-the-shelf smartwatches and smartphones show that key press inference using smartwatch motion sensors is not only fairly accurate, but also comparable with similar attacks using smartphone motion sensors. Additionally, hand movements captured by a combination of both smartwatch and smartphone motion sensors yields better inference accuracy than either device considered individually.

models to learn the uniqueness in wrist motion caused during individual keystrokes depending on known relative location of the key on the screen, and then using the trained classifiers to infer unlabeled (or test) keystrokes. During preliminary experiments, we observed that keystroke induced motion data captured by smartwatch and smartphone sensors differ significantly. Consequently, we thoroughly assess how significantly smartwatch motion sensors elevate the threat of keystroke inference, compared to similar attacks using only smartphone motion data [2]- [4]. We also evaluate the case where the adversary may have gained access to motion sensors on both the smartwatch and smartphone, to see how our attack will perform when motion data from both devices are combined. For the typing scenario where key press events cannot be identified based on the uniqueness of motion sensor activity surge (corresponding to a key press), we present a novel scheme to infer a sequence of key presses based on the transitional movement between individual key presses. We evaluate the proposed attacks in both controlled and realistic typing scenarios. We also briefly discuss possible protection measures against such inference attacks that employ motion sensors as side-channels.
The remainder of this article is structured as follows: First we discuss similar side-channel attacks in the literature in Section II. Then we give an overview of the threat and adversary model in Section III. In Section IV we describe the classification-based keystroke inference framework, followed by its evaluation in Section V. In Section VI we describe the relative transitions-based keystroke inference framework, followed by its evaluation in Section VII. Finally, we discuss implications, limitations and future research directions.

II. RELATED WORK
Inference of private information from various forms of side channels has been an active area of research in the community. Electromagnetic signals emanating from devices have been used to infer private data stored on Smart Cards [7], data transmitted on RS-232 cables [8] and content being played on CRT and LED monitors [9], [10]. Recently, Hayashi et al. [11] showed that it is also possible to remotely reconstruct and eavesdrop on flat panel displays on tablets via measurement of electromagnetic emanations. Similarly, optical emanations from monitors [12] or from eyes [13] have also be used to infer information such as content being displayed or watched. Acoustic or sound signals emanating from devices such as printers have also been used to infer the content being printed on certain models of dot-matrix printers [14].
Availability of several high-precision sensors on modern mobile cyber-physical systems such as smartphones have given rise to additional side-channels [15], thus increasing the risk of private information leakage through such side-channels. Past research efforts have shown how malicious applications can misuse their access rights to these sensors in order to execute various imperceptible side-channel attacks by stealthily capturing information from the physical environment. For example, smartphone cameras can be accessed in an unauthorized fashion to infer sensitive information from user keystrokes [16], [17]. Unauthorized microphone recordings of ambient sound [18] provide a rich source of information that can be used to infer sensitive information about a person's daily life. Activities and locations can be inferred based on characteristic ambient sound patterns, e.g. walking on the streets, or eating in a restaurant [19]. Unauthorized access to GPS sensors can pose obvious risks related to loss of location privacy, such as revealing home/work locations, stalking and location-targeted advertisements [20]. Advanced learning-based techniques were also proposed for predicting users' future movements from previous tracking records of their location activities [21]- [24]. Security and privacy risks associated with front-end sensors, such as, microphones, cameras and GPS, have been comprehensively studied because of the hazards apparent to users.
However, security risks due to sensors obscured from users (e.g., accelerometer, gyroscope, and magnetometer) have largely been overlooked until recently. After modern mobile operating systems introduced user-managed access control on front-end sensors, adversaries shifted attention to sensors which cannot be disengaged by users. It has been shown that malicious applications can track users' movements [25], [26] and activities [27]- [29] by using only smartphone accelerometer readings. It has also been shown that, with the help of standard signal processing and machine learning techniques, it is possible to recognize speakers and parse speech by using gyroscopes on modern mobile devices to measure acoustic signals [30].
Keystroke inference attacks using side-channel information have received significant attention due to their potentially dangerous consequences. Electromagnetic emanations from external keyboards (both wired and wireless) have been used in the past to infer user keystrokes [31]. However, the requirement of extensive setup and expensive monitoring hardware prevents less sophisticated adversaries from carrying out such attacks. Keystroke inference attacks using audio or acoustic signals [32]- [36], on the other hand, have also received significant attention in the literature. Such attacks have proven to be very successful and can be carried out using modest off-the-shelf hardware (e.g., any microphone equipped device). Due to the ubiquitous nature of modern smartphones that are equipped with high-precision microphones, such attacks are much more practical than previously argued.
However, as touchscreen key press events emanate very weak acoustic signals, inference attacks using them is very difficult. Additionally, requirement of undisturbed eavesdropping is another major obstacle in using electromagnetic and acoustic emanations for such attacks. As a workaround to the above limitations, smartphone motion sensors have been used to recover keystroke events on the device. For instance, TouchLogger [2] and TapPrints [37] utilize change in orientation angles of the smartphone, as captured by its accelerometer, to extract appropriate features for keystroke inference. Similarly, ACCessory [3] also attempts to infer keystrokes using the smartphone accelerometer data by employing multiple supervised learning techniques. Alternatively, TapLogger [4] automates the training and logging phases and attempts to work stealthily on the smartphone. Smartphone motion sensors have also been used to detect keystroke events on other external devices/keyboards in proximity [38]. Despite these research efforts, side-channel privacy threats (especially, keystroke inference attacks) posed by wearable devices such as smartwatches have received far less attention. Recently, Wang et al. [39], Liu et al. [40], Wang et al. [41] and Maiti et al. [42] have explored new attacks to infer user keystrokes or key presses on external physical keyboards/keypads by using smartwatch motion sensors. In contrast to these research efforts, our work focuses on keystroke inference on hand-held mobile keypads by employing smartwatch motion sensors.
In our preliminary work in this direction [43], we designed and evaluated a basic supervised learning-based classification framework to infer keystrokes on a mobile/smartphone keypad. The evaluation was limited to the typing scenarios where key press events were identifiable based on surge in wrist motion activity. In this paper, we present new and improved attack frameworks, which include attack on an additional typing scenario. We made significant changes to the classificationbased attack framework, such as enriching the feature vectors and using a robust ensemble classification scheme. Our new relative transition-based attack framework is designed to infer sequences of key presses, where individual key press events cannot be detected by using the unique surge in the wrist motion activity (captured on the smartwatch motion sensor). We conduct extensive empirical evaluation for both frameworks with the help of multi-sensor typing data contributed by human subject participants using two different smartwatch hardware. We also evaluate the efficacy of our attack framework in a non-controlled natural typing setting, Further, we also analyze our attack framework for gain in inference accuracy (if any) when typing-related motion data from both the smartwatch and smartphone is combined.

III. ATTACK DESCRIPTION
In this research effort, we focus on three of the most popular typing (or tapping) scenarios in mobile hand-helds or smartphones [44]. We consider a user typing on a smartphone's numeric touchscreen keypad while wearing a smartwatch on one of his/her hand. In the first case, smartwatch and smartphone are on the same hand and the user types with the hand not holding the smartphone (see Figure 1(a)), also referred by us as SH-NHHT scenario. In the second case, smartwatch and smartphone are again on the same hand and the user types with a finger (generally, thumb) of the smartphone holding hand (see Figure 1(b)), also referred by us as SH-HHT scenario. In the above two scenarios, the action of tapping a key on the smartphone keypad results in a unique motion of the wrist (on the smartphone holding hand) for each keystroke, which can be captured by the motion sensors (e.g., accelerometer and gyroscope) of the smartwatch and used to identify the tapped keystroke. In the third case, smartwatch and smartphone are on different hands and the user types with the hand not holding the smartphone (see Figure 1(c)), also referred by us as DH-NHHT scenario. Unlike the previous two scenarios, each key tap does not produce a unique motion signature on the wrist of the typing hand (where the smartwatch is situated), and thus it cannot be used to infer the exact keystroke in a fashion similar to the previous two cases. However, assuming that the relative position of keys on the keypad as per the standard layout shown in Figure 1(d) is known and remains static, we can use the relative transitional movement between taps to infer a (sub)sequence of the tapped keys. In addition to the above three, other typing scenarios are also possible, for example, typing with both hands and holding (the phone) and typing with non watch wearing hand. However, in order to limit the scope of our study and to clearly demonstrate the keystroke inference threat posed by smartwatches, we only consider the above three typing scenarios (i.e., SH-NHHT, SH-HHT and DH-NHHT) in this work, which also happen to be very widely adopted by smartphone users. We further justify our focus on these three typing scenarios by more precisely determining the percentage of users that employ these typing methods, and as a result, are impacted by the proposed inference attacks. Based on the data available from a study concerning users' smartphone holding and usage behaviors [44], 32.83% of all users hold and use the phone as shown in Figure 1(b), 28.44% of users hold and use the phone as shown in Figure 1(a), while only 2.11% of users hold and use the phone as shown in Figure 1(c). In this work, we also investigate two variations of scenarios in Figures 1(a) and 1(b), where the phone is held in the other hand as shown in Figures 13(a) and 13(b) respectively. Based on [44], 16.17% of all users hold and use the phone as shown in Figure 13(b), while only 7.56% of users hold and use the phone as shown in Figure  13(a). It should be noted that [44] does not provide any data on which hand (left or right) the smartwatch is traditionally worn, so the above only represents percentages of users based on the hand in which the phone is held and the hand used to type or tap. Now even if the phone holding scenario of Figure 1(c) is ignored (due to its low usage percentage), our framework can potentially impact at least 44.61% of users who wear the smartwatch on the left hand, i.e., those who type as shown in Figures 1(a) and 13(b). Similarly, at least 40.39% of users wearing the smartwatch on the right hand are impacted by our keystroke inference framework, i.e., those who type as shown in Figures 1(b) and 13(a)). Moreover, we also show in Section V-I that the performance of our framework does not vary significantly between a particular typing scenario and its variation (say, between Figure 1(b) and 13(b)). This shows that a significant percentage of users have the potential of being impacted by the proposed keystroke inference framework.
Threat Model: We assume an adversary whose goal is to infer a target's keystrokes on a generic smartphone numeric keypad (as shown in Figure 1(d)), based on the wrist movements perceptible by the target's smartwatch motion sensors. The adversary may gain access to the target's smartwatch by installing a malicious application on it which records the activity of the on-board accelerometer and gyroscope sensors. This step can be achieved by exploiting known software vulnerabilities or by tricking the victim into installing malicious code, e.g., using a trojan software. Based on the fact that most common smartwatch operating systems (e.g., Google's Android Wear, Apple's watchOS, etc.) do not implement access control and/or user notification for motion sensor usage, the malicious application may have unrestricted and undetected access to the on-board accelerometer and gyroscope. As a result, the compromised smartwatch can act as an eavesdropping device which the targets' themselves may place on their wrist, and unsuspectingly have it on their wrist while typing on a smartphone. The malicious application may also maintain a covert communication channel [45] with the adversary, and periodically upload the collected wrist motion data on some adversarial server by means of this channel. The use of a covert channel by the trojan is optional. However, if a covert channel is not used, there is a possibility of this information transfer being easily detected and the trojan being inactivated. We assume that the adversary also has sufficient off-site storage and computational resources to download the raw sensor data, extract significant keystroke events, and execute standard machine learning algorithms in order to classify the keystrokes. For comparison with attacks based on smartphone data, we assume similar adversarial capabilities and actions for the smartphone.
IV. CLASSIFICATION-BASED ATTACK FRAMEWORK The linear accelerometer motion sensor found on smartwatches measures the three dimensional acceleration expe-rienced by the device, excluding the omnipresent force of gravity. During preliminary experimentation with SH-NHHT and SH-HHT scenarios, we observed that key press events can be accurately detected using the surge in linear acceleration during a key press. Based on the observation that taps on different locations of the smartphone screen produces characteristically unique motions on the wrist, our attack framework leverages on supervised machine learning to directly classify the detected key presses. The attack framework (Figure 2) consists of a learning phase followed by an attack phase. Both phases go through similar steps of data collection followed by feature extraction, with the learning phase culminating in training (the classifiers), while the attack phase in classification (using the trained classifiers from the training phase). Next, we describe in detail each component of the proposed framework.
Data Collection and Pre-Processing: We developed an application for Android Wear that continuously samples linear accelerometer measurements on the smartwatch, and runs in the background during experiments. Details of the data collection experiments and technical specifications of the hardware used for the experiments are outlined later in Section V-A. The smartwatch data collection application communicates the linear accelerometer measurements to the host smartphone (with which the watch is paired) using Andorid Wear's Wearable Data Layer API. On the smartphone, another Android application displays a keypad to the users to type sequences of numbers. In the background, the smartphone application chronologically logs all accelerometer measurements received from the smartwatch and any key press events registered on the displayed keypad. It logs two data-streams: (i) timestamped readings of the smartwatch's linear accelerometer (A); and (ii) timestamped key press labels (L). Both A and L are stored locally on the smartphone (which is also paired with the smartwatch) during the data collection process and retrieved later for offline evaluation. Note that in the attack phase of our experiments we use L only to verify the classification accuracy.
Due to the absence of labeled data, the attack phase requires an additional key press event detection mechanism. Figure 3 shows a portion of a raw linear accelerometer data-stream, spanning four key press events in the SH-NHHT scenario. As evident from the graph, each tap agitates the linear accelerometer sensor readings on the three axis, with more prominence along the Y-axis and Z-axis than X-axis. SH-HHT data also exhibits similar traits. We apply this observation to model an algorithm for automating the process of key press event detection. Algorithm 1 sequentially examines the "energy" of each sample i in A as the sum of acceleration on the three axis (Equation 1). The energy value calculated in Algorithm 1 is then used in Algorithm 2 to determine keystroke events.
Algorithm 2 establishes the threshold value as the average peak energy values observed in the time-stamped training set. In the attack phase, once the energy level surpasses the empirically learned threshold (from Algorithm 2), a key press event is recognized and a "keystroke-record" is saved. Each keystrokerecord is intended to represent the wrist motion pattern of a key press event, and consists of few linear accelerometer readings immediately before and after the key press event is recognized. We empirically observed that the movement due to a key press subsides after approximately 350 msecs. Thus, a keystrokerecord of eighteen samples (at 50 Hz sampling frequency) sufficiently captures all motion features related to a keystroke. Taking into consideration some of the milder initial motion, we form each keystroke-record as follows: the sample to surpass the energy threshold is preceded by three samples and followed by fourteen samples, chronologically from A. After a key press event is recognized and the corresponding keystroke-record is saved, the key press detection algorithm resumes its search for next key press. As multiple samples during a key press may cross the energy threshold, ignoring the fourteen samples following the last keystroke-record ensures that the same key press is not recorded multiple times.
Feature Extraction: Our proposed attack infers the numeric key that was pressed based on features of the underlying physical event of wrist motion caused during typing (or tapping) on a smartphone. The features of a keystroke-record must be able to capture as many attributes as possible about the underlying three-dimensional movement caused by a key press. A properly designed feature vector should be similar with other feature vectors of the same key, simultaneously being distinguishable between feature vectors of other keys. We observed that, based on the location of a key on the screen, the degree of movement caused by a tap varies on each of the X, Y , and Z axis of the linear accelerometer ( Figure 4). Interestingly, this movement remains fairly consistent for the same key.
In our preliminarily work [43], we used only 54 basic time domain features of the accelerometer data to identify the uniqueness of each key (and the corresponding key press event), and found those features to be reasonably useful for keystroke inference. In this work, we expand that to a more comprehensive set of features, employing both time and frequency domain features, with a total of 155 different features in our feature vector for each key. We continue to use time domain features of individual axis such as minimum and maximum magnitudes, squared sum of magnitude data below 33 percent and above 67 percent of maximum magnitude

Algorithm 1 Key Press Detection Algorithm
Insert T hisKeyP ress into KeyP resses i = i + 15 end if end for end function  (to measure the duration of major and minor movements), position of maximum and minimum magnitude samples, mean, median, variance, standard deviation, skewness (measure of any asymmetry) and kurtosis (to measure any peakedness), raw accelerometer readings, and their first order numerical derivatives (to measure the rate of change of energy). We also use inter-axis time domain features to capture the correlation between movement on the three axis, such as minimum and maximum magnitudes across all three axis, Frobenius norm, Infinity norm, 1-norm, Euclidean norm, and axis with highest and lowest magnitude for each time sample. Along with the time domain features, we also capture frequency domain features by computing the Fast Fourier Transform (FFT) of individual axis readings of the keystroke-record. The frequency domain features are necessary to identify the different rebounding (or oscillatory) motion of the wrist. Note that in the learning phase, the feature vectors are also labeled, using the timestamped key press labels (L) recorded by the data collection application.
Training and Classification: We model the keystroke inference problem as a multi-class classification problem. Labeled feature vectors are used to train classifiers in the learning phase, whereas unlabeled feature vectors are mapped to the "closest" matching class by the already trained classifiers during the attack or test phase. To train our classifiers, we initially tested five different classification algorithms that are appropriate given the properties of our features: (i) simple linear regression (SLR), (ii) random forest (RF), (iii) k-nearest neighbors (k-NN), (iv) support vector machine (SVM), and bagged decision trees (BDT). However, each of these classification techniques has its own advantages and shortcomings, leading us to adopt an ensemble classification approach. Ensemble approaches have proven to be more accurate and robust than any single classification algorithm [46], [47]. We consider an extremely broad set of classification algorithms in our ensemble method, as a result of which, the errors made by constituting classifiers are highly uncorrelated. We include parametric algorithms (SLR, SVM), as well as non-parametric algorithms (k-NN, RF, BDT). Our ensemble method involves both linear (SLR, SVM) and non-linear (k-NN, RF, BDT) techniques. Moreover, RF and BDT are strong ensemble classifiers in themselves which makes our classification framework even more robust. Such a diverse set of classification algorithms increases the likelihood of improvements in classification accuracy over a single algorithm.
During the training phase, multi-class classifiers of each constituting classification algorithm are trained separately using the labeled training data. After all the classifiers have been trained using the labeled data, feature vectors of unlabeled keystroke-records are classified using these trained classifiers (in the attack phase) using an ensemble strategy. Finally, a majority wins ensemble strategy is used to determine the final classification result ( Figure 5).

V. EVALUATION OF CLASSIFICATION-BASED ATTACKS
In this section, we present the findings from our evaluation of the classification-based attack framework.

A. Experimental Setup
Our initial data collection experiments involve 12 participants, aged between 19-32 years. The identity of these participants are anonymized as P 1 , P 2 , . . . , P 12 . We employ a Samsung Gear Live smartwatch equipped with an InvenSense MP92M 9-axis Gyro + Accelerometer + Compass sensor. Smartwatch was worn on left hand for SH-NHHT (Figure 1(a)) and on right hand for SH-HHT (Figure 1(b)). Participants use the virtual numeric keypad of a Motorola XT1028 smartphone (Figure 1(d)) for typing. Linear accelerometer of the smartwatch was sampled at 50 Hz. We used the Weka 3.7.12 [48] libraries for both training and testing the classifiers. MATLAB R2014a was used to compute most of the time and frequency domain features.
We also evaluate the performance of our keystroke inference framework in several additional settings: (i) a more natural Fig. 4: The intuition behind our classification-based attack is that taps on different locations of the smartphone screen produces characteristically unique motions on the wrist. Accordingly, taps on each number on the keypad should be identifiable based on the uniqueness in the resultant wrist motion. or uncontrolled typing scenario (Section V-F), (ii) using a different smartwatch hardware (Section V-G), (iii) employing an additional type of motion sensor, i.e., gyroscope (Section V-D), and (iv) typing on a QWERTY or alphabetic keypad (Section V-H). For these last four experimental settings, we collected additional data from different sets of participants, and in certain cases using a different smartwatch and/or smartphone hardware. The participant and data collection procedure details for these additional experimental settings appear in their respective sections.

B. Constructing and Testing the Classifiers
We construct our classifiers based on different training datasets of labeled keystroke-records generated by the participants. An audio stream of uniformly distributed random numbers between 0 to 9 guided the participants in typing. To prevent fatigue, participants were given optional breaks, during which they were allowed to set down the phone on the table and some participants even went out of the room. However, they returned to approximately the same holding position after the break. We comparatively evaluate the classification accuracy (the percentage of correct prediction divided by the total number of predictions) of our classifiers for the following three training/testing scenarios: • One vs. One: In this case, we measure the percentage of successful inferences on an individual participant, with classifiers trained from the training set of the same participant. Target set size is 100 (10 per key) and training set size is 200 (20 per key). One vs. One is not only a best case scenario, but also represents how the attack will perform if the adversary is able to collect target-specific training data. • One vs. Rest: In this case, we measure the percentage of successful inferences on an individual participant, with classifiers trained from the training set of the rest of the participants (not including the target participant). Target set size is 100 (10 per key) and training set size is 2200 (220 per key). One vs. Rest is a typical scenario where the adversary has a target, but is unable to obtain labeled training data from the target. • All vs. All: In this case, we measure the percentage of successful inferences on all participants, with classifiers trained from training set of all participants. Target set size is 1200 (120 per key) and training set size is 2400 (240 per key). All vs. All is helpful in understanding how our attack framework will perform if the adversary constructs a heterogeneous training data set to infer keystrokes from multiple non-specific targets.
Classification results for One vs. One are shown in Figure 6. One vs. One classification accuracy ranged fairly high between 94% and 77% for SH-NHHT, and between 93% and 75% for SH-HHT, with an average of 84.58% and 83.5%, respectively. However, classification accuracy drops noticeably in One vs. Rest. As shown in Figure 6, One vs. Rest classification accuracy ranged between 82% and 63% for SH-NHHT, and between 78% and 65% for SH-HHT, with an average of 70.08% and 71.16%, respectively. The achieved All vs. All classification accuracy was 88.16% and 85.83% for SH-NHHT and SH-HHT, respectively. Overall, these results validate our claim that smartwatch motion sensors are a feasible sidechannel for inferring keystrokes on mobile touchpads.
We also recorded the mean computation time in each training/testing scenario (Table I). All training and testing operations were executed on a laptop featuring a 2.7 GHz dual-core Intel i5 processor and 8 GB of working memory. Due to the use of ensemble classification technique, the total computation time is the sum of time taken by the five constituting classification algorithms. The average total computation time in One vs. One scenario was less then 12 minutes, about 21 minutes in One vs. Rest, and about 34 minutes in All vs.  All scenario. Moreover, the total computation time can be further reduced if the different classification algorithms are executed in parallel (with suitable hardware support). These results show that the above keystroke inference attacks can be carried out by an attacker using reasonable computation resources in a fairly short amount of time.

C. Reduced Sampling Frequency:
We also briefly investigate how our attack will perform at reduced sampling rate (25 Hz and 10 Hz), a more realistic scenario for low-cost wearables, equipped with less precise sensors. We repeat the experiments outlined in Sections V-A and V-B with smartwatch data sampled at a reduced frequency, and Figure 7 shows the drop in accuracy of our attacks for both the SH-NHHT and SH-HHT scenarios. For example, One vs. One classification accuracy in SH-NHHT dropped from 84.58% to 72% when sampling frequency was reduced to 25 Hz and to 23% when sampling frequency was reduced to 10 Hz. Similarly, the other scenarios also observed drop in classification accuracy with reduction in sampling frequency, but percentage of successful classification can be considered fairly substantial even at a sampling frequency of 25 Hz.

D. Comparison with Smartphone-Based Attacks
Previous research efforts on keystroke inference attacks by using smartphone sensor data [4], [37] (or data collected from the target's smartphone sensors) also used similar learningbased multi-class classification frameworks. This motivated us to apply our attack framework on smartphone data and compare the results with those carried out using smartwatch data. This enables us to understand how much more or less vulnerable a motion sensor-based side-channel originating on a smartwatch makes us, as compared to known motion-based side-channels on the target users' smartphone. We conduct similar experiments (as in Sections V-A and V-B) by using smartphone linear accelerometer data sampled at 50 Hz, rather than using the smartwatch data. Figures 8 and 9 shows the accuracy of our attack for SH-NHHT and SH-HHT scenarios. On comparing with previous results from Section V-B, it can be observed that the keystroke inference attacks in SH-NHHT resulted in slightly better average classification accuracy when smartwatch motion data was used. Whereas in SH-HHT, classification accuracy results are mixed, and nearly equal, for both the smartwatch and smartphone data. In summary, these results demonstrate that the threat of motion-based keystroke inference may be increased in certain typing scenarios due to smartwatches. An interesting pattern of classification accuracy can be observed (see Fig. 10) for inference using only smartphone data in SH-HHT. We observe that the classification accuracy for certain keys (based on their location) are distinctly higher than others. Interestingly, this occurrence is not recognizable for the smartwatch dataset. This may be due to the fact that keys farther away from the thumb impels the user to bend the phone towards the thumb. As a result, significantly greater movement of the phone occurs, compared to keys that are near the thumb.
In order to conduct an exhaustive comparison between the inference threat posed by different motion sensors present on a smartwatch and smartphone, we carry out additional experiments using the gyroscope data which is another widely studied side-channel for keystroke inference [2], [37], [49]. Due to the absence of a gyroscope sensor on the Motorola XT1028, we used another smartphone for this experiment, namely a Motorola XT1096 (paired with the Samsung Gear Live). The same experiment as above was repeated by 12 new participants, each typing 100 randomly dictated numbers. For this experiment, we recorded keystroke related motion data, comprising of both linear accelerometer and gyroscope measurements, from both the smartphone and the smartwatch. We derived 59 time and frequency domain features from the three-dimensional gyroscope data of both devices, such as minimum and maximum values, the mean value, variance, skewness, kurtosis, vertex angles, number of spikes, peak intervals, attenuation rate, etc. These features were selected from the literature on activity detection [50], [51] and keystroke inference [49]. Figure 11 shows the One vs. Rest classification accuracy results when solely the gyroscope features are used compared to when they are used in combination with features derived from the linear accelerometer measurements. The mean classification accuracy is marginally lower when using    only the smartwatch gyroscope, compared to the smartphone gyroscope (SH-NHHT: 59.91% vs. 61.25%, SH-HHT: 59.66% vs. 64.75%). However, we can observe that after combining multiple motion sensors (linear accelerometer and gyroscope) on a device, the keystroke inference threat on the smartwatch is greater than the one on the smartphone (mean classification accuracy, SH-NHHT: 69.91% vs. 61.41% and SH-HHT: 71% vs. 66.08%).

E. Combining Smartwatch and Smartphone Data
After comparing the accuracy of keystroke inference attacks using individually both the smartwatch and smartphone motion data, we were intrigued to study the impact of combining or fusing motion sensor data from both devices in order to further reduce the number of classification errors. As most modern smartwatch operating systems and applications require the watch to be paired with a smartphone, such an attack is quite realistic. The feature vectors of same keystroke-records from both the devices were merged to obtain new feature vectors containing 310 features. We rebuild the classifiers with the larger feature vectors, and re-ran the previous experiments (as outlined in Sections V-A and V-B). Results of these experiments (outlined in Table II) show that indeed accuracy improved when the features from both smartwatch and smartphone were combined. For example, the One vs. One  classification accuracy in SH-NHHT was 90.66%, compared to 83.5% and 84.0% when individual smartwatch or smartphone data were used, respectively. Similar improvements can be observed in other scenarios as well. However, the improvement was relatively marginal, which can be attributed to the convergence in the learning process. Therefore, combining or fusing data from both smartwatch and smartphone may be more beneficial when the adversary has fewer training data.

F. A More Realistic Setting: Natural or Non-Controlled Typing
In all of the experiments so far, the participants were being directed (to tap) by an audio stream. Because participants have to hear the audio and then act on it, a minor delay or disturbance may be introduced in each key press. Moreover, such a kind of typing or tapping does not invoke (and capture) users' natural typing behavior and speed. To evaluate a more natural typing behavior, we conduct another experiment where a new set of 10 participants were instructed to type their phone number followed by their residential zip code (a total of 15 numbers). These two pieces of information can be readily recollected by participants, thus eliminating any delay and/or disturbance while typing. This also enables us to capture more realistic typing or tapping data from users. However, for prediction we continue to use the classifiers trained earlier in the guided experiments (Section V-B). The new data was processed by the same attack framework to extract keystrokerecords and build feature vectors. We obtained a mean classi- Fig. 12: An example where rebounding motion of a key press overlapped with the next key press. fication accuracy of 52% and 61% for SH-NHHT and SH-HHT, respectively. It was observed that the primary cause of drop in classification accuracy resulted from faster typing, where the rebounding motion of few key presses overlapped with their next key press (see Figure 12). Such instances were observed more often when two consecutive key presses were for number adjacent to each other on the keypad. Although the classification accuracy of naturally typed numbers is not as high as in the guided experiments, it is high enough to be a significant threat.

G. Cross Device Performance
In order to further evaluate how the proposed attack framework performs across different commercial wrist wearable or smartwatch hardware, we test our trained classifiers (from Section V-B) on keystroke motion data obtained from a smartwatch of a different make and model. This simulates a situation where an adversary trains classification models using one type of smartwatch hardware and then employs those models to infer the keystrokes of a target user who is using a completely different (possibly, unknown) smartwatch. Such a situation is much more realistic. For this set of experiments, we used a LG Urbane W150 smartwatch that has a InvenSense M651 accelerometer and gyroscope sensor and collected keystroke motion data from 12 completely new participants. Motion data corresponding to 100 keystrokes were collected from each of the new participants, and tested using the classifiers trained earlier in Section V-B. The new data was collected at the same sampling frequency of 50 Hz. Results ( Figure 6) show that while mean classification accuracy dropped slightly on the Urbane W150 (SH-NHHT: 70.08% vs. 67.41%, SH-HHT: 71.16% vs. 70.83%), the variance is significantly lower in case of the Gear Live (SH-NHHT: 26.62 vs. 56.26, SH-HHT: 17.24 vs. 77.0). Although such a trend is intuitive, it nevertheless shows that keystroke inference using the propose framework is still feasible with reasonable accuracy even in such a realistic setting.

H. Extending to QWERTY Keypads
Up until this point, our primary focus has been keystroke inference attacks on numeric mobile keypads. We now briefly investigate how our proposed attack framework performs against alphanumeric mobile keypads with the standard QW-ERTY layout. Intuitively, as the keys on a standard smartphone QWERTY keypad are relatively smaller and placed closer to each other (compared to keys on the numeric keypad), keystroke prediction may suffer from high confusion with neighboring keys [37]. We collected 1248 alphabet keystrokes from a completely new set of 12 participants using the LG Urbane W150 smartwatch, with equal distribution of alphabets (48 each). We then re-ran the training and attack modules in the One vs. Rest setting, with 75% data used for training and 25% data used for testing. Table III summarizes the classification accuracy of the 26 alphabets, along with two most confused keys predicted for each alphabet. As anticipated, the classification accuracy is significantly lower on the QWERTY keypad (compared to the numeric keypad), with an average accuracy of 30.44%. While the low classification accuracy of individual keys is prohibitive in carrying out effective inference attacks, it is important to note that the most confused keys are usually neighboring to the actual key. It is possible that we may be able to further improve the accuracy of these inference attacks by analyzing keyboard characteristics and/or performing a dictionary-based search [38], [42].

I. Variations of the SH-NHHT and SH-HHT Attack Scenarios
In addition to the SH-NHHT and SH-HHT scenarios presented in Figures 1(a) and 1(b), there is an additional variation for each of these scenarios, as shown in Figures 13(a) and 13(b). For SH-NHHT, the scenario 1(a) assumes that the smartphone and smatwatch is on the left hand (and users type with the right hand). A variation of this SH-NHHT scenario is having the smartphone and smatwatch on the right hand and typing with the left hand (13(a)). A similar variation (13(b)) can also be envisioned for the SH-HHT scenario 1(b). We would like to analyze whether the performance of our proposed keystroke inference framework differs significantly for these variations. The same experiment as in Section V-B was repeated by 12 new participants, each typing 100 randomly dictated numbers per variation. A two-tailed t-test on the One vs. Rest classification accuracies for the SH-NHHT variations in Figures 1(a)  Both results are not significant at p<0.05, implying that our attack framework is not dependent on these variations. Therefore, an adversary can still use the same framework to carry out the inference attacks for these variations by simply retraining the classifiers.

VI. RELATIVE TRANSITIONS-BASED ATTACK FRAMEWORK
As discussed earlier, unlike the SH-NHHT and SH-HHT scenarios, key press events in the DH-HHT scenario ( Figure  1(c)) cannot be uniquely and accurately detected on the smartwatch. To overcome this problem in the DH-HHT scenario, we leverage on the observation that transitional movement between each pair of keys produces characteristically unique motions on the wrist, which can be accurately captured by the smartwatch. Accordingly, for the DH-HHT scenario, the keystroke inference framework (Figure 14) leverages on supervised machine learning to first classify transitional movements between consecutive key presses. Then, assuming a reasonable distribution of numbers typed, when multiple transitional directions in between a target sequence of key presses are traced on the key pad, we obtain a unique or highly reduced possibilities for the target sequence.
Data Collection and Pre-Processing: The same data collection application that was used for the SH-NHHT and SH-HHT scenarios is also used for the DH-NHHT scenario. Although the data collection process is exactly the same, the pre-processing operations are entirely different for DH-NHHT. Instead of detecting key press events, our goal here is to detect the type of wrist movement transition between every two consecutive key presses. As a result, we use the labeled stream of data to create labeled "transition-records" ( Figure  15) and use them as the training set. To create the training set, all linear accelerometer samples between two consecutive key press events are used as the transition-record.
Transition Classification: We classify transitions based on cardinal directions. The logic behind such a classification is that transitions in the same direction results in similar wrist movement. For example, wrist movement between numbers 4 and 1 would be similar to wrist movement between 6 and 3 (North), wrist movement between numbers 4 and 7 would be similar to wrist movement between 6 and 9 (South), and so on. One classifier is trained for each possible transitional direction, as listed in Table IV: North (N), South (S), East (E), West (W), Northeast (NE), Northwest (NW), Southeast (SE), Southwest (SW) and Repeat (O). To achieve higher inference ability through tracing (explained later), the transition classifications must also be evenly populated. The number of possible transitions in each of the above nine categories follows a fairly even distribution, varying between 9 and 14.
As an adversary will not have access to labels L, in the attack phase we use a variable-length moving window to check and determine the occurrences of transitions. The moving window is used to traverse (in steps of one sample) the linear accelerometer data A in chronological order, and classify each window of linear accelerometer samples into one of the nine directions. The length of the window was varied from 10 samples (200 msec at 50 Hz) to 100 samples (2 sec at 50 Hz), to capture the variable length intervals possible between key presses. When ten or more consecutive windows were classified to be in the same direction, the classification result was recorded and the centroid was used as the key press time to form transition-records.
Feature Extraction: Contrary to the previous direct Fig. 15: Time series of key press events in DH-NHHT, and their corresponding linear accelerometer readings. In DH-NHHT scenario, the wrist (along with the smartwatch) continues to move in between key press events. As a result, key press events cannot be identified or characterized based on spikes in energy level.
classification-based attacks, where each key press event was denoted in a fixed time period, transition periods between two key presses can vary widely depending on typing habit, keypad size, key pairs, etc. As a result, many of the time domain features used in SH-NHHT and SH-HHT scenarios cannot be applied for DH-NHHT. Thus, we rely mainly on frequency domain features, such as FFT of individual axis readings of the transition-record, their mean, correlation, spectral rolloff, spectral centroid, spectral flux and power spectral density estimates, to learn and classify transitions. Tracing and Recovery: To infer a target sequence of key presses, the proposed framework tries to "trace" the transitions between key presses on the numeric keypad. Tracing eliminates all non-fitting key-pairs (the pair of keys that may have been pressed before and after a transition) for each transition of the target sequence, where the fitness of a keypair is determined by the preceding and following transitions. In case tracing results in a uniquely identified key-pair for a transition, the keys pressed before and after that transition can be directly inferred. In other cases where tracing results in multiple possible key-pairs for a transition, the keys pressed before and after that transition can either be inferred by multiple trials or from the other adjoining key-pairs (only if the adjoining key-pairs are uniquely identified).
After the transitions are classified, tracing of keys can be performed using one of the following strategies: leftmargin=*

VII. EVALUATION OF RELATIVE TRANSITION BASED ATTACK
In this section, we present the findings from our evaluation of the relative transition based attack framework.

A. Experimental Setup
The same experimental setup and participants as in Section V-A were used for DH-NHHT. The only difference was that the smartwatch was worn on the right hand by the participants, and the smartphone was held in the left hand.

B. Constructing and Testing the Framework
We construct our transition classifiers based on training datasets of labeled transition-records generated by the same 12 participants who helped create the classifiers for the SH-NHHT and SH-HHT scenarios. The same audio stream of uniformly distributed random numbers between 0 to 9 guided the participants in typing 100 numbers. Out of the 1200 total numbers typed by all 12 participants, we use 960 numbers (having 948 transitions) for training and rest for testing. We test the accuracy of the transition classifiers and tracing algorithms using two 10-digit long number sequences per participant (24 total test sequences, 240 total numbers, and 216 total transitions). We calculate the accuracy of the different tracing algorithms based on the number of correctly identified key presses in the traced number sequence. In order to infer In this example, we are able to uniquely identify the last 4 key-pairs using bidirectional tracing, which allows unambiguous inference of the last 5 key presses.
a key, at least the preceding or following transitions should be uniquely identified. For example, in the instance shown in Figure 16, the transition 9 to 2 and 2 to 0 both have other contending key-pairs (the incorrect transitions which are not removed by the tracing algorithm because they fit in the overall sequence of transitions). In such cases, it becomes impossible to determine the exact key (2 in this example) pressed in one trial. However, although the transition 2 to 0 have other contending key-pairs, the pressing of key 0 can be inferred with the help of the uniquely identified 0 to 7 transition, following the key press. In case both the preceding or following transitions are uniquely identified, the adversary can be more confident about the inference. One may also notice that the first and last number in a sequence are harder to infer, as there exists only one transition for each.
of the 240 test numbers were unambiguously identified using the bidirectional tracing (43.75% accuracy). We relate the relatively low inference accuracy to three primary reasons: (a) incorrectly classified transitions introduce error in not one but two key presses, (b) unclassified transitions do not introduce error but there is no remedy to fill in the missing information, and (c) even a very small number of contending key-pairs makes it impossible to determine the exact key pressed. Because most of today's information systems acknowledge natural human mistakes and allows multiple trials to validate security tokens (pin, password, card number, etc.), adversaries can easily take advantage of it to try all possible number sequences derived from the output of bidirectional tracing. Accordingly, we evaluate the inference accuracy using multiple trials (solving from left to right), up to the maximum number of trials required to correctly infer the full number sequence. For example, in the instance shown in Figure 16, there can be 21 possible sequences derivable from the output of bidirectional tracing (listed in Table V). Results of multiple trials are presented in Figure 17, where we see that more ambiguous sequences require additional number of trials (in the worst case). We do not restrict the adversary to a certain number of attempts (which would be system dependent) because the actual sequence may or may not be tried in the limited number of attempts. Instead, we evaluate the worst case scenario, where the adversary has to try all possible sequences derived from the output of bidirectional tracing. Note that we evaluate this using only the 7 bidirectionally traced sequences for which all the predicted transitions are correct.

C. Combining Smartwatch and Smartphone Data
As smartphones cannot capture transitional wrist movements of the typing hand, we cannot merge feature vectors like it was done in SH-NHHT and SH-HHT scenarios. As an alternative, we found a novel way to combine smartphone motion sensor data due to key taps, which was used in the classification-based attacks (as evaluated in Section V-E), with the smartwatch transition-records obtained in DH-NHHT scenario. Based on the previous observation that classifying transition-records itself is highly accurate (88.42% accuracy), we continue using the same attack framework. However, to overcome the limitations faced in the inference process (after tracing is completed), classified smartphone keystrokerecords may be used to choose from the multiple candidate sequences obtained as the output of the tracing algorithm. We evaluate this attack using linear accelerometer readings of the smartphone, which were additionally collected during the DH-NHHT experiments of previous section. Keystroke-records and feature vectors are extracted from the smartphone data as it was done for SH-NHHT and SH-HHT. Elimination of contending key-pairs and filling up of undetected transitions with the help of classified smartphone keystroke-records (combined with reasonably accurate classification of keystroke-records), resulted in 82.50% unambiguous inference of key presses. This is a substantial improvement in the inference accuracy, compared to the 43.75% accuracy obtained earlier without the help of classified smartphone keystroke-records.

D. A More Realistic Setting: Natural or Non-Controlled Typing
Similar to SH-NHHT and SH-HHT experiments, the participants were being directed by an audio stream in the above DH-NHHT experiments, which may introduce a minor delay or disturbance in each key press. As a result, we conducted a similar natural or non-controlled typing experiment in the DH-NHHT scenario, where a completely new set of 12 participants were instructed to type their phone number followed by their residential zip code (15 numbers, 14 transitions). In this setting, out of a total of 168 transitions-records, 134 were classified correctly (79.76% accuracy). The remaining incorrect or unclassified transitions introduced error in test sequences of 10 participants. Out of a total of 180 key presses, 69 were unambiguously identified using the bidirectional tracing, thus giving an accuracy of 38.33%.

A. Limitations
Posture and Ambient Movement: In practice, wrist movement patterns may change drastically based on the target user's body posture and orientation. In other words, the key press features while sitting may differ substantially from the key press features while laying down. One main limitation of our attack framework is that it is not robust against such different body postures and orientation. In order to overcome this, an attacker must train multiple classification models using data corresponding to different user postures and orientations, and then apply the appropriate one for the victim. This, if the attacker knows what was the victim's posture while typing. Similarly, if the target user is moving (for example: walking, running, sitting inside a car or train, etc.) while typing, keystroke events in the accelerometer/gyroscope data may get masked and our framework may not be able to correctly infer them. However, we must point out that this issue is not specific only to our attack framework, but other frameworks in the literature suffer from a similar drawback. Power Consumption: Another limiting factor of our attack can be the power consumption rate on the smartwatch, due to the continuous recording of sensor data at a high frequency. For instance, the 300 mAh battery inside the Samsung Gear Live dropped from 100% to 69% in an hour, while recording linear accelerometer readings at 50 Hz. This limitation is less evident in case of smartphones due to their significantly higher battery capacity. To carry out a stealthy attack using the smartwatch, an attacker may have to either reduce the sensor sampling rate, or devise a mechanism to start the recording only when the potential victim is typing. Both Hand Typing: We cover three major typing styles in this paper, while missing the case where a user holds the smartphone and types using both hands. In this scenario, the motion captured by the smartwatch will vary depending on which thumb is used to type a key, and which hand the smartwatch is worn on. The movement captured in this typing scenario will yield very different results and requires a new inference technique. We plan to work in this direction in the future. Threats to Validity: Most of our experimental results were obtained from analysis of keystrokes typed in a relatively controlled setting, where participants were dictated on what to type. As a result, it is possible that those results may not be representative of how our attack framework may perform in more natural typing scenarios. However, we must point out that we do investigate the efficacy of our attack framework in several natural typing scenarios (in Sections V-F and VII-D), and the obtained results show that our inference framework has reasonable accuracy in these scenarios as well.

B. Defenses
Defending against side-channel attacks is a much debated topic [52]. Although modern mobile and wearable operating systems offer access control on some sensors, sensors such as accelerometer and gyroscope cannot be disengaged by the user. Moreover, most mobile applications do not require explicit permissions (either at install or run time) in order to access these sensors. A straightforward defense approach is to safeguard all sensors using system or user-defined access controls. However, such a static access control will become increasing complex to manage and will not protect against applications that gain legitimate access to these sensors. Reducing the frequency at which applications can sample data from these sensors is another potential defense mechanism. A system-level monitoring mechanism that tracks the context and frequency of sensor accesses, and appropriately flag unwanted accesses requested by applications, could also serve as a useful defense tool.

C. Enhancements
Random Walk Tracing: This is a tracing algorithm we propose for use with very long number sequences typed in DH-NHHT scenario. In this tracing algorithm, a random subsequence of varying length is selected and bidirectional tracing is applied. The process is repeated several times such that every transition is covered multiple times, and each key press may end up having multiple candidate keys. Majority voting may be used to determine the final predicted keys (only if a key press has multiple candidate keys). This tracing algorithm will greatly minimize the propagation of any error that may be introduced by a transition misclassification.

IX. CONCLUSION
In this paper, we comprehensively investigated the feasibility of keystroke inference attacks on mobile numeric keypads by using smartwatch motion sensor data as an information side channel. We proposed two supervised learning-based frameworks to infer keystrokes from smartwatch motion data in three popular mobile holding and typing scenarios. We empirically evaluated the performance and efficacy of our proposed inference frameworks under various experimental settings (i.e., controlled versus natural typing), by using different types of smartwatch hardware, by using different types of motion sensors (i.e., accelerometer versus gyroscope) and by fusing motion data from multiple sources (i.e., smartphone and smartwatch). We also evaluated the performance of our attack framework on alphanumeric mobile keypads with a QWERTY layout. Results from our various experimental studies have shown that typing-induced motion data captured by smartwatch sensors can be employed as an effective side-channel to infer keystrokes on mobile keypads.
Jibo He received his Ph.D. degree in Engineering Psychology from the University of Illinois, and Bachelor's degrees from Peking University. He is the director of the Human Automation Interaction Lab at Wichita State University. His research interests include human factors, driver distraction, aviation safety, eye movement, human-machine interaction, usability, and mobile devices.
Igor Bilogrevic is a Research Scientist at Google, which he joined in 2014. He earned his Ph.D. on the privacy of context-aware mobile networks from EPFL in 2014. From 2010 until 2012, he worked in collaboration with the Nokia Research Center on privacy in pervasive mobile networks, encompassing data privacy, social community privacy, location privacy and information-sharing. In 2013, he spent the summer at PARC (a Xerox Company), where he worked on topics related to private data analytics. His main research interests include applications of machine learning to privacy problems, private data analytics, contextual intelligence, and applied cryptography. He is co-inventor on several privacyrelated patents.