FingerPass: Finger Gesture-based Continuous User Authentication for Smart Homes Using Commodity WiFi

(1)

FingerPass: Finger Gesture-based Continuous User Authentication for Smart Homes Using Commodity WiFi

Hao Kong

^∗

Shanghai Jiao Tong University [email protected]

Li Lu

^∗

Jiadi Yu

^†

Yingying Chen

WINLAB, Rutgers University [email protected]

Linghe Kong

Minglu Li

ABSTRACT

The development of smart homes has advanced the concept of user authentication to not only protecting user privacy but also facilitating personalized services to users. Along this direction, we propose to integrate user authentication with human-computer interactions between users and smart household appliances through widely-deployed WiFi infrastructures, which is non-intrusive and device-free. In this paper, we propose FinдerPass which leverages channel state information (CSI) of surrounding WiFi signals to continuously authenticate users through finger gestures in smart homes. We investigate CSI of WiFi signals in depth and find CSI phase can be used to capture and distinguish the unique behavioral characteristics from different users. FinдerPass separates the user authentication process into two stages, login and interaction, to achieve high authentication accuracy and low response latency simultaneously. In the login stage, we develop a deep learning-based approach to extract behavioral characteristics of finger gestures for highly accurate user identification. For the interaction stage, to provide continuous authentication in real time for satisfactory user experience, we design a verification mechanism with lightweight classifiers to continuously authenticate the user’s identity during each interaction of finger gestures. Experiments in real envi- ronments show that FinдerPass can achieve 91.4% authentication accuracy, and 186.6ms response time during interactions.

CCS CONCEPTS

• Security and privacy → Mobile and wireless security; • Net- works → Home networks.

KEYWORDS

User authentication, finger gesture, WiFi signals, smart home

∗Hao Kong and Li Lu are the co-first authors

†Jiadi Yu is the corresponding author

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Mobihoc ’19, July 2–5, 2019, Catania, Italy

ACM ISBN 978-1-4503-6764-6/19/07. . . $15.00 https://doi.org/10.1145/3323679.3326518

ACM Reference Format:

Hao Kong, Li Lu, Jiadi Yu, Yingying Chen, Linghe Kong, and Minglu Li.

2019. FingerPass: Finger Gesture-based Continuous User Authentication for Smart Homes Using Commodity WiFi. In The Twentieth ACM International Symposium on Mobile Ad Hoc Networking and Computing (Mobihoc ’19), July 2–5, 2019, Catania, Italy. ACM, New York, NY, USA, 10 pages. https:

//doi.org/10.1145/3323679.3326518

1 INTRODUCTION

With the development of Internet of Things (IoT), smart household appliances are increasingly pervasive and common in home environments, making smart homes a practical realization. According to a report [18], the deployment rate of smart household appliances is 32.0% in 2018, and expected to hit 53.1% by 2022. Smart household appliances store various sensitive information such as personal interests, hygiene habits, health status, which could facilitate a variety of customized services. However, such potentially leaked information could cause unauthorized access to personal data and derivation of personal lifestyles. Thus, it is essential to provide secure user access to smart appliances in home environments.

Personal Identification Number (PIN) [1] and biometric-based approaches (e.g., fingerprint [4], voiceprint [28], face recognition [15], etc.) are the most widely deployed user authentications. These approaches are successful, but they only provide one-off user authentication, and are not sufficient for scenarios where continuous privacy protection is necessary. Moreover, biometrics are vulnerable to replay attacks. To provide continuous protection, some works [12, 25] explore the human-computer interactions between users and appliances to implement continuous user authentication for smart homes. But these methods require either wearable wrist sensors or pre-deployed infrastructure, which are intrusive for users and induce a high cost. Recently, WiFi-based user authentication attracts considerable attention, because of the widespread deployment of WiFi infrastructures in indoor environments. Researches [16, 27] utilize WiFi signals to distinguish users based on daily human activities. However, these approaches are only realized based on coarse-grained movements (e.g., gaits, daily activities).

In smart homes, the fine-grained finger gesture rather than coarse-grained human movements is a natural and common way of human-computer interactions. Almost all information or service requests to smart household appliances are issued by finger gestures from users. Toward this end, our goal is to secure each request of finger gestures, i.e., enable continuous user authentication based

(2)

on each human-computer interaction, so as to achieve continuous privacy protection. Recently, many works extend the usage of WiFi signals rather than communications, such as indoor location [26], recognition of human activities [24], and breathing rate monitoring [10]. This inspires us to leverage WiFi signals for finger gesture-based user authentication, which is non-intrusive and device-free. To implement the finger gesture-based continuous user authentication through WiFi signals, we face several challenges in practice. First, we should mitigate the always-existed interference induced by unconscious finger motions in CSI of WiFi signals to extract robust features of finger gestures. Second, the authentication system needs to be capable of accurately identify each individual based on extracted unique behavioral characteristics for secure access control. Finally, the user authentication should provide a real-time response for satisfactory user experiences due to user authentication integrated with human-computer interactions.

In this paper, we first investigate the feasibility of leveraging WiFi signals for finger gesture-based user authentication in depth. By an- alyzing Channel State Information (CSI) of WiFi signals induced by finger gestures, we find that the behavioral characteristics of different users can be presented in CSI phase of WiFi signals. Also, since the user authentication is integrated with human-computer interactions, we are inspired to maintain the real-time response during each finger gesture-based user authentication for satisfactory user experiences. Toward this end, we propose a finger gesture-based continuous user authentication system, FingerPass, which leverages CSI of WiFi signals to continuously authenticate users during fin- ger gesture-based interactions. First, FingerPass pre-processes the received CSI of WiFi signals and segments the signals into episodes for every finger gesture through amplitude differential, and then recognizes different finger gestures through Support Vector Machine (SVM). To achieve high authentication accuracy and satisfactory user experience simultaneously, the whole authentication process of FingerPass is divided into two stages, i.e., the login and interaction stages. The login stage identifies a user’s identity based on a specific login finger gesture from multiple registered users. In order to ensure high user authentication accuracy in the stage, we propose a deep learning-based approach, i.e., Long Short-Term Memory-based Deep Neural Network (LSTM-based DNN), to mitigate the interfer- ence induced by unconscious motions and further capture unique behavioral characteristics of finger gestures from CSI phase of WiFi signals for user identification. After a successful login, the user can interact with the system in the interaction stage. To provide continuous protection in real time, FingerPass verifies the user’s identity during each interaction of finger gestures. We propose a verification mechanism integrated with the lightweight Support Vec- tor Domain Description (SVDD) to achieve the real-time continuous authentication. Experiments demonstrate that FingerPass is reliable for continuous user authentication in home environments.

We highlight our contributions as follow.

• We find that utilizing CSI phase of WiFi signals can authenticate user’s identity based on each user’s finger gestures, and propose a finger gesture-based user authentication system, FingerPass, which utilizes CSI of WiFi signals to authenticate users based on finger gesture interactions.

• We develop a deep learning-based method to mitigate the always- existed interference induced by unconscious finger motions for

exploring robust sequential relationship of finger gestures, and further extract unique behavioral characteristics underlying the sequential relationship for user identification.

• We design a verification mechanism to secure each interaction of finger gestures, i.e., enable continuous authentication for human- computer interactions, which achieves real-time response and high authentication accuracy simultaneously.

• We conduct experiments in home environments. The results show that FingerPass can achieve an authentication accuracy of 91.4%, and a response time of 186.6ms during interactions.

2 RELATED WORK

WiFi Signal-based Applications. Recently, WiFi-based sensing attracts considerable attentions because of the wide deployment of WiFi infrastructures in home environments. Previous studies explore WiFi signals for crowd counting [30], breathing rate monitoring [10], indoor location [26], etc. Furthermore, due to the ability of capturing subtle movement with CSI of WiFi signals, more recent works propose to recognize human activity [23], even fine-grained finger gestures [9, 21], through WiFi signals.

User Authentication. Personal Identification Number (PIN) [1]

is a common user authentication method, but it is easily leaked to others. To overcome the vulnerability of PIN-based authentication, previous works propose biometric-based authentications, such as fingerprint [4], voiceprint [28], and face recognition [15], etc. How- ever, these works are all vulnerable to replay attacks, which are easy to carry out, requiring neither sophisticated equipment nor specific expertise. Moreover, the biometric-based authentications are not appropriate for continuous user authentication, due to poor user experiences under frequent active authentications.

User Authentication for Smart Homes. Recent works [12, 25] realize user authentication based on human movements for smart homes. These works either require users to wear intrusive sensors or pre-deploy infrastructures to sense human activity for user authentication. Such strong requirements hinder the wide deployment of these works in practical home scenarios. More recently, some works [16, 27] explore the widely-existed WiFi signals to identify the daily human activity (e.g., walking gait) to authenticate users. However, these works are limited to coarse-grained activities (e.g., walking), which cannot provide continuous authentication during human-computer interactions.

Unlike existing works, our work aims to capture the unique behavioral characteristics of fine-grained finger gestures during human-computer interactions, for continuous user authentication in smart homes, which is non-intrusive and device-free.

3 PRELIMINARY

In this section, we first describe the attack scenario in smart homes, and then explore commodity WiFi for continuous user authentication with satisfactory user experiences.

3.1 Attack Scenario in Smart Homes

As the popularity of smart homes, smart household appliances would not only store individual privacy, but also provide personalized services for specific family members. Traditional user authentication for smart household appliances is usually a one-off process,

(3)

0 0.8 1.6 2.4 Time (s) 12

16 20 24

CSI Amplitude

User 1 User 2

(a) Waving right.

0 0.8 1.6 2.4

Time (s) 8

12 16 20

CSI Amplitude

User 1 User 2

(b) Circling left.

Figure 1: CSI Amplitude of two users when performing two finger gestures.

i.e., authenticate user’s identity only once during login. However, users usually do not log out such smart household appliances when they are temporarily suspended. This may result in two representa- tive attack scenarios in smart homes. The first scenario is that the individual information may be leaked to adversaries. For example, a malicious guest in the home may eavesdrop privacies of family members from smart household appliances. The second scenario is that the personalized services provided for specific users may be mistakenly provided to other unsuitable users. For example, children may interact with smart household appliances out of curiosity during the absence of adults, which makes it possible for children to use unsuitable services (e.g., adult movie, online shopping, etc.).

The traditional user authentication cannot provide a continuous guarantee for privacy protection to prevent the two attack scenarios. Therefore, it is necessary to secure each interaction between users and smart household appliances, i.e., enable a continuous user authentication during the use of smart household appliances.

3.2 Authentication Feasibility via Finger Gesture Sensing using WiFi

The finger gesture is a natural and common way of human-computer interactions between users and smart household appliances. In smart homes, WiFi infrastructures are widely deployed in home environments. The Channel State Information (CSI) of WiFi signals [5] describes the channel properties of a WiFi signal’s propagating path, which can be utilized to recognize different finger gestures through CSI amplitude [9, 21]. Hence, it is natural to first investigate the feasibility of utilizing CSI amplitude of WiFi signals for finger gesture-based user authentication.

To validate whether CSI amplitude can be used to distinguish different users, we conduct an experiment involving two volunteers in a lab. Each volunteer is required to perform two different finger gestures (i.e., waving right and circling left) three times. There is a wireless Access Point (TP-LINK-WDR5620) and a laptop (HP Pavil- ion 14) equipped with Intel WiFi Link 5300 NIC. The distance be- tween the AP and the laptop is 1m. The finger gestures are required to be conducted in the middle of the two devices. Figure 1 shows CSI amplitude at 20^thsubcarrier when two volunteers perform two different finger gestures three times respectively. We can see that there are significant differences on CSI amplitudes for the two finger gestures, which is consistent with existing works[9, 21]. However, the differences between different users are not distinct enough to separate the curves, which indicates that the CSI amplitude cannot

0 0.8 1.6 2.4

Time (s) -0.3

-0.2 -0.1 0 0.1 0.2 0.3

Relative Phase

User 1 User 2

(a) Waving right.

0 0.8 1.6 2.4

Time (s) -0.3

-0.2 -0.1 0 0.1 0.2 0.3 0.4

Relative Phase

User 1 User 2

(b) Circling left.

Figure 2: Relative phase of two users when performing two finger gestures.

be used to distinguish different users. This is because subtle differences between users are overridden by the differences between different finger gestures. Specifically, a finger gesture blocks the propagating path of WiFi signals between a transmitter and a receiver, which induces significant energy attenuation of received WiFi signals. Thus, the CSI amplitude would change significantly due to the energy attenuation. However, since the finger block- age depicts the coarse-grained characteristics of finger gestures, it would override the fine-grained behavioral uniquenesses of different users, i.e., the subtle differences between users cannot be exhibited in CSI amplitude of WiFi signals.

In order to distinguish different users through CSI of WiFi signals, we consider another measure, i.e., CSI phase of WiFi signals, which can express the movement of an object in the propagating path of WiFi signals [23]. However, the absolute CSI phase has an unpredictable offset due to hardware imperfection [29]. Hence, we employ relative phase to eliminate the offset and further reveal fine-grained behavioral characteristics. The relative phase at the k^thsubcarrier can be represented as:

∠ ˆH_k= −2π

λ Δd, (1)

whereΔd is the length difference of two transmitting paths, and λ is the signal wavelength. The relative phase can reveal more fine-grained behavioral characteristics due to the cm-scale λ [16].

To explore the feasibility of user authentication through CSI phase of WiFi signals, we analyze the data collected from the previous experiment. Figure 2 shows the relative phase of WiFi signals when the two volunteers perform the two finger gestures respectively. Different from CSI amplitude, there are differences between different users when performing the same finger gesture, which can be used to distinguish different users. The result indicates that utilizing CSI phase of WiFi signals is feasible to authenticate user’s identity. This is because the differences between different users mainly depend on behavioral characteristics, such as moving distance, speed, and orientation of a user’s fingers and palm. The moving fingers and palm reflect WiFi signals, which would change the length of a propagating path of the WiFi signals. Such a length change of propagating path can be exhibited as CSI phase shift of WiFi signals [2]. This indicates that the CSI phase of received WiFi signals would express how fingers and palm move according to users daily habits. Therefore, the phase shift induced by the path length changes can depict the fine-grained behavioral characteristics of users when performing finger gestures, which can be further used to authenticate users identities.

(4)

Through the analysis above, we find CSI phase of WiFi signals can depict differences between different users due to the exhibition of behavioral characteristics. Therefore, we employ CSI amplitude to recognize different finger gestures, while leverage CSI phase to authenticate users’ identities.

3.3 Finger Gesture-based Continuous User Authentication in Real Time

As mentioned above, we aim to integrate user authentication into the finger gesture-based interactions between users and smart household appliances, i.e., enable continuous user authentication.

In such a scenario, the satisfactory user experience is an important aspect of designing the user authentication system. When inter- acting with smart household appliances, users always require a real-time response of each interaction rather than waiting for a long time. Thus, the continuous user authentication should meet the real-time response requirement for the satisfactory user experience. To design the user authentication within a real-time response, we first consider the workflow of a typical human-computer interaction system with one-off user authentication. The first step of such a system is an identification process of a user’s identity for user login, which is actually a multi-class problem. After a successful login, all subsequent services of the system are provided based on the current login user’s identity. To extend the typical system to continuous user authentication, we consider enabling a verification process during each interaction, i.e., verify whether the current user is the logged-in user or not. Thus, the user authentication during interactions is actually a binary classification.

Toward this end, to simultaneously ensure continuous authentication and real-time requirements, the whole authentication process is considered as two stages, i.e., the login stage and the interaction stage, which corresponds to user identification and verification respectively. The login stage is a one-off user identification which identifies a user’s identity based on a specific login finger gesture.

In the stage, there is almost no interaction request issued by users.

Hence, a relatively long response time for a high accuracy identification would not suffer significant degradation of user experiences.

The interaction stage verifies the identity of the current user according to each interaction of finger gestures. To ensure a satisfactory user experience during interactions, the system should respond to finger gesture interactions in real time.

In practice, challenges emerge when implementing the finger gesture-based continuous user authentication through WiFi signals. First, the CSI phase of WiFi signals affected by finger gestures contains not only behavioral characteristics, but also unconscious motions that cannot represent the characteristics of users. Hence, the unique features of behavioral characteristics should be extracted for robust user identification during login. Second, in order to sat- isfy user experience, user verification during interactions can be achieved by a lightweight method for a real-time response. However, using the lightweight method to verify users may affect authentication performance due to the limitations of the method itself.

Thus, a verification mechanism should be explored for improving the verification accuracy as well as ensuring the real-time response during interactions.

LSTM-based DNN Feature Extraction Multi-class Classifier

Training

DNN Model

Interaction Stage

SVM-based Finger Gesture Recognition Signals Segmentation

SVDD-based User Verification Mechanism SVDD-based One-class

Classifier Training

SVDD Model Register Stage

Signals Segmentation

Signals Pre-processing Signals

Pre-processing Login Stage

SVM-based Finger Gesture Recognition

User Identification and Spoofer Detection LSTM-based DNN Feature Extraction Signals Segmentation

Signals Pre-processing

User Verification for Interaction User Identification

for Login

Figure 3: Architecture of FingerPass.

4 SYSTEM DESIGN

In this section, we present the design of a finger gesture-based user authentication system, FinдerPass, which leverages CSI of WiFi signals to continuously authenticate users based on finger gestures with high accuracy and real-time response in smart homes.

4.1 System Overview

Figure 3 shows the architecture of FingerPass, which includes a register stage, a login stage, and an interaction stage. The register stage is an off-line training process, and the login and interaction stages are on-line authentication processes.

In the register stage, the system collects multiple finger gestures from family members as the training data for model construction.

First, FingerPass processes the received WiFi signals to mitigate multipath effects through Inverse Fast Fourier Transform (IFFT) and Butterworth filter, and selects specific subcarriers which are sensitive to movements. Then, the pre-processed signals are segmented into episodes of finger gestures based on the amplitude differential of in received WiFi signals. Afterward, FingerPass apply Long Short- Term Memory-based Deep Neural Network (LSTM-based DNN) to construct a DNN model for user identification during login. Finally, Support Vector Domain Description (SVDD) is applied to construct lightweight SVDD model for user verification during interactions.

In the login stage, FingerPass identifies a user’s identity based on a specific login finger gesture. Specifically, FingerPass first processes and segments signals, which is the same as that in register stage, and then recognizes the login finger gestures through Support Vector Machine (SVM) based on CSI amplitude. Next, FingerPass extracts unique behavioral characteristics of the user from CSI phase through LSTM-based DNN feature extraction, and applies the trained DNN model for user identification and spoofer detection.

In the interaction stage, each finger gesture-based interaction is first recognized as an interaction request, and then authenticated to provide a security guarantee. Specifically, FingerPass first processes and segments signals, then recognize finger gestures, which is similar to that in the login stage. Then, a verification mechanism with trained SVDD models is applied for continuous user authentication during the finger gesture-based interactions.

4.2 Signal Pre-processing

In this section, we first describe how to mitigate multipath effects, and then give the method of selecting sensitive subcarriers.

(5)

7LPHPV

&6,$PSOLWXGH

,QGH[RI6OLGLQJ:LQGRZV

$POLWXGH'LIIHUHQWLDO

(SLVRGH (SLVRGH

7KUHVKROG

Figure 4: Illustration of signals segmentation leveraging am- plitude differential.

4.2.1 Multipath Mitigation. Except for the targeted finger, the nearby moving objects (e.g., walking people) also reflect the omni- directional WiFi signals, i.e., the multipath effect. Such a multipath effect could interfere the received WiFi signals of finger gestures, and thus reduce the robustness of the finger geature-based user authentication. Therefore, it is necessary to mitigate the multipath effect from received WiFi signals. The signal reflections from the distant dynamic movements usually have longer propagation de- lays before arriving at a receiver. Hence, we remove these signals components with a large time delay to mitigate multipath effects [21]. Specifically, given CSI of WiFi signals at each subcarrier in the frequency domain, we obtain the power delay profile through the n-point Inverse Fast Fourier Transform (IFFT). The previous study shows indoor environments have the maximum delay less than 500ns[7]. Hence, we remove the signal components with a delay longer than 500ns in the power delay profile, which mitigates multipath effects caused by distant dynamic movements. Note that although the IFFT operation reduces the time resolution of received WiFi signals, the mitigation of multipath effect based on IFFT con- tributes more on resisting the interference from ambient moving objects and improving the robustness of user authentication.

4.2.2 Subcarriers Selection. Although WiFi infrastructures usually provide multiple subcarriers for communication (e.g., 30 subcarriers of Intel WiFi Link 5300 NIC), not all of them contribute to capturing human movements [16], due to the insensitivity to specific envi- ronmental changes. In order to reduce the coverage of insensitive subcarriers on unique features of finger gestures, FingerPass needs to select sensitive subcarriers from the m subcarriers on CSI of WiFi signals. Specifically, we select k subcarriers whose variance values exceed mean variance values as sensitive subcarriers. For each sensitive subcarrier, we use the proportion of variance values wias weights to combine each subcarrier so as to get a combined sensitive subcarrier. Given W =_k

i=1w_i which represents the sum of sensitive subcarriers’ variance values, the combined carrier H is calculated as H =_k

i=1wi

WH_i, where H_idenotes CSI of i^thsubcarrier. The combined subcarrier with high sensitivity for finger gestures could enhance the feature extraction of unique behavioral characteristics from different users.

4.3 Finger Gesture Detection and Recognition

In this section, we describe the segmentation of CSI amplitude of received WiFi signals to detect each finger gesture, and the recognition of different finger gestures.

4.3.1 Signals Segmentation. Different finger gestures would induce different values on CSI amplitude. The top part of figure 4 shows CSI amplitude of WiFi signals induced by two different finger gestures.

Compared with the second finger gesture, the amplitude value of the first finger gesture is not obvious, which is similar to the white noises in CSI amplitude. This may results in that some finger gestures cannot be detected, or some white noises in CSI amplitude would be mistakenly detected as a finger gesture. However, we can observe that CSI amplitude induced by different finger gestures all have significant changes. Hence, we consider utilizing the amplitude change for signals segmentation.

To depict the change of CSI amplitude, we define amplitude differential, i.e.,

D(n) =

(n+1)L−1

t =nL

|(C_{t +1}− C_t)| , n ∈ [0, N − 1], (2)

where D(n) denotes the amplitude differential of n^thsliding win- dow, L is the length of a sliding window, C_tis a CSI amplitude value at time t, and N is the number of sliding windows. The bottom part of figure 4 shows the amplitude differentials of two different finger gestures. We can see that although the amplitude value of the first finger gesture is not obvious, the amplitude differential of the finger gesture is significantly different from that of white noises. Thus, we can utilize the amplitude differential for signal segmentation of finger gesture episodes. Specifically, FingerPass utilizes sliding windows to compare the amplitude differential with a predefined threshold for capturing the starting and ending point of a finger gesture. The threshold can be obtained through empirical studies.

4.3.2 Finger Gesture Recognition based on SVM. As mentioned in Section 3.3, the interaction between users and smart household appliances requires a real-time response for a satisfactory user experience. Existing works about finger gesture recognition based on CSI of WiFi signals mainly employs Dynamic Time Warping (DTW) [9, 21]. However, the computational complexity of DTW- based recognition is O(Knm) [14], where K is the number of known finger gestures, n and m are the numbers of sampling points in the matching finger gesture episodes respectively. Such a computational complexity cannot meet the real-time requirement.

To meet the requirement of real-time response, we employ Sup- port Vector Machine (SVM) [20] for finger gesture recognition.

Combined with a one-versus-one strategy, FingerPass constructs a multi-classifier, whose computational complexity of finger gesture recognition is O(K²n). Usually, the number of finger gestures is limited in a human-computer interaction system, which leads to a far smaller K than the number of sampling points n. Therefore, the time consumption of the SVM-based method would be significantly lower than that of DTW, which results in a faster response during human-computer interactions for a satisfactory user experience.

4.4 User Identification through Deep Learning for Finger Gesture-based Access Control

In smart homes, before a user interacts with smart household appli- ances, FinдerPass should first obtain the identity credential in login stage. The user authentication in the login stage can be considered

(6)

Hidden Layers

···

LSTM

LSTM LSTM

···

LSTM

LSTM LSTM

···

LSTM

LSTM LSTM

Gesture-Level Motion-Level User-Level O

X Signal Fragmentation

Sigmoid Function

? Output

Layer

Input Layer

···

CSI Signal

Spoofer Detection

CSI Collection

Layer 3 RNN :3H₃(F ) ₂

h¹1

c¹1 c¹2

h¹2

c²1

h²1

c²2

h²2

c³1

h³1

c³2

h³2

c³N-1

h³N-1

c²N-1

h²N-1

h¹N-1

c¹N-1

v

˥ t

Layer 1 RNN :₁H₁(X)

Layer 2 RNN :2H₂(F ) ₁

Figure 5: Architecture of a three-layer LSTM-based deep neu- ral network.

as an identification problem, i.e., identify the user by a specific login finger gesture, which is actually a multi-class problem.

As mentioned in Section 3.2, CSI phase of WiFi signals depicts behavioral characteristics of each user through finger gestures. Dur- ing performing a finger gesture, there always exist unconscious finger motions, which can interfere the received CSI of WiFi signals.

Since the finger gesture is fine-grained movement, such a subtle interference induced by the unconscious motions could significantly affect the robustness of finger gesture-based user authentication.

To extract unique features from the behavioral characteristics for robust user identification, it is necessary to reduce the interference of these unconscious motions. Usually, a finger gesture exhibits strong relationship between the previous and subsequent finger motions, i.e., the sequential relationship. The unconscious motions of finger gestures are different with normal sequential relationships, i.e., they induce instant significant shifts, which are neither related with previous motions, nor induce sequential effects to the subsequent motions. Thus, we can extract unique features from each user’s finger gestures and eliminate the impact of such unconscious motions through the sequential relationships.

Feature Extraction. To utilize the sequential relationships in CSI phase induced by finger gestures, we propose a three-layer Long Short-term Memory-based Deep Neural Network (LSTM- based DNN) to extract features of each user’s finger gestures for robust user identification. Figure 5 shows the architecture of user identification through the three-layer LSTM-based DNN.

In the proposed DNN model, each hidden layer consists of a Re- current Neural Network (RNN) with LSTM units, which abstracts the input as a set of feature representations. Traditional RNN only maintains the short-term previous information, which leads to the loss of sequential relationships that occurred long ago [3]. Thus, traditional RNN cannot fully depict the unique features from behavioral characteristics. In order to capture all previous information, including long-term and short-term information of finger gestures, we employ LSTM units [6] rather than the typical neural unit in RNN for finger gesture-based user identification. For each hidden layer, LSTM unit can map the input Z_t^l in time slot t and layer l into a feature representation f_t, i.e., f_t = д(PZ_t^l+ b), where д() is a activation function, P is the weight matrix, b is a bias. The input Z^l_tcontains three informations, i.e., Z = [x_t^l, h^l_{t −1}, c^l_{t −1}]^T, where x_t^lis the current information, h^l_{t −1}is the short-term information, and c^l_{t −1}is the long-term information. Thus, RNN with LSTM units

retains both long-term and short-term previous information of finger gestures, which could express more sequential relationships of finger gestures for robust user identification.

Given CSI phase profiles of a user’s finger gestures, each layer of the DNN model contains an RNN Hi, which abstracts the input into a set of feature representations as output. We first partition the input CSI phase profiles X into N small fragments x(t). The fragmented CSI phase profiles of finger gestures are represented as X = [x(1), x(2), ..., x(N )], where x(t) is the fragmented CSI phase profiles in time slot t (t ∈ [1, N ]). The inputs of the first layer are the fragmented CSI phase profiles X of users’ finger gestures, and the gesture-level features F1can be extracted as output by H1(X) in the first layer. Then, the output F1of the first layer is fed to the second layer. H2(F1) in the second layer further extracts the motion- level features F2(e.g., speed, angle, and time). Finally, H3(F2) in the last layer takes the output F2 of the second layer as input, and extracts the user-level features as an output O, which represents user’s unique features and can be used for user identification.

User Identification. FingerPass employ the sigmoid function [8] in output layer through the extracted features to identify a user’s identity. When a user attempts to login, FingerPass calculates the posterior probability P(U_k| O) with the output feature O of the user and every registered user’s features. For output O of the current user from user-level and each registered user U_k, the posterior probability P(U_k | O) is calculated as:

P(U_k| O) = P(O | U_k)P(U_k)

P(O | U_k)P(U_k) + P(O | U_k)P(U_k), (3) where P(U_k) is the prior probability of user U_k, and P(O | U_k) is the likelihood of the feature O given label U_k. The posterior probabilities is under the constraints that 0 P(U_k | O) 1 and P(U_k | O) + P(U_k | O) = 1. Given K user classes, sigmoid function outputs K posterior probabilities. With the objective function k = arg max_{k ∈K}P(U_k| O), the user with feature O will be identified as the user U_k.

Spoofer and Unexpected Body Movements Detection. Ex- cept for identifying the user’s identity from multiple register users, FingerPass needs to detect unexpected spoofers in the login stage.

Through the proposed DNN, the subtle differences between finger gestures of a spoofer and all registered users can be extracted, which can be utilized for spoofer detection. Specifically, when a spoofer attempts to login the system through a finger gesture, the features of the spoofer are extracted by LSTM-based DNN. Since the feature of spoofer would not match that of any registered user, the posterior probability of spoofer in output layer should be significantly lower than that of all registered users. Hence, we use a threshold to detect spoofers. Specifically, we define score_kas the similarity between the feature O of attempted login finger gesture and features U_kof all registered finger gestures, i.e., score_k = {1 | P(U_k | O) > λ, 0 | P(U_k | O) <= λ}. If_K

k=1score_k = 0, the attempted login user with the feature O is identified as a spoofer. Such an approach is also able to detect the unexpected body movements issued by the user. Since the body movement is quite different from a valid finger gesture, the features extracted by DNN would exhibit significant differences. Therefore, even though an unexpected body movement is recognized as a finger gesture, FinдerPass can further identify

(7)

1 2 3 4 5 6 7 8 9 10 The Value of m

50 60 70 80 90 100

Authentication Accuracy (%)

The optimal splicing number

Figure 6: Verification accuracy under different m.

such a body movement as a spoofer’s request, and thus trigger no further permission and interaction response.

4.5 User Verification through SVDD for Finger Gesture-based Interaction

To provide continuous privacy protection and consistent personal services delivery, FingerPass should perform user authentication during each interaction of finger gestures in the interaction stage.

As mentioned in Section 3.3, the system needs to ensure real-time response so as to achieve a satisfactory user experience. Thus, user authentication in the interaction stage can be simplified as a verification problem, i.e., regards a current user as a valid user or an invalid user, which is actually a binary classification.

Verification Mechanism. In order to ensure a real-time re- sponse, we utilize a one-class classifier, Support Vector Domain Description (SVDD) [22], to verify the current user’s identity. How- ever, the SVDD classifier based on single finger gesture has lower classification accuracy since it is a light-weight method. Usually, a user tends to interact with smart household appliances multiple times after the user successfully logins. Based on such an intuition, we propose a verification mechanism, which leverages not only the current finger gesture interaction, but also the previous finger gestures that passed the verification during the ongoing interactions, to continuously authenticate the user’s identity for each interaction.

Thus, the verification mechanism gradually improves the accuracy of SVDD-based classifier as more interactions occur.

In the training process of SVDD-based classifier, FingerPass not only trains single-gesture classifier for every single finger gesture, but also splices finger gestures to train multi-gesture classifiers.

Specifically, we first align all CSI phases of finger gestures through the interpolation method to make the input’s length consistent, and then feed the aligned relative phases into the classifier for training. Assume there are n finger gestures, i.e., д₍₀₎,..., д_(n−1), Fin- gerPass trains finger gesture classifiers for single finger gesture and splicing finger gestures, i.e., for each finger gesture, Finger- Pass trains n single-gesture classifiers, cд(0),..., cд(n−1); for the splic- ing of two gestures, FingerPass trains n² two-gesture classifiers, c_д₍₀₎_д₍₀₎, c_д₍₀₎_д₍₁₎,..., c_д₍₀₎_д_(n), c_д₍₁₎_д₍₀₎,..., c_д_(n−1)_д_(n−1); ... ; for the splic- ing of m finger gestures, FingerPass trains n^mm-gesture classifiers, c_д₍₀₎_д₍₀₎...д₍₀₎, c_д₍₀₎_д₍₀₎...д₍₁₎,..., c_д_(n−1)_д_(n−1)...д_(n−1). Through the training above, we can obtain_m

i=1nⁱ =ⁿ⁽¹⁻ⁿ_1−n^m⁾classifiers including the single-gesture classifiers and the multi-gesture classifiers.

In the verification process, when a user performs the t^th fin- ger gesture interaction д_(t), FingerPass utilizes classifiers of m − 1 previous finger gestures that passed the verification during the on- going interactions combined with current finger gesture, i.e., c_д_(t),

(c) Kitchen User

Laptop Access Point

(a) Living Room (b) Bedroom

Figure 7: Experimental setup of three home environments.

cд(t −1)д(t),..., cд(t−m+1)д(t −m+2)...д(t), to get m preliminary verification results. Then, FingerPass utilizes a voting mechanism that leverages the results of each classifier to obtain a final user verification result.

Specifically, for each classifier, a user verification result μ_i ∈ {1, 0}

is obtained, where μ_i = 1 and μ_i = 0 denote a successful and un- successful verifications respectively. Then, FingerPass votes on all the results, and decides the final user verification result accord- ing to the maximum voting results, i.e., result = {1 |_m

i=1μi >

m2, 0 |_m

i=1μ_i ^m₂}, where result = 1 and result = 0 represent successful and unsuccessful verifications respectively. Similar to the unexpected body movement detection in the login stage, the interaction stage can also recognize an unexpected body movement as a spoofer’s request, due to the significant difference between a valid finger gesture and the unexpected body movement.

The computational complexity of SVDD-based classifier is O(n), where n is the size of each finger gesture sample. Considering m − 1 previous finger gestures are utilized in our verification mechanism, i.e, m classifiers are applied in total, the computational complexity of user verification in the interaction stage is O(nm²). To explore the optimal value of m, we conduct experiments with data collected from real environments. Figure 6 shows the verification accuracy under different m. When we only use a single gesture to verify a user, i.e., m = 1, the verification accuracy is only 68.5%. As more previous finger gestures are utilized, the verification accuracy first increases and reaches the highest accuracy of 89.2% when m is 3, and then decreases due to over-fitting. Since the optimal value m is much less than n, the computational complexity of user verification in the interaction stage can be regarded as O(n). Therefore, through the SVDD-based verification mechanism with a small splicing number, FingerPass can realize high user authentication accuracy and real- time response in the interaction stage.

5 PERFORMANCE EVALUATION

In this section, we evaluate the performance of FingerPass in real home environments.

5.1 Experimental Setup and Methodology

We implement FingerPass on a laptop, i.e., a HP Pavilion 14, which is equipped with an Intel WiFi Link 5300 NIC for providing 30 subcarriers on CSI of WiFi signals. A commercial wireless access point (AP), i.e., a TP-LINK-WDR5620, is employed as the WiFi signal transmitter, which continuously emits the 802.11n WiFi signals. We conduct the experiments in three different home environments, i.e., a living room, a bedroom, and a kitchen. The sizes of the three rooms are 5.8m × 4.2m, 3.8m × 3.4m, and 3.4m × 2.2m respectively.

The distances between AP and laptop in the three rooms are 3.0m, 2.0m, and 1.0m respectively. Figure 7 shows the layouts of the AP, laptop and other furniture in the three home environments.

(8)

1) Wave Left 2) Wave Right 3) Wave Up 4) Wave Down

5) Circle Left 6) Circle Right 7) Zoom Out 8) Zoom In

Figure 8: Illustration of eight different finger gestures.

We select 8 commonly used finger gestures, which are widely used in human-computer interaction systems, as shown in figure 8.

Our experiments involve 7 volunteers. Since the average number of people per family in the US is 3.14 [19], we ask 5 of the volunteers as registered users in the experiments, and the rest 2 as spoofers, which could meet the needs of most families. All of the volunteers are required to perform finger gestures towards the laptop with a distance of 0.5m. This is because users are natural to stand/sit in front of the smart appliance (i.e., laptop in our experiment) during the interaction between users and appliances.

We define several evaluation metrics:

• Response Accuracy. The probability that both the finger gesture and user’s identity are recognized and authenticated correctly.

• Response Time. Assume the CSI of WiFi signals induced by a user’s finger gesture is derived at time T_e, and the time that the system responds the user’s interaction is T_dev. The response time of the system is defined as T = T_dev− T_e.

• Confusion Matrix. Each row and each column of the matrix denotes the ground truth and the authentication result of FingerPass respectively. The i^th-row and j^th-column entry of the matrix shows the percentage of samples that are authenticated as the j^thuser while actually are the i^thuser.

• Authentication/Recognition Accuracy. The probability that a user/finger gesture who is A is exactly identified as A.

• False Accept Rate. The probability that a user not a registered user is authenticated as a registered user.

• False Reject Rate. The probability that a user not a spoofer is authenticated as a spoofer.

5.2 Overall Performance

Figure 9 shows response accuracies in the login and interaction stages. We can see that the average response accuracies of the login and interaction stages are 91.3% and 88.6% respectively. The overall response accuracy of FingerPass is 90.0%. This result demonstrates that FingerPass can achieve satisfactory performance for both in- teraction and authentication. Moreover, it can be observed that the response accuracies in the three home environments are similar, which indicates FingerPass is robust to different distances between transmitter and receiver as well as different home environments.

Figure 10 shows CDFs of response time in the login and interaction stages respectively. Note that the response time contains both finger gesture recognition and user authentication. We can see that for 85% finger gestures, the response times are less than 200ms in the interaction stage, while that are in the range of[800, 1200]ms in the login stage. Previous research validates that the latency of [50, 200]ms in modern touch systems is an appropriate response time for a satisfactory user experience [11]. Since the login stage

Living Room Bedroom Kitchen

0 20 40 60 80 100

Response Accuracy (%)

Environment Login6WDJH

Interaction 6WDJH

Overall

0 200 400 600 800 1000 1200 1400

0.0 0.2 0.4 0.6 0.8 1.0

CDF

Response Time (ms) ,QWHUDFWLRQ 6WDJH

/RJLQ 6WDJH

Figure 9: Response accuracy in different environments.

Figure 10: CDF of time in lo- gin and interaction stages.

0.93

0.00

0.04

0.00

0.00 0.04

0.85

0.07

0.04

0.00

0.00 0.00

0.04

0.81

0.00

0.11

0.04

0.00 0.00

0.00

0.96

0.04

0.00

0.00 0.07

0.04

0.00

0.89

0.00

0.00 0.04

0.04

0.00

0.89

0.00 0.04

0.04

0.00

0.93

U1 U2 U3 U4 U5 S6 S7

Ground Truth U1

U2 U3 U4 U5 S6 S7

Authentication Result

U1 U2 U3 U4 U5

User 0

20 40 60 80 100

Authentication Accuracy (%) Living Room

Bedroom Kitchen

Figure 11: Confusion matrix of user authentication in lo- gin stage.

Figure 12: Authentication ac- curacy of FingerPass in inter- action stage.

has no interaction requests, the relatively long response time would not degrade user experiences. For the interaction stage, it achieves a real-time response of touch system level, which satisfies good user experiences of human-computer interaction.

5.3 Performance of User Authentication

Figure 11 shows the confusion matrix of FingerPass in the login stage. We can see that FingerPass can achieve an average authenti- cation accuracy of 93.3% in identifying the registered user, and that of 90.0% in spoofer detection. The average accuracy of the login stage in user authentication is 92.6% with a standard derivation of 4.43%. This indicates FingerPass can achieve a high accuracy of user authentication in the login stage, which validates the reliability and efficiency of user identification.

Figure 12 shows authentication accuracies of FingerPass in the interaction stage under the three home environments. We can ob- serve that FingerPass can achieve average authentication accuracies of 91.3%, 89.8% and 89.2% under the three home environments respectively, and the standard derivations are 1.6%, 1.7% and 1.3%

respectively. This result indicates that FingerPass can accurately authenticate the logged-in user’s identity in the interaction stage.

Figure 13 shows false accept rates and false reject rates of Fin- gerPass in the login stage under different environments. We can see that the average false accept rate of FingerPass under the three environments is only 3.5%, which demonstrates that FingerPass is re- liable to identify a spoofer in the login stage. Moreover, the average false reject rate under the three home environments is 3.8%, which indicates that FingerPass hardly misidentifies a registered user in the login stage, which ensures a satisfactory user experience.

Figure 14 shows CDFs of interaction numbers for misidentify- ing a logged-in user and identifying a non-logged-in user in the interaction stage. We see that over 90% non-logged-in users can be authenticated within 3 interactions. It indicates that FingerPass is not vulnerable to other users or spoofers, which demonstrates the

(9)

False Accept Rate False Reject Rate 0

1 2 3 4 5 6

False Accept Rate (%)

Living Room Bedroom Kitchen

0 1 2 3 4 5 6

False Reject Rate (%)

Figure 13: False accept rate and false reject rate in login stage under different home environments.

0 5 10 15 20 25 30 35 40

0.0 0.2 0.4 0.6 0.8 1.0

CDF

Number of Interactions Identify a Non-Logged-in User Misidentify a Logged-in User

Figure 14: CDFs of interac- tion numbers to misidentify a logged-in user and identify a non-logged-in user.

Wave LeftWave R ight

Wave UpWave DownCircle LeftCircle RightZoom OutZoom In 0

20 40 60 80 100

Recognition Accuracy (%)

FingerPass DTW-based

Figure 15: Recognition accu- racy of FingerPass and DTW- based method for different finger gestures.

Wave LeftWave R ight

Wave UpWave DownCircle LeftCircle RightZoom OutZoom In 0

100 200 300 400 500 600

Response Time (ms)

FingerPass DTW-based

Figure 16: Response time of FingerPass and DTW-based method for different finger gestures.

2 4 6 8 10

Number of Training Samples 0

20 40 60 80 100

Login Stage (Living Room) Login Stage (Bedroom) Login Stage (Kitchen) Interaction Stage (Living Room) Interaction Stage (Bedroom) Interaction Stage (Kitchen)

(a) Size of training sets.

50 150 250 350 450

Sampling Rates (Hz) 0

20 40 60 80 100

(b) Sampling rates.

0.1 0.4 0.7 1 1.3 1.6 1.9 2.2

Distance between User and Receiving Antennas (m) 0

20 40 60 80 100

(c) Distance between user and antennas.

0 60 120 180 240 300 360

Orientation of an Ambient Person (degree) 0

20 40 60 80 100

(d) Orientation of an ambient person.

Figure 17: Authentication accuracy under different impacts.

reliability of FingerPass. We also observe that when the interaction numbers are below 10 times, about 90% users are still not mistak- enly logged out by FingerPass, which indicates that the continuous authentication rarely affects the normal use of the logged-in user.

5.4 Performance of Finger Gesture Recognition

To evaluate the performance of finger gesture recognition, we implement the DTW-based method [9, 21] as a baseline, and compare the recognition accuracy and response time between DTW-based method and FingerPass. We can see from figure 15 that the aver- age recognition accuracy of FingerPass is quite similar to that of DTW-based method. FingerPass can achieve an average recogni- tion accuracy of 88.7%, which indicates its reliability of interaction through recognizing finger gestures. Moreover, from figure 16, it can be observed that the average response time of FingerPass is 56.1ms, which meets the real-time requirement. But the response time of the baseline is above 350ms, which exceeds the time range for a satisfactory user experience (i.e.,[50, 200]ms). The results show that FingerPass can achieve high accuracy of finger gesture recognition while satisfying the real-time requirement.

5.5 Performance under Different Impacts

Impact of Training Set Size. The size of the training set is the number of users’ performing times of finger gestures for registering. Too much performing times would affect user experiences in the register stage. Figure 17(a) shows authentication accuracies in the login and interaction stages with different training set sizes in the three home environments. We can see that as the size of the training set increases, the authentication accuracies first increase, and then go stable in both stages. When users perform above 8 finger gestures for registering, FingerPass can achieve average au- thentication accuracies of over 80% in the two stages. More finger

gesture performing times would not contribute an improvement in authentication accuracy. Since the performing times of 8 is accept- able for users, it demonstrates that the register stage of FingerPass is also consistent with user experiences.

Impact of Sampling Rate. In our experiment, the laptop is equipped with an Intel WiFi Link 5300 NIC as the WiFi signal receiver, which can receive WiFi signals under a sampling rate ranging in[10, 2000]Hz. To explore the appropriate sampling rates for FingerPass, we evaluate the performance of FingerPass under different sampling rates. Figure 17(b) shows the authentication accuracy of the login and interaction stages in the three home environments respectively. We can see that the authentication accuracy of FingerPass first increases and then goes stable as the sampling rate increases. When the sampling rate approaches 250Hz, Finger- Pass can achieve an authentication accuracy of over 85% in the three home environments respectively. Such a sampling rate is ca- pable for most smart household appliances. Thus, FingerPass can be widely applied for user authentication in smart homes.

Impact of Distance between User and Receiving Antennas.

To explore whether FinдerPass could achieve a satisfactory perfor- mance in a comfortable interaction distance, we first investigate the common interaction distances between user and appliances for smart homes. Usually, smart TV has the longest interaction distance, whose average size is 42.8 inches in 2017 [17], and the distance between such a smart TV and users with the best viewing experience is below 1.6m [13]. Hence, we conduct experiments to explore the performance of FingerPass in the distance within 2.4m that can meet the requirements for most appliances. Since the user is required to experiment between the AP and the laptop, and the distances between the two are fixed at 1m, 2m, and 3m respectively in the three homes, the experimental distance range between user and laptop in each home is set separately. Figure 17(c) shows the