The activities in daily life are not always only performed once (i.e., stirring the cof-fee would repeat until the cofcof-fee powder melted in the hot water). And we can also see some gesture on object in the receipt may repeat (i.e., spooning two cup of butter). The different times that the users performed on everyday object did not affect the accuracy of distinguishing different gestures in our study 2. We did not restrict the participant how many times they need to repeat the stirring gesture, and it can still distinguish this ges-ture from others. However, the detection of repetition is still a must-to-do when introduce MidasTouch in our life.
The amount of change in yaw, pitch, roll of two IMUs depends on which gesture is performed. And sometimes it is hard to observe i.e., stirring gesture only trembles the fingertip. So we need another data for observing the pattern of repetition. Morriset al.used the one-axis accelerometer readings to detect the repetition of exercises [23]. Despite of different mounted position and different type of gesture, We referred the implementation of this work. We conducted a small study for finding the repetition in gesture. 7 participants
Figure 5.1: (a) The autocorrelation can detect period in a series of signal, it compares signal with itself at different lagged time (b) The pouring water data of one of all users has multiple peak in its autocorrelation result, so we can not calculate the right repetition
range from 22- to 24-years-old (Mean= 23; STD= 0.8) were recruited in the study. All users are right-handers. They were asked to wear the Midas Touch device perform four task on three handy kitchen utensils. Three different handy kitchen utensils: knife, fork, plate, and spoon, were used in the study. Four tasks are using the fork, using the knife, spooning up water, stirring water (using the spoon). The requirement of this study has one different from previous studies that the user need to repeat their gestures 10 times before they ended one data collection. Overall, we collected 7 (participants) x 4 (gestures with 10 repetitions) x 10 (repetitions) = 280 series of IMU data in total.
We use the autocorrelation to find the period of repetition. This method was widely used for finding repeat or abnormal signals [24] [25] [26]. And Morriset al.successfully used this method to find a repetition from accelerometer readings [23]. The autocorrelation use the signal compare to its lagged signal. When the signal has period, the output of the autocorrelation would has some peaks in it. The y-axis can be seem as the similarity of two signals. The peak means the lagged time may be the period of the repetition of the signal. We first pick the axis which had the max change of acceleration (sum the absolute value of readings in the same axis), and the function of NumPy to do autocorrelation.
Figure 5.2 shows the prediction of each gesture’s repetition of different participants.
We found that the stirring water gesture can be predicted well. However, the spooning up water gesture is hard to be predict by our method. We further saw the origin autocorrelation
result(Figure 5.1(b)) of the gesture and found that there are not only one local-maximal in one period. The multiple peaks in one period is the main cause of error. And we think the cause of multiple peaks in one period is that the gesture is similar in its one repetition.
Figure 5.2: The mean prediction of repetition of each user’s gesture, x-axis represents P1 to P7, y-axis represents predicted repetition
5.2 Editor
In the connecting everyday things application example that we mentioned, the user can define the relationship between everyday objects by performing the daily activities.
Through the recording process is mainly defined by our device itself, it still need an inter-face to edit and show the relationship between these objects to help the user to memorize the whole daily activities which is a specific workflow(e.g. making dishes).
Figure 5.3: The web-based system overview
5.2.1 Web-based System
We implemented a web service which can provide the user a interface to edit, manage their daily activities. The system architecture can be separated in following parts:
1. Gesture recognition by MidasTouch. Collecting raw data.
2. Web server receive data from MidasTouch and recognize which gesture is perform-ing, sending result to user’s mobile phone.
3. The user’s phone receive the now performing gesture on specific object, and the user can draw a icon for it. Save the result to server.
4. When the user performing the same gesture, the system can infer the next gesture according to previous saved workflow.
With the help of the editing tool, the user can generate a series of gestures on everyday objects as a workflow. And the workflow can be transfer to others. In the connecting everyday things application example, the workflow of how to make a cup of cake was transfer to someone else. The other can learn skills by the prompt on smartwatch.
Figure 5.4: The replay interface connects to our web-based system
5.2.2 Editing Interface
The editing interface has several consideration:
1. The user can minimize text input
2. Everyday objects is numerous, the interface needs to accommodate them 3. Some workflows repeat in our live, it need to be reused
To minimize text input, we defined touch gesture as binding things together. When the user pouring water to a cup, he needs to touch the target cup to make the system understand which cup is filling. And we use the noun event representing a gesture performed on a everyday object. The user can define a icon relate to the event.e.g. pouring water from a bottle. Besides, when the next event came, the system can auto establish the relationship by timestamp or binding gesture. After all editing is done, the user can see a whole picture about the workflow that just performed. Like we have mentioned above, some workflow repeats in our live. The user can use a global view to see what workflow he/she has established, and he/she can merge/split different workflows by dragging and dropping.
The editing way is very like video editing software, it has a pool call material, and it can be merge into one. In the whole editing process, the user does not input text. The users use
Figure 5.5: The drawing style UI for users making a icon for their gesture on objects
Figure 5.6: The drag-and-drop manipulation and workflow as material let users easily merge or split the workflows
their image memory to recall what they have done. And the users do not need to establish relationship by theirselves. We think the drag-and-drop and drawing manipulation can reduce the effort, and the users can focus on what they are doing.
5.2.3 Connect to Smart Things
Reality Editor [27] shows a way to bind different smart things together, but it needs to connect micro controller to these things and it does not connect to non-smart things.
Binding smart thing on everyday object is limited by the diversity of APIs on nowadays smart things. We make a prove-of-concept interface to connect everyday object with smart things, it can control two smart things, Sonos speaker and philips hues. In the editing
interface, it can choose smart device if now coming event is a binding gesture. So the user can bind opening door to lighting color(R,G,B) of philips hues. On the doorknob of the inner side, it can be binded with turning off the light. With the editing tool for smart things, we can auto complete some daily routines.
Figure 5.7: The user use the editor to connect playing smart speaker to the picking off the earphone
Figure 5.8: The user use the editor to connect opening the book to the turning on the light
5.2.4 Display Devices
The editing result need a display device to show the user what is the next work to do.
The glasses sounds good but it is not popular enough, so we prompt the user next work to do by the smart watch interface. In the receipts tutorial scenario, one user performing
a complete workflow of making a cup of cake, and another user follow the workflow that previous user established. He/she can cook the same cake by following the prompt from smart watch.
Figure 5.9: The predicting flow chart