穿戴式互動展演創新應用與即時追蹤技術研究 - 政大學術集成

全文

(1)國立政治大學資訊叅學系 Department of Computer Science National Chengchi University 碩士論文 Master’s Thesis. 穿戴式互動展演創新應用與即時追蹤技術研究 Interactive Performance Using Wearable Devices: Real-time Tracking Technology and Innovative Applications. 研究生：鄭仲祐指導教授：蔡子傑. 中華民國一零五年一月 January 2016.

(2) 穿戴式互動展演創新應用與即時追蹤技術研究 Interactive Performance Using Wearable Devices: Real-time Tracking Technology and Innovative Applications 研究生：鄭仲祐. Student：Chung-Yu Cheng. 指導教授：蔡子傑. Advisor：Tzu-Chieh Tsai. 國立政治大學資訊叅學系碩士論文. A Thesis submitted to Department of Computer Science National Chengchi University in partial fulfillment of the Requirements for the degree of Master in Computer Science. 中華民國一零五年一月 January 2016.

(3)

(4)

(5) 穿戴式互動展演創新應用與即時追蹤技術研究摘要近年來越來越多虛實整合技術不斷地湧出，像是電影阿凡達或虛實互動型態的表演這樣的表演會根據事先預錄好的虛擬角色進行演出，但要成功地演出需要演員們不斷的練習. 另外,許多電影利用攝影機. 捕捉人體姿態來與虛擬角色互動，但此方法受限於燈光環境以及障礙物此篇論文運用穿戴式裝置與新一代無線網路藍芽 4.0,提出即時追蹤技術套用於這類表演,可使得演出更加豐富並且能即興創作. 然而，. 目前受限於穿戴式平台上的藍芽訊號強度更新頻率每叆只有 5 至 10 次且傳輸容量有限，所以本篇論文結合體態感測裝置輔助無線網路藍芽 4.0,提升對穿戴者的即時追蹤能力實驗結果與真實位置只有 0.3 至 0.5 叆的延遲時間,並在校內進行兩場互動式展演作驗證成果未來期許可以將此技術運用於多人展演場，讓更多使用者可以互動體驗關鍵字：藍牙 4.0 虛實互動表演即時體態捕捉即時追蹤穿戴式裝置. i.

(6) Interactive Performance Using Wearable Devices: Real-time Tracking Technology and Innovative Applications Abstract Recently, more and more interactive performance technologies appear such as Avatar or virtual-character integrated art perform. Such performs are based on pre-made animations and physics simulations. However, this kind of shows need a lot of practice, and it is impossible that audience or performers play with or interact with the virtual characters. In. addition,. many moviemakers use high quality cameras to distinguish body postures. Although cameras can record anything with high precision, it is constrained on the light and obstacles of the environments. If we, somehow, can capture the motion of the performers in real time, then we are able to interact with virtual characters and make improvisation possible. This thesis aims to use the wearable sensors and the Received Signal Strength Indication (RSSI) of BLE to track in real-time.. However, the update rate of RSSI is limited to 5~10 per. second. This thesis proposed a tracking technique which combines with wearable motion sensors to assist BLE localization.. The tracking lag. can be reduced to only 0.3~0.5 seconds, and also real performance was experimented in the campus. In the future, we hope to use this technique on interactive performance with many people in different places. Keywords: Bluetooth 4.0; virtual-character integrated interactive perform; Real-time motion capture; Real-time Tracking; Wearable devices. ii.

(7) 致謝辭. 首先想感謝蔡子傑教授這兩年來不斷的提攜我，從每一個計畫中點出我所少考慮的細節，讓我受益很多無論是課業或者是論文及生活經驗給予莫大的協助，而老師總會與我們分享做事的態度，讓我學到很多，非常感謝老師總是能不厭其煩地指導我鼓勵我，並時常給予我信心和建議，讓我能順利的完成碩士論文，謝謝蔡子傑老師. 感謝口試委員周承. 老師吳曉光老師江清泉老師和陳伶志老師百忙之中. 能抽空前來，給我寶貴的建議和指導，使得論文能更加完整感謝我實驗室的每一位同學，博班學長們總是會在我的研究上. 序漸進地講解要如何進行，哪一塊. 需要補充，碩士班的同學們總會在我身心疲憊時帶我去運動讓腦筋更清楚，並且有專案時會一同來討論如何完成另外要感謝數位內容學程的黃心健教授和陶亞倫教授，能夠與這兩位台灣藝術大師一同合作是件非常榮幸的事情，在每次的展演中教授們總是陪在我們身邊並且指導我們演出時每一個關鍵的環節，讓展演順利呈現，在此也感謝每一位與我們合作的數位內容學生們，為了每一場的展演他們總是盡心盡力地去安排與規劃，在此深深佩服他們如此敬業的精神！. 感謝這幾年來支持我的家人，肯讓我一個人來政治大學打拚，提供我所需的資源在我求學過程中，總是無條件地幫助我，讓我能無憂無慮的專心學習，而在我心情低落時，總是鼓勵我和包容我，有您們的支持是我最大的勇氣與動力，謝謝您們最後我要謝謝大家對我的包容和體諒，以及對我的關心，祝福大家順利健康快樂，謝謝你們，我愛你們. iii.

(8) TABLE OF CONTENT. TABLE OF CONTENT ................................................................................................ iv LIST OF FIGURE......................................................................................................... vi LIST OF TABLE........................................................................................................... vi CHAPTER 1 Introduction.............................................................................................. 1 1.1 Background ...................................................................................................... 1 1.2 Motivation ........................................................................................................ 2 1.3 Purpose ............................................................................................................. 2 1.4 Organization ..................................................................................................... 3 CHAPTER 2 Related Work ........................................................................................... 4 2.1 Indoor Localization .......................................................................................... 5 2.2 Gait Analysis .................................................................................................... 7 2.3 Inertial Sensor Measurements and RSSI.......................................................... 8 CHAPTER 3 Localization Algorithm ............................................................................ 9 3.1 System Environment ...................................................................................... 10 3.1.1 Acceleration ........................................................................................ 11 3.1.2 Feet Movement ................................................................................... 14 3.1.3 Beacon Location ................................................................................. 16 3.2 Fault tolerance ................................................................................................ 22 3.2.1 Deployment ......................................................................................... 23 3.2.2 Frame Rate .......................................................................................... 24 3.2.3 Coordinators ........................................................................................ 25 CHAPTER 4 Experimental Evaluation ....................................................................... 26 iv.

(9) 4.1 Simulation Setup ............................................................................................ 26 4.2 Simulation Results ......................................................................................... 29 4.2.1 Absolute position ................................................................................ 29 4.2.2 Real performance in Campus .............................................................. 36 CHAPTER 5 Conclusions and Future Work................................................................ 40 REFERENCES ............................................................................................................ 41. v.

(10) LIST OF FIGURE FIGURE 1: THE SPECIFIC AND MODE OF BLUETOOTH AND BLES. .................................... 5 FIGURE 2: THE SKETCH MAP OF THE EXPERIMENTAL ENVIRONMENT. .............................. 6 FIGURE 3: THE ALGORITHM OF HUMAN MOVING DISTANCE. ............................................ 7 FIGURE 4: HAND MOVEMENT MOTION TRACKING IN 2D EXPERIMENT SETUP. .................. 9 FIGURE 5: THE ROTATION AXIS OF THE ACCELERATION. (A) ORIGINAL AXIS (B) ROTATION AXIS(C) Y-Z PLANE (D) X-Z′ PLANE. ............................................................................... 12. FIGURE 6 : THE TOY CAR MOVE 20 CENTIMETERS FROM RIGHT TO LEFT AT THE PLANE . 13 FIGURE 7: THE ACCELERATION, VELOCITY, DISTANCE, YAW, PITCH AND ROLL OF THE TOY. CAR IS RIGHT, AND THE DISTANCE IS ABOUT 20CM. ........................................................ 13. FIGURE 8: THE EXPERIMENT OF MOVING PITCH IN 500MS, AND THE VELOCITY DISTANCE WILL BE WRONG. ........................................................................................................... 14. FIGURE 9: THE DISTANCE BETWEEN LEGS CALCULATED BY ROTATION. ......................... 15 FIGURE 10: THE SIMULATION OF HUMAN MOVEMENT. ................................................... 16 FIGURE 11: THE FOUR DIFFERENT BLE DONGLES MADE IN TAIWAN. ............................. 17 FIGURE 12: THE THREE DIFFERENT BLES MADE IN CHINA. ........................................... 17 FIGURE 13: THE FOUR DIFFERENT BLES SIGNALS RECEIVED BY FOUR BLES DONGLES. 19 FIGURE 14: THE POSITION WE CALCULATED BY 8 BLES. ............................................... 20 FIGURE 15: THE IMU SENSORS USED TO SERVE AS THE AUXILIARY TOOL. ..................... 21 FIGURE 16: THE IMU SENSORS COMBINE WITH THE RSSI SYSTEM. .............................. 22 FIGURE 17: THE SKETCH MAP OF 8 BLES IN THE FLOOR. ............................................... 23 FIGURE 18: THE SKETCH MAP OF 11 BLES IN THE FLOOR. ............................................. 23 FIGURE 19: THE SKETCH MAP OF 21 BLES IN THE FLOOR. ............................................. 24 FIGURE 20: THE INPUT COLUMN OF THE BLUETOOTH’S UUID. ....................................... 25 vi.

(11) FIGURE 21: THE COORDINATORS, RASPBERRY PI............................................................ 25 FIGURE 22: THE BLUETOOTH LOW ENERGY SERVES AS PERIPHERAL. ............................ 26 FIGURE 23: THE COORDINATE HARDWARE, RASPBERRY PI. ............................................ 27 FIGURE 24: THE DIAGRAM OF DETECTING BLUETOOTH CREATED BY JSXGRAPH. ......... 28 FIGURE 25: THE HUMAN MODEL CREATED BY UNITY. .................................................... 29 FIGURE 26: THE DIAGRAM OF BLUETOOTH ON THE FLOOR IN THE 650 * 200 CM. .......... 29 FIGURE 27: THE ERROR RATE OF BLES. THE ERROR VALUE SHOWS THE DIFFERENCE FROM REAL POSITION TO DETECT POSITION. THE 21 BLES IS THE MORE PRECISE THAN OTHERS.31. FIGURE 28: THE ERROR RATE OF BLES. THE ERROR VALUE SHOWS THE DIFFERENCE FROM REAL POSITION TO DETECT POSITION. THE 21 BLES IS THE MORE PRECISE THAN OTHERS.31. FIGURE 29: THE RESULT OF USING FILTER TO GET MORE STABLE SIGNAL BY ONE SIGNAL.32 FIGURE 30: THE RESULT OF USING FILTER TO GET MORE STABLE SIGNAL BY TWO SIGNALS. ...................................................................................................................................... 33 FIGURE 31: THE RESULT OF REAL MOVEMENT FROM LEFT TO RIGHT.(A) LEFT (B)RIGHT (C)MIDDLE (D) LEFT BUT WITH SOME DELAY TIME ......................................................... 33 FIGURE 32: USING ONLY ONE RASPBERRY PI AND IMU TO GET THE POSITION. ............... 34 FIGURE 33: USING THE TWO RASPBERRY PI AND IMU TO GET THE POSITION. ................ 34 FIGURE 34: THE RESULT OF REAL MOVEMENT FROM LEFT TO RIGHT. ............................. 35 FIGURE 35: THE ARCHITECTURE OF THE OVERALL EXPERIMENT. .................................. 36 FIGURE 36: (A) SYSTEM ARCHITECTURE AND SHOW ON THE PERFORMANCE (B) WEARABLE SENSORS WORKS WITH XSENS. ...................................................................................... 37. FIGURE 37: AUDIENCE CAN WEAR OUR SENSORS TO INTERACT WITH VIRTUAL CHARACTERS. ...................................................................................................................................... 38 FIGURE 38: (A) SYSTEM ARCHITECTURE AND SHOW ON THE PERFORMANCE ................. 39. vii.

(12) LIST OF TABLE TABLE 1: POSITION, VELOCITY, AND ACCELERATION. .................................................... 11 TABLE 2: DEPLOY DIFFERENT BLES AND GET ITS POSITION .......................................... 30 TABLE 3: THE TOTAL TIME FROM RASPBERRY PI TO SERVER. ........................................ 36. viii.

(13) CHAPTER 1. Introduction. 1.1 Background In recently years, wearable items grow more and more popular, such as apple Watch, google glass, and so on. With these wearable items, many interesting things can be fulfilled. For example, there are many art performances using virtual characters and various lights to create amazing performance [1]. To perform this amazing show, the performers need a lot of time to practice. It is innovative to let performers wear sensors to make the effect on acousto-optic interaction. In this way, we plan to survey and develop wearable sensors worn by everyone. Many movies use this technology to make animation more live and more expressive, such as Ted, mixed virtual characters acted by human who wears motion capture sensors in real life. Besides, the technology of AR/VR grows more well-developed skill, and the virtual characters are more and more real in 3D animation. It is innovative to use the technology of AR/VR for interacting with interactive storytelling by wearable sensors and makes user feel more reality to move body or rotate arms to control virtual characters.. 1.

(14) 1.2 Motivation Based on pre-made animations and physics simulations, there are many performers playing with virtual characters. In [2], the performer uses his sword to beat the virtual animals on the screen.. This kind of show needs a lot of practice and audience or. novice may not play with or interact with the performance.. If we, somehow, can. capture the motion and position of the performers in real time, then we are able to interact with virtual characters. The performance can be improvisation, and the experience will be more real. Besides, a commercial product Xsens MVN motion capture[3] is utilized for the performance to get users posture, so it is workable to get body signal from wearable sensors.. In this way, we survey many sensors in the. market to see the difference and compare the flexibility from Xsens. The location of the user is important issue in the performance, but there are seldom researches about how to use wearable sensors to detect user’s location in real time. Therefore, this paper presents the algorithm about the detection of user location from IMU and wireless sensors. Finally, using this localization technology in interactive performance is my study in these researches.. 1.3 Purpose Global Positioning System (GPS) is universal tool to localize in the world, but there are still many limitations.. We can use GPS location to know where we are in. global by satellite navigation, but how about in the building?. It. can. know. which building or in which floor, but it can’t find where you are in the room.. in. In. these case, we can use wireless sensors such as Wi-Fi , Zigbee, and Bluetooth to locate users position by Received Signal Strength Indicator (RSSI). 2. I also try many.

(15) different ways such as acceleration from human body and rotation between legs, but there are some problems which can’t overcome. Finally, we adopt Bluetooth low energy as the way to localize. Here are some reasons we consider to choose the Bluetooth.. First, the cost of Wi-Fi is so expensive to deploy and its battery life is. worse than Zigbee and Bluetooth. There are also many wifi AP in the environment, so it may interrupt the experiment that’s why I don’t adopt. Another, the Zigbee is not so universalness that cell phones don't equip with and its transmit rate is slower than Bluetooth, although its effective transmition range is longer than Bluetooth. All in all, the Bluetooth Low Energy, which can minimize the energy, is suited for wearable sensors, so we use Bluetooth Low Energy to get user’s position and transmit signal.. One of BLE’s most feature is more precise when closing to the. BLE, so I can roughly know where you are when getting closer to the BLEs.. The. signal range of BLEs is from 0.01m to 10m, and be more precise inner 0.3m especially.. However it is not stable and precise on the performance, and it still. suffers from update rate since it would be 25 fps at least in the performance.. For. this reason, we propose the algorithm and use IMU sensors to improve the frame rates.. 1.4 Organization The rest of the thesis is organized as follows. Chapter 2 introduces the overviews of the related works, including the Indoor Localization, Gait Analysis and the Inertial Sensor Measurements combined with RSSI. algorithms and main features of the methodology. and discusses of the simulation results.. Chapter 3 shows the details of Chapter 4 presents the analyze. Finally, the conclusions and future works. are presented in Chapter 5. 3.

(16) CHAPTER 2 Related Work. Localization techniques can be classified into Indoor and outdoor environments. GPS could be classified on the outdoor localization, because GPS performs well in outdoor open–sky environments but tall buildings prevent users from getting a line-of-sight to GPS satellites.. So its accuracy degrades in urban environments.. One of the limitations of GPS is the Network, for users need to connect to the servers. To get more precise, GPS needs more power to localize, so the battery is another issue.. On the contrary, indoor localization has become more and more popular in. recently years.. It always uses wireless technologies like Ultra-wideband (also. known as UWB), WLAN.. UWB is a radio technology pioneered by Robert A.. Scholtz[4] and others which used at a very low energy level for short-range.. In. addition, UWB localization is triangulation like TOA (time of arrival), and TDOA (time difference of arrival).. Triangulation uses the geometric properties of triangles. to estimate the target location and needs line of sight between transmitter and receiver. In this way, obstacle will cause inaccurate and UWB’s transmission distance is too short, so we need to use a lot of UWB tags.. On. the. other. hands,. WLAN. (include BLE) localization method is more popular for indoor localization by comparing the signal strength between users and the tags we placed. At. the. beginning, Bluetooth Low Energy (BLE) started as part of the Bluetooth 4.0 Core 4.

(17) Specification [5] in early 2010.. BLE is a dual-mode device compatible with both. Classic and low energy peripherals as Figure 1.. Figure 1: The specific and mode of Bluetooth and BLEs. Picture originated from: [5] In addition to low energy cost, BLE can be feasible to many devices such as phones, headphone and Keyboard. BLE also provides many services to transfer data in GATT protocols. Therefore, after surveying many sensors in the markets, we adopt BLEs as the source and use it on localization.. 2.1 Indoor Localization There are many theories about how to localize, and we use RSSI from BLE to localize human trace. Honggui Li [6] proposes the method how to use the RSSI to localize by low-cost 3D positioning with Bluetooth. The proposed method also has the merits of low cost, low power consumption, high usability and high location precision. Besides, the nonlinear least square used to calculate the user’s location in 3D indoor position. The least square is a basic model, so there are many different models to improve the precise.. The Weighted and Generalized Least Square. algorithm and the Weighted and Extended Total Least Square are also the different solutions which combine the weights to improve from the least square. The four Bluetooth low energy devices are uniformly deployed on the four 3D 5.

(18) positions of a “cubic” room, the four reference devices are placed on four camera stands, and fall in a 9m×9m square with different height as the Figure 2.. Figure 2: The sketch map of the experimental environment. Picture originated from: [6] Experimental results of Location estimation based on weighted least square algorithm are as the following.. The average and standard deviation of absolute precision of. location estimation for hardware experiment is 2.2739m and 1.445m. Another issue is which place is much better to put beacon. Sudarshan S. Chawathe [7] proposes the placement method about where to put beacon indoor. The maximum density of beacon is limited on the budget and deployment effort. We need to know which places are interesting locations, then using algorithm to deploy our beacon in the area. The most important problem is the processing time for receiving the Bluetooth. Bluetooth-based localization method is use the device 6.

(19) discovery which is inherently a slow process, so the discovery process prescribed by the Bluetooth standard requires a total of 10.24 seconds for completion. The response time is the problem that’s I need to overcome.. 2.2 Gait Analysis Bogdan Muset et al.[8] proposed the method how to use accelerometer and gyroscope sensors to get human moving distance as the Figure 3. By using the rotation from legs, I can roughly calculate the distance human move. Although this algorithm can only calculate particular motion pattern, it can improve the frame rate from localization. The update rate of accelerometer and gyroscope is about 33 frames, which is more than beacons. With this paper’s algorithm, the frame rate and precise of localization can be more enhancement.. Figure 3: The algorithm of human moving distance. Picture originated from: [8] Jia Ming Deng et al.[9] also proposed the method how to use accelerometer and gyroscope from IMU sensors such as MPU6050 to implement tracking system. By 7.

(20) eliminating the gravity of accelerometer, it can work for tracking and has many advantages, such as low cost, small size, high reliability, and strong anti-jamming capability.. Jia Ming Deng et al also ensure that the IMU sensors – MPU6050 can. work for tracking and motion capture. In this way, I adopt the MPU6050 as the IMU sensor, which provides many libraries to use such as the pitch, roll, and yaw to detect the movement from human.. 2.3 Inertial Sensor Measurements and RSSI Gaddi Blumrosen et al. [10] propose the way to trace human body by RSSI and Inertial Sensor Measurements.. The main aim of this paper is to demonstrate the. potential of utilizing RSSI in measurement of human movement such as hand movement tracking as the Figure 4.. Besides, the feature of this paper is use. the aggregation of RSSI and IMU data, and the raw data of RSSI filtered by Complementary Kalman Filter (CKF), which can improve the tracking accuracy by almost 50%. The RSSI based location mean and standard errors were 6.76 and 3.11 cm. In addition, the location estimations based on the Kalman filter in steady state, are lower by around 30%, with a mean and standard error of 4.13 and 1.96 cm. According to this paper, I believe it is workable to use RSSI from BLE and combine with Inertial Sensor Measurements by other filter to improve the precise in localization.. 8.

(21) Figure 4: Hand movement motion tracking in 2D experiment setup. Picture originated from: [10]. 9.

(22) CHAPTER 3 Localization Algorithm We want to locate user’s position in the building. At the beginning, the IMU can calculate the distance from the movement of the human body, but I meet problems about cumulative errors from sensors. Because the limitations of hardware from IMU sensors still can’t overcome, I try other ways to get user’s position.. The RSSI. from BLE is another method to localize user’s position without cumulative errors. Therefore, this chapter will introduce the methods used to localize users in section 3.1.. The algorithm to aggregate Inertial Sensor Measurements and RSSI. will present in section 3.2.. However, some problems may occur during the. performance, so the fault tolerance is important issue and will present in section 3.3.. 3.1 System Environment First, I use the acceleration from IMU system to get distance by integrating. However, it suffers from the cumulative errors seriously. from legs to simulate the movement from human. But,. Then, I use the rotation the. patterns. of. human. movement are so complex that I must find other ways to enhance the localization. Finally, the RSSI from BLE is a workable method and it can detect human without cumulative errors.. This chapter presents three steps to localize user. position and provides some fault tolerance to robust the system.. 10.

(23) 3.1.1 Acceleration At first, there are many sensors with accelerate can be used. When I get the signal from sensors, I found the gravity is a very serious factor in computing distance. As we know, the relation between acceleration, velocity, and position as following:. Table 1: The formula of the position, velocity, and acceleration. Derivative Form. Integral Form,. r �. Position. Velocity. v � =. Acceleration. r � = � + ∫ � �′. � �. v � =� +∫. � � = � �. a � =. �′. vx(index) = vx(index-1) + ax(index)* time;. (1). distx(index) = distx(index-1) + vx(index-1)* time + 0.5*ax(index)* time^2;. (2). With these formulate, I try to use the acceleration from sensor as followings: The acceleration signal can be modeled as: �. Where. =. �� +. �� ,. �,. �. +. and. ��. (3). �� are the angular velocity vector, the related angular. velocity vector bias, and a white noise. To reduce the noise, I try to use the Kalman filter [11] to get more precise. which I want to filter out.. However, the gravity from earth is another factor, Therefore, we can get the acceleration without gravity. by rotating the axis as the Figure 5.. The gravity from earth is fixed, but the sensor. may rotate to get partial value from gravity. In this way, by rotating the axis can make the gravity on the virtual Z axis, named as the 11. ′. as follows..

(24) (a). (b). (c) (d) Figure 5: The rotation axis of the acceleration. (a) original axis (b) rotation axis(c) y-z plane (d) x-z ′ plane.. In y-z plane (c), by rotating φ makes partial gravity of y focus on the new � ′ . φ = tan− ′. = √. �. (4). �. +. (5). In x-� ′ plane (d), by rotating θ makes partial gravity of x focus on the new � ′′ ,. which gets all gravity.. 12.

(25) θ = tan−. �. �′. = tan−. �. (6). √� +�. To verify the distance which was calculated by the acceleration after filtering the gravity, I use the toy car to run 20 cm as the Figure 6 and the distance which calculated from acceleration using low pass filter can be precise as the Figure 7.. Figure 6 : The toy car move 20 centimeters from right to left at the Plane. Figure 7: The acceleration, velocity, distance, yaw, pitch and roll of the toy car is right, and the distance is about 20cm. However, human may move by various ways in real life. 13. The accumulative errors.

(26) from sensors are the critical defect as the Figure 8.. The distance integrated from. acceleration may grow up even the sensor holds still. Owing to moving suddenly to other sides such as the roll or pitch move suddenly, the gravity may be calculated to get wrong velocity and distance.. Figure 8: The experiment of moving pitch in 500ms, and the velocity distance will be wrong. Finally, I try other ways and get the more features of MPU6050 sensors from this experience. 3.1.2 Feet Movement The combination of gyroscope and acceleration can determine the rotation from sensors, so by calculating the rotation from legs can be one method to get user’s 14.

(27) position. To get the position from legs, we need to wear the sensors on the legs. The sensors can detect the pitch, roll, and yaw from legs.. By calculating the. angles between legs can roughly calculate the step distance, as the Figure 9.. Figure 9: The distance between legs calculated by rotation. By step and step as the following to calculate the distance from legs, we can roughly know the position from the users. distance = L * Math.sqrt(2*(1 - Math.cos(RightUpLegX_move*Math.PI/360))); (7) Besides, I used the mathlab tools to simulate the rotation between legs of human 15.

(28) as Figure 10 from the scholar of Chia-Feng Lu [12].. However,. the. patterns. of. human movements are so various that it is not easy to get all the patterns of human movements.. Although I can’t use this method to capture the movement from. human, this technique can improve the frame rates and the precise.. Figure 10: The simulation of human movement. Picture originated from: [12] 3.1.3 Beacon Location The RSSI from the BLE can localize the user’s position, but how can we get RSSI from BLE?. The OS of the raspberry pi is based on the Linux, so I find the. API of BlueZ, which provides the library of Bluetooth low energy to get information such as RSSI and the Uuid.. The peripheral, acting as transmitter, can send its RSSI. to let central detect its signal strength. However, the RSSI from peripheral is unstable and drift. I tried to use four different BLE dongles as Figure 11 to receive three different BLE sensors as Figure 12, and found some features of BLE.The RSSI 16.

(29) from BLE is more precise in 30 cm.. The central sensor Esense D704 made from. CSR plc (formerly Cambridge Silicon Radio) is better to receive signal than others, and the peripheral sensor of BOLITEK BLE is more stable to transmit RSSI as Figure 13.. IOGEAR V4.0. SeeHot V4.0. KooPin V4.0. Esense D704. Figure 11: The four different BLE dongles made in Taiwan.. MBTV4. HC – 08. HM-10. Figure 12: The three different BLEs made in China.. 17.

(30) 195. 180. name: MBTV4 165. 150. 120. -50. 135. 5 15 30 45 60 75 90 105. 0 name: MBTV4 name: HM-10 name: HC-08 name: BOLUTEK. -100. 195. name: HC-08 name: MBTV4 180. 165. 150. 105. 135. -50. 120. 5 15 30 45 60 75 90. 0. name: MBTV4 name: HM-10 name: HC-08 name: BOLUTEK. -100. 18.

(31) 195. 180. name: MBTV4 165. 150. 120. -50. 135. 5 15 30 45 60 75 90 105. 0 name: MBTV4 name: HM-10 name: HC-08 name: BOLUTEK. -100. 195. 180. name: MBTV4 165. 120. 150. -50. 135. 5 15 30 45 60 75 90 105. 0 name: MBTV4 name: HM-10 name: HC-08 name: BOLUTEK. -100. Figure 13: The four different BLEs signals received by four BLEs dongles. After getting the RSSI from BLE and understanding the feature of RSSI, the RSSI is more stable in 90 cm, and can be identified in 30 cm for the curve is more skew.. Although the RSSI is unstable, I can get some regular in the RSSI and try. following steps to calculate user’s position. Step 1: First of all, the mapping of RSSI to real distance is as followings, which 19.

(32) provides by the scholar of Sandeep Mistry [13]. Distance =. .5∗. �/� �. ��. −. (8). Which measurePower means the RSSI of one meter. I can get real distance and RSSI to calculate the measurePower as the test data, and use this test data in my experiment to get the rough distance by measuring the RSSI.. To deal with drift signals from RSSI, I use the moving average filter to. reduce the noise from RSSI and improve its identification. However, the RSSI is not stable, so we try the following algorithm to enhance. At the beginning, we use the triangle localization, but it is not precise for some BLEs are not precise. Because BLE can be detected precisely within 30 cm, I sort all the BLE to get the first, second and third nearest BLEs. To use these three BLE efficiently, I give different weights on three near BLEs to improve the precision, where the weights depend on the RSSI of each BLE.. Figure 14: The position we calculated by 8 BLEs. Step 2: After sorting all BLEs and determine three near BLE namely BLE 、BLE 、. BLE ,where BLE is the nearest from the user and so on, then using weighting on 20.

(33) BLE and BLE to get the First_keypoint as Figure 14. First_keypoint = BLE +. BL. BL. * Vector. +BL. to BLE . (9). ,. , where Vector. ,. is from BLE. The First_keypoint can be used as the reference node to combine with BLE . To get more precise, I use more weighs on the BLE to depend user’s position.. The calculation of real position from user is following: Real position = BLE +. Step 3:. BL. BL. + i. _. y. * Vector. i. , i. _. y. (10). i. By this way, we can get user’s position from the RSSI. Due to the fluctuating feature of RSSI, I also utilized the rotation of user leg movement information measured by IMU sensors as Figure 15. That is, I combine the rotation of legs from MPU6050 as the credibility to reduce the errors of RSSI. used to filter the noise and smooth the signals. Position i. a. = Credibility * Position. I. The low pass filter [14] is. The calculation is as follows. + (1- Credibility)* Position. i a−. ,. where Credibility is the change of legs from MPU6050, and range is from 0.5 m to. 1m.. (11). Figure 15: The IMU sensors used to serve as the auxiliary tool. 21.

(34) Besides, I try to use the feet movement, which calculates distance from rotation of legs, to improve the frame rates.When human moves, the position can receive from one of RSSI or IMU.. The RSSI is my major algorithm to calculate position, and. the IMU sensors can add some values between RSSI. Because the IMU sensors may suffer from the cumulative errors, I use the RSSI signals to overcome these errors and combine these two system, INS system and RSSI system, by the low pass filter as Figure 16.. Figure 16: The IMU sensors combine with the RSSI system.. 3.2 Fault tolerance In order to deal with some problems in the middle of performance, I think some problems may occur in the show. How many BLEs should I deploy? It depends on how precise the performance needs. Besides, the frame rates of BLE is only 5~20 fps, how can I do to make sure the frame rates can meet my needs? The sensors may be broken, what should I do to avoid this happen? In this chapter, I will introduce many methods to deal above questions.. 22.

(35) 3.2.1 Deployment At the first, I deployed 8 BLEs to localize the as the Figure 17.. Figure 17: The sketch map of 8 BLEs in the floor. These BLEs can detect user’s position, but the error rate is so high that the position may drift and unstable between middle of BLEs. To improve the correct rate, I deploy 11 BLEs, which I put more three BLEs in the middle of BLEs as the Figure 18.. Figure 18: The sketch map of 11 BLEs in the floor. Second, the 11 BLEs can improve the precise and reduce the errors and noise from RSSI, but it still suffers from drift. In this way, I deploy 21 BLEs in the floor as the Figure 19.. 23.

(36) Figure 19: The sketch map of 21 BLEs in the floor. Finally, there are 21 BLEs in the floor, so the BLEs can detect user position. However, there are some problems about frame rate. Because of the update rates of RSSI is just 5 to 20 frames, I use some methods to solve this problem. 3.2.2 Frame Rate In this chapter, I use some methods to ensure the frame rate can meet my needs. First, the coordinator, raspberry pi, has four USB ports and I use two BLE dongles to receive BLE signals.. If one of BLE dongles breaks, it can use the other. BLE dongle to receive. In addition, I also compare the time of signals from these BLE dongles. In this way, I can compare the time of every signal and get the better signal during the limited time. Second, the BLE may run out of power. To solve these problems, I implement an interface to change or erase the use of BLEs by Uuid as Figure 20.. With. these. interfaces, I can control these BLEs to decide which node I need, and know which BLE is not work.. 24.

(37) Figure 20: The input column of the Bluetooth’s uuid. 3.2.3 Coordinators In the performance the coordinators, Raspberry pi, are responsible for collecting the signal of the Bluetooth Low energy, and transmit to the server. However,. the. coordinators, Raspberry pi, may run out of power, so I prepare two coordinators to backup. In addition, it can also detect BLE signals by four BLE dongles. With two coordinators, I can also detect and compare the user position which is more precise and adopt it.. If one of the Raspberry pi is broken, the other pi can support to make. the performance work successfully as the Figure 21.. Figure 21: The coordinators, raspberry pi.. 25.

(38) CHAPTER 4 Experimental Evaluation In this chapter, the results of the various localizations will be presented and the evolution test and algorithm will show in chapter 4.2. After trying to use the RSSI to trace human movement, I found that the update rate of RSSI is not enough to meet the needs. Fortunately, the IMU sensors can be used to improve the frame rate. By using these wearable sensors, it becomes possible to use the RSSI to detect the user’s position. Therefore, I will also compare the results of only using RSSI and the combination of the IMU and RSSI.. 4.1 Simulation Setup At the beginning, I used little BLE peripherals and only one BLE central to localize. The BLE peripheral is a device which can broadcast the RSSI to let BLE central scan and detect the signals as the Figure 22.. Figure 22: The Bluetooth low energy serves as peripheral. 26.

(39) The BLE central uses the Bluetooth protocols, GATT (Generic Attribute Profile), to scan near devices for connection and each BLE peripheral has its universally unique identifier(UUID) for identification. The coordinator is the raspberry pi 2 model B as the Figure 23. The raspberry pi is a little and light pc, which equips with a 900MHz quad-core ARM Cortex-A7 CPU, 1GB RAM, 4 USB ports, 40 GPIO pins, full HDMI port, Ethernet port, combined 3.5mm audio jack and composite video, camera interface (CSI), display interface (DSI), micro SD card slot, and videoCore IV 3D graphics core.. Figure 23: The coordinate hardware, raspberry pi. When the raspberry pi coordinates all signals, pi needs to send these data to the server. To connect with server, the raspberry pi equips with Wifi dongle to transmit data to server. The Server is a mac pro equips with 2.26 GHz (P8400) Intel Core 2 Duo Penryn with 3 MB on-chip L2 cache, 2 GB (two 1 GB) memory. resource to deal with complex operation.. Finally, the server has more. Therefore, some complex algorithm in. the server can be operated in real time. There is a problem about how to know the results of the localization. To see how 27.

(40) precise of the algorithm and debug results, I use the JSXGraph to create points, circles and lines.. In this way, I can see where the user is as the Figure 24. Besides,. I upload the results to the server. When the server receives the results, other tools such as Unity can use these data to draw many amazing graph as the Figure 25.. Figure 24: The diagram of detecting Bluetooth created by JSXGraph.. 28.

(41) Figure 25: The human model created by Unity.. 4.2 Simulation Results. 4.2.1 Absolute position I try to use three cases in the 650 * 200 � cubic room as the Figure 26.. Figure 26: The diagram of Bluetooth on the floor in the 650 * 200 cm. The following is the results about deploying different BLEs to localize user position 29.

(42) as table 2. The algorithm is in chapter 3, using weighting with each nodes, as Figure 24.. As. the Figure 27 shows, the more BLEs used the more precise I can calculate from user’s location. I put the BLE dongle on the floor to receive signals, the distance from BLE dongle to BLEs are 30, 50, and 100cm. user’s location.. Within 30 cm, I can really find the. In the 21 beacons, I ignore one beacon to make the distance from. BLE dongle to BLE is 100cm.. What the performance needs are not only stable. but also high frame rates. Therefore, 21 beacons were used in performance. Table 2: Deploy different BLEs and get its position. Number of 8. 11. 21. 2m or 1.414m. 1m. BLE Distance. 2m. from BLEs Distance. 100cm. 30cm. 10cm. 100cm. 30cm. 10cm. 100cm. 30cm. 10cm. (250,150). (250,220). (250,240). (250,150). (250,220). (250,240). (250,150). (250,220). (250,240). (263,210). (299,163). (245,171). (257,187). (298,181). (238,199). (256,204). 32 cm. 50.6 cm. 49 cm. 53.4 cm. 57 cm. 15 cm. 36.4 cm. from BLE Real position Calculate. (390,218). (242,189). value Error. 155 cm. 32 cm. Graph. 30.

(43) (. V a E l r u r e o r c m. ). 180 160 140 120 100 80 60 40 20 0. 8顆 11 顆 21顆. 10 cm. 30 cm. 100 cm. real position. Figure 27: The error rate of BLEs. The error value shows the difference from real position to detect position. The 21 BLEs is the more precise than others. The architecture of the flow char is as the Figure 28 shows. There are two ways to calculate the position. One is the BLE, and another is IMU sensor.. Figure 28: The error rate of BLEs. The error value shows the difference from real position to detect position. The 21 BLEs is the more precise than others. After receiving the raw data of the RSSI, I use the algorithm of the filter to reduce the noise and improve the precise by using the IMU in Equation 11. 31. I actually test the.

(44) localization algorithm and walk in the cubic room from 50m to 650m and return. After the experiment, I compare the RSSI signals and use the camera to record my movement. The results of using only one Bluetooth dongle to receive the RSSI and use the low pass filter can solve the drift of the RSSI are as the Figure 29 shows. In addition, the raspberry pi uses the two Bluetooth dongles to receive more RSSI as the Figure 30 shows. In addition, I also use the camera to capture the real movement and the mapping virtual character as the Figure 31 shows.. one signal 600 500 400 raw data. 300. filter 200 100 0 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27. centimeter. 700. time(s). Figure 29: The result of using filter to get more stable signal by one signal.. 32.

(45) two signals 700. centermeter. 600 500 400 raw data. 300. filter. 200 100 0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44. 0 time(s) Figure 30: The result of using filter to get more stable signal by two signals.. (a). (c). (b). (d) Figure 31: The result of real movement from left to right.(a) Left (b)Right (c)middle (d) left but with some delay time This paper focuses on the movement of the user especially the delay time, and the drifts of the RSSI may affect the result. In order to localize the user position, I present the algorithm which take little seconds to calculate the result. The low pass filter is a good solution to filter the noise and just takes 0.08s to get the user position. 33.

(46) Although it may not so precise that error may be 15cm to 57cm, it can follow up the movement of the user.. In addition, I also use the IMU sensors to improve its. precise, so the algorithm can get user’s position in real-time as the Figure 32 and Figure 33.. one signal with IMU 700. centimeter. 600 500 400 300. raw data. 200. filter. 100 0 0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48 50 52 54 56 58 60 62 64 66 68 70. time(s) Figure 32: Using only one raspberry pi and IMU to get the position.. two signals with IMU 700 600. centimeter. 500 400 raw data. 300. filter 200 100 0 0 2 4 6 8 10121416182022242628303234363840424446485052545658606264666870727476. time(s) Figure 33: Using the two raspberry pi and IMU to get the position. 34.

(47) In addition, I also use the camera to capture the real movement and the mapping virtual character as the Figure 34 shows.. Figure 34: The result of real movement from left to right. The architecture of the overall experiment is as the Figure 35.. First,. the. BLE. peripheral sent signal to the Raspberry pi.. Second, the Raspberry pi filters these. signal and sent to the Server by wireless.. Finally, the server process these signal. and sent the real position by message queue. The cost time during the pi to server and server calculate the position from users is as the Table 3.. 35.

(48) peripheral Figure 35: The architecture of the overall experiment. Table 3: The total times from Raspberry pi to server. Step. time delay. From pi to server. 0.23. Server process the siganl. 0.08. Transfer by Message queue. 0.006. 4.2.2 Real performance in Campus We used the Wearable Item Service runtimE (WISE) server as the platform. As shown in Figure 36a, the WISE items are responsible for gathering motion data of the performers and transmitting to a WISE Coordinator via BLE (Bluetooth Low Energy) protocol in real time. Besides, our team used Unity to create virtual characters and let performers wear our sensors to control performance content. As shown in Figure 36b, we performed in our campus (May 19, 2015). One wears Xsens and the other wears our sensors. After the show, every audience can try to wear our sensors to experience (see Figure 37) as he/she is the performer. In addition to the stage show, our team also used AR to create virtual character interacting with real people by wearable sensors (May 20, 2015) (see Figure 38).. 36.

(49) (a). (b) Figure 36: (a) System architecture and show on the performance (b) wearable sensors works with Xsens.. 37.

(50) Figure 37: Audience can wear our sensors to interact with virtual characters.. 38.

(51) (a). (b) Figure 38: (a) System architecture and show on the performance (b) AR coordinate with WISE and let audience wear our sensors to interact.. 39.

(52) CHAPTER 5. Conclusions and Future Work. In this thesis, we present the localization in door and use this technique in the performance. To get less noise in the hardware, I tried many sensors and found best one to use, but there are still some problem about the frame rate less than 20 fps and the drift in RSSI. To solve this problem, I try to use the acceleration to calculate the distance and use the rotation between legs to get the movement. However, there are some problems need to overcome by using these sensors, so I try another way to get the position.. I deploy 21 BLEs and combine the IMU sensors to give the weight in. the low pass filter and reduce the noise to get more precise. Finally, the error value is about 30 cm. Although the error is not better than other scholar, what I care is the delay time. To make the virtual character follow the users the time is very important, so I develop the algorithm to calculate the users’ position in the real time. Finally, the delay time of the final test in the real performance is about 0.3~0.5s without drift.. We performed in the campus twice and joined the Digital Art Center. in Taipei. Besides, we also let audience wear our sensors on the arms to experience the show. In the future, we hope to use this wearable technology to interact with more people in the different places.. 40.

(53) REFERENCES [1]. Britain's Got Talent. Available:. https://www.youtube.com/watch?v=A7IMKWvyBn4 [2]. International event of cool Japan. Available:. https://www.youtube.com/watch?v=nNushriHQ4Q&feature=youtu.be [3]. Daniel Roetenberg, Henk Luinge, and Per Slycke, “Xsens MVN: Full 6DOF. Human. Motion. Tracking. Using. Miniature. Inertial. Sensors.. XSENS. Technologies,”version Apr 3, 2013. [4]. UWB Technical Overview. Available:. https://en.wikipedia.org/wiki/Ultra-wideband#cite_note-1 [5]. 2015 Bluetooth SIG. Available: https://www.bluetooth.org/en-us. [6]. Li, H. (2014). Low-Cost 3D Bluetooth Indoor Positioning with Least Square.. Wireless Personal Communications, 78(2), 1331-1344. [7]. Chawathe, S. S. (2008, October). Beacon placement for indoor localization. using bluetooth. In Intelligent Transportation Systems, 2008. ITSC 2008. 11th International IEEE Conference on (pp. 980-985). IEEE. [8]. Muset, B., & Emerich, S. (2012). Distance Measuring using Accelerometer and. Gyroscope Sensors. Carpathian Journal of Electronic and Computer Engineering, 5(83), 2012. [9]. Deng, J., Qiu, J., Zhong, Z., & Wan, Z. (2015, January). Three-dimensional. Trajectory Tracking System Based on Compass and Gyroscope. In International Conference on Education, Management, Commerce and Society (EMCS-15). Atlantis 41.

(54) Press. [10] Blumrosen, G., & Luttwak, A. (2013). Human body parts tracking and kinematic features assessment based on RSSI and inertial sensor measurements. Sensors, 13(9), 11289-11313. [11] Brookner,. E.. Tracking. and. Kalman. Filtering. Made. Easy. 1998;. Wiley-Interscience: New York, NY, USA, April 1998. [12] Chia-Feng Lu, Mathlab learning : http://www.ym.edu.tw/~cflu/CFLu_course_matlabgui.html [13] Sandeep Mistry masters in Bluetooth services. Available: http://en.gravatar.com/mistrysandeep [14] Ahn, D., Park, J. S., Kim, C. S., Kim, J., Qian, Y., & Itoh, T. (2001). A design of the low-pass filter using the novel microstrip defected ground structure. Microwave Theory and Techniques, IEEE Transactions on, 49(1), 86-93. [15] Jeff Rowberg. “I2Cdevlib. MPU-6050 6-axis accelerometer/gyroscope”. Accessed 28-May- 2014. 2013. URL: USC Viterbi School of Engineering. Archived from the original 2012-03-21. [16] Liu, T., Inoue, Y., & Shibata, K. (2009). “Development of a wearable sensor system for quantitative gait analysis. Measurement”, 42(7), 978-988. [17] Dąbek, P. (2013). “Evaluation of low-cost MEMS accelerometers for measurements of velocity of unmanned vehicles.” Pomiary, Automatyka, Robotyka, 17, 102-113. [18] Takeda, R., Tadano, S., Natorigawa, A., Todoh, M., & Yoshinari, S. (2009).” Gait posture estimation using wearable acceleration and gyro sensors.” Journal of biomechanics, 42(15), 2486-2494. [19] Drezner, Z., Drezner, T., & Wesolowsky, G. O. (2009). “Location with acceleration–deceleration distance.” European Journal of Operational Research, 42.

(55) 198(1), 157-164. [20] Tuck, K. (2007). “Tilt sensing using linear accelerometers.” Freescale Semiconductor Application Note AN3107. [21] Charlon, Y., Fourty, N., & Campo, E. (2013). “A telemetry system embedded in clothes for indoor localization and elderly health monitoring.” Sensors, 13(9), 11728-11749. [22] Čapkun, S., Hamdi, M., & Hubaux, J. P. (2002). “GPS-free positioning in mobile ad hoc networks.” Cluster Computing, 5(2), 157-167. [23] Lee, J. S., Su, Y. W., & Shen, C. C. (2007, November). “A comparative study of wireless protocols: Bluetooth, UWB, ZigBee, and Wi-Fi. In Industrial Electronics Society, 2007.” IECON 2007. 33rd Annual Conference of the IEEE (pp. 46-51). IEEE. [24] Lee, Y. H., Ho, K. W., Tseng, H. W., Lo, C. Y., Huang, T. C., Shih, J. Y., & Kang, T. H. (2014). “Accurate Bluetooth Positioning Using Large Number of Devices Measurements.” In Proceedings of the International MultiConference of Engineers and Computer Scientists (Vol. 2). [25] Chandrasiri, R., Abhayasinghe, N., & Murray, I. (2013, October). “Bluetooth Embedded Inertial Measurement Unit for Real-Time Data Collection for Gait Analysis.” In International Conference on Indoor Positioning and Indoor Navigation (Vol. 28, p. 31st). [26] Wang, Y., Yang, X., Zhao, Y., Liu, Y., & Cuthbert, L. (2013, January). “Bluetooth positioning using RSSI and triangulation methods.” In Consumer Communications and Networking Conference (CCNC), 2013 IEEE (pp. 837-842). IEEE. [27] Liu, J., Chen, C., Ma, Y., & Xu, Y. (2013, September). “Energy analysis of device discovery for bluetooth low energy.” In Vehicular Technology Conference (VTC Fall), 2013 IEEE 78th (pp. 1-5). IEEE. 43.

(56) [28] Baniukevic, A., Jensen, C. S., & Lu, H. (2013, June). “Hybrid indoor positioning with wi-fi and bluetooth: architecture and performance.” In Mobile Data Management (MDM), 2013 IEEE 14th International Conference on (Vol. 1, pp. 207-216). IEEE.. 44.

(57)