Chapter 2 Hardware System and Platform
2.3 Related Performance
2.3.1 National Shanghai Music Festival
國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
(Topic), and the message body. As can be seen from this, the IP address of the receiving end is not seen at all, and the visible length is much lighter.
2.3 Related Performance
2.3.1 National Shanghai Music Festival
In 2017 Oct, Our team was invited to the National Shanghai Music Festival. At that time, it was led by professor Tsai and Tao. That week, there were more than 800 people who enjoy playing our performance. The performance used the equipment include the systems of the first year of the wearable project which contained the computer visual recognition technology to calculate the position of other users within a drift distance by webcams.
Figure 5 National Shanghai Music Festival’s wigwam
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
Figure 6 Member list of our team
Figure 7 National Shanghai Music Festival’s area in wigwam
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
Figure 8 People in the wigwam
At the time, we had a built a wigwam in the outer space and its content a square about 10m * 10m. In addition, there are four devices and VR for each people who had experience time is about 5 to 10 minutes. The experiencer can move freely within the square body composed of three webcams, and finally, the exit is assisted by the staff.
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
Figure 9 Member of Shanghai Music Festival 2.3.2 Tamsui Music Festival
The Tamsui Music Festival is a performance of the exhibition that corped with National Taiwan University and the National Cheng-Chi University Computer and Science team to work together at the warehouse next to the Tamsui MRT station. The static exhibition has a special robot called Pepper and we have the wearable device experience for people. The dynamic exhibition for the robot and the hoster interaction that is the performer wearing the wearable device performance.
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
Figure 10 Static exhibition of wearable device
Figure 11 Interactive between robot and host
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
Figure 12 Member of Tamsui music festival team
2.3.3 High School Promotion
In 2018 spring, we promoted to two schools and provides their students with teaching that they can use devices and the students can have experienced the role of their own drawing into a movable module on the screen and let them write the background. After the demonstrastion, we supported new user interface, DIPS, which include performance time-line controller and the role’s rules which can control the player’s behavior.
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
Figure 13 High school student discussing their idea for the demo
Figure 14 Drawing background of performance
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
Chapter 3 Background
3.1 The inertial positioning system [10] [11]
The inertial positioning system is just a name of positioning methods. There are many method to do it. We mainly focus on the navigation system and the step calculation system. In the past few years, we have used RSSI to assist in calculating the position. At that time, some IMU sensors were used as the receiving node to measure the signal strength, and the IMU's acceleration sensor, Euler angle sensing were used to calculate the actual position of the user.
The figure shows the status of the RSSI to calculate. You can know that the left side represents the current Euler angle which we can calculate it. Furthermore, because of the error in using RSSI to calculate distance, basically how high the accuracy of the sensor will determine the upper and lower limits of the system itself.
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
Figure 15 Two IMU device’s angle range [12]
3.2 Navigation application in flying area
On ship or airplane, the navigation system [9] uses its own heading Euler angle and its own speed to determine which direction to move and distance. When the ship is going to turn a few degrees in that direction, we can use the Triangulation location method to calculate the position of the next moment. It seems to be concise, but there are bound to be a lot of errors if you use a sensor. This is an issue worth us challenging.
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
Figure 16 Navigation application example
3.3 Foot step detection system
Foot step detection system is a system which can use IMU devices detect user is moving or not. The basic idea is to judge gravity accel value, but we can not know that user is forward or backward.
3.4 Position system by using machine learning [13] [14]
There are many ways to locate in front of the article, one of the methods is machine learning as a basis for calculation. In this area when everyone is calling AI, this thing is gradually a trend. Since the algorithm, we are thinking about or the observed rules are not easy to distinguish in some cases, can we use machine learning to calculate it?
Machine learning can be regarded as a classification algorithm in a broad sense, which can also be used to classify an action gesture or a position to take an estimate.
‧
A more common application is to use AI to estimate the user's possible location is.
The training data is usually signal strength, sensor information, reference coordinates, etc...
3.5 RSSI
Now that this wireless communication is very convenience, there are many wireless signal technologies that we can use. It can also be used for signal strength positioning by using wireless signals. RSSI (Received Signal Strength Indicator) is the difference between received signal strength and power in the node. It’s distance formula is [15]
d = 𝟏𝟎(𝒂𝒃𝒔(𝑹𝑺𝑺𝑰−𝑨)/ (𝟏𝟎∗𝒏)) [1]
d: distance
RSSI: signal strength
A: one meter’s signal strength between receiver and sender n: Environment decrease parameter
RSSI positioning is sensitive and it easily influences by signal problems. That why many papers try to solve it in the future or now. So, We will combine more than two positioning technologies and choose one that we discussed the above of the article as an aid.
3.6 Conclusion
The reason why the inertial positioning system is difficult to be very accurate is that the accuracy of the whole system depends mainly on the original accuracy of the IMU sensor. Today, the sensor is not very good, no matter how you use it will not
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
change the result. By the way, the filter algorithm is either a way of calculating the position and its total performance will not double or increase a lot. In the end, it will only approach a reasonable accuracy.
First, the filter has adjusted the devices first, and then the Newton physics formula is calculated for the distance. This physics formula plays some measurable methods and finally gets the position. But in fact, there are ways to increase its accuracy, which is to increase the information of the sensor, but our paper does not explore this piece, we only focus on using a six-axis sensor to locate it.
‧
Chapter 4 Related Research
4.1 Related filter algorithm or method
4.1.1 Fast Fourier Transform
The Fast Fourier Transform [16] is a way to quickly calculate the discrete Fourier transform (DFT) or inverse transform of a signal sequence in the signal era. Fourier analysis converts the signal from the original domain (usually time domain) to the representation of the frequency domain. But our signal is not regular. You can explain it by using a formula.
FFT’s formula is defined below
𝑿𝒌=∑𝑵−𝟏𝒏=𝟎𝑿𝒏𝒆−𝒊𝟐𝝅𝒌𝑵𝒏 k=0,……….N-1 [2]
K: signal sequence E: phase
Before we use the Fourier transform, we need to know that our signal sequence is a power term of 2 or not, and the sampling frequency must be greater than the original frequency.
After we use it, then we can get combinations of real and imaginary numbers.
After the result. We can learn about certain messages by observing the characteristics of the frequency domain. Figure 2.3.1 shows the result of fast Fourier transform on a random signal. In most cases, we calculate the so-called Amplitude as a decibel. The upper graph in Figure 2.3.1 shows the spectrum generated by fast Fourier transform (composed of many frequencies), the energy distribution from 0 HZ to 50 HZ, and the
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
middle graph shows the phase change of the entire signal at this frequency. The result of directing the fast Fourier transform directly corresponds to the result of the positive and negative frequencies, which is finally symmetrical.
Figure 17 Fast Fourier Transform
4.1.2 Moving Filter
The moving filter method [17] is basically an easy and effective method, which can make the waveform of the entire sensor smoother. In more detail, the prediction using the moving average method can smooth out the impact of sudden fluctuations in demand on the prediction results. The moving filter formula is defined below
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
𝑭𝑻 = (𝑨𝑻 + 𝑨𝑻−𝟏 + ….+ 𝑨𝑻−𝑵)
𝑵 [3]
𝐴𝑇: Signal’s time sequence N: Period of sliding window 𝐹𝑇: New signal sequence
But this method will have some problem：
1: if you increase the period number, the result will be not sensitive.
2: It can not show the result currectly in the future.
3: It needs lots of data in the past.
Figure 18 Accel signal and new signal by using moving filter
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
4.1.3 Complementary Filter
Basically, the complementary filter [18], as it named, uses a gyroscope inside the IMU sensor to merge with the angle that uses acceleration to produce a relatively stable value. The integral error of the gyroscope can be eliminated. More importantly, this method is easy to implement in the program and has the flexibility of adjustment. The complementary filter formula is defined as
𝑨𝒏𝒈𝒍𝒆𝒏𝒆𝒘 = 𝒂 × (𝑨𝒏𝒈𝒍𝒆𝒈𝒚𝒓𝒐) + (𝟏 − 𝒂) × 𝑨𝒏𝒈𝒍𝒆𝒂𝒄𝒄, 𝟎 < 𝐚 < 𝟏 [4]
𝐴𝑛𝑔𝑙𝑒𝑛𝑒𝑤: New angle calculate by formula 𝐴𝑛𝑔𝑙𝑒𝑔𝑦𝑟𝑜: Euler angle from gyro
a: constance
𝐴𝑛𝑔𝑙𝑒𝑎𝑐𝑐: The angle calculate by accel value [19]。
Figure 19 Signal update by using complement filter
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
4.1.4 Gravity Remove
In many wearable papers, they do not mention whether the sensor is a gravity acceleration sensor or an acceleration sensor, and there is no mention of whether to filter the gravity. First of all, without considering the gravity acceleration sensor, the information obtained by the sensor should be
𝑨𝒄𝒄_𝐕𝒂𝒍𝒖𝒆𝑺𝒆𝒏𝒔𝒐𝒓 = 𝑨𝒄𝒄𝒆𝒍𝑽𝒂𝒍𝒖𝒆+ 𝑮𝒓𝒂𝒗𝒊𝒕𝒚𝑽𝒂𝒍𝒖𝒆+ 𝑵𝒐𝒔𝒊𝒆 [5]
But we just want the value without noise
𝑨𝒄𝒄_𝐕𝒂𝒍𝒖𝒆𝑺𝒆𝒏𝒔𝒐𝒓= 𝑨𝒄𝒄𝒆𝒍𝑽𝒂𝒍𝒖𝒆+ 𝑮𝒓𝒂𝒗𝒊𝒕𝒚𝑽𝒂𝒍𝒖𝒆 [6]
As you can see from the above formula, we should remove the information from the second half to the zone, leaving only the value of the required acceleration. It will be good for calculations. In addition, in fact, gravity is also unnecessary. The removal method is as shown in the figure below.
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
Figure 20 How to remove gravity on sensor
4.2 Related Research paper
4.2.1 Method 1
In the paper "Real-time Relative Directional Positioning Using Wearable Devices"
[20], it is mentioned that due to the space system of the sensor and our earth space system, in reality, there are two different coordinate systems, if you want to calculate the position, it is necessary to convert the space system. The coordinate transformation matrix is
c: cos, s: sin
‧
The three-axis acceleration send by the sensor is multiplied in the form of a matrix and the space transformation matrix to obtain the converted acceleration matrix, and then the Newton formula is used to calculate the position.
Table 2 Newton Formula
Derivative Form Integral Form,
Position r(𝑡)
In the paper” Full-futured Pedometer Design Realized with 3-Axis Digital Accelerometer” [21]. They mention that if you want to use foot step judge system, need to use another method to judge it. They caller their method “activity axis method”, and also show that human’s moving style:
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
Figure 21 Human’s walking pattern
Figure 4.5 shows the alteration mode of the acceleration of the vertical axis and the parallel axis of the sensor when a normal person walks. Usually, the acceleration will increase or decrease to form a regularity, so they put forward the concept of the most active axis. The most active axis is because the sensor must change dramatically in the direction of the force applied, as shown in the following figure:
Figure 22 Activity accel value
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
However, they define a dynamic threshold that equal to the maximum acceleration plus the minimum acceleration and divide by two. In addition, it will update its value to achieve its purpose. The figure below shows how they are calculated
Figure 23 Their dynamic threshold algorithm
4.3 Conclusion
Method 1 is not perfect for sensor processing and no controller can limit its sensor calculations, resulting in only knowing the approximate position. The advantage is that because the conversion matrix of the space transform was used in the paper and most of Newton's research is used, there is little concern about it.
Method 2 because they do not have to do positioning algorithm themselves, but it is more like a step counter software and calculates calories and distances, etc. The disadvantage is that the way of positioning is less than the advantage is that there is a way to distinguish the active axis to calculate the distance. Therefore, we will integrate the information we can use and re-develop the algorithm regarding the literature.
‧
Chapter 5 Paper Method
The method we use is to fit the IMU sensor near the ankle and use the swinging of the foot and the angular rotation to determine the direction of movement. First of all, we must know that relying on the IMU sensor to get the step or the forward and backward is basically a certain misjudgment. The reason is that the time from the start of the human foot to the end of the displacement cannot be accurately grasped, and almost 90% of the information is judged. In the IMU sensor itself, this is the source of misjudgment.
5.1 Method 1
First, this method is suitable for time sampling to calculate steps and movements.
We will pass the collected sensor data into the displacement module in the JSON data format. In addition, we will perform preliminary filtering on the data of the gyro Euler angle and acceleration, and then use the coordinate transformation matrix to obtain a new one. The coordinate system is the information that needs to be calculated.
One of the fast Fourier transform methods for filters has been mentioned in the previous section, which we use in judging the step. Because the fast Fourier transform basically converts the signal from the time domain to the frequency domain, for the sensor signal, the acceleration signal is converted into the frequency domain. However, this paper does not use it on the filtered signal, it will be used to determine whether the section has a stepping movement.
‧
Function 1 : FFT_Step_Function 𝐈𝐧𝐩𝐮𝐭: 𝐴𝑐𝑐𝑒𝑙 𝑣𝑎𝑙𝑢𝑒, 𝐴𝑥𝑖𝑠
𝐎𝐮𝐭𝐩𝐮𝐭: 𝑇𝑟𝑢𝑒 𝑜𝑟 𝐹𝑎𝑙𝑠𝑒.
1. 𝐷𝑜 𝐹𝐹𝑇
2.For each axis 𝒄𝒂𝒍𝒄𝒖𝒍𝒂𝒕𝒆 𝒂𝒗𝒆𝒓𝒂𝒈𝒆 𝒓𝒆𝒂𝒍 𝒑𝒂𝒓𝒕 𝒗𝒂𝒍𝒖𝒆 𝒘𝒊𝒕𝒉𝒐𝒖𝒕 𝟎𝑯𝒛 𝟑. 𝒊𝒇 𝒂𝒗𝒆𝒓𝒂𝒈𝒆 𝒗𝒂𝒍𝒖𝒆 𝒈𝒓𝒆𝒂𝒕𝒆𝒓 𝒕𝒉𝒂𝒏 𝒕𝒉𝒓𝒆𝒔𝒉𝒐𝒍𝒅
4. return True (Step success)
5.else
6. return False(Step not success)
The fast Fourier transform itself requires the input of the length of the 2 power terms, and the normal start of the human pace is usually between 0.5 and 1.2 seconds, so we set the length of the sample to 64 (sensor data transmission) The speed is 100 strokes per second, which is about 0.64 seconds. It will be judged once by the fast Fourier transform. Then, the 64 pieces of data will have the so-called imaginary and real values through the converted data. We will add the real part of the data except for 0Hz (because 0Hz is the value of DC).
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
Figure 24 Method 1 Flow Diagram Algorithm 2 : Moving Algorithm
Input: Accel Value , Euler Angle(from Sensor collector) Output: Position Information
1. Collect Sensor Accel , Euler Angle 2. Do Axis Transform
3. Storage new data with array 4. If array length greater than 64 5. If Function1 is True
6. For each axis accel value
7. Calculate velocity and position 8. Send result
9. Clean all array
‧
multi-determined, and the position error is too large. At that time, our standard was that when the user wore the device, it was free to sway and move around. When the user finally walked to the original position, the calculated position of the program could not deviate more than 30cm. But this is just one of the reasons we have abandoned the use of acceleration to calculate the position, because the error is too large, and this is the value calculated after filtering. So we turned to think that if the distance of each movement is fixed, it will cause the error to leave only the Euler angle and the sampling pace. So now the moment is to say, we use machine learning to get this thing done.In general, our signals are time series, and because it is impossible to know the current movement status through each point. So we use Method 1 as a base to make changes.
First, we use Method 2 as the base, as long as the average FFT of each axis is recorded every 0.64 seconds. The idea is that the FFT can be thought of as the energy distribution of the signal, and the average value continues to help us determine step. In addition, we use the decision tree to implement a system of direction judgment. The decision tree is used because it can better solve the problem of so-called multiple if....else. Since Method 1 is used as the base, the FFT determines whether or not the pace becomes the main way to collect data. In general, decision trees can make more diverse classifications. When a decision tree is not working, you can use the decision tree forest to solve the problem.
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
First, after positioning the front, back, left, and right, we begin to collect the average FFT energy and the Euler angle of each axis that the person walks forward as the training data, and make the in-situ data a tag of 0, before the first, and then 2, the left is 3 right and 4 is like this. Its format is:
Figure 25 Training dataset
Figure 26 Decision Tree and Result.
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
Figure 27 Method 2 Flow Diagram
Algorithm 3 : Moving Algorithm Mk-ll
Input: Accel Value , Euler Angle(from Sensor collector) Output: Position Information
1.Collect Sensor Data 2.Do Axis Transform
3.Storage new data with array 4.If array length greater than 64 5. If FFT_Step_Function is True 6. DecTreePred(result) 7. Send result
8. Clean all array
‧
5.2 Method 2 Conclusion
According to the basic rules of human walking, sampling the sensor data with 0.64 seconds each time has a great chance to capture incomplete signal information.
Therefore, the final judgment of the movement is usually problematic, and the most common occurrence is the misjudgment of the number of movements of the footsteps.
Because of this, we have abandoned the data collection method based on time sampling and adopted another more reasonable method. In addition, because our demand is in a certain performance space, the displacement error can not exceed 30 cm. Basically, the filters listed in part 4.1 can be effectively filtered. However, in the case of a large number of steps and a long time, it is impossible to keep the error within 30 cm without superimposing, so we finally improve the partial algorithm of the displacement. After that, we try to use CNN which contained three level mode and we used relu , elu, softmax as activation function. This model contained three level mode and we used relu , elu, softmax as activation function to train this dataset, and the result were not as good as we want.
Figure 28 CNN result
‧
Figure 29 Accel waves shape
5.3 Method 3
In method 1, 2 we know that using time sampling will cause a lot of false positives.
In method 3 we use "Cross Zero" to sample the movement of the footsteps. Cross Zero is the number of times the signal value oscillates around 0 in a certain period of time.
We collected a amount of information about the walking information and recorded the actual number of steps taken and the number of steps judged by the computer to determine how much Cross Zero's Threshold is to be taken. In addition, we also refer to the concept of the so-called most active axis, the most active axis can determine the maximum amplitude of the three axes of the sensor. Its algorithm is
Algorithm 4 : The most Active axis
Algorithm 4 : The most Active axis