In our system there are two major parts: Hardware and Software. The Hardware Part is mainly about IMU sensors. IMU sensors are hanged on the user body to detect the pose and movement of human body. The Software Part consists of two major phases: Training Phase and Online
Phase. In Training Phase, there are two major steps: data filtering and training pattern forma-tion. Data filtering is to pass filters to reduce noises and errors. Training pattern formation is to form a training pattern by segmenting meaningful part of measurement and convert the data into patterns for further use in online phase. Training Data Formation consist of two steps: pat-tern segmentation and patpat-tern acquisition. Patpat-tern Segmentation is to cut the continuous IMU measurements automatically into meaningful segments. Pattern Acquisition is to convert the meaning segments into several types of training patterns for online use. In online phase, there are three major steps: data filtering, current pattern formation and behavior detection. The first two steps are the same as the training phase but the IMU measurement is from the user using our system for behavior recognition online. For the first two steps, we can obtain the current pattern in the end of the step. The third step is called behavior detection. Behavior detection is to match the training pattern in training pattern database with current pattern. We propose three matching methods including: Pattern-Matching, Longest Common Substring and Longest Common Subsequence. Finally, behavior detection will calculate the trigger similarity accord-ing to our matchaccord-ing methods and the system will trigger the behavior accordaccord-ing to the trigger similarity.
System Architecture
Behavior Trigger
Training Pattern Database Matching Pattern Longest Common Substring
Pattern Segmentation Pattern Segmentation
IMU
Other Filters… Other Filters…
Figure 5.1: System Architecture.
5.2 Hardware: IMU Sensor Measurement
In our system, IMU sensors are placed on the three different place of human body. We put them on the chest, the waist and the foot. The IMU sensor on the chest is put on the front pocket of the shirt. The IMU sensor on the waist is tied on the belt of pants. The IMU sensor on the Foot is stuck on the ankle. We measure the vertical acceleration and pitch readings from the IMU sensors simultaneously. Then, we forward these readings from IMU sensors to our behavior detection algorithm of both elevator detection and stairs detection. In elevator detection, we use the measurement of vertical acceleration, and in stairs detection, we use the measurement of pitch.
5.3 Software I: Training Phase
Training phase is to obtain the features of human behaviors in advance for further detection. We call the features of human behaviors training pattern. After we obtain the training patterns, we add them into the training pattern database. Training pattern database is a database containing the training patterns obtained from the training process. These data represents the behaviors and will be used for matching in online phase. We have two steps in this process: data filtering and training pattern formation. Firstly, data filtering is to do filtering in order to reduce noise, such as low pass filter. Then, we will obtain and segment the measurements into the feature of human behaviors from observation. Finally, we form a training pattern by defining the pattern and the method of transforming into a desired pattern. In the following, we introduce the three steps of Training Processes and explain them in both elevator detection and stairs detection.
5.3.1 Data Filtering
Filtering step is to process the data we obtain from IMU sensors. In this step, we process the data in order to reduce the noise of data and transform the data into the form that will be used in training pattern formation step.
Elevator Detection
In the data filtering step of elevator detection, we process our patterns by low pass filter(LPF).
low pass filter can be addressed as following equation:
a0i := α × ai+ (1 − α) × a0i−1 (5.1) aiis the i-th element before LPF processing. a0iis the i-th element after LPF processing. a0i−1is the (i-1)-th element after LPF processing. α is a parameter controlling the frequency to be cut off. We use low pass filter to filter out the high frequency noise. During our observation, since frequency of walking is much higher than the frequency of elevator’s movement, we can filter out the noise which walking may make as in Fig. 5.2.
Training Phase – Data Filtering
• Low Pass Filter (Optional)
– Eliminate the noise from walking.
a’
i:= α × a
i+ (1-α) ×a’
i-1• Other Filters, such as High Pass Filter (Optional)
α=0.05
Figure 5.2: Low Pass Filter reduces high frequency noises.
Stair Detection
In the data filtering step of stair detection, according to the observation, the stair behavior mea-surement of pitch value is clear enough so that no filtering is needed in stair detection.
5.3.2 Training Pattern Formation I: Pattern Segmentation
Pattern segmentation step is to obtain the useful data for determining specific human behaviors.
In this step, we focus on what measurement should be used and what should be obtained for detection.
Elevator Detection
In the pattern segmentation step of elevator detection, we use the vertical acceleration measure-ment to obtain the all combination of the elevator’s moving pattern. For example, if there’s an elevator which can move from the first floor to the third floor, we then record the up 1 floor pat-tern, up 2 floor patpat-tern, down 1 floor pattern and down 2 floor pattern. During our observation,
the elevator moving up 1 floor will all be similar when the floor height is the same. However, during the case of different floor heights, more patterns will be needed. For example, same as the elevator in the previous example, we need the pattern up from first floor to second floor and the pattern up from the second floor to third floor instead of the up 1 floor pattern in the previous example. We record patterns of every elevator we need to detect.
Stair Detection
In the pattern segmentation step of stair detection, we use the pitch measurement to obtain the features used in stair detection. In this step, we obtain three types of human behavior related to stair detection including walking upstairs, walking downstairs and walking on the floor. During our observation, we find out that the pattern between two local maximum which will repeat when the same behavior is performing. This way, the repeat pattern is the feature we want to gather in this step. To obtain a more general pattern, we will do the same behavior for several times and average them. We will have a deeper look in Pattern Formation step of averaging patterns.
5.3.3 Training Pattern Formation II: Pattern Acquisition
Pattern acquisition step is to define a pattern and describe the method we use to form a pattern.
In order to be used in behavior detection of online phase, we have three type of patterns: full pattern(FP), boundary discrete pattern(BDP) and time discrete pattern(TDP).
Full Pattern
Elevator Detection In pattern acquisition step of elevator detection, we have two type of patterns: full Pattern and half detection parameters. To form a full pattern, we first define a training pattern TBehavior = {a1, a2, ..., ak} with dynamic size of k. Element ai of TBehavior is the i-th measurement of vertical acceleration in the training pattern of specific Behavior. We directly use the measurements from the data filtering step with no further process. To retrieve the half detection parameters, we also use the patterns from data filtering step. However, we use only the beginning concave of patterns of elevator going up and elevator going down since the target of half detection is for early detection without knowing the number of floors moved.
This way, we need to obtain the length LU P, LDOW N and the sum SU P, SDOW N of vertical acceleration of concave up as the beginning concave of elevator going down and concave down
as the beginning concave of elevator going up. We choose the shortest movement of elevator to get these parameters. For example, if there is an elevator which can move from the first floor to third floor. We record the length and sum of the elevator moving only one floor up and down.
This is because during our observation, the concave grows larger when it travels more floors.
This way, we only need to obtain the smallest one as boundary.
Stair Detection To form a full pattern, we first define a training pattern TBehavior = {p1, p2, ..., pk} with dynamic size of k. Element pi of training pattern is the i-th measurement of pitch mea-surement in the training pattern of specific Behavior. We use the meamea-surements from data filtering step. In order to get a generic pattern, we need to average the pattern. To average the patterns, we need to interpolate the patterns of the same Behavior to the length of kmax the longest pattern, and then we average them to generate the desired generic training pattern of the behavior.
Boundary Discrete Pattern
In order to obtain boundary discrete pattern, we need to transfer measurement into discrete way, we first get the maximum value DM AX and the minimum value DM IN of each pattern. Then, we separate the range between DM AX and DM IN into several subrange by discrete factor DF . For example, let DF = 5, we can separate the the value between DM AX and DM IN into 5 subrange. As Fig. 5.3, we define Discrete Level that define the Lowest subrange be 1 and define the Highest subrange be 5 as discrete factor DF . Now for every measurement in the pattern, we can transfer every measurement to discrete level. This way, we can transform every pattern into the discrete form. To form a boundary discrete pattern, we first define a discrete training
Training Phase – Training Pattern Formation
(2) Pattern Acquisition
The # of distinct level between Minimum and Maximum of a pattern.
Figure 5.3: Example of Discrete Factor and Discrete Level.
patternDT PBehavior = {d1, d2, ..., dk}. Element di of training pattern is the i-th measurement
of discrete level in the full pattern of specific Behavior. We use the measurements from data filtering step. Now, we want to obtain a sequence called boundary discrete pattern TBehavior0 from the discrete training pattern. The Discrete Level Sequence Pattern is empty in the begin-ning and we add the first discrete level of a discrete traibegin-ning pattern into the sequence. Then, from the second element to the last element, we only add the discrete level into the sequence when the discrete level is different from the previous one. The detailed obtaining algorithm is described in Algorithm 1.
Algorithm 1 Sequence Obtaining Algorithm of Boundary Discrete Pattern Input: DT PBehavior
if LastInsert 6= dithen Insert di into TBehavior0 .
By Algorithm 1, we can obtain the boundary discrete pattern TBehavior0 of each behavior.
Finally, we put these patterns into the training pattern database for further matching use.
Time Discrete Pattern
To form a time discrete pattern, we also use discrete training pattern introduced previously.
Then we want to obtain a sequence called time discrete pattern TBehavior00 . This time discrete pattern is different from the previous one. We add a time constraint, time lasting factor β to the sequence obtaining algorithm with consideration of time domain. While the previous algorithm insert the discrete level only when the discrete level changes, the algorithm in this method insert the discrete level when the discrete level changes or when the discrete level keep the same at a
certain length of time set by time lasting factor. The detailed algorithm is described in Algorithm 2.
Algorithm 2 Sequence Obtaining Algorithm of Time Discrete Pattern Input: DT PBehavior, β
if LastInsert 6= dithen Insert di into TBehavior00 .
By Algorithm 2, we can obtain the time discrete pattern TBehavior00 of each behavior. Finally, we put these patterns into the training pattern database for further matching use.
5.4 Software II:Online Phase
Online phase is to obtain the current pattern and match with training pattern to determine whether the similarity is high enough to trigger the behavior. We have three steps in this pro-cess: data filtering, current pattern formation and behavior detection. Firstly, data filtering is to filter the noise from the current IMU measurement. Then, we will obtain the current pattern from the current IMU measurement by segmenting the meaningful parts and transfer into useful data structure. Finally, we use behavior detection methods to match current pattern with training patterns. Behavior detection methods will then calculate the trigger similarity and the system
will trigger the behavior according to the trigger similarity. In the following, we introduce the three steps of online process and explain them in both elevator detection and stairs detection.
5.4.1 Data Filtering & Current Pattern Formation
In online phase, data filtering is totally the same with training phase. Current pattern pattern formation is almost the same as training pattern formation but the input measurement of IMU is from the current realtime user measurement for behavior recognition.
5.4.2 Behavior Detection I: Pattern Matching
Elevator Detection
In elevator detection, our methods are to detect the elevator is going up or down and how many floors is the elevator going up or down. In order to detect both, we propose our pattern matching method. However, pattern matching only detects when the elevator completes its movement.
This way, we propose our half detection in order to detect whether the elevator is going up or down in the half way, so that we are able to detect the elevator’s behavior in the middle of the elevator’s movement, but this method is not able to detect how many floors are the elevator moved. Therefore, with the combination of pattern matching and half detection, we are able to detect the direction of elevator’s movement in the half way and determine the number of floors the elevator moving.
Pattern Matching In the pattern matching, we use the full pattern from the training process.
Then, we define a current pattern C = {a1, a2, ..., an} with the constant windows size n, and n is set to the maximum length of training pattern for further matching use. Element aiof training pattern is the vertical acceleration measurement value at time i. For every new input from IMU sensor, we add the newest value to the end of the current pattern, and remove the oldest value in the front of the current pattern.
Finally, we start to match current pattern with training patterns. For every training pattern, we calculate the error distance between current pattern and the training pattern by the following equation:
Let size of TBehavior = k, Ci = ithelement of C, and TBehavior,i = ithelement of TBehavior
For every Behavior,
After we calculate the error distance of current pattern and every training pattern, we can trigger the detection by the trigger boundary(TB). We define trigger boundary T BBehavior as a value for a elevator behavior to trigger the detection.
Algorithm 3 Full Pattern Detection Algorithm Input: TBehavior, C
Output: T riggerSimilarity for every TBehavior do
Calculate ErrDist(C, TBehavior)
if (ErrDist(C, TBehavior) < TBehavior) then T riggerSimilarity = 100%.
else
T riggerSimilarity = ErrDist(C,TTBehaviorBehavior). end if
end for
As Algorithm 3, by using pattern matching, we are able to detect the elevator behavior with direction and number of floors and output the behavior trigger similarity to detection trig-ger. However, the detection can be triggered only when the movement of elevator is complete.
Therefore, we propose half detection method to solve this problem.
Half Detection In half detection, we use half detection parameters from the training process.
half detection parameters include the length LU P, LDOW N and the sum SU P, SDOW N. Now we use the current pattern C from pattern matching. For elevator detection of going up, we use the LU P and SU P. We define U pSU M (C) as following:
Also, for elevator detection of going down, we use the LDOW N and SDOW N. We define DownSU M (C) as following:
After we calculate U pSU M (C) and DownSU M (C), we can trigger by comparing the value of U pSU M (C), SU P and DownSU M (C), SDOW N. If U pSU M (C) is larger than SU P, up detection triggers, and if DownSU M (C) is larger than SDOW N, down detection triggers.
Algorithm 4 Half Detection Algorithm Input: C
Output: T riggerSimilarity if U pSum > SU P then
T riggerSimilarityU P = 100%.
else
T riggerSimilarityU P = U pSumS
U P %.
end if
if DownSum > SU P then
T riggerSimilarityDOW N = 100%.
else
T riggerSimilarityDOW N = DownSumS
DOW N %.
end if
As Algorithm 4, by using half detection, we are able to trigger the elevator detection of up and down in the half way of elevator movement. Therefore, half detection can assist pattern matching to have the early detection while pattern matching provides higher accuracy and the number of floors going up or down.
Stairs Detection
In the pattern matching, we use the training pattern T in full pattern from training process as training pattern. Now we define a current pattern C = {p1, p2, ..., pn} with the dynamic windows size n, and n is set to the length between two local maximum. In order to avoid the constant shift problem caused by the IMU unit, we need to do the shifting to make sure the patterns we are going to match are on the same baseline. This way, we shift the pattern to its average as following equation:
P atternshif t= P attern − Average(P attern) (5.5) Then we can obtain the shifted patterns T and C. The length of each pattern is different, so we need to transfer the patterns into the same length. This way, we do the interpolation to expend the shorter pattern to the same size as the longer pattern. Finally, we can calculate the Error
Distance by the following equation:
Then, we can use Algorithm 3 to output the behavior trigger probability.
5.4.3 Behavior Detection II: Longest Common Substring
In the longest common substring, we use the boundary discrete pattern (BDP) from training process as training pattern. First, we construct a current pattern of boundary discrete pattern C’
as T’ in training process. Now we have two boundary discrete patterns from training process and online process. Then, we use the longest common substring algorithm to calculate the length of longest common substring |LCS|. Then we can calculate the trigger similarity between these two patterns as following equation:
Similar(T0, C0) = 2 × |LCStr(T0, C0)|
|T0| + |C0| (5.7)
By calculating the trigger similarity, we can take this similarity and output it to the detection trigger.
5.4.4 Behavior Detection III: Longest Common Subsequence
In the longest common subsequence [20], we use the time discrete pattern (TDP) from training process as training pattern. First, we construct a current pattern of time discrete pattern C” as T” in training process. Now we have two time discrete pattern from training process and current pattern. Then, we use the longest common subsequence algorithm to calculate the length of longest common subsequence |LCS|. Then we can calculate the similarity between these two patterns as following equation:
Similar(T00, C00) = 2 × |LCSeq(T00, C00)|
|T00| + |C00| (5.8)
By calculating the trigger similarity, we can take this similarity and output it to the detection trigger.
5.4.5 Detection Trigger
Detection trigger collects the trigger similarities from the online phase of each IMU sensors.
According to these data, the detection trigger will process these data with the weighted factors
of each behavior to trigger the final detection behavior. The final detection result will be made in this part.
Chapter 6
Implementation and Experimentation
In this chapter, we will talk about how our design decisions are made. In order to apply the method into the real world, we need to choose parameters and make decisions from tradeoffs.
In this chapter, we will talk about how our design decisions are made. In order to apply the method into the real world, we need to choose parameters and make decisions from tradeoffs.