Chapter 4 Design and Implementation 17
4.2 Software Design and Implementation
4.2.3 Common Sense Semantics
Although dividing the table into cells can reduce the probability of multiple objects on one cell, the situation mentioned in Section 3.2 may still happen. To address this situation, we add common sense semantics to extend the inference routines that can disambiguate the multi-objects on one cell problem. For example, if there are one cup and one plate on the same cell, and the user pours tea from the pot to the cup; the Weight-Change event of {cup, plate} will be reported by the Event Interpreter. According to the common sense, tea should be poured into the cup rather than the plate. Therefore, the behavior inference engine matches the weight-decrease of the pot to weight-increase of {cup, plate} and generates Transfer(pot, cup, w) behavior.
Chapter 5
Experimental Set-up and Results
5.1 Evaluation metric, dining scenarios, and dining set-tings
We have conducted several experiments to evaluate the accuracy of our dietary tracking table under different dining scenarios. The evaluation metric, weight accuracy, measures how well the system can correctly recognize the amount of weight from different food items consumed by the dining participants. It is determined by how well the system can correctly recognize the high-level dietary behaviors: specifically the food transfer event and eat event. Therefore, the intermediate evaluation metric, behavior accuracy, is listed as well.
weight accuracy= Σweight of recognize food intake Σweight of actual food intake
behavior accuracy= # of recognized behaviors
# of actual conducted behaviors
Note that both behaviors are associated with attributes defined in Table 3. The transfer 25
26 CHAPTER 5. EXPERIMENTAL SET-UP AND RESULTS
event has three attributes (source object, destination object, weight), whereas the eat event also has three attributes (user, source object, weight). Correct event recognition is defined as the event’s attributes, except the weight attribute, are correctly identified. Since the weight measurements have inherent sensor errors, they are evaluated separately. Specifi-cally, the behavior accuracy is the number of behaviors recognized divided by the number of behaviors conducted by participants. The weight accuracy is the sum of measured weight divided by the sum of actual weight corresponding to dietary behaviors.
The experiments involve three participants. The first two participants are graduate students from our research team who are familiar with our system. The third participant is a graduate student from our department, who is not familiar with our system.
Dining scenarios (# participants, predefined vs. random activity sequences)
We have designed four different dining scenarios. The varying parameters are (1) the number of dining participants and (2) whether dietary behaviors are predefined or random. As the number of dining participants increases, we expect that they will generate higher number of non-overlapping and concurrent events. Predefined activities mean that the dining participants repeat some pre-arranged sequences of dietary steps which we expect in normal dietary behaviors. The predefined activities may include both sequential and concurrent activities. The exact activity sequences depend on the dining settings described in later subsections. Random activities mean that the dining participants are more or less free to follow their natural eating behaviors within the assumptions of our system defined in Section 3.4.
Dining Settings (afternoon tea vs. dinner)
There are two dining settings: an afternoon tea setting and a Chinese-style dinner
5.1. EVALUATION METRIC, DINING SCENARIOS, AND DINING SETTINGS 27
(a) (b)
Figure 5.1: Afternoon tea scenario showing the placements of table objects and partici-pants
(a) (b)
Figure 5.2: Chinese-style dinner scenario showing the placements of table objects and participants
28 CHAPTER 5. EXPERIMENTAL SET-UP AND RESULTS
setting. The dinner setting is more complex than the afternoon tea setting since it involves a larger number of food containers. We describe these two settings in more details as follows. In the afternoon-tea scenario, participant(s) enjoyed an afternoon tea with a cake, a pot of tea, sugar, and milk. The objects (food containers) on the intelligent table are shown in Fig. 5.1-(a) in page 27, including a tea pot, a cake plate, a sugar jar, a milk creamer, personal cake plates, and tea cups. The personal cake plates and tea cups are placed on the cells in front of each participant. The cake plate is placed on one center cell. The tea pot, the sugar jar, and the milk creamer are placed together on another center cell. Possible high-level dietary behaviors are transferring-cake (to a personal plate), pouring-tea (to a personal cup), eating-cake (from a personal plate), and drinking-tea (from a personal cup). In the dinner scenario, three dining participants enjoyed a sumptuous dinner with three shared dishes, one shared soup bowl, and a shared rice bowl.
The objects (food containers) on the table are shown in Fig. 5.2-(a) in page 27, including these shared plates & bowls, as well as personal bowls located on cells in front of each of three participants. Possible high-level dietary behaviors are transferring-food (to a personal bowl) and eating-food (from a personal bowl). Note that given the weight and type of the food items consumed, it is relatively straight-forward to compute the calorie count by looking up a nutritional table for these food items.
We describe the result for each of four dining scenarios as follows. A summary of their experiment results are shown in Table 5.1 in page 29.
5.2. DINING SCENARIO #1: AFTERNOON TEA - SINGLE USER - PREDEFINED
ACTIVITY SEQUENCE 29
Table 5.1: Experimental results for 4 dining scenarios & their recognition accuracy.
Scenarios Event Statistics Results
tea 1 Predefined 73 12 100%
-#2 Afternoon
tea 2 Predefined 162 24 100%
-#3 Afternoon
tea 2 Random 913 78 79.49%
-#4 Chinese style
dinner 3 Random 1811 162 83.33% 82.62%
5.2 Dining Scenario #1: Afternoon Tea Single User -Predefined Activity Sequence
The first dining scenario involves the afternoon tea setting, single user, and predefined activity sequence. The predefined activity sequence is consisted of the following steps:
(1) cut a piece of cake and transfer it to the personal plate; (2) pour tea from the tea pot to the personal cup; (3) add milk to the personal cup from the creamer; (4) eat the piece of cake from the personal plate; (5) drink tea from the personal cup; and (6) add sugar to the personal cup from the sugar jar. This 6-steps sequence is repeated twice during the experiment. The results are shown in Table 5.1. Based on our measurements, the dietary behavior’s recognition accuracy (i.e., transfer & eat events) is 100%. This result is expected, given that the predefined activity sequence has been anticipated and tested extensively during our prototyping. In addition, this scenario involves only a single user with no concurrent interactions.
30 CHAPTER 5. EXPERIMENTAL SET-UP AND RESULTS
5.3 Dining Scenario #2: Afternoon Tea - Two users - Pre-defined Activity Sequence
The second dining scenario also involves the afternoon tea setting and predefined activity sequence, but with two users performing concurrent activities. The predefined activity se-quence is consisted of the following steps: (1) A cuts cake and transfers it to A’s personal plate; (2) B pours tea from the tea pot to B’s personal cup; (3) A pours tea to A’s personal cup while B cuts a piece of cake and transfers it to B’s personal plate; (4) A adds sugar from the sugar jar to A’s personal cup while B adds milk from the creamer to B’s personal up; (5) A eats cake and B drinks tea; (6) B eats cake from B’s personal plate while A drinks tea from A’s personal cup; and (7) A pours tea from the tea pot to both A’s and B’s personal cups. This 7-steps predefined activity sequence is repeated twice during the experiment. The results are shown in Table 5.1. Based on our measurements, the dietary behavior recognition accuracy is 100%. This result shows that our table is accurate in recognizing concurrent activities from multiple participants.
5.4 Dining Scenario #3: Afternoon Tea Two Users -Random Activities
The third dining scenario involves the afternoon tea setting and two users, but with random dietary activities. Random activities mean that the table participants are more or less free to perform any impromptu dietary behaviors for 913 seconds over the table within the bound of our assumptions described in Section 2.4. The results are shown in Table 5.1.
Based on our measurements, the recognition accuracy is 79.49%. Table 5.2 in page 31 shows the recognition accuracy for each of the two dietary behaviors. The eat events have
5.4. DINING SCENARIO #3: AFTERNOON TEA - TWO USERS - RANDOM
AC-TIVITIES 31
better recognition accuracy than the transfer events, because they can be directly deduced by personal container’s Weight-Change event.
To determine the causes for the misses in activity recognition, we videotaped the af-ternoon tea scenario. By analyzing the video in combination with the system event logs, we derive four main leading causes shown in Table 6. They are described as follows.
Table 5.2: The accuracy of activity recognition under afternoon tea scenario #3 Dietary Behavior # of Actual Events Recognition Accuracy
Transfer event 41 70.73%
Eat event 37 89.19%
Table 5.3: Causes of miss recognition in afternoon tea scenario #3. There are 78 activities analyzed from the video log. The number of misses counts both false positives and false negatives.
Causes of misses # of misses of
transfer events
# of misses of eat
events Total
(c1) Event interference within the
weigh-ing cell’s weight stabilization time 6 2 8
(c2) Weight matching threshold 2 0 2
(c3) Slow RFID sample rate 3 0 3
(c4) Noise from weighing cell 1 2 3
Total of misses 12 4 16
(c1) Event interference within the weighing cell’s weight stabilization time: for activities such as putting down an object on the table, it takes about 1.5 seconds for our weighing sensor to output a stable weight value. If two events occur on the same cell and their time interval is less than the weighing cell’s stabilization time, our system cannot differentiate these two Weight-Change events. Instead, our system will incorrectly recognize them as a single Weight-Change event. Consider the case
32 CHAPTER 5. EXPERIMENTAL SET-UP AND RESULTS
where user A puts down the tea pot at cell X while user B immediately (within 1.5 seconds) grabs a sugar cube from the sugar jar on the same cell X. There are actually two Weight-Change events of amount (Δw1) and of amount (−Δw2). However, due to two events interfering with each other within the weight stabilization time, our system can only detect one erroneous Weight-Change event of amount|Δw1−Δw2|.
(c2) Weight matching threshold: the current threshold value is set to be four grams to filter out noises in the weight readings from weighing cells. However, in some cases, such as transferring one cube of sugar, this threshold value may still be too large. As a result, it may lead to false weight matching involving unrelated weight transfers of similar amounts. Consider the example that user A is removing a cube of sugar from the sugar jar. This results in a Weight-Change of approximately four grams in the sugar jar. At the same time, user B is transferring food weighted approximately eight grams. Eight grams is twice as much as four grams, but they are still within the weight matching threshold. Therefore, this leads to false weight matching. To address this issue, we may change the weight matching threshold to be a percentage of transferred weight rather than an absolute value of four grams.
(c3) Slow RFID sample rate: we have found cases when a user picks up a cup and quickly puts it down. This interval is less than the amount of time the RFID reader performs one round of reading over nine antennas. Therefore, a Weight-Change event is generated without any corresponding RFID-Presence event. This leads to false inference.
(c4) Noises from weighing cells: although we ask users not to touch the table, some still do during the experiment out of personal habits. This leads to erroneous generation of Weight-Change events.
5.5. DINING SCENARIO #4: CHINESE-STYLE DINNER - THREE USERS -
RAN-DOM ACTIVITIES 33
5.5 Dining Scenario #4: Chinese-style dinner - Three users - Random activities
The fourth dining scenario involves the Chinese-style dinner setting, three users, and ran-dom dietary activities for 1811 seconds. Similar to the third scenario, three participants perform impromptu dietary behaviors within the bound of our assumptions described in Section 2.4. The results are shown in Table 4. Based on our measurements, the recogni-tion accuracy is 83.33%. Note that increasing number of table participants only slightly increases the activity rate. The reason is that as the number of table participants increases, out of politeness they try to go the dishes less frequently to avoid in-the-air conflicts over the dishes.
Table 5.4 in page 34 shows the recognition accuracy (for the transfer and eat events) and weight accuracy for each of dietary behaviors. The weight accuracy is computed as the ratio between the measured and the actual weight transferred or consumed during dietary behaviors. Both the recognition and weight accuracy for the food transferring behaviors are between 80 85%, except for dish A, which is fluid-covered food. The reason for lower accuracy on transferring fluid-covered food is that juices from the fluid-covered food can easily drip from the chopsticks during food transfer (from a very lousy chopstick user). The juice dripping leads to erroneous generation of Weight-Change events with both positive and negative values, causing mismatches in the weight matching algorithm.
Furthermore, the weight accuracy of transferring dish A is low at 68.42%, because these transfer recognition misses can accumulate to a large weight sum. Similar to the afternoon tea scenario, the eat events have better recognition accuracy because they can be directly deduced from the personal container’s Weight-Change event.
To determine the causes for the misses in activity recognition, we videotaped the Chinese-style dinner scenario and analyzed the video in combination with the system
34 CHAPTER 5. EXPERIMENTAL SET-UP AND RESULTS
Table 5.4: The accuracy of the Chinese-style dinner scenario #4
Dietary Behavior # of times Recognition Accuracy Weight Accuracy
Transfer dish A events 19 73.68% 68.42%
Transfer dish B events 29 79.31% 78.75%
Transfer dish C events 23 82.61% 79.19%
Transfer rice events 12 83.33% 81.88%
Transfer soup events 19 84.21% 80.16%
Eat events 60 88.33% 91.23%
Overall 162 83.33% 82.62%
event logs. We derive five main leading causes shown in Table 8. They are described as follows.
Table 5.5: Causes of miss recognition in Chinese-style dinner scenario #4. There are 162 activities analyzed from the video log. The number of misses counts both false positives and false negatives.
Causes of misses # of misses of
transfer events
# of misses of eat
events Total
(c1) Segmented weight-change events 5 0 5
(c2) Eating before transferring food on
personal containers 5 5 10
(c3) Weight matching ambiguity 7 0 7
(c4) Noises from weighing cells 3 2 5
(c5) Slow RFID sample rate 3 0 3
Total of misses 23 7 30
(c1) Segmented Weight-Change events: during a lousy food transfer where a user drops a part of food back into the container or on the table, the weight matching algorithm fails because of the difference between weight change values of the container and the personal plate. In addition, such category also includes a case (which didn’t happen in our experiment) that a user holding a personal bowl in the air and scoop
5.6. DISCUSSION 35
soup from a soup bowl to the personal bowl several times. In this case, the weight matching algorithm also fails because it cannot match a weight decrease with a several weight increases.
(c2) Eating before transferring food on personal containers: this occurs when a user picks up a serving of food from a shared plate. However, before the user completes the transfer to his/her personal plate, he/she eats a bite of food. This violates one of our assumptions in Section 3.4 that eating must come from food in the personal plates. In this case, weight matching method fails to recognize the food transfer event due to the disappearing weight on the intermediate bite. Although the users are told about this restriction, some of them still do it out of personal habits.
(c3) Weight matching ambiguity: weight matching ambiguity occurs when two unrelated Weight-Change events of similar weight values are mismatched by our system. If two people transfer food with approximately the same weight, this introduce weight decrease from two share containers equal to weight increase in two personal con-tainers. The system cannot accurately pair up the source and destination containers in these two concurrent food transfers.
(c4) Noises from weighing cells: the same as (c4) in afternoon tea scenario.
(c5) Slow RFID sample rate: the same as (c3) in afternoon tea scenario.
5.6 Discussion
Our experimental results have shown reasonable recognition accuracy of around 80%, which is at least as good as the accuracy of the traditional dietary assessment methods.
36 CHAPTER 5. EXPERIMENTAL SET-UP AND RESULTS
5.6.1 Methods to Reduce Recognition Misses
Below we proposed some methods to address some of the main causes of inaccuracy from our experimental results, and relax some of the assumptions and restrictions. Note that some of the restrictions can be solved by making simple design changes.
Design (Hardware) Change
In our experiment, users may grab food directly from the shared containers without transferring it to personal containers first ((c2) in Section 5.5). The system therefore cannot tell who has consumed what food. However, this problem can be solved if we tagged personal utensils with RFID tag. If a user picks up food with RFID-tagged utensils, the system could identify which user picks up the food and then regard the interaction as eat interaction instead of a transfer food interaction, if there is no weight match between a source container and a personal container.
There is also Weight matching ambiguity ((c3) in Section 5.5) between concurrent transfer interactions. Again, this problem can be solved if we tagged personal utensils with RFID. Our system can use the RFID-tagged utensils as a complement evidence to the Weight Matching Algorithm to infer a transfer interaction. By sensing a RFID-tagged utensil around when a share container decreases its weight and sensing the utensil around again when a personal container increases weight, a transfer interaction can be inferred
Software Change
In addition, Segmented Weight-Change events (in (c1) in Seciton 5.5) can be addressed by extending weight matching algorithm with Bin packing. Since the current version of Weight Matching Algorithm does only one-to-one weight matching, it can not match segmented Weight-Change events if there is one weight decrease of a share container and
5.6. DISCUSSION 37
multiple weight increases of other containers. Bin packing algorithm can extend weight matching algorithm into one-to-many or even many-to-many matching.
5.6.2 Removing No Cross-cell Objects Assumption
Here, we proposed a method to remove the no cross-cell objects assumption. Since we divide the table into nine cells, a container may be placed across different cells. Multiple containers’ weight change can be identified by the sum of weight change of the cells under them. Sometimes, they may be placed across the same cell. When weight change happens to the cell that has multiple containers crossing on, we can not decide how much weight gain or loss is contributed by which of these containers, because only the amount of total weight changes on the cell is determined.
Weight distribution history belong to one container was updated every determined food transfer event happens. Each time simultaneous food transfer event happen and these two containers cross the same cell, we can take advantage of each container’s non-crossed cell and weight distribution history to derive crossed cell’s weight distribution. Consider the scenario as shown in Fig. 5.3. Assuming containers A and B across the same cell #5 are put on the cell one by one, we can identify which cells are crossed by each container, obtain the weight distribution of each container on its crossed cells, and then record the information into weight distribution history (Fig. 5.3-(a)(b)). After all containers being placed on the table, suppose weight loss are detected on cell #5 and #2 at the same time, we can look up the weight distribution history and find that cell #5 and #2 are crossed by container A. Then the sum of weight loss on cell #5 and #2 is viewed as the weight loss on container A, and weight distribution history is updated accordingly (Fig. 5.3-(c)). If cell #2, #4, and #5 all detect weight loss at the same time, we can find that container A and B have weight change simultaneously. Then we use weight distribution history and
38 CHAPTER 5. EXPERIMENTAL SET-UP AND RESULTS
Figure 5.3: Illustration of proposed method to remove no cross-cell objects assumption.
weight loss on cell #2 and #4 as determinative proportional scale to estimate weight loss
weight loss on cell #2 and #4 as determinative proportional scale to estimate weight loss