Chapter 6 Augmented Reality for Navigation
6.4 Algorithm of Indoor Navigation by Augmented Reality
In this section, we summarize the processes described in the previous sections as a total process the process of indoor navigation by augmented reality, as described in Algorithm 6.4 below.
Algorithm 6.4 Indoor navigation by augmented reality.
Input: A scene image.
Output: An augmented image.
Steps
Step 1. Obtain the user’s orientation, and the user’s location from the server-side system.
Step 2. Obtain the pitch angle from the orientation sensor of the client device.
Step 3. Create the projection matrix by the method described in Algorithm 6.1.
Step 4. Obtain visiting target information from the server-side system.
Step 5. Display all visiting targets by Algorithm 6.2.
Step 6. Search the desired destination by a keyword to obtain a planned path.
Step 7. Display the planned path by Algorithm 6.3.
108
6.5 Experimental Results
Figure 6.11 and Figure 6.12 show two results of overlaying visiting target information on scene images. The figure includes an omni-image captured from a fisheye camera, the detected location and orientation, and the augmented image shown on the user’s mobile device. A user can understand the surrounding environment by the visiting target information on the augmented image.
(a) (b)
(c)
Figure 6.11 An augmented image with visiting target information. (a) An omni-image. (b) Detected location and orientation. (c) The augmented image shown on user’s mobile device.
109
(a) (b)
(c)
Figure 6.12 An augmented image with visiting target information. (a) An omni-image. (b) Detected location and orientation. (c) The augmented image shown on user’s mobile device.
Figure 6.13 shows a result of overlaying a navigation path on scene images. The figure includes four augmented images which are at different locations and in different orientations. A user can understand how to reach the desired destination by following the navigation path shown by the arrow and the line segment. When the destination is out of the screen, the navigation path may be invisible in an augmented image. At that time, the system will display the destination on the edge of the screen to indicate the correct direction.
110
(a) (b)
(c) (d)
Figure 6.13 An augmented image with a navigation path. (a)(b)(c) The augmented images at three different locations. (d) When the destination is outside of the screen, the name of the destination will display on the edge of the screen (shown as the yellow stroke text); this image shows that the destination is on the rear of the user.
111
Chapter 7
Experimental Results and Discussions
7.1 Experimental Results
In this section, we will show some experimental results of the proposed indoor AR navigation system. The experimental environment is in the Computer Vision Lab at National Chiao Tung University. The environment map is shown in Figure 7.1, which includes eight visiting targets (shown as green regions) and two fisheye cameras (shown as blue circles).
出口
馬賽克畫
飲水機 博班區
碩一區 碩二區
休息區 Camera-2 Camera-1
電視
Figure 7.1 The environment map of the experimental environment.
112
7.1.1 Result of Real Navigations
A. Browsing visiting targets at a certain location
Figure 7.2 shows a result of browsing surrounding visiting targets at a certain location. The omni-images captured from the fisheye cameras are shown on the left-hand side of this figure, and the augmented images shown on the user’s mobile device are shown on the right-hand side. At first, the user faced the left side of the experimental environment, where we can see two visiting targets displayed on the screen (as shown in Figure 7.2(a)). Then, the user began to turn to the left-hand side, and we can see that the overlaying texts are moving to the right-hand side as the user was turning (as shown in Figure 7.2(b)-(j)).
(a)
(b)
Figure 7.2 A result of browsing visiting targets at a certain location. The left-hand side is the images captured from the fisheye cameras, and the right-hand side is the augmented images shown on the user’s mobile device.
113
(c)
(d)
(e)
(f)
Figure 7.2 A result of browsing visiting targets at a certain location. The left-hand side is the images captured from the fisheye cameras, and the right-hand side is the augmented images shown on the user’s mobile device (cont’d).
114
(g)
(h)
(i)
(j)
Figure 7.2 A result of browsing visiting targets at a certain location. The left-hand side is the images captured from the fisheye cameras, and the right-hand side is the augmented images shown on the user’s mobile device (cont’d).
115
B. Navigation by a navigation path.
In this section, we show a result of navigation according to a navigation path. A user stood at a location as shown in Figure 7.3(a), and the detected location and orientation are shown in Figure 7.3(b). Figure 7.3(c) shows the augmented image seen by the user. Then, the user searched the environment map for a visiting target, and there appeared a yellow stroke text on the right-hand side of the bottom edge of the augmented image (as shown in Figure 7.3(d)). The user could then understand that the destination is on the right rear, so the user began to turn to the right-hand side. As the user was turning, we can see that the destination was moving to the right-hand side of the user (as shown in Figure 7.3(e)). Finally, the user saw the destination and the navigation path when he turned to the correct direction (as shown in Figure 7.3(f)).
Therefore, the user began to follow the navigation path to move. As shown in Figure 7.4, the user faced the left-hand side of the environment to move. When the user moved to the location as shown in Figure 7.4(b), he was closer to another camera of the environment. Therefore, the system shifted to use the other camera to track the user as shown in Figures 7.4(c) and 7.4(d). Figure 7.5 shows the four augmented images corresponding to the four locations as shown in Figures 7.4(a) through 7.4(d), respectively.
116
(a) (b)
(c) (d)
(e) (f)
Figure 7.3 A result of navigation by a navigation path. (a) A user was at a certain location. (b) The detected location and orientation. (c) The augmented image seen by the user. (d) The augmented image shown when the user searched a visiting target, and there is a yellow stroke text shown on the right-hand side of the bottom edge of the augmented image, which indicates the direction of the destination. (e) The augmented image shown when the user is turning to the right-hand side. (f) The augmented image shown when the user is turning to the correct direction.
117
(a)
(b)
(c)
(d)
Figure 7.4 A user following the path shown in Figure 7.3(f) to move.
118
(a) (b)
(c) (d)
Figure 7.5 The four augmented images corresponding to the four locations as shown in Figures 7.4(a) through 7.4(d), respectively.
7.1.2 Result of Precision Measurement
We show a result of precision measurement of human location detection in this section. As shown in Figure 7.6, we chose several locations in the experimental environment, and we let a person stand at these locations and detect the locations by the proposed human location detection method. The result is shown in Table 7.1, which includes the actual locations of the chosen locations which are measured manually and the detected locations by the proposed method. The average error of the computed locations is 15cm, which is small enough for the proposed system to locate a user.
119
Table 7.1 Error of human location detection (unit: cm) Location # (1) Actual (2) Computed
120
Table 7.2 shows another result of precision measurement in the transformation from the ICS to GCS. We measured 11 line segments on the ground, and computed the length by transforming the two end points of each line from the ICS to the GCS.
These lines segments we chose are shown in Figure 7.7. The average error rate is 2.79%, which is small enough and shows that the proposed transformation technique actually works for real applications.
Figure 7.7 Line segments used for the line length measurement.
Table 7.2 Error of line length measurement.
Line #
121
Table 7.3 shows the result of precision measurement for the proposed process of human orientation detection by the color edge mark. We choose six locations in the experimental environment to for this experiment. We let a person stand at these locations, and the person faced towards the mobile device to four directions at each location. Finally, we computed the orientation vector by the proposed method, and the error is the angle between the actual orientation vector and the computed orientation vector. As shown in the result, we can see that the errors are almost below 8o, and the average is below 4o. This shows that the proposed method actually works for ral applications. However, the errors of some cases exceed 8o. The main cause of these higher errors is that the color edge mark will be projected into a small region in the omni-image when the distance between the color edge mark and the camera becomes larger, so it will not always be segmented successfully from the image completely.
1 2
3
5 4
6
Figure 7.8 Locations used for precision measurement in human orientation detection.
122
Table 7.3 Error of human orientation detection.
Location #
123
7.2 Discussions
The experimental results of the proposed indoor navigation system presented previously show that we can utilize fisheye cameras to detect the user’s location and orientation. Meanwhile, a user can understand the surrounding environment and conduct the navigation by the proposed AR techniques.
However, the proposed system still has some problems. As a user is moving far away from a fisheye camera, the detected locations will become more and more unstable. This is because the detected locations are computed by interpolation of four calibration points, but the pixels of farer objects will have higher distortion. Therefore, the actual distance between two neighboring pixels become larger at a far location from the camera, and the error of a few pixels might cause the interpolation result to be inaccurate. A similar problem will occur on the color edge mark detection. As the user is moving away from a fisheye camera, the region of the color edge mark in the omni-image will become smaller and hard to detect. A possible way to solve these problems is to use more cameras in the environment. Therefore, when a user moves away from a camera, it can be detected from another camera which is closer to the user. Furthermore, when a user’s feet are covered by obstacles, cameras may not be able to detect correct foot locations and may result in incorrect detected locations.
This problem can be solved by the same solution mentioned above, which is to use more cameras. Another possible way is to detect the head point of the user, and then we can use the height of the user to estimate the foot location.
Furthermore, our experimental environment is just a small region, so the client-side system can be connected to the server-side system through the Wi-Fi wireless network, which has a smaller access range. However, if we want to apply the proposed system in a larger environment, we might have to use a mobile
124
telecommunication network, such as a 3G or 4G network, to connect the both sides.
The mobile telecommunication network has a larger access range than the Wi-Fi wireless network. Using mobile telecommunication networks can also reduce the costs of building Wi-Fi wireless networks.
Finally, the proposed system can handle only one user at a time. If we want to enhance the capability for multiple user usages, we have to distinguish different users in the environment. A possible solution is to analyze images captured from the mobile device, and detect the features in the images to identify different users at different locations.
125
Chapter 8
Conclusions and Suggestions for Future Works
8.1 Conclusions
An indoor navigation system by augmented reality and down-looking omni-vision techniques using mobile devices has been proposed. To design such a system, several techniques have been proposed as summarized in the following.
1. A modified method for point transformation from an omni-image to the global coordinate system has been proposed, which is modified from a space-mapping technique [16]. The proposed method can provide point transformation for larger pixel region in omni-images than the adopted method, by which we can increase the utilization of the pixels of the omni-image.
2. A method for human localization in indoor environments has been proposed, by which we can obtain a user’s location and orientation in an indoor environment.
The orientation detection algorithm integrates three different techniques to detect the orientation of a user, and each of the techniques can make up the deficiencies of the others.
3. A method for path planning for indoor environments has been proposed, which is based on the analysis of the floor plan drawing of an indoor environment. By this method, the system can provide a navigation path starting from a user’s location to his/her desired destination.
4. A method for indoor AR navigation by overlaying visiting target information on
126
the real objects in scene images has been proposed, by which a user can understand the surrounding environment in an AR way from the overlaying visiting target information.
5. A method for indoor AR navigation by overlaying a navigation path on the floor in scene images has been proposed, by which a user can follow a navigation path shown on the screen and reach his/her desired destination in an AR way.
The experimental results shown in the previous chapters have revealed the feasibility of the proposed system.
8.2 Suggestions for Future Works
According to our experience obtained in this study, several issues and possible extensions of the proposed system worth further studies are listed in the following:
1. Designing a background/foreground separation algorithm which can adapt to different lighting conditions and moving styles of non-human objects.
2. Seeking a solution to the problem of human location detection in the situation that a user’s feet are covered with obstacles.
3. Proposing a method for human orientation detection with higher precision and better stability, which can be accomplished by matching the image captured from the user’s mobile device with a pre-learned database to determine the orientation.
4. Providing the capability for processing multiple environment maps, which can provide human localization in different floors of an indoor environment.
5. Enhancing the system capability for indoor environments with multiple system users, which can distinguish different users in an indoor environment.
6. Including more useful information in an environment map, such as merchandise, food, etc. A user can search by a keyword for what he/she wants rather than just
127
searching for the name of a visiting target.
7. Collecting the images captured from cameras on users’ mobile devices, and using them to establish a virtual environment database, by which users can browse an indoor environment without going there.
128
References
[1] C. Lukianto, C. Honniger, and H. Sternberg, "Pedestrian Smartphone-Based Indoor Navigation Using Ultra Portable Sensory Equipment," in Proceedings of International Conference on Indoor Positioning and Indoor Navigation (IPIN), Zurich, Switzerland, 2010, pp. 1-5.
[2] B. Ozdenizci, K. Ok, V. Coskun, and M. N. Aydin, "Development of an Indoor Navigation System Using NFC Technology," in Proceedings of Fourth International Conference on Information and Computing (ICIC), Phuket Island, Thailand, 2011, pp. 11-14.
[3] L. C. Huey, P. Sebastian, and M. Drieberg, "Augmented Reality Based Indoor Positioning Navigation Tool," in Proceedings of IEEE Conference on Open Systems (ICOS), Langkawi, Malaysia, 2011, pp. 256 - 260.
[4] A. Mulloni, D. Wagner, D. Schmalsteig, and I. Barakonyi, "Indoor Positioning and Navigation with Camera Phones," Pervasive Computing, IEEE, vol. 8, pp.
22-31, 2009.
[5] M. Werner, M. Kessel, and C. Marouane, "Indoor positioning using smartphone camera," in Proceedings of International Conference on Indoor Positioning and Indoor Navigation (IPIN), Guimaraes, Portugal, 2011, pp. 1-6.
[6] H. Hile and G. Borriello, "Positioning and Orientation in Indoor Environments Using Camera Phones," IEEE Computer Graphics and Applications, vol. 28, pp.
32-39, 2008.
[7] S. Henderson and S. Feiner, "Exploring the Benefits of Augmented Reality Documentation for Maintenance and Repair," IEEE Transactions on
129
Visualization and Computer Graphics, vol. 17, pp. 1355 - 1368, 2011.
[8] M. C. Juan, C. Botella, M. Alcaniz, R. Banos, C. Carrion, M. Melero, and J. A.
Lozano, "An Augmented Reality System for treating psychological disorders:
Application to phobia to cockroaches," in Proceedings of the Third IEEE and ACM International Symposium on Mixed and Augmented Reality, Arlington, USA, 2004, pp. 256 - 257.
[9] D. Kalkofen, E. Mendez, and D. Schmalstieg, "Interactive Focus and Context Visualization for Augmented Reality," in Proceedings of IEEE and ACM International Symposium on Mixed and Augmented Reality, Nara, Japan, 2007, pp. 191-201.
[10] K. Jongbae and J. Heesung, "Vision-Based Location Positioning using Augmented Reality for Indoor Navigation," IEEE Transactions on Consumer Electronics, vol. 54, pp. 954-962, 2008.
[11] T. Miyashita, P. Meier, T. Tachikawa, S. Orlic, T. Eble, V. Scholz, A. Gapel, O.
Gerl, S. Arnaudov, and S. Lieberknecht, "An Augmented Reality Museum Guide," in Proceedings of IEEE International Symposium on Mixed and Augmented Reality, Cambridge, United Kingdom, 2008, pp. 103-106.
[12] H. C. Chen and W. H. Tsai, "Optimal security patrolling by multiple vision-based autonomous vehicles with omni-monitoring from the ceiling," in Proceedings of 2008 International Computer Symposium, Taipei, Taiwan, Republic of China, 2008, pp. 196-201.
[13] J. Borenstein and Y. Koren, "The Vector Field Histogram-Fast Obstacle Avoidance for Mobile Robots," IEEE Transactions on Robotics and Automation, vol. 7, pp. 278-288, 1991.
[14] J. Y. Hwang, J. S. Kim, S. S. Lim, and K. H. Park, "A Fast Path Planning by Path Graph Optimization," IEEE Transactions on Systems, Man and Cybernetics, Part
130
A: Systems and Humans, vol. 33, pp. 121-129, 2003.
[15] J. Bruce and M. Veloso, "Real-Time Randomized Path Planning for Robot Navigation," in IEEE/RSJ International Conference on Intelligent Robots and Systems, Lausanne, Switzerland, 2002, pp. 2383 - 2388.
[16] T. Akenine-Moller, E. Haines, and N. Hoffman, "Pespective Projection," in Real-Time Rendering, Third Edition, T. Akenine-Moller, ed., 2008, pp. 92-97.
[17] A. Senior, A. Hampapur, Y.-l. Tian, L. Brown, S. Pankanti, and R. Bolle,
"Appearance Models for Occlusion Handling," in Proceedings of 2nd IEEE Workshop on Performance Evaluation of Tracking and Surveillance, Hawaii, USA, 2001.