Chapter 2 System Design and Processes
2.4 System Processes
2.4.2 Navigation Process
The second part of the proposed guidance system is the navigation process. In Section 2.4.1, we mentioned how we learn about environment through the learning process. Accordingly, we can estimate the position of the vehicle on the environment
map and implement our tour guidance system in the navigation process. First of all, we use the captured omni-image to detect the vertical line-shaped objects in the environment. This process will be introduced elaborately in Chapter 4. Then, by using the learning information and the detected features, the system can localize the current position of the vehicle in the environment map. The detailed process will be introduced in Chapter 5. Next, the system can calculate the position of the building and augment the building information on the passenger-view image. The detailed process will be introduced in Chapter 6. The entire navigation process is shown as a flowchart in Figure 2.8.
Figure 2.8 Flowchart of proposed tour guidance system.
Chapter 3
Learning of Environments
3.1 Ideas of Proposed Environment Learning Techniques
In this chapter, we describe the details of the method we propose to generate the environment map for use in the proposed AR-based tour guidance system. In order to complete the system, we must construct the environment map for use in the navigation phase, which includes the information about the path of the tour, the line feature detected for vehicle localization, and the building information.
The first part of environment learning is the construction of a real-world map. We choose the “OpenStreetMap” to construct our environment map. The OpenStreetMap is an open data commons where peoples can modify the map free on the internet like Wikipedia. We use the real-world map acquired from there as the base of the environment map for this study, and define features of the environment for my system.
In more detail, each feature we define will be marked with an icon on the map. The selected path is also marked on the map. In addition, the system can also show the vehicle position on the map during the tour so that the user can see the map clearly.
Next, we learn the line-shaped features for vehicle localization. In order to make our system more accurate, we have to get more information about the features. In addition, we not only learn the position of each feature on the map but also learn the information of the feature about how the camera on the vehicle can “see” It. The detail will be described in Section 3.3.
Furthermore, we learn the building information for the system to show in the AR image. The building information includes the building name, the building area in the map, the area where the camera can see, and so on.
Finally, we merge all the data of the environment as the environment map that the system can use for navigation in the tour. The detail of environment learning will be described in the following.
3.2 Coordinate Systems Used in This Study
In this section, we will introduce the coordinate systems used in this study, which describe the relations between the used devices and the environment map. The following are the four coordinate systems used in this study.
(1) World coordinate system (WCS): denoted as (x, y, z) as shown in Figure 3.1(a). The origin OW of the WCS, a pre-defined point on the ground, is regarded as the starting position of the path traversed by the vehicle during the learning and navigation processes.
(2) Camera coordinate system (CCS): denoted as (X, Y, Z) as shown in Figure 3.1(b). The origin Om of the CCS, a focal point of the hyperboloidal-shaped mirror, lies on the X-Y plane which is coincident with the image plane. The Z-axis coincides with the optical center of the lens of the upper CMOS camera in the omni-imaging device.
(3) Image coordinate system (ICS): denoted as (u, v) as shown in Figure 3.1(c).
The u-v plane of this system coincides with the image plane with the origin OC located at the center of the image plane.
(4) Map coordinate system (MCS): denoted as (Mx, My) as shown in Figure
3.1(d). The MCS is used to represent the environment map. The Mx-My plane coincides with the image plane of the floor. The origin is at the left-top position of the image plane.
Z
Y X
OW
Evironment
(a) X
Y Z Om
Om
(b)
v O
Cu
(c)
Y O X
MAP
(d)
Figure 3.1 The three coordinate systems used in the proposed system. (a) The global coordinate system. (b) The camera coordinate system. (c) The image coordinate system.
3.3 Construction of Environment Map
In this section, we will introduce the method we propose to construct the
environment map. The environment map is like a database, which contains the information that we use in the navigation process.
3.3.1 Information Included in Environment Map
The information put in the environment map includes the real-world map, the navigation path, the vertical lines in environment, the building information. The real-world map is constructed by the use of the “OpenStreetMap” in a website, as shown in Figure 3.2. It is constructed by the geometry and the text describing the buildings. In addition, we define a path on the map using piecewise line segments.
The information of the path includes the positions of the end points of the path and the length of it. Moreover, the features for matching along the path are also learned.
Next, we define the features for each line segment, which includes its position on the map, the relation of the path and the feature, and the angle between the feature and the road. Finally, building information for showing the AR image is learned, which includes the area of the building, the name of the building, the relation between the building and the path.
Figure 3.2 User interface for real-world map construction by use of OpenStreetMap.
3.3.2 Creation of Database for Environment Map
In order to create a database for use by the system in the navigation process, we have to save the data orderly. First, we construct a 2-dimension (2D) environment map by the “OpenStreetMap” and choose a part of the area on the map which is big enough to include the park environment. Next, we set an appropriate scale and a data type for the map. In this study, the scale of the map is 1 centimeter to 10 meters, and the data type of the map is set to be “.jpeg” as shown in Figure 3.3.
Figure 3.3 The real-world map we use in the proposed system.
Furthermore, we save the data of the path into a data structure we designed.
Specifically, we divide the path into some line segments, each being represented by two points. Moreover, the length of any line segment is also calculated and saved in the database. The length distp of each line segment p with end points at coordinates (x1, y1) and (x2, y2) is calculated by the following equation:
2 2 1 2 2
1 ) ( )
(x x y y
distp . (3.1)
In addition, the orientation p of the line segment p is also calculated and saved, which is calculated by the following equation:
)
where the orientation θp means the angle between the line segment and the horizontal direction in the map.
The subsequent major task is to save the information about the features which match with the path. In more detail, we save the number of features matched and the tag of each feature we defined.
Also, the system has to “learn” as well the vertical lines for vehicle localization.
For this, it saves the position of each vertical line found in the environment as a point in the map. Also, while the vehicle is moving on the path, only on a part of the path can be “seen” by the cameras on the vehicle; therefore, for each detected vertical line, we save the orientations in which the first time and the last time the system in the vehicle can detect the vertical line. In other words, we learn the range of the views in degrees in which the vertical line can be seen.
Finally, the system learns the building for showing the AR image. For this, the information of the building we save into the system includes the geometry of the building drawn by lines, the building name, and the tag of the building we defined.
3.4 Learning of Environment Features
In this section, we introduce the proposed methods to learn the information about the environment. The learning process is a necessary step for the system to get ready to run.
3.4.1 Learning of Navigation Paths
A path for car driving in the real world is not just a straight line, but in the study, we use piecewise line segments to describe a path. Furthermore, the system “sees”
each line segment as a unit which includes a lot of information about it. The detail of learning a navigation path is described in the following algorithm.
Algorithm 3.1 Learning of a navigation path.
Input: A real world map.
Output: An environment map with line segments drawn on it as a selected path and the information of the path in a type of data structure.
Steps:
Step 1. Choose a line segment which can be used to compose a desired path by defining the start point and the end point on the map for the line segment.
Step 2. Calculate the length of the line segment by using Equation (3.1) and save it into the data structure of the path.
Step 3. Calculate the angle of the line segment by using Equation (3.2) and save it into the data structure of the path.
Step 4. Add all the features that the vehicle driving on the line segment of the path can “see,” and save the number of the features also, into the data structure of the path.
Step 5. Repeat Steps 1 to 4 until all the line segments of the desired path are chosen.
Step 6. Draw all the line segments as a path on the map.
It noted that the action of adding the features in Step 4 will be described in more detail in Section 3.4.2. After learning the path, the system can use the path data together with other learned data to navigate in the environment.
3.4.2 Learning of Vertical Lines in Environments
The learning of vertical lines includes two parts. The first part is to learn a line feature at a time. The second is for the system to learn multiple line features simultaneously.
It is noted that learning the path should be conducted before learning the vertical line features. Furthermore, the most important step in our system is to calculate the angle of the feature orientation on the map. In more detail, an illustration showing the vehicle on the road is shown in Figure 3.4(a). And an illustration of a detected feature, a light pole, is shown in Figure 3.4(b). We can easily calculate the angle of the orientation of the feature as can be seen from the illustrations. The following algorithm describes the first part of the proposed learning process learning of vertical-line features along the selected path.
θ
Light pole
Car Orientation
(a)
θ
Light pole Car Orientation
(b)
Figure 3.4 The vehicle on a path while detecting a feature. (a) An illustration of the vehicle driving on the path. (b) An omni-image with a detected feature a light pole.
Algorithm 3.2 Learning of vertical line features.
Input: A real-world map with the path information and an omni-image I1 acquired by
with the omni-imaging device on the vehicle.
Output: An environment map with the along-path features drawn on it and the information of the features in a type of data structure.
Steps:
Step 1. Drive the vehicle along each line segment li along the path and detect the vertical lines in image I1 using the feature detection method described in Chapter 4.
Step 2. While the vertical lines are being detected, stop the vehicle and do the following steps.
2.1 Measure the position of the feature on the map using the scale of the map and save it into the environment map by associating the feature with the corresponding line segment li of the path.
2.2 Compute the orientation by which the system detects the feature for the first time, call it the first-angle of the feature, and save it into the environment map.
2.3 Drive the vehicle forward until the feature can no longer be detected by the system.
2.4 Compute the orientation by which the system detects the feature for the last time, call it the last-angle of the feature, and save it into the environment map.
Step 3. Repeat Steps 1 and 2 until the vehicle arrives at the end point of the path.
Step 4. Draw all the features on the map.
Next, learning multiple features simultaneously is different from learning one at a time. The learning of multiple features conducted in this study is a new method. For vehicle localization using multiple line features, we propose to use the longest
common subsequence (LCS) algorithm in this study, which is based on the dynamic programming technique. It enables the vehicle to drive on the path by using the data learned from Algorithm 3.2 and the path information.
In some cases, the vehicle on the path may detect many features at a time like the case illustrated in Figure 3.5(a). And the system has to learn these multiple features by saving all the angles of the features as illustrated in Figure 3.5(b). In more detail, we calculate the orientation of a feature by the following equation:
p g
f
(3.3)
where θp means the angle between the line-segment path and the horizontal direction in the map that is learned by Algorithm 3.1; θg is the feature direction with respect to the horizontal direction in the map; and θf is the feature direction with respect to the
Figure 3.5 Illustrations of multiple feature detection. (a) Illustration of detected feature
on the map. (b) Detected angle of features. (c) Illustration of calculating the angle of θf. We use the detected line features to localize the vehicle by matching the angles of the features using the LCS algorithm to. The details are described in Chapter 5. In the following, the algorithm for learning multiple line feature data to localize the vehicle is described.
Algorithm 3.3 Learning of multiple line features.
Input: A real-world map with the path and feature information.
Output:A table T of the multiple line feature data.
Steps:
Step 1. Start with the first line segment l of the pre-selected path in the map.
Step 2. Start with the first point/pixel of line segment l.
Step 3. Calculate the angle between l and each feature which matches l by do the following steps.
3.1 Choose the feature associated with l whose data are learned from Algorithm 3.2.
3.2 Calculate the feature angle by using Equations 3.2 and 3.3 which setting the point/pixel of the vehicle and the point/pixel of the feature as two end points.
3.3 Save the result into table T.
3.4 Repeat Steps 3.1 to 3.3 until all the features matching l are learned.
Step 4. Move to the next point/pixel along the line segment.
Step 5. Repeat Steps 3 and 4 until the end point/pixel of the current segment is reached.
Step 6. Repeat Steps 2 to 5 until the end line segment of the path is reached.
3.4.3 Learning of Building Information
Since the system must show the building information in the AR image, an algorithm for learning the building is necessary and is derived in the following. An illustration of the algorithm is shown in Figure 3.6(a). The learning of the building corner is shown in Figure 3.6(b). In more detail, we learn the edge line of the building by connecting two corners. And it noted that the edge line we learn is the side which the vehicle can “see”. In other worlds, there is no need to learn the side which can’t be seen when driving on the path. Then, the system can calculate the position of the building while driving on the path by the result of learning the edge line segments.
The details of showing the AR image will be described in Chapter 6.
Car
Figure 3.6 Learning of buildings. (a) An illustration of learning the building. (b)The result of learning the building in the map.
Algorithm 3.4 Learning of building information.
Input: A real-world map image with the path information.
Output:A table of the data of learned buildings.
Steps.
Step 1. Drive the vehicle on the path and detect the building(s) in the acquired image.
Step 2. If a building is detected, stop the vehicle and do the following steps.
2.1 Save the position of the building corner Pi on the map by measuring it and using the scale of the map to calculate the position of Pi on the map.
2.2 Repeat Step 2.1 until all building corners Pi are learned.
2.3 Define a line by connecting every two corners Pi. 2.4 Save the building name in the data structure.
Step 3. Repeat Steps 1 and 2 until the vehicle arrived at the end point of the path.
3.5 Experimental Results
The map of our experimental environment is shown in Figure 3.7. It is the final result of learning all features and the path. In more detail, we will present the results of applying the proposed algorithm in this chapter step by step. First, in the part of learning the navigation path, we defined a path in the environment as shown in Figure 3.8. After learning the path, we have a lot of information about it. Next, in the part of learning vertical line features, we define points of features in the environment as shown in Figure 3.9. Furthermore, the different colors of the features represent different kinds of them, such as light pole and edge line on building walls, etc. After the learning processes are completed, the system can use the resulting information to locate the vehicle. Finally, in the part of learning building information, we show the result in Figure 3.7. All the learned data are saved in the system in many data structures we defined. Before the system starts to run, the data will be loaded into the
system.
Figure 3.7 The environment map we use in the proposed system.
Figure 3.8 The environment map with the path.
Figure 3.9 The environment map with the path and features.
Chapter 4
Automatic Detection of Vertical Lines in Environments with an Omni-camera
4.1 Introduction
In this chapter, we describe the proposed method for detecting vertical-line features in omni-images around the vehicle. Vertical-line features include light poles on street sidewalks or edge lines on building walls. Sometimes, such features can even be tree trunk or some non-artificial object. In other worlds, all the objects with the shapes of vertical lines can be features for use in our system. So our system can match most application environments.
At first, our main idea of vertical-line feature analysis is that a vertical line in real world, when mapped into the omni-image, becomes a radius line in the image. In more detail, the object we see in the omni-image will generally be distorted. But only the vertical line in the omni-image will not be so. So we take advantage of this characteristic and regard the vertical line as a feature for use in our study.
Secondly, a vertical line in the real world is not just a line in the mathematical meaning; in other worlds, it has a certain width and may not be totally straight everywhere on the line. It might be a broken line as well. So it has a lot of problems to be solved before it can be well detected in omni-images by a software program. In this study, we propose many techniques to solve the problems and a method using the