CHAPTER 2 Ideas Of Proposed Methods And System Design
2.5 System Processes
2.5.2 Navigation Process
The second part of our guidance system is the navigation process. In Section 2.5.1, we mentioned how we create a guidance map through the learning process. Accordingly, we can estimate the position of the video surveillance car on the guidance map and implement our tour guidance system in the navigation process. First of all, we use the captured omni-images to detect the landmarks at the sidewalk. Then, by using the pano-mapping table, the 3D information of the landmarks can be estimated. The process will be introduced elaborately in Chapter 4. Next, the turning direction of the surveillance vehicle is checked by an optical flow method. We use the turning direction to keep track the current node corresponding to the vehicle on the graph of the guidance map. The
detailed process will be introduced in Chapter 6. After the laptop COMA receives the position of the currently-visited landmark and the vehicle turning direction, we integrate these data to obtain the position of the video surveillance vehicle and get the information of the neighboring buildings. Finally, we need to send the names of buildings to the pad. For this, at first through the AP, we can use the local wireless network to connect the laptop COMA and the pad. The laptop COMA then sends the names of the nearby buildings to the pad. Then, we display the names of the buildings as an image shown on the pad and project the image onto the car windshield. Moreover, we propose a method to estimate the position of the projected images on the windshield. The user can use the method to place the pad conveniently. The detailed process will be introduced in Chapter 5. As shown in Figure 2.11, we can know the flowchart of navigation process.
Figure 2.11 Flowchart of tour guidance
Chapter 3
Creation of Guidance Map
3.1 Introduction
In this chapter, we describe the details of the method we propose to generate the guidance map for use in the proposed augmented reality based tour guidance. Using the omni-camera device and the PTZ-camera device affixed to the roof of the video surveillance vehicle, we can create the guidance map quickly.
The proposed guidance map includes the center points of the landmarks and some feature points of objects of interests in the guidance area. We can use prominent points on any objects, like buildings, lamps, chairs, etc., as feature points. However, in this study, we just choose the corner points of buildings as the feature points because buildings are the most obvious objects in the guidance area and we can get prominent feature points of buildings conveniently in images.
In general, we use a ruler and a protractor to measure manually the distance between the PTZ-camera and every feature point, as well as the orientation of the camera direction with respect to the line from the PTZ-camera to the feature point. On the other hand, in this study we use the omni-camera to detect and estimate the location of each landmark automatically, which is defined to the position of the center of the landmark. The detail will be described in Chapter 4.
In addition, Wang and Tsai [9] proposed a method for angular mapping for camera calibration to compute the angular information of points in the PTZ-image.
The PTZ-camera can be used to estimate the directions of feature points by the method. However, we have to combine the information of the landmark with that of the feature points. Therefore, we propose a method to convert PTZ-camera
coordinates into omni-camera coordinates. The detail will be described in Sections 3.2 and 3.3.2.
By the aforementioned techniques, local maps of landmarks can be created. In Section 3.3.2, we describe a graphical user interface (GUI) which we design in this study for a user to create the local map of every landmark. Moreover, a method for combining multiple local maps into a single global map is also proposed in this study, which we describe in Section 3.3.3.
3.2 Integration of Information Acquired by PTZ-camera and Omni-cameras
3.2.1 Review of Adopted Calibration Method for PTZ-camera
In order to get the orientation of every feature point with respect to the PTZ-camera direction, Wang and Tsai [9] proposed a nonlinear angular mapping method to conduct angular transformations between real-world space points and acquired PTZ image points. The method is adopted for use in this study. It is reviewed here at first.
First of all, we introduce the involved coordinate systems, as shown in Figure 3.1.
The image coordinate system (ICS) defined for PTZ-images is described by image coordinates (u, v), and the spherical coordinate system (SCS) defined for the real-world space is described by the space coordinates (ρ, θ, φ). The SCS is a 3D polar coordinate system whose coordinates can be computed in terms of those of a Cartesian coordinate system (CCS) described by (i, j, k). The origin S of the SCS is
also the origin of the CCS, and the ij-plane of the CCS is parallel to the uv-plane of the ICS. Reversely, the coordinates (i, j, k) of the CCS can be represented by the coordinates (ρ, θ, φ) of the SCS as well. Specifically, as illustrated in Fig. 3.1, a space point P with coordinates (i, j, k) in the CCS may transformed into a corresponding one in the SCS with coordinates (ρ, θ, φ), where ρ is the distance between point P and the origin S of the SCS; θ is the longitude angle between the positive k-axis and the line from the origin S to point P projected onto the ik-plane, and φ is the latitude angle between the ik-plane and the line from the origin S to point P.
In order to calibrate the PTZ camera for the purpose of computing the longitude and latitude angles of space points which appear in the acquired PTZ-image, a grid board is used in this study. It has m vertical lines and n horizontal lines, and is attached on a wall which is perpendicular to the ground. During the calibration process, we measure the angles between the PTZ-camera direction and the lines from Figure 3.1 An illustration of transformation between image coordinate system (ICS)
and spherical coordinate system (SCS) and Cartesian coordinate system.
the camera to the intersection points in the grid board, so the longitude and latitude angle values of all the intersection points can be known in advance. Then, we use an interpolation method to compute the orientation θ of each non-intersection point p in the PTZ-image with respective to the PTZ camera direction. This orientation θ is just the angle between the PTZ-camera direction and the line from the position of the PTZ-camera to p in the PTZ-image. In this way, the longitude and latitude values of each point p in the PTZ-image can be obtained. In Figure 3.2, we can see the respective positions of the grid board and the camera used in this method. And two image views of the board from the camera are shown in Figure 3.3. The intersections of the lines are marked by yellow points.
(a) (b)
Figure 3.2 Camera calibrations by a vertical grid board. (a) An illustration of attaching the lines on the wall. (b) The intersections seen by camera are marked by yellow points.
After knowing the longitude and latitude values of the non-intersection points in the PTZ-image by nonlinear angular mapping, the PTZ-camera calibration work is completed. These longitude and the latitude values of every point in the image is
finally saved into a bmp image, like that shown in Figure 3.3(c).
(a) (b)
(c)
Figure 3.3 Camera views of Figure 3.2. (a) View of Figure 3.2(a). (b) View of Figure 3.2(b). (c) The bmp image with camera calibration information.
3.2.2 Conversion of PTZ-camera Coordinates into Omni-camera Coordinates
The omni-camera coordinate system (OCCS) is regarded as the world coordinate system (WCS) with coordinates (x, y, z), and the PTZ-camera coordinate system (PCCS) as the SCS, in this study. However, we also have the PTZ-camera in the WCS;
therefore, in order to use the information of the PTZ-image coordinates integrally, we have to transform these coordinates into the OCCS. In this section, we propose a
scheme to convert the PTZ-camera coordinates into coordinates in the OCCS. In this way, we can get the relative position between the omni-directional imaging device and the PTZ-camera device easily.
In more detail, we use Figure 3.4 to show an illustration of the layout of the video surveillance car roof where the distance between the omni-camera and the PTZ-camera is obtained by manual measurement. The red circles in the layout are the positions of the camera devices. Accordingly, if the front omni-camera device is assumed to be located at the origin of the WCS, then the position of the PTZ-camera device is (223, 0, 0). That is, the offset between the two devices is (223, 0, 0).
Figure 3.4 an illustration of the layout of the video surveillance car roof.
Furthermore, as shown in Figure 3.5, the practical field of view of the omni-camera is limited. Nevertheless, even if a point P is too far from the practical field of view of the omni-camera, we can still get it by the PTZ-camera. Therefore, we have to convert the coordinates of the PCCS into corresponding ones of the OCCS.
The detail is described as follows.
Figure 3.5 The WCS used in this study which includes the SCS.
As shown in Figure 3.6, we can acquire the tilt angle ϕc and the pan angle θc
from the PTZ-camera device. The value θc is the angle between the camera direction and the direction of the y-axis, and the value φc represents the vertical tilting angle of the camera. The tilt angle ϕ and the pan angle θ of a detected space point in a PTZ-image can be acquired through the calibration method mentioned previously.
The distance D between the PTZ-camera and the detected point can be obtained by manual measurement, and the distance ρ is the horizontal distance between the PTZ-camera and the detected point. Finally, we want to estimate the world coordinates (x, y, z), and the process we propose in this study is described as an algorithm in the following. Refer to Fig. 3.5 for the notations used in the algorithm.
(a) (b)
(c) (d)
Figure 3.6 the angle between the detected point and the PTZ-camera (a) the PTZ-camera has a tilt angle of ϕc < 0 (b) the PTZ-camera has a tilt angle of ϕc > 0 (c) the PTZ-camera has a pan angle of θc < 0 (d). the PTZ-camera has a pan angle of θc >
0
Algorithm 3.1: computing the world coordinates of a feature point.
Input: A feature point p captured by the PTZ-image and its distance D to the PTZ-camera which is obtained manually in advance.
Output: The world coordinates (x, y, z) of the feature point in the PTZ-image.
Step 1. Calculation of distance ρ and height z ---
Using the tilt angles ϕc and ϕ , we can compute the distance ρ and the height z in the following way:
(a) if the ϕc < 0 as shown in Figure 3.5(a), compute ρ and z by:
Step 2. Calculation of x and y in the world coordinate system ---
Use the distance ρ and the pan angles θc and θ, compute the world coordinates x
where the parameter L is the position of the PTZ-camera on the x-axis of the WCS and K is the position of the PTZ-camera on the y-axis of the WCS, both known in advance by manual measurement.
Using the above algorithm, we can get the world coordinates (x, y, z) of each feature point appearing in the captured PTZ-image.
3.3 Creation of Guidance Map
3.3.1 Creation of Database for Guidance Map
The guidance map is important to the proposed augmented reality based guidance system for use in tour guide driving. We have to navigate to visit the sites in the guidance area precisely according to the guidance map, and monitor the entire top-view guidance environment. A display of the guidance map proposed in this study is designed to include various data, such as:
(1) the center points of landmarks which are labeled on the guidance map;
(2) the feature points in the guidance area;
(3) the current position of the video surveillance car.
Landmarks are regarded as navigation points in this study. In order to recognize the different landmarks, every landmark is assigned a label. We can use the labels of landmarks to check the correct positions of detected landmarks and navigate in the guidance area correctly. In the guidance map, we select a landmark to be the start point of a guidance tour with the start label of 0.
On the other hand, feature points can be anything theoretically. However, in our guidance system, we want to get the information of nearby buildings. So we choose edge points of buildings to be our feature points, as mentioned previously. A user can input the names of the selected feature points to the proposed system. Moreover, we have designed a “learning GUI” for the user, which will be described in Section 3.3.2.
In addition, by using the camera devices, the position of a landmark can be computed.
Furthermore, we measure the distance between the PTZ-camera and each feature point by hand, also mentioned previously. Accordingly, the position of feature points on the WCS can be obtained by Algorithm 3.1. Finally, the location of the video
surveillance vehicle can also be marked on the guidance map. In this way, we can always check the position of the video surveillance vehicle and update its corresponding point on the guidance map for display and observation. An illustration of the guidance map is shown in Figure 3.7.
Figure 3.7 Illustration of proposed guidance map.
3.3.2 Creation of Guidance Map with PTZ-camera and Omni-camera
In order to create our guidance map, we have to generate local maps at first. A local map includes a landmark and a number of related feature points, with the landmark being defined as the origin on the local map. So, for each landmark, a local map is created. Also, we record the label of the landmark, and the name and the coordinates of every feature point chosen. On the other hand, a local network has been set up in the surveillance vehicle as mentioned previously, which integrates the two laptops for between-computer communication. The structure of the network is illustrated in Fig. 3.8. Through this network, we can use the two camera devices to get the landmark coordinates on the WCS and the coordinates of every feature point on
the local map. The detail of this process is described as an algorithm below.
Algorithm 3.2: computing the positions of a landmark and related feature points on the local map using images acquired with the two camera devices.
Input: two omni-images I1 and I2 acquired by the omni-imaging devices and a perspective-view image I captured by the PTZ-camera.
Output: the coordinates of a landmark Plm and those of the feature points selected of nearby buildings on a local map with respect to Plm.
Steps.
Step 1. Detect the image points of landmark Plm appearing in I1 and I2 and compute accordingly the coordinates (x1, y1, z1) of the center point of the landmark using a detection method described in Chapter 4.
Step 2. Click a feature point P in the PTZ-image, and measure the distance d between P and the PTZ-camera by hand.
Step 3. According to the discussions in Section 3.2.2, compute the world coordinates (x2, y2, z2) of the feature point P in the SCS of the PTZ-camera in the following way:
Figure 3.8 Structure of the communication between two laptops used in this study.
, x2 = ×d sin(ϕc±ϕ) sin(× θc ±θ)+L; (3.9) the coordinates (x3, y3) of every related feature point in a local map. We record these computed data in the local map of Plm.
3.3.3 Conversion of Local Maps into a Global Map
The global map is used for tour guidance in this study and it is designed to integrate all the local maps. Moreover, as shown in Figure 3.9, when we create the local maps, the last feature point of the local map and the first feature point of the next local map are taken to be an identical feature point, and this overlapping point can be connected to merge the two local maps. Moreover, we define the landmark on the first local map as the origin point on the global map. We divide the process of converting all the local maps into a global map into four steps, as described in the following algorithm.
Algorithm 3.4: Integrate all landmarks with the global map.
Input: a local map set L = {L1, L2, …, Ln} Output: a global map G.
Steps.
Step 1. Translate the coordinates (x0, y0) of the landmark of local map L1 into coordinates (0, 0) of the desired global map G.
Step 2. Translate the coordinates (xm, ym) of every feature point pm in local map L1 the same feature point in global map G, as illustrated in Figure 3.8.
3.2 Calculate the translation values j and k by j = xpl− xpf;
maps L2 through Ln have been merged into the global map G.
Step 4. Take the final global map G as output.
Pl
Pf
Figure 3.9 Between two local maps there is an overlapping point.
3.3.4 Algorithm for generating the guidance map
The following algorithm describes the detail of the proposed method for creating the guidance map as shown in Fig. 3.11.
Algorithm 3.5: a method for creating the guidance map in the learning process.
Input: the upper omni-image set Iu = {Iu1, Iu2, …, Iun} and the lower omni-image set Il
= {Il1, Il2, …, Iln}, and the PTZ-image set I = {I1, I2, …, In}, taken at n pre-selected landmarks along the way of a guided tour.
Output: a guidance map.
Steps.
Step 1. Set i = 1 initially to process images acquired at the first-visited landmark Plmi
.
Step 2. Obtain the position of the currently-visited landmark Plmi
in images Iui and Ili as shown in Figure 3.10(a) using a method described in Chapter 4, and assign a label to Plmi.
Step 3. Click any number of feature points in the PTZ-image Ii using the interface as shown in Figure 3.10(b), compute the position of the acquired feature points, assign names to the buildings including these feature points, and create a local map by Algorithm 3.3.
Step 4. Set i = i + 1 and Go to Step 2 until i > n which means that all the landmarks have been visited and their images processed.
Step 5. Use all the local maps to generate a rough global map as shown in Figure 3.10(a) using Algorithm 3.4.
Step 6. Create the borders of the buildings by connecting the feature points of the same names in the rough guidance map to create a refined guidance map as shown in Figure 3.10(b) as the desired output.
(a)
Figure 3.10 The GUI designed in this study for creating local maps (a) The interface for using the omni-cameras. (b) The interface for using the PTZ-camera.
(b)
Figure 3.10 The GUI designed in this study for creating local maps (b) The interface for using the PTZ-camera. (b) The interface for using the PTZ-camera.(cont’d)
(a) (b)
Figure 3.11 Creating the guidance map. (a) Various feature points on the global map. (b) The guidance map after organizing these feature points.
Click a feature point
Chapter 4
circular-shaped landmark in the omni-image around a video surveillance vehicle. Also described is a scheme we propose for drawing the contour of an elliptical shape on the omni-image after detecting the shape of a circular landmark in an omni-image, which appears approximately as an ellipse in the image. Specifically, the landmark is detected from the omni-image by color detection and ellipse fitting. Then, the 3D location of the center point of the landmark is estimated. Finally, the relative position of the detected landmark with respect to the video surveillance vehicle is computed.The proposed method is divided into four major stages and a flowchart of the method is shown in Fig. 4.1.
In the first stage, the process of detecting a circular shape in the real world consists of four major steps: (1) approximation of a circle in an omni-directional
In the first stage, the process of detecting a circular shape in the real world consists of four major steps: (1) approximation of a circle in an omni-directional