Thesis Organization

The remainder of this thesis is organized as follows. In Chapter 2, the hardware and processes of this system will be introduced. And in Chapter 3, the proposed method for constructing the mapping tables for fisheye cameras will be described.

In Chapter 4, the construction steps of an environmental map and the proposed method for obstacle avoidance are described, and four strategies of correcting the positions and directions of vehicles are also described. In Chapter 5, the method of finding specific partial regions in an image to compute the position of a person and the technique for prediction of the person’s next movement are described. The hand-off problem is solved in this chapter, too.

The experimental results of the study are shown in Chapter 6, and some discussions are also included. At last, conclusions and some suggestions for future works are given in Chapter 7.

Chapter 2

System Configuration

2.1 Introduction

The hardware and software which are used in this study will be introduced in this chapter. The hardware includes the autonomous vehicle we use and the fisheye cameras and wireless network equipments. The software includes the programs for the processes of gathering the information of an environment, constructing an environment map, avoiding obstacles when patrolling in an environment and calculating the position of a person automatically.

2.2 Hardware

The autonomous vehicle we use in this study is a Pioneer 3-DX vehicle made by MobileRobots Inc., and an Axis 207MW camera made by AXIS was equipped on the vehicle as shown in Figure 2.1. The Axis 207MW camera is shown in Figure 2.2.

The Pioneer 3-DX vehicle has a 44cm38cm22cm aluminum body with two 19cm wheels and a caster. It can reach a speed of 1.6 meters per second on flat floors, and climb grades of 25^o and sills of 2.5cm. At slower speeds it can carry payloads up to 23 kg. The payloads include additional batteries and all accessories. By three 12V

fully charged initially. A control system embedded in the vehicle makes the user’s commands able to control the vehicle to move forward or backward, or to turn around.

The system can also return some status parameters of the vehicle to the user.

The Axis 207MW camera has the dimension of 855540mm (3.3”2.2”1.6”), not including the antenna, and the weight of 190g (0.42 lb), not including the power supply, as shown in Figure 2.4. The maximum resolution of images is up to 12801024 pixels. In our experiment, the resolution of 320240 pixels is used for the camera fixed on the vehicle and that of 640480 pixels is used for the one affixed on the ceiling. Both of their frame rates are up to 15 fps. By wireless networks (IEEE 802.11b and 802.11g), captured images can be transmitted to users at speeds up to 54 Mbit/s. Each camera used in this study is equipped with a fish-eye lens that expands the field of view of a traditional lens in general.

(a) (b)

Figure 2.1 The vehicle used in this study is equipped with a camera. (a) A perspective view of the vehicle. (b) A front view of the vehicle.

(a) (b)

Figure 2.2 The camera system used in this study. (a) A perspective view of the camera.

(b) A front view of the camera.

The Axis 207MW cameras are fisheye cameras. They are also affixed on the ceiling and utilized as omni-cameras, as shown in Figure 2.3. A notebook is used as a central computer to control the processes and calculate needed parameters from the information gathered by the cameras.

Figure 2.3 An Axis 207MW camera is affixed on the ceiling.

Figure 2.4 A notebook is used as the central computer.

Communication between the hardware components mentioned above is via a wireless network, a WiBox made by Lantronix equipped on the vehicle, in order to deliver and receive the signals of the odometer as shown in figure 2.5.

(a) (b)

Figure 2.5 The wireless network equipments. (a) A wireless access point. (b) A WiBox made by Lantronix.

2.3 System Process

In the proposed process of constructing the mapping tables for omni-cameras, we calculate the relative positions and rotation angles between the cameras. Afterward, mapping tables are constructed automatically for every camera, and a point-correspondence technique integrated with an image interpolation method is used to calculate the position of any object appearing in the image.

In the proposed process of environment learning, the information is gathered by a region growing technique first. The information includes the position of obstacles and open spaces where a vehicle can drive through. Afterward, the positions will be converted into the global coordinate system, and an environment map will be constructed by composing the coordinates of all obstacles appearing in the vehicle navigation environment.

In the proposed process of security patrolling in an indoor environment, each vehicle is designed to avoid obstacles on the navigation path. If there are some obstacles on a path of the vehicle, the vehicle system will plan several turning points to form a new path for the vehicle to navigate safely. After the vehicle patrol for a while, it will diverge from its path because the vehicle suffers from mechanical errors.

In the proposed process of vehicle path correction, we calculate the position of a vehicle in the image coordinate system by the images taken by the omni-cameras affixed on the ceiling, and then convert the coordinates into global coordinates and modify accordingly the value of the odometer in the vehicle.

In the proposed process of person following, the position of an intruding person’s feet is calculated also by the images taken by omni-cameras on the ceiling, and then

system will calculate the relative position and the rotation angle between the vehicle and the person, and adjust accordingly the orientation and speed of the vehicle to achieve the goal of following the person.

The major processes of the system are summarized and listed below:

1. Construct mapping tables for every top-view cameras.

2. Acquire environmental information by top-view cameras and construct the environment map.

3. Correct mechanic errors continuously in each cycle.

4. Plan a path to avoid obstacles in the environment.

5. Detect, predict, and compute the position of any intruding person by top-view omni-cameras continuously.

6. Handle the camera hand-off problem to keep tracking any intruding person using a single camera at a time.

Chapter 3

Adaptive Space Mapping Method for Object Location Estimation Subject to Camera Height Changes

3.1 Ideas of Proposed Adaptive Space Mapping Method

In this study, we use multiple fish-eye cameras affixed on the ceiling to keep an indoor environment under surveillance. The cameras are utilized to locate and monitor the autonomous vehicle, and trace the track of any suspicious person when he/she comes into the environment. When using these omni-cameras, we want to know the conversion between the ICS (image coordinate system) and the GCS (global coordinate system). So we propose a space mapping method and construct a mapping table for converting the coordinates of the two coordinate systems.

Because the indoor environment under surveillance is unknown at first, we propose further in this study another space mapping method by which the cameras can be affixed to different ceiling heights for use, which we call height-adaptive space mapping method. Besides, multiple fish-eye cameras are used in the mean time to monitor an environment in this study, so calculating the relative positions and angles between every two cameras which have overlapping fields of view (FOV’s) is needed

and is done in this study. Finally, a point-correspondence technique integrated with an image interpolation method is used to convert the coordinates between the ICS and the GCS.

3.2 Construction of Mapping Table

In this section, we propose a method of constructing a basic mapping table for use at a certain ceiling height. The mapping table we use contains 1515 pairs of points, each pair including an image point and a corresponding space point. And the data of each pair of points in the table includes the coordinates (x1, y1) of the image point in the ICS and the coordinates (x₂, y₂) of the corresponding space point in the GCS. We use a calibration board which contains 15 vertical lines and 15 horizontal lines to help us constructing the mapping table.

First, we take an image of the calibration board, find the curves in the image by curve fitting, and calculate the intersection points of the lines in the image. Then, we measure manually the width of the real-space interval between every two intersection points, and compute the coordinates of the intersection points in the GCS in terms of the value of this interval width. Afterward, we affix the cameras to the ceiling, and modify the global coordinates in the mapping table by the height of the ceiling. The details of these tasks will be described in Sections 3.2.1 and 3.2.2. The major steps from constructing a space mapping table to calculating the position of a certain object are described below.

Step 1. Set up a camera (not necessarily attached to the ceiling) at a known height

from the floor and take an image of the calibration board under the camera.

Step 2. Find the curves in the image of the calibration board by curve fitting, and calculate the coordinates of the intersection points of the curves in both the ICS and the GCS, forming a basic space mapping table.

Step 3. Attach the camera to the ceiling.

Step 4. Calculate the coordinates of the intersection points of the calibration board in the GCS utilizing the height of the ceiling assumed to be known and the content of the basic space mapping table, forming an adaptive space mapping table.

Step 5. Calculate the relative positions and rotation angles between the cameras.

Step 6. Calculate the position of any object under the camera by a point-correspondence technique integrated with an image interpolation method using the adaptive space mapping table.

In both the basic and adaptive space mapping tables, we only record the coordinates of the corresponding intersection points between the ICS and the GCS.

When we want to calculate the position of a certain object which is not right on the above-mentioned intersection points, a point-correspondence technique integrated with an image interpolation method will be applied (Step 6 above). The detail will be described in Section 3.4. We only have to calculate the coordinates in the GCS and construct the adaptive mapping table once after the cameras are attached to the ceiling (Step 4 above). The adaptive mapping table can be stored in a computer and can be used any time.

3.2.1 Proposed Method for Constructing A Basic Mapping Table

The basic mapping table of a camera should be constructed first before the camera can be used for environment monitoring. Assume that the camera is attached at a fixed height h (not necessarily to the ceiling). To construct the basic mapping table, at first we put a calibration board on the ground right under the camera. The calibration board contains at least 15 horizontal lines and 15 vertical lines, and the intervals between every two lines are all the same, as shown in Figure 3.1. Then, we use the camera to take an image of the calibration board, extract the quadratic curves in the image by minimum mean-square-error (MMSE) curve fitting, and find the intersections of these curves, as shown in Figure 3.2.

Figure 3.1 The calibration board used for basic table construction with 15 horizontal lines and 15 vertical lines.

These intersection points are described by image coordinates, which are recorded in the basic mapping table. And the height h of the camera is recorded, too. We

to the origin in the GCS, as shown in Figure 3.3.

Figure 3.2 Finding the intersection points in the image of the calibration board.

Figure 3.3 The mapping of the origins of the ICS and GCS.

Afterward, we measure manually the width of the real-space interval Wcali

between every two intersection points on the calibration board, and the global coordinates of the intersections can be calculated by the following way.

First, assume that the upmost and leftmost point in the image is just the projection of the origin of the GCS. So, the global coordinates of this point are (0, 0).

The x-axis is in the horizontal direction, and the y-axis is in the vertical direction.

Hence the coordinates of the intersection points in the GCS can be calculated simply as a multiple of the value Wcali as shown in Figure 3.4. Or more specifically, the global coordinates of the (i, j)-th intersection point in Figure 3.4 may be computed as (i Wcali, j Wcali), where the origin is regarded to be the (0, 0)-th intersection point.

(0, 0) (W_cali, 0)

(Wcali, Wcali)

(W_cali, 2W_cali) (0, Wcali)

(0, 2Wcali)

(2W_cali, 0)

(2Wcali, Wcali)

(2Wcali, 2Wcali)

Figure 3.4 Calculating the coordinates of intersection points in the GCS by Wcali.

If the cameras are used at a height which is the same as that used during the stage of constructing the basic mapping table, we can then use this table to calculate the position of any object under the camera by a point-correspondence technique integrated with an image interpolation method, which will be described in next section.

But if the camera is used at a different height, the global coordinates in the basic mapping table should be modified. Otherwise, the object position we calculate later will be wrong. The method will be described in Section 3.2.3.

3.2.2 Using a Point-correspondence

Technique Integrated With An Image Interpolation Method to Locate

Objects

After the basic mapping table is constructed, we can know the corresponding coordinates of the above-mentioned intersection points in the ICS and the GCS. But when we want to calculate the position of a certain object which is not right on any intersection point, a point-correspondence technique integrated with an image interpolation method is applied in this study. As shown in Figure 3.5, we want to calculate the global coordinates of the point I which is not right on the intersection point. The detail is described as an algorithm in the following.

P_ij

Pi(j+1)

P_(i+1)j

P_(i+1)(j+1) L₀

L₁

L2

L₃ t

s

q I r

M_h

M_v

Figure 3.5 Calculating the coordinates of a non-intersection point by interpolation.

Algorithm 3.1: Calculating the coordinates of an object in the GCS.

Input: A mapping table and the image coordinates of a non-intersection point I.

Output: The global coordinates of the space point G corresponding to I.

Steps:

Step 1. Derive the equations of the lines L0, L1, L2 and L3 in the ICS by the image coordinate data of the points P_ij, P_(i+1)j, P_(i+1)(j+1) and P_i(j+1), as shown in Figure 3.5.

Step 2. Derive the equations of lines M_h and M_v in the ICS.

Step 3. Calculate the image coordinates of the intersection points s, r, t, and q.

Step 4. Calculate the global coordinates G of point I in the GCS.

The above algorithm is just an outline, whose details are now explained. In Step 1, the image coordinates of Pij, P(i+1)j, P(i+1)(j+1) and Pi(j+1) are known, so the equations of lines L₀, L₁, L₂ and L₃ in the ICS can be derived. In Step 2, the slope of M_h is the average of those of lines L1 and L3, and the slope of Mv is the average of those of lines L₀ and L₂. The equation of M_h and M_v in the ICS can be derived by these slopes and I.

Then, the intersection point s of L1 and Mv can be computed accordingly, so are r, t, and q in similar ways. Finally, the global coordinates (G_x, G_y) of I can be calculated by the following formulas according to the principle of side proportionality under the assumption that the space area enclosed by the four corners P_ij, P_(i+1)j, P_(i+1)(j+1) and Pi(j+1) is not too large so that the linearity inside the area holds:

)

where (X, Y) are the global coordinates of P computed in a way described previously,

and d(q, I) means that the distance between points q and I. Note that (X, Y) = (i Wcali, j Wcali) as mentioned previously in Section 3.2.1 if P_ij is the (i, j)-th intersection point in the calibration board.

3.2.3 Proposed Method for Constructing An Adaptive Mapping Table

If the camera is used at a height different from that used in constructing the basic mapping table, then when calculating the global coordinates of the intersections found on the calibration board, the value of the interval should be changed to another value Wreal as shown in Figure 3.6; otherwise, the computation results will be incorrect.

Thereafter, the method of calculating the global coordinates of space points is the same as discussed previously.

(0, 0) (W_real, 0)

(Wreal, Wreal)

(W_real, 2W_real) (0, Wreal)

(0, 2Wreal)

(2W_real, 0)

(2Wreal, Wreal)

(2Wreal, 2Wreal)

Figure 3.6 Calculating the coordinates of intersection points in the GCS by W_real.

To compute W , we make the following observation first. When the camera is

attached to a lower height, the FOV’s will decrease, and the number of points found in Figure 3.2 is the same, so the value of Wreal will also decrease. On the other hand, when cameras are attached to a higher height, the FOV’s will increase, and the value of Wreal will increase, too. Accordingly, we derive a formula to calculate Wreal in the following. Note that if we use multiple cameras in this study, the global coordinates should be revised additionally, because there may be some rotation between the cameras. The detail of this problem will be described in Section 3.3.

As depicted in Figure 3.7 there is a camera lens and an object O1 with height h1

on the left. Hence, the formation of an image of O₁ will be on the right side. The distance between point B and C is d1, the distance between point C and E is f, and the distance between E and F is a. The height of the projection of the object on the image is i. Because the two triangles ABC and CFG are similar in shape, we can obtain the following equation from the principle of side proportionality:

a

Figure 3.7 The imaging process of an object O1.

image is on the right side with height i, too. But the distance between O₂ and lens is

Figure 3.8 The imaging process of object O2.

By Equations 3.3 and 3.4, we know

a

By Equations 3.5 and 3.6, we know

b

according to Equations 3.4 and 3.6 as follows:

b

h by Equations 3.7 and 3.8 as follows:

b

Substituting Equation 3.11 into the above for the value of b, we get

(3.16) leads to the following formulas:

2

We may rewrite the formulas above more clearly by other symbols as follows:

ca li board as mentioned before; H_cali is the height between the camera and the calibration board, alse mentioned previously; Hreal is the real height at which the camera is affixed now; and W_real is the desired new interval for use now to replace the value Wcali in Equations (3.1) and (3.2).

3.3 Using Multi-cameras to Expand the Range of Surveillance

Because the range of surveillance of a single camera is finite, we use multiple cameras to expand the range of surveillance in the study. First, the adaptive mapping tables should be constructed for every camera by the method mentioned above. But if there are displacements and rotations between the cameras, then the global coordinates of the mapping table should be modified further. The major steps of constructing a modified mapping table are listed below, and the detail will be described in the rest of this section.

1. Calculate the relative position of two cameras.

2. Calculate the relative rotation angle of two cameras.

3. Calculate the global coordinates of intersection points in the calibration board in the expanded range.

3.3.1 Calculating Relative Position of Cameras

We design a special target board with a shape of a very small rhombus region on the ground as shown in Figure 3.9, and put it at the upmost and leftmost intersection area of the two cameras as shown in Figure 3.10. Because we want to calculate the global coordinates of all the intersections in the image via the coordinates of the center of this upmost and leftmost intersection area, we need to calculate the global

coordinates of this center point first.

Then, an image of the target board is taken with the first camera. The target board area in the image are found out by the information of color, and the coordinates

Then, an image of the target board is taken with the first camera. The target board area in the image are found out by the information of color, and the coordinates

在文檔中 以天花板上多環場攝影機輔助自動車作室內安全監控之研究 (頁 20-0)