Chapter 2 System Configuration and Navigation Principles
2.4 Vehicle Guidance Principle and Proposed Process
When the learning job has been done, all vehicles can start to perform the security patrolling task. The entire guidance process proposed in this study is shown in Figure 2.6.
At first, the system reads all files that are obtained from the learning phase and contain information about the environment, autonomous vehicles, and monitored objects. According to the distances between all pairs of MPs, this system then plans random paths for each autonomous vehicle. If all differences between the paths of two vehicles do not exceed a threshold value which ensures the loads of all autonomous vehicles being balanced, the security patrolling task can be carried out.
Because autonomous vehicles suffer from accumulation of mechanical errors, we need to locate them constantly. When a vehicle runs a fixed length of distance, it must be located by the top-view omni-cameras. By the values of the vehicles’ odometers, this system calculates the centroid of each vehicle from an image captured by a top-view omni-camera. The other function of the camera is to monitor vehicles to see whether they are still under control. If any vehicle loses control of its action, the system will stop all vehicles and send an alarm message to the user. Otherwise, the odometer of the vehicle is corrected and then the vehicle proceeds to move to its goal node.
While the vehicles are carrying out the security patrolling, there could be collisions between vehicles. Therefore, the detection of collisions is necessary. This system computes the distance between two vehicles in every cycle of a fixed time duration and determines if they are too close. If true, the paths of the vehicles will be
21
Figure 2.6Flowchart of proposed navigation process.
A mission for the autonomous vehicles in this system is to take pictures of all the monitored objects during the navigation process. As a vehicle goes to a MP, it means that the vehicle will be in front of a monitored object. Therefore, the direction of the vehicle must be adjusted to face the object. Then the camera equipped on the vehicle takes a picture at the moment. The picture is transmitted to the control system by the wireless network and saved into an image file. When all the vehicles have accomplished their own patrolling paths, one cycle of security patrolling is finished.
Then, the system will plan another set of new random paths for all the autonomous vehicles again.
22
Chapter 3
Learning Strategies for Navigation by Semi-automatic Driving
3.1 Ideas of Proposed Techniques Used in Learning
In this study, two cameras with fish-eye lenses fixed on ceilings are utilized to locate and monitor all autonomous vehicles. Before the use of the cameras, they must be calibrated. For this purpose, we propose in this study a point-correspondence technique integrated with an image interpolation method without conducting the conventional task of calculating the projection matrix for transforming points between 2-D image and 3-D global spaces. At first, by a mouse a user points out some landmarks in an image of a calibration target which is selected to be the tile pattern on the floor of our experimental environment. The landmarks we use in this study are the crossing points of the grid formed by the tile pattern. Such crossing points for use as corresponding points are abundant which yield better calibration accuracy in the proposed point-correspondence technique for camera calibration. The detail is described in Section 3.2.
In an environment where autonomous vehicles navigate, it is indispensable to use some turning points in the navigation path to ensure no collision between the vehicles and the walls. To compute the turning points, the corner points of the walkable area are first utilized to acquire all rectangular regions within the entire area. Each region
23
is then represented by its upper-left and lower-right points. With these points, the system can judge whether the vehicles can move straightly between any pair of nodes where the vehicles visit (including the turning points). If two nodes belong to different regions, the vehicle will be guided to pass some turning points. In other words, the turning point is a medium that enables the vehicles to navigate between any pair of nodes without incurring collisions with the walls. Therefore, a turning point is selected to be the intersection of the centerlines of two overlapping regions or the center of the overlapping boundary of two adjacent ones. The details of the proposed techniques about processing rectangular regions and computing turning points are described in Section 3.3.1.
Additionally, to take the pictures of monitored objects by cameras equipped on the autonomous vehicles, all nodes and directions with respect to the objects must be recorded. In this study, a learning technique is proposed to guide vision-based vehicles to capture pictures at suitable spots and directions. Two top-view omni-cameras are used. The process is described in detail in Section 3.3.2.
3.2 Calibration of Top-View
Omni-Cameras with Fish-Eye Lenses
Each camera used in this study is equipped with a fish-eye lens. All images captured by the camera are warped. So the traditional camera calibration method of obtaining a global-space point via a projection-based transformation cannot be utilized directly; the cameras must be calibrated by another method, as mentioned
24
previously. For this, we propose a point-correspondence technique integrated with an image interpolation method. By the way, it is noted that the correct coordinates in the global space can be obtained from a warped image directly, as done in this study.
3.2.1 Review of Conventional Camera Calibration Technique
In general, a projection matrix is utilized in conventional methods to do the job of camera calibration. There are two kinds of parameters in the matrix, which must be calibrated, namely, the intrinsic and the extrinsic parameters. The intrinsic parameters do not depend on the position and orientation of a camera in the global space and include the focal length f, the image center point (u0, v0), the aspect ratio (Sx, Sy), and the skew error θs of the camera. Because the coordinate system of a camera and the global space may not be the same, the extrinsic parameters related to the rotation angle θ and the translation vector ( , , )t t t of the camera must be x y z calibrated.
Based on the intrinsic and extrinsic parameters, the relation between points in 2-D image and 3-D global spaces may be described by Eq. (3.1) below [23], where the point (u, v)T is in the image coordinate system and the point (x, y, z)T is in the global coordinate system:
25
0 0
transformation transformation matrix of matrix of
1 intrinsic parameters extrinsic parameters 1
3.2.2 Proposed Calibration Technique
In the proposed calibration technique, we divide the patrolling area on the floor of the environment into multiple grids at first and all corners of them are called reference points. These points are the crossing points of the boundaries of the rectangular-shaped tiles on the floor. For every reference point, both of its coordinates in an image and in the global space must be recorded, describing a pair of corresponding points between the image and the global spaces.
In order to acquire more corresponding points faster, we calculate all quadratic curves in the image of the patrolling floor area, of which the intersections are exactly the desired reference points. Note that because the images are captured by the cameras equipped with fish-eye lenses, a straight line in the global space appears as a quadratic curve in the image. Therefore, the technique is feasible. A quadratic curve can be calculated by three points at least. This property is utilized to find all curves.
More specifically, we use a minimum mean-square-error (MMSE) method to calculate all the curves. Assume that a curve L is to be computed, which includes three parameters a, b, c. If points (x1, y1), (x2, y2), ..., (xn, yn) belong to the curve L, we may acquire n curves which can be represented as a matrix in the form of Aw bJK K=
, as
26
Through a series of simplifications from Eq. (3.3), we can acquire a result as
(A A w A bT )ˆ = TK
Therefore, the curve L may be computed to be
2
Before calculating a curve, a user has to input the index of the curve and indicate that the curve is horizontal or vertical. As a curve is obtained, the pixels passed by the curve must record the index of the curve. The process of acquiring all quadratic curves is described as Algorithm 3.1 below. An example is shown in Figure 3.1 which is a result of acquiring some horizontal and vertical quadratic curves by the algorithm and each curve is calculated by four points on it.
27
Algorithm 3.1 Calculating a quadratic curve.
Input: An image I, the number n of points needed to calculate a curve, the index k of the curve, and being horizontal or vertical for the curve.
Output: Pixels passed by the quadratic curve.
Steps:
Step 1. Point out n points on the curve in the image I.
Step 2. Calculate the curve by the MMSE criterion as described previously.
Step 3. Record index k and the property of being horizontal or vertical into the table of pixels passed by the curve.
Figure 3.1Calculating quadratic curves.
When all curves are obtained, we check all pixels in the image. If a pixel is passed both by a vertical curve and by a horizontal one, the pixel is taken to be one of the reference points. The width and height of a grid in the global space are also taken as input by the user. According to the lengths and the total numbers of horizontal and vertical curves, the coordinates of all reference points in the global space can be
28
obtained. The entire process of acquiring all corresponding points between the image and global spaces is described as Algorithm 3.2 below. A result of finding all corresponding points from the image of one top-view omni-camera is shown in Figure 3.2.
Algorithm 3.2 Acquiring corresponding points between 2-D image and 3-D global spaces.
Input: An image I, the total number nx of vertical curves, the total number ny of horizontal curves, and the width w and height h of a grid in the global space.
Output: The coordinates of all corresponding points.
Steps:
Step 1. Repeat Algorithm 3.1 until enough curves are obtained.
Step 2. Find all intersections from the result of Step 1 and record their coordinates in the image coordinate system.
Step 3. Calculate all coordinates, with respect to the intersections obtained from Step 2, in the global coordinate system according to nx, ny, w and h, and then record them.
29
Figure 3.2 Finding all corresponding points.
With the corresponding points, a bilinear interpolation method is used in this study to obtain the global-space coordinates with respect to the points in the warped image. Some assumptions involved are shown in Figure 3.3. The points
( 1) ( 1)( 1) ( 1)
, , ,
ij i j i j i j
P P+ P+ + P + are the corners of the grid which point I belongs to. The lines L0, L1, L2 and L3 are straight lines through these corners. Mh is the line passing through point I and its slope is the mean of those of line L1 and line L3. Mv is the line passing through point I and its slope is the mean of those of line L0 and line L2. The points q and r are the intersections of Mh with L0 and L2, respectively. The points s and t are the intersections of Mv with L1 and L3, respectively.
30
Figure 3.3Point I in a distortion image.
For a point in a warped image, we must judge which grid the point belongs to and calculate Mh, Mv, q, r, s and t. The ratios of distances are utilized to obtain the global-space coordinates with respect to the point. The equation is Eq. (3.7) below:
( , )
point I and pointP , respectively. The lengths unit_width and unit_height are the ij widths and the heights of a grid in the global space, respectively. The distance
( , )
d q I is the length from point q to point I. A graphic illustration is shown in Figure 3.4 and the process of acquiring the coordinates in the global space is described as Algorithm 3.3.
31
Figure 3.4 A relation between the image taken by the omni-camera and the global space.
Algorithm 3.3 Computation of coordinates in the global space.
Input: A point I in a warped image, the coordinates of all corresponding points, the width unit_width and the height unit_height of a grid in the global space.
Output: The coordinates of the point G in global space with respect to the point I.
Steps:
Step 1. Judge which grid point I belongs to.
Step 2. Calculate lines L0, L1, L2 and L3 that are the straight lines through the corners of the grid to which point I belongs, as illustrated by Figure 3.3.
32
Step 3. Calculate line Mh that passes through point I with its slope being the mean of those of line L1 and line L3.
Step 4. Calculate line Mv that passes through point I with its slope being the mean of those of line L0 and line L2
Step 5. Calculate points q and r that are the intersections of line Mh and the edges of the grid.
Step 6. Calculate points s and t that are the intersections of line Mv and the edges of the grid.
3.3 Information for Security Patrolling
3.3.1 Proposed techniques of learning patrolling environment
Information about the patrolling environment includes rectangular regions, which the floor shape is composed of, and the turning points among the regions. To find all rectangular regions, a user must key in all the corners of the patrolling area in the clockwise order. Because each point in the area belongs to one rectangular region at
33
least, a vertical line, called scanning line, is utilized to check whether some points in the vertical line do not belong to one region at least. If yes, another new rectangular region will be found for the points. The entire process and details are described as Algorithm 3.4 below.
Algorithm 3.4 Finding all rectangular regions.
Input: All corners of the patrolling area.
Output: All rectangular regions of the patrolling area.
Steps:
Step 1. Set a vertical line L through the leftmost corner of the patrolling area and scan the area from left to right.
Step 2. Find the intersections of L and the boundaries of the area, and record them.
Record the corners of the boundary merely, if there is an overlap between L and the boundary.
See the example shown in the following, where points a, b and c recorded.
Figure 3.5Intersections of L and the boundaries of the area.
Step 3. Divide these intersections into groups from top to down with each group including two points.
34
See the example of Step 2 above for an illustration, where two groups (a, b) and (b, c) are obtained.
Step 4. Judge where to set a line segment l.
Step 4.1. Check whether the right part of a group is the inner range of the patrolling area and a rectangular region in the range has not been found.
If yes, go to Step 4.2; else, continue to check the next group.
Annotate that the technique of checking whether the right part of a group is the inner range of the patrolling area is described in Algorithm 3.5.
Step 4.2. Set l in the following way.
Step 4.2.1. Shift the middle point M of the group one pixel to the right.
Step 4.2.2. Extend M along a vertical direction until colliding with the boundaries, where the line l is set.
See the example shown in the following, where line de is l mentioned above.
Figure 3.6Setting a line segment l.
Step 5. Continue scanning the line segment l to the left until colliding with the boundary of the patrolling area.
See the example shown in the following.
35 scanning
l Region1
Figure 3.7Scanning the line segment l to the left.
Step 6. Continue scanning the line segment l to the right until colliding with the boundary of the patrolling area.
See the example shown in Step 5, where Region2 was obtained.
scanning l
Region2 Region1
Figure 3.8Scanning the line segment l to the right.
Step 7. Continue scanning the vertical line L to the right and repeat Step 2 until L reaches the rightmost boundary of the patrolling area.
Algorithm 3.5 Judging an inner range.
Input: All corners C of the patrolling area, and two points P whose x-coordinates are
36
the same.
Output: An indication whether the right part of P is an inner range of the patrolling area.
Steps:
Step1. Calculate the middle point M of the two points P.
Step2. Shift M one pixel to the right.
Step3. Set a vertical line LM through point M.
Step4. Check the numbers of intersections of the line LM and the boundary.
If both the number of the upper part and that of the lower one of M are odd and either part does not include an overlapping line, it is decided that the right part of P is an inner range of the patrolling area.
See the graphic illustration shown in the following.
Case 1: the right part of points a and b is not an inner range of the
Figure 3.9The right part of points a and b is not an inner range.
Case 2: the right part of points b and c is an inner range of the patrolling area.
37 ( , )x y1
( , )x y2
1 2
( 1, )
2 y y M x+ + a
b
c L
Figure 3.10 The right part of points a and b is an inner range.
Now, we show a result of Algorithm 3.4. A patrolling area is shown in Figure 3.11. The total number of rectangular regions in the patrolling area is seven and they are shown in Figure 3.12 through Figure 3.18 step by step.
Figure 3.11 A patrolling area.
According to these rectangular regions, we can compute all desired turning points. If two regions are overlapping or adjacent, we can obtain a turning point. The point is the intersection of two vertical and horizontal centerlines of the regions. The process of finding all turning points is described as Algorithm 3.6 and the result in the patrolling area of Figure 3.11 is shown in Figure 3.19. The eight circles in Figure 3.19 are exactly the turning points, in which each blue one is the center of the overlapping
38
boundary of two adjacent regions.
Region1
Figure 3.12The first rectangular region.
Region2
Figure 3.13 The second rectangular region.
Region3
Figure 3.14The third rectangular region.
39
Region4
Figure 3.15The fourth rectangular region.
Region5
Figure 3.16 The fifth rectangular region.
Region6
Figure 3.17The sixth rectangular region.
40
Region7
Figure 3.18 The seventh rectangular region.
Algorithm 3.6 Finding all turning points.
Input: All rectangular regions R in an area.
Output: All turning points.
Steps:
Step1. Take a pair of overlapping or adjacent regions from the set R.
If the two regions are overlapping, go to Step2; else, go to Step4.
Step2. Calculate all vertical and horizontal centerlines of the two regions.
Step3. Find an intersection within these centerlines, which is exactly a turning point of the area.
Step4. Calculate the center of overlapping boundary, which is exactly a turning point of the area.
Step5. Repeat Step 1 until all regions in R have been checked.
41
Figure 3.19 All turning points in the area.
3.3.2 Learning of poses of vehicles with respect to monitored objects
To learn the poses of autonomous vehicles with respect to monitored objects, we designed a user interface, as shown in Figure 3.20. The image in the interface is the real-time view of the camera installed on an autonomous vehicle.
Before learning, a vehicle is parked at the origin of the global space. Because the vehicle suffers from mechanic errors, a user must constantly locate it by the top-view omni-cameras in the period of the learning phase. We can use a joystick or the buttons in the user interface to control the moves of the vehicle. As the vehicle has moved a short distance, the user must press the “Localize” button in the interface. For an example, Figure 3.21 shows a situation that the vehicle is ready to be located. After the user presses the “Localize” button, the system will calculate the centroid of the
Before learning, a vehicle is parked at the origin of the global space. Because the vehicle suffers from mechanic errors, a user must constantly locate it by the top-view omni-cameras in the period of the learning phase. We can use a joystick or the buttons in the user interface to control the moves of the vehicle. As the vehicle has moved a short distance, the user must press the “Localize” button in the interface. For an example, Figure 3.21 shows a situation that the vehicle is ready to be located. After the user presses the “Localize” button, the system will calculate the centroid of the