Chapter 1 Introduction
1.5 Thesis Organization
The remainder of this thesis is organized as follows. In Chapter 2, we describe the system configuration of the vehicle used as a test bed, as well as the principle of vehicle guidance with operations of obstacle avoidance and danger condition detection. The proposed method of vision-based vehicle location estimation for correcting records of the odometer in the vehicle is described in Chapter 3. The proposed method of real-time obstacle avoidance by goal-direction minimum path following is described in Chapter 4. In Chapter 5, we discuss two kinds of danger conditions and describe the proposed methods for danger condition monitoring. In Chapter 6, the proposed method for security monitoring of paintings on walls are described. The experimental results are shown in Chapter 7. Finally, some conclusions and suggestions for future works are given in Chapter 8.
Chapter 2
System Configuration and Navigation Principles
2.1 Overview
Security surveillance in indoor environments is paid more and more attention recently even in ordinary families. Using a vision-based autonomous vehicle to perform security surveillance is a good idea because it can save manpower and the vehicle can patrol all day without taking a rest. In general, there are usually obstacles, narrow paths, and unexpected danger conditions in indoor environments. Therefore, a small and dexterous vehicle is a wonderful choice for this study. A small vehicle is convenient to monitor the space under tables, beds, cabinets, etc., and can navigate for a larger range.
It is indispensable to give the vehicle vision ability to accomplish security surveillance. In this study, we take a pan-tilt-zoom camera as an eye of the vehicle.
Although the vision of the vehicle is monocular cause of only one camera, we use computer vision techniques to overcome the problem in this study. In Section 2.2, the features of a pan-tilt-zoom camera and the reason of using such a camera are described in detail.
The vehicle system used in this study as a test bed is composed of a small vehicle and a pan-tilt-zoom camera There are some communication and control equipments for users to communicate with the vehicle and to control the vehicle to achieve
functions. The entire architecture including hardware and software is introduced in Section 2.3.
We also need a principle to guide the vehicle when the vehicle detects danger conditions in security patrolling in indoor environments. The principle of the vehicle guidance adopted in the study is described in Section 2.4.
2.2 Advantage of Using Pan-tilt-zoom IP Cameras
A traditional pinhole camera used on the vehicle is lack of dexterity because the view of the camera is fixed. It means that if we want to look at another situation out of the view, we should move the vehicle to face the situation first. With a new pan-tilt-zoom camera, we can look at anywhere by turning the camera around without moving the vehicle first. A pan-tilt-zoom camera can perform panning, tilting, and zooming actions as illustrated in Figure 2.1. It means that we can observe surroundings and get the conditions in them quickly without changing the vehicle direction. We also obtain a wider range of field of view because the camera can pan and tilt. Equipped with a pan-tilt-zoom camera on the vehicle overcomes the disadvantage that the view of a vehicle is fixed and restricted to be a lower area. Such a disadvantage is caused by attaching a fixed camera on a small vehicle at a lower position. On the contrary, a vehicle equipped with a PTZ camera can monitor not only objects at lower positions but also objects at higher areas, like paintings.
Furthermore, because we transmit image data in a wireless way, another disadvantage of a traditional pinhole camera is image data transmission through
analog radio, which usually produces noise, especially in a longer transmission. In addition, the image data we deal with is in digital form and not in analog form, so the image data need to be converted into digital form for a computer to deal with. In this study, we adopt an IP camera which transmits image data through a wired network.
The image data transmitted by the IP camera is in digital form and with less noise.
Figure 2.1 Action of a pan-tilt-zoom camera.
2.3 System Configuration
In order to perform functions and complete missions in this study, we use a mini-vehicle as a test bed and install a PTZ camera on it as its monocular vision.
Because we want to control the whole system remotely, some wireless communication equipments are necessary. We also need a power source to energize the whole system.
The hardware architecture and used components of the test bed are described in Section 2.3.1. Besides the hardware, the software is useful to help us develop the system and provide an interface for users to control the vehicle. The software we used in the study is described in Section 2.3.2.
2.3.1 Hardware Configuration
camera, the vehicle, and the computer. In this study, the camera communicates through a wired network, the vehicle communicates through a wireless network, and the computer communicates through both. So we use a wireless access point to connect these hardware components. The entire system is illustrated in Figure 2.2 and the appearance of the hardware architecture in our system is shown in Figure 2.3.
Figure 2.2Structure of the system
The PTZ IP camera used in this study is AXIS 213 PTZ made by AXIS, as shown in Figure 2.4. This is a camera with a height of 130mm, a width of 104 mm, a depth of 130mm, and a weight of 700g with panning, tilting, and zooming functions.
The pan angle range is 340 degrees and the tilt angle range is 100 degrees. It can zoom 26 times optically and 12 times digitally. The image grabbed in our experiments is of the resolution of 320×240 pixels. It also provides a 10Base/100BaseTX Ethernet network and needs an external power source of 13V DC and 1.8A. Because the
camera is not directly connected to the computer used to control the system, we use a wireless access point to connect these two components.
(a)
(b) (c) Figure 2.3 The vehicle used in this study. (a) The bevel view of the vehicle. (b) The
front view of the vehicle. (c) The left side view of the vehicle.
The mini-vehicle used in this study is an Amigo Robot made by ActivMedia Robotics Technologies, Inc. The length, width, and height of the vehicle are 33cm, 28cm, and 21cm, respectively. There are two larger wheels and one auxiliary small wheel at the tail of the vehicle. The maximum speed of the vehicle is 75cm/sec and the maximum rotation speed is 300 degrees/sec. There are eight ultrasonic sensors, an odometer, and an embedded hardware system in this mobile vehicle. The ultrasonic sensors are not used in this study. The odometer provides the coordinates and the
direction of the vehicle in the navigation. The origin of the coordinates is the starting position of the vehicle. There is a 12V battery in the vehicle to supply the power of the vehicle system.
(a)
(b) (c) Figure 2.4 The pan-tilt-zoom camera used in this study. (a) The bevel view of the
camera. (b) The front view of the camera. (c) The right side view of the camera.
There is one wireless device in the vehicle and another in the PC. The commands of the remote system are transmitted to the wireless signal receiver by an access point that meets the 802.11b standard. By using the access point as a medium, the commands can be transmitted from the computer to the vehicle and the camera.
2.3.2 Software Configuration
We use the ARIA, which is an object-oriented programming interface for the ActivMedia Robotics’ line of intelligent mobile robots to control the vehicle actions easily, such as move, rotation, and stop. The lowest-level data and information of the vehicle are also retrieved easily by means of the ARIA interface. In other words, any developer can use the ARIA as an interface to communicate with the embedded system of the vehicle. And we use the Borland C++ builder as the development tool in our experiments.
The AXIS Company also provides a development tool called AXIS Media Control SDK for AXIS 213 PTZ. With the SDK, we can get image previews and the current image from the camera. We can also perform the panning, tilting, and zooming actions easily through the SDK. It is convenient for us to develop any functions with the images grabbed from the camera.
2.4 Vehicle Guidance Principle
Before the vehicle patrols in an indoor environment, a navigation map must be created first. The node information of a learned path is indispensable for creating the navigation map. There are two types of nodes in this study. One is used for correction of the vehicle location. The other is used for security monitoring of paintings. We save the node information and camera position when the vehicle moves to a node for correction of the vehicle location. We also save the node information, camera position and painting data when the vehicle moves to a node of security monitoring of a painting. After a manual learning process, a set of learned data, including node
information, are saved in the system. The navigation map is created by using the node information and is used to guide the vehicle in indoor environments. In order to guide the vehicle along the learned path, the vehicle moves sequentially from one node to another according to the navigation map. While the vehicle patrols in indoor environments, there are three things which require attention. One is whether the vehicle reaches the next node or not. The next is whether the vehicle encounters an obstacle or not. The last is whether the vehicle detects any danger conditions or not.
An illustration of the vehicle navigation process is shown in Figure 2.5.
Figure 2.5 Flowchart of proposed navigation process
When the vehicle reaches the next node, it corrects its location by a house corner first. If there is a nearby painting which is monitored, the vehicle uses learned
painting data to detect whether the painting still exists or not. If the detection process of the painting fails, the system issues a warning signal to the user.
The vehicle might encounter a variety of situations in patrolling. In order to react to conditions in real time, we analyze the images grabbed by the camera continuously to detect the following two conditions. If an obstacle is detected on the patrolling path, a new path for obstacle avoidance and destination approaching is planned. If a danger condition is detected, a warning message is announced to relevant people who can prevent the danger condition from getting worse. The proposed vehicle guidance method is described in the following as an algorithm.
Algorithm 2.1. Vehicle guidance procedure.
Input: A set of learned navigation nodes.
Output: The vehicle moving action.
Steps:
Step 1. Create the navigation map by the learned navigation nodes.
Step 2. Guide the vehicle to the next navigation node and detect obstacles or danger conditions such as fire and lighting failure.
Step 3. Whenever the vehicle reaches a navigation node, move the camera to face a house corner.
Step 4. Estimate the vehicle location by the house corner and correct the odometer of the vehicle.
Step 5. If the location of the vehicle indicates that the vehicle is too far from the navigation node, then guide the vehicle to the navigation node.
Step 6. Go to step 2 if there is a next navigation node.
Chapter 3
Vehicle Guidance by Vehicle Location Estimation Using A House Corner Detection Technique
3.1 Overview
The location of a vehicle is important information to guide the vehicle to navigate correctly in an indoor environment. Installation of an odometer on a vehicle is useful to record the position and the orientation of the vehicle in every navigation cycle. Because of the existence of possible mechanic errors, the position data obtained from the odometer might be inconsistent with the real position. In order to make a vehicle navigate more stably, vision-based vehicle location estimation is helpful to correct the error.
Using binocular cameras to simulate human stereo vision for vehicle location estimation is a method. But in a controllable indoor environment, using a monocular camera is sufficient for vehicle location estimation if special standard marks are adopted. This kind of technique depends on the use of image analysis under certain reasonable assumptions. A number of methods will be reviewed in Section 3.2. One of these methods is vehicle location estimation by house corner detection proposed by Chou and Tsai [8]. The proposed vehicle location estimation method in this study is a simplified version derived from Chou and Tsai. The principle and feasibility of the method are described in Section 3.3.
In order to detect house corner edges and to determine automatically the coefficients of the edge line equations which are used in the vehicle location estimation, several image processing techniques are employed. The techniques and processes are described in Section 3.4.
3.2 Review of Vehicle Location Estimation Techniques
In a known indoor environment, the use of special landmarks and monocular images captured by a camera is an ordinary technique for vehicle location estimation.
As shown in Figure 3.1 (a), a diamond shape whose boundary consists of four identical thick line segments all with a known length is the standard landmark proposed by Fukui [6]. The two diagonals are arranged to be vertical and horizontal, respectively. However, one restriction is that the lens center of a camera should be put at the same height as the diamond center and the camera optical axis is pointed through the diamond center. Under the restriction, the boundary of the diamond images taken by the camera is extracted, and the lengths of the two diagonals are computed. Based on the two diagonals in the image, the location of the camera is finally derived, i.e., the location of vehicle is obtained. Courtney and Aggarwal [9-10]
relaxed the restriction of the method by a reasonable assumption that the height of the camera is known.
Another method proposed by Magee and Aggarwal [7] is to take a sphere on which two perpendicular great circles are drawn as a standard landmark for vehicle location estimation. An illustrated is in Figure 3.1 (b). This method also possesses the
same restriction that the camera optical axis must be pointed through the sphere center before the image of the sphere is taken. According to the size of the projected circle of the sphere, the distance from the camera to the sphere center and the direction of the camera are computed. From the points on the projections of the great circles that are closest to the center of the sphere outline, the vehicle location is finally computed accordingly in terms of the sphere coordinates.
Chou and Tsai [8] utilized a set of house corner edges as a standard landmark.
This landmark exists naturally in a house, and is not entirely artificial-created. Such a landmark is visible from a house floor, and most show as an identical geometric structure of a “Y” shape like Figure 3.1 (c). The projections of the three lines going through the corner on the image plane are then extracted to estimate the vehicle location. A reasonable assumption is the distance from the camera to the ceiling is known. Under this assumption, the coefficients of the equations of the house corner edge lines are substituted into a set of location formulas. Finally we can estimate the vehicle location by results yielded by the location formula set.
(a) (b) Figure 3.1 Three different landmarks. (a) A diamond-shaped standard mark used in
[6]. (b) A sphere used for robot location in [7]. (c) The perspective projection of a house corner used in Chou and Tsai [8].
(c)
Figure 3.1 Three different landmarks. (a) A diamond-shaped standard mark used in [6]. (b) A sphere used for robot location in [7]. (c) The perspective projection of a house corner used in Chou and Tsai [8].
3.3 Vehicle Location Estimation by House Corner Detection
We use only two edge lines instead of all the three ones of a house corner to estimate the vehicle location. The pattern used in the study is illustrated in Figure 3.2.
The vehicle is equipped with a PTZ camera to get a house corner image in each navigation cycle. In order to describe the proposed method conveniently, we introduce some coordinate systems in Section 3.3.1. The principle is to use the coefficients of the equations of the edge lines through the corner point to estimate the vehicle location. The detail of the principle is described in Section 3.3.2.
The relation between the vehicle location parameters and the coefficients of the edge line equations are discussed in detail in Section 3.3.3. The derivation of the vehicle location parameters is also described.
Figure 3.2 The two red lines of the perspective projection of a house corner used in the study.
3.3.1 Coordinate Systems
The three coordinate systems used in this study are introduced at first. By these coordinate systems, it will be clear and convenient to describe a vehicle location. The definitions of the three coordinate systems are described in the following.
(1) The global coordinate system (GCS, denoted as X-Y-Z): In the global coordinate system, we assume that the two perpendicular lines of a corner on the ceiling are the X and Y axes, the vertical line is the Z axis, and the corner point is the origin Go of the global coordinate system.
(2) The camera coordinate system (CCS, denoted as U-V-W): In the camera coordinate system, we establish the U, V, and W axes. The U-V plane is parallel to the image plane, and the U axis is parallel to the X-Y plane of the global coordinate system. The origin Co is located at the camera lens center and the W axis is aligned to be parallel with the camera optical axis.
(3) The image coordinate system (ICS, denoted as u-v): The image plane is located at W = f, where f is the focus length of the camera. The image plane of the image coordinate system is coincident with the u-v plane and the
origin Io is the image plane center.
The relations among the three coordinate systems are illustrated in Figure 3.3.
Figure 3.3 The three coordinate systems used in this study.
3.3.2 Principle of Proposed Method
The three equations of the edge lines through the corner point in terms of image coordinates (u, v) are described by up +b vi p+ =ci 0, where i=1, 2, 3. The desired vehicle location will be described by three position parameters Xc, Yc, and Zc and two direction parameters ψ and θ, where Zc is the distance from the camera to the ceiling and is assumed to be known, θ is the pan angle of a camera, and ψ is the tilt angle of the camera with respect to the global coordinate system. The five vehicle location parameters can be derived in terms of the four coefficients , , and
of the edge line equations in the corner image. Finally the vehicle location could be estimated after the coefficients of the edge line equations are computed.
b1 c1 b2 c2
3.3.3 Location Estimation by Coefficients of Edge Line Equations
In this section, we will derivate the relation between the global coordinates and the coefficients of the edge line equations in the image coordinate system. At first, we transform the global coordinates into the camera coordinates. The transformation consists of four steps.
Step 1 Translate the origin of the global system by
(
X Y Zc, c, c)
to the origin of the camera system in the following way:( )
Step 4 Reverse the
Z
axis in the following way such that the positive direction of theZ
axis is identical to the negative direction of the W axis1 0 0 0 coordinates , the transformation in the two coordinate systems can be described in short as: following two equations to describe u
(u vp, p) P
p and vp according to the triangulation principle:
x Eliminating the variable x, we can get the equation for the projection of the
x Eliminating the variable x, we can get the equation for the projection of the