Chapter 1 Introduction
1.3 Overview of Proposed Approach
In this study, we try to design a vision-based vehicle system for security patrolling in indoor environments. In order to achieve the mission of security patrolling, to recognize surrounding objects and know the vehicle positions are necessary. The chief tools are the images captured by the camera and the odometer in the vehicle hardware system. An overall framework of the proposed system is illustrated in Figure 1.1.
The proposed vehicle system for security patrolling includes three major processes: setup of the vehicle system, learning of paths and objects by user control, and vehicle navigation.
Setup of the vehicle system is the basis of navigation. It includes location mapping calibration and mechanic error correction. After fixing the camera on the
vehicle, a mapping calibration technique for using 2D image analysis is used to calculate the distance between the vehicle and an object. On the other hand, because the vehicle is a machine, it is unavoidable that there exist mechanic errors. Although the odometer provides the position of the vehicle, the real position of the vehicle is different from the value provided by the odometer due to the mechanic error. By the way of correcting the value provided by the odometer, navigation can be made more accurate. A mechanic error correction model is proposed to solve the problem.
Moreover, the mechanic error causes the vehicle moving far away from original straight path. We propose a straight navigation technique to deal with this problem.
Before the vehicle navigates, how to detect objects from the image is essential.
Because the computer can not identify objects likes people, an improved snake algorithm is proposed in this study to detect objects in captured images. Although some researches [2-6] improve the snake algorithm, these methods are complicated and slow. We design a faster method to meet the real time requirement. In order to recognize objects, we utilize three kinds of object features in this study; they are color, shape, and coordinates. We propose three methods to compute these three features, respectively.
In the learning process, the vehicle is controlled to move to desired places by using a user interface designed in this study. Some coordinates of the vehicles which are denoted as nodes are recorded along the learning path. The user may identify objects or doors to be monitored from the images when the vehicle moves to neighborhood of objects. The system will record the features of the objects which are illustrated in last paragraph automatically.
According to the node data and the position of the objects and doors, a navigation map is created for use of the vehicle navigating automatically. The vehicle
them. As soon as the vehicle can not detect the monitored object, a warning message will be sent out from the system right away. When detecting the object, the system compares it with the learned data further. If the features are not correct, the same warning is announced. In the same manner, the vehicle moves to the neighborhood of the door to check whether the door is opened or not.
Due to the mechanic error, the vehicle might move far away from the original path gradually. We propose two methods to improve the stability of navigation. One is to use the coordinates of learned objects as an auxiliary tool to adjust the position and direction of the vehicle. The other is a line following technique. After completing the check of the objects and doors, one circle of security patrolling is finished.
Location mapping calibration and mechanic
error correction
Path learning by user control
Creation of learned path map
Security patrolling
Features of objects and doors saving
Figure 1.1 A flowchart of proposed system.
1.4
1.5
Contributions
Some major contributions of this study are listed as follows.
(1) A mechanic error correction method is designed.
(2) A technique for reducing mechanic error effect on straight navigation is proposed.
(3) A fast object detection method in indoor environments by using an improved snake algorithm is proposed.
(4) Three methods for object feature computing are proposed.
(5) A strategy for learning paths, objects, and doors is proposed.
(6) A method for path map creation by learned data is designed.
(7) A line following method which can reduce accumulative errors for navigation is proposed.
(8) Methods for object matching and door situation recognition are proposed.
(9) A method for correcting vehicle positions in the navigation session is proposed.
Thesis Organization
The remainder of this thesis is organized as follows. In Chapter 2, the system configuration of the vehicle and the principles of learning and navigation are described. In Chapter 3, the proposed methods for location mapping calibration and mechanic error correction are described. The proposed methods for object detection in the images and feature computing are described in Chapter4. The proposed learning strategy and path map creation method are described in Chapter 5. The proposed
recognition, and judgment of the door situation are described in Chapter 6. Some experimental results are given in Chapter 7. Finally, conclusions and some suggestions for future works are included in Chapter 8.
Chapter 2
System Configuration And Navigation Principles
2.1 Introduction
In order to achieve the goal of security patrolling by an autonomous vehicle, conditions around the vehicle are investigated. There are usually many obstacles and narrow paths in indoor environments. Hence, it is a wonderful choice that the size of a vehicle is small and its action is dexterous. The smaller the size of the vehicle, the larger range the vehicle can navigate. Moreover, if the size of the vehicle is small enough, it can monitor the space under tables, beds, cabinets, etc.
In this study, a small vehicle is used as a test bed. Beside the vehicle structure, a camera is installed on the vehicle. For a user to achieve the function of controlling the vehicle, some communication and control equipments are required. The entire hardware equipment and software used in this study are introduced in Section 2.2.
After introducing the equipment, the vehicle navigation principles and some detailed relevant processes are described in the following sections. In Section 2.3, we will describe the process of learning. Path and object data which are necessary for security patrolling are recorded in the learning process. In Section 2.4, we will describe the process of security patrolling in which the vehicle monitors concerned objects and checks whether a door is opened or not. As soon as an unusual situation is detected, a warning is issued.
2.2 System Configuration
The vehicle is shown in Figure 2.1.
(a)
(b)
Figure 2.1 The vehicle used in this study. (a) The front of the vehicle. (b) The flank of the vehicle.
Recently, wireless networking and communication techniques advance quickly.
More and more communication products are made with wireless transmission. In this study, we use an Amigo robot, a mini-vehicle made by ActiveMedia Robotics
Technologies Inc., on which a camera is installed, as the testbed of our research. We utilize wireless transmission equipment to control the vehicle and get images captured by the camera. We will describe the hardware system in Section 2.2.1 and the software used in this study in Section 2.2.2.
2.2.1 Hardware Configuration
The entire system is illustrated in Figure 2.2. The first part is the vehicle system.
The length, width, and height with the camera of the vehicle are 33cm, 28cm, and 21cm, respectively. There are two larger wheels and one auxiliary small wheel at the rear of the vehicle. The maximum speed of the vehicle is 75cm/sec and the maximum rotation speed is 300 degrees/sec. There are eight ultrasonic sensors, an odometer, and an embedded hardware system in this mobile vehicle. The ultrasonic sensors are not used in this study. The odometer provides the coordinates and direction of the vehicle in the navigation. The origin of coordinates is the starting position of the vehicle.
There is a 12V battery in the vehicle to supply the power of the vehicle system.
The second part is the vision system. There is a wireless 2.4GHz camera which can transmit analog signals on the vehicle. A receiver and an imaging frame grabber are connected to the computer. The receiver receives analog signals from the camera and a digital image can be obtained by the imaging frame. The image grabbed in our experiments is of the resolution of 320×240 pixels.
A personal computer with Centrino 1.4GHz CPU, a 768MB DDR RAM, and a 5400 rpm 40GB HDD is used as a remote control and monitoring system; a kernel program can be executed on it to give commands to the vehicle through a wireless transmission system.
There is one wireless device in the vehicle and another in the PC. The commands
that meets the IEEE 802.11b standard. By using the access point as a medium, the commands can be transmitted from the PC to the vehicle.
Figure 2.2 Structure of system.
2.2.2 Software Configuration
lications-programming interface for The ARIA is an object-oriented control app
Wireless Device
Access Point
Wireless Transmission System
PC
Remote System Vehicle
Camera 9V Battery
12V Battery Embedded Hardware System
Vehicle System
Imaging frame grabber Receiver Vision System
ActivMedia Robotics’ line of intelligent mobile robots. The ARIA is written in the C++ language. We use it to control and retrieve the status of the vehicle. The operation system in the PC is Microsoft Windows XP and we use Borland C++
Builder as the development tool in our experiments.
2.3 Learning Principle And Proposed Process of Learning
Before security patrolling, a learning process is necessary. Navigation path, object, and door data are recorded in advance. The entire learning process in this study is shown in Figure 2.3.
A user control interface is designed for use in controlling the vehicle and choosing monitored objects and doors. The user controls the vehicle to navigate in indoor environments and move to the front of objects for choosing monitored objects.
The main recorded data include two categories, path and object data. The door is considered as an object in this study. As soon as the learning process ends, all data are stored in the storages of the computer such that the learning process is only executed once and data can be used repeatedly.
When the vehicle patrols in rooms and corridors, the computer records the path data provided by the odometer, and denote them as nodes. The monitored objects are selected in the image captured by the camera when the user controls the vehicle to patrol. The features of the objects are also computed automatically from the images.
There are three kinds of object features are recorded in this study. They are color, shape, and coordinates respectively.
The process of door learning is similar to that of ordinary object learning. The computer can record more than one object when the vehicle is at a single position. By means of turning the vehicle at same position, we can learn neighboring objects of the vehicle. The user can choose objects continuously along the path until finishing a learning process.
Start learning
Manual control
Data collection
Figure 2.3 A flowchart of proposed learning process.
End learning
Odometer data
Feature data of objects and doors
Saving data in the storage Choosing objects
and doors Driving vehicle
Map created by learned data
After finishing learning process, we can get a navigation map by way of combining the path data and object positions. The vehicle utilizes this map to monitor the doors and objects consecutively along the navigation path.
2.4 Vehicle Guidance Principle And Proposed Process
In the navigation process, the vehicle monitors the objects and doors consecutively by using the map created from learned data. The vehicle moves to a fixed position and stops to check the existence of the monitored objects. As soon as an unusual situation is occurred, a warning is issued. After checking the objects at a location, the vehicle moves to the next position where there are monitored objects or doors, and then continues checking the existence of the monitored objects or whether the door is opened. An illustration of the vehicle navigation process is shown in Figure 2.4.
In order to patrol along the learning path, the vehicle moves along each node in the map one by one. To move from a node to another, a line following technique is used to reduce navigation errors.
When the vehicle moves to a node where there is a nearby monitored object, the computer detects the object by using the position of the learned object. As soon as the computer detects an object from the image, its features are compared with those of the learned objects. If the features are incorrect, a warning is issued. Of course, when the computer can not detect the object, the same warning is also issued. Because the vehicle might navigate far away from the original path gradually, we use a
vision-based technique to reduce accumulative errors for navigation. When the vehicle detects an object and consider it as a learned object, we use its position recorded in the learning process to correct the position of the vehicle for reducing accumulative errors.
Start navigation
Vehicle patrolling
Figure 2.4 A flowchart of proposed navigation process
Although the door is considered an object, the method of recognizing the door End navigation
Door identification Object detection
Whether correct Whether closed
Whether unchecked objects
No Yes
A warning is announced
Yes No
Yes
Correct the vehicle position
No
Navigating unfinished path
situation is different from the method of object matching. After the vehicle moves to a suitable position recorded in the learning process, the computer checks the door situation to see if it is opened or closed. After the vehicle patrols the entire route and checks all objects and doors, a security patrolling process is finished.
Chapter 3
Location Mapping Calibration and Mechanic Error Correction
3.1 Introduction
When a vehicle navigates in indoor environments, the position and distance are important data for the navigation. By using these data, the vehicle can arrive at suitable positions to search monitored objects or doors and navigate along a learned path. In order to get position data and distance values, a camera and an odometer are used in this study, which are equipped on the vehicle.
At first, we utilize an image captured by the camera to obtain the relative position between an object and the vehicle. We use a 2D mapping method to achieve this goal. The detailed process is described in Section 3.2.
The odometer provides the positions of the vehicle in the environment. It records the coordinates and the direction angles of the vehicle when the vehicle navigates.
Possible mechanic errors might cause the real positions of the vehicle to be unequal to the expected values provided by the odometer. Hence, in Section 3.3, we will propose a correction method to reduce the mechanic error so that the vehicle can navigate more stably.
Before describing the above-mentioned methods, we first introduce some coordinates system and the definition of the direction angle of the vehicle for use in this study. We introduce the coordinate systems in Section 3.1.1 and the definition of
the direction angle of the vehicle in the space is described in Section 3.1.2. By using the direction angle, a transformation function between the coordinate systems is described in Section 3.1.3.
3.1.1 Coordinate Systems
Three coordinate systems are utilized in this study to describe the locations of the vehicle and objects. The coordinate systems are shown in Figure 3.1. The definitions of these systems are stated in the following.
u v
I
Room
x y
G
(a) (b)
Vy
Vx
V
Vx
Vy
V
(c)
Figure 3.1 Three coordinate systems in this study. (a) ICS (b) GCS (c) VCS.
(1) The image coordinate system (ICS): denoted as u−v. The u−v plane is coincident with the image plane and the origin I of the ICS is placed at the center of the
image plane.
(2) The global coordinate system (GCS): denoted as x-y. The x-axis and the y-axis are defined to lie to on the ground, and the origin G of the global coordinate system is a pre-defined point on the ground. In this study, we define G as the starting position of the navigation.
(3) The vehicle coordinate system (VCS): denoted as Vx-Vy. The Vx-Vy plane is coincident with the ground. The Vx-axis is parallel to two wheels of the vehicle.
The Vy-axis is parallel to the body of the vehicle. And the origin V is placed at the middle of the line segment that connects the two contact points of the two driving wheels with the ground.
3.1.2 The Direction Angle
The direction angle of the vehicle is defined in the global coordinate system, the x-y plane. The angle denoted as θ represents the rotation degree of the vehicle in the global system and plays a very important role in the coordinate transformations.
θ is the angle between the positive direction of the x-axis and the front of the vehicle. The direction angle θ is set to be zero at the beginning of navigation. The range of θ is between 0 and π if θ is in the first and second quadrants, as illustrated in Figure 3.2(a) and (b). It is between 0 and −π if θ is in the third and forth quadrants, as illustrated in Figure 3.2(c) and (d).
3.1.3 Coordinate Transformation
By using the direction angle and the coordinates of the vehicle in the GCS, we can describe the coordinate transformation between the vehicle coordinate system and the global coordinate system by the following equations.
V cosx Vy sin p x= × θ − × θ+ x
V sinx Vy cos p y= × θ + × θ+y
where xp and yp are the coordinates of the vehicle in the GCS. It is shown in Figure 3.3.
θ
x y
θ
0
π/ 2
π/ 2
− πor−π
(a) (b)
θ
x
θ 0
π/2
π /2
− π or−π
(c) (d) Figure 3.2 Definition of the direction angle (a)(b) θ is positive. (c)(d) θ is negative.
x y
GCS
θ
Vx
Vy
Figure 3.3 The coordinate transformation between the GCS and VCS.
3.2 Location mapping Calibration
The camera is the only sensor of the vehicle to gather the features of the environment, and the techniques of learning and guidance are based on visual perception. Location mapping calibration and image analysis techniques are indispensable in this study. A real location data acquisition method by Location mapping calibration and image mapping is proposed to obtain the relative positions of the vehicle and the surrounding environment precisely.
We use a point set P = {P00, P01, …,Pmn} whose coordinates are known in advance in the VCS attached on the floor, and their corresponding point set V = {V00, V01, …,Vmn} appearing in the image, to compute the VCS coordinates of a set of pixels. The detailed process is described in the following algorithms.
Algorithm 3.1. Real location data acquisition by image taking and mapping.
Input: An image I, as shown in Figure 3.5(a).
Output: A point set V = {V00, V01, …,Vmn} and another point set P = {P00, P01, …,Pmn} Steps:
Step 1. Attach some straight lines on the floor, as shown in Figure 3.4. Set the length between every two vertical lines to be 30 cm and the length between every two horizontal lines to be 30 cm, too, where every rectangle is a 30×30 square. There exists a distance Div equal to 80cm in the front of the vehicle, which the camera can not view.
Step 2. For those tessellated regions shown in the image which are not surrounded by the lines, extend the lines such that every quadrilateral is complete. Mark red points on the intersections of the lines, as shown in Figure 3.5(b).
Step 3. Record the coordinates of each red point Pij(uij, vij) in the ICS and group all such points into the set P.
Step 4. Measure relative coordinates Vij(xij, yij) manually in the VCS of the point Pij
and group such points into the set V.
Div
Figure 3.4 An illustrated of attaching the lines on the floor.