Chapter 4 A Method for Detection of Monitored Objects by Image
4.3 Object Feature Extraction
4.3.2 Ellipse Fitting for Shape Representation
4.3.2 Ellipse Fitting for Shape Representation
Shapes of objects are usually different, such as circle, triangle, ellipse, rectangle, etc. Moreover, many object shapes are irregular. Hence, if all shapes can be represented by using only one shape, it will be convenient and fast. In this study, we use the ellipse shape to represent all object shapes. Although the shapes of objects are different, we always can compute an ellipse shape to fit the object shape.
2 1
According to the ellipse equation, x , there are two unknown variables
a and b which are the horizontal and the vertical axis length. We can distinguish different objects by different values of a and b. Because the center of the ellipse is usually taken to be the origin of the coordinate system, we must change the coordinates of the object pixels and the origin is moved to the center of object region such that the equation 22 + 22 =1
b y a
x can be used.
The center of the object region is denoted as , and calculated by the following equation:
where variables, x and yi i, are the coordinates of the object pixels in the ICS. For all the object pixels , they are moved to the new coordinates, denoted as and
, by using the equation and the least square method to solve variables a and b by the following equation:
x′i
)
Finally, we take a and b to represent this object. An experimental result is shown in Figure 4.10.
Figure 4.10 The ellipse representation of a safe.
4.3.3 Coordinate Transformation
The coordinates of an object are useful for navigation guidance. When the vehicle navigates along a route, it can detect the object according to coordinates of the learned object to help navigation on the right path. In this study, we use the coordinates of the object in the GCS as navigation data.
After detecting an object in an image, we use Equation (4.20) to compute the center of object pixels, such as the red point shown in Figure 4.11. By using a region-growing technique to find out the floor region in the image, the object can then be located on the floor, such as the green point shown in Figure 4.11. Computing the
coordinates of green points in the VCS by the 2D mapping technique as illustrated in Chapter 3, we can get relative positions of the vehicle and the object.
Using the coordinates and the direction angle of the vehicle in the GCS, we can compute the coordinates of the object in the GCS by the following equations:
cos sin
obj obj obj car
Gx =Vx ⋅ θ −Vy ⋅ θ +Gx ; (4.23)
sin cos
obj obj obj car
Gy =Vx ⋅ θ +Vy ⋅ θ +Gy , (4.24)
where Gxobj and Gyobj are the coordinates of the object in the GCS, Vxobj and Vyobj are the coordinates of the object in the VCS, and Gxcar and Gycar are the coordinates of the vehicle in the GCS. Finally, θ is the direction angle of the vehicle in the GCS.
(a) (b)
(c) (d) Figure 4.11 The locations of objects (a) An original image in which a ball is on the
floor. (b) An original image in which a TV is on the wall. (c) An experimental result of (a). (d) An experimental result of (b).
Chapter 5
Learning Strategies for Indoor Navigation by Manual Driving
5.1 Introduction
Before the vehicle navigates, it is a necessary process that we control the vehicle to record the paths and objects. Because indoor environments are usually complicated and objects are placed at different positions, building a complete navigation map is necessary. Hence, to create the navigation path and choose monitored objects is a primary work of security patrolling by vehicle navigation. We will describe how to build navigation data by manual driving in this chapter.
In Section 5.2, we will first describe the control rules and the entire manual learning process simply. It includes all the steps that the user may use a control interface to control the vehicle to patrol in indoor surroundings and point out which object has to be monitored by using the control system.
Two kinds of navigation data are used in this study. One is path data and the other object data. Although we can get the position of the vehicle by the odometer value any time, how to represent the entire path by using useful and simple values is a problem. In Section 5.3, we will describe how to gather path data when the user controls the vehicle to navigate in indoor environment. Every object has its own color and shape; if different kinds of objects use different learning methods, the work will be annoying to the user. We design a simple object learning method to solve this
problem in this study, as illustrated in Section 5.4. The door is considered as an object, and monitoring of it is also described in Section 5.4.
After learning whole data, we have to utilize the data to build information for security patrolling. In Section 5.5, we will describe how to use path data and the positions of learned objects to create a navigation path which is then used when the vehicle navigates automatically.
5.2 Control Rules And Entire learning Process
In this study, the user controls the vehicle to navigate by the following fives types of actions.
(1) Moving forward.
(2) Moving backward.
(3) Turning left at the original position.
(4) Turning right at the original position.
(5) Stop.
Because the vehicle only owns three wheels and the rear wheel is an auxiliary wheel, rotation of the vehicle depends on the two side-wheels. Also, the vehicle can rotate at any position. Therefore, we can control the vehicle to move to the neighborhood of an object and a door, and let it turn to the front of the object or the door to learn relative object features.
The control rules for the vehicle are described as follows.
(1) When a user wants to turn rightward or leftward, the vehicle should be made still
first. If the vehicle is moving, the user has to stop the vehicle and then turn the vehicle.
(2) Because the view of camera is fixed, there is a limit of distance in the front of the vehicle, 60cm, in which the camera can not take clear images of the scene.
Hence, the distance between the vehicle and a learning object has to be kept larger than 60cm.
(3) The process of object learning should be done when the vehicle is still.
(4) The object region should be around the center of an image.
The entire learning process is described as following algorithm and a example of learning process is shown in Figure 5.1.
Figure 5.1 Proposed learning process.
Algorithm 5.1. Learning process.
Input: The user control interface in the PC.
Output: Learned data.
Steps:
Step 1. Control the vehicle to move from a start position using the control interface.
Step 2. Let the vehicle move to the front of a monitored object or a door.
Step 3. Choose the monitored object by using the user control interface in the control system, a personal computer in this study. Compute the features of objects or doors, and show the result on the control interface.
Step 4. If the learning result of the object and door are not satisfactory, repeat Step 3 until satisfactory, and then save the features of the objects and doors.
Step 5. Repeat Steps 2 through 4 until all the features of the monitored objects are grasped and saved.
Step 6. If another object is to be learned, repeat Steps 2 through 5; else, continue following steps.
Step 7. Decide whether the learning process should be continued, and continue driving and repeat Step 2 through 6 if so; else control the vehicle to move to the destination and finish the learning process.
Step 8. Save the learned data.
Store navigation data in the storage
Figure 5.2 Flowchart of manual learning process.
5.3 Process of Learning Navigation Paths
When the vehicle moves along a path, the odometer values are changed continuously. The odometer provides the rotation angles and the vehicle coordinates (x, y) with the coordinate origin being the beginning position of the vehicle. We only record vehicle coordinates as path data in this study.
Although the coordinates are changing all the time, we just save some coordinates (x, y) which are called node Ni in this study. Two types of data are stored in node Ni. Besides the coordinates, a number denoted as NNumber and used to mark the order of the object is also saved. The number is computed from 0.
Saving the path data is different from saving the object data. The user has to point out an object manually in the acquired image on the interface in the object learning process, but the user does not have to do so in the path learning process.
When the user controls the vehicle to move ahead or back, the vehicle system will automatically collect values of the coordinates (x, y) and the moving direction.
Each node of a path is marked with a serial number. After finishing learning, we have a set of notes, denoted as Npath. The process of recording the path data is described as an algorithm in the following.
Algorithm 5.2 . Path node collection.
Input: The coordinates provided by the odometer in the vehicle.
Output: A set of nodes denoted by Npath ={N0, N , N , …, N }. 1 2 t
Steps:
Step 1. Record the first node as (x0, y0) = (0, 0) into the set Npath and mark the node
as N with index 0, when the vehicle is at the starting position. 0
into the set N
Step 2. Record the node N (x , y )i i i path by taking the values of the odometer (x, y) and mark the node Pi with the next index number, when the vehicle is at one of the following three situations:
(1) when the direction angle and coordinates are corrected, as described in Section 3.3;
(2) when the user controls the vehicle to turn;
(3) when the user controls the vehicle to learn the data of certain objects.
into the set N
Step 3. Record the finally node Nt path and mark it as Nt by the next index number when the learning process is finished.
Step 4. Save all the nodes of the set Npath into the PC.
According to Section 3.3, the navigation path of vehicle is a curve when the vehicle moves ahead. In the learning process, the direction angle and coordinates of the vehicle are corrected automatically. In this study, we design the direction angle and coordinates to be corrected once each time when the vehicle moves for the distance of 250cm. We record a node each time the vehicle comes to a stop in a learning course. Besides the start position which the vehicle begins to move ahead, there are two types of nodes which should be recorded. One is the position where the vehicle turns back to a straight path, as denoted by node s1 in Figure 5.3(a). The other is the position where the vehicle already comes back to a straight path, as denoted by node s2 in Figure 5.3(a). Recorded nodes are shown in Figure 5.3(b) when the vehicle navigates along a straight path. And we use the same method to record nodes when the vehicle moves backward.
As an illustration of the result of applying Algorithm 5.2, we show an example of recorded nodes in Figure 5.4 that a navigation path shown in Figure 5.1 mentioned in Section 5.2. We can see that all critical nodes of the three situations are recorded in
addition to the start and the end nodes. All nodes are marked with index numbers according to the order of patrolling. And the start node and the end node are the same node in this example. The index numbers are useful for path map creation and object detection. We will describe them in detail in Section 5.5.
(a)
(b)
Figure 5.3 Recorded nodes when the vehicle moves straightly. (a) Nodes in one circle of the navigation correction. (b) Nodes in a longer path.
Figure 5.4 An experimental result of path learning with critical nodes recorded.
5.4 Process of Learning Monitored Objects and Doors
5.4.1 Process of Learning Objects
When the user controls the vehicle to move to the front of objects, the user must use the mouse connected to the PC to choose an object which appears in the image.
As soon as the user chooses the object, the object data are computed automatically and saved. The set of object data is denoted by LearnOi and i = 1, 2, 3, …, n. There are six kinds of data to be saved in LearnO . They are: i
(1) The color set denoted as LearnCobj,i. (2) The shape set denoted as LearnSobj,i.
(3) The GCS coordinate set denoted as LearnGCobj,i.
(4) The ICS coordinate set of object centers denoted as LearnCICobj,i. (5) The set of light source of the floor, denoted as LearnCfloor,i.
(6) The number of path nodes NodeNumber.
Hence, we have LearnO = {LearnCICi obj,i, LearnCobj,i, LearnSobj,i, LearnGCobj,i, LearnCfloor,i, NodeNumber}.
The methods for computing the color, shape, and coordinates of the object have already been described in Section 4.4. After the vehicle records all the data of monitored objects LearnO , we save the data into the set LearnOi object. The entire learning process of the object is described as follows.
Algorithm 5.3 . Learning of object features.
Input: A color image I captured by the camera on the vehicle.
Output: A set of object data LearnOobject = {LearnO , LearnO , LearnO ,…, 0 1 2
LearnO }. n
Steps:
Step 1. Control the vehicle to move to the front of the first object and turn toward it such that the image of the object can be taken.
Step 2. Use the mouse to choose an object from the image, as shown in Figure 5.5(a) and enclose the object by a rectangle.
Step 3. Release the button of the mouse for the computer to perform the improved snake algorithm to capture the object, as shown in Figure 5.5.
Step 4. Compute the elements of the object data in LearnO as follows: 0
Step 3.1. Record the object center coordinates in the ICS into LearnCICobj,i
by using Equation (4.5) with LearnCICobj,i computed by
LearnCICobj,i = { Learn_uobj_centerj,i, Learn_vobj_centerj,i }. (5.1) Step 3.2. Record the mean and standard deviation values of the R, G, and B
values into the color set LearnCobj,i by using Equations 4.13 and 4.14 with LearnCobj,i computed as
, , ,
LearnC LearnR LearnG LearnB LearnR LearnG
LearnB
= ,,
(5.2)
Step 3.3. Record the shape data of the object into LearnSobj,i in terms of the horizontal axis Learn_aobj,i and the vertical axis Learn _bobj,i as illustrated in Section 4.4.2:
LearnSobj,i = {Learn_aobj,i, Learn _bobj,i}. (5.3) Step 3.4. Record the coordinates of the object in the GCS into the set
LearnGCobj,i in terms of Learn_xobj,i and Learn_yobj,i as illustrated in Section 4.4.3:
LearnGCobj,i = { Learn_xobj,i, Learn_yobj,i}. (5.4)
, floor i
LearnR LearnGfloor i,
LearnCfloor,i, denoted by , , and
, floor i
LearnB .
, , ,
, { floor i, floor i, floor i}
floor i
LearnC = LearnR LearnG LearnB . (5.5)
Step 3.6. Record the number of path nodes NodeNumber which the vehicle has located.
should be saved into LearnO Step 5. Decide whether the data of object LearnO0 object
or not. If the result of object segmentation is not satisfactory, repeat Steps 2 through 4.
into LearnO
Step 6. Save the object data LearnO0 object and control the vehicle to move to the next object and repeat Steps 2 through 5 to save object data LearnO into LearnOi object until all data of monitored objects are collected and saved.
Step 7. Save LearnOobject into the PC and finish the learning process.
(a) (b) Figure 5.5 Learning process of choosing an object manually. (a) Choosing an object.
(b) An experimental result of computing object features data.
Because of the lighting effect of the environment, the R, G, and B values of an object are not always the same when the camera takes the object images at different
times. It causes erroneous results when the vehicle navigates using the object color feature to conduct object matching. We utilize an offsetting technique to solve this problem. We record the color features of a floor area in an object learning process. In Figure 5.6, the red area is the floor area, and we compute the R, G, and B means of the interior area of the rectangle. As the object matching process is done when the vehicle navigates, we utilize the differences of the R, G, and B values of the floor region to modify the thresholds used in the object matching process.
Figure 5.6 A selected floor region.
5.4.2 Process of Learning Doors
When a user controls the vehicle to move, he/she can choose doors as monitored objects. After the features of a door are recorded, the vehicle can check their situations in a security patrolling navigation. When the vehicle learns the features of a door, we record its color data (denoted as LearnCdoor,i) and coordinates (denoted as LearnGCdoor).
All door data are saved in a set Di and the date sets of all doors in the navigation are saved in a set LearnD = {D , D , D , …,D0 1 2 n}. The entire learning process for a door is described in the following.
Algorithm 5.4. Learning of a door.
Input: An image I, and the number of nodes NNumber.
Output: A set of door data D = {LearnCi door,i, LearnGCdoor,i, NodeNumber, baseline}.
Steps:
Step 1. Point out a door on the image I using a cursor, as shown in Figure 5.7(a).
Step 2. Use a region growing technique to find out the door region DoorR, as shown in Figure 5.7(b).
(a) (b) Figure 5.7 A learning process of choosing a door manually. (a) Choose a door. (b) A
door region is shown.
Step 3. Compute the means of the R, G, and B values of DoorR and save them in the set LearnCdoor as follows:
, , ,
, { floor i, floor i, floor i}
door i
LearnC = LearnR LearnG LearnB . (5.6)
step 1. Compute the coordinates of the door in the GCS and save them in the set LearnGCdoor,i as follows:
, { _ ,, _
door i door i door i
LearnGC = Learn x Learn y ,}. (5.7)
Step 4. Record the index number of the node as NodeNuber = NNumber.
Step 5. Detect the edges of I by applying the Sobel operator to get an edge image S.
Step 6. Detect the door edge edoor, the right baseline edge er, and the left baseline edge e of the door, as shown in Figure 5.8. l
Step 7. Compute the slopes adoor, a , and a of er l door, e , and er l, using a line fitting technique by the following equation:
2 2
( )
e e e
e e
u v n uv
a x n x
−
= −
∑ ∑ ∑
∑ ∑
(5.8)Step 8. Compare a and a with ar l door by the following equation:
(5.9)
| a − ar door | ≤ | a − al door |.
If the above equation is true, then set baseline = 1; else, set baseline = 0. The variable baseline is used to illustrate which baseline is used in the navigation process.
As shown in Figure 5.8, we utilize a baseline of a wall and compare it with the edge of the door to test whether it is open or not, as done in Step 9 in the above algorithm. Hence, if the edge of the wall is not parallel to the door edge when the door is closed, we can not use its baseline to compare with the door edge. So we compute the slopes of two baselines and the door edge, and choose a baseline whose slope is close to the door edge for use in the comparison. The slope of er is closer to the slope of edoor than that of e , as seen in Figure 5.8. l
Figure 5.8 An experimental result of detection of three edges.
5.5 Process of Automatic Path Map Creation from Learned Data
After ending the learning process, the path data Npath, the object data LearnOobject, and the door data LearnDdoor are already saved. We use the index number NNumber of each node and the total number of nodes NodeNumber to create a path map for later navigation sessions, as shown in Figure 5.9. By using the index numbers NNumber of the nodes, the vehicle can move along the navigation path.
After ending the learning process, the path data Npath, the object data LearnOobject, and the door data LearnDdoor are already saved. We use the index number NNumber of each node and the total number of nodes NodeNumber to create a path map for later navigation sessions, as shown in Figure 5.9. By using the index numbers NNumber of the nodes, the vehicle can move along the navigation path.