Chapter 3 Location Mapping Calibration and Mechanic Error
3.3 Proposed Mechanical Error Correction Method
3.3.5 Correction of Backing off
Due to the mechanic error the vehicle also backs off along a curve path. The navigation path when the vehicle backs off is shown in Figure 3.17. We use the same method as mentioned in last sections to correct the errors in this section.
Expected path Actual path
Figure 3.17 An illustration of Backing path.
As first, using curve fitting to compute the curve equation y = ax2 + bx + c, as shown in Figure 3.18. The equation is different from the equation computed when the vehicle moves ahead. The distance between the starting position and the now position is record as a negative value. The coefficients of a, b, and c are computed to be a = 0.0001, b = 0.002, and c = 0.0216, respectively, in this study.
Back path curve
-5 0 5 10 15 20
-450 -400 -350 -300 -250 -200 -150 -100 -50 0
cm
Figure 3.18 The path curve of backing off.
We still use the same method to correct the coordinates and direction angle of the vehicle. The correction equations are different from those of (3.10) and (3.11). The equation of computing the actual position (x′1, ) of the vehicle become as: y′1
1 y 0
x′ = − ⋅D cosθ−d ⋅sinθ+ ; x (3.17)
1 0
y′ = − ⋅D sinθ +d ⋅cosθ+ . y (3.18)
Finally, the process of backing along the straight line is the same with the process of moving ahead.
Chapter 4
A Method for Detection of Monitored Objects by Image Analysis
4.1 Introduction
We use a vision-based vehicle to monitor concerned objects and doors in this study. How to recognize objects and doors are an important problem in this study.
Hence, before describing the vehicle navigation process, we will first describe the image processing and pattern recognition techniques we use for solving this problem in this chapter.
There are two main processes to recognize an object in an image. One is to detect the object region, and the other is to compute the feature data of the object. To detect an object, we propose the use of an improved snake algorithm to accomplish the task.
The details of the method are described in Section 4.2.
After detecting objects, we have to compute useful feature data and save them for use in the learning process such that we can use them to monitor the objects in the security patrolling process. We will describe how to compute the feature data in Section 4.3.
4.2 Process of Monitored Objects Detection
In this section, we will describe how to detect an object region in an image. We tried to use a famous object detection technique, called “Snake Algorithm” [1] in this study. In Section 4.2.1, we will briefly describe the basic concept of the snake algorithm. But the snake algorithm has many weaknesses when it is used in the complicated environments; hence, we will propose an improved version of the method to detect objects in indoor environments in Section 4.2.2.
Some complicated environment backgrounds could influence the result of object detection; we will describe a feasible method proposed in this study in Section 4.2.4.
In Section 4.2.3, we will describe the details of the proposed improved snake algorithm.
4.2.1 Brief Description of Snake Algorithm
The performance of the snake algorithm is like the behavior of an elastic band.
Imagine that there is an elastic band in the image and an object exists in it. By repeating continuous operations, the elastic band becomes narrow and approaches the object edge. Finally, the elastic band will enclose the edge of the object and stop computing, as illustrated in Figure 4.1. We now describe the principle of the snake algorithm below.
The snake algorithm uses a set of control points, effectively connected by straight lines. Each control point has a position, specified by coordinates (x, y) in the ICS, and the snake is entirely specified by the number and coordinates of its control points. By means of adjusting the positions of the control points and computing the
energy of the snake at every moment, computation stops finally when the energy achieves the minimum value. The strength of traction is determined by the energy.
When the control points are moved toward their center continuously, some image properties will influence energy computation and cause the control points to stop moving.
(a) (b) Figure 4.1 Performance of the snake algorithm. (a) The behavior of elastic band and
an object in it. (b) Final result of repeating continuous computations.
The energy for a snake exists in two parts, the internal and external energies.
That is,
∫
∫
+= E (s) E (s)
Esnake internalv externalv (4.1)
where v(s) = (x(s), y(s)) which specifies image coordinates. The internal energy Einternal is the part that depends on the intrinsic properties of the snake, such as its length or curvature. The external energy Eexternal depends on factors such as the image structure, and the particular constraints the user has imposed. Each control point has its internal and external energies and Esnake is the total energy of all the control points.
The equation of the internal energy is written as
2 2
( ) ( )
internal s ss
E = ⋅α v s + ⋅β v s . (4.2)
The subscript s denotes the derivative. The above equation is the sum of the first derivative and the second derivative of the control point. Also, the values α and β are the coefficients determined by the designer.
The first derivate can be written as
2 2 2 2
From the above equation, we can understand that when two control points are more close, the value of the first derivate will be smaller. Hence, the first derivate accounts for stretching of the snake and will have a large value when there is a gap between the control points. So, when the distribution of the control points becomes smaller, the internal energy will have a smaller value, as shown in Figure 4.2 (a) and (b).
The second derivate can be written as
2 2
The second order term accounts for bending and will be large in a region of high curvature. When the edge of the object is smooth, the internal energy will have a smaller value, as shown in Figure 4.2(c) and (d). Therefore, when the control points are closer to each other and their distribution shape is smooth, a smaller internal energy is obtained.
(a) (b)
(c) (d) Figure 4.2 A illustration of internal energy. (a) A pattern has a larger internal energy.
(b) A pattern has a smaller internal energy. (c) A pattern has a larger internal energy. (d) A pattern has a smaller internal energy.
Now we will focus on the external forces on the snake. These determine its relationship to the image. Suppose that we want a snake to latch on to bright structures in the image. We usually use the magnitude of the gradient in image processing.
4.2.2 Description of Improved Snake Algorithm
In the last section, we have described the principle of the snake algorithm. We will now describe the proposed method in this section.
Before computing the Esnake energy, the locations of the control points have to be decided at first. In this study, the shape distributed by the control points is a rectangle at the beginning in the image plane. A larger object in the image will have a larger
locations of the control points, as shown in Figure 4.3. The center of the control point control points in the image plane. Each control point moves toward the center point in each cycle of computation. The moving path is computed as a vector in the following way:
After computing Esnake each time, the control points move to the center point gradually. The new position of the control points in each moving step becomes:
Through repeated computations, the shape of the control point distribution is no longer a rectangle. The distribution will be deformed gradually and the control point is close to the center point Pcenter(x , y0 0). Until the Esnake reaches a minimum value, .the process of the snake algorithm ends.
Figure 4.3 Contribution of control point.
The strength of shrinking comes from internal energy computation. Besides the method that Section 4.2.1 mentions, we add the so-called centripetal force into the internal energy. The improved internal energy is computed in the following way:
2 2 2 2
The detailed mathematical operation is specified by:
2 2
We utilize the distance between a control point and Pcenter 0, y0) as an element for computing internal energy. When the control point is closer to the point Pcenter(x0, y0), the distance becomes smaller gradually so that the internal energy has a smaller value. Hence, using centripetal force causes the control points to gather together.
The internal energy causes the control points to gather, so we have to compute the external energy so that the moving of the control points can be stopped. The final result is that the control points surround an object. The Sobel operator is shown in Figure 4.4 and its equation is shown in Eq.(4.10). The following specifies its operation formula:
−1 −2 −1 −1 0 1
denotes gray value in the position (u,v), z
where z5 1 denotes gray value in (u − 1,v − 1),
and so on.
By using the Sobel operator, the edge of an object becomes obvious, as shown in Figure 4.5. Hence, utilizing the value of the Sobel operator of the pixels on the control point’s position, we can get the external energy of a control point, as illustrated in the following way:
external =
E λ × [minus (value of Sobel operator of the control point in the image)
(4.11)
2]
where λ is a coefficient. Finally, the total energy, i.e., the snake energy, is computed as follows:
where n is the number of control points.
Because the data are discrete, the integration is substituted by a summation. The snake energy Esnake is the sum of each control point’s internal and external energy.
The internal energy is smaller when the control points are closer to the center such that the snake energy has smaller value. As soon as the control points bump into the edges, the external energy has a larger value such that the snake energy value
becomes larger; in the mean while the snake energy computation is stopped because the minimum snake energy is reached.
Figure 4.5 The result of using the Sobel operator.
The coefficients, α, β, γ, and λ play important roles in the snake energy. The magnitudes of the coefficients α and β influence the control points’ pulling force. The magnitude of coefficient γ influences the cohesion of the control points, and the magnitude of coefficient λ influences the result of the captured object edge. In this study, the magnitudes of α, β, γ, and λ are set to be 2, 3, 2 and 1, respectively.
4.2.3 Computing Object Region
After detecting an object, we have to recognize the object region from the image.
Through the above repeated calculations, the control points surround the object edge pixels finally. Utilizing the coordinates of two adjacent control points to compute a straight equation y = ax + b, we can find out the pixels of the object edge by using this equation. A detailed process is described in the following algorithm.
Algorithm 4.1. Object region extraction.
Input: A set of control points P.
Steps:
Step 1.2 Compute the value y of the edge point coordinates between P and Ps t
by using the value of x in the following way:
to x
y = ax + b, x = xt s. (4.14)
Step 2. Find out the leftmost pixel and rightmost pixel denoted as (xL, yL)and (xR, yR ) after comparing the x coordinate of each edge pixel.
Step 3. Find out the maximum and minimum y coordinates of each value of xi, denoted as (xui, yui) and (xdi, ydi).
(x , y ), if its x coordinate is between x and x
Step 4. For each pixel, Pi i i L R and its value
of y satisfies the following inequality, then P is considered as an object pixel: i
(4.15) ydi ≤ y ≤ yi ui.
4.2.4 Eliminating Noise of Background
Because of the effect of the noise and edges in the background, the experimental result of object detection is usually not satisfactory, as shown in Figure 4.6. If a wall or the floor under the object is not clear, it usually influences the result of the moving of the control points. Hence, we have to improve the method for object detection.
(a) (b) Figure 4.6 Effect of background on object detection (a) Before detection. (b) Result of
detection.
Besides the methods of Section 4.2, we add the strength of pull and drag between the control points. We hope that the distance of adjacent control points can keep inside a reasonable domain. In order to achieve this goal, the number of the control points is determined by a beginning rectangle size. According to the length of a rectangle’s boundary and the coordinates of the four endpoints of the rectangle, we compute every control point position on the edge of the rectangle so that every control point can keep a fixed distance to each other. Utilizing the distance of the control points, we can adopt a threshold Dt such that the distance between two adjacent control points are kept inside D when the control points are moving along the path. t
As illustrated in Figure 4.7(a), the control point , and the two points, and are adjacent points of P and by the following equations:
D1 D2
points Pi−1 and Pi+1, we can get a center position mi between them by using the
control point is changed to the new position specified byPi vi 4 Figure 4.7 Correction of control points. (a) Before correction. (b) After correction.
An experimental result of the above-mentioned method is shown in Figure 4.8.
(a) (b) Figure 4.8 Results of improved method. (a) Detecting a ball. (b) Detecting a tub
The vehicle sometimes will navigate on a decorative floor, as shown in Figure 4.9(a). Before detecting the object, we have to filter the floor region by using a region growing technique. A result is shown in Figure 4.9(b). When the position of control point is on the floor region, it is moved without being stopped until it leaves the floor region.
(a) (b) Figure 4.9 Elimination of a floor region. (a) A decorative floor. (b) An experimental
result.
4.2.5 Detailed Object Detection
We have already described the concept behind the proposed improved snake algorithm in the above sections. The detail of the proposed object detection process is described in the following algorithm.
Algorithm 4.2. Object detection.
Input: An object image I.
Output: An object region ObjR.
Steps:
Step 1. Specify manually four end points of a rectangle in the ICS as initial control points and compute the coordinates of the four points.
Step 2. Compute the coordinates of the other control points according to the coordinates of the four endpoints.
Step 3. Compute the center point Pcenter by using the coordinates of all the control points.
Step 4. Use a region growing technique to filter out the floor region.
Step 5. Compute the internal energy Einternal and the external energy Eexternal of every control point. And compute the original snake energy Esnake by summing up all the values of Einternal and Eexternal.
Step 6. For each control point, do the following steps.
Step 6.1. Compute next position of the control point on the shrinking path and its internal energy Einternal and external energy Eexternal and the new snake energy newEsnake by summing up Einternal and Eexternal.
Step 6.2. Compare the original snake energy Esnake with newEsnake.
Step 6.3. If original energy Esnake is smaller than newEsnake, the control point is not moved to the next position. Else moving the control point to the next position and substitute Esnake with the new snake energy newEsnake. Step 6.4. Compute the distances D and D1 2 between the control point and its two
adjacent points.
Step 6.5. Sum up D and D . If their sum is larger than a threshold D1 2 t, then adjust the coordinates of the control point by using the method described in Section 4.2.3 and compute the new snake energy.
Step 7. When reaching the minimum energy, stop moving the control points.
Step 8. Compute the object edges using the coordinates of the control points.
Step 9. Detect object region ObjR using the coordinates of the object edge points.
4.3 Object Feature Extraction
After detecting an object from an image, we hope to gather useful features of the object for object matching. Through comparing the object features with those of a
learned object, we can achieve the purpose of security patrolling. The object features for use in this study are color, shape, and coordinates in the GCS. Among the features, the coordinates are used as vehicle navigation data, and the others are used for object matching. In Section 4.3.1, we will illustrate how to use simple mathematics to describe color features. We will use an ellipse to fit the shape of each object, as illustrated in Section 4.3.2. Finally, a coordinate transformation is described in Section 4.3.3 for computing the coordinates of objects in the GCS.
4.3.1 Color Feature
Each object has its own colors which are the most useful feature. We can use it for object matching in the security patrolling process. About color data, we adopt the RGB color model and choose certain statistics to describe color features in this study.
After segmentation of object pixels, we compute the means of the R, G, and B colors, respectively, and denote them as Robj, Gobj, and Bobj. The means can be used to describe the whole color situation and are influenced slightly by noise or some fragmented edges. The equation is shown as follows:
)
Representing the color deviation of the object, we use the standard deviation to show this feature. We compute the standard deviations of the R, G, and B values, respectively, which are denoted as , , and . The equation for this is shown as follows:
sd
Robj Gobjsd Bobjsd
)
4.3.2 Ellipse Fitting for Shape Representation
Shapes of objects are usually different, such as circle, triangle, ellipse, rectangle, etc. Moreover, many object shapes are irregular. Hence, if all shapes can be represented by using only one shape, it will be convenient and fast. In this study, we use the ellipse shape to represent all object shapes. Although the shapes of objects are different, we always can compute an ellipse shape to fit the object shape.
2 1
According to the ellipse equation, x , there are two unknown variables
a and b which are the horizontal and the vertical axis length. We can distinguish different objects by different values of a and b. Because the center of the ellipse is usually taken to be the origin of the coordinate system, we must change the coordinates of the object pixels and the origin is moved to the center of object region such that the equation 22 + 22 =1
b y a
x can be used.
The center of the object region is denoted as , and calculated by the following equation:
where variables, x and yi i, are the coordinates of the object pixels in the ICS. For all the object pixels , they are moved to the new coordinates, denoted as and
, by using the equation and the least square method to solve variables a and b by the following equation:
x′i
)
Finally, we take a and b to represent this object. An experimental result is shown in Figure 4.10.
Figure 4.10 The ellipse representation of a safe.
4.3.3 Coordinate Transformation
The coordinates of an object are useful for navigation guidance. When the vehicle navigates along a route, it can detect the object according to coordinates of the learned object to help navigation on the right path. In this study, we use the coordinates of the object in the GCS as navigation data.
After detecting an object in an image, we use Equation (4.20) to compute the center of object pixels, such as the red point shown in Figure 4.11. By using a region-growing technique to find out the floor region in the image, the object can then be located on the floor, such as the green point shown in Figure 4.11. Computing the
coordinates of green points in the VCS by the 2D mapping technique as illustrated in Chapter 3, we can get relative positions of the vehicle and the object.
Using the coordinates and the direction angle of the vehicle in the GCS, we can compute the coordinates of the object in the GCS by the following equations:
Using the coordinates and the direction angle of the vehicle in the GCS, we can compute the coordinates of the object in the GCS by the following equations: