Autonomous Land Vehicle
Navigation in Indoor
Environments by Person
Following Using Sequential
U
Pattern Recognition
䢇 䢇 䢇 䢇 䢇 䢇 䢇 䢇 䢇 䢇 䢇 䢇 䢇 䢇 䢇 䢇 䢇 䢇 䢇 䢇 䢇 䢇 䢇 䢇 䢇 䢇 䢇 䢇 䢇 䢇 䢇 䢇 䢇 䢇 䢇
† Ching-Heng Ku, Wen-Hsiang Tsai
Department of Computer and Information Science National Chiao Tung University
Hsinchu, Taiwan 300 Republic of China
Received November 7, 1997; revised October 17, 1998; accepted December 5, 1998
Ž .
A new approach to autonomous land vehicle ALV navigation by the person follow-ing is proposed. This approach is based on sequential pattern recognition and com-puter vision techniques, and maintenance of smoothness for indoor navigation is the main goal. The ALV is guided automatically to follow a person who walks in front of the vehicle. The vehicle can be used as an autonomous handcart, go-cart, buffet car, golf cart, weeder, etc. in various applications. Sequential pattern recognition is used to design a classifier for making decisions about whether the person in front of the vehicle is walking straight or is too right or too left of the vehicle. Multiple images in a sequence are used as input to the system. Computer vision techniques are used to detect and locate the person in front of the vehicle. By sequential pattern recognition, the relation between the location of the person and that of the vehicle is categorized into three classes. Corresponding adjustments of the direction of the vehicle are computed to achieve smooth navigation. The approach is implemented on a real ALV, and successful and smooth navigation sessions confirm the feasibility of the approach.
䊚 1999 John Wiley & Sons, Inc.
U
This work was supported by the National Science Council under Grant NSC86-2213-E009-114.
†
To whom all correspondence should be addressed.
( ) ( )
Journal of Robotic Systems 16 5 , 249᎐262 1999
1. INTRODUCTION
Many methods1᎐ 8 have been proposed for ALV navigation in indoor environments. These methods depend on special features in environments, such as lines,1,2 corners,3,4 models,5,6 or special
land-marks.7,8 When an ALV navigates in different
in-door environments, these features may not always be available. In such cases, the ALV has to change its navigation strategy to suit new environments. In this study, a navigation method by a person follow-ing is proposed, by which a person can lead the ALV to any place the person wants to reach, and this creates many possible applications of the ALV system. The system can be used as an autonomous handcart, go-cart, buffet cart, shopping cart, dust cart, golf cart, weeder, etc. Besides, the system can also be used as a learning system to collect informa-tion in certain environments, including open paths or locations of obstacles. On the other hand, because the indoor environment is usually complicated and small, the navigation methods of the road follow-ing,9,10 vehicle following,11,12 or car following13᎐ 15 proposed for outdoor environments are usually un-suitable to indoor navigation. Navigation by the person following instead is more convenient in such cases.
In the area of human body tracking, some meth-ods have been proposed, such as stereo and motion measurements,16 multiclass statistical modeling of
17 Ž
color and shape, and the so-called Pfinder
‘‘per-. Ž .18
son finder’’ and Spfinder ‘‘stereo person finder’’ . These methods are not useful for autonomous navi-gation because the detection of an entire human body shape spends a lot of time and is not necessary for real-time navigation. In this study, a special shape attached on the back of the person is used for fast detection and tracking of the person to shorten the processing time.
Besides, some systems19᎐ 21 for following a
per-son have been proposed. Matsuda et al.19 proposed
a tracking control system for an autonomous mobile robot with multiple sensors in which supersonic distance sensors and infrared direction sensors are configured. Hanebeck and Schmidt20 proposed a
high performance multisonar system which can be used for tracking a walking person. These two sys-tems do not use the information of the visual sen-sor. Mori and Sano21 proposed a guide dog robot which follows a moving pedestrian by tracking the jacket or trousers image of the person. This system does not use the information of the relation between consecutive images.
In this study, a new approach to the person following for autonomous land vehicle navigation using image sequences is proposed. The major goal is to achieve smooth navigation by sequential pat-tern recognition techniques. To guide the ALV to follow the person who walks in front of the vehicle, the location of the person is detected from input images first. The relation between the location of the person and that of the vehicle is then categorized into three classes. The first class is that the location of the person is too right or too left to the vehicle. The second is that both the person and the vehicle move in a straight trajectory. In addition, the last class is that the location of the person does not belong to the above two classes. In the sequential pattern recognition process, two features are pro-posed for use in a quadratic classifier. According to the classification result, corresponding adjustments of the direction of the vehicle are proposed. The proposed method can solve the problem of the drift location due to small differences between the vehi-cle heading and the heading of the person. The direction of the vehicle is not adjusted in the case of drift location when the location of the person is close to the trajectory of the vehicle. The vehicle keeps its direction until the location of the person is too right or too left of the vehicle. The adjustment of the ALV speed is implemented using the fuzzy technique.22 It is hoped that the ALV could navigate smoothly when the person walks in different direc-tions, so that the turning angle of the front wheels of the ALV need not be adjusted immediately in every cycle. Furthermore, due to the use of a rectan-gular shape on the back of the person, the comput-ing time for the detection of the location of the person is shortened and the walking direction of the person is easily detected. In short, the main contri-bution of this study is the use of sequential pattern recognition and the detection of the simple shape to achieve fast navigation with smooth trajectories.
A flowchart of the person following system is shown in Figure 1. The navigation system is com-posed of a vision system, a decision system, and a control system. In the vision system, input images are captured by a color CCD camera that is mounted on the vehicle. Additionally, image processing and computer vision techniques are employed to detect feature points from the image of the person who walks in front of the vehicle and calculate the three-dimensional coordinates of the feature points. The location and speed of the person are computed accordingly and used in the control system. In the decision system, the sequential pattern recognition
Figure 1. Flowchart of the proposed system.
technique is used to decide whether the person walks straight or is too right or too left of the vehicle. The control system is triggered by the deci-sion system. The three-dimendeci-sional coordinates of the location of the person are used in the control system to control the turning angle of the front wheels of the ALV. Besides, the speed of the person and the distance between the person and the vehicle are used to adjust the speed of the vehicle by a fuzzy control technique.22 A more detailed
de-scription of the navigation process is described as follows.
Step 1. Image acquirement: Acquire an image of the front view of the vehicle, which in normal cases, includes the person who walks in front of the vehicle.
Step 2. Feature point detection: Detect some feature points of the rectangular shape attached on the back of the person by a region growing tech-nique using the color information.
Step 3. Coordinate transformation: Transform the image coordinates of the feature points obtained in Step 2 into the 3D space coordinates to calculate the speed and location of the person.
Step 4. Sequential pattern recognition: Ap-ply the sequential pattern recognition technique to make a decision about whether the person walks straight or is turning, based on the use of a quadratic classifier.
Step 5. Speed calculation of the person: Calcu-late the speed of the person from the locations of the
person in consecutive cycles as a reference for ad-justing the speed of the vehicle.
Step 6. Wheel angle control: Adjust the turning angle of the front wheels of the vehicle according to the location of the person when the system decides that the person is turning.
Step 7. Vehicle speed control: Apply the fuzzy control technique to adjust the speed of the vehicle according to the speed of the person and the dis-tance between the person and the vehicle.
In the remainder of this article, the proposed person following technique is described in section 2, including a review of the sequential pattern recogni-tion technique, the descriprecogni-tion of the proposed clas-sifier, and the features used for the classifier. Wheel angle adjustment strategy is described in section 3. Employed image processing techniques for the de-tection of feature points are described in section 4. Experimental results are found in section 5. Conclu-sions are given in section 6.
2. PROPOSED DECISION MAKING BY SEQUENTIAL PATTERN RECOGNITION
In the process of ALV navigation by the person following, the turning angle of the front wheels of the vehicle is decided not only by the location of the person but also by the turning status of the person. When a person means to walk straight, hershe may not be able to march in a real straight trajectory. That is, the location of the person may not be always right in front of the vehicle and might some-times be a little bit on the right side or on the left of the moving direction of the vehicle. In such cases, the turning angle of the front wheels of the vehicle should keep at 0⬚. It is undesirable to turn the ALV when the person is not too left or too right of the direction of the vehicle. On the contrary, if the location of the person is too left or too right, the result of the classifier should mean to adjust the direction of the ALV even when the heading differ-ence is small. In short, the main concept here is that the direction of the vehicle should not be adjusted even when there is a small difference between the vehicle heading and the heading of the person.
To implement the above idea, we develop in this study a method based on the sequential pattern recognition technique to decide whether the person walks straight or not. This method does not take as input just a single location of the person; instead, a sequence of the locations of the person extracted
from an image sequence is used to provide suffi-cient information for decision making. The method results in more precise decisions and guides the vehicle to navigate more smoothly.
2.1. Principle of Using Sequential Pattern Recognition for the Person Following Navigation
The principle of sequential pattern recognition is to use a sequence of input data and postpone decision making until sufficient confidence is accumulated. The decidable and undecided areas in the feature space are separated by two threshold values. By this method, the turning angle of the vehicle is not adjusted in every cycle; turning is activated only when a decision is made.
Traditionally, a two-class classifier used for making decisions takes the form,
Ž . h X )0 ªXg1 Ž .1 Ž . h X -0 ªXg2 Ž . where X is an input feature vector, h X is a discriminant function, and denote the two1 2 classes, and ‘‘ª’’ means ‘‘decide.’’ In sequential
23 Ž .
pattern recognition, the value of h X is not com-pared with zero but with two threshold values, a and b. We assume that a is smaller than b. When the value of the discriminant function is smaller than a or larger than b, the input feature vector is classified and a decision is made. Otherwise, the
Ž .
value of h X is judged to be located on an unde-cided area and decision making is postponed. In
Ž .
this situation, the value of h X is computed for the next input feature vector and added to the prior
Ž .
value of h X . The discriminant function of the classifier is
m
Ž . Ž .
Sms
Ý
h Xi 2is1
where m denotes the number of all the input feature vectors observed so far. In addition, the classifier is of the form,
SmFaªXXsg1
Ž .
a-S -bªtake the mq1 th samplem Ž .3
bFS ªXm Xsg2
For our case here, means the first class that the1 location of the person is too right or too left of the vehicle, and means the second class that both2
the person and the vehicle move in a straight trajec-tory. Whenever a decision is made, S is reset to 0m and m to 1. This means that S is reinitialized and am new decision making process is started. If the vector
X does not belong to or , the vector X1 2
belongs to an undecided area, which is the last class, and the next sample is taken.
Ž .
The two threshold values, a and b, in 3 control the error of the sequential pattern recognition result.24As the absolute values of a and b become
larger, the error reduces, while the number m of observations required to reach the decision in-creases. If the values of a and b are equal to zero, the three classes are combined into two classes and the undecided area does exist. Also, the vector X will belong either to or to according to the1 2
Ž . Ž .
value of classifier h X in Eq. 1 , and a larger error will occur. For example, when a person walks straight but his location is a little right of the vehi-cle, the motion of the person does not belong to 1 or , and the next sample should be considered. If2 the discriminant function makes a decision at this time, the decision result may be wrong and errors occur. Hence, if b is larger than zero and a is smaller than zero, then the larger the range between
a and b, the larger the range of the last class. This
will force the discriminant function to use more samples to make a decision and the error caused in making decision will be reduced.
The values of a and b are assigned according to the experiment. If the value of a is taken to be too small, the person will be out of the camera view because it will take too much time to make a deci-sion that the person belongs to . If the value of a1 is, on the contrary, taken to be too large, the dis-criminant function judges most of the time that the person belongs to so that the vehicle adjusts its1 direction frequently and smooth navigation is hard to achieve. According to the value of the discrimi-nant function Sm and the two threshold values a and b, the relation between the location of the person and that of the vehicle is categorized.
2.2. Features for Sequential Pattern Recognition
Ž .
By 3 , a decision is made according to the value of
S which is calculated from the sequential values ofm
Ž .
the function h X . In this section, the feature vectori
X selected for use in this study is described.
Conceptually, if the X coordinate value l of the location of the person in the vehicle coordinate
Ž .
system VCS is larger than zero, the person is regarded to stand on the right side of the vehicle;
otherwise, he is regarded to be on the left side of the vehicle. When a person stands on the right side of the vehicle, hershe may turn left, right, or go straight. This means that l is not sufficient for decision making about the movement of the person. We thus choose another feature which is the length
w of the projection of the body of the person onto
the X axis of the VCS. Because the value of w is calculated from the real width w of the body of theb person, it is not affected by locations of the person. This means that these two features are independent. When the person walks straight, the value of w is stable and keeps on almost the same value. On the contrary, the value of w becomes smaller when the direction of the person is not parallel to the vehicle. So, the value of w is a good feature for predicting the direction of the person. An illustration of the values of l and w when the person turns right is shown in Figure 2, in which P and P , to bel r
described in detail in section 4, denote two detected feature points of the back of the person. w is theb
length between P and P . Assume that their coordi-l r
Ž . Ž .
nates are x , y , zpl pl pl and x , y , zpr pr pr in the VCS,
Ž .
respectively. The valuer of l is xplqx r2 andpr
Ž .
that of w is xpryx . The selected feature vectorpl
X is
l Ž .
Xs w 4
2.3. Classifier for Sequential Pattern Recognition
Since the feature vector X is two-dimensional, the decision boundary of the classifier is a
two-dimen-Figure 2. Top view of the vehicle coordinate system ŽVCS and illustration of the X coordinate value l of the. location of the person and the length w of the projection of the body of the person when he turns right.
sional curve in the L-W space. When the person stands on the right side of the vehicle, l is larger than zero; otherwise, l is smaller than zero. Besides, when the person walks straight, w is equal to the width w of the body of the person; otherwise, w isb smaller than w .b
In this study, it is found that the use of the two values of l and w is enough to identify three classes
Ž .
and derive the classifier h X . The first class is that the location of the person is too right or too left of the vehicle. We use the width w of the vehicle as av
boundary. If the absolute value of the X coordinate of the location of the person is larger than w , thev location of the person is regarded to belong to the first class; otherwise, the direction of the person has to be considered. The second class is that both the person and the vehicle move in a straight trajectory. In the optimal situation, this means that w is equal to w . However, the direction of the person mayb drift. We use a half value of w as a boundary. Ifb the value of w is smaller than a half value of w ,b
the location of the person is not regarded to belong to the second class. It means that the person is turning right or left. Based on the above
observa-Ž .
tion, we design a classifier h X to be a parabola
Ž . Ž .
curve which passes through 0, wbr2 and w , w ,v b
Ž . as illustrated in Figure 3. The equation for h X
Ž
is derived accordingly as follows the details are . omitted , wb wb 2 Ž . Ž . h X sWy
ž
2/
L y 5 2 2 wv Ž .According to the classifier h X , the L-W plane is separated into three areas, S, T , and T , as illus-r l
Ž .
Figure 3. Illustration of the classifier, h X . w and wb v
trated in Figure 4. The area S means that the person walks straight and belongs to . The areas T and2 r
T mean that the location of the person is too rightl
and too left of the vehicle, respectively, and belongs to . If the location of the person is in the area T1 r
or T , the direction of the vehicle has to be changed.l
It does not mean that the person is turning right or
Ž .
left. Besides, if the person is not too left l-0 and
Ž .
the person turns right w small , the feature vector
X is located near the boundary of the classifier
Ž . Ž .
h X and the absolute value of h X is a little Ž . larger than zero. In this case, according to Eq. 3 , the vector X belongs to an undecided area, i.e., to class 3, and the vehicle keeps the current direction and the person is still in the camera view. If the
Ž .
person is too left l-0 and the person turns right Žw small , then the location of the person belongs to. , and the vehicle adjusts the direction of the1
vehicle according to the location of the person. Hence, the location of the person will not be out of the camera view in the next cycle. If the person walks very fast so that the person is out of the camera view, the vehicle will stop.
With the three classes, corresponding adjust-ments of the direction of the vehicle are described in the next section.
3. WHEEL ANGLE CONTROL
The wheel angle control is triggered after the deci-sion that the person belongs to the class or is1 2 made. If the person belongs to the second class ,2 the wheel angle of the vehicle is adjusted to be 0⬚. If the person belongs to class 3, the wheel angle of the vehicle keeps on the current one. If the person
Figure 4. The area of S means that the person walks
Ž .
straight class 2 . The areas of T and T mean that ther l
location of the person is too right and too left of the
Ž .
vehicle, respectively, class 1 .
belongs to the first class , the direction of the1 vehicle is adjusted to follow the person. The turning angle ␦ of the front wheels of the vehicle is
com-Ž .
puted according to the coordinates x , ypb pb of the location of the person in the VCS.
When the location of the person belongs to ,1
it does not mean that the person always turns 90⬚. The person might walk straight but with his loca-tion too right of the vehicle. In such a case, the direction of the vehicle is adjusted. Otherwise, the person will be out of the camera view. The angle ␦ is calculated according to the current location of the
Ž .
person in the vehicle coordinate system VCS and the adjustment of the direction of the vehicle to-ward the person is made accordingly. The deriva-tion of the turning angle ␦ for the front wheels of the vehicle is based on the kinematic trajectory of the vehicle.
As shown in Figure 5 which is a top view of the VCS, assume that the vehicle is located at A and
Ž .
the person is located at P with coordinates x , yb pb pb
in the VCS. Let the vehicle move a distance S from location A to location P by turning an angleb ␦. Let
S be the kinematic trajectory of the vehicle and ␦ be
the angle that we want to obtain. By assuming that
Ž .
xpb and ypb have been calculated see section 4 , the values of P and u in Figure 5 can be computed as
Figure 5. Analysis of the path S on which the vehicle navigates from location A to location P by turning anb
Ž .
angle ␦. The coordinates of location P are x , yb pb pb in the VCS.
follows, 2 2 Ps x qy
'
pb pb Ž .6 ypb y1 ustanž /
xpbwhere P, x , and ypb pb compose a right triangle and
u is the corresponding angle of y . On the otherpb
hand, P, R, and u can be acquired by the following equations25:
'
Ž . Ž . PsR 2 1ycos r 7 d Ž . Rs 8 sin␦ r Ž . us y␦y 9 2 2where R is the rotation radius, d is the distance between the front wheels and the rear wheels, r is the corresponding angle of S, and ␦ is the turning
Ž . Ž .
angle of the front wheels. From Eqs. 7᎐ 9 , we can obtain P Gs 2 d Ds yu 2 Ž10. sin D y1 ␦stan
ž
/
cos DqGwhere d is a constant, and P and u are calculated Ž .
according to 6 . After substituting the values of P, Ž .
u, D, and G into the last equation in 10 , we get a
more detailed equation for␦ as follows,
y y1 pb sin
ž
ytanž /
/
2 xpb y1 ␦stan 2 2 x qy'
y y1 pb pb pb cosž
ytanž /
/
q0
2 xpb 2 d Ž11. 4. VISION SYSTEMIn this study, it is desired to control the vehicle to navigate smoothly by following a person who walks
in front of the vehicle. An important source of information used in this study is the location and the direction of the person. Only by the use of correct coordinates of the location of the person can the proposed system drive the vehicle to navigate correctly. Because the detection of the whole human body wastes a lot of time, we attach a rectangular shape on the shirt of the person for fast detection and tracking of the person to shorten the processing time. Images with the shirt of the person are grabbed by a color CCD camera. The two upper corners of the rectangular shape are detected by image pro-cessing techniques. We have two reasons for detect-ing just the two upper corners of the rectangular shape instead of the four. One is that the two upper corners are sufficient to provide the width of the shape. The other is that the two lower corners are out of the camera view when the person is close to the vehicle. The second reason is related to the position of the camera. In this study, the camera is mounted on the front of the vehicle. Besides, the position of the camera is set at an average height of human beings and with a little tilt downward to the ground. Hence, the whole shape can be detected when the distance between the person and the vehi-cle is larger than 1 or 2 m. In the experiment, the position of the camera is 35 cms higher than the feature points to be detected. When the two upper corners are detected in the image, the three-dimen-sional coordinates of the detected corners in the vehicle coordinate system are calculated to obtain the location of the person. In this study, the height of the detected corners is assumed to be known by measuring it directly in advance. The camera cali-bration is implemented using the information of vanishing lines.26 The coordinate system27 and the
transformations between them are adopted from refs. 28 and 29 for use in this study.
4.1. Feature Point Detection by Image Processing Techniques
The feature point we use for locating the person are two points on the back of the person which repre-sent proportionally the width of the person. As shown in Figure 6a, a rectangular shape is attached on the shirt of the person. The color of the rectangu-lar shape is yellow. We segment the shape out of the back of the person by thresholding. Next, a region growing method is used to detect the border of the rectangle. Finally, the intersection point of
Ž .
Figure 6. The experimental result of feature point detection. a is a captured image with Ž .
the segmented rectangular shape. b Detected feature points.
the left border and the upper border, and that of the right border and the upper border, are detect-ed as the two desirdetect-ed feature points, as shown in Figure 6b.
4.2. Three-Dimensional Coordinate Calculation by Computer Vision Techniques
In this study, the ALV navigation environment is described by three coordinate systems, the image
Ž .
coordinate system ICS , the camera coordinate
sys-Ž .
tem CCS , and the vehicle coordinate system ŽVCS ,.27as shown in Figure 7.
The transformations between the ICS, the CCS, and the VCS are described as follows. Assume that any point P in the image plane has the CCS
coordi-Ž . Ž .
nates u ,P yf, w , where u , w specify the coor-P P P
dinates in the ICS and f is the focus length. We get
Ž .
the VCS coordinates x , y , zP P P of point P in the image4 as
Ž .
xPsu cosP cos qsin sin sin
Ž . Ž
qf sin cos qw sin sin cos P
. qcos cos qxd
Ž . Ž .
yPsu sinP cos qcos sin sin 12
Ž . Ž
yf cos cos yw cos sin cos P
. qsin sin qyd
Ž . Ž .
zPsu cosP sin yf sin qw cos cos qzP d
Ž .
where x , y , zd d d are the VCS coordinates specify-ing the translation vector from the origin of the VCS to the origin of the CCS, , , and are the pan,
Ž . Figure 7. The three coordinate systems, including the image coordinate system ICS ,
Ž . Ž .
tilt, and swing angles, respectively, of the camera with respect to the VCS. These camera parameters are calibrated26 in advance. In addition, the
equa-tion of a line LP, which passes the lens center and
P, is
xyxd yyyd zyzd
Ž .
s s sk 13
xPyxd yPyyd zPyzd
As shown in Figure 8, let point P⬘ be the intersection point of the plane zsh and the line LP.
Ž .
By substituting zsh into Eq. 13 , the desired VCS
Ž .
coordinates x , y , zP⬘ P⬘ P⬘ of point P⬘ can be solved to be hyzd Ž . xP⬘sx qd xPyxd zPyzd hyzd Ž . yP⬘sy qd yPyyd zPyzd Ž14. zP⬘sh
In this study, assume the image coordinates of
Ž .
the feature points P and P arel r u , wpl pl and Žu , w , respectively. In addition, the height zp p. p of
r r b
the detected feature points is assumed to be known.
Ž . Ž .
The coordinates x , y , zpl pl pl and x , y , zpr pr pr of the features points P and P are obtained by Eqs.l r Ž12 and 14 with h. Ž . sz in the vehicle coordinatep
b
Ž .
system. Hence, the coordinates x , y , zpb pb pb of the location of the person Pb can be calculated as
follows, xplqxpr xpbs 2 yplqypr ypbs 2 Ž15. zpbsz szpl pr 5. EXPERIMENTAL RESULTS
The proposed method has been implemented and used to steer a real ALV autonomously to follow a person, as shown in Figure 9. Smooth navigation was demonstrated while the person walks in differ-ent directions. In our experimdiffer-ents, the width w andv
Figure 9. The vehicle is guided automatically to follow a person who walks in front of the vehicle.
Figure 8. Illustration of the coordinate transformations between the camera coordinate
Ž . Ž . X
system CCS and the vehicle coordinate system VCS when the point P is located in the plane zsh.
Ž . Figure 10. A situation where the location of the person is too right of the vehicle. a The
Ž .
result of the image processing. b The location = of the value of the discriminant function in the L-W plane. The coordinates, x and y , of the location of the person inb b
the VCS are 87.94 and 201.07, respectively. The value w is 33. The value of the Ž .
discriminant function h X isy166.23 and the turning angle, ␦, of the front wheels is 10.22⬚.
the length of the ALV are 40 and 120 cm, respec-tively. The length between the front wheels and the rear wheels of the ALV is 82 cm. The width w ofb the rectangular shape is 30 cm. The X-Y-Z
coordi-Ž .
nates of the camera in the VCS are 17.7, 3.7, 167.9 . The pan, tilt, and swing angles of the camera are y0.007, 0.005, and 0.03, respectively. In the experi-Ž . ment, the threshold values a and b used in Eq. 3 arey15 and 15, respectively.
The proposed relations between the location of the vehicle and that of the person and the corre-sponding actions taken by the ALV are verified in the experiments. Figures 10 and 11 illustrate that the location of the person is too right and too left, respectively, and both cases belong to the first class . Figure 10 shows that the person walks in a1
straight trajectory and Figure 11 shows that the person turns left. The values of the discriminant
Ž . Figure 11. A situation where the location of the person is too left of the vehicle. a The
Ž .
result of the image processing. b The location = of the value of the discriminant function in the L-W plane. The coordinates, x and y , of the location of the person inb b
the VCS are y61.21 and 202.15, respectively. The value w is 23. The value of the Ž .
discriminant function h X isy80.03 and the turning angle, ␦, of the front wheels is y7.46⬚.
Ž .
Figure 12. The situation that the person walks straight. a The result of the image Ž .
processing. b The location= of the value of the discriminant function in the L-W plane. The coordinates, x and y , of the location of the person in the VCS areb b y2.38 and
Ž . 232.39, respectively. The value w is 28. The value of the discriminant function h X is 17.90 and the turning angle,␦, of the front wheels is 0⬚.
function are both smaller than the threshold value
a. The feature vectors X are located in the areas of
Ž
T and T , respectively, in the L-W plane denotedr l
.
with ‘‘=’’ . In Figure 10, the two lower corners of the shape are out of the camera view and the two upper corners are detected stably. Figure 12 shows that the person walks straight and the case belongs to the second class . The value of the discrimi-2 nant function is larger than the threshold value b and the feature vector is located in the area S in the
L-W plane. Figure 13 shows that the feature vector
is located in the undecided area. The value of the discriminant function is in the range from the threshold value a to the threshold value b. The direction of the vehicle keeps on the current one. Figure 14 shows a sequence of experimental images in a real indoor environment. A top view of a sequence of the trajectory of the vehicle and that of the person is shown in Figure 15. In the experimen-tal results, if the value of ␦ is smaller than zero,
Ž . Figure 13. The situation that the feature vector is located in the undecided area. a The
Ž .
result of the image processing. b The location = of the value of the discriminant function in the L-W plane. The coordinates, x and y , of the location of the person inb b
the VCS are y27.62 and 224.54, respectively. The value w is 25. The value of the Ž .
Figure 14. A sequence of experimental images in the corridor of a real indoor environ-ment.
then it means that front wheels of the vehicle turn left; otherwise, the vehicle turns right.
The accuracy in the computation of the location of the person is shown in Table I, which includes the location of the person measured by hand and that computed by the calibrated vision system. The distance error rate in the table is defined as the ratio of the Euclidean distance between the measured position of the person and the computed position of the person to that between the measured position of the person and the origin of the VCS, which is expressed as 2 2 Ž . Ž .
'
xeyx q y yyc e c Ž . s 15 2 2'
xe qye Ž .where x , ye e are the VCS coordinates of the
mea-Ž .
sured location of the person and x , yc c are the VCS coordinates of the computed location of the person. The average distance error rate computed from all of the error rates in the table is 5.6%, which is acceptable for safe navigation.
6. CONCLUSIONS
In this study, a new approach to ALV navigation by the person following in indoor environments has been proposed. This approach uses sequential pat-tern recognition and computer vision techniques to achieve smooth navigation. A rectangular shape at-tached on the back of the person is detected by the
computer vision technique and two effective fea-tures are used in calculating the location of the person. Then sequential pattern recognition is used in making decisions about whether the person walks straight or is too left or too right of the vehicle. This
Figure 15. A top view of the experimental indoor envi-ronment and a continuous navigation session, in which each black circle indicates the location of the person in a navigation cycle and= means the location of the vehicle.
Table I. Computed location of the person vs. measured location of the person. Measured location of the Computed location of the
Ž . Ž . Ž .
person x , ye e person x , yc c Distance error rate %
Ž . Ž . 1 10, 100 11.06, 93.88 6.2 Ž . Ž . 2 15, 200 16.40, 186.81 6.6 Ž . Ž . 3 10, 300 14.31, 311.92 4.2 Ž . Ž . 4 60, 200 66.80, 187.70 6.7 Ž . Ž . 5 5, 250 3.96, 233 6.8 Ž . Ž . 6 y10, 150 y7.97, 145.22 3.5 Ž . Ž . 7 y20, 220 y24.75, 229.25 4.7 Ž . Ž . 8 40, 180 49.73, 174.33 6.1
approach has been implemented on a real ALV, and successful and smooth navigation sessions confirm the feasibility of the approach.
The authors thank Kuang-Hsiung Chen in the puter Vision Laboratory of the Department of Com-puter and Information Science at National Chiao Tung University for helpful discussions and assistance in the experiments of this research.
REFERENCES
1. M.S. Chang, P.Y. Ku, L.L. Wang, and W.H. Tsai, Indoor autonomous land vehicle guidance by line following using computer vision techniques, Proc 1989 Workshop Comput Vision, Graph and Image Process-ing, Taipei, Taiwan, Republic of China, 1989, pp. 135᎐144.
2. B. Matthews, M. Ruthemeyer, D. Perdue, and E.L. Hall, Line following for a mobile robot, Proc SPIE—The Int Soc Opt Eng, Vol. 2588, 1995, pp. 610᎐617.
3. W.J. Ke and W.H. Tsai, Indoor autonomous land vehi-cle guidance by corner tracking using computer vi-sion, Proc 1991 Workshop on Comput Vivi-sion, Graph and Image Processing, Tainan, Taiwan, Republic of China, 1991, pp. 133᎐139.
4. Y.M. Su and W.H. Tsai, Autonomous land vehicle guidance for navigation in buildings by computer vision, radio, and photoelectric sensing techniques, J
Ž .
Chinese Institute Eng, 17 1994 , 63᎐73.
5. H.I. Christensen, N.O. Kirkeby, S. Kristensen, L. Knudsen, and E. Granum, Model-driven vision for
Ž .
indoor navigation, Robot Autonomous Syst, 12 1994 , 199᎐207.
6. L.L. Wang, P.Y. Ku, and W.H. Tsai, Model-based guidance on the longest common subsequence algo-rithm for indoor autonomous vehicle navigation
us-Ž .
ing computer vision, Automat Construction, 2 1993 , 123᎐137.
7. T. Fukuda, S. Ito, N. Oota, F. Arai, Y. Abe, K. Tanaka, and Y. Tanaka, Navigation system based on ceiling landmark recognition for autonomous mobile robot,
Proc IECON Int Conf Ind Electron, Control, and In-strumentation, 3, 1993, pp. 1466᎐1471.
8. A. Gilg and G. Schmidt, Landmark-oriented visual navigation of a mobile robot, IEEE Trans Ind Electron,
Ž .
41 1994 , 392᎐397.
9. C. Thrope, M.H. Hebert, T. Kanade, and S.A. Shafer, Vision and navigation for Carnegie-Mellon NAVLAB,
Ž .
IEEE Trans Pattern Anal Machine Intell, 10 1988 , 362᎐373.
10. J.D. Crisman and C.E. Thorpe, SCARF: A color vision system that tracks roads and intersections, IEEE Trans
Ž .
Robot Automat, 9 1993 , 49᎐58.
11. C.C. Chien, M.C. Lai, and R. Mayr, Vehicle following controller design for autonomous intelligent vehicles, Conference on Intelligent Robotics in Field, Factory, Service, and Space, Houston, TX, Vol. 1, 1994, pp. 212᎐221.
12. D. Noll, M. Werner, and W. von Seelen, Real-time vehicle tracking and classification, Proc Intell Veh Symp, 1995, pp. 101᎐106.
13. K.I. Kim, S.Y. Oh, S.W. Kim, H. Jeong, J.H. Han, C.N. Lee, B.S. Kim, and C.S. Kim, An autonomous land vehicle PRV II: progresses and performance enhance-ment, Proc Intell Veh ’95 Symp, Detroit, MI, 1995, pp. 264᎐269.
14. M. Schwarzinger, T. Zielke, D. Noll, M. Brauckmann, and W. von Seelen, Vision-based car-following: detec-tion, tracking, and identificadetec-tion, IEEE Intelligent Ve-hicles Symposium, 1992, pp. 24᎐29.
15. H. Schneiderman, M. Nashman, A.J. Wavering, and R. Lumia, Vision-based robotic convoy driving, Mach
Ž .
Vis Appl, 8 1995 , 359᎐364.
16. H.K. Nishihara, H.J. Thomas, and E. Huber, Real-time tracking of people using stereo and motion, Proc SPIE—The Internat Soc Opt Eng, Vol. 2183, 1994, pp. 266᎐273.
17. C. Wren, A. Azarbayejani, T. Darrell, and A. Pentland, Pfinder: Real-time tracking of the human body, Proc SPIE—Integration Issues Large Commercial Media Delivery Syst, Vol. 2615, 1996, pp. 89᎐98.
18. A. Azarbayejani, C. Wren, and A. Pentland, Real-time 3-D tracking of the human body, Proc IMAGE’COM 96, France, May 1996.
19. K. Matsuda, M. Kamihira, M. Ohchi, and T. Fu-rukawa, Design and implementation of tracking control system of autonomous mobile robot with
multiple sensors, Rep Fac Sci Eng Saga Univ, 24 Ž1995 , 73᎐82..
20. U.D. Hanebeck and G. Schmidt, A new high perfor-mance multisonar system for fast mobile robots, IROS Proc IEEErRSJrGI Internat Conf Intell Robots and Syst Advanced Robotic Syst Real World, Vol. 3, 1994, pp. 1853᎐1860.
21. H. Mori and M. Sano, A guide dog robot Harunobu-5—following a person, Proc IROS, IEEErRSJ Internat Workshop Intelli Robots and Syst Intell Mech Syst, Vol. 1, 1991, pp. 397᎐402.
22. C.H. Ku and W.H. Tsai, Smooth autonomous land vehicle navigation in indoor environments by person following using sequential pattern recognition, fuzzy speed control and computer vision techniques, Proc Joint Conf Fifth Int Conf Automation Technol 1998 Int
Ž .
Conf Prod Res Asia Meeting , Grand Hotel, Taipei, Taiwan, Republic of China, July 1998, p. 115.
23. K. Fukunaga, Introduction to statistical pattern recog-nition, 2nd ed., Academic Press, San Diego, CA, 1990. 24. A. Wald, Sequential analysis, Wiley, New York, 1947. 25. S.D. Chang and W.H. Tsai, Model-based guidance of autonomous land vehicles in indoor environments by structured light using vertical line information, J Electr
Ž .
Eng, 34 Dec. 1991 .
26. L.L. Wang and W.H. Tsai, Camera calibration by vanishing lines for 3D computer vision, IEEE Trans
Ž .
Pattern Anal Machine Intell, 13 1991 , 370᎐376. 27. S.D. Cheng and W.H. Tsai, Model-based guidance of
autonomous land vehicle in indoor environments by structured light using vertical line information, J Electr
Ž .
Eng, 34 1991 , 441᎐452.
28. R.M. Haralick and L.G. Shapiro, Computer and robot vision, Vol. 2, Addison-Wesley, Reading, MA, 1993. 29. A. Watt, 3D computer graphics, Addison-Wesley,
