利用自動車作基於視覺之室內安全巡邏

全文

(1)國立交通大學資訊科學系碩士論文. 利用自動車作基於視覺之室內安全巡邏 Vision-Based Security Patrolling in Indoor Environments Using Autonomous Vehicles. 研究生：陳明哲指導教授：蔡文祥. 教授. 中華民國九十四年六月.

(2) 利用自動車作基於視覺之室內安全巡邏 Vision-Based Security Patrolling in Indoor Environments Using Autonomous Vehicles. 研究生：陳明哲. Student：Ming-Che Chen. 指導教授：蔡文祥. Advisor：Wen-Hsiang Tsai. 國立交通大學資訊科學系碩士論文. A Thesis Submitted to Department of Computer and Information Science College of Electrical Engineering and Computer Science National Chiao Tung University in partial Fulfillment of the Requirements for the Degree of Master in Computer and Information Science June 2005 Hsinchu, Taiwan, Republic of China. 中華民國九十四年六月.

(3) 利用自動車作基於視覺之室內安全巡邏. 研究生: 陳明哲. 指導教授: 蔡文祥博士. 國立交通大學資訊科學研究系. 摘要. 本研究主要是提出一套基於電腦視覺技術，可讓自動車航行在室內環境中且具有安全監控的能力。我們利用一台小型自動車為實驗平台，並且利用無線操控的方式讓自動車航行在室內的環境中。我們設計了一套人簡單且有效的學習方式，讓使用者可以操控自動車行走且自由的選擇要被監控的物品或是房門，自動車將會自動的建立導航的地圖。在學習完路線、物品和門的資料之後，我們運用了一套完整的導航策略來完成安全巡邏的任務。這個策略包括了監控物品的安全偵測和房門情況的識別，利用自動車判斷物品是否遭竊或者是門被開起，我們能夠立刻發出警報給使用者，已達成安全巡邏的任務。最後我們以成功的學習與導航實驗結果證明本系統的完整性與可行性。. I.

(4) Vision-Based Security Patrolling in Indoor Environments Using Autonomous Vehicles Student: Ming-Che Chen. Advisor: Dr. Wen-Hsiang Tsai. Department of Computer and Information Science National Chiao Tung University. ABSTRACT. A vision-based approach to security patrolling in indoor environments using autonomous vehicles is proposed. A small vehicle with wireless control and image grabbing capabilities is used as a test bed. Three stages of security patrolling are proposed. First, a simple learning strategy is designed for flexible and effective learning of reachable spots and monitored objects in indoor environments. Accordingly, a planned path is obtained, and monitored objects and doors are specified by analyzing user commands. Next, following the learned path, the vehicle can accomplish specified navigation sessions. Two different kinds of methods, mechanic error correction modeling and vehicle position modification by positions of monitored objects, are proposed for navigation accuracy maintenance. Finally, an object matching algorithm is used for checking the existence of monitoring objects and the opening status of doors. All the experimental results show flexibility and feasibility of the proposed approach for the application of indoor security patrolling.. II.

(5) ACKNOWLEDGEMENTS. I am in hearty appreciation of the continuous encouragement, support, and technical guidance received from my advisor, Dr. Wen-Hsiang Tsai, not only in the development of this thesis, but also in every aspect of my personal growth. Thanks are due to Mr. Chih-Jen Wu, Mr. Shi-Chei Hung, Mr. Lien-Yi Weng, Mr. Shi-Yi Wu, and Mr. Yuei-Cheng Chuang for their numerous discussions and suggestions. Appreciation is also given to the colleagues of the Computer Vision Laboratory in the Department of Computer and Information Science at National Chiao Tung University for their suggestions and help. I also extend my profound thanks to my family during my school life for their lasting love, support, and encouragement. I dedicate this dissertation to my parents.. III.

(6) CONTENTS ABSTRACT.......................................................................................... II CONTENTS ........................................................................................III LIST OF FIGURES ........................................................................ VII Chapter 1 Introduction.............................................................................1 1.1. Motivation.................................................................................................1. 1.2 1.3 1.4 1.5. Survey of Relative Studies........................................................................2 Overview of Proposed Approach ..............................................................3 Contributions.............................................................................................6 Thesis Organization ..................................................................................6. Chapter 2 System Configuration And Navigation Principles...............8 2.1 2.2. Introduction...............................................................................................8 System Configuration ...............................................................................9 2.2.1 Hardware Configuration ..............................................................10 2.2.2 Software Configuration................................................................11 2.3 Learning Principle And Proposed Process of Learning ..........................12 2.4 Vehicle Guidance Principle And Proposed Process ................................14. Chapter 3 Location Mapping Calibration and Mechanic Error Correction............................................................................. 17 3.1. Introduction.............................................................................................17 3.1.1 Coordinate Systems .....................................................................18 3.1.2 The Direction Angle.....................................................................19 3.1.3 Coordinate Transformation ..........................................................19 3.2 Location mapping Calibration ................................................................21 3.3 Proposed Mechanical Error Correction Method .....................................25 3.3.1 Brief Description of Proposed Error Correction Model ..............26 3.3.2 Curve Fitting for Navigation Path................................................28 3.3.3 Coordinates And Angle Correction..............................................30 IV.

(7) 3.3.4 3.3.5. Straight Navigation Technique.....................................................32 Correction of Backing off ............................................................35. Chapter 4 A Method for Detection of Monitored Objects by Image Analysis ................................................................................. 38 4.1 4.2. Introduction.............................................................................................38 Process of Monitored Objects Detection ................................................39 4.2.1 Brief Description of Snake Algorithm .........................................39 4.2.2 Description of Improved Snake Algorithm..................................42 4.2.3 Computing Object Region ...........................................................46 4.2.4 Eliminating Noise of Background ...............................................47 4.2.5 Detailed Object Detection............................................................50 4.3 Object Feature Extraction .......................................................................51 4.3.1 Color Feature ...............................................................................52 4.3.2 4.3.3. Ellipse Fitting for Shape Representation .....................................53 Coordinate Transformation ..........................................................54. Chapter 5 Learning Strategies for Indoor Navigation by Manual Driving .................................................................................. 56 5.1 5.2 5.3 5.4. Introduction.............................................................................................56 Control Rules And Entire learning Process ............................................57 Process of Learning Navigation Paths ....................................................60 Process of Learning Monitored Objects and Doors................................63 5.4.1 Process of Learning Objects ........................................................63 5.4.2 Process of Learning Doors...........................................................66 5.5 Process of Automatic Path Map Creation from Learned Data................69. Chapter 6 Security Patrolling in Indoor Environments..................... 70 6.1 6.2 6.3 6.4. Introduction.............................................................................................70 Navigation Process..................................................................................71 Navigation Strategy for Straight-Line Sections ......................................73 Object Security Monitoring Process .......................................................75 6.4.1 Proposed Monitored Object Detection Method ...........................76 6.4.2 Proposed Object Matching Method .............................................78 6.4.3 Detailed Object Monitoring Algorithm........................................81 6.5 Detection of Door Opening.....................................................................84 6.6 Improved Guidance Precision in Navigation by Learned Object Location 86 V.

(8) Chapter 7 Experimental Results And Discussions.............................. 90 7.1 7.2. Experimental Results ..............................................................................90 Discussions .............................................................................................95. Chapter 8 Conclusions and Suggestions for Future Works ............... 96 8.1 8.2. Conclusions.............................................................................................96 Suggestions for Future Works.................................................................97. References ............................................................................................... 99. VI.

(9) LIST OF FIGURES Figure 1.1 A flowchart of proposed system...................................................................5 Figure 2.1 The vehicle used in this study. (a) The front of the vehicle. (b) The flank of the vehicle. ..................................................................................................9 Figure 2.2 Structure of system. .................................................................................... 11 Figure 2.3 A flowchart of proposed learning process. .................................................13 Figure 2.4 A flowchart of proposed navigation process ..............................................15 Figure 3.1 Three coordinate systems in this study. (a) ICS (b) GCS (c) VCS.............18 Figure 3.2 Definition of the direction angle (a)(b) θ is positive. (c)(d) θ is negative..20 Figure 3.3 The coordinate transformation between the GCS and VCS. ......................21 Figure 3.4 An illustrated of attaching the lines on the floor.........................................22 Figure 3.5 A method of finding image coordinates of tessellated points in the grabbed image. (a) A grabbed image with tessellated points. (b) The tessellated points marked by red points. .....................................................................23 Figure 3.6 An illustration of the interpolation method. (a) A region contains the point I in the ICS. (b) The projection of the region in (a) onto a floor region......25 Figure 3.7 Illustrations of navigation path deviation. (a) Moving ahead. (b) Backing off. .............................................................................................................26 Figure 3.8 An illustration the correction model. ..........................................................28 Figure 3.9 Flowchart of correction process. ................................................................28 Figure 3.10 An illustration of recording the coordinates. ............................................29 Figure 3.11 A figure of the curve. ................................................................................30 Figure 3.12 An illustration of correction detailed........................................................31 Figure 3.13 An illustration of computing correct angle...............................................32 Figure 3.14 An illustration of the correction path........................................................33 Figure 3.15 Computation of corrected path .................................................................34 Figure 3.16 An Illustration of corrected navigation path.............................................35 Figure 3.17 An illustration of Backing path.................................................................36 Figure 3.18 The path curve of backing off...................................................................36 Figure 4.1 Performance of the snake algorithm. (a) The behavior of elastic band and an object in it. (b) Final result of repeating continuous computations......40 Figure 4.2 A illustration of internal energy. (a) A pattern has a larger internal energy. (b) A pattern has a smaller internal energy. (c) A pattern has a larger internal energy. (d) A pattern has a smaller internal energy. ....................42 Figure 4.3 Contribution of control point......................................................................44 VII.

(10) Figure 4.4 Sobel operators. ..........................................................................................45 Figure 4.5 The result of using the Sobel operator. .......................................................46 Figure 4.6 Effect of background on object detection (a) Before detection. (b) Result of detection...............................................................................48 Figure 4.7 Correction of control points. (a) Before correction. (b) After correction...49 Figure 4.8 Results of improved method. (a) Detecting a ball. (b) Detecting a tub......49 Figure 4.9 Elimination of a floor region. (a) A decorative floor. (b) An experimental result..........................................................................................................50 Figure 4.10 The ellipse representation of a safe. .........................................................54 Figure 4.11 The locations of objects (a) An original image in which a ball is on the floor. (b) An original image in which a TV is on the wall. (c) An experimental result of (a). (d) An experimental result of (b). ...................55 Figure 5.1 Proposed learning process. .........................................................................58 Figure 5.2 Flowchart of manual learning process........................................................59 Figure 5.3 Recorded nodes when the vehicle moves straightly. (a) Nodes in one circle of the navigation correction. (b) Nodes in a longer path. .........................62 Figure 5.4 An experimental result of path learning with critical nodes recorded........62 Figure 5.5 Learning process of choosing an object manually. (a) Choosing an object. (b) An experimental result of computing object features data. .................65 Figure 5.6 A selected floor region................................................................................66 Figure 5.7 A learning process of choosing a door manually. (a) Choose a door. (b) A door region is shown. ................................................................................67 Figure 5.8 An experimental result of detection of three edges. ...................................68 Figure 5.9 An example of navigation map...................................................................69 Figure 6.1 Flowchart of security patrolling process ....................................................72 Figure 6.2 Line following navigation. .........................................................................75 Figure 6.3 Illustration of turn angle computing. ..........................................................77 Figure 6.4 An experimental result of object region. ....................................................78 Figure 6.5 An ideal experimental result of the vehicle turning to the object...............78 Figure 6.6 An experimental result of detecting no object............................................79 Figure 6.7 Situations of object detection. ....................................................................82 Figure 6.8 Flowchart of object matching. ....................................................................83 Figure 6.9 An illustration of door detection. (a) The door is opened. (b) The door is closed. .......................................................................................................85 Figure 6.10 Coordinate Transform between the VCS and the object coordinate system. (a) A sidelong view of the VCS. (b) A vertical view of the object coordinates. ...............................................................................................89 Figure 6.11 An illustration of navigation after correcting the position of the vehicle. 89 VIII.

(11) Figure 7.1 An interface of the experiment. ..................................................................90 Figure 7.2 The learning images....................................................................................91 Figure 7.3 An illustration of learned data. ...................................................................91 Figure 7.4 A navigation process...................................................................................92 Figure 7.5 The experimental result of security monitoring. ........................................93. IX.

(12) Chapter 1 Introduction. 1.1 Motivation The development of autonomous vehicles or mobile robots is paid much attention recently. Many researchers are devoted to developing functions of vehicles or robots such that they can do works conducted by human beings. Among applications of the autonomous vehicle, security patrolling is a practical function for human beings. We often install cameras in the house to monitor the indoor situation nowadays. We only depend on looking at videos to find out thieves when we discover that thieves broke into the house or objects were stolen. It takes much time and manpower, and lacks efficiency. Moreover, the positions of cameras are always fixed such that there might exist corners where the camera view cannot cover. The use of a vision-based autonomous vehicle system is a good choice to solve the above problem. Because the vehicle only spends electric power or fuel, it can substitute for manpower to do security patrolling cheaply. By computer control, the vehicle can repeat identical steps, never feels tired, and makes errors. Hence, the vehicle is able to patrol the whole day and will not rest like human beings. Another advantage is that when the vehicle detects an unusual situation; the monitoring system can send warning messages to guards or hosts immediately through signal transmission equipment. So the works of monitoring will cost less time and money. So the research goal of this study is to design an intelligent system for security 1.

(13) patrolling by means of a vehicle. It is desired to design the system to be capable of indoor navigation and security checks, including object detection and door situation recognition.. 1.2 Survey of Relative Studies To achieve the mission of security patrolling in indoor environments, learning navigation paths and recording features of monitored objects is required before the vehicle can navigate automatically. Since scenes of indoor environments consisting of rooms, corridors, and objects are usually complicated, the vehicle must have the ability of computing the distance between objects. Lai [9] proposed a curve fitting technique and a modified interpolation technique to perform 2D-to-3D distance transformations. By this method, we can know the distances between the vehicle and the surroundings in real space through captured images. Li [10] proposed learning methods for dealing with complicated surroundings as well as strategies for autonomous vehicle navigation techniques. The user controls the vehicle to patrol and analyze the captured image in the learning process, and a navigation map is created dynamically. The vehicle can keep away from obstacles when it navigates automatically. Moreover, Chen [12] proposed two navigation modes and a fuzzy guidance technique. A navigation map is created by two kinds of learned data and the fuzzy guidance technique is applied to achieve obstacle avoidance. About security patrolling navigation, Liu [13] proposed a method for use in building corridors by multiple-camera vision and automatic vehicle navigation techniques. In the learning process, the vehicle has to record the features of monitored objects. It is usually difficult to segment objects from complicated background in 2.

(14) images. Kass, WitKin, and Terzopoulos [1] proposed a prototype of the snake algorithm to segment patterns in the image. However, there are some drawbacks in the snake algorithm. Some researches [2-6] tried to improve this algorithm by using more complicated techniques. After detecting an object from the image, the features of the object is recorded in the learning process. In order to represent the shapes of objects, Sussman [7] mentioned the technique of fitting an ellipse to a set of data points by using the least square fitting technique. And Pilu, Fitzgibbon, and Fisher [8] proposed the ellipse representation of object pixels by using the edge pixels of the object. There are methods of coordinate transformations in the space and vector operations techniques for use in Fraleigh and Beauregard [14]. There are some image processing techniques consulted in Gonzalez and Woods [15].. 1.3 Overview of Proposed Approach In this study, we try to design a vision-based vehicle system for security patrolling in indoor environments. In order to achieve the mission of security patrolling, to recognize surrounding objects and know the vehicle positions are necessary. The chief tools are the images captured by the camera and the odometer in the vehicle hardware system. An overall framework of the proposed system is illustrated in Figure 1.1. The proposed vehicle system for security patrolling includes three major processes: setup of the vehicle system, learning of paths and objects by user control, and vehicle navigation. Setup of the vehicle system is the basis of navigation. It includes location mapping calibration and mechanic error correction. After fixing the camera on the 3.

(15) vehicle, a mapping calibration technique for using 2D image analysis is used to calculate the distance between the vehicle and an object. On the other hand, because the vehicle is a machine, it is unavoidable that there exist mechanic errors. Although the odometer provides the position of the vehicle, the real position of the vehicle is different from the value provided by the odometer due to the mechanic error. By the way of correcting the value provided by the odometer, navigation can be made more accurate. A mechanic error correction model is proposed to solve the problem. Moreover, the mechanic error causes the vehicle moving far away from original straight path. We propose a straight navigation technique to deal with this problem. Before the vehicle navigates, how to detect objects from the image is essential. Because the computer can not identify objects likes people, an improved snake algorithm is proposed in this study to detect objects in captured images. Although some researches [2-6] improve the snake algorithm, these methods are complicated and slow. We design a faster method to meet the real time requirement. In order to recognize objects, we utilize three kinds of object features in this study; they are color, shape, and coordinates. We propose three methods to compute these three features, respectively. In the learning process, the vehicle is controlled to move to desired places by using a user interface designed in this study. Some coordinates of the vehicles which are denoted as nodes are recorded along the learning path. The user may identify objects or doors to be monitored from the images when the vehicle moves to neighborhood of objects. The system will record the features of the objects which are illustrated in last paragraph automatically. According to the node data and the position of the objects and doors, a navigation map is created for use of the vehicle navigating automatically. The vehicle moves along the path and detects the objects and doors one by one by the positions of 4.

(16) them. As soon as the vehicle can not detect the monitored object, a warning message will be sent out from the system right away. When detecting the object, the system compares it with the learned data further. If the features are not correct, the same warning is announced. In the same manner, the vehicle moves to the neighborhood of the door to check whether the door is opened or not. Due to the mechanic error, the vehicle might move far away from the original path gradually. We propose two methods to improve the stability of navigation. One is to use the coordinates of learned objects as an auxiliary tool to adjust the position and direction of the vehicle. The other is a line following technique. After completing the check of the objects and doors, one circle of security patrolling is finished.. Location mapping calibration and mechanic error correction Path learning by user control. Features of objects and doors saving. Creation of learned path map. Security patrolling. Figure 1.1 A flowchart of proposed system.. 5.

(17) 1.4 Contributions Some major contributions of this study are listed as follows. (1) A mechanic error correction method is designed. (2) A technique for reducing mechanic error effect on straight navigation is proposed. (3) A fast object detection method in indoor environments by using an improved snake algorithm is proposed. (4) Three methods for object feature computing are proposed. (5) A strategy for learning paths, objects, and doors is proposed. (6) A method for path map creation by learned data is designed. (7) A line following method which can reduce accumulative errors for navigation is proposed. (8) Methods for object matching and door situation recognition are proposed. (9) A method for correcting vehicle positions in the navigation session is proposed.. 1.5 Thesis Organization The remainder of this thesis is organized as follows. In Chapter 2, the system configuration of the vehicle and the principles of learning and navigation are described. In Chapter 3, the proposed methods for location mapping calibration and mechanic error correction are described. The proposed methods for object detection in the images and feature computing are described in Chapter4. The proposed learning strategy and path map creation method are described in Chapter 5. The proposed vehicle navigation method, the strategies for navigation accuracy maintenance, object 6.

(18) recognition, and judgment of the door situation are described in Chapter 6. Some experimental results are given in Chapter 7. Finally, conclusions and some suggestions for future works are included in Chapter 8.. 7.

(19) Chapter 2 System Configuration And Navigation Principles. 2.1 Introduction In order to achieve the goal of security patrolling by an autonomous vehicle, conditions around the vehicle are investigated. There are usually many obstacles and narrow paths in indoor environments. Hence, it is a wonderful choice that the size of a vehicle is small and its action is dexterous. The smaller the size of the vehicle, the larger range the vehicle can navigate. Moreover, if the size of the vehicle is small enough, it can monitor the space under tables, beds, cabinets, etc. In this study, a small vehicle is used as a test bed. Beside the vehicle structure, a camera is installed on the vehicle. For a user to achieve the function of controlling the vehicle, some communication and control equipments are required. The entire hardware equipment and software used in this study are introduced in Section 2.2. After introducing the equipment, the vehicle navigation principles and some detailed relevant processes are described in the following sections. In Section 2.3, we will describe the process of learning. Path and object data which are necessary for security patrolling are recorded in the learning process. In Section 2.4, we will describe the process of security patrolling in which the vehicle monitors concerned objects and checks whether a door is opened or not. As soon as an unusual situation is detected, a warning is issued.. 8.

(20) 2.2 System Configuration The vehicle is shown in Figure 2.1.. (a). (b) Figure 2.1 The vehicle used in this study. (a) The front of the vehicle. (b) The flank of the vehicle.. Recently, wireless networking and communication techniques advance quickly. More and more communication products are made with wireless transmission. In this study, we use an Amigo robot, a mini-vehicle made by ActiveMedia Robotics 9.

(21) Technologies Inc., on which a camera is installed, as the testbed of our research. We utilize wireless transmission equipment to control the vehicle and get images captured by the camera. We will describe the hardware system in Section 2.2.1 and the software used in this study in Section 2.2.2.. 2.2.1 Hardware Configuration The entire system is illustrated in Figure 2.2. The first part is the vehicle system. The length, width, and height with the camera of the vehicle are 33cm, 28cm, and 21cm, respectively. There are two larger wheels and one auxiliary small wheel at the rear of the vehicle. The maximum speed of the vehicle is 75cm/sec and the maximum rotation speed is 300 degrees/sec. There are eight ultrasonic sensors, an odometer, and an embedded hardware system in this mobile vehicle. The ultrasonic sensors are not used in this study. The odometer provides the coordinates and direction of the vehicle in the navigation. The origin of coordinates is the starting position of the vehicle. There is a 12V battery in the vehicle to supply the power of the vehicle system. The second part is the vision system. There is a wireless 2.4GHz camera which can transmit analog signals on the vehicle. A receiver and an imaging frame grabber are connected to the computer. The receiver receives analog signals from the camera and a digital image can be obtained by the imaging frame. The image grabbed in our experiments is of the resolution of 320×240 pixels. A personal computer with Centrino 1.4GHz CPU, a 768MB DDR RAM, and a 5400 rpm 40GB HDD is used as a remote control and monitoring system; a kernel program can be executed on it to give commands to the vehicle through a wireless transmission system. There is one wireless device in the vehicle and another in the PC. The commands of the remote system are transmitted to the wireless signal receiver by an access point 10.

(22) that meets the IEEE 802.11b standard. By using the access point as a medium, the commands can be transmitted from the PC to the vehicle.. Vision System Receiver. 9V Battery. Camera. Imaging frame grabber. Vehicle. 12V Battery. Embedded Hardware System. Vehicle System. Wireless Device. Access Point. PC. Remote System Wireless Transmission System. Figure 2.2 Structure of system.. 2.2.2 Software Configuration The ARIA is an object-oriented control applications-programming interface for. 11.

(23) ActivMedia Robotics’ line of intelligent mobile robots. The ARIA is written in the C++ language. We use it to control and retrieve the status of the vehicle. The operation system in the PC is Microsoft Windows XP and we use Borland C++ Builder as the development tool in our experiments.. 2.3 Learning Principle And Proposed Process of Learning Before security patrolling, a learning process is necessary. Navigation path, object, and door data are recorded in advance. The entire learning process in this study is shown in Figure 2.3. A user control interface is designed for use in controlling the vehicle and choosing monitored objects and doors. The user controls the vehicle to navigate in indoor environments and move to the front of objects for choosing monitored objects. The main recorded data include two categories, path and object data. The door is considered as an object in this study. As soon as the learning process ends, all data are stored in the storages of the computer such that the learning process is only executed once and data can be used repeatedly. When the vehicle patrols in rooms and corridors, the computer records the path data provided by the odometer, and denote them as nodes. The monitored objects are selected in the image captured by the camera when the user controls the vehicle to patrol. The features of the objects are also computed automatically from the images. There are three kinds of object features are recorded in this study. They are color, shape, and coordinates respectively.. 12.

(24) The process of door learning is similar to that of ordinary object learning. The computer can record more than one object when the vehicle is at a single position. By means of turning the vehicle at same position, we can learn neighboring objects of the vehicle. The user can choose objects continuously along the path until finishing a learning process.. Start learning. Manual control. Driving vehicle. Data collection. Choosing objects and doors. Feature data of objects and doors. Odometer data. Saving data in the storage. Map created by learned data. End learning. Figure 2.3 A flowchart of proposed learning process.. 13.

(25) After finishing learning process, we can get a navigation map by way of combining the path data and object positions. The vehicle utilizes this map to monitor the doors and objects consecutively along the navigation path.. 2.4 Vehicle Guidance Principle And Proposed Process In the navigation process, the vehicle monitors the objects and doors consecutively by using the map created from learned data. The vehicle moves to a fixed position and stops to check the existence of the monitored objects. As soon as an unusual situation is occurred, a warning is issued. After checking the objects at a location, the vehicle moves to the next position where there are monitored objects or doors, and then continues checking the existence of the monitored objects or whether the door is opened. An illustration of the vehicle navigation process is shown in Figure 2.4. In order to patrol along the learning path, the vehicle moves along each node in the map one by one. To move from a node to another, a line following technique is used to reduce navigation errors. When the vehicle moves to a node where there is a nearby monitored object, the computer detects the object by using the position of the learned object. As soon as the computer detects an object from the image, its features are compared with those of the learned objects. If the features are incorrect, a warning is issued. Of course, when the computer can not detect the object, the same warning is also issued. Because the vehicle might navigate far away from the original path gradually, we use a. 14.

(26) vision-based technique to reduce accumulative errors for navigation. When the vehicle detects an object and consider it as a learned object, we use its position recorded in the learning process to correct the position of the vehicle for reducing accumulative errors.. Start navigation. Vehicle patrolling. Yes. Object detection. Door identification. Whether correct. Whether closed. No. Yes Correct the vehicle. No. Yes. A warning is announced. position. Whether unchecked objects. No Navigating unfinished path. End navigation. Figure 2.4 A flowchart of proposed navigation process. Although the door is considered an object, the method of recognizing the door 15.

(27) situation is different from the method of object matching. After the vehicle moves to a suitable position recorded in the learning process, the computer checks the door situation to see if it is opened or closed. After the vehicle patrols the entire route and checks all objects and doors, a security patrolling process is finished.. 16.

(28) Chapter 3 Location Mapping Calibration and Mechanic Error Correction. 3.1 Introduction When a vehicle navigates in indoor environments, the position and distance are important data for the navigation. By using these data, the vehicle can arrive at suitable positions to search monitored objects or doors and navigate along a learned path. In order to get position data and distance values, a camera and an odometer are used in this study, which are equipped on the vehicle. At first, we utilize an image captured by the camera to obtain the relative position between an object and the vehicle. We use a 2D mapping method to achieve this goal. The detailed process is described in Section 3.2. The odometer provides the positions of the vehicle in the environment. It records the coordinates and the direction angles of the vehicle when the vehicle navigates. Possible mechanic errors might cause the real positions of the vehicle to be unequal to the expected values provided by the odometer. Hence, in Section 3.3, we will propose a correction method to reduce the mechanic error so that the vehicle can navigate more stably. Before describing the above-mentioned methods, we first introduce some coordinates system and the definition of the direction angle of the vehicle for use in this study. We introduce the coordinate systems in Section 3.1.1 and the definition of. 17.

(29) the direction angle of the vehicle in the space is described in Section 3.1.2. By using the direction angle, a transformation function between the coordinate systems is described in Section 3.1.3.. 3.1.1 Coordinate Systems Three coordinate systems are utilized in this study to describe the locations of the vehicle and objects. The coordinate systems are shown in Figure 3.1. The definitions of these systems are stated in the following.. Room. v. I. y u G. (a). x. (b) Vy. V. Vx. V Vy Vx. (c) Figure 3.1 Three coordinate systems in this study. (a) ICS (b) GCS (c) VCS.. (1) The image coordinate system (ICS): denoted as u−v. The u−v plane is coincident with the image plane and the origin I of the ICS is placed at the center of the. 18.

(30) image plane. (2) The global coordinate system (GCS): denoted as x-y. The x-axis and the y-axis are defined to lie to on the ground, and the origin G of the global coordinate system is a pre-defined point on the ground. In this study, we define G as the starting position of the navigation. (3) The vehicle coordinate system (VCS): denoted as Vx-Vy. The Vx-Vy plane is coincident with the ground. The Vx-axis is parallel to two wheels of the vehicle. The Vy-axis is parallel to the body of the vehicle. And the origin V is placed at the middle of the line segment that connects the two contact points of the two driving wheels with the ground.. 3.1.2 The Direction Angle The direction angle of the vehicle is defined in the global coordinate system, the x-y plane. The angle denoted as θ represents the rotation degree of the vehicle in the. global system and plays a very important role in the coordinate transformations.. θ is the angle between the positive direction of the x-axis and the front of the vehicle. The direction angle θ is set to be zero at the beginning of navigation. The range of θ is between 0 and π if θ is in the first and second quadrants, as illustrated in Figure 3.2(a) and (b). It is between 0 and −π if θ is in the third and forth quadrants, as illustrated in Figure 3.2(c) and (d).. 3.1.3 Coordinate Transformation By using the direction angle and the coordinates of the vehicle in the GCS, we can describe the coordinate transformation between the vehicle coordinate system and the global coordinate system by the following equations.. 19.

(31) x = Vx × cos θ − Vy × sin θ + x p y = Vx × sin θ + Vy × cos θ + y p where xp and yp are the coordinates of the vehicle in the GCS. It is shown in Figure 3.3.. y. π /2 θ 0. π or −π. θ x. −π / 2. (a). (b) π /2. π or −π. θ. 0. x. θ −π /2. (c). (d). Figure 3.2 Definition of the direction angle (a)(b) θ is positive. (c)(d) θ is negative.. 20.

(32) y Vy Vx θ. x. GCS. Figure 3.3 The coordinate transformation between the GCS and VCS.. 3.2 Location mapping Calibration The camera is the only sensor of the vehicle to gather the features of the environment, and the techniques of learning and guidance are based on visual perception. Location mapping calibration and image analysis techniques are indispensable in this study. A real location data acquisition method by Location mapping calibration and image mapping is proposed to obtain the relative positions of the vehicle and the surrounding environment precisely. We use a point set P = {P00, P01, …,Pmn} whose coordinates are known in advance in the VCS attached on the floor, and their corresponding point set V = {V00,. V01, …,Vmn} appearing in the image, to compute the VCS coordinates of a set of pixels. The detailed process is described in the following algorithms. Algorithm 3.1. Real location data acquisition by image taking and mapping.. Input: An image I, as shown in Figure 3.5(a). Output: A point set V = {V00, V01, …,Vmn} and another point set P = {P00, P01, …,Pmn} Steps: 21.

(33) Step 1. Attach some straight lines on the floor, as shown in Figure 3.4. Set the length between every two vertical lines to be 30 cm and the length between every two horizontal lines to be 30 cm, too, where every rectangle is a 30×30 square. There exists a distance Div equal to 80cm in the front of the vehicle, which the camera can not view. Step 2. For those tessellated regions shown in the image which are not surrounded by the lines, extend the lines such that every quadrilateral is complete. Mark red points on the intersections of the lines, as shown in Figure 3.5(b). Step 3. Record the coordinates of each red point Pij(uij, vij) in the ICS and group all such points into the set P. Step 4. Measure relative coordinates Vij(xij, yij) manually in the VCS of the point Pij and group such points into the set V.. D iv. Figure 3.4 An illustrated of attaching the lines on the floor.. 22.

(34) (a). (b) Figure 3.5 A method of finding image coordinates of tessellated points in the grabbed image. (a) A grabbed image with tessellated points. (b) The tessellated points marked by red points.. We now have known the VCS coordinates of red points. As for the VCS coordinates of the other pixels, we use an interpolation method to obtain them, as described in the following algorithm. Algorithm 3.2. Interpolation for computing lateral distances.. Input: A point I(u,v) in the ICS, a point set P, and another point set V. Output: The coordinates of I in the VCS. Steps: Step 1. Compute the varibales a and b of the line equation y = ax + b for lines L0, L1, L2, and L3 by using ICS coordinates of four endpoints, Pij, P(i+1)j, Pi(j+1), and. 23.

(35) P(i+1)(j+1) in the following ways, as illustrated in Figure 3.6. v2 − v1 . u2 − u1. (3.1). v1 × u2 − v2 × u1 . u2 − u1. (3.2). a= b=. Step 2. Decide whether the point I is in the region surrounded by four endpoints, Pij,. P(i+1)j, Pi(j+1), P(i+1)(j+1) by substituting (u, v) for (x, y) of the line equation in the following ways. (a0 ⋅ u + b0 − v) ⋅ (a2 ⋅ u + b2 − v) ≤ 0 .. (3.3). (a1 ⋅ u + b1 − v) ⋅ (a3 ⋅ u + b3 − v) ≤ 0 .. (3.4). Step 3. If the inequalities (3.3) and (3.4) are satisfied, the point I is in this region. Else, repeat Step 2 to check the next region. Step 4. By using a horizontal line equation y = v which passes the point I, we obtain two intersections H(uh, vh) and K(uk, vk) as shown in Figure 3.6(a). Step 5. By using a vertical line equation x = u which passes the point I, we obtain two intersections S(us, vs) and T(ut, vt). Step 6. Use an interpolation method to obtain the VCS coordinates (xI, yI) of I by the following equations:. uh − u ⎧ ⎪30 × ( j − 3) + 30 × (1 − u − u ), if j ≤ 3; ⎪ h k xI = ⎨ u u − k ⎪30 × ( j − 3) + 30 × ( ), if j > 3. ⎪⎩ u h − uk yI = Div + 30 × i − 7 + 30 × (. vs − v ). vs − vt. (3.5). (3.6). Where i is serial number of the horizontal lines and j is serial number of the vertical lines. Although the image is distorted slightly, we consider every line as a straight line. Every region is not rectangular as appearing in the image; therefore, we use the interpolation method to compute xI and yI respectively. 24.

(36) L1. Pij. Pi(j+1). T. L0. P(i+1)j. L2. I. K. Vij. VI. H S. L3. Vi(j+1). P(i+1)(j+1 ). (a). V(i+1)j. V(i+1 )(j+1). (b). Figure 3.6 An illustration of the interpolation method. (a) A region contains the point I in the ICS. (b) The projection of the region in (a) onto a floor region.. 3.3 Proposed Mechanical Error Correction Method There exist mechanic errors when the vehicle navigates back and forth, no matter how people control the vehicle or how the vehicle navigates automatically. Whenever the control instruction is moving ahead or backing off, the vehicle will move away from the original straight path gradually and the coordinates and the direction angle of the vehicle provided by the odometer will be different from the actual position and angle of the vehicle. A method is proposed for solving the above problem. A mathematical model is set up for correcting mechanic errors. In Section 3.3.1, we will describe the concept of the method briefly. Utilizing this mathematical model; we can adjust the coordinates and the direction angle of the vehicle provided by the odometer, as described in Section 3.3.2. Finally, we hope the vehicle can move along straight path automatically; hence, we propose a method to solve this problem, as described in Section 3.3.3. 25.

(37) Because there exists a similar problem when the vehicle backs off, we use the same concept to correct navigation errors. It is described in Section 3.3.4.. 3.3.1 Brief Description of Proposed Error Correction Model Due to mechanic errors, a straight path of the vehicle navigation will become a curve path. A detailed situation is illustrated in Figure 3.7. In Figure 3.7(a), the vehicle starts moving and the expected path is a straight line in advance. After moving a distance, the vehicle moves along path d instead of the expected path c. The same situation occurs when the vehicle backs off. In Figure 3.7(b), the vehicle backs along path f instead of path e.. . j. . j. Expected path. Expected path. Actual path. Actual path. (a). (b). Figure 3.7 Illustrations of navigation path deviation. (a) Moving ahead. (b) Backing off.. Besides the last problem, the coordinates and direction angle provided by the odometer do not show the actual position of the vehicle in the GCS due to the mechanic error. As shown in Figure 3.8, the vehicle moves from the starting position. P0 and arrives at position Pc finally. But the coordinates provided by the odometer are the point P1 instead of Pc and the direction angle is the original angle of the starting 26.

(38) position. Actually, because the vehicle moves along the curve path, the vehicle has rotated for an angle θ. Because the path of the vehicle is a curve, we compute a curve equation to represent the path. We measure the deviation distances in advance to build a mathematical equation as a correction model such that the vehicle can modify navigation errors dynamically. The concept is described as follows. At first, we get some the vertical deviation distance values measured manually when the vehicle navigates forward a distance. According to the values, a second-order equation y = ax2 + bx + c is obtained by a curve fitting technique. Among the equation, x is the distance between the starting position P0 and the position. P1, and y is the deviation distance from the expected path. Moreover, In Section 3.3.2, we will describe how to get the coefficients a, b, and c of the equation. This correction equation always accompanies the vehicle whether the vehicle is controlled by a user or navigates automatically. To correct the direction angle of the vehicle, we use the first derivative equation to compute the rotation angle. It is like a concept in physics mentioning that when the object is in a curvilinear motion, by using a tangent line, the direction of the vehicle can be computed. Based on such a principle, we can find a tangent line L1 as shown in Figure 3.8. We compute the slope of line L1 by using the first derivative of y = ax2 +. bx + c. The process of modifying the values of the vehicle position is shown in Figure 3.9 and a brief computing process is described as follows. A distance xd is gathered by computing the length between P0 and P1. Substituting xd into the equation y = ax2 + bx + c, we can get the deviation value yd. By using xd and the slope. dy = 2ax + b , we can get an deviation angle θ the vehicle dx. had turned in the GCS. Hence, by using coordinate transformation techniques, we can. 27.

(39) use xd, yd and θ to get actual position Pc. The details are described in Section 3.3.3. Every time the vehicle starts moving, the above method is carried out. As long as the vehicle stops moving, the correct values of xd, yd and θ must be computed again in the next moving circle.. y = ax 2 + bx + c Pc. L1. yd. P0. ⎠. P1. L0. xd Figure 3.8 An illustration the correction model.. Coordinates. Start correction. y´ = 2ax + b. Compute the navigation distance xd. xd. Compute the deviation distance yd. yd. Compute the deviation angle ¬. ¬. Compute correct coordinates. Correct the direction angle ¬¬. ¬¬. Correct the coordinates of the odometer. y = ax2 + bx + c. End correction. Figure 3.9 Flowchart of correction process.. 3.3.2 Curve Fitting for Navigation Path In Section 3.3.1, we have described the concept of the correction model. According to the discussion in the last section, to build the second-order curve equation is the most basic requirement. All correction computation is based on this. 28.

(40) curve equation, so we describe how we compute the coefficients of the equation first in this section. A detailed process is described in the following algorithm. Algorithm 3.3. Curve fitting for navigation path.. Input: A navigation path. Output: An equation y = ax2 + bx + c. Steps: Step 1.. Measure a deviation distance ydv every 0.5m when the vehicle navigates, as shown in Figure 3.10.. Figure 3.10 An illustration of recording the coordinates.. Step 2. Record the deviation distance ydv when the vehicle navigates a distance x, and denote the data as P1j(xj, y1,j) where xj is navigation distance computed between the starting position and the new position provided by the odometer and y1,j is a deviation distance. We record thirteen data when the vehicle moves a 6m distance. Finally, we group the data into a point set P1 = {P1,0, P1,1, …, P1,12}. Step 3. Repeat Step 1 two times and group the data into two sets P2 and P3. Step 4. There are three deviation values at each distance xj. Compute a mean set P in the following way; 3 3 3 1⎧ 3 ⎫ P = ⎨∑ Pi ,0 , ∑ Pi ,1 , ∑ Pi ,2 , ..., ∑ Pi ,12 ⎬ . 3 ⎩ i =1 i =1 i =1 i =1 ⎭. 29. (3.7).

(41) where the data in the set P are denoted as P j ( x j , y j ) . Step 5. By using the least square method, we compute the coefficients of the optimum curve y = ax 2 + bx + c according to x j and y j , as shown in Figure 3.11.. cm. Path curve. 30 25 20 15 10 5 0 0. 50. 100. 150 200. 250. 300 350. 400. 450 500. 550 600. 650 cm. Figure 3.11 A figure of the curve.. Because the deviations are slightly different in each navigation path, we measure each deviation three times and use their means to compute the curve equation. In this study, a, b, and c are computed to be a = 0.00008, b = −0.0089, and c = 0.5828. The distance unit we use is cm.. 3.3.3 Coordinates And Angle Correction After computing the path curve equation, the main goal is to adjust the odometer values of a navigating vehicle. Utilizing the curve equation; we can conjecture that the vehicle has shifted how much distance from a straight line when it is still to move a certain distance. A detailed description of the adjustment is described as follows. In Figure 3.12, the vehicle starts moving from point P0 ( x0 , y0 ) toward the goal point P1 ( x1 , y1 ) in the GCS. The direction of the vehicle in the starting position is θ.. 30.

(42) Due to mechanic errors, the final position the vehicle arrives at is point P1′( x1′ , y1′ ) . Although the position of the vehicle is P1′ and the angle has become θ + α , the odometer shows that the position and angle are P1 and θ . The point P1′ and angle. α is the most important data we want to modify the error of the odometer. We compute the distance D as follows: D = ( x1 − x0 ) 2 + ( y1 − y0 ) 2 .. (3.8). Utilizing D and the curve equation y = ax 2 + bx + c , the deviation distance d y is derived to be as follows: d y = aD 2 + bD + c .. (3.9). Then we use the coordinate transformation technique to get the point P1′ as follows: x1′ = D ⋅ cosθ − d y ⋅ sinθ + x0. (3.10). y1′ = D ⋅ sinθ + d y ⋅ cosθ + y0. (3.11). y P1′. dy. L0. P1. α +θ P0. θ D x. G. Figure 3.12 An illustration of correction detailed.. 31.

(43) In the following, we want to get the rotation angle α , as shown in Figure 3.13. Utilizing the tangent slope y′ = 2ax + b , we can get vector b = [1, 2aD + b] in the error correction model. Using the unit vector a = [1, 0] , we can get the angle α by the following formula:. α = cos −1 (. a •b ab. ).. (3.12). Finally, we correct the coordinates of the vehicle to be P1′ and correct the angle to be θ + α . P1′. b. α. P1. a. x. Figure 3.13 An illustration of computing correct angle.. 3.3.4 Straight Navigation Technique Because the vehicle always navigates along a curve path, in this section, we propose a technique to control the vehicle to navigate along a straight line automatically. The most important concept behind our technique is that we control the vehicle to return to the original straight path, as illustrated in Figure 3.14. The path of the vehicle is shown as a red line in Figure 3.14. Assume that the vehicle starts from point P0 , and plans to navigate along path L0 . According to the curve equation y = ax 2 + bx + c , we can get a distance dy when. 32.

(44) the vehicle has moved a distance Dt . We use these data to compute a moving distance and a rotation angle such that the vehicle can turn toward path L0 and go back to this path. Although we can correct the values of the odometer at any time, it is impossible to do this in the practical operation. Hence, we correct the mechanic error when the vehicle stops moving. When the vehicle goes to point P1′ , the vehicle stops and corrects the values of the odometer. To utilize the tangent line, we compute the rotational angle α . Point P2′( x2′ , y2′ ) which is on the path L0 is the goal for the vehicle to go back to path L0 . We compute the distance P1′P2′ by the following equation: P1′P2′ =. dy sinα. (3.13). .. P1′. dy. α P0. 2α P2′ P2. Dt. L0. φ L1. L2. Figure 3.14 An illustration of the correction path. The vehicle should turn 2α toward path L0 and go through the distance P1′P2′ back to L0 , as the path L2 shown in Figure 3.3.4.1. Unfortunately, the. vehicle does not arrive at point P2′ because of the curve path. If the vehicle goes toward point P2′ , the path will be like the blue line shown in Figure 3.15 and the terminal will be Pt finally, as shown in Figure 3.15. Hence, the vehicle has to turn an angle of φ such that it can navigate along the curve and arrive at P2′ finally. As 33.

(45) shown in Figure 3.15, the planed path is L1 and the expected goal is P2 ( x2 , y2 ) , such that the vehicle can arrive at P2′ finally. From the triangle U P1′P2′Pt , we have known the length P1′P2′ , hence we can compute the length Pt P2′ by the following curve equation: 2. Pt P2′ = a P1′P2′ + b P1′P2′ + c .. (3.14). And the angle ∠Pt P1′P2′ can be computed as follows:. ∠Pt P1′P2′ = tan −1 (. Pt P2′ P1′P2′. ).. (3.15). Let ∠Pt P1′P2′ be equal to ∠P2′P1′P2 which is denoted as φ , then we can get U Pt P1′P2′ ≈ U P2′P1′P2 . Hence, we can get the following relation:. P1′Pt P1′P2′. =. P1′P2′ P1′P2. (3.16) .. If the angle ∠Pt P1′P2′ is small enough, then we can get P1′Pt ≈ P1′P2′ . Hence, we can get P1′P2′ ≈ P1′P2 . Therefore, we get a navigation distance by computing the distance between P1′ and P2′ .. P1′. Pt L0. P2′ P2. φ. L2. L1. Figure 3.15 Computation of corrected path. 34.

(46) Summarizing the above description, we describe the correction process as follows: Step 1.. Start the vehicle from point P0 . According to value of the odometer, the vehicle stops at the point P1′ after going through a distance of Dt .. Step 2.. Correct the coordinates and the angle of the vehicle.. Step 3.. Compute the turn angle φ by using Equation (3.15) and navigate a distance P1′P2′ .. Step 4.. Turn the vehicle for the angle of 2α + φ toward the straight path L0 and let it navigate for a distance of P1′P2′ to arrive at point P2′ and stop.. Step 5.. Correct the coordinates and the angle of the vehicle again.. Step 6.. Turn the vehicle such that the direction of the vehicle is the same as the direction in the beginning position P0 at the point P0 in the GCS.. The above process is just only one correction circle among the navigation. If the path is longer, the vehicle has to repeat the above process many times. The final path which is the red line is shown in Figure 3.3.4.3. It is periodic and looks like a trigonometric function of mathematics.. Figure 3.16 An Illustration of corrected navigation path. 3.3.5 Correction of Backing off Due to the mechanic error the vehicle also backs off along a curve path. The navigation path when the vehicle backs off is shown in Figure 3.17. We use the same method as mentioned in last sections to correct the errors in this section. 35.

(47) Expected path Actual path. Figure 3.17 An illustration of Backing path.. As first, using curve fitting to compute the curve equation y = ax2 + bx + c, as shown in Figure 3.18. The equation is different from the equation computed when the vehicle moves ahead. The distance between the starting position and the now position is record as a negative value. The coefficients of a, b, and c are computed to be a = 0.0001, b = 0.002, and c = 0.0216, respectively, in this study.. Back path curve 20 15 10 5 cm. 0 -450. -400. -350. -300. -250. -200. -150. -100. -50. -5. 0. Figure 3.18 The path curve of backing off.. We still use the same method to correct the coordinates and direction angle of the vehicle. The correction equations are different from those of (3.10) and (3.11). The equation of computing the actual position ( x1′ , y1′ ) of the vehicle become as: x1′ = − D ⋅ cosθ − d y ⋅ sinθ + x0 ;. (3.17). y1′ = − D ⋅ sinθ + d y ⋅ cosθ + y0 .. (3.18). 36.

(48) Finally, the process of backing along the straight line is the same with the process of moving ahead.. 37.

(49) Chapter 4 A Method for Detection of Monitored Objects by Image Analysis. 4.1 Introduction We use a vision-based vehicle to monitor concerned objects and doors in this study. How to recognize objects and doors are an important problem in this study. Hence, before describing the vehicle navigation process, we will first describe the image processing and pattern recognition techniques we use for solving this problem in this chapter. There are two main processes to recognize an object in an image. One is to detect the object region, and the other is to compute the feature data of the object. To detect an object, we propose the use of an improved snake algorithm to accomplish the task. The details of the method are described in Section 4.2. After detecting objects, we have to compute useful feature data and save them for use in the learning process such that we can use them to monitor the objects in the security patrolling process. We will describe how to compute the feature data in Section 4.3.. 38.

(50) 4.2 Process of Monitored Objects Detection In this section, we will describe how to detect an object region in an image. We tried to use a famous object detection technique, called “Snake Algorithm” [1] in this study. In Section 4.2.1, we will briefly describe the basic concept of the snake algorithm. But the snake algorithm has many weaknesses when it is used in the complicated environments; hence, we will propose an improved version of the method to detect objects in indoor environments in Section 4.2.2. Some complicated environment backgrounds could influence the result of object detection; we will describe a feasible method proposed in this study in Section 4.2.4. In Section 4.2.3, we will describe the details of the proposed improved snake algorithm.. 4.2.1 Brief Description of Snake Algorithm The performance of the snake algorithm is like the behavior of an elastic band. Imagine that there is an elastic band in the image and an object exists in it. By repeating continuous operations, the elastic band becomes narrow and approaches the object edge. Finally, the elastic band will enclose the edge of the object and stop computing, as illustrated in Figure 4.1. We now describe the principle of the snake algorithm below. The snake algorithm uses a set of control points, effectively connected by straight lines. Each control point has a position, specified by coordinates (x, y) in the ICS, and the snake is entirely specified by the number and coordinates of its control points. By means of adjusting the positions of the control points and computing the. 39.

(51) energy of the snake at every moment, computation stops finally when the energy achieves the minimum value. The strength of traction is determined by the energy. When the control points are moved toward their center continuously, some image properties will influence energy computation and cause the control points to stop moving.. (a). (b). Figure 4.1 Performance of the snake algorithm. (a) The behavior of elastic band and an object in it. (b) Final result of repeating continuous computations.. The energy for a snake exists in two parts, the internal and external energies. That is,. E snake = ∫ Einternal v ( s) + ∫ Eexternal v ( s). (4.1). where v(s) = (x(s), y(s)) which specifies image coordinates. The internal energy Einternal is the part that depends on the intrinsic properties of the snake, such as its length or curvature. The external energy Eexternal depends on factors such as the image structure, and the particular constraints the user has imposed. Each control point has its internal and external energies and Esnake is the total energy of all the control points. The equation of the internal energy is written as. 40.

(52) 2. 2. Einternal = α ⋅ vs ( s ) + β ⋅ v ss ( s) .. (4.2). The subscript s denotes the derivative. The above equation is the sum of the first derivative and the second derivative of the control point. Also, the values α and β are the coefficients determined by the designer. The first derivate can be written as dvi ds. 2. 2. ≈ vi − vi −1 = ( xi − xi −1 ) 2 + ( yi − yi −1 ) 2 , i = 1, 2, ..., n .. (4.3). From the above equation, we can understand that when two control points are more close, the value of the first derivate will be smaller. Hence, the first derivate accounts for stretching of the snake and will have a large value when there is a gap between the control points. So, when the distribution of the control points becomes smaller, the internal energy will have a smaller value, as shown in Figure 4.2 (a) and (b). The second derivate can be written as d 2 vi ds 2. 2 2. ≈ vi -1 − 2vi + vi +1 = ( xi −1 − 2 xi + xi +1 ) 2 + ( yi −1 − 2 yi + yi +1 ) 2 .. (4.4). The second order term accounts for bending and will be large in a region of high curvature. When the edge of the object is smooth, the internal energy will have a smaller value, as shown in Figure 4.2(c) and (d). Therefore, when the control points are closer to each other and their distribution shape is smooth, a smaller internal energy is obtained.. 41.

(53) (a). (b). (c). (d). Figure 4.2 A illustration of internal energy. (a) A pattern has a larger internal energy. (b) A pattern has a smaller internal energy. (c) A pattern has a larger internal energy. (d) A pattern has a smaller internal energy.. Now we will focus on the external forces on the snake. These determine its relationship to the image. Suppose that we want a snake to latch on to bright structures in the image. We usually use the magnitude of the gradient in image processing.. 4.2.2 Description of Improved Snake Algorithm In the last section, we have described the principle of the snake algorithm. We will now describe the proposed method in this section. Before computing the Esnake energy, the locations of the control points have to be decided at first. In this study, the shape distributed by the control points is a rectangle at the beginning in the image plane. A larger object in the image will have a larger rectangle, and vice versa. The terminal of control point shrinking is the center of the 42.

(54) locations of the control points, as shown in Figure 4.3. The center of the control point Pcenter(x0, y0) is computed as: n. Pcenter ( x0 , y0 ) = (. n. ∑x ∑y i. i =1. n. ,. i =1. n. i. ).. (4.5). where n is the number of the control points, and xi and yi are the coordinates of the control points in the image plane. Each control point moves toward the center point Pcenter ( x0 , y0 ) in each cycle of computation. The moving path is computed as a vector V ( xv , yv ) in the following way:. ⎡ xv ⎤ ⎡ xi − x0 ⎤ V =⎢ ⎥=⎢ ⎥. ⎣ yv ⎦ ⎣ yi − y0 ⎦. (4.6). After computing Esnake each time, the control points move to the center point Pcenter ( x0 , y0 ) gradually. The new position of the control points in each moving step becomes: xi =. xv − 1 × xv + x0 , xv. (4.7) yi =. yv − 1 × yv + y0 . yv. Through repeated computations, the shape of the control point distribution is no longer a rectangle. The distribution will be deformed gradually and the control point is close to the center point Pcenter(x0, y0). Until the Esnake reaches a minimum value, .the process of the snake algorithm ends.. 43.

(55) Figure 4.3 Contribution of control point.. The strength of shrinking comes from internal energy computation. Besides the method that Section 4.2.1 mentions, we add the so-called centripetal force into the internal energy. The improved internal energy is computed in the following way: 2. 2. Einternal. 2 dv d 2v = α ⋅ i + β ⋅ 2i + γ ⋅ V . ds ds. (4.8). The detailed mathematical operation is specified by: Einternal = α ( xi − xi −1 ) 2 + α ( yi − yi −1 ) 2 + β ( xi −1 − 2 xi + xi +1 ) 2 +. β ( yi −1 − 2 yi + yi +1 ) 2 + γ ( xi − x0 ) 2 + γ ( yi − y0 ) 2 .. (4.9). We utilize the distance between a control point and Pcenter(x0, y0) as an element for computing internal energy. When the control point is closer to the point Pcenter(x0, y0), the distance becomes smaller gradually so that the internal energy has a smaller value. Hence, using centripetal force causes the control points to gather together. The internal energy causes the control points to gather, so we have to compute the external energy so that the moving of the control points can be stopped. The final result is that the control points surround an object. The Sobel operator is shown in Figure 4.4 and its equation is shown in Eq.(4.10). The following specifies its operation formula:. 44.

(56) −1. −2. −1. −1. 0. 1. 0. 0. 0. −2. 0. 2. 1. 2. 1. −1. 0. 1. Figure 4.4 Sobel operators.. I (u, v) = ( z7 + 2 z8 + z9 ) − ( z1 + 2 z2 + z3 ) + ( z3 + 2 z6 + z9 ) − ( z1 + 2 z4 + z7 ). (4.10). where z5 denotes gray value in the position (u,v), z1 denotes gray value in (u − 1,v − 1), and so on. By using the Sobel operator, the edge of an object becomes obvious, as shown in Figure 4.5. Hence, utilizing the value of the Sobel operator of the pixels on the control point’s position, we can get the external energy of a control point, as illustrated in the following way: Eexternal = λ × [minus (value of Sobel operator of the control point in. (4.11). 2. the image) ] where λ is a coefficient. Finally, the total energy, i.e., the snake energy, is computed as follows: n. n. i =1. i =1. i i Esnake = ∑ Einternal + ∑ Eexternal. (4.12). where n is the number of control points. Because the data are discrete, the integration is substituted by a summation. The snake energy Esnake is the sum of each control point’s internal and external energy. The internal energy is smaller when the control points are closer to the center such that the snake energy has smaller value. As soon as the control points bump into the edges, the external energy has a larger value such that the snake energy value 45.

(57) becomes larger; in the mean while the snake energy computation is stopped because the minimum snake energy is reached.. Figure 4.5 The result of using the Sobel operator.. The coefficients, α, β, γ, and λ play important roles in the snake energy. The magnitudes of the coefficients α and β influence the control points’ pulling force. The magnitude of coefficient γ influences the cohesion of the control points, and the magnitude of coefficient λ influences the result of the captured object edge. In this study, the magnitudes of α, β, γ, and λ are set to be 2, 3, 2 and 1, respectively.. 4.2.3 Computing Object Region After detecting an object, we have to recognize the object region from the image. Through the above repeated calculations, the control points surround the object edge pixels finally. Utilizing the coordinates of two adjacent control points to compute a straight equation y = ax + b, we can find out the pixels of the object edge by using this equation. A detailed process is described in the following algorithm. Algorithm 4.1. Object region extraction.. Input: A set of control points P. Output: An object region ObjR. 46.

(58) Steps: Step 1. For any two adjacent points Ps(xs, ys) and Pt (xt, yt), do the following steps. Step 1.1 Compute a straight line equation y = ax + b which crosses Ps and Pt by the following equation: ( a, b) = (. y t − y s y s xt − y t x s , ) , if ( xt − x s ) ≠ 0 . xt − x s xt − x s. (4.13). If xt − xs = 0, the equation becomes x = xt. Step 1.2 Compute the value y of the edge point coordinates between Ps and Pt by using the value of x in the following way: y = ax + b, x = xt to xs.. (4.14). Step 2. Find out the leftmost pixel and rightmost pixel denoted as (xL, yL) and (xR, yR ) after comparing the x coordinate of each edge pixel. Step 3. Find out the maximum and minimum y coordinates of each value of xi, denoted as (xui, yui) and (xdi, ydi). Step 4. For each pixel, Pi(xi, yi), if its x coordinate is between xL and xR and its value of y satisfies the following inequality, then Pi is considered as an object pixel: ydi ≤ yi ≤ yui.. (4.15). 4.2.4 Eliminating Noise of Background Because of the effect of the noise and edges in the background, the experimental result of object detection is usually not satisfactory, as shown in Figure 4.6. If a wall or the floor under the object is not clear, it usually influences the result of the moving of the control points. Hence, we have to improve the method for object detection.. 47.

(59) (a). (b). Figure 4.6 Effect of background on object detection (a) Before detection. (b) Result of detection.. Besides the methods of Section 4.2, we add the strength of pull and drag between the control points. We hope that the distance of adjacent control points can keep inside a reasonable domain. In order to achieve this goal, the number of the control points is determined by a beginning rectangle size. According to the length of a rectangle’s boundary and the coordinates of the four endpoints of the rectangle, we compute every control point position on the edge of the rectangle so that every control point can keep a fixed distance to each other. Utilizing the distance of the control points, we can adopt a threshold Dt such that the distance between two adjacent control points are kept inside Dt when the control points are moving along the path. As illustrated in Figure 4.7(a), the control point Pi ( xi , yi ) , and the two points, Pi −1 ( xi −1 , yi −1 ) and Pi +1 ( xi +1 , yi +1 ) are adjacent points of Pi. We compute the distances D1 and D2 by the following equations: D1 = xi − xi −1 + yi − yi −1 ;. (4.16). D2 = xi − xi +1 + yi − yi +1 .. (4.17). If the inequality D1 + D2 > Dt is true, then we will modify the position of the control point Pi , as shown in Figure 4.7(b). Utilizing the coordinates of the control 48.

(60) points Pi −1 and Pi +1 , we can get a center position mi between them by using the equation of mi =. Pi + Pi +1 . After computing a vector vi = mi − Pi , the position of the 2. 3 control point Pi is changed to the new position specified by Pi + vi . 4 Pi −1. Pi −1. 3 vi 4. D1. Pi. Pi. mi D2. Pi +1. Pi +1. (a). (b). Figure 4.7 Correction of control points. (a) Before correction. (b) After correction.. An experimental result of the above-mentioned method is shown in Figure 4.8.. (a). (b). Figure 4.8 Results of improved method. (a) Detecting a ball. (b) Detecting a tub. The vehicle sometimes will navigate on a decorative floor, as shown in Figure 4.9(a). Before detecting the object, we have to filter the floor region by using a region growing technique. A result is shown in Figure 4.9(b). When the position of control point is on the floor region, it is moved without being stopped until it leaves the floor region. 49.

(61) (a). (b). Figure 4.9 Elimination of a floor region. (a) A decorative floor. (b) An experimental result.. 4.2.5 Detailed Object Detection We have already described the concept behind the proposed improved snake algorithm in the above sections. The detail of the proposed object detection process is described in the following algorithm. Algorithm 4.2. Object detection.. Input: An object image I. Output: An object region ObjR. Steps: Step 1.. Specify manually four end points of a rectangle in the ICS as initial control points and compute the coordinates of the four points.. Step 2. Compute the coordinates of the other control points according to the coordinates of the four endpoints. Step 3. Compute the center point Pcenter by using the coordinates of all the control points.. 50.