⚳
ġ䩳
ġṌ
ġ忂
ġ⣏
ġ⬠
ġ
ႝπำࣴز܌!
റ
!
γ
!
ፕ
!
Ў
!
ӭख़གޕӝԄᇶշӼӄًၩၗ೯ૻس
!
An Assistant Safety Telematics System with Integrated Multiple Sensors
ࣴ!ز!ғǺ؇η!
ࡰᏤ௲Ǻ݅ᐩ!!௲!
!
!
!
ӭख़གޕӝԄᇶշӼӄًၩၗ೯ૻس!
An Assistant Safety Telematics System with Integrated Multiple
Sensors
ࣴ ز ғǺ؇η StudentǺTzu-Kuei Shen
ࡰᏤ௲Ǻ݅ᐩ ௲ AdvisorǺProf. Chin-Teng Lin
୯ ҥ Ҭ ೯ ε Ꮲ
ႝπำࣴز܌
റ γ ፕ Ў
A Dissertation
Submitted to Institute of Electrical Control Engineering College of Electrical and Computer Engineering
National Chiao Tung University in Partial Fulfillment of the Requirements
for the Degree of Doctor
in
Electrical Control Engineering November 2010
Hsinchu, Taiwan, Republic of China
ӭख़གޕӝԄᇶշӼӄًၩၗ೯ૻس!
ᏢғǺ؇η!!
ࡰᏤ௲
Ǻ݅ᐩ!௲!
!
୯ҥҬ೯εᏢႝπำࣴز܌ റγ
ᄔ
ा
! ! ! ! ! !
߈ԃٰᒿѱΓαکً፶ኧҞόᘐޑቚуǴѱޑҬ೯ୢᚒຫٰຫ
ᝄख़Ǵၸӭޑً፶ԋΑҬ೯Ꮮ༞ޑݩǴҬ೯ཀѦ٣ࡺޑวғΨ׳уޑ
ᓎᕷǴ೭٤٣ࡺԋΑΓ҇ғڮᆶౢޑཞѨǴᡣঁΓѳ֡ӧᙴᕍ
ޑҔε൯ޑቚуǴԋΑᚳεޑޗԋҁᆶॄᏼǴ٠फ़ե୯ৎޑᡏ
ᔮᝡݾૈΚǶӢԜǴӚঁӃ୯ৎΕΑ࣬ӭޑΓΚӧࣴزඵችࠠၮᒡ
س(Intelligent Transportation Systems, ITS)ǴITS!ޑЬाҞޑࢂճҔӃࣽ
מܭً፶ϷၰၡࡼǴڐշᎯᎭჹً፶ϐڋǴа෧Ͽ٣ࡺǴቚՉً
Ӽӄ٠ၲډගଯҔၡਏᆶૈ෧ᅹޑҞǶҁፕЎࣁֹӭख़གޕ
ӝԄᇶշӼӄًၩၗ೯ૻسǴځύхً֖፶ӼӄᇶշسᆶၡαӼӄᅱ
ٿҽǶً፶ӼӄᇶշسճҔൂᗭങ᜔ᓐឪቹᐒჹً፶Չڬᜐም
ᛖނୀෳᆶॹًᇶշᎯᎭǶڬᜐምᛖނୀෳسࢂၸϸࢀኳࠠஒቹႽ
аႝတຎࣁ୷ᘵǴԾෳॹًਔޑ࣬ჹ౽ໆǴܭฝय़ෳр҂ٰ
ޑՉॉၞǶၡαᅱޑҽǴЬाࣁ٣ҹᅱسǴ໒วቹႽ൪Ε
ԄسѳѠҔܭԏၡα࣬ᜢၗૻǴ٠җၡୁൂϡௗԏᆶ༼Ǵӆҗ ETSD
ᆶًᐒՉၗૻҬࢬǴӝًᆶߕ߈ၡαၗૻ๏ᎯᎭޣୖԵǴࡌҥଆ
ֹޑᇶշӼӄًၩၗ೯ૻسǶ!
An Assistant Safety Telematics System with Integrated Multiple Sensors
Student
ǺTzu-Kuei
Shen
Advisors
ǺProf. Chin-Teng Lin
!
Institute of Electrical Control Engineering
National Chiao Tung University
ABSTRACT
In recent years, the urban population and vehicles increase continuingly.
The problems of city traffic are more and more serious, such as too many
vehicles cause traffic jams and accidents frequently. These accidents do not only
make people lost their life and fortune, but also waste lots of medical resources.
Upper disastrous influences make enormous social costs, besides they also
debase whole national economic competitiveness. For dealing with these
problems, Intelligent Transportation System (ITS) becomes an important policy
in each country. The main objective of ITS is to develop high-end technology on
the electrical equipment in vehicles and traffic applications. Drivers can reduce
the probability of traffic accidents and improve self-driving safety via
controlling high-end assist driving technology and then achieve the goals of
increasingly efficiency in road freight and energy saving and carbon reduction.
This dissertation presents a whole integrated multi-sensors telematics safety
system and parking assistant system. The obstacle detection system one is
transferred image coordinate into world coordinate by fisheye lens inverse
perspective mapping modal (FLIPM) and follows the property of moving
obstacle to position candidates’ location. The parking assistant system is based
on computer vision algorithm via motion vector, and estimates the curve in the
path of vehicle. The other segment is intelligent intersection surveillance system.
Our concern is to consider a whole intersection events monitor system. It
collects the traffic data from local intersection by embedded platform and then
arranges these data for road side unit (RSU) to communicate with on board unit
(OBU) in vehicles via DSRC protocol to set up an assistant safety telematics
system.
Acknowledgement
ӧཥԮҬεറγᏢғࢲޑ೭ϖԃ္Ǵҗ૱ޑགᖴࡰᏤ௲!݅ᐩԴৣ๏ϒךᏢ ೌᆶࡑΓೀ٣ޑӭ௴วǶӧࣴزޑၸำύගٮΑֹ๓ޑഢᆶᔅշǴӕਔΨᙖҗ ӭӝբᐒǴ٬ךૈஒፕᆶჴ୍่ӝǴԶჴӭബཥཷۺޑཥᔈҔǴ٠ගϲךޑ ࣴزᆶჴբૈΚǶӕਔǴҭགᖴҭα၂ہ!ླྀكࢩԴৣǵ!ד҉Դৣǵ!݅ЎണԴৣǵ! ᎄሎඦԴৣǵ!٫ᑫԴৣᆶ!ᇀകറγӧԭԆϐύܜޜᆿᖏࡰᏤǴ๏ϒךӭࡌ٬ ҁፕЎ׳ᖿֹ๓Ƕ! ೭ϖԃӭޑറγғࢲ္Ǵ२ाགᖴ!ᇀകറγ๏ϒךғࢲǵౢӝբکሦ ୱޑӭࡌقǴаϷ!णখᆢറγܭࣴزၸำᆶำԄኗቪਔޑӭᔅշǴவգॺٿΓޑ يךᏢډΑӭࣴวᆶ࣮Γǵ٣ǵނޑБݤᆶྗ߾ǶԜѦǴाགᖴࡌᆶฑၲӕᏢ ೭ϖԃӭٰޑឫЋӅǴόፕࢂፐǵࣴز܈ࢂғࢲգॺࢂന٫ޑუՔǶќѦǴ ᗋाགᖴܿᓄǵയඵᆶᆬᏢǴჴᡍ࠻ԖգॺޑڐշǴωёаֹԋӭޑӝբਢǴ ᡣჴᡍ࠻ૈԖӵϞޑԋ݀ǶќѦǴΨाགᖴӭޑᏢۂޑഉՔǴӧ೭ϖԃ္εৎӕ ᏢಞǵࣴزᆶፕޑၸำύǴᡣ۶Ԝ׳Ǵᡣךӆ೭٤кჴޑВη္੮Πऍӳޑ ӣᏫǶ! ךाགᖴךޑР҆ǵٿՏёངޑۂۂᆶךޑۢРۢ҆ǴգॺޑᜢЈᆶᔅԆωૈᡣך ӧറγޑᏢғఱ္ЈคᜰǴёаЈठΚܭᏢǴ٠ճֹԋፕЎǶќѦǴाձ གᖴԴஇےᆗǴགᖴی೭ሶߏਔ໔ޑ࣬ഉᆶᆶЍǴیޑЍࢂךँઇᓍᆶܭௗڙ ཥ٣ނࡷᏯޑচΚǶനࡕǴाགᖴζٽऎǴیޑрғᡣך׳ԖΚѐֹԋറγᏢՏǶ നࡕǴаԜፕЎ๏܌ԖᜢЈךޑΓǶ! !Table of Contents
Page Chinese Abstract ... i English Abstract ... ii Acknowledgement ... v Table of Contents ... vi Figures ... viii Tables ... x 1. Introduction ... 11.1 Background and Motivation ... 1
1.2 Objective and Methods ... 2
1.3 Organization ... 2
2. Related Work ... 4
2.1 Telematics ... 4
2.2 Inverse Perspective Mapping (IPM) ... 10
2.3 Obstacle Detection ... 12
2.4 Object Tracking ... 14
3. Structure of Safety Assistant Driving Telematics System ... 16
3.1 System Overview ... 16
3.2 On Board Vision-Based Detection System ... 19
3.2.1 Embedded Platform Description ... 19
3.2.2 Sinffer Equipment ... 22
3.3 Intelligent Surveillance System ... 25
3.3.1 Improved Intelligent Surveillance System ... 25
3.3.2 Vision-based incident detector... 26
4. Vision-Based Intelligent Technique ... 28
4.1 Fisheye lens inverse perspective mapping (FLIPM) ... 28
4.1.1 The Modified Normal Lens IPM Method ... 29
4.1.2 Fisheye Lens Inverse Perspective Mapping (FLIPM) ... 33
4.2 Obstacle Detection with single Camera... 37
4.2.1 The Pre-Process ... 37
4.2.2 Profile image ... 37
4.2.3 The temporal FLIPM difference image ... 39
4.2.4 Road Detection ... 42
4.2.5 Ground Movement Estimation ... 48
4.2.6 Obstacle Feature Searching algorithm ... 54
4.2.7 Histogram Post-processing ... 58
4.2.8 Object Tracking and Information Extraction ... 59
4.3 Dynamic Distance Gauge (DDG) ... 66
4.3.1 System architecture of the DDG ... 66
5. Experiment Results ... 70
5.1 Obstacle Detection Experiment ... 70
5.1.1 Comparisons about the Normal Lens IPM Method ... 72
5.1.2 The Experimental Configurations ... 73
5.1.3 Results in Various Environments ... 73
5.1.4 Accuracy Evaluation of Obstacle Distance ... 82
5.3 Entire System Experiment ... 85
5.3.1 Environment description ... 85
5.3.2 Protocol Test Results ... 86
5.3.3 Intersection Test Results ... 88
5.4 Discussions ... 91
6. Conclusions ... 92
Figures
Figure 1: Harmonized 5.9 GHz DSRC BAND PLAN ... 5
Figure 2: DSRC Performance envelops ... 6
Figure 3: The structure of SAE J2735 ... 7
Figure 4: A package processing step... 7
Figure 5: Structure of Safety Driving Assistant Telematics System... 16
Figure 6: Structure of Telematics service platforms ... 17
Figure 7: Current application on telematics system ... 18
Figure 8: Vehicle Sensors ... 19
Figure 9: Structure of Intelligent Image Processing Embedded Platform ... 19
Figure 10: Structure of Embedded Platform PCB Design ... 20
Figure 11 : S100 WAVE Box ... 22
Figure 12: S100 detail Spec ... 22
Figure 13: Sirit Sniffer Card (PCMCIA Interface) ... 23
Figure 14: OBU Service and Navigation Structure ... 23
Figure 15: The flowchart of transmitting message ... 24
Figure 16: Traditional intelligent surveillance system ... 25
Figure 17: Improved intelligent surveillance system ... 25
Figure 18: Fisheye lens inverse perspective mapping structure ... 28
Figure 19 : The vertical line projection of Eq. (4.1) ... 30
Figure 20: The projected result of Eq. (4.5) ... 30
Figure 21: The figures and expected results ... 31
Figure 22: The geometrical relations of the image and world coordinate system for deriving our equations. ... 32
Figure 23 : The original and adjusted scope ... 34
Figure 24: Illustrations for distortion images ... 35
Figure 25: The flowchart of image pre-processing ... 37
Figure 26: The results in the profile searching process ... 38
Figure 27: Illustrations for the temporal FLIPM difference image ... 39
Figure 28: Results of edge detection and its corresponding optical flow ... 41
Figure 29: Results of corner detection and its corresponding optical flow ... 42
Figure 30: Block diagram of feature point extraction ... 42
Figure 31: A color ball i in the L*a*b* color model whose center is at (Lm, *am, *bm) and with radius λmax ... 44
Figure 32: Sampling area and color ball with a weight which represents the similarity to current road color. ... 45
Figure 33: Pixel matched with first B weight color balls which are the most represent standard color. ... 46
Figure 34: Results of road detection ... 47
Figure 35: Results of feature point extraction. The upper image is result of road detection, and lower image is position of feature points... 48
Figure 36: Flowchart of ground movement estimation ... 48
Figure 37: Difference between optical flow of original image and those of bird’s view image when vehicle is moving straight. ... 49
Figure 38: Difference between optical flow of original image and those of bird’s view image when vehicle is turning. ... 50
Figure 39: Differences between optical flows of obstacle and those of planar object. ... 50
Figure 40: Histogram distribution of optical flow in world coordinate system ... 51
Figure 42: Procedure of the compensated image building ... 53
Figure 43: Chart of temporal coherence ... 54
Figure 44: The results in the feature searching procedure by using profile images ... 55
Figure 45: The results of the feature searching procedure using temporal difference FLIPM images ... 56
Figure 46: The flowchart of feature searching ... 57
Figure 47: The block diagram of histogram post-processing ... 58
Figure 48 : Illustrative figures of the trapezoid histogram distributions. ... 58
Figure 49: The processes of histogram post-processing. x axis means the polar histogram’s angle and y axis means the accumulation on each angle. ... 59
Figure 50 : The block diagram of object tracking and information extraction... 59
Figure 51: Flowchart of Obstacle Detection ... 60
Figure 52: Results of image difference between current image and compensated image ... 61
Figure 53: Flowchart of obstacle localization ... 61
Figure 54: the obstacle candidate image and the corresponding vertical-orientated histogram ... 62
Figure 55: Procedure of creating vertical-orientated histogram ... 63
Figure 56: Procedure of obstacle verification ... 64
Figure 57: Transformation between image coordinate and world coordinate ... 64
Figure 58: Scale measure between world coordinate and real length ... 65
Figure 59: Block diagram of distance measurement ... 65
Figure 60: System architecture of the proposed DDG algorithm ... 66
Figure 61: Allocation of motion detection areas. ... 67
Figure 62: The scheme of block matching ... 67
Figure 63: The results of the normal lens IPM equations ... 72
Figure 64: The set-up location of camera. ... 73
Figure 65: The results of FLIPM and obstacle detection in different scenes. ... 76
Figure 66: Results of obstacle tracking in Scenery 1 ... 77
Figure 67: Results of obstacle tracking in Scenery 3 ... 77
Figure 68: Results of obstacle warning in the lateral direction ... 78
Figure 69: Results of obstacle warning in the rear direction. ... 78
Figure 70: Results of obstacle warning with moving objects. ... 79
Figure 71: The results in different environments with heavy noise. ... 81
Figure 72: Lane marking for distance measurement ... 82
Figure 73: Testing results of DDG in various environments. ... 83
Figure 74: Test Street Location... 85
Figure 75 Experiment structure for testing transmission rate... 86
Figure 76: WSM Captured Message ... 86
Figure 77: WSM header ... 87
Figure 78 : The locations of camera on setting intersection ... 88
Figure 79: The location of fix camera on intersection ... 88
Figure 80: Simulating a pedestrian crossing intersection ... 89
Tables
Table 1: Content sets of SAE J2735 ... 8
Table 2: VSC-A Basic Safety Message ... 9
Table 3: Features of BlackFin 561 DSP ... 20
Table 4: SK34B Electrical Characteristics ... 21
Table 5: SAE J2735 WSM Message Protocol ... 23
Table 6: The specifications of our working platform ... 70
Table 7: The runtime in each processing step ... 70
Table 8: Comparisons of different obstacle algorithms ... 71
Table 9: Experimental result of distance measurement ... 83
Table 10: Parameters of Testing ... 86
Table 11: Static test (600 packages per test) ... 87
Table 12: Active test (600 packages per test, 30 Km/hr) ... 87
Table 13: The minimum distance of stop view and overtake view ... 89
1. Introduction
1.1 Background and Motivation
In recent years, there has been a dramatic proliferation of vehicle and population in the world. This city phenomenon has caused many traffic and environmental protection related problems, such as collision, traffic jam, traffic offence, exhaust emission and so on. In order to solve these problems, many advanced countries have started to develop Intelligent Transportation System (ITS). Up to now, ITS can be divided into automotive electronics technology and traffic control system two research fields. Driver’s fatigue, drowsiness, inattention, and distraction are reported a major causal factor in many traffic accidents. Due to the drivers lost their attention, they had markedly reduced the perception, recognition and vehicle control abilities. Since, related studies had become a major interest research topic in automotive safety engineering. Previously, vehicle detect obstacle via radar sensor, but its response speed is not immediately. Collision happening occurs if the warning signal of radar sensor is presented with a little delay time. Nowadays, mounting a fisheye lens camera on the bumper is a general method to let driver to see reverse direction scene. However, fisheye lens scene has a serious distortion formation of image. This effect makes driver to estimate distance and control the direction of vehicle difficultly. In the other field, microwave, ground loop and radar are now three kinds of popular methods for traffic control applications. All of them do not reconstruct traffic events specifically. On the roads, camera and digital video recorder (DVR) are used to monitor road condition and catch traffic offense. Management costs too much to employ people attending these surveillance systems. Since, a part of researches interesting in vision based traffic control and analysis start to aim the objectives of ITS, such as vehicle detector, event detector, self-adaptive traffic light control and so on. When we can get so much useful traffic related information from these sensors, we may begin integrating all of them to assist drivers keep danger away. Therefore, the concept of telematics is to exchanging information by any kind of detector sensor on the vehicle and road intersection. It a whole system focuses on communicating data, for instance, traffic condition, driver’s spirit state and navigation information etc.. Summarily, combing the advantage of vision-based sensor and
1.2 Objective and Methods
This thesis presents a whole safety driving assistant telematics system including obstacle detection with fisheye lens camera on vehicle and road condition surveillance system. We develop a vision-based obstacle detection system by utilizing our proposed fisheye lens inverse perspective mapping (FLIPM) method. The new mapping equations are derived to transform an images captured by a fisheye lens camera into an undistorted remapped ones under practical circumstances. In the obstacle detection, we make use of the features of vertical edges on objects from remapped images to indicate the relative positions of obstacles. In order to obtain a suitable feature, adaptive road recognizing is the first step to extract obvious useless compensation points and mitigate interference by shadow and illumination changing. Our obstacle detection can export a warning signal on the screen within a limited distance from nearby vehicles while the detected obstacles are even with the quasi-vertical edges.
Road condition surveillance system contains intelligent detector and communication module. We improve traditional surveillance system by integrating intelligent detectors and developing road side unit (RSU) and on board unit (OBU) to exchange traffic information. We set up multiple cameras on the intersection to capture video streams from different directions. First, we make use of adaptive Gaussian Mixture Model (GMM) to form basic background. We will track foreground objects’ location and predict their future path for estimating collision occurrence. According foreground objects’ moving speed and appearance, we classify pedestrian and vehicle to record the behavior of this intersection. We also establish a world coordinate system to map each camera view field. Upper information will be collected by RSU and transfer them to OBU for driver to remind possible danger. Our messages follow WAVE/DSRC SAE J2735 protocol and the effective transmission distance is over 100 meters long. This intelligent system also contains back-end storage equipment and control human interface with DVR and content management system.
1.3 Organization
This thesis is organized as follows: Chapter II gives an overview of related work about this research realm and discusses the traditional methods. Chapter III introduces our proposed system divided into three parts. First, we will show a new design of intelligent surveillance system. On the front-end, we present the structure of OBU and show the hardware configuration on a sport utility vehicle. On the back-end, RSU collects and process information
captured from multiple intelligent sensors. Chapter IV presents technologies used in each intelligent sensor. The sensors on vehicle contain FLIPM, obstacle detection with single camera and dynamic distance gauge (DDG) three techniques, and on intersection contain collision estimation, moving objects classification and coordinate correction. Chapter V shows experimental results of the implementation of our proposed algorithm and the successful working on actual environment. Finally, we made a conclusion of this study in Chapter VI.
2. Related Work
Our safety assistant driving system has four related researches. The first is telematics fundamental structure including WAVE/DSRC international protocol. Next, we will aim at inverse perspective mapping methods and indicate unsuited spot. Besides, correcting distortion image captured with fisheye lens is also an interesting study. There are two kinds of methods to deal with this problem. Third is the most important subject in safety assistant driving system, and we will confer deeply past methods with different sensors. Finally, we restrict the scope of tracking algorithm without recognition and classifying method. In this chapter, we will review some researches related to these main modules.
2.1 Telematics
Telematics includes two important compositions, one is telecommunications and the other one is informatics. Since Global Positioning System technology is applied to navigate on vehicle driving, the first telematics system would be produced. Nowadays, there are twelve kinds of popular contents for telematics applications, including telematics education, vehicle tracking, trailer tracking, cold store freight logistics, fleet management, satellite navigation, mobile data and mobile television, wireless vehicle safety communications, emergency warning system for vehicles, intelligent vehicle technology, car clubs and auto insurance etc. [1]. In our telematics system, we follow the international standard protocol called Wireless Access in the Vehicular Environment (WAVE)/ Dedicated Short Range Communication (DSRC). It follows IEEE 802.11p and IEEE 1609 international communication standard protocol, and suitable to exchanging information in short distance for safety assistant driving. The communication module is a multi-channels structure, and its licensed band is 5.8 to 5.9GHz. There are seven channels with 10MHz Bandwidth, and the harmonized 5.9 GHz DSRC Band PLAN as shown in Figure 1.
Figure 1: Harmonized 5.9 GHz DSRC BAND PLAN Source: Industrial Technology Research Institute
Standard of IEEE 802.11p is based on CALM [2] and ASTM [3] E2213-03. The former is an European system to provide a standardized set of air interface protocols and parameters for medium and long range, high speed ITS communication using one or more of several media, with multipoint and networking protocols within each media, and upper layer protocols to enable transfer between media. Its communication modes are on V2I, I2I and V2V. The other one follows International standard organization with US playing a major role. E2213-03: Standard Specification for Telecommunications and Information Exchange Between Roadside and Vehicle Systems - 5 GHz Band Dedicated Short Range Communications (DSRC) Medium Access Control (MAC) and Physical Layer (PHY) Specifications. It purposed to provide wireless communications over short distances between information sources and transactions stations on the roadside and mobile radio units, between mobile units, and between portable units and mobile units. DSRC performance envelopes are shown in Figure 2.
Figure 2: DSRC Performance envelops
Source: Industrial Technology Research Institute
IEEE 1609 (Full use) contains five services including 1609.1, 1609.2, 1609.3, 1609.4 and 1609.11. 1609.1 is remote management service and 1609.2 is security services for applications and management messages. 1609.3 is networking services, and 1609.4 is multi-channel operation. Finally, 1609.11 is over-the-air data exchange protocol for ITS.
DSRC SAE J2735 standard protocol defines Message Sets, Data Frames and basic Data elements, and exchange data with ASN.1 (Abstract Syntax Notation one) DER (Distinguish Encoding Rules). It is a popular communication method to encode and decode around the world, and easy to combine with other WAVE/DSRC equipment. Here we only discuss two kinds of related messages protocol to represent SAE J2735. One is Basic Safety Message (BSM), and the other one is Emergency vehicle Alert (EVA). DSRC SAE J2735 is based on Wave Short Message Protocol (WSMP) that contains many related safety driving message sets. In general, the structure of Safety Message Handler (SMH) is shown in Figure 3. The function of SMH is to transfer and receive data from upper and down layers. Supporting safety application message protocol with decode and encode capability can auto choose suitable coding rules for different applications either be receiving or transmitting mode. It also can filter some useless information and abandon them to only keep defined kinds of data.
Figure 3: The structure of SAE J2735 Source: DSRC SAE J2735 Rev 31.
There is an important issue on correctly processing and maintain all kinds of safety information from multiple vehicles. In Figure 4, we show a package processing step, each package has a unique identifier to make the differentiation with other sources.
Table 1 shows a list of SAE J2735 defined messages. All messages have their own purpose and fitting protocol.
Table 1: Content sets of SAE J2735 Contents 1 MSG_Al_a_Carte (ACM) 2 MSG_BasicSafetyMessage 3 MSG_CommonSafetyRequest (CSR) 4 MSG_EmergencyVehicleAlert (EVA) 5 MSG_IntersectionCollisionAvoidance (ICA) 6 MSG_MapData (MAP) 7 MSG_NEMA_Corrections (NEMA) 8 MSG_ProbeDataManagement (PDM) 9 MSG_ProbeVehicleData (PVD) 10 MSG_RoadSideAlbert (RSA) 11 MSG_RTCM_Corrections (RTCM)
12 MSG_SignalPhaseAndTiming Message (SPAT) 13 MSG_SignalRequestMessage (SRM)
14 MSG_SignalStatusMessage (SSM)
15 MSG_TravelerInformation Message (TIM) 16 MSG_BasicSafetymessage_Verbose (BSV)
Basic safety driving information contains location, speed, and direction of moving or static vehicle. It is composed with two different kinds of elements. One contains upper necessary information of vehicle, and the other one depends on user selection. In Table 2, it is the VSC-A Basic Safety Message composing structure.
Table 2: VSC-A Basic Safety Message VSC-A Basic Safety Message
Part I
Message Sequence Number Temporary ID
Time
Position Latitude, Longitude, Elevation, Accuracy Vehicle Speed, Heading, Steering Wheel Angle Vehicle Accelerations, Yaw Rate
Brake Status
Vehicle Length, Width Part II
Vehicle Events Object Vehicle Path History Object Vehicle Path Prediction Object
Vehicle Relative Positioning RTCM 1002 Data Object
In the newest version of SAE J2735 Rev. 31, it already defines 16 message protocols, 71 data structures and 147 basic data units. All of them are applied on V2V and V2R message formats in application layer. The objective is to standardize the message format on RSU and OBU with different hardware platform.
2.2 Inverse Perspective Mapping (IPM)
In general, the objective of camera calibration is to extract the intrinsic and extrinsic information of the camera and the extracted information could be used to reconstruct the 3D world coordinate. Nevertheless, the performance of camera calibration would depend on the perspective eggect, lens distortion, and the number of cameras. An alternative method, namely inverse perspective mapping (IPM), was proposed to reconstruct the 3D world coordinates by using a single camera only. Broggi et al. [4, 5] utilized the IPM method and stereo cameras to detect obstacles in front of the vehicle, and implemented the parallel processor for image checking and analysis (PAPRICA) system Single Instruction Multiple Data (SIMD) computer architecture, to construct their obstacle and lane detection system, called GOLD (Generic Obstacle and Lane Detection) [5]. The GOLD implemented in the ARGO (derived from Argo and Argus, a research group from Italy) experimental vehicle made automatic driving possible. Ji [6] utilized IPM to get the 3D information of the front vehicle, and Cerri and Grisleri [7] presented the stabilized sub-pixel precision IPM image and the time correlation to estimate the possible driving space on highways. Muad et al. [8] used IPM to implement lane tracking and gave discussions of the factors which might have the influences on IPM. Tan et al. [9] combined IPM and the optical flow to detect obstacles for the lateral blind spot of the vehicle. Jiang et al. [10] proposed the fast IPM algorithm and used it to detect lanes and obstacles. Nieto et al. [11] introduced how to stabilize IPM images by using vanish point estimation. However in their approaches based on IPM, the planar objects such as lane markings were eliminated and the prominent objects like quasi-triangle pairs were reserved.
Our algorithm used the IPM’s property; therefore, the polar histograms derived from the IPM images could help to obtain the information of images in 1-D distributions. For separating from non-planar obstacles, we also constructed a novel method to detect and localize obstacles. With the intrinsic and extrinsic parameters from camera calibration, the obstacle detection system could establish a transformation table for mapping the coordinates of real-road surfaces into the distorted image coordinates. The objective of IPM method was to remove the perspective effects caused by cameras, and the higher performance of IPM methods made it possible to achieve better image processing results. Since IPM methods have been proven to be more efficient and applicable to real traffic conditions, we would focus on developing an accurate IPM algorithm for both normal lens and fisheye lens by improving the previous IPM methods. Our obstacle detection system aimed at detecting obstacles with either vertical or quasi-vertical edges. In fact, the obstacles with the significant height in vertical or
quasi-vertical edges could be mapped to the radial lines of the transformed bird-view images. As a result, we could deal with the transformed images to extract the profile of edges and obtain the polar histogram for post-processing.
!!
2.3 Obstacle Detection
The performance of those detection methods would obviously depend on the height, width, distance, and shape of an obstacle. There have been some other methods proposed for obstacle detection. Lai [12] used both of vision and the ultrasonic sensors on the mobile robot to detect the wall in the indoor environment. For the pedestrian detection, Curio et al. [13] used the contour, local entropy, and binocular vision to detect pedestrians. Bertozzi et al. [14] utilized stereo infrared cameras and three steps including warm area detection, edge-based detection, and v-disparity computation to detect pedestrians and used the morphological and thermal characteristics of heads to validate the presence of pedestrians. Though infrared cameras could perform well in either daytime or nighttime, the applications would be still restricted because of the higher prices of those cameras. There have existed many kinds of features such as symmetry, color, shadow, corner, Vertical/horizontal edges, texture, and vehicle light for vehicle detection [15]. Kyo et al. [16] used edges to detect possible vehicles and further validated the vehicles by the characteristics of symmetry, shadow, and differences in the gray-level average intensity, and Denasi and Quaglia [17] used pattern matching to detect and validate vehicles. These methods would usually fail if the obstacles did not match the defined models. For the general obstacle detection task, the optical flow-based and stereo-based methods have been most popular in recent researches. The optical flow based methods would detect obstacles by analyzing the differences between the expected and real velocity fields. Krueger et al. [18] combined the optical flow with odometry data to detect obstacles, but the optical flow-based methods would have the higher computational complexity and might fail if the relative velocity between obstacles and the detector was too small. For the stereo-based methods, Forster and Tozzi [19] utilized disparities of obstacles to detect obstacles and used a Kalman filter to track obstacles. However, stereo methods are highly dependent on the accuracy of identification of correspondences in the two images. In other words, searching the pairs of homogeneous points was much tougher for stereo-based methods. In recent years, there were two important subjects, including improving the accuracy of compensation estimation and obstacle detection. After an IPM image was acquired, a serious problem on resolution between the original and remapped images might be caused. Therefore, how to get an appropriate compensation result would be difficult, especially in our fish-eye lens approach. In Yang et al. [20], the compensation estimation was gained by the recursive method in trials and errors. Firstly, he chose randomly two pixels with a predefined distance to compare the optical flow values until gaining twenty pairs, and then used the median pair to be the value of compensation estimation. However, the
IPM remapped images may cause a serious problem for computing the optical flow values in case of the worse resolution. Furthermore, even if the recursion method was used to avoid choosing non-planar pixels, it was still probably to get similar or non-planar points when the values of optical flows were very close. In our approach, we adopted the edge features and images with time difference to improve the above problems in both static and dynamic environments. For dynamic environments, since the non-planar edge features may change more vibrantly than planar edge features, the values of compensation estimation can be easily determined by the compensated image with the minimum number of candidate pixels of obstacles. To improve stability and robustness of our system, we considered both the time interval and the earlier k frames to average and update the latest compensation estimation. For obstacle detection, in Ma et al. [21] approach, he adopted the pedestrian features and symmetrical property to search the possible positions of obstacles in the region of interest. Although the performance of their system was acceptable, the results would be not stable and robust with the detection rate in 58% 92%. That was because the pedestrians’ feet steps might be influenced by lane markings, shadows of trees, and any other planar noises.
2.4 Object Tracking
Besides foreground segmentation, objects tracking is another key module of surveillance systems. The purpose of tracking module is to track moving objects from one frame to another in an image sequences. And, a tracking algorithm needs to match the observed objects to the corresponding objects detected previously. Useful mathematical tools for objects tracking include the Kalman filter, the condensation algorithm, the dynamic Bayesian network, the geodesic method, etc. Hu et al. [22] presented there are four major categories of tracking algorithms: region-based tracking algorithms, active-contour-based tracking algorithms, feature-based tracking algorithms, and model-based tracking algorithms. Firstly, region-based tracking algorithms [23] were dependent on the variation of the image regions corresponding to the moving objects. The motion regions were usually detected by subtracting the background from the current image. Secondly, active contour-based tracking algorithms represented the outline of moving objects as contours. These algorithms had been successfully applied to vehicle tracking [24]. Thirdly, feature-based tracking algorithms performed the recognition and tracking of objects by extracting elements, clustering them into higher level features, and then matching the features between images. The global features used in feature-based algorithms include centroids, perimeters, areas, some orders of quadratures, and colors [25], etc. Fourthly, model-based tracking algorithms localized and recognized vehicles by matching a projected model to the image data. Tan et al. [26] proposed a generalized Hough transformation algorithm based on single characteristic line segment matching an estimated vehicle pose.
Besides, much research presented tracking algorithms with different categories integrated together for better tracking performance. McKenna et al. [27] proposed a tracking algorithm at three levels of abstraction: regions, people, and groups in indoor and outdoor environments. Each region has a bounding box and regions can merge and split. A human is composed of one or more regions under the condition of geometric constraints, and a human group consists of one or more people grouped together. Cucchiara et al. [28] presented a multilevel tracking scheme for monitoring traffic. The low-level consists of image processing while the high-level tracking is implemented as knowledge-based forward chaining production system. Veeraraghavan et al. [29] used a multilevel tracking approach with Kalman filter for tracking vehicles and pedestrians at intersections. The approach combined low-level image-based blob tracking with high-level Kalman filtering for position and shape estimation. An intermediate occlusion-reasoning module served the purpose of detecting occlusions and filtering relevant measurements. Chen et al. [30] proposed a learning-based automatic framework to support the
multimedia data indexing and querying of spatio-temporal relationships of vehicle objects. The relationships were captured via unsupervised image/video segmentation method and object tracking algorithm, and modeled using a multimedia augmented transition network (MATN) model and multimedia input strings. Useful information was indexed and stored into a multimedia database for further information retrieval and query. Kumar et al. [31] presented a tracking algorithm combined Kalman filter-based motion and shape tracking with a pattern matching algorithm. Zhou et al. [32] presented an approach that incorporates appearance adaptive models in a particle filter to realize robust visual tracking and recognition algorithms. Nguyen et al. [33] proposed a method for object tracking in image sequences using template matching. To update the template, appearance features are smoothed temporally by robust Kalman filters, one to each pixel.
In regard to the cameras of surveillance systems, there are fixed cameras, active cameras and multiple cameras used for capturing the surveillance video. Kang et al. [34] presented an approach for continuous tracking of moving objects observed by multiple, heterogeneous cameras and the approach processed video streams from stationary and Pan-Tilt-Zoom cameras. Besides, much research used fixed cameras for the convenience of system construction and combining with the traditional surveillance system.
In this dissertation, we combined region-based and feature-based tracking methods and used plentiful features as effective inputs of tracking analysis. This proposed algorithm can do a good job to handle multi-objects with occlusion events or split events.
3. Structure of Safety Assistant Driving
Telematics System
In this chapter, we will present our system structure and the details of hardware platform in each part. The system structure is composed of two sub-systems: On Board Vision-Based Detection System and Intersection Intelligent Surveillance System. In section 3.1, we use a diagram of the global system to show two sub-systems and their functions. In section 3.2, we present the structure on the vehicle with vision-based sensors and how they work. In section 3.3, we present the RSU structure with communication module in intelligent surveillance system with embedded platform.
3.1 System Overview
Figure 5: Structure of Safety Driving Assistant Telematics System
In Figure 5, there is an entire structure of our proposed system. The upper part is set on the intersection. Digital Video Recorder (DVR) responds to capture video from each camera on the different location. Intersection Intelligent Surveillance System analyzes and recognizes the content of video stream in four intelligent functions, moving object recognition, moving object tracking, collision prediction and coordinates calibration. Content Management System (CMS) also connects to DVR to receive video stream from DVR and configure parameters on Intelligent Surveillance System. In the end, Road Side Unit (RSU) gathers information
together and encodes message, follows SAE J2735 protocol, to broadcast the intersection condition.
The lower part of Figure 5 is established on a vehicle. The on Board Vision-Based Detection System contains three intelligent functions, side collision warning (SCW), parking assistant system (PAS) and Obstacle Detection. SCW and PAS both are already developed on an embedded platform. Each of them captures the composite NTSC/PAL analog video signal from cameras, and exports the detection results via serial communication or CAN Bus to On Board Unit (OBU). Besides, the embedded platform also exports video stream with detection result on the screen to caution drivers. Obstacle Detection is a PC-Based Program, and it processes the video from reserve, right and left side of vehicle. Upper three directions’ outcomes will be rearranged their priorities with the detection results of intersection condition. After all, driver can obtain the entire driving related information and makes a decision when an incident is occurring.
Figure 6: Structure of Telematics service platforms Source: Institute for Information Industry
Figure 6 is the structure of OBU and RSU, they are two important facilities. The objective is to design a transmitting protocol for broadcasting local intersection condition to drivers. By WAVE/DSRC protocol, we can realize V2R and V2V applications via IEEE 1609.3 and IEEE 1609.4. Besides, we also follow DSRC SAE J2735 standard to deal with related safety information, and package them by ASN.1 encoding rules.
Figure 7: Current application on telematics system
Although ITS is a popular research, high accuracy traffic application detectors are still rare. The first useful traffic related information for driver is GPS. Because of the property of GPS signal, it is more accurate in country than in city. Besides, there are several kinds of useful information on the road that were already defined. In recent years, vision-based traffic detectors such as vehicle detector (VD), Image incident detection (IID) and self-adaptive traffic sign are developed for ITS applications. In our proposed safety driving assistant telematics system, we add vision-based intelligent detectors to provide safety detector results from the sensors on vehicle and intersection.
3.2 On Board Vision-Based Detection System
Figure 8: Vehicle Sensors
From Figure 8, it shows the sensors location and the assistant driving signal flow. There are three vision-based detectors, including side collision warning (SCW), parking assistant system (PAS), Obstacle detection. SCW and PAS are already integrated on embedded platform with CAN Bus protocol, and obstacle detector can work on laptop 10 fps. There is a WAVE Box on the vehicle to receive the information of intersection condition. WAVE Box communicating with notebook via RS232 protocol.
3.2.1
Embedded Platform Description
DSP BF561 ADV 7180 Video Deccoder ADV 7179 Video Encoder I2C Parallel Peripheral Interface(PPI) Parallel Peripheral Interface(PPI)
Power Regulator Module (PWM + Linear Power IC)
3.3V For DSP I/O 1.2V For DSP Core Power 1.8V RS232 (TX,RX, GND) UART GPIO TTL Logic IC SDRAM 炵 FLASH 32-Bits Data Bus
Figure 10: Structure of Embedded Platform PCB Design
In Figure 10, there is an embedded platform for SCW, PAS and intersection vision-based instant detector applications. It contains an ADI Blackfin BF561 Duo-Core DSP, Power Regulator, Video Decoder, Video Encoder, Memory, and I/O Modules. Detail features of Blackfin 561 are shown in Table 3.
Table 3: Features of BlackFin 561 DSP
Features Peripherals Dual symmetric 600MHz high performance
Blackfin cores 328K bytes of on-chip memory
Two parallel input/output peripheral interface units supporting ITU-656 video gluesless interface to analog front end ADCs Each Blackfin core includes:
Two 16-bit MACs, two 40-bit ALUs, four 8-bit video ALUs, 40-bits shiffer
Two dual channel, full duplex synchronous serial ports supporting
eight stereo I2S channels RISC-like register and instruction model for
ease of programming and compiler-friendly support
Dual 16-channel DMA controllers and one internal memory DMA controller
Advanced debug, trace, and performance monitoring
0.8V-1.2V core VDD
3.3V and 2.5V tolerant I/O
12 general-purpose 32-bit timer/counters, with PWM capability
256-ball mini-BGA and 297-ball PBGA package options
SPI-compatible port
UART with support for IrDA® Dual watchdog timers
48 programmable flags
On-chip phase-locked loop capable of 1ǘ to 63 ǘ frequency multiplication
Source: ADI Blackfin 561 Datasheet
In this embedded platform, we consider the reference design from ADI BF561 EZ-KIT Lite manual. Due to our application, we change decoder chip into ADV 7180 for simple video
stream decode steps, and redesign the power regulator module for 12V input. DSP controls video decoder and encoder by I2C protocol, and gets digital video data via 8 bits parallel peripheral Interface. In DSP inside, we can program the DMA register to control the DSP PPI catching data by itself automatically without wasting too much computing power on this kind of I/O data flow.
Power regulator module in this design, we choose the 2-channel PWM power regulator chip LM2642. At first, when 7V-24V (regular is 12V) is sent in the board, we string a schottky rectifiers and radial leaded PTC fuse to protect circuit. Table 4 is the SK34B’s electrical characteristics, it limits the basic current forward and flow rate. FRX-090-60 is a fuse, its function is to avoid short connect occurrence. Besides protect circuit, we also consider the stability. We use DLW5BSN351SQ2 and two inductances to keep input voltage before entering LM2642 more smoothly. This embedded system total needs four kinds of voltages, including 5V, 3.3V, 1.8V and 1.2V. LM2642 outputs DSP needed two kinds of main power 3.3V and 1.2V. 3.3V supports DSP’I/O, RS232, RS485, TTL logic IC and 1.8V regulator’s input voltage. 1.2V only supports DSP core power. 5V and 1.8V are both regulated by linear power regulator IC LM7805 and RT9166-18PX.
Table 4: SK34B Electrical Characteristics Electrical Characteristics
Average forward current IF(AV) 3.0A TJ=120ɮC
Maximum surge current I FSM 100A 8.3ms half-sine
Max repetitive reverse current IR 2A F=1KHz
Max peak forward voltage (SK32B-SK34B)
VFM .50V IF=3A, TJ=25ɮC
Max peak reverse current IRM .5mA VRRM, TJ=25ɮC
Typical junction capacitance CJ 250pF VR=5.0V, TJ=25ɮC
As a result of our applications, end-user needs to see the exporting scream with intelligent detection result. For exporting a stable video output, we add a video OP after ADV 7179. Furthermore, we design a low-pass filter to reduce the noise influence.
3.2.2
Sinffer Equipment
Figure 11 : S100 WAVE Box
For verifying the contents fits IEEE protocol, we choose Savari’s S100 WAVE Box and Sirit’s WSM Sniffer Card. The detail spec is shown in Figure 12.
Figure 13: Sirit Sniffer Card (PCMCIA Interface) Table 5: SAE J2735 WSM Message Protocol
1 1 1 1 1 4 2 Variable WSM Version Security Type Channel Number Data Rate TxPwr_level Provider Service Identifier WSM Length WSM DAta
Figure 14: OBU Service and Navigation Structure
Figure 14 is the structure of OBU for Service and Navigation. RSU and OBU have a communication bridge by DSRC protocol via their WAVE device. For fitting protocol, when it transmits message, Safety Message handler will provide up layer related message for choosing encoding rule. So message can be distributed different decoder, for example BER, DER and XER. For the same reason, when receiving a message either from OBU or RSU in communication layer, first step needs to do is to judgment what kind of coding rule, and then use the suitable decoder to gain the message.
As mentioned previously, in our propose system, WAVE device supports DSRC ability for each safety applications. The working method is to develop a WSMP-IP converter on the WAVE device. When Safety application needs to send the message, it only sends UDP
shown in Figure 15.
Figure 15: The flowchart of transmitting message
If OBU works successfully, it will receive traffic condition from many RSU on different intersections and detection result of the vehicle sensors. As a result of too much information, driver should get a sorted event list and suggestion
3.3 Intelligent Surveillance System
In this section, we propose a surveillance system with intelligent functions on an embedded platform. In traditional surveillance system, each Video Stream Input peer needs to connect to CMS for updating the current video stream as shown in Figure 16.
Figure 16: Traditional intelligent surveillance system
3.3.1 Improved Intelligent Surveillance System
In traditional structure, it needs a lot of manpower to pay attention on monitor, although the false alarm rate is less. Besides, this kind of design lacks adaptability for detectors to connect. Since, we propose an entire system with vision-based incident process platform and CMS. The vision-based incident process platform contains image incident detector and image incident collector. The former analyzes the image and then classifies incidents. It marks a sign on the video stream to distinguish the incident types. The image incident collector contains two parts. One is image storage server, and the other one is incident information storage server. The incident information storage server becomes a communication bridge between image storage server and incident detector, and also produces a trigger signal when some incident occurs.
In Figure 17, red region means our proposed intelligent surveillance system. The blue region is vision-based incident process platform with incident detector and collector. Hence, we design RSU as a vision-based incident detector. After analyzing and classifying incident in video stream, it integrates all the information and uploads the results with correct content protocol to telematics transmitter via Ethernet. And then the WAVE Box will broadcast information in WAVED\DSRC channel to remind drivers to take care intersection condition.
3.3.2 Vision-based incident detector
In this section, we will discuss the basic functions on the vision-based incident detector. Due to the intersection traffic applications, we developed collision prediction and object tracking two algorithms. Since we utilize the fix camera, adaptive GMM background update method is suitable to extract foreground objects. Although GMM method needs heavy computing power, we modified the update period by the video frame rate per line. It means the cycle of update rate is about 16 seconds when the GMM background update method is running in single line update mode. Intersection traffic also have a serious problem of such kind of background method is the static object. When a vehicle stops before stop line to wait the traffic light, traditional GMM method make the wrong background image easily. Hence, our adaptive GMM Background algorithm have changeable update rate to adjust the parameters.
On our embedded platform, we capture composite NTSC/PAL video stream signal, and then ADV7180 decodes the analog signal to YUV 4:2:2 digital data into DSP Blackfin 561. In SDRAM, we allocate four image buffers for different processes. First frame is input frame, and it stores the data via DMA. Second frame is overlay frame, it overlays the lines or patterns on the image. Third frame is output frame, and it is prepared to export to ADV7179. The last frame is preprocessing frame, and it copies the data from input frame for image processing.
For our applications, tracking algorithm is the most important process. We extract the foreground and make a connected component image Ic. We accumulates Ic in each iteration to
make accumulation image Ia. If pixel in Ic is foreground pixel, the value of Ia will add 1 until
to threshold Tha, otherwise Ia will decrease 1 until 0. Tha is equal to 50 in our applications,
and the value also can be estimated by the height of camera. The tracking index is connected component’s label. From different time images, we first refer the objects’ locations and ranges. If the location is close enough and range has overlap, it marks as a candidate object. The second step is to refer Ia image, from this image, we have the pixel’s variance in past 50
frames. Therefore, we can filter some error foreground pixels and stable the tracking sequence. Finally, we package the whole information, and transfer the package via UART to PC client. At the same time, we also display the intersection incident and objects’ location and trail on the scream.
4. Vision-Based Intelligent Technique
In this chapter we present four vision-based algorithms. In section 4.1 we will present a whole FLIPM method. It considers an optical path with mathematical calculating. In section 4.2 we will promote obstacle detection with single camera. Using the mapping image captured from the result of section 4.1, we analyze the characteristic to attend our objective. Finally, in section 4.3 we realize a vision-based dynamic distance gauge system for parking assistant system.
4.1 Fisheye lens inverse perspective mapping (FLIPM)
Figure 18: Fisheye lens inverse perspective mapping structure
Our overall systematic structure is illustrated in Figure 18. The obstacle detection is performed after obtaining the bird-view images of road surfaces captured by the camera mounted on the lateral side of the vehicle. The edge profile of road surfaces in bird-view images or temporal FLIPM difference image should be acquired, and then the segment searching algorithm will use the edge profile to get the feature radial lines which indicate the obstacles. After searching the feature elements, the polar histogram which represents the direction and size of obstacles will be computed. The histogram post-processing will also be used to filter out
some noises and obstacles with shorter height. We still have to identify the detected obstacles and extract the relative information of the obstacle after the obstacle tracing process. After all the processes, we can obtain the final results in the output videos.
4.1.1 The Modified Normal Lens IPM Method
To find more practical applications and set up the appropriate mapping equations in our system, we modify the previous approaches proposed by Broggi et al. [2] and make the obstacle detection system more complete. Let u and v represent the image coordinate system and X,Y,Z be the world coordinate system where (X,Y, 0) indicates the road surface. L, D, H are the coordinates of the camera in the world coordinate system whileθ and γ are the camera’s tilt and pan angles, respectively.
α
,β
are the horizontal and vertical aperture angles. m and n indicate the height and width of an image. O is the optic axis vector, and ηx,ηy are the vectorsrepresenting the optic axis vector O projected on the road surface and its perpendicular vector.
D ) 1 -n 2 v -sin( * ) 1 -m 2 u -cot( * H Y L ) 1 -n 2 v -cos( * ) 1 -m 2 u -cot( * H X + + + = + + + = α α γ β β θ α α γ β β θ (4.1)
From Eq. (4.1), the vertical straight line in the image coordinate system can be represented by the set of pixels whose v coordinate value is constant. If we assume that
1 -n 2 v -Kv=
γ
α
+α
is constant, then Eq. (4.1) will be simplified to Eq. (4.2).
D sin(Kv) * ) 1 -m 2 u -cot( * H Y L cos(Kv) * ) 1 -m 2 u -cot( * H X + + = + + = β β θ β β θ (4.2)
After simple calculations, we can obtain Eq. (4.3) from Eq. (4.2), which is shown in Figure 19.
X-L=(Y-D)*cot(Kv) (4.3)
The equation Eq. (4.3) means that a vertical straight line in the image which represents the vertical edge of obstacles or other planar markings in the world coordinate system will be projected into a straight line whose prolongation will pass the vertical projection point of the camera on the world surface.
Figure 19 : The vertical line projection of Eq. (4.1)
Similarly, the horizontal straight line in the image coordinate system can be represented by the set of pixels whose u coordinate value is a constant. If we assume
1 -m 2 u -Ku=
θ
β
+β
is constant, then Eq. (4.1) will be also simplified to Eq. (4.4).D ) 1 -n 2 v -sin( * K D ) 1 -n 2 v -sin( * cot(Ku) * H Y L ) 1 -n 2 v -cos( * K L ) 1 -n 2 v -cos( * cot(Ku) * H X + + = + + = + + = + + = α α γ α α γ α α γ α α γ (4.4)
Thus, we can derive Eq. (4.5) from Eq. (4.4), which is shown in Figure 20. 2 2 2 K D) -(Y L) -(X + = (4.5) The equation Eq. (4.5) means that a horizontal straight line in the image will be projected to an arc on the world surface.
In order to modify the original IPM model, we propose a new pair of transformation equations for two expected results. First, a vertical straight line in the image will still be projected to a straight line whose prolongation will pass the vertical projection point of the camera on the world surface. Second, a horizontal straight line in the image will be projected to a straight line instead of an arc on the world surface. The results can be verified by the similar triangle theorem. With some prior knowledge such as the assumptions on flat roads, intrinsic and extrinsic parameters, we will be able to reconstruct a 2D image without the perspective effect. The illustrated figures and expected results are shown in Figure 21
(a)
(b) (c) Figure 21: The figures and expected results
(a) perspective effect removing (b) a vertical straight line in the image will be projected to a straight line whose prolongation will pass the vertical projection point of the camera on the
By referring to the notations, the diagrams of relationship between the image coordinate system and the world coordinate system are shown in Figure 22. We will derive a new pair of transformation equations by simple mathematical computations in triangular functions. From Figure 22(a) and (b), we can obtain (6) and (7).
β
β
θ
-1 -m 2 u 1 = → ) cot( * H H0 = θ → ) cot( * H H H0 + 1= θ+θ1 → (4.6) x η (a) (b) x η y η (c) (d)Figure 22: The geometrical relations of the image and world coordinate system for deriving our equations. α α θ -1 -n 2 v 2 = → ) sec( * ) tan( ) tan(θ2' = θ2 θ+θ1 → (4.7)
Figure 22(c) describes how the points in the first quadrant of the image coordinate system will be projected onto the road surface. If the world coordinate of camera is (0, 0, H), we will finally obtain Eq. (4.8) by the geometrical descriptions in Figure 22(c) (d) and the length of each segment listed below.
) cot( * H H0 =
θ
→ ) cot( * H H H0+ 1=θ
+θ
1 → ) sec( * H H2 = oγ
→ ) sec( * ) H (H H H2+ 3 = 0+ 1γ
→ ) sec( * ) tan( * ) H (H W W0 + 1= 0+ 1θ
2θ
+θ
1 → ) tan( * H W2 = 0γ
→ ) sec( * ) tan( * H W W2+ 3 = 0θ
2θ
+θ
1 →[
cos( ) s ( )*tan( )*sin( )]
* ) cot( * H ) sin( * W H H X= 2+ 3+ 1γ
=θ
+θ
1γ
+θ
+θ
1θ
2γ
⇒ ec[
-sin( ) s ( )*tan( )*cos( )]
) cot( * H ) cos( * W Y= 1 γ = θ+θ1 γ + θ+θ1 θ2 γ ⇒ ec (4.8)Now, we have obtained the forward transformation equations, and the backward transformation equations shown in Eq. (4.9) can also be obtained easily by some mathematical computations in inverse functions.
θ
γ
γ
θ
) -H ) sin( * Y -) cos( * X ( cot-1 1 = ⇒ ; ) ) csc( * H ) cos( * Y ) sin( * X ( tan 1 1 -2 θ θ γ γ θ + + = ) -) H ) sin( * Y -) cos( * X ( (cot * 2 1 -m u -1γ
γ
θ
β
β
+ = ⇒ ) ) ) csc( * H ) cos( * Y ) sin( * X ( (tan * 2 1 -n v 1 1 - α θ θ γ γ α + + + = ⇒ (4.9)4.1.2 Fisheye Lens Inverse Perspective Mapping (FLIPM)
Forster et al. [19] proposed a camera spherical projection model to implement the endoscope image formation process and utilized the warping transformation equations to correct the radial distortion. The warping transformation equation pairs and its inverse pairs are shown in Eq. (4.10) and Eq. (4.11). The coordinate (X,Y,Z) is the position of point in the 3D world coordinate system, (u1,v1) is the coordinate in the un-distorted image, and (u,v) is the
) tan( * ) H (H W0 = 0 + 1
γ
→2 2 2 2 2 2 Y X f Y * R v ; Y X f X * R u + + = + + = (4.11)
Where f is the focal length of camera, and R is the radius of the sphere. We modify and redefine that model for our applications in this dissertation. We regard the X1-Y1 plane as an undistorted image plane and the u-v plane as the distorted one, thus we can derive the modified equations in Eq. (4.12) and Eq. (4.13).
( )
( )
( )
( )
cos sin * * v -u -R v * f v cos sin * * v -u -R u * f u 2 2 2 2 2 2 1 2 2 2 1 1 2 1 2 2 2 1 2 2 2 1 ⎪ ⎪ ⎩ ⎪ ⎪ ⎨ ⎧ + = = + = = θ θ θ θ u v k v u k (4.12) )) ( tan ) ( (tan 1 k v v u f v * R v )) ( tan ) ( (tan 1 k u v u f u * R u 2 2 1 2 1 1 2 1 2 1 2 1 2 2 1 2 1 1 2 1 2 1 2 1 ⎪ ⎪ ⎩ ⎪ ⎪ ⎨ ⎧ + + = + + = + + = + + =θ
θ
θ
θ
(4.13) Where R f k1 = , and ) f u ( tan ) R ( sin -1 1 2 2 1 -1 = − = v u θ and ) f v ( tan ) R ( sin -1 1 2 2 1 -2 = − = u v θare the angles between the lines connected the horizontal or vertical direction projection point of an image point with the optical center on the optical axis. The equation Eq. (4.14) instead of Eq. (4.13) will be used through this dissertation since Eq. (4.13) may produce many non-pixel-values of the image. We also can obtain the distorted or un-distorted images no matter if the focal length is known or not by tuning the parameter k1.
1. The Complete Fisheye Lens Inverse Perspective Mapping
A fisheye lens inverse perspective mapping (FLIPM) algorithm consists of two parts, the forward and backward mapping algorithm. The objective of the forward mapping algorithm is to search the dimensions or ranges of remapped images, which can be illustrated in Figure 23.
1 θ β β -1 -m 2 u
The dimensions of scopes are only related to the view-ranges of a camera, that is to say, either the use of the normal lens or fisheye lens with fixed tile and pan angle will determine the factors of influences. In order to reduce the computational loadings in use of the tangent and secant triangular functions, we restrict the scope of a camera by narrowing down its view-range. Without loss of generality, we still keep the broadest range of scopes and minimize discarding far and fringe information. Furthermore, we narrow down the view-angles by using Snell’s Law as shown in Eq. (4.14) where IR simulates the index of refraction and controls the scopes of resultant ranges. The range of IR is between 1.3~1.7 for glass-based lens.
) sin( * IR ) -1 -n 2 sin(v ) sin( * IR ) -1 -m 2 sin(u 2 1 θ α α θ β β = = (4.14)
The angles θ1andθ2 can be substituted into Eq. (4.9) to compute the extreme values about the coordinate values of X, Y, and in this way we will obtain the dimension of the remapped image. The backward mapping algorithm is different from the forward one because a plus of the radial distortion correction step should make it more rational. We firstly consider the ideas of the backward mapping algorithm by Figure 24.
Figure 24: Illustrations for distortion images
(a) the real scene image, (b) the distorted image, and (c) the desired image
Since the images captured by the fisheye cameras which can be shown in Figure 24(a) have the perspective effects and distortions, we have to remove those undesired effects to acquire the available images just like Figure 24(b) in pursuit of Figure 24(c) where the perspective effect and distortion have been completely removed. Thus so, we can derive the backward mapping algorithm by modifying Eq. (4.9) as Eq. (4.15). We also complete the
perspective effect removed images. -) H ) sin( * Y -) cos( * X ( cot-1 1 θ γ γ θ = ) ) csc( * H ) cos( * Y ) sin( * X ( tan 1 1 -2 θ θ γ γ θ = + + (4.15)
4.2 Obstacle Detection with single Camera
In this section we develop an obstacle detection algorithm by using both spatial and temporal information of the FLIPM method. We use a single fisheye camera mounted on the lateral side of the vehicle to detect obstacles. The definitions of obstacles in this dissertation are the objects with the height shorter than a threshold and with non quasi-vertical edges. The straight line in the vertical direction in the images represents the vertical edges of obstacles in the world coordinate system, and will be projected to a straight line whose prolongation will pass the vertical projection point of the camera on the world surface. To illustrate our systematic mechanism more clearly, we will introduce the obstacle detection algorithm in the following parts, including some image pre-processing steps, feature selection, histogram analysis, object tracking, and information extraction.
4.2.1 The Pre-Process
Figure 25: The flowchart of image pre-processing
We have to simplify the image patterns for our following procedures by some image preprocessing techniques shown in Figure 25. At first, the remapped image will be smoothed by mean filter to reduce the noises resulted from FLIPM transformation. Our developed equations in FLIPM have the advantages of IPM in removing the information of height and can help to detect the obstacles on the surface of roads. We also propose two different strategies toward feature extraction. We use the profile image which will be introduced next to extract the feature series when the detected objects and our cameras are relatively motionless, otherwise we acquire the features by the obstacle-sensitive temporal FLIPM difference image which will be clarified in Section 4.3.
and detect edges by simple Sobel operations. The binary images can be obtained by thresholding after edge detecting of the remapped image, and we have to use the morphological operations on dilation and erosion to get the useful edges for our processes. As for extraction of the feature segments, we remodel the thinning algorithm introduced in [20] in thinning the binary edges in order to meet our real-time needs in the applications of ITS. We turn to use the center pixel of a mask to extract the exterior profile of a pattern without checking the conditions
of patterns iteratively. Figure 26 shows the processed results of our profile image searching.
Figure 26: The results in the profile searching process (a) the remapped image (b) the profile image.
4.2.3 The temporal FLIPM difference image
The objective of temporal FLIPM difference process is to simulate the stereo vision of captured images. The stereo IPM can keep the non-plane objects and remove the plane objects such as lane-markings, shadows by comparing the differences between the left and right remapped image, which will be illustrated in Figure 27.
(a) (b)
(c)
Figure 27: Illustrations for the temporal FLIPM difference image
(a) the planar object patterns and (b) non-planar object patterns (c) Moving non-planar object patterns
In Figure 27(a), it presents a result of planar object patterns. The planar pattern in IPM image has a shift movement in different time; hence we can easily remove the planar pattern via accurate enough shift movement to compensate the different images. Figure 27(b) shows a result of a static non-planar object on the IPM image. By the projection effect, the non-planar point also can be projected to the ground with the farer distance. Therefore the difference of
