Chapter 5 Vehicle Detection and Tracking for Traffic Parameter Estimation
6.3 Vehicle Tracking and Traffic Parameter Estimation with Shadow Suppression 96
Figure 6-11 shows the system architecture of the proposed traffic parameter estimation system with shadow suppression. The system consists of five parts: the foreground segmentation module, shadow suppression module, vehicle detection module, vehicle tracking module, and traffic parameter estimation module. The foreground segmentation module constructs the background image for foreground segmentation from image sequences, as described in Chapter 3. The shadow suppression module separates the cast shadow from the moving regions for improving the quality of foreground segmentation, as presented in Chapter 4. The vehicle detection module detects the moving vehicles for vehicle counting and tracking initialization, as described in Chapter 5. The vehicle tracking module tracks the moving vehicle for speed estimation. Finally the traffic parameter estimation obtains useful traffic parameters by analyzing detection and tracking result, as shown in Section 5.3.3.
The frame rate adopted in the experiment is 15 frames per second. The pixel resolution of each test frame is 352x240 pixels. Figure 6-12 illustrates an example of image tracking of cars and trucks on an intercity expressway. In the scene, the shadow strength is medium, and the shadow size is large. The ITMS will fail to detect and track the moving vehicles if three is no shadow suppression tool in use. Figs. 6-12(b)-(l) shows that the proposed method successfully separates the shadow from the moving region. In Fig. 6-12(b), two cars are
97
Background Estimation Moving Vehicle
Segmentation
Shadow Suppression
Vehicle Detection
Camera Calibration
Vehicle Tracking
Traffic Parameter Estimation Traffic Image Sequence
Fig. 6-11. Block diagram of the traffic parameter estimation system.
detected in the detection window, and one car is tracked. One car leaves the detection window and an active contour is initiated for tracking in Fig. 6-12(d). Three cars are tracked in Fig.
6-11(f). One car is tracked, and a truck is detected in the detection window, as shown in Fig.
6-12(h). The truck leaves the detection window and an initial contour is generated for tracking in Fig. 6-12(j). In Fig. 6-12(l), the truck is tracked and a car is detected in the detection window. The experimental results demonstrate that the traffic monitoring system successfully tracks the moving vehicles with shadow suppression. Partial results can be found at:
http://isci.cn.nctu.edu.tw/video/JCTai/shadow_track.wmv.
In this example, useful traffic parameters are estimated. In a time duration of 23 seconds, a total of 16 vehicles were detected (the ground truth is 18 vehicles). The accuracy of vehicle number estimation and tracking are satisfactory, thanks to the shadow suppression and the detection window for handling vehicles that travel across lane boundaries. Table 6-2 shows the experimental results of the estimation of vehicle speed. In the table, the ground truth,
98
(a) (b)
(c) (d)
(e) (f)
(g) (h)
(i) (j)
(k) (l)
Fig. 6-12. Experimental results of vehicle tracking with shadow suppression in an expressway.
(Detected shadows are indicated in white area and detected vehicles are indicated in darker area.)
99
Table 6-2 Experimental Results of Vehicle Speed Estimation
Vehicle Index
Estimated speed (km/hr)
Ground truth (km/hr)
Error %
1 84.1 80.6 4.3%
2 86.0 84.9 1.4%
3 86.4 85.7 0.8%
4 84.9 86.8 -2.2%
5 86.8 86.4 0.4%
6 101.3 103.5 -2.2%
7 82.9 87.6 -5.3%
8 100.6 95.7 5.1%
9 73.1 73.1 0.0%
10 75.0 76.8 -2.3%
11 77.0 77.3 -0.4%
12 79.3 77.6 2.2%
13 84.1 82.9 1.4%
14 54.9 54.0 1.7%
15 66.0 66.6 -0.9%
16 64.8 64.5 0.5%
Average 80.4 80.2 0.3%
which was manually measured from image sequences, is also presented for comparison.
Because the initial contour generated by the proposed method is well-suited for the active-contour-based image tracking with shadow suppression; the error of average speed estimation is within 5%. The traffic parameters are estimated as follow: the average speed is 80.4 km/hr (the ground truth is 80.2 km/hr), the traffic flow rate is 2504.3 car/hr and the density is 31.3 car/km.
100
6.4 Turn Ratio Estimation with Shadow Suppression
Figure 6-13 shows the system architecture of the proposed turn ratio estimation system.
The system consists of five parts: the foreground segmentation module, shadow suppression module, vehicle detection module, optical flow detection module, and vehicle turn ratio module. The first three preceding modules are the same as described in Fig.6-12. The direction of movement detected by the optical flow detection module is used to determine whether the detection window detects the moving vehicle, as presented in Section 5.4. The turn ratio module computes the turn ratio by using the vehicle count of each vehicle detection module, as described in Section 5.4.
In this experiment, the left turn ratio of oncoming vehicles is estimated at a T-shape intersection as shown in Fig. 6-14. Figures 6-14(a)-(c) depict a test result of shadow suppression and the motion vector estimation. The image sequences are captured during
Fig. 6-13. Block diagram of the vehicle turn ratio estimation system.
101 (a)
(b)
(c)
Fig. 6-14. Experimental results of turn ratio estimation with shadow suppression at an intersection.
Three types of driving direction of vehicles are distinguished: (a) Moving right. (b) Moving left. (c) Moving straight ahead.
sunset. In the scene, the shadow strength is large, and the shadow size is medium. The actual shadow has been successfully separated from the moving vehicle by the proposed method. In the figures, the small arrows indicate the detected motion direction of interesting points of the vehicles. An average motion vector is obtained by considering the motion vectors of all interesting points in the specific region. The direction of the big arrow shows the average motion vector of vehicles in the specific region. Figure 6-14(a) shows two oncoming vehicles
102
passing the specific region. One vehicle moves left and passes through the detection window in Fig. 6-14(b). Figure 6-14(c) shows an ongoing vehicle passes through the specific region.
The detected motion accurately reflects the motion type of vehicles in this region and the shadow has been suppressed in traffic monitoring. According to the information of the estimated direction, the system can identify where the vehicles come from and determine the turn ratio accordingly. In this test, the actual turn ratio is 7% and the calculated value is 7.2%.
The averaged error is about 3.7%. Partial results can be found at:
http://isci.cn.nctu.edu.tw/video/JCTai/shadow_ratio.wmv.
6.5 Summary
This chapter evaluates the proposed GBH, traffic parameter estimation, and shadow detection approaches through experimental results. Five experiments have been carried out to validate the performance of the proposed scheme. In the first three experiments, the vehicles are detected and tracked for traffic parameter estimation under a rare shadow condition. In the rest experiments, the proposed method of shadow suppression successfully removes the overlapping shadows attached to moving vehicles for traffic image analysis. The moving vehicles are successfully segmented using our proposed method, showing that our tracking algorithm and turn ratio estimation work satisfactorily under shadow condition. Traffic parameters are obtained with high accuracy.
103
Chapter 7
Conclusion and Future Work
7.1 Dissertation Summary
To measure reliable traffic parameters, a series of designs and implementations of image processing are proposed in this thesis. The extracted traffic parameters are average speed, traffic flow, traffic density, and turn ratio. The image processing methodologies consist of camera calibration, single Gaussian background modeling, vehicle detection and tracking, optical-flow-based turn ratio measurement, and shadow suppression.
We propose a novel algorithm to automatically calibrate a PTZ camera that overlooks a traffic scene. The geometric information of parallel lane markings is used to derive the focal length equation for camera calibration. The computed focal length is further used to obtain the pan and tilt angles of camera. Furthermore, an image processing procedure has been developed to locate the parallel lane markings. The simulation results reveal that the error rates of position estimation are within 2.3% under the presence of reasonable translational and height errors. In actual experiments, twelve feature samples in a road scene were selected for distance measurement. The maximum error of the distance measurement is within 5% and the absolute mean of the error is below 2.39%.
To estimate a single Gaussian model of background pixel in real time, a group-based histogram algorithm has been designed and implemented. The method is highly effective for building the Gaussian background model from traffic image sequences. It is robust to errors
104
caused by sensing noise and slow-moving objects. The computational load of the GBH is 35% less than the popular GMM approaches, and is thus more suitable for the real-time requirement of surveillance applications. The performance of the proposed algorithm has been evaluated and successfully applied to an ITMS.
An automatic contour initialization procedure has been developed for image tracking of multiple vehicles based on an active contour and image measurement approach. First, a detection-window-based method is proposed to detect moving vehicles of various sizes and generate their initial contours for image tracking in a multi-lane road. The proposed method is not constrained by the lane boundary. Kalman filtering techniques are then applied for active contour-based image tracking of various vehicles. The automatic contour initialization and tracking scheme have been tested for traffic monitoring. Experimental results reveal that the proposed method successfully tracks motorcycles as well as cars on an urban multi-lane road.
Traffic parameters are successfully extracted by using the developed method. The error of average vehicle speed estimation is within 5%.
A method is proposed to automatically estimate the turn ratio at an intersection by using techniques of optical flow calculation and detection window. An algorithm for estimating motion vector based on corner correlation has been designed to meet the real-time requirement. The automatic turn ratio measurement has been tested for traffic monitoring in Hsinchu Science Park, Taiwan. Experimental results show that the proposed method successfully detects vehicles and estimate the motion vectors of moving vehicles at an intersection. Turn ratio is measured with a satisfactory accuracy.
To increase the performance of moving vehicle segmentation, we propose a shadow-region-based statistical nonparametric method to construct a ratio model for shadow pixel detection. The generation of this shadow model is more effective than existing methods.
105
Two types of spatial analysis have been further designed to enhance the performance of shadow suppression. Practical experimental results show that the shadow detection method outperforms existing methods both in the shadow suppression rate and the computation time.
The proposed scheme has been successfully applied to an ITMS. Traffic parameter estimation with shadow suppression works satisfactorily. Traffic parameters such as traffic flow, turn ratios, traffic densities, and vehicle speeds have been obtained with acceptable accuracy. The absolute error rate is less than 5.3% for vehicle speed estimation and within 3.7% for the turn ratio estimation.
7.2 Future Directions
Some directions for future study are recommended below:
(1) In the future, practical applications of PTZ cameras will be further studied. For instance, most image-based traffic surveillance methods adopt a virtual window to detect vehicles [71]. If the view of a PTZ camera is changed, then the position and size of the window must be adjusted manually. Using the dynamic calibration procedure developed in this thesis, the detection window can be arranged automatically. On the other hand, the effects of lens distortion and having a non-fixed principal point need to be handled in order to increase the accuracy of PTZ camera calibration.
(2) Several directions on vehicle detection and tracking deserve further study in the future. On one hand, heavy occlusion of vehicles influences the accuracy of image measurement.
Methods need to be developed to distinguish individual vehicles. Color information of individual tracked cars can be very useful for solving this problem [80]. On another hand, in order to increase the accuracy of foreground segmentation, we will focus on the study of new methods to select adaptive thresholds for handling the change of environment
106
illumination on the road. To achieve a dependable performance, a Neuro-Fuzzy classifier can be desirable for an ITMS.
(3) For future shadow-detection studies, more emphasis will be directed to increase the robustness of the shadow detection under various illumination conditions. First, the Gaussian ratio model built under a specific illumination condition might fail under a considerably different illumination. In traffic monitoring applications, it will be beneficial to build a database of ratio models for different illumination conditions. Additionally, shadow pixels that lie nearmoving vehicles or overlap other vehicles might be misclassified as moving-vehicle pixels. The pixels of the same shadow region may have similar color information to the traffic imagery. The color distribution can be used to find uniform sub-regions and hence the uniform sub-regions can be used to verify the actual shadow region [81].
107
Appendix A
Derivation of Focal Length Equation
In this appendix, the focal length equation will be derived by using only two parallel lines in the image. As shown in Fig. 2-1(a), lines L1 and L2 are two parallel lines, L1 and L2
intersect X-axis and Y-axis at P1, P2, P3 and P4; the corresponding coordinates of these points in image plane are expressed as follows:
u
2can be rewritten as
v =
1108
Applying trigonometric function properties, one can easily find the relationship between
r, s and t. Computing r
2 +f
2s
2and r2t2, one can obtainRearranging (A.12), we obtain
φ
109
Substituting (A21) into (A13), we obtain
θ
Using the vanishing point constraints and trigonometric function properties, one can proceed to derive equations that will determine the equation containing sec2θ and, finally,
110
Rearranging (A.26), we arrive at:
2 +bm+c=0
am . (A.27)
where m is f2 and other variables are listed Table 2-1. This governing equation is presented in Section 2.2 as the focal length equation.
Next, let us discuss how camera parameters affect the sign of coefficient a. For simplicity, the sign of at2is discussed instead of the sign of a
)
(A.28) reveals that the magnitude of the tile angle and pan angle affect the sign of coefficient
a ; the details are listed below:
• a>0, if
φ
>θ
,111
• a=0, if
φ
=θ
orY
3 =0,• a<0, if
φ
<θ
.It is clear that the difference between the absolute value of tilt angle and pan angle determines the sign of coefficient a . When a=0, the focal length equation becomes linear and the focal length can be easily estimated. This completes the derivation of the focal length equation.
When the vanishing point is far from the image center or disappears (for instance, when the tilt angle equals 90 deg) in the image frame, (A.27) cannot be used to find the focal length. Instead the focal length can be easily obtained by the perspective projection equation:
w h w
f
= p , (A29)where wp is the width between parallel lanes in the image frame.
112
Appendix B
Conversion between Pixel
Coordinates and World Coordinates
In this appendix, we derive the transformation between pixel coordinates and world coordinates. We will explain how focal length and tilt angle are used to obtain the world coordinates of a feature in the ground plane.
A pixel coordinate (u,v) is expressed as a function of world coordinate (X,Y,Z)
113 v v v v f Y h
= −
0 0
sin2φ . (B.6)
Substituting (B.6) into (B.1), it is easy to obtain
v v
v u f h v
v v u f X h
= −
− +
=
0 0
0 1) sin
sinφ ( φ . (B.7)
From (B.6) and (B.7), one can transform the pixel coordinates into their world coordinates.
114
Appendix C
RGB Color Ratio Model of Shadow Pixels
In an outdoor daytime environment, there are two light sources, namely, a light point source and a diffused extended light source. In the following derivation, the road is assumed as to be Lambertian, with a constant reflectance in a traffic scene. The radiance
L
lit of the light reflected at a given point on a surface in the scene is formulated as [82]) respectively; λ is the wavelength;iis the angle of incidence between the illumination direction and the surface normal at a considered point; e is the reflection angle between the surface normal and the viewing direction; and g is the phase angle illumination direction and the viewing direction. When sunlight occlusion creates shadows, the radiance
L
shadow of the reflected light becomes)
where )
L′
a(λ
is the ambient reflection term in presence of the occluding object. To simplify the analysis, the design assumes that the ambient light coming from the sky is not influenced by the presence of the occluding objects, that is,L
a′(λ
)=L
a(λ
).The model is derived based on an RGB mode. The color components of the reflected intensity reaching the RGB sensors at a point (x,y) in the 2D image plane can be expressed
115 wavelengths λ. We assume that the scene radiance and the image irradiance are the same because the situation is considered a Lambertian scene under uniform illumination [83]. For a point in direct light, the sensor measurements are
[ ]
When a point is in the shadow, the sensor measurements are
∫
Λ similar in a traffic scene, for each object point of the road surface, the RGB measurement ratio between the lit and the shadow condition is approximately constant:) .
116
Bibliography
[1] B. McQueen and J. McQueen, Intelligent Transportation Systems Architectures.
Norwood, MA: Artech House, 1999, pp. 19-49.
[2] K. Hayashi and M. Sugimoto, “Signal control system (MODERATO) in JAPAN,” in
Proc. IEEE Int. Conf. Intell. Transport. Syst., Tokyo, Japan, 1999, pp. 988 – 992.
[3] G. K. H. Pang, K. Takabashi, T. Yokota and H. Takenaga, “Adaptive route selection for dynamic route guidance system based on fuzzy-neural approaches,” IEEE Trans.
Veh. Technol., vol. 48, no. 6, pp. 2028 – 2041, Nov. 1999.
[4] V. Kastrinaki, M. Zervakis and K. Kalaitzakis, “A survey of video processing techniques for traffic applications,” Image and Vision Computing, vol. 21, no. 4, pp.
359-381, Dec. 2003.
[5] R. Cucchiara, M. Piccardi and Paola Mello, “Image analysis and rule-based reasoning for a traffic monitoring system,” IEEE Trans. Intell. Transport. Syst., vol.
1, no. 2, pp. 119-130, Jun. 2000.
[6] N. J. Ferrier, S. M. Rowe and A. Blake, “Real-time traffic monitoring,” in Proc.
Second IEEE Workshop on Applications of Computer Vision, Sarasota, Florida, 1994,
pp. 81–88.[7] D. Koller, K. Daniilidis and H. H. Nagel, “Model-based object tracking in monocular image sequences of road traffic scenes,” International Journal of Computer Vision, vol. 10, no.3, pp. 257-281, Jun. 1993.
[8] H. Veeraraghavan, O. Masoud and N. Papanikolopoulos, “Computer vision algorithms for intersection monitoring,” IEEE Trans. Intell. Transport. Syst., vol. 4, no. 2, pp.78-89, Jun. 2003.
[9] K. T. Song and J. C. Tai, “Dynamic calibration of pan-tilt-zoom cameras” IEEE
Trans. Syst., Man, Cybern. B, vol. 36, no. 5, in press.
[10] D. Beymer, P.F. McLauchlan, B. Coifman and J. Malik, “A real-time computer vision system for measuring traffic parameters,” in Proc. IEEE Comput. Vis. and
Pattern Recogn., San Juan, Puerto Rico, 1997, pp. 495-501.
[11] O. Masoud, N. P. Papanikolopoulos and E. Kwon, “The use of computer vision in monitoring weaving sections,” IEEE Trans. Intell. Transport. Syst., vol. 2, no. 1, pp.
18–25, Mar. 2001.
[12] J. C. Tai, S. T. Tseng, C. P. Lin and K. T. Song, “Real-time image tracking for automatic traffic monitoring and enforcement applications,” Image and Vision
Computing Journal, vol. 22, no. 6, pp. 485–501, Jun. 2004.
[13] Z. Zhu, G. Xu, B. Yang, D. Shi and X. Lin, “VISATRAM: a real-time vision system for automatic traffic monitoring,” Image and Vision Computing Journal, vol. 18, no.
10. pp. 485–501, July 2000.
117
[14] A. M. Sabatini, V. Genovese and E. S. Maini, “Toward low-cost vision-based 2D localisation systems for applications in rehabilitation robotics,” in Proc. IEEE Int.
Conf. Intell. Robots and Syst., Lausanne, Switzerland, 2002, pp. 1355–1360.
[15] F. Y. Wang, “A simple and analytical procedure for calibrating extrinsic camera parameters,” IEEE Trans. Robot. Automat., vol. 20, no. 1, pp. 121–124, Feb. 2004.
[16] S. Ying and G. W. Boon “Camera self-calibration from video sequences with changing focal length,” in Proc. IEEE Int. Conf. Image Processing, Chicago, Illinois, 1998, vol. 2, pp. 176–180.
[17] E. Izquierdo, “Efficient and accurate image based camera registration,” IEEE Trans.
Multimedia, vol. 5, no. 3, pp. 293–302, Sept. 2003.
[18] L. L. Wang and W. H. Tsai, “Camera calibration by vanishing lines for 3-D computer vision,” IEEE Trans. Pattern Anal. Machine Intell., vol. 13, no. 4, pp.
370–376, Apr. 1991.
[19] T. Echigo, “A camera calibration technique using three sets of parallel lines,”
Machine Vision and Applications, vol. 3, no. 3, pp. 159–167, 1990.
[20] E. K. Bas and J. D. Crisman, “An easy to install camera calibration for traffic monitoring,” in Proc. IEEE Conf. Intell. Transport. Syst., Boston, Massachusetts, 1997, pp. 362–366.
[21] C. Zhaoxue and S. Pengfei, “Efficient method for camera calibration in traffic scenes,” IEE Electron. Lett., vol. 40, no. 6, pp. 368–369, Mar. 2004.
[22] A. H. S. Lai and N. H. C. Yung, “Lane detection by orientation and length discrimination,” IEEE Trans. Syst., Man, Cybern. Part B, vol. 30, no. 4, pp. 539–548, Aug. 2000.
[23] T. N. Schoepflin and D. J. Dailey, “Dynamic camera calibration of roadside traffic management cameras for vehicle speed estimation,” IEEE Trans. Intell. Transport.
Syst., vol. 4, no. 2, pp. 90–98, Jun. 2003.
[24] N. Li, J. Bu and C. Chen, “Real-time video object segmentation using HSV space,”
in Proc. IEEE Conference on Image Processing, Rochester, New York, 2002, pp.
85-88.
[25] R. Cucchiara, M. Piccardi and P. Mello, “Image analysis and rule-based reasoning for a traffic monitoring system,” IEEE Trans. on Intelligent Transportation Systems, vol. 1, no. 2, pp.119-130, 2000.
[26] C. Eveland, K. Konolige, and R. Bolles, “Background modeling for segmentation of video-rate stereo sequences,” in Proc. of the IEEE Computer Society Conference on
Computer Vision and Pattern Recognition, Santa Barbara, 1998, pp. 266-271.
[27] J. B. Zheng, D. D. Feng, W. C. Siu, Y. N. Zhang, X. Y. Wang and R. C. Zhao, “The accurate extraction and tracking of moving objects for video surveillance,” in Proc.
International Conference on Machine Learning and Cybernetics, Beijing, China,
2002, pp. 1909-1913.[28] P. Kumar, K. Sengupta and A. Lee, “A comparative study of different color spaces
118
for foreground and shadow detection for traffic monitoring system,” in Proc. IEEE
5th International Conference on Intelligent Transportation Systems, Singapore, 2002,
pp. 100-105.[29] C. Stauffer and W.E.L. Grimson, “Learning patterns of activity using real-time tracking,” IEEE Trans. on Pattern Analysis and Machine Intelligence, vol. 22, no. 8,
[29] C. Stauffer and W.E.L. Grimson, “Learning patterns of activity using real-time tracking,” IEEE Trans. on Pattern Analysis and Machine Intelligence, vol. 22, no. 8,