Due to the different goals being emphasized in each publish, the comparison of research focus is shown in Table 2.1. The value, Yes, in the filed means that the publishes listed in the second column mentioned that they did contributions in the corresponding module.
2.2 Features Types
Gradient features:
Gradient features are usually the most important since the features are insensitive to scale, rotation, size, or colors. After thresholding the gradient values, edges would be analyzed in many publishes [12,15,19–21,25,33,36,42,51,53,55,58,62–64,67,79,80,83,91,98,
103,104,114,120]. Moreover, vertical edges [15,20,25,33,36,55,64,79,80,83,91,120] would be the most popular gardient feature to represent the license plate areas. By limiting the colors of license plates, Xu et al. extract color edges from an RGB image; Chang et al. [58]
extract color edges from an rgb image; Yang et al. [68] extract color edges from an HSV image.
Statistical features:
Statistical features for LPD or OCR are usually composed of the covariance matrix [84], the density [8,16,25–27,36,44,48,76,79,81,82,85,87,91,120], density variance [85,87,91,120], Haar-like features [85,87–89,91,111,120], or Gabor features [23,117]. The region density may be measured from the edges [8,36,76,82] or the gradients [91,120] of the license plate region. Wu et al. [86] use the frequency of zero crossing on the map of edges.
Shape-based features:
Shape-based features are usually composed of the skeleton [117], the size [10,20,36,81,94, 111], the width or the height [10,27,31,36,53,54,79,86], the aspect ratio [8,20,25,27,36,53, 60,70,76,79,81,82,86,104,109,111], the rectangularity [8,76,82], or the orientation [25,91,94]
of the license plate or plate characters. Although these features may not be scale-invariant or rotation-invariant, they are insensitive to many environment changes. The plate orien-tation calculated using least second moments would be adopted in some publishes [109].
Moreover, the symmetric property [22,38] of license plate regions is measured to eliminate the false positives.
2.3 Extracting Methods
Morphological methods:
Morphological operators, such as local or global thresholding, Sobel [19,24,78,83,95], thin-ning [113], smoothing [10,88,89,93,106,110,121], opethin-ning or closing [10,36,62,67,79,80,88, 89,107,110], or differencing [10,88,89], are widely used in license plate extraction or charac-ter segmentation because the operators does not require complex and heavy mathematical calculations.
Transformation methods:
Hough transformation is a method for detecting the borders of the license plates [24,35,53, 55,104]. Martin and Borges [31] use bottom-hat transformation to enhances the charac-ters. Hou et al. [110] measure the differences of top-hat and bottom-hat transformations to extract the license plate characters. Wu et al. [86] use bottom-hat transformation to enhance the texture in the input image. Hsieh et al. [81] and Guo et al. [122] per-form wavelet decomposition in each block of the input image and generate four subbands (smoothing, horizontal, vertical, and diagonal).
Projection:
License plates or characters could be extracted by analysing the horizontal and vertical projections [12,34,35,42,51,69,80,93,95,99,104,107]. Because characters of license plates are usually arranged in horizontal lines, some publishes [19,20,31,33,54,60,62,64,81,94, 115,123] only use vertical projection histograms to segment plate characters.
Mean Shift:
The mean shift algorithm [124] is a nonparametric clustering technique which does not require prior knowledge of the number of clusters, and does not constrain the shape of the clusters. Some publishes [8,76,82] would use the mean shift filter to extract candi-dates of license plates. Kim et al. [71] adopted the continuously adaptive mean shift algorithm (CAMShift)[xxx] to extract regions of license plates in the result after plate color measurement.
Sliding window:
Each sub-window [63,78,85,91,104,106,109,111,122] in the input image would be extracted and identifed by heuristic rules or classification methods. Anaqnostopoulos et al. [109]
proposed a sliding concentric windows(SCWs) to detect regions of license plates.
Vector quantization(VQ):
Based on vector quantization, Zunino and Rovetta [77] encode each row of the input image for locating license plates. [23]
Hidden Markov Chain(HMC):
Franc and Hlavac [112] use the Hidden Markov Chain (HMC) model to describe a relation between the image and its corresponding segmentation.
2.4 Classification Methods
Template matching:
By thresholding the minimum distance calculated from the pre-defined database, tem-plate matching algorithms [8,9,13,15,18,20,49,60,64,69,76,82,94,96,99,107,121,123] verified an input pattern as a plate or a character.
K nearest neighbors (KNN):
Cano and Perez-Cortes [11,78] classify every pixels of the input image based on KNN method. Guo et al. use a k-means cluster to identify license plate blocks.
Cascade classifier:
For object detection, Viola and Jones [125] propose an cascade classifier trained by Ad-aBoost [126]. A cascade classifier can be taken as a degenerate decision tree. Some publishes [85,87–89,91,111,120] adopt the cascade classifier to identify license plates.
Neural network(NN):
Various neural network architectures [12,16,28,39,41,41,44,48,55,56,58,70,84,90,95–97,102, 105,108,109,122] are proposed and implemented for plate identification or character recog-nition. Artificial neural network(ANN) trained by a backpropagation algorithm is used for plate character recognition in [58,97,105,106,121]. Guo et al. [122], Yuan et al. [90], and Chacon and Zimmerman [21] classify license plates based on pulse coupled neuron model(PCNN) [127], a kind of artifical neural network model. Anaqnostopoulos et al. [109]
trained a two-layer probabilistic neural network(PNN) to identify plate characters. Hu
et al. [115] use a PNN to identify low-dimension test samples segmentated from actual license plate images.
Hidden Markov Model(HMM):
For character recognition, Llorens et al. [103] and Duan et al. [104] use a HMM model with the observations of the ratio of foreground pixels in a window.
Support vector machine(SVM):
Support vector machine with radial basis functions (RBFs) is usually used for plate clas-sification [122] or plate character recognition [111,116]. Direct pixel values of an input region are scaled and the size of the region is normalized. Otherwise, Kim et al. [71]
adopted a support vector machine (SVM) with a polynomial kernel as the color texture classifier.
Genetic algorithms(GA):
Yoshimori et al. [22,37,38] and Yohimori et al. [49] change the threshold values based on genetic algorithms to extract license plates. Karungaru et al. [121] use a genetic algorithm to select the three parameters, position, size, and orientation of the input characters.
Xiong et al. [46] apply GA to seach the possible license plate area in the whole image.
Chapter 3 Detection of License Plates under Two Special Situations
3.1 License Plates with Different Appearances
This section proposes an approach to developing an automatic license plate detection system with different appearances. The car images are taken from various positions in outdoors. Because of the variations of angles from the camera to the car, the license plates will have various locations and rotation angles in an image. In the license plate detection phase, since the colors of characters and of the license plate background are generally different, the magnitude of the gradients is used to detect candidate license plate regions.
The license plates are usually located on the bumper. In the car images, there are several horizontal lines. If we use the horizontal gradients, it will be difficult to separate the regions of license plates from the bumper. Thus, the magnitude of the vertical gradients is used to detect the candidate license plate regions. These candidate regions are then evaluated based on three geometrical features: the ratio of width and height, the size and the orientation. The last feature is defined by the major axis. The various rotated character images of a specific character can be normalized to the same orientation based on the major axis of the character image. Two different appearances of the license plates are shown in Fig. 4 Experimental results show that the license plates detection method can correctly extract all license plates from 102 car images taken in outdoors.
Figure 4: Two different appearances of license plates.
The remaining parts of this section are organized as follows. Section 3.1.1 studies the motivation and modules of the orientation normalization and inverse rotation trans-formation. Section 3.1.2 presents the procedure of license plates detection. Section 3.1.3 shows experimental results and Section 3.1.4 includes some concluding remarks.
3.1.1 Orientation Normalization
This section aims to detect the license plates of the car image with various locations and to recognize the rotation-free characters in the license plates. It is useful to derive the major axis which shows the orientation of the image to detect and recognize license plates.
In the license plate detection phase, the major axis is measured in possible license plate region to evaluate the possibility to be a license plate region. Figure 5 shows the major axis on each image of the character R in five different rotation angles, where the dash lines represent the major axis of the character image.
Figure 5: The images of character R in different orientations. The dash lines represent the major axis.
When the rotation angles of a specific character image are between 90o and 270o such as Figs. 5(d)-(f), the normalized character images is inverse. However, the situation is regardless because the license plate images would not have rotation angles between 90o and 270o.
In general, the new position for the pixel in the original image after rotating an angle
is defined below [128]
The rotation transformation will result in some holes after the rotation transforma-tion is applied. In Fig. 6(b) the destinatransforma-tion pixel 2 is mapped from two source pixels, while the destination pixel 1 is mapped from none of the source pixel. These undefined destination pixels produce holes in the image. To solve the problem, for each pixel of the rotated image, the relative origin pixel is checked to see if it is the black pixel of the character image. If it is, the pixel of the rotated image is marked as character pixel;
otherwise it is marked as non-character one.
Figure 6: The rotation result of an image block. (a) The original image; (b) The rotated image with a -30-degree transformation.
The processes of orientation detection will be discussed in the following.
A. Orientation Detection
In the binary image, we first define that the mass is the black pixels whose gray level is 1. The moment of mass of the binary image is the distribution of the mass throughout the binary image. Horn [129] mentioned that the first moment of mass which is defined
as mass times distance could be used to derive the center location of the mass and the second moment of the mass could be measured the distribution of mass relative to axes through the center of the mass. And the orientation, of the mass is derived from the least second moment of the mass. Then, the major axis of the mass can be achieved from the orientation and the center. The steps to derive the orientation from the binary image are described in the following.
The first moment of mass in the binary image is defined as
C = (xc, yc) =
where g(x, y) is the black point (x, y) which gray level is 1 in the binary image.
The second moment of mass in the binary image is equal to mass times square of the distance from the black point in the binary image to a line as shown below:
S =
∫ ∫
r2g(x, y)dxdy (3.3)
where r is the perpendicular distance from the black point (x, y) to a line L. In Fig. 7, for a particular line in the binary image, two parameters are defined: the distance from the origin to the closest point on the line, and the angle between the x-axis and the line, which is measured counterclockwise. The equation of the line is presented as follows.
x sin θ− y cos θ + t = 0 (3.4)
Note that the line intersects the x-axis at sin θ−t and the y-axis a cos θ+t . The closest point on the line to the origin is located at −t sin θ, +t cos θ. Suppose that the point (x0, y0 is
Figure 7: The coordinate diagram.
located on the line The equations for the point (x0, y0) on the line L are as the following:
x0 =−t sinθ + r cos θ and y0 = +t cos θ + r sin θ.
(3.5)
Given an arbitrary black point (x, y) in the binary image, the shortest distance between (x, y) and the line L is defined as
r2 = (x− x0)2+ (y− y0)2
= t2+ 2t (x sin θ− y cos θ) − 2r (x cos θ + y sin θ) + r2+ (x2+ y2) .
(3.6)
Totally differentiating with respect to we obtain
r = x cos θ + y sin θ. (3.7)
Substituting the equation 3.7 back into the equation 3.6 lead to
r = x sin θ− y cos θ + t. (3.8)
By substituting the equation 3.8 back into the equation 3.3, the second moment of
mass can be derived as
S =
∫ ∫
(x sin θ− y cos θ + t)2g(x, y)dxdy. (3.9)
Because the second moment of mass crosses the center of the binary image, we can substitute the equation 3.2 into the equation 3.9 and totally differentiating with respect to t, we obtain
(xcsin θ− yccos θ + t) = 0, (3.10)
where (xc, yc) is the center of the binary image. Without losing the generality, we can change the coordinate to x0 = x− xc and y0 = y− yc the equation 3.10 can be rewritten as follows:
x sin θ− y cos θ + t = x0sin θ− y0cos θ, (3.11)
and we can substitute the equation 3.11 back into the equation 3.9, then the new equation is obtained.
S =
∫ ∫ (
(x0)2sin2θ + 2 (x0y0) sin θ cos θ + y02(cos 2θ− 1))
g(x, y)dx0dy0 (3.12)
Total differentiating S with respect to θ and we can obtain
tan 2θ =
∫ ∫ 2(x0y0)dx0dy0
∫ ∫ x02dx0dy0−∫ ∫
y02dx0dy0 (3.13)
Finally, we obtain the orientation θ,
θ = 1
The major axis in the binary image is defined as the line which slope is θ and the
major axis crosses the center.
3.1.2 License Plates Detection
The first step of the license plate detection system is to detect the license plate regions of the input car images. Due to the similar colors of the license plate background and that of the car body, it is difficult to detect the boundary of the license plate from the input car images in outdoors. Because the color of characters is different from that of the license plate background, the gradients of the original image are adopted to detect candidates of license plate regions. Figure 8 shows the processing flow of license plates detection.
Figure 8: The system diagram.
This section presents the details of license plates detection. The procedure consists of four major steps: (1) detection of possible license plate regions, (2) possibility mea-surement, (3) merging of broken regions, (4) inverse rotation transformation. The last step, inverse rotation transformation, has already been described in the previous section.
The details of the remaining steps are explained as follows.
A. Detection of Possible License Plate Regions
At the first step of the license plate detection phase, the possible license plate regions are detected from the vertical gradients of the input car images. The vertical gradients are derived by multiplying with a mask value for each pixel and its neighboring pixels. In the vertical gradients image, the license plate region is the area with large local variance. The local variances of the vertical gradients image are measured with a local window mask. In
this section, in order to cover the characters in the license plate of the input car images, the size of the local window mask is set as 11× 7. The smaller the window size is, the more possible the license plate regions are separated, while the larger the window size is, the over detected license plate regions occur. Figure 9 shows the possible license plate regions with three different window sizes: 7× 3, 11 × 7 and 15 × 11.
Figure 9: (a) The license plate image; (b)-(d) The possible license plate regions with the sizes of window mask, 7× 3, 11 × 7 and 15 × 11.
The pixel is defined as 1 for possible license plate regions. When we threshold the local variance image, the image of possible license plate regions is obtained. Figure 10(a) shows the image of a car with a license plate, where the colors of the license plate background and that of the car body are similar. Figure 10(b) displays the vertical gradient image of Fig. 10(a). Figure 10(c) and Fig. 10(d) demonstrate the local variance image of Fig. 10(b) and the possible license plate regions, respectively.
There may be some noise in the images of possible license plates such as holes and single dots. An opening operation of morphological analysis, in which the dilation oper-ation is performed after an erosion operoper-ation, is applied in order to reduce the undesired effect of noise and to separate the regions that were slightly connected.
Figure 10: (a) The car image with a license plate; (b) The vertical gradients of Fig. 10(a);
(c) The local variance of Fig. 10(b); (d) The possible license plate regions.
B. Possibility Measurement
To detect the most possible license plate regions from the candidate plate regions, the geometrical properties of the license plate are introduced to measure the possibility value.
The following defines the geometrical features:
• Area: If the candidate region is large, it is more likely being a license plate. A higher possibility value represents a more possible license plate region. The possibility of the area is defined as ∑Ns
Ns, where Ns is the number of boundary rectangle of the possible license plate region, s.
• Orientation: As described before, the orientation of each possible license plate re-gion can be measured. A license plate usually appears as a horizontal rectangle.
The smaller the orientation of the possible license plate region is, the higher the possibility value is. The possibility of the orientation is given by 9090−θ, where θs is
the orientation of the possible license plate region, s.
• Density: The ratio between the black regions and the area of the bounding rectangle is defined as the density of the license plate region. The license plate is always a rectangle. A higher density value means that the region is more likely to be a rectangle and to be viewed as a license plate region. The possibility of the density is defined as Bs/Ns, where Bs is the number of the possible license plate region, s.
For each possible license plate region, s, the possibility value p(s), is defined as the weighted sum of the above three features, as shown below.
p(s) = ω1 Ns
∑Ns + ω290− θs
90 + ω1Bs
Ns (3.15)
where ωi is the weighting coefficient. We need to select proper ωi that can keep a high detection rate. These values are determined according to experimental results. In this study, ω1 = 0.2, ω2 = 0.3, and ω3 = 0.5 are adopted.
C. Merging of Broken Regions
After the detection of all candidate license plate regions, a license plate is probably sepa-rated into several adjacent regions. In Fig. 11(a), since the distance between the characters F and 4 in the license plate is larger than the threshold of the window mask defined above, two separated candidate license plate regions are generated. These separated regions have to be merged to extract the accurate license plate region.
Assume that s1 and s2 are two possible license plate regions and s is the merged region of s1 and s2. Regions s1 and s2 are merged when the following two rules are satisfied.
• The distance between s1 and s2 is smaller than a threshold value.
Figure 11: (a) The car image with a license plate; (b) The separated possible license plate regions.
• The possibility value of the merged region s is larger than both of s1 and s2.
The merging operation is repeatedly performed until no regions could be merged. Then, the region with the largest possibility value is viewed as the license plate region.
3.1.3 Experimental Results
The system proposed in this chapter has been applied to 102 images with 104 license plates, involving vehicles at different pan/tilt angles. We implemented the proposed system on a Pentium II 300MHz PC with C++ language under Windows environment and used Nikon 5700 digital camera as an input device. For the license plate detection method, Fig. 12 shows the original car images, the possible license plate regions and the result image of license plate detection. There are 108 totally license plate images extracted from the test images.
Figure 12: (a) The car images with license plates; (b) The possible license plate regions;
(c) The license plate detection results.
3.1.4 Conclusion
In this chapter, we have proposed an automatic license plate detection system with dif-ferent appearances. In conventional license plate detection methods, it is difficult to determine the license plate with large pan and tilt angles. The proposed methods use major axis information which is non-sensitive to rotation variance to detect the license plate. The major axis is determined by the orientation which is the second moment of the mass and center which is the first moment of the mass in the binary image. Then, the input images can be taken from large pan and tilt angles relative to the car in outdoors.
Experiments carried out on some samples of outdoors car images show the feasibility of using the proposed methods to detect the license plates. The system proposed can be applied in general security systems and car violation prevention systems.