1 Introduction
1.2 Organization and Contribution
The following chapters in this dissertation are organized as follows.
In Chapter 2, some definitions of luminance/color contrast for different image analysis applications are introduced. Also, for the color segmentation issue, some existing image segmentation algorithms and evaluation methods in the literature are introduced.
In Chapter 3, for the automatic inspection of defects on LCDs, a luminance contrast based formula, named SEMU formula, is introduced. A SEMU-based optimal thresholding mechanism is then proposed to automatically inspect Cluster defects on LCDs.
In Section 4.1, the definition of a directional color contrast is proposed first.
Then, with the directional color contrast, the definition of color contrast in the CIE L*a*b* color space is proposed. Also, a technique that can estimate the directional color contrast of color edges in a color image is proposed. On the other hand, a definition of “visible color difference” in the CIE L*a*b* color space is introduced to represent the effects of color difference in human visual perception. Then, based the defined color contrast, a so-called visible color difference is proposed to differentiate visible color difference from invisible color difference. In Section 4.2, some visual rating experiments are performed to verify the correlation between segmentation quality and degree of over-segmentation and/or degree of under-segmentation. Then, based on the defined “visible color difference” formula in Section 4.1, which are used to estimate the degree of over-segmentation, the degree of under-segmentation, and segmentation quality, some quantitative visual error measures are proposed.
These quantitative measures are also used in the evaluation of color
segmentation. In Section 4.3, based on the directional color contrast introduced in Section 4.1, a color segmentation algorithm is proposed. On the other hand, uniting with the proposed quantitative evaluation method mentioned in Section 4.2, a new color segmentation algorithm is proposed.
Finally, conclusions are drawn in Chapter 5.
CHAPTER 2
Backgrounds
______________________________________________
Luminance/color contrast has played an important role in the development of image analysis techniques. In Section 2.1, some existing definitions of luminance/color contrast for different applications will be introduced first. On the other hand, since in this dissertation the evaluation of image segmentation results is especially discussed, the backgrounds for image segmentation algorithms and related evaluation methods are briefly introduced in Section 2.2.
2.1 Luminance/Color Contrast
To measure luminance/color contrast, several definitions of luminance/color contrast have already been proposed for various applications. In Section 2.1.1, we first introduce some existing definitions of luminance contrast. Then, in Section 2.1.2, among a variety of color difference definitions, we introduce the CIE L*a*b* color difference. Based on the CIE L*a*b* color difference, we will propose in the following chapters a few more definitions about color difference.
2.1.1 Luminance Contrast
In this section, we introduce four commonly used definitions related to luminance contrast: 1) Luminance Contrast (Weber Contrast), 2) Luminance Ratio, 3) Peak-to-Peak Contrast (Michelson Contrast, Modulation), and 4) Just Noticeable Difference (JND). The details for each of these four definitions are as follows.
1) Luminance Contrast, or Weber Contrast:
To define the relationship between the luminance of an interested area and the luminance of the surrounding area, a measure called Weber contrast is defined in [1]:
CW = (LS–Lb) / Lb. (2.1)
Here, LS is the luminance of the interested area and Lb is the luminance of the adjacent background. In [1], a thorough technical discussion of this type of measure can be found.
When the background is brighter than the interested area, CW is negative and ranges from zero to –1. On the contrary, when the background is darker, CW is a positive number that can be potentially very large. Hence, the sign and value of Cw can be used to describe the relationship between the luminance of an interested area and the luminance of the adjacent background.
2) Luminance Ratio:
To specify the difference between bright parts and dark parts in a photo, another measure called Contrast Ratio is often used. This contrast ratio is defined as:
CR = LS / Lb. (2.2)
In practice, a logarithmic variation of this CR definition, as expressed in (2.3), is also widely used.
log(CR) = log(LS) – log(Lb). (2.3)
In Equation (2.2), the contrast ratio is defined as LS divided by Lb, where LS and Lb have the same definitions as that in (2.1). The Lb in the denominator is often a reasonable estimate of the adapted luminance level around this stimulus level in human eyes. Instead of the luminance difference away from the background, the numerator is defined as the luminance of the interested area, Hence, this CR definition has a rather different mathematical behavior from the Weber contrast.
3) Peak-to-Peak Contrast (Michelson Contrast, Modulation):
To determine the strength of a signal luminance relative to the noise luminance level [1], a measure which evaluates the relation between the spread and the sum of signal and noise is introduced. Here, we have
Modulation = (Lmax - Lmin) / (Lmax + Lmin) (2.4)
In the context of human vision, such kind of noise fluctuation could be caused by scattered lights introduced into the view path by a translucent element, which partly obscures the scene behind it.
4) JND (Abbreviation for Just Noticeable Difference) [2]:
For a certain stimulus, the smallest change in the stimulus that can be perceived by human is called the JND. The JND measure is usually used in the study of Psychophysics. Specifically, this definition is often used to indicate a measure in the statistical sense that the probability of being “perceptible” is 50%. For different applications, the definition of the JND measure could be different. Especially in the FPD (Flat Panel Display) visual inspection problem that is to be discussed in this dissertation, the JND over a uniform luminance background is regarded as a gauge of luminance contrast.
2.1.2 CIE L*a*b* Color Difference
On the other hand, for color images, color difference is usually adopted as a gauge of color contrast. Among a variety of color difference definitions, we focus on the CIE L*a*b* color difference. In Section 2.1.2.1, the CIE L*a*b* color space is to be introduced. The color space transformation from the RGB color space to the CIE L*a*b* color space is also described. Then, in Section 2.1.2.2, we introduce the definition of color difference in this CIE L*a*b* color space.
2.1.2.1 CIE L*a*b* Color Space
In the analysis of colors, a proper choice of the color space is a key issue. In this dissertation, we demand three major requirements in the selection of color space: 1) separation of achromatic information from chromatic information, 2) uniform color space, and 3) similar to human visual perception. Among various color spaces, we pick the CIE L*a*b* color space. In this color space, L* represents luminance, while a* and b* represent color components. The definitions for L*, a*, and b* are expressed as follows.
,
The detail about the CIE L*a*b* space can be easily found in many color related books, like [6]. In Figure 2.1, we show an illustration of this CIE L*a*b* color space.
Fig. 2.1 Illustration of the CIE L*a*b* color space [7].
On the other hand, Fig. 2.2 illustrates the color transformation from the CIE L*a*b*
color space to the RGB color space. To digitize colors in photos, the (L*,a*,b*) values within an area of interest are first converted to (Rt, Gt, Bt) by using Equations (2.5)(2.6) (2.7)(2.8), and (2.9).
are the coefficients of the color transformation from the RGB color space to the XYZ color space.
Fig. 2.2 Color space transformation from the CIE L*a*b* space to the RGB color space.
Then, for the compression of data size and the compensation of the nonlinear effect of the monitors, the (Rt, Gt, Bt) values are further mapped to the (R, G, B) values by applying the Gamma correction process expressed in Equation (2.10).
) G channels of the output monitor.
2.1.2.2 Color Difference in CIE L*a*b* Color Space
In the CIE L*a*b* space, the differences of color components between Patch X and Patch Y are defined by their differences in lightness (∆L*), redness-greenness (∆a*), and yellowness-blueness (∆b*). That is, The overall color difference between these two samples is defined as the Euclidean distance in the CIE L*a*b*. That is,
2 The differences in color can also be defined by the differences in lightness (∆L*), chroma (∆Cab*), and hue (∆Hab*) [8]. In mathematics, we have
2.2 Introduction of Image Segmentation
In Section 2.2.1, we will introduce some existing image segmentation algorithms first.
Then, evaluation methods for color segmentation are introduced in Section 2.2.2.
2.2.1 Image Segmentation Algorithms
In recent years, plenty of efforts have been focusing on the segmentation of color images. In general, current image segmentation algorithms can be roughly classified into three major categories: 1) image domain-based techniques; 2) feature space-based techniques; and 3) physics-based techniques [13]. For image domain-based techniques, like [14], the similarity of neighboring pixels or the discontinuity of local information is used as the gauge for segmentation. Adjacent pixels with small intensity/color variations are merged together, while pixels with large enough variations are split apart. For feature space-based techniques, like [26], the data distribution of the entire image plays a crucial role. Clustering or grouping techniques are usually applied over the data distribution to allot image data into groups. On the other hand, for physics-based techniques, the adopted mathematical tools are basically the same as the former two kinds of techniques, while an underlying physical model is used to account for the reflection properties of colored matter [40].
2.2.1.1 Image Domain-Based Approaches
In general, for image domain-based techniques, the similarity of neighboring pixels or the discontinuity of local information is used as the gauge for segmentation. These image domain-based methods could be roughly classified into two kinds of methods:
1) edge-based methods and 2) region-based methods. Edge-based methods usually split apart pixels with large enough data variations, while region-based methods
merge together adjacent pixels with small intensity/color variations.
2.2.1.1.1 Edge-Based Methods
As mentioned above, edge-based methods usually split apart pixels with large enough data variations. So far, plentiful methods have been proposed to detect image boundaries/edges. For example, in [16], to detect boundaries in a given image, three criteria are introduced by the author: 1) good detection, 2) good localization, and 3) single response to a single edge. To have good boundary detection, the detector maximizes the signal-to-noise ratio; on the other hand, to have good localization of boundaries, the detected edge points should be as close to the position of the true edge as possible. With these three criteria, the author finds the optimal detector based on numerical optimization. In [14] and [18], the authors proposed a boundary detection scheme based on “edge flow”. This scheme utilizes a predictive coding model to identify the direction of change in colors and textures at each image location at a given scale. An edge flow vector at each image location is then constructed. By propagating the edge flow vectors, the boundaries can be detected at the image locations where two opposite directions of flow are encountered in a stable state. On the other hand, in [17] and [19], the gradient vector of intensity/color data is used as a gauge to locate boundaries. Usually, the locations with large gradient values are regarded as the locations of edges.
2.2.1.1.2 Region-Based Methods
Different from edge-based methods, region-based methods merge adjacent pixels with small intensity/color variations into regions. In [15] and [23], colors in an image are first quantized into several representative classes. These image pixels are then replaced by the corresponding color-class labels. Then, a criterion using the class-map
is proposed to estimate the data variations in local windows. A region growing method is used to segment the image based on the estimated data variations. In [20][21][22][25], image gradient is used as a gauge to estimate image boundaries/edges. By applying the watershed transform on the gradient magnitude, primitive regions can be produced. Then, region competition, region growing, and/or region merging are used to produce the final segmentation results.
2.2.1.2 Feature Space-Based Approaches
For feature space-based techniques [26], the data distribution of the entire image plays a crucial role. Clustering or grouping techniques are usually applied over the data distribution to allot image data into groups. In [26] and [33], given a color image, a neighborhood is modeled as a distribution of colors. Then, the authors tried to show that the increase in the accuracy of the representation causes higher-quality results for low-level vision tasks on complicated natural images, especially as the size neighborhood increases. In [27][30][32], the authors proposed a general non-parametric technique to analyze a complex multimodal feature space and to delineate arbitrarily shaped clusters in the feature space. A recursive mean shift procedure is used to detect the modes of the distribution. Based on the estimated modes of the distribution, an image segmentation algorithm is performed by classifying images pixels into corresponding clusters. On the other hand, in [29] and [34], the authors treated image segmentation as a graph partitioning problem and proposed a so-called “normalized cut” method to segment the graph. The normalized cut method measures the overall dissimilarity between different groups as well as the overall similarity within the groups. In [31], image segmentation is formulated as a data clustering problem based sparse proximity data. Dissimilarities of pairs of textured regions are computed based on a multiscale Gabor filter representation. Then,
an optimization framework, which uses statistical tests as a measure of homogeneity, is proposed for unsupervised texture segmentation. In [35], the authors extended the general idea of a histogram to the homogeneity domain. Then, uniform regions are identified via multi-level thresholding on the homogeneity histogram. On the other hand, in [36][37][38], based on statistical frameworks with the assumption of Gaussian/Non-Gaussian densities, image analysis techniques, such as expectation maximization, independent component analysis, or Data-Driven Markov Chain Monte Carlo, are applied for image segmentation.
2.2.1.3 Physics-Based Approaches
For physics-based techniques, the adopted mathematical tools are basically the same as the former two kinds of techniques, while an underlying physical model is used to account for the reflection properties of colored matter [40]. In [13] and [40], several photometric invariant similarity measures are proposed to assist image analysis techniques in handling undesired imaging conditions, such as shading, shadows, illumination changes, and highlights.
2.2.2 Evaluation Methods for Image Segmentation
Color segmentation is a crucial step in image analysis and pattern recognition. The performance of color segmentation may significantly affect the quality of an image understanding system. So far, hundreds of color segmentation algorithms have already been developed to deal with various kinds of image-related applications [13][41]. For these color segmentation algorithms, the automatic setting of controlling parameters is usually a difficult task. Currently, these control parameters are often adjusted by the users in an interactive and tiresome manner. Moreover, the selection of control parameters is also image-dependent. For most color segmentation algorithms, there exists no parameter setting that is universally applicable.
On the other hand, it is well known that performance evaluation of segmentation algorithms is critical and essential in the development of image understanding systems. However, as compared with the tremendous efforts spent in the development of segmentation algorithms, relatively fewer efforts have been made on the subject of image segmentation evaluation [42][43][44][45][46]. As shown in Fig. 2.3, Zhang classified existing evaluation methods for image segmentation into three categories [42][43]: 1) analytical methods; 2) discrepancy methods; and 3) goodness methods.
Analytical methods directly evaluate segmentation algorithms by analyzing their principles, requirements, utilities and complexity, etc [42][43]. Due to the lack of a general theory for image segmentation, analytical methods work well only for some particular models or for some desirable properties of the algorithms. Moreover, these analytical methods themselves are seldom used alone. On the contrary, both discrepancy methods and goodness methods evaluate the performance of segmentation by judging the quality of segmentation results directly. Especially, discrepancy methods measure the difference between the segmentation result and a
reference result, which is usually an expected result or a ground truth [47][48]. On the other hand, as illustrated in Fig. 2.3, goodness methods evaluate the segmentation results directly with certain quality measures, without the use of any reference result.
Fig. 2.3 Approaches for Evaluating Image Segmentation [42][43].
Among these three evaluation categories, the third type of methods, the goodness methods, is considered a more practical approach due to its direct evaluation of segmentation results without any user-dependent ground truth [49][50]. In this dissertation, we’ll focus on this type of evaluation methods. Moreover, we believe that, some evaluation principles and formulas in these evaluation methods can be used to facilitate the development of segmentation algorithm. Hence, some goodness measures that had been proposed before are to be used in the development of a segmentation algorithm in this dissertation.
So far, several goodness methods have already been proposed [42][43][49][50].
One type of method is to evaluate segmented results with the use of the “Peak Signal to Noise Ratio” (PSNR) function, which is defined as follows:
( )
f = MSE( )
I where I is an N×M color image f(x,y), and f’(x,y) is denoted as the segmented result, with the color of each segmented region being filled with the average color of that region. Then, with a specified region number, the desired segmented results are defined to be the results with the maximal PSNR value, or equally with the minimum MSE value. In general, this type of method tends to measure the within-region color difference between the original color image and the segmented result. However, due to the lack of consideration regarding the color contrast between adjacent regions, this type of method tends to prefer over-segmented results, which include many small regions. This is because a larger value of MSE would be expected when we merge several small regions into a large one.
To avoid producing overly segmented results, the factor of region area is considered in the approach proposed by Liu and Yang [49]. In their approach, an evaluation function named F function is defined as:
( ) ( ) ∑
where I is the image to be segmented, R is the number of regions in the segmented image, ei is the color error of the ith region, Ai is the area of the ith region, and N, M represent the length and width of the image. Here, ei is defined as the sum of the Euclidean distance in the RGB color space between the color vectors in the original image and the color vectors in the segmented image, in the ith region. Consequently, with a smaller F-function value, the segmented result is regarded to be better.
Although these methods take the factor of region area into account to avoid the
segmented results from including too many small regions, the relation between color difference and region area is not clearly treated. Hence, these F-function based methods may include both over-segmented regions and under-segmented regions at the same time. Furthermore, since these evaluation functions only consider the color differences within regions but ignore the color contrast between regions, they may cause some undesired circumstances. For example, the segmented result with each pixel being labeled as an independent region may be treated as a “perfect” result without any color difference error.
Based on Equation (2.21), two further improved evaluation functions are proposed in [50] that are defined as:
( ) ( ) ∑ [ ( ) ] ∑
where R(Ai) represents the number of regions with the area size Ai. In both equations, the areas of regions are considered to punish these segmentation results with too many small regions. Similarly, the number of segmented regions is also included, aiming to achieve segmented results with an appropriate number of homogeneous regions.
For the above evaluation functions, two primary requirements are adopted to define “preferred” segmentation results: smaller color difference and fewer segmented regions. However, color difference and the number of segmented regions are two very
For the above evaluation functions, two primary requirements are adopted to define “preferred” segmentation results: smaller color difference and fewer segmented regions. However, color difference and the number of segmented regions are two very