ORIGINAL ARTICLE
Directional textures auto-inspection using principal
component analysis
Ssu-Han Chen&Der-Baau Perng
Received: 9 July 2010 / Accepted: 29 December 2010 / Published online: 13 January 2011 # Springer-Verlag London Limited 2011
Abstract This paper describes a global image restoration scheme using a principal component analysis that can be used to inspect defects in directional textured surfaces automatically. Decomposing the gray level of image pixels into an ensemble of row vectors, the input spatial domain image is transformed into principal component space so that the directional textures are well approximated by firstk major components and their corresponding weight vectors, named truncated component solution (TCS). Then the local defects will be revealed by applying image subtraction between the original image and the TCS. This procedure blurs all directional textures and pre-serves only the local defects that were initially embedded in the input image. These defects, if any, are finally extracted by thresholding. Experiments on a variety of product surfaces with directional textures such as straight, slanted, orthogonal, slanted orthogonal, and oblique linear primitives were conducted to demonstrate the effectiveness and robustness of the proposed method. Furthermore, some preliminary experiments were also conducted to demonstrate the proposed scheme was insensitive to horizontal and vertical shifting, changes in illumination, and image rotation.
Keyword Directional texture . Principal component analysis . Defect inspection . Machine vision
1 Introduction
A directional texture is a set of line primitives in some regular or repetitive arrangement over an entire texture region. Non-destructive visual inspection approaches for discriminating textures from abnormalities have been applied to a variety of surfaces such as machined surfaces, semiconductor products, natural wood, and textiles [1]. This type of non-destructive visual inspection is quite valuable in industry for assisting with or replacing subjective and repetitive manual inspection processes. Global approaches to inspecting defects in directional textured surfaces usually start with a forward transform and filtering, followed by an inverse transform and thresholding as shown in Fig. 1 [2]. Thus, only the local defects that existed in the original input image were preserved, and all directional textures were removed. Possible global approaches to detecting surface defects include the use of discrete Fourier transform [1, 3], discrete wavelet transform [4], singular value decomposi-tion [5], independent component analysis [6], and non-negative matrix factorization [7]. Each of these methods may be a good choice for implicit or qualitative inspection of directional textured surfaces. Because these methods require neither textural features nor any reference image for comparison, they are immune to the limitations inherent in local feature extraction or golden template matching methods [8].
In this paper, we presented a new global image restoration scheme using the principal component analysis (PCA) for automatic surface quality control. PCA converts statistically dependent image elements into independent weight vectors and represents the principal components (PCs) by the linear combinations of the given data and the ordered weight vectors. The
S.-H. Chen (*)
:
D.-B. PerngDepartment of Industrial Engineering and Management, National Chiao Tung University,
HsinChu, Taiwan 30010
e-mail: [email protected] DOI 10.1007/s00170-010-3141-1
magnitude of eigenvalue of each component indicates how much variance of the given image it accounts for. For a directional texture surface, the larger magnitude of eigenvalue it has, the more texture information it includes. The main information of an image can be represented by the first k PCs and the corresponding weight vectors, named truncated component solution (TCS) [9]. All other smaller PCs and the corresponding weight vectors provide the local defects information if they initially are embedded in the surface. Statistically process controlled (SPC) binarization is used to set the upper and lower thresholds for discriminating between uniform and defective regions in the restored image.
The rest of this paper was organized as follows. In Section2, we reviewed the PCA applications in computer vision. In Section3, we described the proposed PCA-based image restoration scheme. In Section 4, we presented the experimentation and corresponding results. Our concluding remarks were given in Section5.
2 PCA applications to computer vision
PCA has found numerous applications in computer vision. Fukanaga introduced PCA to compute the directions of greatest variance for a set of vectors [10]. Gonzalez and Wintz performed a rotation–translation
procedure that can standardize the coordinates of an object by PCA [11]. Jain described PCA for image compression to obtain low bit-rate representations of images by removing redundancy which is inherent in the image data [12]. Murase and Nayar implied PCA for image compression in object recognition [13]. Many researchers used PCA to map samples into a low-dimensional and the most expressive feature space for face representation and recognition [14]. Gottumukkal and Asari developed a face detection system which is capable of detecting faces in real time from a streaming color video based on PCA [15]. Besides, PCA features can be naturally adopted for the task of clustering or classifica-tion in pattern recogniclassifica-tion [16].
3 PCA-based defect detection scheme 3.1 Principal component analysis
The truncation error of PCA is minimized in the mean-square sense. If an imageX of size n×m is considered to be a realization of a two-dimensional random process, the covariance matrices for the PCA should be estimated from an ensemble of images rather than a single image. Following [17] in this paper, we chop up the input image into an ensemble of n row vectors, X=[x1,x2,…, xn]T. So
Spatial Domain Transformed Domain Spatial Domain
Fig. 1 The flowchart of defect detection in directional textures using global approaches
(a)
(b)
(c)
(d)
(e)
an image can also be regarded asm ensembles of a random process and then we can use the PCA for a two-dimensional image as well. The imageX is centered by subtracting off column mean vector mX. The PCA forms the following
linear combinations:
Y ¼ W X mð XÞ ð1Þ
where Y=[y1, y2, …, yn]T is a set of row vectors of n
uncorrelated PCs;W=[wij]n×nis the weight matrix of thejth variable for ith principal component and its columns are orthonormal.
Empirically, we find theW by calculating the eigenvec-tors of covariance matrix of (X–mX) first and order them by
eigenvalues, highest to lowest. This gives the PCs in order of significance. The five artificial images containing straight, slanted, orthogonal, slanted orthogonal, and oblique linear structures are shown in Fig.2a–e and their corresponding first 15 eigenvalues are demonstrated in
Fig. 3. Note that the contours in Fig. 3, the first few eigenvalues, dominate all other eigenvalues, which dramat-ically draw down closing to zero. In most case, the larger the eigenvalue is, the more variation, repetitive and periodical primitives, of X are accounted for. And all the other smaller eigenvalues account for the remaining variation, local defects, ofX.
3.2 PCA-based image restoration
In the image compression or degradation/reconstruction application, the dimensionality of the Y in Eq.1 could be reduced by discarding a set of the eigenvectors that have small eigenvalues. By properly selecting the first k eigenvalues and adopt the corresponding PCs and weight vectors, the restored image will closely approximate to the arranged texture character of original image and named TCS. But in the surface defect inspection application, the dominant eigenvalues should be ignored and adopt the
0 7 14 21 28 35 4 1 2 3 5 6 7 8 9 10 11 12 13 14 15 Number Magnitude of eigen v alue (a) (b) (c) (d) (e) F i g . 3 T h e p l o t o f t h e corresponding first fifteen largest eigenvalues of Fig.2a–e
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
Fig. 4 a The artificial spot defective image. b–h The TCSs generated from reconstructing the PCs and weight vectors with the first one to the first seven eigenvalues respectively
remaining PCs and weight vectors to restore the image for background texture removal. The inverse-transformation formula of PCA is given as,
bX ¼Xmj¼kþ1WT
j yj; ð2Þ
where the restored image bX is reconstructed without introducing the information of first k PCs and weight vectors. To improve the computational efficient of Eq.2, another equivalent inverse-transformation formula is given as,
bX ¼ X Xkj¼1WT j yjþ mX
; ð3Þ
where the restored image bX (in statistics, it means the truncation error of X) is reconstructed by applying image subtraction between the original image and the TCS. From geometric point of view, the objective has gone back to reconstruct a TCS which closely approximates to the texture of the original image by properly selecting the k value and then manifests the discrepancies between the original image and the approximate texture of TCS by image subtraction operation. Remember we have subtracted
mX at the start, thus we need to add it in Eq. 3 for
completeness. As a consequence, the directional textures are eliminated and only local defects are preserved if they initially existed in the surface.
3.3 Selecting the proper number of PCs
Here, the issue of how to determine a proper k value is shown in Eq. 3. We demonstrated the effect of k value on the approximation ability of the PCA for faulty texture surface. Figure4awas an image derived from Fig.2bwith some spot defects on it. Figure 4a was transformed into PCA space and then reconstructed the TCS using variousk values. Fork=1, the rotated line primitives of Fig.4awere not well shaped, and only horizontal linear textures appeared, as shown in Fig. 4b. The phenomenon shown in Fig. 4b is the result of under-specification of k and involves a loss of texture information in TCS. Fork=2 to 4, the background texture of Fig. 4a was approximated relatively well, as shown in Fig.4c–e, even if a few defects and some noise were present. These relatively structured results signify the directional texture of Fig.2b, which was sufficiently represented fork=2 to 4. For even larger values ofk, the background texture was polluted with increasingly significant spot defects, as shown in Fig. 4f–h. These phenomena are the result of over-specification ofk; defects in the TCS were wrongly included.
As shown in Fig. 4b–h, under- or over-specification of thek value definitely distorts the TCS. For this reason, we must select an appropriate value for k in Eq. 3 that will sufficiently approximate the texture structure so that the subtraction function can be used to obtain the restored image. The eigenvalues of each image under inspection are first normalized. Then retain only those components whose normalized eigenvalue are greater than one. This is referred to as the normalized-eigenvalue-greater-than-one (NEGTO) rule in this paper. In our experiments of various images, the components with NEGTO represent latent information of the texture structure. As shown in Fig. 5a, the ordered
-2 0 2 4 6 8 10 12 5 1 2 3 4 6 7 8 9 10 11 12 13 14 15 Number Magnitude of Nor maliz ed eigen v alues
(a)
(b)
Fig. 5 a The first fifteen largest normalized eigenvalues of Fig.4a. b The restored image fromk=4
(a1) (a2) (a3) (a4)
(b1) (b2) (b3) (b4)
(c1) (c2) (c3) (c4)
Fig. 7 a1–a4 One defect-free OLED panel and three defec-tive ones with hole, scratch, and erosion; b1–b4 corresponding restored images; and c1–c4 corresponding binarized images
(a1)
(a2)
(a3)
(a4)
(b1)
(b2)
(b3)
(b4)
(c1)
(c2)
(c3)
(c4)
Fig. 8 a1–a4 One defect-free internal thread and three defec-tive ones with scratch, collapse/ flaw, and collapse/scratch; b1– b4 corresponding restored images; and c1–c4
magnitude of normalized eigenvalues decreases around to zero rapidly and then keeps steady subsequently. The first four normalized eigenvalues were larger than one so that the first four PCs and weight vectors should be included for global texture approximation. Figure5bshows the restored image which is generated from Eq. 3 with k=4. It is observed that the global texture has been removed or blurred and the local spot defectives are preserved. 3.4 Statistical process control binarization
Because the intensities of the pixels in the restored image varied only slightly, the statistical process control (SPC) binarization method is used to establish the upper and lower control limits for highlighting the defects. The SPC binarization method is,
ef x; yð Þ ¼ 255 if m c sbf x; yð Þm þ c s 0 otherwise
(
ð4Þ wherec is a control constant, and μ and σ are the mean and standard deviation, respectively, of the gray levels in the restored image. If a pixel has a gray level that falls between the upper and the lower limits, it is shown as white and is considered to be a texture element to be removed.
Otherwise, it is shown as black and is considered to be a defective element to be preserved. Figure 6 shows the binarized result of Fig. 5b for c=6.5, which clearly produced white directional textures and dark defects. Blob analysis can always be used to remove the small-area noise blobs. The threshold for small-area noise blobs can be set in terms of the requirement specification.
4 Experiment results
We implemented the proposed scheme and conducted experiments to evaluate its performance in detecting defects for different product surfaces. The inspection algorithm was implemented on AMD Athlon 64 X2 dual-core processor 3600+ PC with 2 GB RAM and was programmed by MATLAB. Several types of directional texture patterns with defective or non-defective samples were used, including an organic light-emitting diode (OLED) panel, an internal thread, a polymer light-emitting diode (PLED) panel, a contact lens dioptric pattern, and a diamond mesh. The size of the images under inspection is 180×180 pixels. Some preliminary experiments were also conducted to demon-strate the effects on the proposed scheme of horizontal and vertical shifting, illumination changes, and image rotation.
(a1)
(a2)
(a3)
(a4)
(b1)
(b2)
(b3)
(b4)
(c1)
(c2)
(c3)
(c4)
Fig. 9 a1–a4 One defect-free PLED panel and three defective ones with dark point, particle, and scratch; b1–b4
corresponding restored images; and c1–c4 corresponding binar-ized images
4.1 Directional texture surface inspection
Parameter c of the proposed scheme should initially be selected in a pre-training stage. Fifteen individual training samples (seven defect-free and eight defective samples) were prepared. The number of correct detections was recorded in a supervised manner for values of c in the range 0.5–8.0 at increments of 0.5, and the value of c that resulted in the highest correct detection rate was selected. Thec values for the SPC formula values of 5.5, 4.0, 5.5, 4.5, and 4.0 were selected for the four products.
After the initial c-training stages, some images for follow-up defect inspection were tested. Figs. 7a1–a4,
Figs. 8a1–a4, Figs. 9a1–a4, Figs. 10a1–a4, and Figs. 11a1–a4 show one defect-free and three defective
product surfaces with distinct defects. Figures 7b1–b4,
Figs. 8b1–b4, Figs. 9b1–b4, Figs. 10b1–b4, and
Figs. 11b1–b4 show the corresponding restored images
using the PCA-based image restoration scheme. It took less than 0.17 second in average to inspect such an image. Note that the directional textures of these product surfaces were almost eliminated. Figures 7c1–c4, Figs. 8c1–c4,
Figs. 9c1–c4, Fig. 10c1–c4, and Fig. 11c1–c4 show the
corresponding binarized images generated by SPC binar-ization withc=5.5, 4.0, 5.5, 4.5, and 4.0, respectively. Note
that Figs.7c1–11c1 shows clear responses, and Figs.7c2–
c4 to Figs. 11c2–c4 show the defects that were initially
embedded in the product surfaces.
The proposed scheme indicated the defect location and roughly preserved the defect shape. In real-world applications, the defect location can be used not only for backtracking, but also to analyze the type of defect distribution (e.g., random or clustered) on the surface for in-process monitoring and control. The defect shape can provide extra geometrical features (e.g., area, elongation, centroid, or perimeter) by blob analysis to identify the defect compared to the specification, or for recognizing the defect type by providing discriminating features to a classifier.
4.2 Effect of image shifting
The effect of the proposed PCA-based image restoration scheme in the presence of shifting was evaluated. Samples for this experiment were taken from Figs. 9a1 and a2. Because the PLED sample consisted of a 10×10-pixel rectangular pattern, only the 1-, 3-, 5-, 7-, and 9-pixel(s) vertically or horizontally shifted versions shown in Figs. 12a1–a10 and Figs. 13a1–a10, respectively, were
tested using the fixed parameter c=5.5. The proposed scheme produced clear results and highlighted defects on the
(a1)
(a2)
(a3)
(a4)
(b1)
(b2)
(b3)
(b4)
(c1)
(c2)
(c3)
(c4)
Fig. 10 a1–a4 One defect-free contact lens dioptric pattern and three defective ones with twist, eddy, and dark point; b1–b4 corresponding restored images; and c1–c4 corresponding binar-ized images
surfaces even when the spatial arrangement had been changed, as shown in Figs.12b1–b10 and Figs. 13b1–b10. Thus, the
proposed scheme was able to detect defects in directional textured images and was invariant to horizontal or vertical shifting of the directional texture. This shift-invariance property is quite useful for inspecting large object surfaces in real-world applications. Users could arrange several cameras in an automated optical inspection (AOI) system in parallel across the inspected object. These cameras could be moved to the next portion to continue the inspection process. In general, the textures in each sub-image scanned with different fields of view may not appear in exactly the same manner and may be accompanied by slight shifting. The shift-invariance property of the proposed scheme handles this complex case and is useful for inspecting sub-images of large objects. Moreover, the shifting caused by errors in machine actuators can also be generally tolerated. 4.3 Effect of change in illumination
The effect of the proposed PCA-based image restoration scheme under conditions of varying illumination was
evaluated. Using a uniform light source system, subtrac-tive and addisubtrac-tive illumination changes were applied to Figs. 9a1 and a2, respectively, to simulate the under-exposed and overunder-exposed conditions shown in Figs.14a1, a2, and a3, a4. The proposed scheme to the images was applied using the fixed parameter c=5.5. As shown in Figs. 14b1–b4, the proposed scheme produced clear results and highlighted the defects on the surfaces under all illumination conditions. As shown in the corresponding Fig. 14b1–b4, the proposed scheme pro-duced clear results and highlighted defects on the surfaces even if the optimal parameter c was determined under a specific illumination level. The PCA adaptively recon-struct a TCS whose latent recon-structure (including shape and intensity) was very similar to the original image. As long as the texture and defects of an image were enough (i.e., not distorted), the proposed scheme is robust to varying illumination conditions. This illumination-invariance property of the proposed scheme will reduce the rate of false alarms or missed detections if the AOI system encounters unexpected uniform light source instability in real-world applications.
(a1)
(a2)
(a3)
(a4)
(b1)
(b2)
(b3)
(b4)
(c1)
(c2)
(c3)
(c4)
Fig. 11 a1–a4 One defect-free diamond mesh and three defec-tive ones with a particle, a fiber, and a cut; b1–b4 corresponding restored images; and c1–c4 corresponding binarized images
4.4 Effect of image rotation
The effect of the proposed PCA-based image restora-tion scheme in the presence of image rotarestora-tions was evaluated. Figures 15a1–a5 and 16a1–a5 were obtained by rotating Figs. 9a1 and a2 by 9°, 18°, 27°, 36°, and 45°, respectively. Then the proposed scheme to the images was applied using the fixed parameter c=5.5. After extracting the high-energy components related to the texture and binarizing the restored image, it is clear that the proposed scheme produced good results and highlighted defects on the surfaces, even when the images had been rotated, as shown in Figs. 15b1–b5
and 16b1–b5. The results of Figs.15 and 16 thus show that the proposed scheme is insensitive to image rotation. This rotation invariance will eliminate tedious calibra-tions in real-world applicacalibra-tions because they will require neither an extremely precise AOI hardware fixture for object placement nor an effective AOI software algorithm for image reversal.
5 Conclusions and discussions
6 Conclusions
In automatic surface inspection, we often tackle the problem of detecting small defects embedded in a homogeneous texture. Global approaches are always recommended because they do not rely on textural features extraction from every specific surface. We have described a new global PCA-based image restoration scheme for detecting defects in directional textured surfaces. Chopping up the image into an ensemble of horizontal scan lines, the scheme first transformed the original input space into the principal component space. The linear primitives associated with the first k major components and corresponding weight vectors were determined by NEGTO criterion and then were recon-structed a corresponding TCS for texture approximation. The directional texture could then be removed by
(a1) (a2) (a3) (a4) (a5)
(a6) (a7) (a8) (a9) (a10)
(b1) (b2) (b3) (b4) (b5)
(b6) (b7) (b8) (b9) (b10)
Fig. 12 a1-a5 and a6-a10 were the vertical and horizontal shift-ing versions of Fig.9a1 by 1-, 3-, 5-, 7-, and 9-pixel(s); b1–b5 and b6–b10 were the corresponding binarized images
(a1) (a2) (a3) (a4) (a5)
(a6) (a7) (a8) (a9) (a10)
(b1) (b2) (b3) (b4) (b5)
(b6) (b7) (b8) (b9) (b10)
Fig. 13 a1–a5 and a6–a10 were the vertical and horizontal shifting versions of Fig.9a2 by 1-, 3-, 5-, 7-, and 9-pixel(s); b1– b5 and b6–b10 were the corresponding binarized images
(a1)
(a2)
(b1)
(b2)
(a3)
(a4)
(b3)
(b4)
Fig. 14 a1, a2 and a3, a4, respectively, were derived by changing the illumination of Fig.9a1 and a2; and b1–b4, respectively, was the binarized image of Fig.14a1–a4
implying image subtraction operation between the original image and the TCS and were binarized by SPC binarization method; the textures were removed in the final image while any original defect was distinctly preserved. A defect-free directional texture surface image resulted in a clear response in our experiments; otherwise, distinct defects were clearly identified in precise locations and with rough shapes. This demonstrated the ability of the proposed scheme to detect defects in surfaces with various periodical linear patterns. The proposed global PCA-based restoration scheme was insensi-tive to horizontal and vertical shifting, changes in illumina-tion, or image rotation. Such three insensitive characteristics greatly support the practicality of the proposed scheme for industrial inspection.
7 Discussions
The proposed method can be regarded as a kind of self-refining scheme that can self-reconstruct a referential golden template of original directional texture surface for template matching. Compared with the available template matching methods, the proposed scheme need not have a supervised training phase to produce a golden template, whereas it only requires the sample itself and of course releases from the alignment dilemma. However, the original texture may not be well reconstructed whenever some over large defects seriously destroy the regularity of texture. This remains a subject for further research.
(a1) (a2) (a3) (a4) (a5)
(b1) (b2) (b3) (b4) (b5)
Fig. 15 a1–a5, respectively was obtained by rotating Fig. 9a1 with rotations of 9°, 18°,…, and 45°; and b1–b5, respectively, was the binarized image of Fig.15a1–a5
(a1) (a2) (a3) (a4) (a5)
(b1) (b2) (b3) (b4) (b5)
Fig. 16 a1–a5, respectively, was derived by rotating Fig. 9a2 with rotations of 9°, 18°,…, and 45°; and b1–b5, respectively, was the binarized images of Fig.16a1–a5
Acknowledgements This research is partially supported by the National Science Council, Taiwan, under contract No. NSC 97-2221-E-009-112-MY3.
References
1. Tsai DM, Hsieh CY (1999) Automated surface inspection for directional textures. Image Vis Comput 18(1):49–62
2. Kumar A (2008) Computer-vision-based fabric defect detection: a survey. IEEE Trans Ind Electron 55(1):348–363
3. Perng DB, Chen SH, Chang YS (2010) A novel internal thread defect auto-inspection system. Int J Adv Manuf Technol 47(5– 8):731–743
4. Tsai DM, Chiang CH (2003) Automatic band selection for wavelet reconstruction in the application of defect detection. Image Vis Comput 21(5):413–431
5. Lu CJ, Tsai DM (2005) Automatic defect inspection for LCDs using singular value decomposition. Int J Adv Manuf Technol 25:53–61
6. Lu CJ, Tsai DM (2008) Independent component analysis-based defect detection in patterned liquid crystal display surfaces. Image Vis Comput 26:955–970
7. Perng DB, Chen SH (2010) Automatic surface inspection for directional textures using nonnegative matrix factorization. Int J Adv Manuf Technol 48(5–8):671–689
8. Lin HD, Ho DC (2007) Detection of pinhole defects on chips and wafers using DCT enhancement in computer vision systems. Int J Adv Manuf Technol 34(5–6):567–583
9. Gorsuch RL (1988) Handbook of multivariate experimental psychology. Plenum Press, New York
10. Fukanaga K (1972) Introduction to statistical pattern recognition. Academic, New York
11. Gonzalez RC, Wintz P (1997) Digital image processing. Addison-Wesley, Reading
12. Jain AK (1989) Fundamentals of digital image processing. Prentice-Hall, Englewood Cliffs
13. Murase H, Nayar S (1995) Visual learning and recognition of 3-D objects from appearance. Int J Comput Vis 14:5–24
14. Tang X, Wang X (2004) Face sketch recognition. IEEE Trans Circuits Sys Video Technol 14(1):50–57
15. Gottumukkal R, Asari VK (2003) Real time face detection from color video stream based on PCA method. Proc 32nd Appl Imagery Pattern Recognit Workshop:146–150
16. Jolliffe IT (2002) Principal component analysis. Springer, New York
17. Ranade A, Mahabalarao SS, Kale S (2007) A variation on SVD based image compression. Image Vis Comput 25:771–777
