The hardware system for internal thread extraction is shown in Figure 9. The sequences of internal thread images were captured by a TELI CS8320 black and white camera with a resolution of 640 × 480 and a Matrox Meteor II frame grabber. A CCD with an illumination less than 0.4 lux is recommended for this task. A 7-in Hawkeye Slim rigid industrial endoscope connected to a 90° side-view mirror tube (as shown in Figure 10) was used as the lens. The outside diameter of the endoscope was only 0.20-in and included a compact illumination fiber. The Moritex MHF-G150LR halogen light source supplied white light to the endoscope to enable the CCD to receive clear images in dark cavities. To overcome the line of sight limitation of the 90° side-view adaptor, the mechanism included a rotational servo motor (SM3416D_PLS) and a linear actuator (SmartT integrated module) to observe all the surfaces of the internal thread at different successive angles and depths. The apparatus was connected to a computer. A flowchart of the proposed vision inspection method is given in Figure 11.
The details of this method are described below.
15
Figure 9: Proposed OTPG hardware
Figure 10: The 90° side-view adaptor (from http://www.gradientlens.com/)
16
Image processing by cosine transformation
Are there pixels out of upper control limit? Grade C: collapse or flaw of defect , if N1 = 0 and N2 = 1 Grade D: mixed type defects , if N1 = 1 and N2 = 1
Are there pixels out of lower control limit?
Yes N2 = 1 No
Figure 11: Flowchart of the proposed OTPG algorithm
17 3.2 Registration of the 2D unwrapped image
As shown in Figure 12, the inner surface of internal thread can be fully observed by the proposed OTPG. Even so, image distortion and non-uniform illumination will occur due to the inherent endoscope structure and the cylindrical geometry of the internal threads. The farther away the pixels are from the center of the captured image, the more significant the above phenomena will be. Thus, only the region with little distortion near the center of each image for various angles and depths are used. With appropriate control of the sensor and the associated motion unit, a sequence of 150 × 150 pixel low-distortion images by rotating the fixture in 15° steps could be extracted. The ROI of two successive images under these conditions is restricted to only two non-overlapping neighbors. After the fixture has rotated 360°, the linear actuator raises the fixture to the next level and the rotation is repeated. The procedure continues until images of all the inner surfaces of the internal thread have been captured. Finally, the captured sequence of low-distortion images is used to reconstruct a 2D unwrapped image with size 3600 × 1500. It needs 24 × 10 = 240 sub-images to completely registrate one internal thread with 15.3mm in diameter and 20mm in length. Figure 13 shows the reconstructed 2D unwrapped image of an internal thread obtained by the described approach. In this figure, the crest and root of the internal thread correspond to the wide and narrow white bands, respectively. The flanks of the internal thread appear as gray bands. A complete thread pattern is composed of a wide white band, a gray band, a narrow white band, and another gray band in that order. The inner surface of an internal thread is indeed a type of directional texture that comprises repetitive and periodic thread patterns.
18
Figure 12: A partial wall image of internal thread
Figure 13: Reconstructed 2D unwrapped image of an internal thread
19 3.3 OTPG algorithm
The most common defects in internal threads are collapses on the crest, scratches on the flank, and flaws in the root, as shown in Figure 14. A scratch usually appears as a bulge that will cause an internal thread to bind with an external one, while a collapse or flaw appears as a cavity that will decrease the tight fit of a thread. This section focuses on developing an auto-inspection software, OTPG algorithm, to detect those defects that are embedded in homogeneous thread patterns. An implicit qualitative inspection algorithm is used to detect the embedded defects. The OTPG algorithm includes four major operations: unwrapped image normalization, normalized image segmentation, thread pattern blurring, and defect extraction. These are discussed below.
3.3.1 Unwrapped image normalization
Because the internal thread is at some arbitrary orientation in relation to the OTPG fixture during defect inspection, the start point of the tapping process in the reconstructed unwrapped image will appear at some random location. To ensure that the relative position of the global structure of each unwrapped image coincide, a process that can automatically reorient the start point of the tapping process in the unwrapped image was developed so it is always on the right-hand side of the image. The procedure of normalizing the unwrapped image is described below and illustrated in Figure 14.
The key to normalizing an unwrapped image is to find the start point of the tapping process of the internal thread and place it on the right-hand side of the image. To do this requires a good binary image where the foreground is composed of white bands (crests and roots) and the background is composed of gray bands (flanks). In addition, to locate the initial root in the binary image is necessary. The initial root is generated early in the thread tapping process. Therefore, if the intensity of each pixel of the binary image was tracked one-by-one, scanning from left to right and top to bottom, the frontier foreground element and the corresponding eight-connected elements must be the initial root.
20
Figure 14(a) shows an initial reconstructed unwrapped image of an internal thread. The grayscale closing operator with structure element size k1 × k1 was first applied to fix the interspaces and fill up the holes; the result is shown in Figure 14(b). Then the grayscale image of Figure 14(b) was converted to a binary image using a threshold value calculated with Equation (1) to separate the crests and roots from the flanks,
( )
k2 maxvalue
threshold = G − (1) where G is the universal set of gray values of Figure 14(b) and k2∈[1, max(G) – 1] is an offset constant. This produces the binary image of Figure 14(c). In Figure 14(c), each eight-connected foreground element can be regarded as a blob and apply a row-by-row labeling algorithm from left to right and top to bottom. The row-by-row labeling algorithm is guaranteed to find the initial root of an internal thread because it is the first one to be produced in the tapping process. The blob of the initial root is labeled as the index one and its corresponding right-bottom coordinate (x*, y*), the start point of the tapping process, is recorded as shown in Figure 14(d). Then the coordinate (x*, y*) was mapped onto the unwrapped image of Figure 14(a) and this image was divided into left and right sub-images, as shown in Figure 14(e), based on the x* coordinate. The unwrapped image can be rounded arbitrarily due to the intrinsic cylindrical structure of the internal thread. A normalized image by rounding the coordinate of the start point to the right-hand side was obtained, as shown in Figure 14(f).
(a)
21 (b)
(c)
(d)
(e)
22 (f)
Figure 14: Unwrapped image normalization procedure: (a) unwrapped image, (b) morphological image, (c) binary image, (d) labeled image where the blob with index one is regarded as the initial root of the thread and its corresponding right-bottom coordinate (x*, y*) is the start point of the tapping process, (e) division of the unwrapped image into left and right sub-images based on the (x*, y*) coordinate, and (f) normalized image generated from rounding the coordinate of the start point in (e) to the right-hand side
3.3.2 Normalized image segmentation
Note that the first two and the last thread patterns of an internal thread can be ignored because the beginning and ending stages of the tapping process form relatively unstable and abnormal patterns. Moreover, these thread patterns are not important in the interaction with an external thread for creating a firm fastening. Here, only the remaining thread patterns, named the inspected image, in the normalized image are focused and an automatic segmentation process can be developed. The normalized image segmentation procedure is explained below and illustrated in Figure 15.
The inner surface of an internal thread comprises repetitive and periodic thread patterns.
A scan along the line that extends from the top to the bottom in the vertical direction of (x*, y*) in Figure 14(d) will touch on the initial root first, followed by the initial crest, followed by the second root and then the second crest, etc. This is the ideal case, however, because some noise blobs may interrupt the regularity. Thus an efficient sub-operation for eliminating the
23 noise blobs is necessary.
Based on the results of Figure 14(d), a binary image where each blob has a unique labeling index was obtained. However, there are still some noise blobs in Figure 14(d). To eliminate them, two properties of each blob: the area and the angle were extracted. The areas of blobs were conveyed into the 1D and two groups clustering formula of {Liu and Tsai [39]}
to find out the representatives of noises and crests/roots. The blobs with areas that were closer to the smaller area representative were regarded as ignoring areas and could be removed.
Then the mode of the angles of the remaining blobs was estimated. Each angle was rounded to the second decimal place. The remaining blobs with angles that were distinct with the mode were regarded as blobs with irregular angle and could be removed. In sum, the noise blobs with a small area or irregular angle relative to the angle of the crests and roots are eliminated.
When the blob elimination sub-operation was applied on Figure 14(d), a clear labeled image can be obtained as shown in Figure 15(a). Then by scanning along a line that stretches from the top to bottom of the image in the vertical direction of (x*, y*), the top coordinates of the third- and second-last roots could be recorded. Based on the coordinates of the third- and second-last roots, upper and lower segmented bounds along the horizontal direction, as shown in Figure 15(b) could be determined. Finally, the normalized image is trimmed based on these two segmented bounds, as shown in Figure 15(c), and the corresponding inspected image of size m × n is auto-segmented, as shown in Figure 15(d).
(a)
24 (b)
(c)
(d)
Figure 15: Normalized image segmentation procedure: (a) start with a clear labeled image where the noise blobs have been eliminated, (b) find the third and second-last roots, (c) map two segmented bound onto the normalized image of Figure 14(f), and (d) obtain final image to be inspected
3.3.3 Thread pattern blurring
Up to the present, a set of directional textures in the image was obtained and can be inspected. The DCT-based image restoration technique is well-suited for detecting defects in directional textures. Intuitively, the dominating direction of the thread pattern in the inspected
25
image will correspond to orthogonal straight lines throughout the origin of the DCT spectrum.
The lines associated with high-energy frequency components in the spectral domain are eliminated by reducing them to zero and transforming back to the spatial domain. The procedure will blur all thread patterns and preserve only local defects if they are initially embedded in the inspected image.
3.3.3.1 Inspected image processing by discrete cosine transformation
As shown in Figure 15(d), the thread patterns of the inspected image appear as a type of periodic directional texture. The periodically occurring thread patterns were first characterized according to their frequency components. Several DCT variants have been proposed. These were categorized by Wang into four slightly different transformations: DCT-I, DCT-II, DCT-III, and DCT-IV [40]. The DCT-II was used in this research because of its ability to process images with uneven boundaries. Let f(x, y) be the gray level of the pixel at (x, y) in the inspected image of size m × n. The discrete 2D DCT is,
( ) ( ) ( ) ( ) ( ) ( )
As shown in Figure 16(a), the global thread patterns are easily distinguishable as a concentration of high-energy lines in the spectrum that are orthogonal to the direction of thread pattern in Figure 15(d).
26
The forward DCT was then applied on Figure 15(d) and the spectra was obtained as shown in Figure 16(a); Figure 16(b) showed the corresponding 3D energy plots. The dominant directions in Figure 15(d) were compacted to orthogonal straight lines through the direct current (DC) component, as shown in the corresponding Figures 16(a) or 16(b). In addition, Figures 16(a) or 16(b) clearly showed that the high-energy frequency components are packed around the top left regions. These are all inherent characteristics of the DCT basis function.
3.3.3.2 High-energy frequency components elimination
Since thread patterns and scratch defects are oriented in the same direction in the inspected image, the thread patterns are mixed together with the orthogonal lines in the spectrum. Since orthogonal lines may be due to both thread patterns and scratches, using a band-rejected filter to eliminate the orthogonal is not good approach. In this dissertation, the wide dynamic range of C(u, v) was first mapped into the narrow range of P(u, v) by a logarithmic transformation, and scaled its intensity into an eight-bit gray level using
( )
,[
log(
1 C( )
, 2) ]
Pu v =S + u v (4) where S(·) is a scaling operation. Some high-energy frequency components in the spectrum image can then be determined in terms of a high-energy threshold, and are set to zero described in, where k3∈[1, 255] is a high-energy threshold. The resulting image is shown in Figure 16(c) and the corresponding 3D energy plot is shown in Figure 16(d).
3.3.3.3 Image restoration using inverse discrete cosine transformation (IDCT)
After eliminating the specific high-energy frequency components, the spectrum image was back transformed into the spatial domain using the IDCT,
27 Figure 16(e), it is observed that the homogenous thread patterns have been blurred and the associated gray levels have been compressed into a uniform and limited range. Conversely, the gray levels of scratches on the flank in Figure 15(d) have been retained, as shown in Figure 16(e). Meanwhile, the gray levels of collapses on the crest or flaws in the root in Figure 15(d) have been reduced, as also shown in Figure 16(e).
3.3.4 Defect extraction
Since the scratches are relatively brighter and the collapses or flaws are relatively darker than the blurred thread patterns in the restored image, as shown in Figure 16(e), the statistical process control (SPC) binarization method [32] could be used to set the upper and lower control limits for determining defects from the uniform thread patterns. The SPC binarization method can be described by,
( ) ( )
respectively, of the gray level in the restored image. If a pixel has a gray level that falls between the upper and the lower limits, it is shown as white and is considered to be a thread element that should be removed. Otherwise, it is shown as black and is considered to be a defective element that should be preserved. The result of applying the SPC binarization method on Figure 16(e) is shown in Figure 16(f). Finally, the connected component labeling operator with 8-adjacent is implied again which will have the noise blobs with small areas be removed
28 (a)
(b)
(c)
(d)
29 (e)
(f)
Figure 16: Thread pattern blurring and binarization: (a) the spectra of Figure 15(d), (b) the 3D energy plots of Figure 16(a), (c) the filtered spectra of Figure 16(a) when k3 = 55, (d) the 3D energy plots of Figure 16(c), (e) the restored images, (f) the binarized image
30
4 Experiments and Discussion
In subsection 4.1, the repeatability property of the normalized image was described. A sample internal thread was placed at three arbitrary orientations to test the repeatability of the normalization. In subsection 4.2, some preliminary experiments to evaluate the impact of different values of these parameters were described. In subsection 4.3, ten samples of good and defective internal threads are used to demonstrate the inspection results of the proposed OTPG. Forty-four samples are used to evaluate effectiveness and robustness of the proposed OTPG.
4.1 Repeatability property of the normalized image
To demonstrate the repeatability property of the normalized image, the user can place the internal thread on the fixture at any arbitrary orientation. The original sequence of images is reconstructed by rounding the start point of the tapping process of the unwrapped image to the right-hand side. As shown in Figures 17(a)-(c), three unwrapped images were reconstructed in arbitrary orientations. The three images differed only in their relative position. Applying the proposed image normalization procedure normalized all these images in the same manner, as shown in Figure 17(d). This shows that our normalization procedure can successfully normalize the unwrapped image so that the start point of the tapping process is always on the right-hand side. Such image position normalization also provides a uniform inspected image for defect detection.
31 (a)
(b)
(c)
(d)
Figure 17: Demonstration of repeatability: (a) first unwrapped image, (b) second unwrapped image, (c) third unwrapped image, and (d) corresponding normalized image
32 4.2 Sensitivity analysis of parameter settings
As implemented the OTPG algorithm described in section 3.3 has four major parameters that influence the inspection outcome: the structure element size of grayscale closing operator k1, the offset constant k2, the high-energy threshold k3, and the control constant k4.
4.2.1. Effects of the structure element size of grayscale closing operator k1 and the offset constant k2
The parameter k1 is the rectangular size of the structure element of grayscale closing operator that can mend the interrupted crests or roots in the internal thread image. The parameter k2 is an offset constant that can adjust the binary threshold value in Equation (1) to enable the user to reveal the pixels with gray levels between max(G) – k2 and max(G) in Figure 14(a). The bright bands (crests or roots) and gray bands (flanks) in the morphological image of the internal thread can be well mended and totally separated if both k1 and k2 are selected properly. That permits the analysis of the normalization and segmentation procedures in subsections 3.3.1 and 3.3.2 to proceed successfully. As shown in Figures 18(a)-(c), as the value of k1 or k2 increases, there are more and more crests and roots will join into the connected blobs. On the contrary, as the value of k1 or k2 decreases, neither the crest nor the root will be well shaped or well mended. Those blobs will appear as false crests or roots so that the image of the internal thread will be ambiguous and complicated for later processing.
(a)
33 (b)
(c)
Figure 18: Effect of different offset constants: (a) k1 = 10 and k2 = 7,(b) k1 = 50 and k2 = 11, and (c) k1 = 100 and k2 = 15
To obtain reliable values of k1 and k2, a supervised pre-training session was conducted on 22 samples. Figure 19 and Figure 20 showed the curves of the average area of blobs and the average number of blobs after the blob elimination sub-operation described in subsection 3.3.2 was applied for different combinations of k1 and k2. The curves of different k1 in Figure 19 were dramatically inflating when the values of k2 were larger than 25. Meanwhile, in Figure 20, these curves were decreasing when the values of k2 were larger than 14 due to the bright and gray bands have gradually joined together. The intersection of the intervals, (k2
≧ 25) ∩ (k2 ≧ 14), was took the complement which yielded an interested region, k2 < 14, to be following discussed. When k1 was smaller than 5, the average area of blobs were relative low and the averages number of blobs were relative high due to interrupted crests or
34
roots were not well mended and then were partial eliminated. When k1 was in the range of 7-11, the average area of blobs and the average number of blobs were relative stable due to the crests and roots can be well mended and separated from the flanks reliably. When k1 was larger than 13, the average area of blobs were relative high and the average number of blobs were relative low due to some crests or roots have joined together by the large structure element sizes that some blobs will be misinterpreted as false crests, roots, or flanks. In this way, we could get the stable intervals of k1 in the range of 7-11 and k2 less than 14.
Figure 19: Average area of blobs generated for different values of k1 and k2
Figure 19: Average area of blobs generated for different values of k1 and k2