Some promising new techniques for non-contact internal thread inspection have been developed in recent two decades. Four types of test are briefly described below.
(1) Reflected light test {Hassel [1, 2], Gore [3], Tu et al. [4], Zhao et al. [5], Zhao and Liao [6, 7]}: As show in Figure 4, the mechanism is equipped with a transmitter/receiver probe arranged perpendicular to the thread surface. The transmitter sends a light beam towards the thread, and the reflection goes back to the receiver. The reflectivity is used as a quantifiable feature for defect determination. Field [8] summarized two drawbacks of the reflected light test. First, when thread has been polluted by fluid or oil, varying levels of reflectivity can degrade the detection capability. Second, this technique senses only a small portion of the threaded hole due to the low scanning speed.
Figure 4: Reflected light test system [7]
(2) Magnetic flux leakage test {Wang et al. [9], He et al. [10]}: As shown in Figure 5, a magnetizer magnetizes a part of the internal thread to generate a magnetic leakage field. A detector detects the magnetic leakage field and transforms it into defect signals that can be used to identify defects in threads and mark their positions and ranges. The magnetic flux
7
leakage test, however, can only be used on metal parts.
Figure 5: Magnetic flux leakage test system [9]
(3) Eddy current test {Lin et al. [11]}: As shown in Figure 6, a probe generating an eddy current is inserted into the part with the internal thread. A sensor measures the response signals, which give only a rough indication of the screw defects and positions.
Nevertheless, the eddy current test is sensitive to the material's microstructure, hardness, chemistry, temperature, and geometry.
Figure 6: Eddy current test system [11]
8
(4) Industrial computed tomography test {Liu et al. [12]}: As shown in Figure 7, the tomography of a part is retrieved from multiple directions and the system uses image processing to detect inner faults and measure their geometrical size. As with the reflected light test, this system only scans a small portion of the thread. Moreover, the computed tomography equipment is very expensive.
Figure 7: Industrial computed tomography test system (depicted through the descriptions of [12])
9 2.2 Applications of rigid industrial endoscope
It is impossible for an ordinary detector to look inside small-diameter holes to evaluate internal conditions. George Sumner Crampton developed the first rigid endoscope in 1921 to check for possible flaws inside the rotor of a steam turbine {Lang [13]}. The distinct characteristics of the endoscope are that it can extend the range of vision and change the angle of view to see what could not otherwise be seen. Modern endoscopes are complex, vital non-contact inspection instruments that include an optical-mechatronic sensor and an auto-control scheme. In the past two decades, endoscopes have been used in a wide range of industrial applications. Parenti et al. [14] used them to analyze the combustion in a burner.
Tsushima et al. [15] employed them to inspect the inner walls of steel tubes for corrosion.
Boudjahi et al. [16] developed an integrated system for detecting micorcacks in pipes. Gu [17]
used a video-endoscope in civil aviation maintenance. Biegelbauer et al. [18] proposed a surface reconstruction system for bore holes using an endoscope. Bondarev [19] and Ahn et al.
[20, 21] described numerous endoscope applications in the petroleum and gas industry. These papers form a background for using the rigid endoscope in internal thread inspection.
10
2.3 Surface registration from a sequence of 2D images
Image registration is essential to evaluating the global surface texture. Registration aligns two or more images from different sensors or viewpoints. Some studies have used a sequence of 2D images to reconstruct the surface of an object for specific applications such as:
(1) Reconstructing a large object surface: printed circuit board (PCB) {Perng et al. [22]}, cathode ray tube (CRT) panel {Perng et al. [23]}, organic light-emitting diode (OLED) panel {Perng et al. [24]}, or thin film transistor liquid crystal display (TFT-LCD) panel {Chen and Kuo [25]};
(2) Reconstructing a non-flat object surface: bore hole {Biegelbauer et al. [18]} or router {Perng and Chen [26]};
(3) Reconstructing an object surface with high resolution: integrated circuit (IC) chip {Perng et al. [27]}.
Registration techniques have been developed for many different types of problems. In general, the alignment methods can be separated into two categories according to whether two or more aligned images overlap or not. Both categories require close coordination of the sensor and an associated motion unit. For the non-overlapping method, the region of interest (ROI) must exist in two successive but non-overlapping images {Biegelbauer et al. [18], Perng et al. [24]}. For the overlapping method, the ROI must exist in two successive images that overlap by a specified percentage. The users must predefine the overlapping region in the first image as a template and then apply the pattern matching algorithm to the neighboring image {Lewis [28], Fitch et al. [29]}. In the matching process, the predefined template will slide over the entire target image on a pixel-by-pixel or sub-pixel-by-sub-pixel basis so that the maximum matching score can be found and the corresponding alignment coordinate can be determined {Perng et al. [22, 23, 27]}. Although the registration image of the former method may be rougher than the latter one, it is highly computationally efficient and so was used in this dissertation.
11 2.4 Surface inspection for directional textures
A directional textured surface, such as machined part, semiconductor, natural wood, fabric textile, etc. is an object surface which composes of a set of line primitives in some regular or repetitive arrangement over an entire appearance. Detecting local defects embedded in a directional texture surface is one of the popular researches of computer vision. Numerous approaches to auto-inspect the directional textured surface have been proposed, including statistical, structural, global, and model-based approaches {Kumar [30], Xie [31]}. The global approaches are based on image restoration procedure such as using discrete Fourier transform (DFT) {Tsai and Hsieh [32]}, discrete cosine transform (DCT) {Chen and Perng [33]}, discrete wavelet transform (DWT) {Tsai and Chiang [34]}, singular value decomposition (SVD) {Lu and Tsai [35]}, principal component analysis (PCA) {Perng and Chen [36]}, and independent component analysis (ICA) {Lu and Tsai [37]}. Each of these approaches may be a good choice for inspecting directional textured surfaces. Because these approaches require neither textural features nor any reference image for comparison, they are immune to the limitations inherent in local feature extraction or golden template matching methods.
In general, global approaches to inspecting defects in directional textured surfaces usually start with a forward transform and filtering, followed by an inverse transform and thresholding {Kumar [30]}. These approaches all involve implicit qualitative inspection algorithms {Newman and Jain [38]}. Figure 8 shows a flowchart of such a global approach.
The input image in the spatial domain is first converted to the transform domain where textures exhibit significant high-energy characteristics that can be detected by some pre-defined criteria. After those specific high-energy components have been suppressed in the transform domain, the inverse transform restores the image to the spatial domain. These global approaches thus preserve only the local defects that existed in the original input image, and remove all directional textures.
12
Figure 8: Flowchart of global approach to detecting defects in directional textures
Detecting defects on the directional textured surfaces is merely one of the basic capabilities. It is prominent, but insufficient, to be discharged in our complex world. So other auxiliary capabilities should be also taken into consideration. Based on the experimental results of existing references [32-37] and experiences, some curial features are summarized in Table 2 to compare the auxiliary abilities of the existing approaches.
As shown in Table 2, the ICA-based approach additionally need golden template and can neither indicate the defect location nor preserve the defect shape. It is also the least adaptive one to tackle the unexceptional evens. The DWT-based approach is good, except it cannot detect the defects that parallel to the texture. Both the texture and such parallel type of defect maps into similar wavelet bank(s) and then be suppressed which usually yields a missed-detection. The remaining approaches, DFT-based, DCT-based, SVD-based, and PCA-based, are relatively outstanding. They can provide with all of the mentioned auxiliary capabilities. Among of them, ICA-based approach needs less memory locations and shows more efficient in computation due to it all deals with real number and has fast DCT algorithm.
Based on this reason, an image restoration process based on DCT was used in this dissertation.
13
Table 2 Ability comparison of the state-of-the-art global directional textured surface defect detectors
DFT DCT DWT SVD PCA ICA
Indicate the defect location ○ ○ ○ ○ ○ X
Preserve the defect shape ○ ○ ○ ○ ○ X
Detect the defects that parallel to the texture ○ ○ X ○ ○ ○
Template free ○ ○ ○ ○ ○ X
Shift invariance ○ ○ ○ ○ ○ X
Rotation invariance ○ ○ ○ ○ ○ X
Illumination invariance ○ ○ ○ ○ ○ ○
Suit for line scan system X X X X X ○
Note: “○” represents “Yes”; “X” represents “No”.
14
3 Research Method
3.1 OTPG hardware
The hardware system for internal thread extraction is shown in Figure 9. The sequences of internal thread images were captured by a TELI CS8320 black and white camera with a resolution of 640 × 480 and a Matrox Meteor II frame grabber. A CCD with an illumination less than 0.4 lux is recommended for this task. A 7-in Hawkeye Slim rigid industrial endoscope connected to a 90° side-view mirror tube (as shown in Figure 10) was used as the lens. The outside diameter of the endoscope was only 0.20-in and included a compact illumination fiber. The Moritex MHF-G150LR halogen light source supplied white light to the endoscope to enable the CCD to receive clear images in dark cavities. To overcome the line of sight limitation of the 90° side-view adaptor, the mechanism included a rotational servo motor (SM3416D_PLS) and a linear actuator (SmartT integrated module) to observe all the surfaces of the internal thread at different successive angles and depths. The apparatus was connected to a computer. A flowchart of the proposed vision inspection method is given in Figure 11.
The details of this method are described below.
15
Figure 9: Proposed OTPG hardware
Figure 10: The 90° side-view adaptor (from http://www.gradientlens.com/)
16
Image processing by cosine transformation
Are there pixels out of upper control limit? Grade C: collapse or flaw of defect , if N1 = 0 and N2 = 1 Grade D: mixed type defects , if N1 = 1 and N2 = 1
Are there pixels out of lower control limit?
Yes N2 = 1 No
Figure 11: Flowchart of the proposed OTPG algorithm
17 3.2 Registration of the 2D unwrapped image
As shown in Figure 12, the inner surface of internal thread can be fully observed by the proposed OTPG. Even so, image distortion and non-uniform illumination will occur due to the inherent endoscope structure and the cylindrical geometry of the internal threads. The farther away the pixels are from the center of the captured image, the more significant the above phenomena will be. Thus, only the region with little distortion near the center of each image for various angles and depths are used. With appropriate control of the sensor and the associated motion unit, a sequence of 150 × 150 pixel low-distortion images by rotating the fixture in 15° steps could be extracted. The ROI of two successive images under these conditions is restricted to only two non-overlapping neighbors. After the fixture has rotated 360°, the linear actuator raises the fixture to the next level and the rotation is repeated. The procedure continues until images of all the inner surfaces of the internal thread have been captured. Finally, the captured sequence of low-distortion images is used to reconstruct a 2D unwrapped image with size 3600 × 1500. It needs 24 × 10 = 240 sub-images to completely registrate one internal thread with 15.3mm in diameter and 20mm in length. Figure 13 shows the reconstructed 2D unwrapped image of an internal thread obtained by the described approach. In this figure, the crest and root of the internal thread correspond to the wide and narrow white bands, respectively. The flanks of the internal thread appear as gray bands. A complete thread pattern is composed of a wide white band, a gray band, a narrow white band, and another gray band in that order. The inner surface of an internal thread is indeed a type of directional texture that comprises repetitive and periodic thread patterns.
18
Figure 12: A partial wall image of internal thread
Figure 13: Reconstructed 2D unwrapped image of an internal thread
19 3.3 OTPG algorithm
The most common defects in internal threads are collapses on the crest, scratches on the flank, and flaws in the root, as shown in Figure 14. A scratch usually appears as a bulge that will cause an internal thread to bind with an external one, while a collapse or flaw appears as a cavity that will decrease the tight fit of a thread. This section focuses on developing an auto-inspection software, OTPG algorithm, to detect those defects that are embedded in homogeneous thread patterns. An implicit qualitative inspection algorithm is used to detect the embedded defects. The OTPG algorithm includes four major operations: unwrapped image normalization, normalized image segmentation, thread pattern blurring, and defect extraction. These are discussed below.
3.3.1 Unwrapped image normalization
Because the internal thread is at some arbitrary orientation in relation to the OTPG fixture during defect inspection, the start point of the tapping process in the reconstructed unwrapped image will appear at some random location. To ensure that the relative position of the global structure of each unwrapped image coincide, a process that can automatically reorient the start point of the tapping process in the unwrapped image was developed so it is always on the right-hand side of the image. The procedure of normalizing the unwrapped image is described below and illustrated in Figure 14.
The key to normalizing an unwrapped image is to find the start point of the tapping process of the internal thread and place it on the right-hand side of the image. To do this requires a good binary image where the foreground is composed of white bands (crests and roots) and the background is composed of gray bands (flanks). In addition, to locate the initial root in the binary image is necessary. The initial root is generated early in the thread tapping process. Therefore, if the intensity of each pixel of the binary image was tracked one-by-one, scanning from left to right and top to bottom, the frontier foreground element and the corresponding eight-connected elements must be the initial root.
20
Figure 14(a) shows an initial reconstructed unwrapped image of an internal thread. The grayscale closing operator with structure element size k1 × k1 was first applied to fix the interspaces and fill up the holes; the result is shown in Figure 14(b). Then the grayscale image of Figure 14(b) was converted to a binary image using a threshold value calculated with Equation (1) to separate the crests and roots from the flanks,
( )
k2 maxvalue
threshold = G − (1) where G is the universal set of gray values of Figure 14(b) and k2∈[1, max(G) – 1] is an offset constant. This produces the binary image of Figure 14(c). In Figure 14(c), each eight-connected foreground element can be regarded as a blob and apply a row-by-row labeling algorithm from left to right and top to bottom. The row-by-row labeling algorithm is guaranteed to find the initial root of an internal thread because it is the first one to be produced in the tapping process. The blob of the initial root is labeled as the index one and its corresponding right-bottom coordinate (x*, y*), the start point of the tapping process, is recorded as shown in Figure 14(d). Then the coordinate (x*, y*) was mapped onto the unwrapped image of Figure 14(a) and this image was divided into left and right sub-images, as shown in Figure 14(e), based on the x* coordinate. The unwrapped image can be rounded arbitrarily due to the intrinsic cylindrical structure of the internal thread. A normalized image by rounding the coordinate of the start point to the right-hand side was obtained, as shown in Figure 14(f).
(a)
21 (b)
(c)
(d)
(e)
22 (f)
Figure 14: Unwrapped image normalization procedure: (a) unwrapped image, (b) morphological image, (c) binary image, (d) labeled image where the blob with index one is regarded as the initial root of the thread and its corresponding right-bottom coordinate (x*, y*) is the start point of the tapping process, (e) division of the unwrapped image into left and right sub-images based on the (x*, y*) coordinate, and (f) normalized image generated from rounding the coordinate of the start point in (e) to the right-hand side
3.3.2 Normalized image segmentation
Note that the first two and the last thread patterns of an internal thread can be ignored because the beginning and ending stages of the tapping process form relatively unstable and abnormal patterns. Moreover, these thread patterns are not important in the interaction with an external thread for creating a firm fastening. Here, only the remaining thread patterns, named the inspected image, in the normalized image are focused and an automatic segmentation process can be developed. The normalized image segmentation procedure is explained below and illustrated in Figure 15.
The inner surface of an internal thread comprises repetitive and periodic thread patterns.
A scan along the line that extends from the top to the bottom in the vertical direction of (x*, y*) in Figure 14(d) will touch on the initial root first, followed by the initial crest, followed by the second root and then the second crest, etc. This is the ideal case, however, because some noise blobs may interrupt the regularity. Thus an efficient sub-operation for eliminating the
23 noise blobs is necessary.
Based on the results of Figure 14(d), a binary image where each blob has a unique labeling index was obtained. However, there are still some noise blobs in Figure 14(d). To eliminate them, two properties of each blob: the area and the angle were extracted. The areas of blobs were conveyed into the 1D and two groups clustering formula of {Liu and Tsai [39]}
to find out the representatives of noises and crests/roots. The blobs with areas that were closer to the smaller area representative were regarded as ignoring areas and could be removed.
Then the mode of the angles of the remaining blobs was estimated. Each angle was rounded to the second decimal place. The remaining blobs with angles that were distinct with the mode were regarded as blobs with irregular angle and could be removed. In sum, the noise blobs with a small area or irregular angle relative to the angle of the crests and roots are eliminated.
When the blob elimination sub-operation was applied on Figure 14(d), a clear labeled image can be obtained as shown in Figure 15(a). Then by scanning along a line that stretches from the top to bottom of the image in the vertical direction of (x*, y*), the top coordinates of the third- and second-last roots could be recorded. Based on the coordinates of the third- and second-last roots, upper and lower segmented bounds along the horizontal direction, as shown in Figure 15(b) could be determined. Finally, the normalized image is trimmed based on these two segmented bounds, as shown in Figure 15(c), and the corresponding inspected image of size m × n is auto-segmented, as shown in Figure 15(d).
(a)
24 (b)
(c)
(d)
Figure 15: Normalized image segmentation procedure: (a) start with a clear labeled image where the noise blobs have been eliminated, (b) find the third and second-last roots, (c) map two segmented bound onto the normalized image of Figure 14(f), and (d) obtain final image to be inspected
3.3.3 Thread pattern blurring
Up to the present, a set of directional textures in the image was obtained and can be inspected. The DCT-based image restoration technique is well-suited for detecting defects in directional textures. Intuitively, the dominating direction of the thread pattern in the inspected
25
image will correspond to orthogonal straight lines throughout the origin of the DCT spectrum.
The lines associated with high-energy frequency components in the spectral domain are eliminated by reducing them to zero and transforming back to the spatial domain. The procedure will blur all thread patterns and preserve only local defects if they are initially embedded in the inspected image.
3.3.3.1 Inspected image processing by discrete cosine transformation
3.3.3.1 Inspected image processing by discrete cosine transformation