1 Introduction
1.1 Introduction
An internal thread is a spiral screw pattern cut into the inner surface of a hollow cylinder.
Figure 1 shows a sectional view of the basic internal thread geometry formed by a sequence of thread patterns. A thread pattern is composed of a crest, a root, and two flanks. The crest, the most prominent part of the thread pattern, is the plateau between the two slanted surfaces.
The root is the bottom of the groove between the two slanted surfaces. The flanks are the slanted sides that connect the crest and root; these may be straight or curved.
Figure 1: Characteristic parts of internal thread geometry
The manufacture procedure of an internal thread is to tap the hole by turning the screw tap half turn clockwise and following by quarter turn anti-clockwise till the tapping process is finished. This tapping process allows the material filings to be released from the threads and retains the cutting edges of the screw tap free so as to make taping much easier. However, to keep a high production rate, producers abandon the above tapping steps but only turn clockwise to the end un-intermittently. While in the new tapping process, a screw tap may embed with material filings that shape nonconforming features around the thread with high-risk such as collapses on the crest, scratches on the flank, or flaws on the root. A scratch
2
on the flank is a significant bulge that can cause an internal thread to bind with an external one. A collapse on the crest or a flaw deep inside the root will decrease the ability of an internal thread to provide a tight fit. A comparison about the defects of internal thread was tabulated in Table 1. As shown in Figure 2, the internal threads in nuts have a wide variety of sizes, depths, and shapes. Due to the very nature of machine nuts and limited space, visualizing inspecting the crucial features of an internal thread is not a trivial task. Even the inspectors look on sideway; they may not immediately observe the entire pattern yet.
Accordingly, some implicit blemishes may be disregarded and it is difficult to guarantee the quality of an internal thread.
Table 1: The defects of the internal thread Defect type Location Geometrical shape Inference
Scratch Flank Bulge-shaped To cause an internal thread to bind with an external one
Collapse Crest Flaw Root
Cave-shaped To decrease the ability of an internal thread and can not provide a tight fit
Figure 2: Various types of internal threads (from http://www.nuts.com.tw/)
A thread plug gauge as shown in Figure 3 is usually used to check the functionality of an internal thread by screwing the gauge into every thread manually. However, the screwing operation is labor-intensive, time-consuming, and subject to interference. Furthermore, a
3
contact gauge cannot detect certain defects such as collapses or flaws. A non-contact inspection approach is required for the mass production of internally threaded components for the automobile, shipbuilding, and aerospace industries where high precision is required.
Figure 3: The thread plug gauge
4 1.2 Objectives
Machine vision has rarely been used in the inspection of internal threads due to the unique structure and limited space involved. This dissertation is to develop a novel internal thread inspection mechanism that relies on machine vision to create an optical thread plug gauge (OTPG) capable of overcoming the inherent dimensional limitations of internal threads.
The proposed OTPG used a rigid industrial endoscope, a CCD camera, and a two degree-of-freedom motion control unit to capture a sequence of partial wall images of an internal thread without any mechanical contact. It also used a 2D image registration method to reconstruct the unwrapped images. Then the digital image processing and discrete cosine transformation-based image restoration scheme are used to normalize image, segment image, and enhance defect of the internal thread.
5 1.3 Organization of the dissertation
This dissertation is organized as follows. In section 2, some related researches, such as the non-contact internal thread inspection systems, endoscope applications, image registration techniques, and the state-of-the-art global directional textures inspectors, are reviewed. In section 3, the hardware structure and the software algorithm of the proposed automated optical inspection (AOI) system are described. In section 4, the experimental environment and results were described. The conclusions and further suggestions are given in section 5.
6
2 Literatures Review
2.1 Non-contact internal thread inspection systems
Some promising new techniques for non-contact internal thread inspection have been developed in recent two decades. Four types of test are briefly described below.
(1) Reflected light test {Hassel [1, 2], Gore [3], Tu et al. [4], Zhao et al. [5], Zhao and Liao [6, 7]}: As show in Figure 4, the mechanism is equipped with a transmitter/receiver probe arranged perpendicular to the thread surface. The transmitter sends a light beam towards the thread, and the reflection goes back to the receiver. The reflectivity is used as a quantifiable feature for defect determination. Field [8] summarized two drawbacks of the reflected light test. First, when thread has been polluted by fluid or oil, varying levels of reflectivity can degrade the detection capability. Second, this technique senses only a small portion of the threaded hole due to the low scanning speed.
Figure 4: Reflected light test system [7]
(2) Magnetic flux leakage test {Wang et al. [9], He et al. [10]}: As shown in Figure 5, a magnetizer magnetizes a part of the internal thread to generate a magnetic leakage field. A detector detects the magnetic leakage field and transforms it into defect signals that can be used to identify defects in threads and mark their positions and ranges. The magnetic flux
7
leakage test, however, can only be used on metal parts.
Figure 5: Magnetic flux leakage test system [9]
(3) Eddy current test {Lin et al. [11]}: As shown in Figure 6, a probe generating an eddy current is inserted into the part with the internal thread. A sensor measures the response signals, which give only a rough indication of the screw defects and positions.
Nevertheless, the eddy current test is sensitive to the material's microstructure, hardness, chemistry, temperature, and geometry.
Figure 6: Eddy current test system [11]
8
(4) Industrial computed tomography test {Liu et al. [12]}: As shown in Figure 7, the tomography of a part is retrieved from multiple directions and the system uses image processing to detect inner faults and measure their geometrical size. As with the reflected light test, this system only scans a small portion of the thread. Moreover, the computed tomography equipment is very expensive.
Figure 7: Industrial computed tomography test system (depicted through the descriptions of [12])
9 2.2 Applications of rigid industrial endoscope
It is impossible for an ordinary detector to look inside small-diameter holes to evaluate internal conditions. George Sumner Crampton developed the first rigid endoscope in 1921 to check for possible flaws inside the rotor of a steam turbine {Lang [13]}. The distinct characteristics of the endoscope are that it can extend the range of vision and change the angle of view to see what could not otherwise be seen. Modern endoscopes are complex, vital non-contact inspection instruments that include an optical-mechatronic sensor and an auto-control scheme. In the past two decades, endoscopes have been used in a wide range of industrial applications. Parenti et al. [14] used them to analyze the combustion in a burner.
Tsushima et al. [15] employed them to inspect the inner walls of steel tubes for corrosion.
Boudjahi et al. [16] developed an integrated system for detecting micorcacks in pipes. Gu [17]
used a video-endoscope in civil aviation maintenance. Biegelbauer et al. [18] proposed a surface reconstruction system for bore holes using an endoscope. Bondarev [19] and Ahn et al.
[20, 21] described numerous endoscope applications in the petroleum and gas industry. These papers form a background for using the rigid endoscope in internal thread inspection.
10
2.3 Surface registration from a sequence of 2D images
Image registration is essential to evaluating the global surface texture. Registration aligns two or more images from different sensors or viewpoints. Some studies have used a sequence of 2D images to reconstruct the surface of an object for specific applications such as:
(1) Reconstructing a large object surface: printed circuit board (PCB) {Perng et al. [22]}, cathode ray tube (CRT) panel {Perng et al. [23]}, organic light-emitting diode (OLED) panel {Perng et al. [24]}, or thin film transistor liquid crystal display (TFT-LCD) panel {Chen and Kuo [25]};
(2) Reconstructing a non-flat object surface: bore hole {Biegelbauer et al. [18]} or router {Perng and Chen [26]};
(3) Reconstructing an object surface with high resolution: integrated circuit (IC) chip {Perng et al. [27]}.
Registration techniques have been developed for many different types of problems. In general, the alignment methods can be separated into two categories according to whether two or more aligned images overlap or not. Both categories require close coordination of the sensor and an associated motion unit. For the non-overlapping method, the region of interest (ROI) must exist in two successive but non-overlapping images {Biegelbauer et al. [18], Perng et al. [24]}. For the overlapping method, the ROI must exist in two successive images that overlap by a specified percentage. The users must predefine the overlapping region in the first image as a template and then apply the pattern matching algorithm to the neighboring image {Lewis [28], Fitch et al. [29]}. In the matching process, the predefined template will slide over the entire target image on a pixel-by-pixel or sub-pixel-by-sub-pixel basis so that the maximum matching score can be found and the corresponding alignment coordinate can be determined {Perng et al. [22, 23, 27]}. Although the registration image of the former method may be rougher than the latter one, it is highly computationally efficient and so was used in this dissertation.
11 2.4 Surface inspection for directional textures
A directional textured surface, such as machined part, semiconductor, natural wood, fabric textile, etc. is an object surface which composes of a set of line primitives in some regular or repetitive arrangement over an entire appearance. Detecting local defects embedded in a directional texture surface is one of the popular researches of computer vision. Numerous approaches to auto-inspect the directional textured surface have been proposed, including statistical, structural, global, and model-based approaches {Kumar [30], Xie [31]}. The global approaches are based on image restoration procedure such as using discrete Fourier transform (DFT) {Tsai and Hsieh [32]}, discrete cosine transform (DCT) {Chen and Perng [33]}, discrete wavelet transform (DWT) {Tsai and Chiang [34]}, singular value decomposition (SVD) {Lu and Tsai [35]}, principal component analysis (PCA) {Perng and Chen [36]}, and independent component analysis (ICA) {Lu and Tsai [37]}. Each of these approaches may be a good choice for inspecting directional textured surfaces. Because these approaches require neither textural features nor any reference image for comparison, they are immune to the limitations inherent in local feature extraction or golden template matching methods.
In general, global approaches to inspecting defects in directional textured surfaces usually start with a forward transform and filtering, followed by an inverse transform and thresholding {Kumar [30]}. These approaches all involve implicit qualitative inspection algorithms {Newman and Jain [38]}. Figure 8 shows a flowchart of such a global approach.
The input image in the spatial domain is first converted to the transform domain where textures exhibit significant high-energy characteristics that can be detected by some pre-defined criteria. After those specific high-energy components have been suppressed in the transform domain, the inverse transform restores the image to the spatial domain. These global approaches thus preserve only the local defects that existed in the original input image, and remove all directional textures.
12
Figure 8: Flowchart of global approach to detecting defects in directional textures
Detecting defects on the directional textured surfaces is merely one of the basic capabilities. It is prominent, but insufficient, to be discharged in our complex world. So other auxiliary capabilities should be also taken into consideration. Based on the experimental results of existing references [32-37] and experiences, some curial features are summarized in Table 2 to compare the auxiliary abilities of the existing approaches.
As shown in Table 2, the ICA-based approach additionally need golden template and can neither indicate the defect location nor preserve the defect shape. It is also the least adaptive one to tackle the unexceptional evens. The DWT-based approach is good, except it cannot detect the defects that parallel to the texture. Both the texture and such parallel type of defect maps into similar wavelet bank(s) and then be suppressed which usually yields a missed-detection. The remaining approaches, DFT-based, DCT-based, SVD-based, and PCA-based, are relatively outstanding. They can provide with all of the mentioned auxiliary capabilities. Among of them, ICA-based approach needs less memory locations and shows more efficient in computation due to it all deals with real number and has fast DCT algorithm.
Based on this reason, an image restoration process based on DCT was used in this dissertation.
13
Table 2 Ability comparison of the state-of-the-art global directional textured surface defect detectors
DFT DCT DWT SVD PCA ICA
Indicate the defect location ○ ○ ○ ○ ○ X
Preserve the defect shape ○ ○ ○ ○ ○ X
Detect the defects that parallel to the texture ○ ○ X ○ ○ ○
Template free ○ ○ ○ ○ ○ X
Shift invariance ○ ○ ○ ○ ○ X
Rotation invariance ○ ○ ○ ○ ○ X
Illumination invariance ○ ○ ○ ○ ○ ○
Suit for line scan system X X X X X ○
Note: “○” represents “Yes”; “X” represents “No”.
14
3 Research Method
3.1 OTPG hardware
The hardware system for internal thread extraction is shown in Figure 9. The sequences of internal thread images were captured by a TELI CS8320 black and white camera with a resolution of 640 × 480 and a Matrox Meteor II frame grabber. A CCD with an illumination less than 0.4 lux is recommended for this task. A 7-in Hawkeye Slim rigid industrial endoscope connected to a 90° side-view mirror tube (as shown in Figure 10) was used as the lens. The outside diameter of the endoscope was only 0.20-in and included a compact illumination fiber. The Moritex MHF-G150LR halogen light source supplied white light to the endoscope to enable the CCD to receive clear images in dark cavities. To overcome the line of sight limitation of the 90° side-view adaptor, the mechanism included a rotational servo motor (SM3416D_PLS) and a linear actuator (SmartT integrated module) to observe all the surfaces of the internal thread at different successive angles and depths. The apparatus was connected to a computer. A flowchart of the proposed vision inspection method is given in Figure 11.
The details of this method are described below.
15
Figure 9: Proposed OTPG hardware
Figure 10: The 90° side-view adaptor (from http://www.gradientlens.com/)
16
Image processing by cosine transformation
Are there pixels out of upper control limit? Grade C: collapse or flaw of defect , if N1 = 0 and N2 = 1 Grade D: mixed type defects , if N1 = 1 and N2 = 1
Are there pixels out of lower control limit?
Yes N2 = 1 No
Figure 11: Flowchart of the proposed OTPG algorithm
17 3.2 Registration of the 2D unwrapped image
As shown in Figure 12, the inner surface of internal thread can be fully observed by the proposed OTPG. Even so, image distortion and non-uniform illumination will occur due to the inherent endoscope structure and the cylindrical geometry of the internal threads. The farther away the pixels are from the center of the captured image, the more significant the above phenomena will be. Thus, only the region with little distortion near the center of each image for various angles and depths are used. With appropriate control of the sensor and the associated motion unit, a sequence of 150 × 150 pixel low-distortion images by rotating the fixture in 15° steps could be extracted. The ROI of two successive images under these conditions is restricted to only two non-overlapping neighbors. After the fixture has rotated 360°, the linear actuator raises the fixture to the next level and the rotation is repeated. The procedure continues until images of all the inner surfaces of the internal thread have been captured. Finally, the captured sequence of low-distortion images is used to reconstruct a 2D unwrapped image with size 3600 × 1500. It needs 24 × 10 = 240 sub-images to completely registrate one internal thread with 15.3mm in diameter and 20mm in length. Figure 13 shows the reconstructed 2D unwrapped image of an internal thread obtained by the described approach. In this figure, the crest and root of the internal thread correspond to the wide and narrow white bands, respectively. The flanks of the internal thread appear as gray bands. A complete thread pattern is composed of a wide white band, a gray band, a narrow white band, and another gray band in that order. The inner surface of an internal thread is indeed a type of directional texture that comprises repetitive and periodic thread patterns.
18
Figure 12: A partial wall image of internal thread
Figure 13: Reconstructed 2D unwrapped image of an internal thread
19 3.3 OTPG algorithm
The most common defects in internal threads are collapses on the crest, scratches on the flank, and flaws in the root, as shown in Figure 14. A scratch usually appears as a bulge that will cause an internal thread to bind with an external one, while a collapse or flaw appears as a cavity that will decrease the tight fit of a thread. This section focuses on developing an auto-inspection software, OTPG algorithm, to detect those defects that are embedded in homogeneous thread patterns. An implicit qualitative inspection algorithm is used to detect the embedded defects. The OTPG algorithm includes four major operations: unwrapped image normalization, normalized image segmentation, thread pattern blurring, and defect extraction. These are discussed below.
3.3.1 Unwrapped image normalization
Because the internal thread is at some arbitrary orientation in relation to the OTPG fixture during defect inspection, the start point of the tapping process in the reconstructed unwrapped image will appear at some random location. To ensure that the relative position of the global structure of each unwrapped image coincide, a process that can automatically reorient the start point of the tapping process in the unwrapped image was developed so it is always on the right-hand side of the image. The procedure of normalizing the unwrapped image is described below and illustrated in Figure 14.
The key to normalizing an unwrapped image is to find the start point of the tapping process of the internal thread and place it on the right-hand side of the image. To do this requires a good binary image where the foreground is composed of white bands (crests and roots) and the background is composed of gray bands (flanks). In addition, to locate the initial root in the binary image is necessary. The initial root is generated early in the thread tapping process. Therefore, if the intensity of each pixel of the binary image was tracked one-by-one, scanning from left to right and top to bottom, the frontier foreground element and the corresponding eight-connected elements must be the initial root.
20
Figure 14(a) shows an initial reconstructed unwrapped image of an internal thread. The grayscale closing operator with structure element size k1 × k1 was first applied to fix the interspaces and fill up the holes; the result is shown in Figure 14(b). Then the grayscale image of Figure 14(b) was converted to a binary image using a threshold value calculated with Equation (1) to separate the crests and roots from the flanks,
( )
k2 maxvalue
threshold = G − (1) where G is the universal set of gray values of Figure 14(b) and k2∈[1, max(G) – 1] is an offset constant. This produces the binary image of Figure 14(c). In Figure 14(c), each eight-connected foreground element can be regarded as a blob and apply a row-by-row labeling algorithm from left to right and top to bottom. The row-by-row labeling algorithm is guaranteed to find the initial root of an internal thread because it is the first one to be produced in the tapping process. The blob of the initial root is labeled as the index one and its corresponding right-bottom coordinate (x*, y*), the start point of the tapping process, is recorded as shown in Figure 14(d). Then the coordinate (x*, y*) was mapped onto the unwrapped image of Figure 14(a) and this image was divided into left and right sub-images, as shown in Figure 14(e), based on the x* coordinate. The unwrapped image can be rounded
threshold = G − (1) where G is the universal set of gray values of Figure 14(b) and k2∈[1, max(G) – 1] is an offset constant. This produces the binary image of Figure 14(c). In Figure 14(c), each eight-connected foreground element can be regarded as a blob and apply a row-by-row labeling algorithm from left to right and top to bottom. The row-by-row labeling algorithm is guaranteed to find the initial root of an internal thread because it is the first one to be produced in the tapping process. The blob of the initial root is labeled as the index one and its corresponding right-bottom coordinate (x*, y*), the start point of the tapping process, is recorded as shown in Figure 14(d). Then the coordinate (x*, y*) was mapped onto the unwrapped image of Figure 14(a) and this image was divided into left and right sub-images, as shown in Figure 14(e), based on the x* coordinate. The unwrapped image can be rounded