An image matching algorithm for variable mesh surfaces

(1)

An image matching algorithm for variable mesh surfaces

Tung-Hsien Tsai

a,*

, Kuang-Chao Fan

b

a_{Department of Mechanical Engineering, Hsiuping Institute of Technology, No. 11, Gungye Road, Dali City,}

Taichung 41280, Taiwan, ROC

b

Department of Mechanical Engineering, National Taiwan University, 1, Roosevelt Road Sec. 4, Taipei 10617, Taiwan, ROC Received 30 September 2005; received in revised form 11 April 2006; accepted 23 May 2006

Available online 6 June 2006

Abstract

For measuring an object that has a large surface or a local steeper profile, a variable resolution optical profile measure-ment system that combined two CCD cameras with zoom lenses, one line laser and a three-axis motion stage are sufficient to implement. The measurement system can flexibly zoom in or out the zoom lens to measure the object profile according to the slope distribution of the object. However, one can acquire only one set of sectional measurement points in each mea-surement. To reconstruct the entire object, every set of sectional measurement points acquired at different positions must be matched with one another.

In this paper, an efficient and accurate image matching algorithm of two measured sectional images that has variable mesh surfaces is proposed. Using this image matching algorithm and the variable resolution optical profile measurement system, a larger profile or complicated object can be efficiently and flexibly measured and matched in reverse engineering. Two experimental results obtained from different mesh images matching are sufficient to demonstrate the validity and applicability of this system.

Keywords: Reverse engineering; Non-contact measurement; Surface ﬁtting; Image matching; Variable mesh

1. Introduction

In industry and reverse engineering, optical image measurement systems have been widely used for the measurement of object profile that has com-plex surfaces [1–3]. The application is flexible and quick. However, all optical image measurement sys-tems generally are fixed focus types that are mounted on a fixed three-axis stage [4–6]. In this

kind of system, the resolution per pixel and the ﬁeld of view are constrained by the lens. Tsai and Fan[7]

have developed a variable resolution optical profile measurement system which can vary the focus length by adding a zoom lens to the conventional dual CCD cameras system. For some steeper sur-face areas, the field of view can be further localized by using a zoom lens so that the micro surface pro-files can be detected to extra precise resolution, and the measurement data is more lifelike than a fixed focus type.

doi:10.1016/j.measurement.2006.05.014

* _{Corresponding author. Tel.: +886 4 24961197.}

E-mail addresses: [email protected] (T.-H. Tsai), [email protected](K.-C. Fan).

(2)

Nevertheless, one of the drawbacks of the system is that the measurement always divides into several processes for a large object, especially at the steeper area. Thus, one can acquire only one set of sectional measurement points at each time in the measurement process. To reconstruct the entire object, every set of sectional measurement points acquired at different positions or resolutions must be matched with one another. Most previous research efforts on image matching have been focused on the development of registration efficiency and accuracy, but they are incapable of registering variable mesh surfaces [8– 12]. In this study, a flexible image matching algo-rithm for variable mesh surfaces is developed to solve the above problems. The two experiment results demonstrate the validity and applicability of the proposed algorithm. The details of this algo-rithm are described in the later sections.

2. Development of the variable resolution optical proﬁle measurement system

An object can be measured by using the variable resolution optical profile measurement system (VROPMS) with various resolutions to enhance the flexibility and save the measurement time. This VROPMS can flexibly zoom in or out to measure a 3D profile according to the slope distribution of the object and the accuracy requirement. The pic-ture of scanning probe of our VROPMS is shown inFig. 1. In this section, a summary of mathemati-cal model and mathemati-calibration procedures of VROPMS is described. This procedure is quick and accurate.

2.1. Triangulation measurement principle

Triangulation measurement technology is one of the frequently used technologies for non-contact measurement system. It has the advantages of fast computation and easy operation. The principle of triangulation measurement with one camera is shown in Fig. 2. In the ﬁgure, p(x, y, z) is a point in the world coordinate (X, Y, Z), p0_{(u, v) is its} focused point in the image plane (U, V), where u and v is the horizontal and vertical image coordi-nates respectively, f is the focal length, h is the angle between the X-axis and light direction, and b is the distance between the light source and the lens opti-cal center O. According to the geometriopti-cal optics and similar triangles, the coordinates of point p(x, y, z) can be calculated by using the following formulae: x¼ bu ðf cot h uÞ; y¼ bv ðf cot h uÞ; z¼ bf ðf cot h uÞ: 8 > > > > > > > < > > > > > > > : ð1Þ

If the system parameters (b, f, h) are obtained, then the coordinates of point p(x, y, z) can be calculated. However, this technology possesses the disadvan-tages of complicated system parameter calibration and tedious system setup. In practice, inaccuracy in system parameters and system setup will induce large measurement errors.

2.2. Mathematical model and calibration principle There are two groups of physically meaningful parameters in a pinhole camera. The ﬁrst group is

Fig. 1. The picture of scanning probe of VROPMS.

CCD Z X ) , , (x y z p ) , ( vu p x b o u z f Light Source _θ

Fig. 2. The principle of triangulation measurement with one camera.

(3)

intrinsic parameters and the second one is extrinsic parameters. Maybank and Faugeras[13]and Burner

[14] observed that some of the intrinsic parameters are constant over long periods, especially the aspect ratio and the skew angle which do not change in gen-eral. Faugeras and Toscani[15]suggested that if we try to model the intrinsic parameters by using func-tional relationships on the vertical scale factor, the camera must be calibrated by using a classical method for several zoom positions. Sturm[16] pro-posed that the functional relationship among the intrinsic parameters is linear. Consequently, in a zoom lens system, the extrinsic parameters are con-stant, but the intrinsic parameters will vary accord-ing to the amount of zoom. Therefore, the VROPMS system parameters must be adjusted simultaneously responding to the various zoom posi-tions for exact computation of space coordinates.

A flexible mathematical model and calibration procedure based on the concept of linear relation-ship that were proposed by Tsai and Fan [17] is adopted for establishing the relationship between the 3D space coordinates and the image coordi-nates. This procedure can not only reduce the cali-bration time effectively, but can also flexibly acquire the system parameters at any focus position with satisfactory measurement accuracy. The sche-matic diagram of calibration procedure of VROPMS is shown inFig. 3. The required calibra-tion procedures are divided into two steps. The objective of the first step is to convert the image coordinate from the magnification factor position m to its corresponding image coordinate at m = 1 using the ‘polynomial fitting functions’. The pur-pose of the second step is to use the ‘coordinate

mapping functions’ to transform the 2D image data into corresponding 3D spatial positions at the origi-nal focal plane (m = 1). A standard template fixed on the linear stage is employed for generating cali-bration points in the calicali-bration procedures. The coefficients of the polynomial fitting functions and the coordinate mapping function equations can be determined using least squares approximation. Therefore, using the polynomial fitting functions and the coordinate mapping function equations that were calibrated in advance, the related image coor-dinates for any focus position corresponding to the original focus position and the related space coordinates for each measurement data can be derived. In summary, the VROPMS needs two data conversion steps. In the first step, the image coordi-nates (Um, Vm) for the focus position at magnifica-tion m can be detected using CCD cameras. Substituting (Um, Vm) into the polynomial fitting functions, the corresponding image coordinates ðUm; V

mÞ can be calculated. In the second step, the corresponding space coordinates (Y, Z) can be derived from the image coordinates (U_m; V_m) using the coordinate mapping functions. Lastly, by com-bining the measured Y and Z components of each profile line from the image information with the positioned X component from the scanning stage, the entire profile can be formed with respect to the focus position at magnification m. For the detailed description of this calibration procedure, readers please refer to[17]. From their experimental results in the article, all of the measurement errors are satisfactory.

3. The image matching algorithm of variable mesh surfaces

There is generally no existing absolute reference surface when matching every set of sectional mea-surement surfaces. This study has determined the ﬁrst set of sectional measurement surfaces to be a matching reference surface, the registration of two sets of measurement points is accomplished by reg-istering the second set of measurement points to the reference surface according to the image matching algorithm. There are several registration approaches for solving the image matching problem. Those approaches can be generally divided into three cate-gories: (1) motion control (2) feature-based and (3) nonlinear programming method. Bhanu [18], Vemuri and Aggarwal[19]and Lai et al.[20] deter-mined the coordinate variations between the images P(x,y,z) ) , ( * * m mV U ) , (UmVm m U m V Coordinate Mapping Function Fitting Function

Original Focus Plane

Other Focus Plane

(0,0,0) 1 O m O Space Plane U V Y x z

(4)

according to a known motion of the measurement system or the related position of the particular pat-tern in the images. The image registration is accom-plished by the coordinate variations of the images that have been determined in advance. This motion control registration method is fast and easy opera-tion, but the scope and manner of the image grab-bing are constrained.

Stein and Medioni [21], Fan [22], Li [23], Chua and Jarvis [24] and Higuchi et al. [25] calculated the feature values of the corresponding points for each image to acquire the coordinate transforma-tion matrix of the images by comparing the feature values between the images. The feature values are inﬂuenced by the noise of measurement points and the selective numbers of control points. Neverthe-less, the object proﬁle must have enough obvious features for extraction. Although the feature-based method is automatic and suitable for pattern recog-nition, it is inaccurate and not suitable to register a simple free-form surface.

Besl[8], Chen and Medion[9], Lin and Chen[10], Eggert et al.[11], and Zhang[12]adopted a nonlin-ear programming method to register two image sur-faces. The registration was accomplished by finding the nearest distance between two image surfaces and minimizing the objective function by various opti-mization theories. This registration is accurate and easy for operation. But an appropriate initial value for optimal operation is desired to avoid a local minimum result. Therefore, an efficient and accurate registration approach of the nonlinear program-ming method is an interesting topic for image match. The iterative algorithm is used to find the nearest distance between two matching surfaces exactly. However, this method is time consuming and tedious for finding the convergence solution. In this study, based on the method that was devel-oped by Fan and Tsai[26], an image matching algo-rithm of variable mesh surfaces was proposed for high efficiency and accuracy. The algorithm is divided into five departments as described below. 3.1. Surface reconstruction

Surface reconstruction of the measurement points from multiple range images can be deter-mined by several approaches[27,28]. A bicubic uni-form B-spline interpolation approach was used to describe the ﬁrst set measurement points and construct a B-spline reference patch using four mea-surement points of the ﬁrst set meamea-surement points.

All of the patches in the ﬁrst measurement points can then be joined along their boundaries with C0 and C1 continuity to construct a multiple bicubic B-spline reference patches that are everywhere with C0and C1continuity and are used as the reference surface.

3.2. Initial localization

The shape error of the matching image surface in this paper is defined as the maximum value of the nearest distances from the second set measurement points to the reference (the first set) surface. Obvi-ously, the objective function of fine registration is the sum of the squared nearest distances. For acquiring an appropriate initial value, the initial geometric matching location of the two sets of measurement points was determined by using human–computer interaction. This initial localiza-tion method is convenient and easy for operalocaliza-tion. Two sets of sectional measurement points are located initially by the coordinate translations and rotations. The initial shape error of the location is computed and compared sequentially for locating the images on the appropriate position.

3.3. Finding the nearest distance of the point to the reference surface

In the blending area, all of the second set measure-ment points have the corresponding nearest mesh point and four close neighbor patches around the nearest mesh point. The key point of a nonlinear pro-gramming method is to find the closest distance of the point to the corresponding reference surface accu-rately and quickly. In this study, we adopted an improved algorithm which was developed by Fan and Tsai[26]to find the closest point. The method determines the closest distance between the second set measurement points and the reference surface through the intersection between the normal vector of the reference surface and the second set measure-ment point directly. Fig. 4 shows the concept of searching method for finding the closest distance.

According to the concept of the normal vector, the closest distance between the second set measure-ment point and the reference surface can be determined by solving two nonlinear equations. These equations solution can be determined by using the Newton–Raphson method [29]. Because the Newton–Raphson method is a second-order

(5)

method, the computing time for ﬁnding the closest distance is faster than the Powell algorithm. 3.4. Searching rule of the nearest mesh point

The objective function of optimization is the sum of the squared nearest distances. For finding the closest distance between the second set measure-ment points and the reference surface, the corre-sponding mesh point of the reference surface of the second set measurement points must be found. A flexible searching rule of the nearest mesh point is proposed to search the mesh points exactly. Since the measurement coordinates of X-axis depend upon the scanning interval in a linear scanning mea-surement, all of the measurement points on a scan-ning line stripe have the same X-axis coordinate. By comparing the X-axis coordinate of each second set measurement point with those of the first set, the position of second set measurement point will be located between two scanning line stripes of the first set. The nearest mesh point is determined by calcu-lating the point with the shortest distance of two scanning line stripes. The concept of searching rule of the nearest mesh point is shown inFig. 5. Symbol ‘‘a’’ and ‘‘b’’ are two mesh points, pjand pj+1are the adjacent points of the second set measurement

points. This ﬂexible searching rule includes ﬁve steps as follows:

1. Compare the X-axis coordinates of the second set measurement points and the reference mesh points.

2. Check whether the second set measurement point is located on the outside boundary lines of the blending area of the reference mesh points or not?

3. If the second set measurement point is located on the outside boundary line of the reference mesh points, calculate the distances to the reference mesh points along the column direction (Y-axis) of the boundary line, thus the nearest mesh point is the closest distance point.

4. Else, distinguish the point pj that is located between two scanning line stripes L1and L2. Cal-culate the distances corresponding to point pjto the reference mesh points along the column direc-tion of L1and L2, respectively. Then the nearest mesh point (point ‘‘a’’) is the point that has clos-est distance.

5. Repeat the searching steps from 1 to 4 for the next measurement point pj+1 shown in Fig. 5, and the nearest mesh point of point pj+1is point ‘‘b’’.

The highlight of the searching rule is that only the X-axis coordinates of the second set measure-ment points are compared to those of the reference surfaces. It is fast and useful to exactly ﬁnd the near-est mesh point from various mesh patches.

3.5. Optimization theory

Although the parameters of a rigid body trans-formation matrix of the two sets of sectional mea-surement data can be obtained in the initial registration, the accuracy is not satisfactory. There-fore, a ﬁne registration must be executed for precise surface reconstruction. An optimization theory of a ﬁne registration that composed of the normal vector method, the least squares method and the DFPM searching method (Davidon–Fletcher–Powell method)

[30] are proposed to acquire optimal registration results eﬃciently.

Let (a, b, c) and (x, y, z) be the angular rotations and the linear translations parameters of a rigid body transformation, respectively. The nearest dis-tances between the second set measurement points pi(i = 1 to n) and the corresponding point qiin the

p

q

v

u

Normal Vector

Fig. 4. The concept of the searching method for ﬁnding the closest distance. X a b row col Z j p 1 + j p 1 L L₂

(6)

reference patches are d1, d2, d3,. . .,dn. Then the objec-tive function is Fða; b; c; x; y; zÞ ¼X n i¼1 ½diða; b; c; x; y; zÞ 2 ; ð2Þ where d2_i ¼ kpi qik 2

. Thus, a ﬁne registration is a search of six parameters ((a, b, c) and (x, y, z)) and minimization of the objective function. That is min Fða; b; c; x; y; zÞ ¼ minX

n i¼1 ½diða; b; c; x; y; zÞ 2 : ð3Þ The parameters matrix in the Eq.(3)can be derived by the DFPM searching algorithm. In this study, a computer program based on the above methods was developed to register various sectional measured meshes, and to analyze the shape error of the second set measured data with respect to the reference sur-face. Here, we called it the variable mesh image matching method (VMIMM). The ﬂow chart of the VMIMM is shown inFig. 6.

4. Experimental applications

4.1. Experiment 1: Diﬀerent meshes image matching of a club head

A club head of golf is measured with a scanning pitch of 1 mm in two diﬀerent directions. Two sec-tional measurement points are shown in Fig. 7. There is an unknown cross arrangement overlap in the common area. Using human–computer interac-tion, this overlap is approximately matched by initial localization of the two sets of measurement points, as shown inFig. 8. The error of the maximum shape and that of the root mean square (RMS) of the initial localization are about 1.3710 mm and 0.5865 mm, respectively. After the VMIMM procedure, the error of the optimal maximum shape and the RMS reduce to 1.2242 mm and 0.4484 mm. Subsequently, the coordinates of the second set measurement points can be transformed to the reference coordinates sys-tem. The results of merging two sets of measurement points for the completed shape of the club head is shown inFig. 9.

4.2. Experiment 2: Variable meshes image matching of a sculpture

For a complex object, especially that has a stee-per region, the measurement is in general divided

into several sections with various measurement res-olutions to enhance the measurement accuracy, and also to save the total measurement time. It is in a manner similar to the CAE analysis of the variable mesh.Fig. 10shows the coarse measurement points

Fig. 6. The ﬂow chart of VMIMM.

(7)

of a human sculpture, which was taken by linear scanning measurement with a pitch of 1 mm. The slope distribution of the coarse scanning proﬁle is shown inFig. 11, a steeper slope is found in the dar-ker area. As shown inFigs. 10 and 11, the ambient area of the nose is a local steeper area, the scanning pitch of 1 mm at the nose area seems too larger and the measurement accuracy is not satisfactory. The scope of local steeper area is 5 mm· 15 mm, approximately.

To enhance the accuracy of the nose area, the magniﬁcation of lens was adjusted to 2 and the

scanning pitch 0.5 mm. A linear scanning measure-ment of the nose area was executed to form a micro measurement with a finer resolution. The local mea-surement points are shown in Fig. 12. Both the image resolution and the scanning pitch are different between the micro and macro points, and the blend-ing area of two sets measurement points are a vari-able mesh surface. Two different mesh surfaces in the blending area must be stitched up to acquire

Fig. 10. The coarse measurement points of a human sculpture.

Fig. 11. The slope distribution of the coarse scanning proﬁle. Fig. 8. The blending area and initial localization of a club head.

Fig. 9. The completed shape of the club head surface.

Fig. 12. The local measurement points with lens magniﬁcation of 2.

Fig. 13. The initial localization of two sets of the measurement points.

(8)

completed measurement points with variable resolutions.

The initial registration of two sets measurement points is implemented as shown in Fig. 13. There are 2142 points in the blending area of the matching images. The error of the initial maximum shape and that of the RMS are about 0.5348 mm and 0.2416 mm, respectively. After the ﬁne registration, the error of the maximum shape and the RMS are 0.4631 mm and 0.1724 mm, respectively. The com-putation time of the optimal analysis is about 136 sec. Fig. 14shows the completed shape of this human sculpture surface with diﬀerent meshes. The shading diagram of this human sculpture is shown inFig. 15.

5. Conclusions

The optical proﬁle measurement systems are used ﬂexibly and quickly in reverse engineering for rapid

product prototyping with complex surfaces. A vari-able resolution optical profile measurement system (VROPMS) was developed to measure the object profile according to the slope distribution of the object or the accuracy requirement. This flexible measurement system can collect more image infor-mation and compensate for the loss of data that resulted from using only one CCD camera. In prac-tice, the object was measured initially using VROPMS with a large scanning interval using the original focus position. For some steeper surfaces, the field of view can be further localized by using a zoom lens so that the micro surface profiles can be detected to a very fine resolution.

Generally, the VROPMS was used to measure an object with several sectional measurements formed to generate a cross arrangement mesh or a variable mesh in the blending area. Every set of sectional measurement points acquired at different positions or resolutions must be matched with one another to reconstruct the entire object. In this paper, a flex-ible image matching algorithm for variable mesh surfaces is developed to solve the above problems. In the 3D surface reconstruction process, this vari-able mesh image matching method (VMIMM) can not only register two sectional measurement images with high efficiency and accuracy, but can also deal with meshes that have a cross arrangement or differ-ent image resolutions. Therefore, by combining the VROPMS with the VMIMM for measuring a large or complicated object, the measured points and reconstructed surfaces will be more lifelike and accurate than the conventional fixed focus type. The two experiment results demonstrate the validity and applicability of the proposed algorithm. The algorithm will be found useful in reverse engineering applications.

References

[1] M.J. Tsai, C.C. Hung, Development of a high-precision surface metrology system using structured light projection, Measurement 38 (3) (2005) 236–247.

[2] Y.V. Chugui, Optical dimensional metrology for 3D objects of constant thickness, Measurement 30 (1) (2001) 19–31. [3] C. Liguori, A. Paolillo, A. Pietrosanto, An on-line

stereo-vision system for dimensional measurements of rubber extrusions, Measurement 35 (3) (2004) 221–231.

[4] J.J. Aguilar, F. Torres, M.A. Lope, Stereo vision for 3D measurement accuracy analysis, calibration and industrial applications, Measurement 18 (4) (1996) 193–200.

[5] C.C. Ho, 3D Surface Matching Techniques in Image Measurements, Master Thesis, National Taiwan University, 1997.

Fig. 15. The shading diagram of this human sculpture. Fig. 14. The completed shape of this human sculpture surface with diﬀerent meshes.

(9)

[6] W.C. Tai, M. Chang, Non-contact proﬁlometric measure-ment of large-form parts, Optical Engineering 35 (9) (1996) 2730–27353.

[7] T.H. Tsai, K.C. Fan, A variable resolution optical proﬁle measurement system, Measurement Science and Technology 13 (2002) 190–197.

[8] P.J. Besl, A method for registration of 3D shapes, IEEE Transactions on PAMI 14 (2) (1992) 239–256.

[9] Y. Chen, G. Medion, Object modelling by registration of multiple range images, Image and Vision Computing 10 (3) (1992) 145–155.

[10] A.C. Lin, C.F. Chen, Point-data processing and error analysis in reverse engineering, International Journal of Advanced Manufacturing Technology 14 (1998) 824–834. [11] D.W. Eggert, A.W. Fitzgibbon, R.B. Fisher, Simultaneous

registration of multiple range views for use in reverse engineering of CAD models, Computer Vision and Image Understanding 69 (3) (1998) 253–272.

[12] Z.Y. Zhang, Iterative point matching for registration of free-form curves and surfaces, International Journal of Computer Vision 13 (2) (1994) 119–152.

[13] S.J. Maybank, O.D. Faugeras, A theory of self calibration of a moving camera, International Journal of Computer Vision 8 (2) (1992) 123–151.

[14] A.W. Burner, Zoom lens calibration for wind tunnel measurements, in: Proc. SPIE Conf. on Videometrics IV, Philadelphia, PA, 1995, pp. 19–33.

[15] O.D. Faugeras, G. Toscani, Camera calibration for 3D computer vision, in: Proc. Int. Workshop Machine Vision and Machine Intelligence, Tokyo, 1987, pp. 240–247. [16] P. Sturm, Self-calibration of a moving zoom-lens camera by

pre-calibration, Image and Vision Computing 15 (1997) 583– 589.

[17] T.H. Tsai, K.C. Fan, A mathematical model and procedure for calibrating a 3D measurement system with a zoom-lens

camera, Journal of the Chinese Society of Mechanical Engineers 21 (3) (2000) 223–227.

[18] B. Bhanu, Representation and shape matching of 3D objects, IEEE Transactions on PAMI 6 (3) (1984) 340–351. [19] B.C. Vemuri, J.K. Aggarwal, 3D model construction from

multiple views using range and intensity data, in: Proc. Conf. On Comput. Vision Pattern Recogn., 1986 pp. 435–437. [20] J.Y. Lai, W.D. Ueng, C.Y. Yao, Registration and data

merging for multiple sets of scan data, International Journal of Advanced Manufacturing Technology 15 (1999) 54–63. [21] F. Stein, G. Medioni, Structural indexing: Eﬃcient 3D object

recognition, IEEE Transactions on PAMI 14 (2) (1992) 125–145. [22] T.J. Fan, Describing and Recognizing 3-D Objects Using Surface Properties, Springer-Verlag Inc., New York, 1990. [23] S.Z. Li, Toward 3D vision from range images: An

optimi-zation framework and parallel networks, CVGIP: Image Understanding 55 (3) (1992) 231–260.

[24] C.S. Chua, R. Jarvis, 3D free-form surface registration and object recognition, International Journal of Computer Vision 17 (1996) 77–99.

[25] K. Higuchi, M. Hebert, K. Ikeuch, Building 3-D models from unregistered range images, Graphical Models and Image Processing 57 (4) (1995) 315–333.

[26] K.C. Fan, T.H. Tsai, Optimal shape error analysis of the matching image for a free-form surface, International Journal of Robotic and Computer-Integrated Manufactur-ing 17 (3) (2001) 215–222.

[27] A.P. Armit, Curve and surface design using multipatch and multiobject design systems, Computer-Aided Design 25 (4) (1993) 251–261.

[28] E. Beeker, Smoothing of shapes designed with free-form surface, Computer-Aided Design 18 (4) (1986) 224–232. [29] D.A. Wismer, R. Chattergy, Introduction to Nonlinear

Optimization, North-Holland, 1978.