6.3.1 Simulation Results
The proposed method is validated via computer simulations under different noise levels and with different numbers of samples. The values of parameters in all cases are chosen as follows: f =550, u f =550, v αc=0, u =320, 0 v =240, 0 CBθ =-90°, CBφ
=-10°, CBϕ =-170°, BCt =[510,151,430]T (mm), ΛEθ =21.80°, ΛEφ =53.39°, ΛEt
=[0,-10,30]T (mm), Ba1=[152.41, 138.34, -163.92]T (mm), Ba2 =[ 48.68, 31.47, -153.81]T (mm), and Ba3 is randomly generated.
In the following tests, the relative errors of one focal length are calculated as
, , / ,
u estimated u true u true
f − f f (percentage). The hand-eye rotation and translation errors are the Frobenius norms of the difference between the true and estimated CBtand CBR; in addition, the relative errors are calculated as testimated −ttrue F ttrue F and
estimated − true F true F
R R R (percentages), respectively. The relative errors of the first plane pose are calculated as B 1, B 1,
estimated true F
Πn −Πn . For each condition, 100 independent trials are performed.
Figure 6-7 summarizes the performance of the proposed methods for a single plane case and a multiple plane case (2 planes), as well as the results from closed-form solution and optimization. Eight and four different hand orientations, each with ten pure translating points, are generated for the single plane case and for the 2 plane case, respectively. Eighty points are generated independently for each case. This test assesses the performance under different image pixel noise level perturbations.
Experimental results confirm that the proposed methods can calibrate hand-eye-workspace geometric relationships and camera intrinsic parameters simultaneously. Accuracy of the focal length estimation of using multiple planes outperforms the one of using a single plane since points on multiple planes provide more abundant information for calibrating camera intrinsic parameters. However, the accuracy of geometrical relationships of using single plane is better, since the single plane case has fewer parameters needed to be calibrated. Furthermore, the accuracy is increased after the optimization procedure.
Figure 6-7 Relative errors with respect to the image pixel noise level.
The single plane case (Left) and the multiple plane case (Right). The closed-form solution (Top) and the nonlinear optimization (Bottom).
Ma’s method [36], Malm and Heyden’s method [27][28], and Horaud and Dornaika’s method [16] are selected to compare with the proposed algorithm. The three methods concern camera intrinsic parameters together and can represent conventional calibration methods of using three reference types, as shown in Figure 6-8 respectively. The three methods are modified for eye-to-hand configuration (originally for eye-in-hand). Since Horaud and Dornaika’s method only introduces calibration of relative translation and rotation between the robot and the reference object, the relative translational and rotational errors are the Frobenius norms of the difference between the true and estimated OEtand EOR rather than CBtand CBR, where OEt and OER denote reference object position and orientation relative to the end-effector .Three noise types are added for the following test. The noises are normally distributed with zero mean and the standard deviation representing the noise
Noise level (pixel)
level, 90 points are generated independently for each method. The number of hand orientations, hand pure translating direction and/or plane poses, are listed as,
M1: Proposed method for multiple plane case: 3 plane poses, 3 hand orientations per plane, 1 starting position and 3 translating directions per orientation, 3 positions along each direction.
M2: Ma’s method: 3 non-structure points on the robot hand, 3 hand orientations, 1 starting position and 3 sets of 3 orthogonal pure translations per orientation.
M3: Malm and Heyden’s method: 10 points on the plane reference in the robot hand, 3 hand orientations, 1 starting position and 2 translations per orientation.
M4: Horaud and Dornaika’s method: 10 point on the 3-D reference object, 9 hand poses.
Figure 6-8 Three different calibration references on the hand.
Table 6-2 The types of added noises Types Range of noise
level Image pixel error 0 ~ 1 pixel Hand translation error 0 ~ 1 mm Hand orientation error 0 ~ 0.05 degree
Since the performances of these methods are related to poses and movements of references, i.e. the robot hand motions, some points should be mentioned when generating data. For the proposed method, the end-effector is close to the workspace to decrease the influence of hand orientation error. When generating data for Ma’s method, especially when pixel positions are noisy, the pure translations should not be parallel to any camera axis; otherwise, the estimation of FOE point seriously deviates and following camera calibration results are erroneous. The instructions provided by Malm and Heyden to implement their method place the plane pattern nonparallel to image plane and translate the plane in the direction nonparallel to the plane. Moreover, based on our experiences, manipulating the plane pattern closer to the camera is better for Malm and Heyden’s method and Horaud and Dornaika’s method. According to
effector
72
method is outstanding in terms of camera intrinsic parameters. Horaud and Dornaika’s method can resist hand orientation noise in rotation terms. Ma’s method has a satisfactory rotation estimation capability for all noise types.
Figure 6-9 Relative errors of f , rotation, and translation with u
respect to noise levels added in image pixel position.
Noise level (pixel) added in image position Relative error (%) of fu
Noise level (pixel) added in image position
Relative error (%) of rot.
0 0.2 0.4 0.6 0.8 1
0 0.5 1 1.5
Noise level (pixel) added in image position
Relative error (%) of trans.
0 0.2 0.4 0.6 0.8 1
M1: Proposed method for multiple plane case M2: Ma’s method
M3: Malm and Heyden’s method M4: Horaud and Dornaika’s method
Figure 6-10 Relative errors of f , rotation, and translation u with respect to noise levels added in robot hand translation.
0 0.2 0.4 0.6 0.8 1
0 0.5 1 1.5
Relative error (%) of fu
Noise level (mm) added in robot hand translation
Noise level (mm) added in robot hand translation
Relative error (%) of rot.
0 0.2 0.4 0.6 0.8 1
0 0.5 1 1.5
0 0.2 0.4 0.6 0.8 1
0 1 2 3 4
Noise level (mm) added in robot hand translation
Relative error (%) of trans.
M1: Proposed method for multiple plane case M2: Ma’s method
M3: Malm and Heyden’s method M4: Horaud and Dornaika’s method
Figure 6-11 Relative errors of f , rotation, and translation u with respect to noise levels added in robot hand orientation.
Relative error (%) of fu
0 0.01 0.02 0.03 0.04 0.05
0 0.2 0.4 0.6 0.8
Noise level (degree) added in robot hand orientation
Relative error (%) of rot.
0 0.01 0.02 0.03 0.04 0.05
0 0.2 0.4 0.6 0.8 1
Noise level (degree) added in robot hand orientation
Noise level (degree) added in robot hand orientation
Relative error (%) of trans.
0 0.01 0.02 0.03 0.04 0.05
0 0.2 0.4 0.6 0.8
M1: Proposed method for multiple plane case M2: Ma’s method
M3: Malm and Heyden’s method M4: Horaud and Dornaika’s method
Figure 6-12 Relative errors of f , rotation, and translation u with respect to mixed noise levels.
Figure 6-12 shows that these methods are comparable to each other with mixing the three kinds of noises. Ma’s method has a satisfactory rotation estimation capability for noise. The proposed method is outstanding in terms of camera intrinsic parameters and eye-to-hand translation.
Relative error (%) of fu
Relative error (%) of rot.
Noise level
Relative error (%) of trans.
0 0.2 0.4 0.6 0.8 1
M1: Proposed method for multiple plane case M2: Ma’s method
M3: Malm and Heyden’s method M4: Horaud and Dornaika’s method
The accuracy is also influenced by the number of data. Performances of the proposed methods with a single plane, 2 planes and 3 planes are compared to each other using different numbers of samples. Zero mean Gaussian noise with a 1-pixel standard deviation is added to each point image position. Different numbers of hand orientations are increased to produce different numbers of samples with measurements of camera projections. For each number of samples, 100 independent trials are performed and averages of relative errors are determined. Figure 6-13 summarizes results of one focal length, fu , to represent the system error. The errors decrease with an increasing number of samples. Because of limited data information for camera intrinsic calibration, the error of using single plane is larger than that using the multiple planes. Notably, for the single plane case, the relative errors fall into reasonable levels, i.e. it could be used to save calibration time if the accuracy requirement is met.
Figure 6-13 Relative errors with respect to the number of samples.
Closed-form solution(Top) and Nonlinear optimization(Bottom).
6.3.2 Experimental Results
Figure 6-14 shows the experimental setup. A planar plate, sized 50×50cm, is placed
40 60 80 100 120 140 160 180 200 220 240
3 plane poses2 plane poses
Relative error (%) of fu 3 plane poses2 plane poses
roughly 90cm away from the manipulator. In this configuration, the camera cannot see any part of the manipulator. The laser pointer is a laser diode with an adjustable lens attached on the robot tool.
Figure 6-14 Experimental setup.
Overview (Left) and the laser pointer attached on the end-effector (Right).
Following the steps in Section 2.5, under each hand orientation, this work manipulated the end-effector to a position and then moved it along three directions.
Three positions along each direction were generated, resulting in ten hand positions per orientation. Steps 3 and 4 were performed three times to generate three different hand orientations. The plane pose was then changed twice. Totally, 3 plane poses and 90 hand poses per plane were generated. All movements were designed to project the laser beam onto the planar object, and corresponding images were saved. Data were applied to the proposed closed-form solutions in order to obtain initial values. Next, the proposed nonlinear optimization method refined these values.
Table 6-4 summarizes the results. Six tests were performed in this experiment. The first column in the first test, labeled as Initial, is the estimation of the closed-form solution described in Section 2.3. The second column, labeled Refined, is the estimation of nonlinear optimization. The third column is the standard deviation, indicating the uncertainty of the refined result. The test 2 results of nonlinear optimization using more data are in the right hand column of
Table 6-4. Figure 6-15 summarizes the extrinsic results from the last test. The results of test 3 to test 6 under 2 and 3 plane poses originate from the closed form solution described in Section 2.2. In each test, the last row, labeled by RMS, shows the root
Robot
mean square of the distance of the image from the estimated points to the predicted points. The camera intrinsic parameters were also estimated by applying Camera Calibration Toolbox for Matlab [56]. This camera calibration was performed with a pattern of 35(5×7) grids; all grids are 15mm×15mm. In total, 3 images of the pattern under different orientations were taken. Table 6-3 lists the calibrated camera intrinsic parameter values as references to validate the results in
Table 6-4. The results are consistent with simulation results that the uncertainty of estimations declines with an increasing number of data.
Table 6-3 Results of using Camera Calibration Toolbox for Matlab
Parameter 105 points
Table 6-4 Results with real data of different number of samples
Parameter
Single plane case
Test 1: 30 samples Test 2: 60 samples Unit
Initial Refined σ Refined σ
(fu,fv,αc) (1053.10,1053.10,0.00) (1142.98,1293.16,0.00) (109.35,192.57,0.00) (1047.45,1092.13,0.00) (53.71,73.83,0.00) pixel
(u0,v0) (320.00,240.00) (320.00,240.00) (0.00,0.00) (320.00,240.00) (0.00,0.00) pixel
(k1,k2) (0, 0) (-1.1959,5.5389) (0.4256,3.4803) (-0.7155,2.2629) (0.2823,1.7424) 1
(p1,p2) (0, 0) (-0.0194,0.0488) (0.0133,0.0274) (-0.0020,0.0124) (0.0114,0.0158) 1
(CBθ,CBφ,CBϕ) (141.47,-6.02,-148.04) (144.51,6.48,-144.51) (0.47,2.37,4.47) (142.41,-0.56,-147.54) (0.33,1.72,2.07) deg.
CB Tt [916.73,-0.83,22.95] [985.72,15.37,27.07] [16.55,10.07,93.92] [955.08,-5.47,17.83] [13.08,4.39,48.63] mm
(ΛEθ,ΛEφ) (0.95,88.39) (1.10,88.75) (0.10,0.02) (1.16,88.76) (0.03,0.02) deg.
ΛE Tt [0.00,10.79,41.95] [0.00,10.34,57.54] [0.00,1.91,1.24] [0.00,9.03,56.80] [0.00,0.56,0.36] mm
1 B T
Πa [2.46,8.32,-173.53] [2.81,4.68,-177.22] [0.83,1.22,4.17] [2.90,5.54,-176.68] [0.69,1.06,3.26] mm
RMS 23.8973 2.4336 2.4418 pixel
Parameter
Multiple plane case: Two planes
Test 3: 60 samples (30 samples per plane) Test 4: 80 samples (40 samples per plane) Unit
Initial Refined σ Refined σ
(fu,fv,αc) (816.34,1067.92,-0.17) (1077.97,1065.69,0.02) (30.02,31.10,0.01) (1093.90,1070.21,0.01) (25.92,26.08,0.01) pixel
(u0,v0) (243.57,656.84) (364.68,223.85) (31.59,31.28) (371.29,197.11) (28.14,29.56) pixel
(k1,k2) (0, 0) (-0.5798,1.3944) (0.1712,0.9398) (-0.5519,1.0944) (0.1168,0.6135) 1
(p1,p2) (0, 0) (-0.0017,0.0012) (0.0084,0.0056) (0.0041,0.0001) (0.0081,0.0048) 1
(CBθ,CBφ,CBϕ) (145.98,-8.88,-162.32) (141.28,-0.08,-147.53) (0.92,1.58,1.61) (140.99,0.41,-147.12) (0.74,1.37,1.47) deg.
CB Tt [1018.33,-22.55,-57.55] [946.65,-2.25,17.29] [5.81,3.80,30.72] [944.59,-2.29,20.65] [4.46,2.70,25.15] mm
(ΛEθ,ΛEφ) (1.14,87.62) (1.23,88.71) (0.04,0.02) (1.21,88.72) (0.03,0.02) deg.
ΛE Tt [0.00,4.46,26.46] [0.00,7.71,55.95] [0.69,0.00,0.44] [0.00,8.12,56.25] [0.46,0.00,0.32] mm
1 B T
Πa [31.87,46.04,-306.93] [2.42,2.28,-175.95] [0.79,0.95,3.78] [2.73,2.35,-177.73] [0.69,0.71,3.28] mm
B 2T
Πa [208.20,153.19,-470.50] [186.61,76.43,-448.28] [5.20,3.75,3.98] [191.38,77.50,-451.92] [3.88,2.41,2.87] mm
RMS 143.3003 2.6361 2.5723 pixel
Parameter
Multiple plane case: Three planes
Test 5: 90 samples (30 samples per plane) Test 6: 120 samples (40 samples per plane) Unit
Initial Refined σ Refined σ
(fu,fv,αc) (1191.74,1125.11,0.01) (1088.34,1088.48,0.02) (19.54,20.64,0.01) (1072.34,1071.61,0.01) (17.95,18.83,0.00) pixel
(u0,v0) (446.07,72.58) (361.89,216.20) (23.08,24.09) (349.63,232.21) (20.02,20.58) pixel
(k1,k2) (0, 0) (-0.5182,1.0424) (0.1101,0.5925) (-0.4831,0.8587) (0.1052,0.5671) 1
(p1,p2) (0, 0) (-0.0003,0.0050) (0.0059,0.0034) (-0.0015,0.0016) (0.0049,0.0028) 1
(CBθ,CBφ,CBϕ) (139.57,-3.55,-146.11) (141.49,1.07,-146.97) (0.65,1.10,1.19) (141.42,-0.66,-147.71) (0.54,0.97,1.06) deg.
CB Tt [949.32,-10.57,-4.41] [950.65,0.10,19.89] [3.51,2.46,20.91] [948.89,-2.98,18.48] [2.75,1.99,18.22] mm
(ΛEθ,ΛEφ) (1.18,87.89) (1.25,88.71) (0.03,0.02) (1.22,88.72) (0.02,0.02) deg.
E T
Λt [0.00,3.85,39.76] [0.00,7.46,55.84] [0.53,0.00,0.33] [0.00,7.97,56.28] [0.34,0.00,0.26] mm
1 B T
Πa [2.30,14.86,-231.06] [2.86,2.53,-178.00] [0.62,0.63,2.88] [3.17,3.26,-179.39] [0.56,0.54,2.59] mm
2 B T
Πa [255.58,149.78,-499.21] [183.56,75.08,-446.46] [3.25,2.39,2.54] [186.65,78.33,-448.03] [2.62,1.75,2.03] mm
3 B T
Πa [181.60,37.80,-478.56] [117.54,-2.95,-394.35] [2.40,1.91,2.67] [120.44,-1.05,-397.42] [2.00,1.54,2.19] mm
RMS 122.9256 2.3590 2.3318 pixel
6.4 Simultaneous Hand-Eye-Workspace and Camera