1.1.1 Hand-Eye Calibration
Hand-eye calibration of the eye-in-hand configuration has been discussed for several decades, with many solutions to estimate parameters accurately. While assuming that the camera is well calibrated, most works focus on calibrating geometrical relationships by utilizing a 3D object, 2D pattern, non-structured points, or just a point. Jordt et al.
[35] categorized hand/eye calibration methods depending on the type of calibration reference. Based on their results, the following discussion summarizes four various approaches.
(1) Hand-eye calibration methods using 2D pattern or 3D object [3]-[32]
Most hand/eye calibration methods adopt a reference object with known dimensions or a standard calibration object. Features such as corners or circles on the calibration object are extracted in the images. Each feature is related to a position in the reference coordinate system. The camera pose can then be determined by inversely projecting the features in an image in the camera calibration stage. From this viewpoint, using a 2-D pattern is similar to using a 3-D one. The classical approaches to hand/eye calibration solve the transformation equation in the form, AX=XB, as first introduced by Shiu and Ahmad [3][7]. Tsai and Lenz [4][8] developed the closed-form solution by decoupling the problem into two stages: rotation and translation.
Quaternion-based approaches, such as those developed by Chou and Kamel [5], Zhuang and Roth [10], and Horaud and Dornaika [16], lead to a linear form to solve rotation relationships. To avoid error propagation from the rotation stage to the translation stage, Zhuang and Shiu [12] developed a nonlinear optimization method with respect to three Euler angles of a rotation and a translation vector. Horaud and Dornaika [16] also applied a nonlinear optimization method to solve the rotation quaternion and the translation vector simultaneously. Albada et al. [17] considered robot calibration based on the camera calibration method by using a 2-D pattern.
Daniilidis and Bayro-Corrochano [19][22] developed a method of representing a rigid transformation in a unit dual quaternion to solve rotation and translation relationships simultaneously. While simultaneously considering hand/eye and robot/world problems, Zhuang et al. [13] introduced the transformation chain in the
general form, AX=YB, and presented a linear solution based on quaternions and the linear least square method. Similarly, Dornaika and Horaud [21] included a closed-form solution and a nonlinear minimization solution. Hirsh et al. [23]
developed an iterative approach to solve this problem. Li et al. [32] used dual quaternion and Kronecker product to solve it. The method of Strobl and Hirzinger [30]
requires only orthogonal conditions in grids, which relaxes the requirement for patterns with precise dimensions.
(2) Hand-eye calibration methods using a single point [11], [33]-[35]
Wang [11] pioneered a hand/eye calibration procedure using only a single point at an unknown position. Pure translation without rotating the camera to observe the point yields a closed-form solution for the rotational relationship. Once the rotation is obtained, the translation relationship is solved via least squares fitting. Similar works such as [34] and [35] include finding camera intrinsic parameters. Gatla et al. [33]
considered the special case of pan-tilt cameras attached to the hand.
(3) Hand-eye calibration methods using non-structured features [36]-[38]
This category can be considered an extension of using a single point. Rather than using a specific reference, the method of Ma [36] uses feature points of unknown locations in the working environment and determines the camera orientation by using three pure translations. By moving the camera without rotating, the focus of expansion (FOE) is determined by the movements of points in the images where the relative translation is parallel to a vector connecting the optical center and the FOE point. Directions of the three translations in the camera frame are related to the direction of hand movement in the robot frame. Based on such relationships, the camera orientation relative to the end-effector can be calculated. After the orientation is obtained, the camera position with respect to the robot can be obtained by analyzing the general motions. Similarly, Wei et al. [37] computed the initial parameters and developed an automatic motion planning procedure for optimal movement to minimize parameter variances. The work of Andreff et al. [38] was based on the structure-from-motion (SfM) method. It provided a linear formulation and several solutions for combining specific end-effector motions.
(4) Hand-eye calibration methods using optical-flow [39]-[40]
Optical flow data implicit the information of camera motion. To extract the relative pose from the flows and motions, Malm and Heyden [39]-[40] developed a method by pure translation followed by rotation around an axis. This category can be regarded as a unique case of using non-structured features.
While addressing the eye-in-hand and eye-to-hand configurations for visual servoing, Staniak and Zieliński [41] analyzed how calibration errors influence the control. Calibration of the eye-to-hand configuration for static camera has seldom been discussed since the transformation can be in the same form of eye-in-hand configuration. Most of the above methods are applicable to either eye-in-hand or eye-to-hand configurations. Dornaika and Horaud [21] presented a formulation to deal with the calibration problems for the both configurations, indicating that these two problems are identical. However, this identity is applicable only when the robot hand can be viewed with the eye-to-hand camera. This limits installation flexibility and potential applications. An eye occasionally fails to see a hand due to various requirements. For instance, to sort products on a moving conveyor, the camera is often placed at a distance from the arm to compensate for image processing delay and avoid interference when tracking targets on a rapidly moving conveyor belt. In catching ball systems, e.g., [44], cameras focus on the region of the initial trajectory of the ball and may not see the arm. Sun et al. [45] developed a robot-world calibration method using a triple laser device. Despite enabling calibration of the eye-to-hand transformation when the camera cannot see the arm, their method requires a uniquely designed triple-laser device, thus limiting the flexibility of system arrangement. Moreover, the spatial relationship from the working plane to camera must also be known in advance.
1.1.2 Camera Calibration
Since the camera calibration is fundamental in machine vision, there are rich literatures and resources on camera calibration. The direct linear transformation (DLT) (Abdel-Aziz and Karara [74] and Shapiro [75]) is applied to camera calibration to obtain linear solution. Bacakoglu and Kamel [76] adopted the DLT method and the nonlinear estimation, and developed methods to refine homogeneous transformation between the two steps. Methods based on DLT method basically need a 3D reference.
Tsai [77] developed a radial alignment constraint (RAC) based method using a 2D pattern. Zhuang et al. improved the Tsia’s RAC method and utilized robot mobility to deal with multiple planes. Zhang [50] provided a flexible method which can handle data of different poses of a planar pattern and doesn’t need any robot or linear table. An excellent toolbox named Camera calibration toolbox for Matlab (Bouguet [56]) based on Zhang’s method [50] is available online and is used in this work.
Most work on calibration of a hand-eye system is done by separating camera calibration and hand-eye calibration, subsequently causing error propagation. In many industrial applications, a camera must be recalibrated frequently. Camera calibration requires reference objects that could be a 3-D object [49], a 2-D pattern [50][51], or a 1-D bar [52]. However, in an environment that humans cannot easily enter to place a reference object, separating the calibration might become inefficient. Hence, Ma [36]
developed a method that combines the two calibrations into one process when considering this problem for an active vision system.
1.1.3 Robot Calibration
Calibration of the geometrical parameters of a manipulator is important to ensure the accuracy of the end-effector position. The inaccuracy factors include assembly misalignments, tolerance of mechanical parts, and joint offset. Further, mechanical wear and temperature variation due to long-term operation will also induce drifts of the parameters. To maintain consistent performance, it is often necessary to calibration the manipulator even when it is in production line. As a result, cost-effective and easy-to-use calibration tools and methods are beneficial to enhance the efficiency of manipulator usage in industry.
Many studies of the kinematic identification and calibration were done [57]-[59].
Kinematic calibration methods generally consisted of four processes: kinematic modeling, measurement, identification, and correction [57]. High-accuracy laser interferometer is used for measuring the end-effector position, and then the calibrated parameters were obtained by applying nonlinear least square method [60] and neural networks [61]. The high-accuracy laser interferometer could guarantee tiny measurement error to achieve more accurate kinematic parameters. Rauf et al. [62]
used a partial pose measurement device for kinematic calibration of parallel
manipulators. Renaud et al. [63] used a low-cost vision-based measuring device, which included a calibration board and a camera, to capture the end-effector pose and then calibrated the parallel manipulators. Another vision-based kinematic calibration uses the camera type 3D measurement device to capture the position of an infrared LED mounted on the end-effector [64]. In these vision-based methods, the manipulator must locate in the view of the camera.
In Newman and Osbom’s method, the manipulator was controlled to move along the laser line using an optical quadrant detector [65]. A laser pointer mounted on the end-effector is pointed to the position sensitive detector for workspace calibration [66]
and kinematic calibration [67]. Park et al. [68] estimated the kinematic errors by using a structured laser module, a stationary camera, and they introduce a method based on the derived Jocabian matrices, and an extended Kalman filter. Two laser pointers beamed on the screen, and the camera measured an accurate position of two laser spots on the screen. In this setting (laser pointer/camera/plane), the arrangement for calibration is quite free as long as the laser pointer mounted on the manipulator is projected on the plane which is visible to the camera.