Dissertation organization

In the remainder of this dissertation, the proposed systematic method to derive a set of new and general analytic equations for unwarping images taken from an omni-directional camera with a hyperbolic-shaped mirror (called a hypercatadioptric camera) is described in Chapter 2. In Chapter 3, the proposed new method of

“edge-preserving 8-directional two-layered weighting interpolation” for interpolating unfilled pixels in a perspective-view or panoramic image resulting from unwarping an omni-image taken by a non-SVP hypercatadioptric camera is described. In Chapter 4, the proposed unified approach to unwarping of omni-images into panoramic or perspective-view images is described. In Chapter 5, the proposed robust and accurate calibration method for coordinate transformation between display screens and their images is presented. In Chapter 6, the proposed camera mouse with the vision-based method for computer cursor control using a video camera held in hand in the air is described. Finally, conclusions and some suggestions for future research appear in Chapter 7.

Chapter 2

Analytic image unwarping for omni-directional cameras with hyperbolic-shaped mirrors

2.1 Introduction

It is well known in computer vision that enlarging the FOV of a camera enhances the visual coverage, reduces the blind area, and saves the computation time, of the camera system, especially in applications like visual surveillance and vision-based robot or autonomous vehicle navigation.

There are many ways to design a camera system consisting of CCD sensors, lenses, and mirrors to increase the FOV of the system [5]. An extreme way is to expand the FOV to a full hemisphere by the use of a catadioptric camera, which is an integration of a CCD sensor chip, a convex reflection mirror, and a projection lens. A popular name for this kind of camera, as mentioned previously, is omni-camera, and that for an image taken by it is omni-image. The surface curve of the reflection mirror in such a kind of camera may be conical, spherical, parabolic, or hyperbolic, and the lens may be of the type of orthographic or perspective projection. To simplify the process for unwarping omni-images into commonly-used perspective ones, it is usually desired to design an omni-directional camera in such a way that

the SVP constraint is satisfied [6]. Only some of the possible mirror/lens combinations can fit the SVP constraint, for examples, a combination of a parabolic mirror and a orthographic lens or that of a hyperbolic mirror and a perspective lens [6]. However, because of the difficulty in the alignment of the mirror and the camera lens, many commercial products do not satisfy the SVP constraint. When this constraint is not met, the resulting locus of viewpoints will form a so-called caustic curve [9]. In such a case, the image unwarping work is very complicated. On the other hand, when the parabolic mirror/orthographic lens combination is used; the resulting system is called a paracatadioptric camera [16]. Following this idea of naming the camera system, when the hyperbolic mirror/perspective lens combination is used, the resulting system is called a hypercatadioptric camera in this study. We deal with the image-unwarping problem for a hypercatadioptric camera in a non-SVP system in this study.

More specifically, we propose in this study a systematic method to calibrate the system parameters of a hypercatadioptric camera and derive accordingly a set of equations for accurate image unwarping. In the proposed calibration process, a calibration pattern of the shape of a thin ring is designed and attached at the border of the mirror as an aid. Next, mirror reflection laws as well as system geometry constraints are utilized to derive a set of mapping equations between a pixel in the image coordinate system and a point in the world space. The calibrated system parameters are used as known parameters in the derivation. The derived equations are then used to unwarp accurately an omni-image taken by a hypercatadioptric camera into a perspective-view image from any viewpoint.

A major contribution of this study is that the derived image unwarping equations are

analytic. This is achieved for the first time. With these equations, unwarping of omni-images

taken by a hypercatadioptric camera into perspective-view images will not be confined to the

SVP constraint. And this makes the applicability of the hypercatadioptric camera much wider to various computer vision problems.

The remainder of this chapter is organized as follows. In Section 2.2, we review some basic concepts about SVP omni-directional cameras and some previous works for omni-directional camera calibration. The camera calibration process proposed in this study is described in Section 2.3. In Section 2.4, the corresponding analytic image-unwarping equations are derived. In Section 2.5, some experimental results using simulation data as well as real images are given. Finally, we made a summary of this chapter in Section 2.6.

2.2 Review of Previous Works

For an SVP catadioptric camera, unwarping an omni-image into a perspective version is a process of forward projection from a point Xp on a certain perspective-view plane in the world space to an omni-image point Xi, which can be described by Xi = h(Xp) with h being a

one-to-one mapping function from the world space to the omni-image plane [7][21]. For

example, for a SVP hypercatadioptric camera, the mapping relation between a point Xp(x, y, z) in a world space and its projection point Xi(u, v) in the image plane, as illustrated in Fig. 2.1, is as follows: hyperbolic curve of the mirror surface described as follows:

2

In practical situations, because of the existence of the geometric lens distortion, the projection point Xi(u, v) in the image plane might be shifted erroneously. So, the real position of the point Xi in the image coordinate system should be calibrated by proper geometric correction even in the SVP case for accurate image unwarping. Some techniques about this can be found in [1][13] and are followed in this study. The details are omitted.

On the other hand, to estimate the intrinsic parameters of a paracatadioptric omni-directional camera system, a calibration procedure should be performed before unwarping omni-images into perspective-view ones. In [16][17], using a single view of three lines, Geyer et al. derived analytic calibration solutions for the focal length, the image center,

Oc

Fig. 2.1 An SVP hypercatadioptric camera. Where Ow is the origin of the world coordinate system (also one focus of the hyperbolic curve), and Oc is the optical center (another focus of the hyperbolic curve).

Xp(x, y, z)

Xi(u, v)

View plane

Omni-image

and the aspect ratio of a paracatadioptric camera. In [18], Kang used the consistency condition of pair-wise tracked point features across a sequence of paracatadioptric images to calibrate the same parameters. These approaches basically deal with the calibration problem of an SVP paracatadioptric camera, and misalignment between the mirror and the camera components (including the lens and the CCD sensor) was not considered. That is, the image plane was assumed to be parallel to the base plane of the mirror in these approaches, and only the intrinsic parameters of the cameras were taken into account in the calibration. The quality of the unwarped image is severely degraded when equations derived from a system configuration not meeting such an SVP assumption are used in the unwarping process, although the intrinsic parameters of the camera have been calibrated.

On the contrary, when a non-SVP camera is used, for example, for the reason to increase the FOV, system configuration parameters related to the pose of the mirror relative to the camera, in addition to the intrinsic camera parameters, need be calibrated. In [19], Aliaga developed a calibration model using a beacon-based pose estimation algorithm for a catadioptric camera which includes a parabolic mirror and a perspective lens. The mirror/lens combination in [19] is a non-SVP design, and the adopted camera model, like Tsai’s [13], has eleven parameters (5 intrinsic and 6 extrinsic). But the physical meanings of Aliaga’s extrinsic parameters are different from those of Tsai’s, with the translation vector representing the offset between the center point of the mirror base plane and that of the image plane, and the rotation vector representing the orientation of the mirror base plane with respect to a world space system. Also, the mirror base plane is assumed to be parallel to the image plane. The calibrated data were used to estimate the pose of the camera with respect to the world space system.

A more complete calibration procedure for a catadioptric camera with a parabolic mirror and a perspective lens, which estimates the intrinsic camera parameters and the pose of the mirror relative to the camera, appeared in Fabrizio et al. [20]. The images of two circles on two planes existing in the mirror were used to calibrate the intrinsic camera parameters and the system configuration parameters. But no discussion was made about how to use the calibrated parameters to modify the mapping described by Eqs. (2.1) to get an accurate unwarped perspective-view image from an omni-image.

2.3 Proposed Method for Calibrating Camera Pose with Respect to Mirror

In this section, the proposed method for calibrating the camera pose with respect to the mirror of a hypercatadioptric camera system is described. The system configuration and the relationships among the involved coordinate systems are described first, and the proposed calibration process is presented next. The camera pose with respect to the mirror is derived finally, using the calibrated system parameters.

2.3.1 System Configuration and Coordinate System Relationships

The configuration of a hypercatadioptric camera and the related coordinate systems used in this study are depicted in Fig. 2.2. First, we define a world coordinate system with its origin W taken to be the middle point between the foci of the two arms of the hyperbolic curve defined by the mirror surface. Let b be the distance from W to the tip Tm of the mirror, c the distance from W to a focus Om of an arm of the hyperbolic curve, h the height of the mirror (measured at Tm), and m the radius of the circular-shaped mirror base. Then, a point M(xm, ym,

z

m) on the mirror surface with respect to W can be described by the following equations according to Eq. (2.2):

2 3D camera coordinate system, and the optical axis of the camera is assumed to align with the

z-axis of the world coordinate system. Accordingly, the center O

i(u0, v0) of the 2D image coordinate system, which is the projection point of the optical axis on the image plane described by z = f, is (0, 0). The mirror parameters a, b, h, and m, and the physical size of the CCD sensor may be obtained from the specifications of the hypercatadioptric camera.

Next, we define a base coordinate system on the mirror with its origin taken to be the (uf, vf)

Fig. 2.2 The configuration of a hypercatadioptric camera used in this study.

T

m

c

C P(xb, yb, zb)

Om

center C of the bottom circle of the mirror. The base plane of the mirror is located at the plane

z = 0 of the base coordinate system. A point P(x

b, yb, zb) on the ring-shaped calibration pattern on the base plane with respect to the origin of the camera coordinate system can be expressed as follows:

[x y z]^T = R[xb yb zb]^T+ T (2.4) where R is a 3×3 rotation matrix with three rotation angles

φ

(pitch),

θ

(yaw), and

ψ

(tilt) around the x-, y-, and z-axes of the base coordinate system, respectively, and T is a translation vector described by T = [Tx Ty Tz]^T. Eq. (2.4) respresents a relationship from the base coordinte system to the camera coordinate system. We will transform the relationship into one from the camera coordinate system to the base coordinte system in Section 2.3.3, which represents the pose of the camera with respect to the mirror.

On the other hand, the location of the projection point I(u, v) in the image plane of a point P(x, y, z) in the camera coordinate system can be described as follows:

z f y z v

f x

u

= , = . (2.5)

To correct possible geometric distortion of the lens in the radial direction, the following distortion model [13] is adopted in this study:

u

d = u + Dx, vd = v + Dy (2.6) where ud and vd are the shifted versions of u and v in the image coordinate system, and Dx and

D

_y are the amounts of distortion estimated, according to [13], by

D

_x

= κ u

d

r

²

, D

_y

= κ v

d

r

² (2.7) with r² = ud2 + vd2 and

κ

being the radial distortion factor of the lens. Combining the above

equations, we get the following equations:

coordinates. Finally, since the unit of the image coordinates (u

f, vf) used in the computer, called computer image coordinates hereafter, is “pixel” for discrete images kept in the computer, additional relations between the distorted image coordinates (ud, vd) and the computer image coordinates (uf, vf) must be specified, which may be described by:

y

where Sx and Sy are the coordinate scaling factors for the x and y directions, respectively, and (Cx , C_y) are the coordinates of the origin of the computer image coordinate system. Here, S_x and Sy, and (Cx , Cy) are some parameters related to the physical properties of the CCD sensors and the computer memory, respectively.

2.3.2 Proposed Calibration Process for Estimating Pose Parameters with Respect to Camera

As mentioned previously, we draw a calibration pattern on a paper ring and attach the ring on the mirror mount around the mirror border for use in the subsequent calibration process.

The shape of the calibration pattern consists of an inner circle with a diameter equal to that of the mirror, as well as 16 black marks of short line segments evenly distributed around the circle border. Each short line segment has an end point on the inner circle of the ring, which we call a calibration point. The configuration is shown in Fig. 2.3. An image of this calibration pattern is shown in Fig. 2.4. It is noted that only 12 marks are visible in the FOV of the camera.

The proposed calibration process in this study includes the following major steps.

(1) Acquisition of calibration pattern images

At the beginning of the calibration process, an image of the calibration pattern is taken. An example of calibration pattern images is shown in Fig. 2.4.

(2) Identification of calibration points

The calibration points on the base plane of the calibration pattern are then identified in the image. Let the coordinates of their projection points in the computer image coordinate system be denoted as (ufi, vfi), i = 0, 1, …, n. On the other hand, the base coordinates (xbi, ybi, zbi) of the calibration points are known in advance, with all the values of zbi being equal to zeros because the points are located on the base plane.

(3) Computation of physical parameters

Let the image size in the computer image coordinate system be wi × hi and the CCD sensor size be ws × hs. Then the parameters Sx, Sy and (Cx, Cy) in Eqs. (2.9) are calculated in this study

Fig. 2.3 The calibration pattern designed for use in this study.

Fig. 2.4 An omni-image of the calibration pattern.

in the following way:

(4) Computation of intrinsic and extrinsic parameters

The extrinsic parameters R and T in Eq. (2.4), the intrinsic parameters f in Eqs. (2.5), and the radial distortion factor

κ

in Eqs. (2.8) should be estimated by a certain calibration method.

This is accomplished in this study according to the method proposed in [13]. The steps are sketched here. First, from Eqs. (2.9) we get the distorted image coordinates (udi, vdi) of a calibration point in the computer image coordinate system as follows:

y

where (ufi, vfi) are the corresponding computer image coordinates. Next, we combine Eqs. (2.4) through (2.11) to derive the following equations:

z we can solve R, T,

κ

from Eqs. (2.12) by Tsai’s single view coplanar calibration method [13].

The parameter f is assumed available from the camera specifications.

Fig. 2.5 shows a calibration result of the pose of the base plane with respect to the camera, which includes the values (−2.99, 0.96, 88.67) of the translation vector T in the unit of mm

and the values (0.013, 0.035, 0.007) of the three rotation angles

φ

,

θ

, and

ψ

of the rotation matrix R in the unit of radian. The real coordinates of the 12 calibration points are described by the square-bracketed coordinates [xi

, y

i] in Fig. 2.5. After the calibration, the detected image coordinates of the calibration points are back-projected onto the base plane, the results are described by the angle-bracketed coordinates <xi

, y

i>, which are also shown in Fig. 2.5.

2.3.3 Proposed Calibration Process for Deriving Pose Parameters with Respect to Mirror

The pose of the base plane with respect to the camera is composed of the rotation matrix R and the translation vector T derived above. To obtain the pose of the camera with respect to the mirror, we have to transform Eq. (2.4) into a form similar to those specified in Eqs. (2.1).

The origin of the mirror coordinate system is defined at one focus of the hyperbolic mirror surface (denoted by Om in Fig. 2.2). The mirror plane z = 0 is taken to be parallel to the base plane at a distance of d = (b + h)

−

c. The z-axis of the mirror coordinate system is aligned with the z-axis of the base coordinate system.

It is known that R has the rotation angles (

φ

,

θ

,

ψ

) with respect to the x-, y-, and z-axes Fig. 2.5 The calibration result of a hypercatadioptric camera used in this study.

respectively, and T has the values (Tx, Ty, Tz). To map a point (x, y, z) in the camera coordinate system into a point (xb, yb, zb) in the base coordinate system, the following equation may be applied:

where the new rotation matrix

⎥ ⎥

Because the base plane and the mirror plane are apart with a distance of d, the coordinates (xb, yb, zb) of a point in the base coordinate system with origin Ob are related to the coordinates

Finally, given the coordinates (u, v, f) of a point I in the camera coordinate system where (u, v) is the ideal image coordinates of I, the corresponding coordinates (xi, yi, zi) of I in the mirror coordinate system, according to Eqs. (2.15), may be derived to be:

T

x

2.4 Back-Projection of Image Point

As a summary of the discussions in Section 2.3, we redraw the camera model as shown in Fig. 2.6 from the viewpoint of image projection. In Fig. 2.6, the angles of pitch

φ

c, yaw

θ

c, and tilt

ψ

c are respectively the negative values of the calibrated rotation angles in Eq. (2.4). When the pose of the camera with respect to the mirror is determined in a way as described in Section 2.3.3, a point I(u, v) in the image plane can uniquely determine a reflective ray Rr

from the mirror surface and so a corresponding mirror surface point M(xm, ym, zm). In turn, at point M there will be an incident ray Ri corresponding to Rr with its incident orientation being determined by the mirror surface geometry. Let the direction of Ri be specified by a unit vector denoted by

w

r = [w_u x

w

_y

w

_z]^T. In this section, we will derive a set of equations to specify a mapping F from the coordinates (u, v) of point I to the elements (wx, wy, wz) of the unit vector

w

r . To be simple, we denote this mapping as _u

w

r = F(I). This mapping is _u constrained, according to the optical reflection principle, by the following two rules.

(1) Co-planarity constraint: the unit normal

n r

of the mirror surface at point M and the two

rays, Ri and Rr, are co-planar.

(2) Reflection constraint: the incident angle of Ri is equal to the reflection angle of Rr. In the sequel, all the derived formulas are based on the mirror coordinate system.

2.4.1 Derivation of Unit Normal Vector n r

Fig. 2.7 depicts the unit normal vector

n r

at point M(xm, ym, zm) on a plane passing through the z-axis of the mirror coordinate system with the tilt angle

ϕ

, denoted as Pn. The vector

n r

can be decomposed into two orthogonal vectors nr_m and nr_z, where the vector nr_m

is on a plane PM perpendicular to Pn located at z = zm. and nr_z is parallel to the z-axis of the mirror coordinate system. The tilt angle by definition is equal to

Optical center O(x

cw

, y

cw

, z

cw

)

I(u, v) → (xi, yi, zi)

pitch φ

_c

yaw θ

_c

P(x

w

, y

w

, z

w

) M(x

m

, y

m

, z

m

)

Fig. 2.6 The image projection model, where Om is the origin of the mirror coordinate system, and Ob is the origin of the base coordinate system.

base plane O

b

W O

m

image plane (CCD) d

c

R

i

R

r

m

On the other hand, we want to derive the equation of the mirror surface in the mirror coordinate system. Eqs. (2.3) describes the mirror surface in the world coordinate system. So, a shift −c should be added to the z-value in Eqs. (2.3), resulting in

2 Because the mirror surface is rotationally symmetric in the x- and y-directions, we can consider the polar coordinates (rm, zm) only, i.e., point M may be thought to be located at (rm,

z

m). Also, let the tangent plane at point M perpendicular to

n r

be denoted as PT, and let the intersection line of PT and Pn be denoted as TM. Now, the value of the angle

δ

of TM with respect to the plane PM at point M with polar coordinates (rm, zm) on the mirror surface may be derived, by taking the inverse tangent value of a partial derivative of zm in Eqs. (2.19) with respect to rm, to be

Accordingly, we can derive the values sin

δ

and cos

δ

as follows:

2

2.4.2 Use of Co-Planarity Constraint

The co-planarity constraint on the unit normal

n r

of the mirror surface at point M and the two rays, Ri and Rr, is shown in Fig. 2.8, and can be described by the following equality according to vector analysis:

0

0

where “×” and “⋅” denote the cross and inner product operators for vectors, respectively;

w r

= [(xw

− x

m) (yw

− y

m) (zw

− z

m)]^T specifies the direction of the incident ray Ri; [i j k]^T is a unit vector; and

o r

= [(xcw

− x

m) (ycw

− y

m) (zcw

− z

m)]^T specifies the direction of the reflection ray Rr. By computing the above matrix product and substituting the result with the following notations

Fig. 2.8 The co-planar vectors and the cross product.

n r M(x

m

, y

m

, z

m

)

ϕ

2.4.3 Use of Reflection Constraint

The aforementioned reflection constraint, which indicates the identicalness of the incident angle to the reflection angle, may be expressed by the following equalities:

The aforementioned reflection constraint, which indicates the identicalness of the incident angle to the reflection angle, may be expressed by the following equalities:

在文檔中 攝影機校準及影像轉換技術與其應用之研究 (頁 36-0)