Analytic image unwarping by a systematic calibration method for omni-directional cameras with hyperbolic-shaped mirrors

(1)

Analytic image unwarping by a systematic calibration method

for omni-directional cameras with hyperbolic-shaped mirrors

Sheng-Wen Jeng

a,*

, Wen-Hsiang Tsai

a,b,1

a_{Department of Computer Science, National Chiao Tung University, 1001 Ta Hsueh Road, Hsinchu 300, Taiwan} b_{Department of Computer Science and Information Engineering, Asia University, Liufeng Road, Wufeng, Taichung 41354, Taiwan}

Received 12 November 2004; received in revised form 3 August 2007; accepted 8 August 2007

Abstract

Unwarping an directional image into a perspective-view one is easy for a single-viewpoint (SVP) designed catadioptric omni-directional camera. But misalignment between the components (such as the mirror and the lens) of this kind of camera creates multiple viewpoints and distorts the unwarped image if the SVP constraint is assumed. The SVP constraint is relaxed in this study and a system-atic method is proposed to derive a set of new and general analytic equations for unwarping images taken from an omni-directional cam-era with a hyperbolic-shaped mirror (called a hypercatadioptric camcam-era). The derivation is made possible by careful investigation on the system configuration and precise calibration of involved system parameters. As a verification of the correctness of the derived equations, some of the system parameters are adjusted to fit the SVP constraint, and unwarped images using the resulting simplified camera model are shown to be of no difference from those obtained by a method based on the SVP model. The generality of the proposed method so has extended the image-unwarping capability of the existing methods for the hypercatadioptric camera to tolerate lens/mirror assembly imprecision, which is difficult to overcome in most real applications. Some experimental results of image unwarping are also included to show the effectiveness of the proposed method.

Keywords: Omni-directional camera; Single-viewpoint (SVP) constraint; Image unwarping; Hyperbolic mirror; Hypercatadioptric camera; Camera cal-ibration; Analytic solution

1. Introduction

It is well known in the computer vision ﬁeld that enlarg-ing the ﬁeld of view (FOV) of a camera enhances the visual coverage, reduces the blind area, and saves the computa-tion time, of the camera system, especially in applicacomputa-tions like visual surveillance and vision-based robot or autono-mous vehicle navigation.

There are many ways to design a camera system consist-ing of CCD sensors, lenses, and mirrors to increase the FOV of the system[1]. An extreme way is to expand the

FOV to a full hemisphere by the use of a catadioptric cam-era, which is an integration of a CCD sensor chip, a convex reflection mirror, and a projection lens. A popular name for this kind of camera is omni-directional camera, or sim-ply camera, and that for an image taken by it is omni-directional image, or simply omni-image. The surface curve of the reflection mirror in such a kind of camera may be conical, spherical, parabolic, or hyperbolic, and the lens may be of the type of orthographic or perspective projec-tion. To simplify the process for unwarping omni-images into perspective ones, it is usually desired to design an omni-camera in such a way that the single-viewpoint (SVP) constraint is satisfied[2].

Only some of the possible mirror/lens combinations can ﬁt the SVP constraint, for examples, a combination of a parabolic mirror and an orthographic lens or that of a

*

Corresponding author. Tel.: +886 6 3847167.

E-mail addresses: [email protected] (S.-W. Jeng), [email protected]. edu.tw(W.-H. Tsai).

1 _{Tel.: +886 3 5728368.}

www.elsevier.com/locate/imavis Image and Vision Computing 26 (2008) 690–701

(2)

hyperbolic mirror and a perspective lens [2]. However, because of the diﬃculty in the alignment of the mirror and the camera lens, many commercial products do not sat-isfy the SVP constraint. When this case occurs, the resulting locus of viewpoints will form a so-called caustic curve [3]

and the corresponding image unwarping work becomes very complicated. On the other hand, when the parabolic mirror/orthographic lens combination is used, the resulting system is called a paracatadioptric camera[8]. Following this idea of naming the camera system, when the hyperbolic mirror/perspective lens combination is used, the resulting system is called a hypercatadioptric camera in this paper. In this study, we deal with the image-unwarping problem for a hypercatadioptric camera in a non-SVP system.

More speciﬁcally, we propose in this study a systematic method to calibrate the system parameters of a hypercata-dioptric camera and derive accordingly a set of equations for accurate image unwarping. In the proposed camera cal-ibration process, a calcal-ibration pattern of the shape of a thin ring is designed and attached at the border of the mirror as an aid. Next, mirror reﬂection laws as well as system geom-etry constraints are utilized to derive a set of mapping equations between a pixel in the image coordinate system and a point in the world space. The calibrated system parameters are used as known parameters in the deriva-tion. The derived equations are then used to unwarp accu-rately an omni-image taken by a hypercatadioptric camera into a perspective-view image from any viewpoint.

Several contributions are made by the proposed method. The first is that the derived image-unwarping equations are analytic. This is achieved for the first time, and the compu-tation involved in the image unwarping work will so become faster. Another contribution is that unwarping of omni-images taken by a hypercatadioptric camera into per-spective-view images will not be confined to the SVP con-straint. And this makes the applicability of the hypercatadioptric camera much wider to various computer vision problems. Finally, the generality of the proposed method for non-SVP cameras has extended the image-unwarping capability of the existing methods for the hyper-catadioptric camera to tolerate lens/mirror assembly imprecision, which is difficult to overcome in most real applications.

The remainder of this paper is organized as follows. In Section2, we review the basic concepts about SVP omni-cameras and some previous works for omni-camera cali-bration. The camera calibration process proposed in this study is described in Section 3. In Section 4, the corre-sponding analytic image-unwarping equations are derived. In Section 5, some experimental results using simulation data as well as real images are given. Finally, some conclu-sions are made in Section6.

2. Review of previous works

For an SVP catadioptric camera, unwarping an omni-image into a perspective version is a process of forward

projection from a point Xp on a certain perspective-view

plane in the world space to an omni-image point Xi, which

can be described by Xi= h(Xp) with h being a one-to-one

mapping function from the world space to the omni-image plane [4,5]. For example, for an SVP hypercatadioptric camera, the mapping relation between a point Xp(x, y, z)

in a world space and its projection point Xi(u, v) in the

image plane, as illustrated in Fig. 1, is as follows:

u¼ fðb 2_c2_Þx ðb2_{þ c}2_{Þz 2bc}pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi_x2_{þ y}2_{þ z}2; v¼ fðb 2 c2_Þy ðb2_{þ c}2_{Þz 2bc}pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi_x2_{þ y}2_{þ z}2; ð1Þ

where f is the focal length of the camera lens, and a, b, and c are the parameters of the hyperbolic curve of the mirror surface described as follows:

z¼ c þ b ffiffiffiffiffiffiffiffiffiffiffiffiffi 1þr 2 a2 r ; r2_{¼ x}2_{þ y}2 _ð2Þ with c¼pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffia2_{þ b}2_.

In practical situations, because of the existence of the geometric lens distortion, the projection point I(u, v) in the image plane might be shifted erroneously. So, the real position of the point I in the image coordinate system should be calibrated by proper geometric correction for accurate image unwarping even in the SVP case. Some techniques for this purpose can be found in [6,7] and are followed in this study. The details are omitted.

On the other hand, to estimate the intrinsic parameters of a paracatadioptric omni-camera system, a calibration procedure should be performed before unwarping omni-images into perspective-view ones. In [8,9], using a single view of three lines, Geyer et al. derived analytic calibration solutions for the focal length, the image center, and the aspect ratio of a paracatadioptric camera. In [10], Kang

O

c

f

c

b

O

w

P(x, y, z)

I(u, v)

View plane Omni-image

Fig. 1. An SVP hypercatadioptric camera where Owis the origin of the

world coordinate system (also one focus of the hyperbolic curve), and Ocis

(3)

used the consistency condition of pair-wise tracked point features across a sequence of paracatadioptric images to calibrate the same parameters. These approaches basically deal with the calibration problem of an SVP paracatadiop-tric camera, and misalignment between the mirror and the camera components (including the lens and the CCD sen-sor) was not considered. That is, the image plane was assumed to be parallel to the base plane of the mirror in these approaches, and only the intrinsic parameters of the cameras were taken into account in the calibration. The quality of the unwarped image is severely degraded when equations derived from a system conﬁguration not meeting such an SVP assumption are used in the unwarping pro-cess, even when the intrinsic parameters of the camera have been calibrated.

On the contrary, when a non-SVP camera is used, for example, for the reason to increase the FOV, the system configuration parameters related to the pose of the mirror relative to that of the camera, in addition to the intrinsic camera parameters, need be calibrated. In [11], Aliaga developed a calibration model using a beacon-based pose estimation algorithm for a catadioptric camera which includes a parabolic mirror and a perspective lens. This mirror/lens combination is a non-SVP design, and the adopted camera model, like Tsai’s [6], has eleven parame-ters, five intrinsic and six extrinsic. But the physical mean-ings of Aliaga’s extrinsic parameters are different from those of Tsai’s, with the translation vector representing the offset between the center point of the mirror base plane and that of the image plane, and the rotation vector repre-senting the orientation of the mirror base plane with respect to a world space system. Also, the mirror base plane is assumed to be parallel to the image plane. The cal-ibrated data were used to estimate the pose of the camera with respect to the world space system.

A more complete calibration procedure for a catadiop-tric camera with a parabolic mirror and a perspective lens, which estimates the intrinsic camera parameters and the pose of the mirror relative to the camera, appeared in Fab-rizio et al. [12]. The images of two circles on two planes existing in the mirror were used to calibrate the intrinsic camera parameters and the system conﬁguration parame-ters. But no discussion was made about how to use the cal-ibrated parameters to modify the mapping described by Eqs. (1) to get an accurate unwarped perspective-view image from an omni-image.

3. Proposed method for calibrating camera pose with respect to mirror

In this section, the proposed method for calibrating the camera pose with respect to the mirror of a hypercatadioptric camera system is described. The system configuration and the relationships among the involved coordinate systems are described first, and the proposed calibration process is presented next. The camera pose with respect to the mirror is derived finally, using the calibrated system parameters.

3.1. System conﬁguration and coordinate system relationships

The configuration of a hypercatadioptric camera and the related coordinate systems used in this study are depicted inFig. 2. First, we define a world coordinate sys-tem with its origin W taken to be the middle point between the foci of the two arms of the hyperbolic curve defined by the mirror surface. Let b be the distance from W to the tip Tmof the mirror, c the distance from W to a focus Omof an

arm of the hyperbolic curve, h the height of the mirror (measured at Tm), and m the radius of the circular-shaped

mirror base. Then, a point M(xm, ym, zm) on the mirror

sur-face with respect to W can be described by the following equations according to Eq.(2):

zm¼ b ffiffiffiffiffiffiffiffiffiffiffiffiffi 1þr 2 m a2 r ; r2_m¼ x2 mþ y 2 m; ð3Þ

where a¼pffiffiffiffiffiffiffiffiffiffiffiffiffiffic2_b2_{. The optical center O}

c of the camera

lens is taken to be the origin of the 3D camera coordinate system, and the optical axis of the camera is assumed to align with the z-axis of the world coordinate system. Accordingly, the center Oi(u0, v0) of the 2D image

coordi-nate system, which is the projection point of the optical axis on the image plane described by z = f, is (0, 0). The mirror parameters a, b, h, and m, and the physical size of the CCD sensor may be obtained from the speciﬁcations of the hypercatadioptric camera.

Next, we deﬁne a base coordinate system on the mirror with its origin taken to be the center C of the bottom circle of the mirror. The base plane of the mirror is located at the plane z = 0 of the base coordinate system. A point P(xb, yb, zb) on the ring-shaped calibration pattern on the

base plane with respect to the origin of the camera coordi-nate system can be expressed as follows:

½x y zT¼ R½xb yb zb T þ T ; ð4Þ (uf, vf) I(u, v) (ud, vd) Oi(uo, vo) calibration pattern image plane (CCD) m h b f M(xm, ym, zm) P(xw, yw, zw) Oc W Tm c C P(xb, yb, zb) Om →

Fig. 2. The conﬁguration of a hypercatadioptric camera used in this study.

(4)

where R is a 3· 3 rotation matrix with three rotation an-gles / (pitch), h (yaw), and w (tilt) around the x-, y-, and z-axes of the base coordinate system, respectively, and T is a translation vector described by T = [Tx Ty Tz]T. Eq.

(4)respresents a relationship from the base coordinte sys-tem to the camera coordinate syssys-tem. We will transform the relationship into one from the camera coordinate sys-tem to the base coordinte syssys-tem in Section3.3, which rep-resents the pose of the camera with respect to the mirror.

On the other hand, the location of the projection point I(u, v) in the image plane of a point P(x, y, z) in the camera coordinate system can be described as follows:

u¼ fx

z; v¼ f y

z: ð5Þ

To correct possible geometric distortion of the lens in the radial direction, the following distortion model [6] is adopted in this study:

ud ¼ u þ Dx; vd¼ v þ Dy;

where udand vd are the shifted versions of u and v in the

image coordinate system, and Dxand Dyare the amounts

of distortion estimated, according to[6], by Dx¼ judr2; Dy ¼ jvdr2

with r2_{¼ u}2 dþ v

2

d and j being the radial distortion factor of

the lens. Combining the above equations, we get the fol-lowing equations:

u¼ ð1 jr2_Þu

d; v¼ ð1 jr2Þvd: ð6Þ

In the sequel, (u, v) will be called ideal image coordinates, and (ud, vd) distorted image coordinates. Finally, since the

unit of the image coordinates (uf, vf) used in the computer,

called computer image coordinates hereafter, is ‘‘pixel’’ for discrete images kept in the computer, additional relations between the distorted image coordinates (ud, vd) and the

computer image coordinates (uf, vf) must be speciﬁed,

which may be described by:

uf ¼ Sxudþ Cx; vf ¼ Syvdþ Cy; ð7Þ

where Sxand Syare the coordinate scaling factors for the x

and y directions, respectively, and (Cx, Cy) are the

coordi-nates of the origin of the computer image coordinate sys-tem. Here, Sx and Sy, and (Cx, Cy) are some parameters

related to the physical properties of the CCD sensors and the computer memory, respectively.

3.2. Proposed calibration process for estimating pose parameters with respect to camera

As mentioned previously, we draw a calibration pattern on a paper ring and attach the ring on the mirror mount around the mirror border for use in the subsequent calibra-tion process. The shape of the calibracalibra-tion pattern consists of an inner circle with a diameter equal to that of the mir-ror, as well as 16 black marks of short line segments evenly distributed around the circle border. Each short line

seg-ment has an end point on the inner circle of the ring, which we call a calibration point. The conﬁguration is shown in

Fig. 3. An image of this calibration pattern is shown in

Fig. 4. It is noted that only 12 marks are visible in the FOV of the camera.

The proposed calibration process in this study includes the following major steps.

(1) Acquisition of calibration pattern images At the begin-ning of the calibration process, an image of the cali-bration pattern is taken. An example of calicali-bration pattern images is shown in Fig. 4.

(2) Identiﬁcation of calibration points The calibration points on the base plane of the calibration pattern are then identiﬁed in the image. Let the coordinates of their projection points in the computer image coor-dinate system be denoted as (ufi, vfi), i = 0, 1, . . ., n. On

the other hand, the base coordinates (xbi, ybi, zbi) of

the calibration points are known in advance, with all the values of zbibeing equal to zeros because the

points are located on the base plane.

Fig. 3. The calibration pattern designed for use in this study.

(5)

(3) Computation of physical parameters Let the image size in the computer image coordinate system be wi· hi

and the CCD sensor size be ws· hs. Then the

param-eters Sx, Syand (Cx, Cy) in Eqs.(7)are calculated in

this study in the following way: Sx¼ wi ws ; Sy ¼ hi hs ; Cx¼ wi 2 ; Cy¼ hi 2:

(4) Computation of intrinsic and extrinsic parameters The extrinsic parameters R and T in Eq.(4), the intrinsic parameters f in Eqs.(5), and the radial distortion fac-tor j in Eqs.(6)should be estimated by a certain cal-ibration method. This is accomplished in this study according to the method proposed in [6]. The steps are sketched here. First, from Eqs.(7)we get the dis-torted image coordinates (udi, vdi) of a calibration

point in the computer image coordinate system as follows: udi¼ ufi Cx Sx ; vdi¼ vfi Cy Sy ; ð8Þ

where (ufi, vfi) are the corresponding computer image

coor-dinates. Next, we combine Eqs.(4)–(8)to derive the follow-ing equations: udið1 þ jr2iÞ ¼ ðr11xbiþ r12ybiþ r13zbiþ TxÞf r31xbiþ r32ybiþ r33zbiþ Tz ; vdið1 þ jr2iÞ ¼ ðr21xbiþ r22ybiþ r23zbiþ TyÞf r31xbiþ r32ybiþ r33zbiþ Tz ; ð9Þ where r2 i ¼ u 2

diþ v2di. With suﬃcient known pairs of (udi, vdi)

and (xbi, ybi, zbi), i = 0, 1, . . ., n, we can solve R, T, j from

Eqs.(9)by Tsai’s single view coplanar calibration method

[6]. The parameter f is assumed available from the camera speciﬁcations.

Fig. 5shows a calibration result of the pose of the base plane with respect to the camera, which includes the values (2.99, 0.96, 88.67) of the translation vector T in the unit of mm and the values (0.013, 0.035, 0.007) of the three rota-tion angles /, h, and w of the rotarota-tion matrix R in the unit of radian. The real coordinates of the 12 calibration points are described by the square-bracketed coordinates [xi, yi] in

Fig. 5. After calibration, the detected image coordinates of the calibration points are back-projected onto the base plane, and the results are described by the angle-bracketed coordinatesÆxi, yiæ, which are also shown inFig. 5.

3.3. Proposed calibration process for deriving pose parameters with respect to mirror

The pose of the base plane with respect to the camera is composed of the rotation matrix R and the translation vec-tor T derived above. To obtain the pose of the camera with respect to the mirror, we have to transform Eq.(4)into a form similar to those speciﬁed in Eqs. (1). The origin of the mirror coordinate system is deﬁned at one focus of

the hyperbolic mirror surface (denoted by Om in Fig. 2).

The mirror plane z = 0 is taken to be parallel to the base plane at a distance of d = (b + h) c. The z-axis of the mir-ror coordinate system is aligned with the z-axis of the base coordinate system.

It is known that R has the rotation angles (/, h, w) with respect to the x-, y-, and z-axes respectively, and T has the values (Tx, Ty, Tz). To map a point (x, y, z) in the camera

coordinate system into a point (xb, yb, zb) in the base

coor-dinate system, the following equation may be applied: xb y_b zb 2 6 4 3 7 5 ¼ r0 11 r012 r013 r0₂₁ r0₂₂ r0₂₃ r0 31 r032 r033 2 6 4 3 7 5 x y z 2 6 4 3 7 5 Tx Ty Tz 2 6 4 3 7 5;

where the new rotation matrix

R0¼ r0 11 r012 r013 r0 21 r022 r023 r0 31 r032 r033 2 6 4 3 7 5

is obtained by reversing the signs of (/, h, w) in Eq.(4). Because the base plane and the mirror plane are apart with a distance of d, the coordinates (xb, yb, zb) of a point

in the base coordinate system with origin C are related to the coordinates (xw, yw, zw) of a point in the mirror

coordi-nate system with origin Omby the following equalities:

xw¼ xb¼ xr011þ yr 0 12þ zr 0 13 Tx; y_w¼ yb¼ xr021þ yr022þ zr023 Ty; ð10Þ zw¼ zbþ d ¼ xr031þ yr 0 32þ zr 0 33 Tzþ ðb þ hÞ c:

So, the position (xcw, ycw, zcw) of the camera origin Ocin

the mirror coordinate system may be derived from that of the mirror origin Om by setting (x, y, z) in Eqs. (10) to be

(0, 0, 0):

xcw¼ Tx; ycw¼ Ty; zcw¼ Tzþ ðb þ hÞ c: ð11Þ

Fig. 5. The calibration result of a hypercataoptric camera used in this study.

(6)

Finally, given the coordinates (u, v, f) of a point I in the camera coordinate system where (u, v) is the ideal image coordinates of I, the corresponding coordinates (xi, yi, zi)

of I in the mirror coordinate system, according to Eqs.

(10), may be derived to be: xi¼ ur011þ vr 0 12þ fr 0 13 Tx; y_i¼ ur0 21þ vr022þ fr 0 23 Ty; ð12Þ zi¼ ur031þ vr 0 32þ fr 0 33 Tzþ ðb þ hÞ c:

4. Back-projection of image point

As a summary of the discussions in Section3, we redraw the camera model as shown inFig. 6from the viewpoint of image projection. InFig. 6, the angles of pitch /c, yaw hc,

and tilt wcare respectively the negative values of the

cali-brated rotation angles in Eq. (4). When the pose of the camera with respect to the mirror is determined in a way as described in Section3.3, a point I(u, v) in the image plane can uniquely determine a reﬂective ray Rrfrom the mirror

surface and so a corresponding mirror surface point M(xm, ym, zm). In turn, at point M there will be an incident

ray Ri corresponding to Rr with its incident orientation

being determined by the mirror surface geometry. Let the direction of Ri be speciﬁed by a unit vector denoted by

~wu¼ ½wxwywz T

. In this section, we will derive a set of equa-tions to specify a mapping F from the coordinates (u, v) of point I to the elements (wx, wy, wz) of the unit vector ~wu. To

be simple, we denote this mapping as ~wu¼ FðIÞ. This

map-ping is constrained, according to the optical reﬂection prin-ciple, by the following two rules.

(1) Co-planarity constraint: the unit normal ~nof the mir-ror surface at point M and the two rays, Ri and Rr,

are co-planar.

(2) Reﬂection constraint: the incident angle of Riis equal

to the reﬂection angle of Rr.

In the sequel, all the derived formulas are based on the mirror coordinate system.

4.1. Derivation of unit normal vector ~n

Fig. 7 depicts the unit normal vector ~n at point M(xm, ym, zm) on a plane passing through the z-axis of

the mirror coordinate system with the tilt angle u, denoted as Pn. The vector ~ncan be decomposed into two

orthogo-nal vectors ~nmand ~nz, where the vector ~nmis on a plane PM

perpendicular to Pn located at z = zm and ~nzis parallel to

the z-axis of the mirror coordinate system. The tilt angle by deﬁnition is equal to

u¼ tan1ym xm

: ð13Þ

On the other hand, we want to derive the equation of the mirror surface in the mirror coordinate system. Eqs.(3) de-scribes the mirror surface in the world coordinate system. So, a shift c should be added to the z-value in Eqs.(3), resulting in zm¼ c þ b ffiffiffiffiffiffiffiffiffiffiffiffiffi 1þr 2 m a2 r ; r2_m¼ x2 mþ y 2 m; ð14Þ

where rmmay be regarded as a polar coordinate composed

of the coordinates of xmand ym.

Because the mirror surface is rotationally symmetric in the x- and y-directions, we can consider the polar coordi-nates (rm, zm) only, i.e., point M may be thought to be

Optical center O(xcw, ycw, zcw) I(u, v)→ (xi, yi, zi) pitchφc yawθc tiltψc P(xw, yw, zw) M(xm, ym, zm) base plane Ob W Om image plane (CCD) d c Ri Rr

Fig. 6. The image projection model where Omis the origin of the mirror

coordinate system, and Obis the origin of the base coordinate system.

n

δ rm2 = xm2+ ym2 Om ϕ M(xm, ym, zm) z n TM Pn m n PM PT

(7)

located at (rm, zm). Also, let the tangent plane at point M

perpendicular to ~n be denoted as PT, and let the

intersec-tion line of PT and Pnbe denoted as TM. Now, the value

of the angle d of TMwith respect to the plane PMat point

M with polar coordinates (rm, zm) on the mirror surface

may be derived, by taking the inverse tangent value of a partial derivative of zmin(14)with respect to rm, to be

d¼ tan1 @zm @rm ¼ tan1 brm a ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffir2 mþ a2 p ¼ tan1b 2_r m a2_z m : ð15Þ

Accordingly, we can derive the values sin d and cos d as follows: sin d¼ ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffibrm a4_{þ c}2_r2 m p ;cos d¼ a ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi a2_{þ r}2 m p ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi a4_{þ c}2_r2 m p : ð16Þ

Finally, it is not diﬃcult to derive the unit normal vector at point M(xm, ym, zm) to be~n¼ ½sin d cos u sin d sin u cos d

T

according to the geometry shown inFig. 7.

4.2. Use of co-planarity constraint

The co-planarity constraint on the unit normal ~nof the mirror surface at point M and the two rays, Riand Rr, is

shown in Fig. 8, and can be described by the following equality according to vector analysis:

ð~o ~nÞ ~w¼ 0; or equivalently,

i j k

ðxcw xmÞ ðycw ymÞ ðzcw zmÞ

sin d cos u sin d sin u cos d 2 6 4 3 7 5 ðxw xmÞ ðyw ymÞ ðzw zmÞ 2 6 4 3 7 5 ¼ 0;

where ‘‘·’’ and ‘‘Æ’’ denote the cross and inner product oper-ators for vectors, respectively; ~w¼ ½ðxw xmÞ ðyw ymÞ

ðzw zmÞT speciﬁes the direction of the incident ray Ri;

[i j k]T is a unit vector; and ~o¼ ½ðxcw xmÞðycw ymÞ

ðzcw zmÞT speciﬁes the direction of the reﬂection ray Rr.

By computing the above matrix product and substituting the result with the following notations

Km1¼ ðym ycwÞ cos d þ ðzm zcwÞ sin d sin u;

Km2¼ ðxm xcwÞ cos d þ ðzm zcwÞ sin d cos u;

Km3¼ ðxcw xmÞ sin d sin u ðycw ymÞ sin d cos u;

xn¼ ðxw xmÞ; yn¼ ðyw ymÞ; zn¼ ðzw zmÞ;

we get

Km1xn Km2ynþ Km3zn¼ 0: ð17Þ

4.3. Use of reﬂection constraint

The aforementioned reﬂection constraint, which indi-cates the identicalness of the incident angle to the reﬂection angle, may be expressed by the following equalities:

~ w ~n k~wkk~nk¼ cos q; cos q¼ o * ~n k o*kk~nk; ð18Þ where q denotes the two identical angles. The second equal-ity in Eqs.(18)may be expanded to be

cos q¼ðxcw xmÞ sin d cos u þ ðycw ymÞ sin d sin u ðzcw zmÞ cos d ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ðxcw xmÞ 2 þ ðycw ymÞ 2 þ ðzcw zmÞ 2 q : ð19Þ On the other hand, the three components of the unit vector ~wu¼_k~~w_wk¼ ½wxwywz

T

, by definition, can be calculated as follows: wx¼ xn ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi x2 nþ y2nþ z2n p ; wy¼ y_n ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi x2 nþ y2nþ z2n p ; wz¼ zn ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi x2 nþ y2nþ z2n p :

Then, the ﬁrst equality in Eqs.(18)can be derived to be as follows:

~w ~n

k~wkk~nk¼ ~wu ~n;

¼ ½wxwywzT ½sin d cos u sin d sin u cos dT;

¼ wxsin d cos uþ wysin d sin u wzcos d;

¼ cos q; ð20Þ

where cos q can be computed by Eq.(19)above.

4.4. Calculating direction of incident ray

If the values (xn, yn, zn) are not equal to (0, 0, 0), Eq.(17)

may be rewritten as Km1 xn ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi x2 nþ y2nþ z2n p Km2 y_n ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi x2 nþ y2nþ z2n p þ Km3 zn ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi x2 nþ y2nþ z2n p ¼ 0; which is equivalent to Km1wx Km2wyþ Km3wz¼ 0: ð21Þ

On another hand, the norm of the unit vector ~wuis equal to

1, i.e., w2_xþ w2 yþ w 2 z ¼ 1: ð22Þ

w

(Ri)

o

(Rr)

o

×

n

ρ ρ _P(x w, yw, zw) O(xcw, ycw, zcw)

n

M(xm, ym, zm) ϕ

(8)

Using Eqs. (20)–(22), we can solve the three unknown parameters wx, wy, and wzin the following way.

First, we eliminate the unknown wzin Eqs.(20) and (21)

to get

wy¼ Amwxþ Bm; ð23Þ

where Am¼

Km1cos dþ Km3sin d cos u

Km2cos d Km3sin d sin u

;

Bm¼

Km3cos q

:

Next, we eliminate the unknown wyin Eqs.(20) and (21)to

get

wz¼ Cmwxþ Dm; ð24Þ

where Cm¼

ðKmsin uþ Km2cos uÞ sin d

;

Dm¼

Km2cos q

:

Finally, substituting Eqs.(23) and (24) into Eq.(22) and reducing the result, we get

wx¼ ðAmBmþ CmDmÞ ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ðAmBmþ CmDmÞ2 ð1 þ A2mþ C 2 mÞðB 2 mþ D 2 m 1Þ q ð1 þ A2 mþ C2mÞ : ð25Þ

There are two possible solutions for wx, and using the

rela-tionship between the coordinates (xi, yi, zi) of the image

point and the coordinates (xcw, ycw, zcw) of the optical

cen-ter, all in the mirror coordinate system, we can determine one of them as the correct solution. The details are omitted here. After wx is obtained, wy and wz can be computed

accordingly by Eqs.(23) and (24).

4.5. Calculating coordinates of mirror surface point in terms of image point coordinates

In Sections 4.1–4.4, we have derived the elements (wx, wy, wz) of the unit vector ~wuin terms of the coordinates

(xm, ym, zm) of the mirror surface point M. Here, we further

want to derive (xm, ym, zm) in terms of the coordinates (u, v)

of the image point I to complete the derivations of the for-mulas for specifying the mapping ~wu¼ FðIÞ. The

coordi-nates (u, v) can be calculated from a point (uf, vf) in the

computer image coordinate system by Eqs. (8) and (6). Also, the coordinates (xi, yi, zi) of the image point I in the

mirror coordinate system can be calculated from Eqs.

(12)which are repeated in the following: xi¼ ur011þ vr012þ fr 0 13 Tx; y_i¼ ur0 21þ vr 0 22þ fr 0 23 Ty; zi¼ ur031þ vr032þ fr 0 33 Tzþ ðb þ hÞ c:

Now, referring toFig. 8, we see that both the tilt angle of the point I and that of its back-projection point M on the

mirror surface relative to the camera coordinate system are equal. Let both angles be denoted by /. Then, it is easy to see from the geometry in the ﬁgure that

tan /¼yi ycw xi xcw ¼ym ycw xm xcw ; or equivalently, that y_m¼ ycw xcwtan /þ xmtan /: ð26Þ

Combining Eqs. (12) and (26) and using the following notations K1¼ ycw xcwtan /; K2¼ ðzi zcwÞ xi xcw ; K3¼ zcwþ c xcwK2; K4¼ b2ð1 þ tan2/Þ a2K22; K5¼ b2K1tan / a2K2K3; K6¼ a2b2þ b2K21 a 2_K2 3;

we get, after some derivations and reductions, the follow-ing result for xm:

xm¼ K5 ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi K2₅ K4K6 q K4 : ð27Þ

There are two possible solutions for xmin the above

equa-tion, and we can get the correct one by checking the condi-tion that xmand xiare at the same side with respect to xcw,

or equivalently, that the value of the product (xm xcw)

(xi xcw) is larger than or equal to zero. Also, using Eq.

(26), we can get ym. And ﬁnally the value of zmcan be

cal-culated from Eqs.(14)which are repeated as follows:

zm¼ c þ b ffiffiffiffiffiffiffiffiffiffiffiffiffi 1þr 2 m a2 r ; r2_m¼ x2 mþ y 2 m: ð28Þ 5. Experimental results

We show in this section the experimental results of two unwarping cases with two diﬀerent omni-images as inputs, one being a pseudo-image and the other a real image taken by our hypercatadioptric camera.

5.1. Unwarping of a pseudo-image into perspective views We ﬁrst describe how we create the pseudo-image for the ﬁrst unwarping experiment. For this purpose, we used the calibration data obtained in Section 3, which include the translation parameters (2.99, 0.96, 88.67) (in the unit of mm); the rotation angles (0.013, 0.035, 0.007) (in the unit of radian); the radial distortion factor j = 0.0; and the focal length f = 2.9 mm. Then, we used the mapping equa-tions obtained in Section 4 to warp a pseudo target as shown in Fig. 9 into the image plane to get the desired pseudo omni-image as shown in Fig. 10. The procedure

(9)

was mentioned in Section5.2, and the details are described in the following.

The pseudo target includes two parts, namely, a ground region with the area of 20· 20 m2 and consisting of 400 grids with each being of the size of 1· 1 m2, as well as an L-shaped wall with the height of 1.0 m and a side width of 2.8 m. The L-shaped wall is placed near the center of the ground region. In simulating the image taking work, the target was laid under our hypercatadioptric camera and the normal vector of the ground region at the region center aligns with the z-axis of the mirror coordinate sys-tem. The distance of the region center from the origin of the mirror coordinate system is 2 m. The resulting pseudo-image ofFig. 10is of the size of 640· 480 pixels.

We then describe how we unwarped the pseudo-image into perspective-view images. We selected two perspec-tive-view planes, one being from a side view and the other from the top view. As shown inFig. 11, each perspective-view plane is a rectangular region, which was used to cap-ture the rays back-projected from the image plane. Each rectangular region was divided into 320· 240 units repre-senting a 320· 240 image.

The top-view region RT is 4· 4 m2 in size, parallel to

the x–y plane of the mirror coordinate system. The nor-mal vector of RT at the center CT of RT aligns with the

z-axis of the mirror coordinate system, and CT is 2 m

below the origin of the mirror coordinate system.

Fig. 12(a–c) are the unwarping results in RTwith diﬀerent

calibration parameter settings. Here, Fig. 12(a) was pro-duced using the same translation and rotation parameters as those used in yieldingFig. 10. In Fig. 12(b), the three rotation angles were all set to be zero. And inFig. 12(c), we further set the two translation parameters Tx and Ty

to be zero (meaning perfect alignment of the camera with respect to the mirror). We can see in Fig. 12(a) that our derived equations can be used to unwarp the pseudo-image of Fig. 10 perfectly within the top-view region RT. Fig. 12(b) and (c) tell us that insuﬃcient calibration

of the hypercatadioptric camera will produce distorted unwarping results.

On the other hand, we show some results coming from inappropriate unwarping of the input pseudo-image

Fig. 10using Eqs.(1)under the erroneous assumption that the camera is an SVP system. They are shown inFig. 12(d– f), which are the results coming from the uses of three dif-ferent CCD sensors of sizes 3.2· 2.4, 1.6 · 1.2, and 0.8· 0.6 mm2

, respectively. The actual size of our CCD camera sensor is the ﬁrst one, namely, 3.2· 2.4 mm2and the corresponding unwarping result is the image shown in

Fig. 12(d). But the visible scope of the image region in the ﬁgure is too small to show the entire unwarping result. For the reason of comparison, we therefore assume the other two sensor sizes for our camera to yield Fig. 12(e) and (f) for the purpose of showing the unwarping results more clearly. Note that the sensor size settings of

Fig. 12(e) and (f) are unreasonable for a real CCD sensor. FromFig. 12(f), it is obviously seen that the result is worse than the perfect one shown inFig. 12(a).

Fig. 13(a) through (d) show the unwarping results on four perspective-view planes. Each perspective-view plane

Fig. 10. The warped image of the pseudo target inFig. 9.

P(xw, yw, zw) M(xm, ym, zm) Optical center O(x, y, z) View plane (side view) View plane (top view) image plane (CCD) PI

R

T

C

T f

Fig. 11. View planes deﬁned in the real world for unwarped images.

20m

Fig. 9. A pseudo target of size 20· 20 m with an L-shaped wall at the center position.

(10)

is set parallel to the z-axis of the mirror coordinate sys-tem with a view-angle span of 90 in the x–y plane and at a distance of pffiffiffi2m from the z-axis of the mirror coor-dinate system. The unwarping result is projected into a region in each perspective-view plane with a height of 2 m and a width of 2pffiffiffi2m. Fig. 13(a) through (d) are the unwarping results using our derived equations in the four perspective-view planes. The settings of the

translation parameters and the rotation angles are the same as those for Fig. 12(a) through (d). Fig. 13(e) and (f) are the unwarping results by Eqs. (1) in the same perspective-view planes as those used for Fig. 13(b) and (c) but under the SVP assumption and with the CCD sensor size of 0.8· 0.6 mm2

. Note especially that the ver-tical lines in Fig. 13(e) and (f) can be seen to be slanted erroneously.

Fig. 12. Unwarped images ofFig. 10(top view), (a) has the same T(Tx, Ty, Tz) and R(Rx, Ry, Rz) as those for yieldingFig. 10, (b) has the setting

Rx= Ry= Rz= 0, and (c) has the further setting Tx= Ty= 0, (d–f) Results using Eq.(1)with diﬀerent CCD sensor sizes of 3.2· 2.4, 1.6 · 1.2, and

0.8· 0.6, respectively.

Fig. 13. Unwarped images ofFig. 10from 4 side views. (a–d) Results using the proposed method with the same T and R as those for yieldingFig. 10. (a–d) Results with diﬀerent span of view angle, 1.5p 2.0p, 1.0p 1.5p, 0.5p 1.0p, and 0.0p 0.5p, respectively. (e) and (f) Results using Eqs.(1)with diﬀerent spans of viewing angle, as (b) and (c), respectively.

(11)

5.2. Unwarping of a real image into perspective views

Figs. 14 and 15are the unwarping results of a real image shown previously in Fig. 4 taken by our camera. The parameter settings used for computingFigs. 14 and 15are the same as those used to produceFigs. 12 and 13, except that the pseudo-image is now replaced by the real image. Or more speciﬁcally, Fig. 14(a) through (f) correspond to

Fig. 12(a) through (f), respectively, andFig. 15(a) through

(f) to Fig. 13(a) through (f), respectively. Comparing

Fig. 14(a) with (f), andFig. 15(b) and (c) withFig. 15(e) and (f), respectively, we can see that the unwarping results obtained by our methods are better than those obtained under the SVP assumption. Especially, the vertical lines in

Fig. 15(e) and (f) as well can be seen to be slanted. Such sit-uations are not seen in our results inFig. 15(b) and (c).

It is noted that the images of the above-mentioned experiments were obtained using some methods proposed

Fig. 14. Unwarped images of a real scene inFig. 4(top view), (a) has the T(Tx, Ty, Tz) and R(Rx, Ry, Rz) mentioned in Section3.2after calibration, (b) has

the setting Rx= Ry= Rz= 0, and (c) has the further setting Tx= Ty= 0. (d–f) Results using Eq.(1)with diﬀerent CCD sensor size of 3.2· 2.4, 1.6 · 1.2,

and 0.8· 0.6 mm2_{, respectively.}

Fig. 15. Unwarped images of a real scene inFig. 4(Side view). (a–d) Results using the proposed method with the same T and R as those for yielding Fig. 2.14. (a–d) Results with diﬀerent span of view angle, 1.5p 2.0p, 1.0p 1.5p, 0.5p 1.0p, and 0.0p 0.5p, respectively. (e) and (f) Results using Eqs.

(12)

in our previous paper [13], and the details are omitted here.

6. Conclusions

An approach to systematic calibration and analytic image unwarping for omni-directional non-SVP hypercata-dioptric cameras with hyperbolic-shaped mirrors has been proposed. We used the calibrated parameters of the camera to derive precise unwarping equations. The derived equa-tions have been validated to yield the same unwarping results as those yielded by a perfectly designed SVP camera by adjusting the calibrated parameters to ﬁt the SVP con-straint. Furthermore, we have shown the advantages of our method over the SVP-constrained method for real cameras by some simulation and experimental results. It is mentioned by the way that the resulting image quality of the unwarped perspective-view image is decided by the structure of the omni-camera, the number of grids placed in the deﬁned perspective-view plane in the world space, as well as the location of the view plane with respect to the omni-camera. Future studies may be directed to enhancing the quality of the images according to the above-mentioned three factors as well as employing the unwarping results for real applications.

Acknowledgement

This work was supported ﬁnancially by the Ministry of Economic Aﬀairs under Project No. MOEA 93-EC-17-A-02-S1-032 in Technology Development Program for Academia.

References

[1] S.K. Nayar, Catadioptric Omni-directional Camera, in:Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, San-Juan, Puerto Rico, June 1997, pp. 482–488.

[2] S. Baker, S.K. Nayar, A theory of single-viewpoint catadioptric image formation, International Journal of Computer Vision 35 (2) (1999) 175–196.

[3] R. Swaminathan, M.D. Grossberg, S.K. Nayar, Caustics of catadi-optric cameras, in: Proceedings of eighth IEEE International Conference on Computer Vision, Vancouver, BC, Canada, vol. 2, July 7–14, 2001, pp. 2–9.

[4] K. Yamazawa, Y. Yagi, M. Yachida, Omni-directional imaging with hyperboloidal projection, in: Proceedings of 1993 IEEE/RSJ Inter-national Conference on Intelligent Robots and Systems, Yokohama, Japan, July 26–30, 1993, pp. 1029–1034.

[5] Y. Onoe, N. Yokoya, K. Yamazawa, H. Takemura, Visual surveillance and monitoring system using an omni-directional video camera, in: Proceedings of 14th International Conference on Pattern Recognition, Brisbane, Australia, vol. 1, August 16–20, 1998, pp. 588–592.

[6] R.Y. Tsai, An eﬃcient and accurate camera calibration technique for 3D machine vision, in: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Miami Beach, Florida, 1986, pp. 364–374.

[7] J. Salvi, X. Armangue´, J. Batle, A comparative review of camera calibrating methods with accuracy evaluation, Pattern Recognition 35 (2002) 1617–1635.

[8] C. Geyer, K. Daniilidis, Paracatadioptric camera calibration, IEEE Transactions on Pattern Analysis and Machine Intelligence 24 (5) (2000) 687–695.

[9] C. Geyer, K. Daniilidis, Catadioptric camera calibration, in: Pro-ceedings of seventh IEEE International Conference on Computer Vision, Corfu, Greece, vol. 1, September 20–25, 1999, pp. 398–404. [10] S.B. Kang, Catadioptric self-calibration, in: Proceedings of IEEE

Conference on Computer Vision and Pattern Recognition, Hilton Head, South Carolina, vol. 1, June 13–15, 2000, pp. 201–207. [11] D.G. Aliaga, Accurate catadioptric calibration for real-time pose

estimation in room-size environments, in: Proceedings of eighth IEEE International Conference on Computer Vision, Vancouver, BC, Canada, vol. 1, July 07–14, 2001, pp. 127–134.

[12] J. Fabrizio, J.P. Tarel, R. Benosman, Calibration of panoramic catadioptric sensors made easier, in: Proceedings of third Workshop on Omni-directional Vision, Copenhagen, Denmark, June 02, 2002, pp. 45–52.

[13] S.W. Jeng, W.H. Tsai, Construction of perspective and panoramic images from omni-images taken from hypercatadioptric cameras for visual surveillance, in: Proceedings of 2004 IEEE International Conference on Networking, Sensing, and Control, Taipei, Taiwan, March 21–23, 2004, pp. 204–209.