• 沒有找到結果。

Camera calibration

N/A
N/A
Protected

Academic year: 2022

Share "Camera calibration"

Copied!
96
0
0

加載中.... (立即查看全文)

全文

(1)

Camera calibration

Digital Visual Effects Yung-Yu Chuang

with slides by Richard Szeliski, Steve Seitz,, Fred Pighin and Marc Pollefyes

(2)

Outline

• Camera projection models

• Camera calibration

• Nonlinear least square methods

• A camera calibration tool

• Applications

(3)

Camera projection

models

(4)

Pinhole camera

(5)

Pinhole camera model

(optical center) origin

principal point

P (X,Y,Z) p

(x,y)

(6)

Pinhole camera model













 









0 1 1

0 0

0 0

0

0 0

0

~

1 Z

Y X f

f

Z fY fX y

x

Z y fY

Z x fX

principal point

(7)

Pinhole camera model

















 









0 1 1

0 0

0 0

1 0

0 0

0 1

1 0

0

0 0

0 0

~

1 Z

Y X f

f

Z fY fX y

x

principal point

(8)

Principal point offset

















 









0 1 1

0 0

0 0

1 0

0 0

0 1

1 0

0 0

0

~ 1

0 0

Z Y X y

f

x f

Z fY fX y

x

 

I X

K

x ~ 0

intrinsic matrix

principal point

only related to camera projection

(9)

Intrinsic matrix

• non-square pixels (digital video)

• skew

• radial distortion





1 0

0 0

0

0 0

y f

x f

K





1 0

0

0 0

0

y f

x s

fa K

Is this form of K good enough?

(10)

Distortion

• Radial distortion of the image

– Caused by imperfect lenses

– Deviations are most noticeable for rays that pass through the edge of the lens

No distortion Pin cushion Barrel

(11)

Camera rotation and translation

t

R





 





Z Y X

Z Y X

3 3

' ' '

 

















1 1 0

0 0

0

~ 1

0 0

Z Y X y

f

x f

y x

t R

 

R t X

K x ~

extrinsic matrix

(12)

Two kinds of parameters

• internal or intrinsic parameters such as focal length, optical center, aspect ratio:

what kind of camera?

• external or extrinsic (pose) parameters including rotation and translation:

where is the camera?

(13)

Other projection models

(14)

Orthographic projection

• Special case of perspective projection

– Distance from the COP to the PP is infinite

– Also called “parallel projection”: (x, y, z) (x, y)

Image World

(15)

Other types of projections

• Scaled orthographic

– Also called “weak perspective”

• Affine projection

– Also called “paraperspective”

(16)

Illusion

(17)

Illusion

(18)

Fun with perspective

(19)

Perspective cues

(20)

Perspective cues

(21)

Fun with perspective

Ames room

Ames video BBC story

(22)

Forced perspective in LOTR

(23)

Camera calibration

(24)

Camera calibration

• Estimate both intrinsic and extrinsic parameters. Two main categories:

1. Photometric calibration: uses reference objects with known geometry

2. Self calibration: only assumes static scene, e.g.

structure from motion

(25)

Camera calibration approaches

1. linear regression (least squares) 2. nonlinear optimization

(26)

Chromaglyphs (HP research)

(27)

Camera calibration

(28)

Linear regression

 

R t X MX

K

x ~ 

(29)

Linear regression

• Directly estimate 11 unknowns in the M matrix using known 3D points (Xi,Yi,Zi) and measured feature positions (ui,vi)

(30)

Linear regression

(31)

Linear regression

(32)

Linear regression

Solve for Projection Matrix M using least-square techniques

(33)

Normal equation

Given an overdetermined system

b Ax

b A

Ax

A

T

T

the normal equation is that which minimizes the sum of the square differences between left and right sides

(34)

Linear regression

• Advantages:

– All specifics of the camera summarized in one matrix – Can predict where any world point will map to in the

image

• Disadvantages:

– Doesn’t tell us about particular parameters – Mixes up internal and external parameters

• pose specific: move the camera and everything breaks

– More unknowns than true degrees of freedom

(35)

Nonlinear optimization

• A probabilistic view of least square

• Feature measurement equations

• Probability of M given {(ui,vi)}

P

(36)

Optimal estimation

• Likelihood of M given {(ui,vi)}

• It is a least square problem (but not necessarily linear least square)

• How do we minimize L?

P L

(37)

Optimal estimation

• Non-linear regression (least squares), because the relations between ûi and ui are non-linear functions of M

• We can use Levenberg-Marquardt method to minimize it

  R t X

K u

u

u  ~ ˆ 

known constant

We could have terms like in this

f cos 

unknown parameters

(38)

Nonlinear least square

methods

(39)

Least square fitting

number of data points

number of parameters

(40)

Linear least square fitting

y

t

(41)

Linear least square fitting

y

t

t x x

t M t

y ( )  ( ; x ) 

0

1

model parameters

(42)

Linear least square fitting

y

t

t x x

t M t

y ( )  ( ; x ) 

0

1

model parameters

(43)

Linear least square fitting

y

t

t x x

t M t

y ( )  ( ; x ) 

0

1

model parameters

)

; ( )

(

i i

x

i

x y M t

f  

residual prediction

(44)

Linear least square fitting

y

t

t x x

t M t

y ( )  ( ; x ) 

0

1

3 2 1

)

0

;

( t x x t x t

M x   

is linear, too.

model parameters

)

; ( )

(

i i

x

i

x y M t

f  

residual prediction

(45)

Nonlinear least square fitting

t x t

x x e

e x t

M ( ;x)  3 14 2 model

x T

x x

x , , , ] [ 1 2 4 4

 parameters x

)

; ( )

(x i i x

i y M t

f  

x t x t

i x e x e

y3 14 2

 residuals

(46)

Function minimization

It is very hard to solve in general. Here, we only consider a simpler problem of finding local minimum.

Least square is related to function minimization.

(47)

Function minimization

(48)

Quadratic functions

Approximate the function with a quadratic function within

a small neighborhood

(49)

Quadratic functions

A is positive definite.

All eigenvalues are positive.

For all x, xTAx>0.

negative definite

A is indefinite A is singular

(50)

Function minimization

Why?

By definition, if is a local minimizer,

h F(x*h)F(x*)

x*

) h O(

) (x F' h

) F(x h)

F(x*   *T *2

is small enough

(51)

Function minimization

(52)

Function minimization

(53)

Descent methods

(54)

Descent direction

(55)

Steepest descent method

the decrease of F(x) per unit along h direction

hsd is a descent direction because hTsd F’(x) = -F’(x)2 <0

(56)

Line search

minimum is

) (

that so

Find

0 h)

F(xα

) (

' )

0 (

0 0

h x

F x h

x F

h) F(x

T α

α

 

 

 

 

) (x F'

h   0

(57)

Line search

Hh h

h h

T

T

0 )

(

' x0  hF

hT α

) (x F'

h   0

 

0 ) (

) (

) (

'

0 ''

0 '

0

Hh h

h h

h x

F x

F h

h x

F h

T T

T T

T

α

α α

(58)

Steepest descent method

isocontour gradient

(59)

Steepest descent method

It has good performance in the initial stage of the iterative process. Converge very slow with a linear rate.

(60)

Newton’s method

(61)

Newton’s method

• Another view

• Minimizer satisfies

Hh h

g h

x h

x

h

T T

2 ) 1

( )

( )

(  F   F  

E

0 )

(

' h

*

E

0 )

(

' hgHhE

g H

h  

1

(62)

Newton’s method

• It requires solving a linear system and H is not always positive definite.

• It has good performance in the final stage of the iterative process, where x is close to x*.

g H

h  

1

(63)

Gauss-Newton method

• Use the approximate Hessian

• No need for second derivative

• H is positive semi-definite

J J

H

T

(64)

Hybrid method

This needs to calculate second-order derivative which might not be available.

(65)

Levenberg-Marquardt method

• LM can be thought of as a combination of steepest descent and the Newton method.

When the current solution is far from the correct one, the algorithm behaves like a

steepest descent method: slow, but guaranteed to converge. When the current solution is close to the correct solution, it becomes a Newton’s method.

(66)

Nonlinear least square

).

ˆ ( with ˆ ,

Here, minimal.

is distance

squared

that the so

vector parameter

best the

find try to

, ts measuremen of

set a

Given

p x

x x

p

x

f

T

 

(67)

Levenberg-Marquardt method

(68)

Levenberg-Marquardt method

• μ=0 → Newton’s method

• μ ∞ → → steepest descent method

• Strategy for choosing μ

– Start with some small μ

– If F is not reduced, keep trying larger μ until it does – If F is reduced, accept it and reduce μ for the next

iteration

g I)h

J

(J

T

   

(69)

Recap (the Rosenbrock function)

2 2 2

2) 100( )

1 ( )

,

(x y x y x

f

z     

Global minimum at (1,1)

(70)

Steepest descent

g x

x

k1

k

 

Hh h

h h

T

T

(71)

1

xk

xk

x1

x2 xmin

 g F x'

 

k

(72)

x1

x2

g

1

xk

xk

xmin

(73)

In the plane of the steepest descent direction

Hh h

h h

T

T

1

xk xk

(74)

Regularized Least- Squares

Steepest descent (1000 iterations)

(75)

Gauss-Newton method

• With the approximate Hessian

• No need for second derivative

• H is positive semi-definite

g H

x

x

k1

k

1

J J

H

T

(76)

x1

x2

g H - 1

1

xk

xk

xmin

(77)

Regularized Least- Squares

Newton’s method (48 evaluations)

(78)

Levenberg-Marquardt

• Blends steepest descent and Gauss-Newton

• At each step, solve for the descent direction h

• If μ large, , steepest descent

• If μ small, , Gauss-Newton

g I)h

J

(J

T

   

g h  

g J)

(J

h  

T 1

(79)

Regularized Least- Squares

Levenberg-Marquardt (90 evaluations)

(80)

A popular calibration tool

(81)

Multi-plane calibration

Images courtesy Jean-Yves Bouguet, Intel Corp.

Advantage

• Only requires a plane

• Don’t have to know positions/orientations

• Good code available online!

Intel’s OpenCV library: http://www.intel.com/research/mrl/research/opencv/

Matlab version by Jean-Yves Bouget:

http://www.vision.caltech.edu/bouguetj/calib_doc/index.html

Zhengyou Zhang’s web site: http://research.microsoft.com/~zhang/Calib/

(82)

Step 1: data acquisition

(83)

Step 2: specify corner order

(84)

Step 3: corner extraction

(85)

Step 3: corner extraction

(86)

Step 4: minimize projection error

(87)

Step 4: camera calibration

(88)

Step 4: camera calibration

(89)

Step 5: refinement

(90)

Optimized parameters

(91)

Applications

(92)

How is calibration used?

• Good for recovering intrinsic parameters; It is thus useful for many vision applications

• Since it requires a calibration pattern, it is often necessary to remove or replace the

pattern from the footage or utilize it in some ways…

(93)

Example of calibration

(94)

Example of calibration

(95)

Example of calibration

• Videos from GaTech

• DasTatoo, MakeOf

• P!NG, MakeOf

• Work, MakeOf

• LifeInPaints, MakeOf

(96)

PhotoBook

MakeOf

PhotoBook

參考文獻

相關文件

– Select View  Cameras  Set Active Object as Active Camera – Load background images. – Set the parameters of

If the subset has constant extrinsic curvature and is a smooth manifold (possibly with boundary), then it has an explicit intrinsic lower curvature bound which is sharp in

&#34;Extensions to the k-Means Algorithm for Clustering Large Data Sets with Categorical Values,&#34; Data Mining and Knowledge Discovery, Vol. “Density-Based Clustering in

The relationship between these extra type parameters, and the types to which they are associated, is established by parameteriz- ing the interfaces (Java generics, C#, and Eiffel)

• The Tolerable Upper Intake level (UL) is the highest nutrient intake value that is likely to pose no risk of adverse health effects for individuals in a given age and gender

The Hull-White Model: Calibration with Irregular Trinomial Trees (concluded).. • Recall that the algorithm figured out θ(t i ) that matches the spot rate r(0, t i+2 ) in order

• As n increases, the stock price ranges over ever larger numbers of possible values, and trading takes place nearly continuously. • Any proper calibration of the model parameters

structure for motion: automatic recovery of camera motion and scene structure from two or more images.. It is a self calibration technique and called automatic camera tracking