### Camera calibration

Digital Visual Effects
*Yung-Yu Chuang*

*with slides by Richard Szeliski, Steve Seitz,, Fred Pighin and *
*Marc Pollefyes*

**Outline**

• Camera projection models

• Camera calibration

• Nonlinear least square methods

• A camera calibration tool

• Applications

**Camera projection **

**models**

**Pinhole camera**

**Pinhole camera model**

(optical center) origin

principal point

**P**
*(X,Y,Z)*
**p**

*(x,y)*

**Pinhole camera model**

0 1 1

0 0

0 0

0

0 0

0

~

1 *Z*

*Y*
*X*
*f*

*f*

*Z*
*fY*
*fX*
*y*

*x*

*Z*
*y* *fY*

*Z*
*x* *fX*

principal point

**Pinhole camera model**

0 1 1

0 0

0 0

1 0

0 0

0 1

1 0

0

0 0

0 0

~

1 *Z*

*Y*
*X*
*f*

*f*

*Z*
*fY*
*fX*
*y*

*x*

principal point

**Principal point offset**

0 1 1

0 0

0 0

1 0

0 0

0 1

1 0

0 0

0

~ 1

0 0

*Z*
*Y*
*X*
*y*

*f*

*x*
*f*

*Z*
*fY*
*fX*
*y*

*x*

###

^{I}

^{X}**K**

**x** ~ 0

**intrinsic matrix**

principal point

only related to camera projection

**Intrinsic matrix**

• non-square pixels (digital video)

• skew

• radial distortion

1 0

0 0

0

0 0

*y*
*f*

*x*
*f*

**K**

1 0

0

0 _{0}

0

*y*
*f*

*x*
*s*

*fa*
**K**

**Is this form of K good enough?**

**Distortion**

• Radial distortion of the image

– Caused by imperfect lenses

– Deviations are most noticeable for rays that pass through the edge of the lens

No distortion Pin cushion Barrel

**Camera rotation and translation**

**t**

**R**

*Z*
*Y*
*X*

*Z*
*Y*
*X*

3 3

' ' '

###

1 1 0

0 0

0

~ 1

0 0

*Z*
*Y*
*X*
*y*

*f*

*x*
*f*

*y*
*x*

**t**
**R**

###

^{R}

^{t}

^{X}**K**
**x ~**

**extrinsic matrix**

**Two kinds of parameters**

*• internal or intrinsic parameters such as focal *
length, optical center, aspect ratio:

*what kind of camera?*

*• external or extrinsic (pose) parameters *
including rotation and translation:

*where is the camera?*

**Other projection models**

**Orthographic projection**

• Special case of perspective projection

– Distance from the COP to the PP is infinite

– Also called “parallel projection”: (x, y, z) (x, y)→

Image World

**Other types of projections**

• Scaled orthographic

– Also called “weak perspective”

• Affine projection

– Also called “paraperspective”

**Illusion**

**Illusion**

**Fun with perspective**

**Perspective cues**

**Perspective cues**

**Fun with perspective**

Ames room

Ames video BBC story

**Forced perspective in LOTR**

**Camera calibration**

**Camera calibration**

• Estimate both intrinsic and extrinsic parameters. Two main categories:

1. Photometric calibration: uses reference objects with known geometry

2. Self calibration: only assumes static scene, e.g.

structure from motion

**Camera calibration approaches**

1. linear regression (least squares) 2. nonlinear optimization

**Chromaglyphs (HP research)**

**Camera calibration**

**Linear regression**

###

^{R}

^{t}

^{X}

^{MX}**K**

**x** ~

**Linear regression**

**• Directly estimate 11 unknowns in the M matrix **
*using known 3D points (X*_{i}*,Y*_{i}*,Z** _{i}*) and measured

*feature positions (u*

_{i}*,v*

*)*

_{i}**Linear regression**

**Linear regression**

**Linear regression**

Solve for Projection Matrix M using least-square techniques

**Normal equation**

Given an overdetermined system

**b** **Ax**

**b** **A**

**Ax**

**A**

^{T}

###

^{T}

the normal equation is that which minimizes the sum of the square differences between left and right sides

**Linear regression**

• Advantages:

– All specifics of the camera summarized in one matrix – Can predict where any world point will map to in the

image

• Disadvantages:

– Doesn’t tell us about particular parameters – Mixes up internal and external parameters

• pose specific: move the camera and everything breaks

– More unknowns than true degrees of freedom

**Nonlinear optimization**

• A probabilistic view of least square

• Feature measurement equations

**• Probability of M given {(u**_{i}*,v** _{i}*)}

*P*

**Optimal estimation**

**• Likelihood of M given {(u**_{i}*,v** _{i}*)}

• It is a least square problem (but not necessarily linear least square)

*• How do we minimize L?*

*P*
*L*

**Optimal estimation**

• Non-linear regression (least squares), because
*the relations between û**i** and u**i* are non-linear
**functions of M **

• We can use Levenberg-Marquardt method to minimize it

## ^{R} ^{t} ^{X}

^{R}

^{t}

^{X}

**K** **u**

**u**

**u** ~ ˆ

known constant

We could have terms like in this

*f* cos

unknown parameters

**Nonlinear least square **

**methods**

**Least square fitting**

number of data points

number of parameters

**Linear least square fitting**

y

t

**Linear least square fitting**

y

t

*t* *x* *x*

*t* *M* *t*

*y* ( ) ( ; **x** )

_{0}

###

_{1}

model parameters

**Linear least square fitting**

y

t

*t* *x* *x*

*t* *M* *t*

*y* ( ) ( ; **x** )

_{0}

###

_{1}

model parameters

**Linear least square fitting**

y

t

*t* *x* *x*

*t* *M* *t*

*y* ( ) ( ; **x** )

_{0}

###

_{1}

model parameters

### )

### ; ( )

### (

_{i}

_{i}**x**

*i*

*x* *y* *M* *t*

*f*

residual prediction

**Linear least square fitting**

y

t

*t* *x* *x*

*t* *M* *t*

*y* ( ) ( ; **x** )

_{0}

###

_{1}

3 2 1

### )

0### ;

### ( *t* *x* *x* *t* *x* *t*

*M* **x**

is linear, too.
model parameters

### )

### ; ( )

### (

_{i}

_{i}**x**

*i*

*x* *y* *M* *t*

*f*

residual prediction

**Nonlinear least square fitting**

*t*
*x*
*t*

*x* *x* *e*

*e*
*x*
*t*

*M* ( ;**x**) _{3} ^{1} _{4} ^{2}
model

*x* *T*

*x*
*x*

*x* , , , ]
[ _{1} _{2} _{4} _{4}

parameters **x**

)

; ( )

(**x** _{i}_{i}**x**

*i* *y* *M* *t*

*f*

###

^{x}

^{t}

^{x}

^{t}###

*i* *x* *e* *x* *e*

*y* _{3} ^{1} _{4} ^{2}

residuals

**Function minimization**

It is very hard to solve in general. Here, we only consider a simpler problem of finding local minimum.

Least square is related to function minimization.

**Function minimization**

**Quadratic functions**

Approximate the function with a quadratic function within

a small neighborhood

**Quadratic functions**

**A is positive definite.**

All eigenvalues are positive.

For all x,
x^{T}Ax>0.

negative definite

A is indefinite A is singular

**Function minimization**

Why?

By definition, if is a local minimizer,

** h** **F(x**** ^{*}**

**h)**

**F(x**

^{*}**)**

**x*******

**)**
**h**
**O(**

**)**
**(x**
**F'**
**h**

**)**
**F(x**
**h)**

**F(x**** ^{*}**

****

^{*}

^{T}****

^{*}

^{2}is small enough

**Function minimization**

**Function minimization**

**Descent methods**

**Descent direction**

**Steepest descent method**

* the decrease of F(x) per *
unit along h direction

**→**

h_{sd }is a descent direction because h^{T}_{sd }F’(x) = -F’(x)^{2 }<0

**Line search**

minimum is

) (

that so

Find

0 **h)**

**F(x** *α*

###

###

###

) (

' )

0 (

0 0

**h**
**x**

**F**
**x** **h**

**x**
**F**

**h)**
**F(x**

**T** *α*

*α*

**)**
**(x**
**F'**

**h** _{0}

**Line search**

**Hh** **h**

**h** **h**

**T**

###

**T**

###

0 )

(

' **x**_{0} ** h**
**F**

**h**^{T}*α*

**)**
**(x**
**F'**

**h** _{0}

###

0 ) (

) (

) (

'

0 ''

0 '

0

**Hh**
**h**

**h**
**h**

**h**
**x**

**F**
**x**

**F**
**h**

**h**
**x**

**F**
**h**

**T**
**T**

**T**
**T**

**T**

*α*

*α*
*α*

**Steepest descent method**

isocontour gradient

**Steepest descent method**

It has good performance in the initial stage of the iterative process. Converge very slow with a linear rate.

**Newton’s method**

**→**

**→**

**→**

**→**

**Newton’s method**

• Another view

• Minimizer satisfies

**Hh** **h**

**g** **h**

**x** **h**

**x**

**h**

^{T}

^{T}### 2 ) 1

### ( )

### ( )

### ( *F* *F*

*E*

### 0 )

### (

### ' **h**

^{*}

### *E*

### 0 )

### (

### ' **h** **g** **Hh** *E*

**g** **H**

**h**

^{}

^{1}

**Newton’s method**

• It requires solving a linear system and H is not always positive definite.

• It has good performance in the final stage of the iterative process, where x is close to x*.

**g** **H**

**h**

^{}

^{1}

**Gauss-Newton method**

• Use the approximate Hessian

• No need for second derivative

• H is positive semi-definite

**J** **J**

**H**

^{T}**Hybrid method**

This needs to calculate second-order derivative which might not be available.

**Levenberg-Marquardt method**

• LM can be thought of as a combination of steepest descent and the Newton method.

When the current solution is far from the correct one, the algorithm behaves like a

steepest descent method: slow, but guaranteed to converge. When the current solution is close to the correct solution, it becomes a Newton’s method.

**Nonlinear least square **

### ).

### ˆ ( with ˆ ,

### Here, minimal.

### is distance

### squared

### that the so

### vector parameter

### best the

### find try to

### , ts measuremen of

### set a

### Given

**p** **x**

**x** **x**

**p**

**x**

*f*

*T*

###

###

###

###

###

**Levenberg-Marquardt method**

**Levenberg-Marquardt method**

• μ=0 → Newton’s method

• μ ∞ → **→ steepest descent method**

• Strategy for choosing μ

– Start with some small μ

– If F is not reduced, keep trying larger μ until it does – If F is reduced, accept it and reduce μ for the next

iteration

**g** **I)h**

**J**

**(J**

^{T}###

**Recap (the Rosenbrock function)**

2 2 2

2) 100( )

1 ( )

,

(*x* *y* *x* *y* *x*

*f*

*z*

*Global minimum at (1,1)*

**Steepest descent**

**g** **x**

**x**

_{k}_{}

_{1}###

_{k}###

**Hh** **h**

**h** **h**

**T**

###

**T**

###

**1**

**x****k**_{}

**x****k**

*x*1

*x*2 **x**^{min}

** g** ^{F x}^{'}

###

_{k}*x*1

*x*2

**g**

**1**

**x****k**_{}

**x****k**

**x**min

**In the plane of the steepest descent direction**

**Hh** **h**

**h** **h**

**T**

###

**T**

###

###

**1**

**x****k**_{} **x**_{k}

**Regularized Least-**
**Squares**

**Steepest descent (1000 iterations)**

**Gauss-Newton method**

• With the approximate Hessian

• No need for second derivative

• H is positive semi-definite

**g** **H**

**x**

**x**

_{k}_{}

_{1}###

_{k}###

^{}

^{1}**J** **J**

**H**

^{T}*x*1

*x*2

**g**
**H**
**-** ^{}^{1}

**1**

**x****k**_{}

**x****k**

**x**min

**Regularized Least-**
**Squares**

**Newton’s method (48 evaluations)**

**Levenberg-Marquardt**

• Blends steepest descent and Gauss-Newton

• At each step, solve for the descent direction h

• If μ large, , steepest descent

• If μ small, , Gauss-Newton

**g** **I)h**

**J**

**(J**

^{T}###

**g** **h**

**g** **J)**

**(J**

**h**

^{T}^{}

^{1}

**Regularized Least-**
**Squares**

**Levenberg-Marquardt (90 evaluations)**

**A popular calibration tool**

**Multi-plane calibration** ** **

Images courtesy Jean-Yves Bouguet, Intel Corp.

Advantage

• Only requires a plane

• Don’t have to know positions/orientations

• Good code available online!

– Intel’s OpenCV library: http://www.intel.com/research/mrl/research/opencv/

– Matlab version by Jean-Yves Bouget:

http://www.vision.caltech.edu/bouguetj/calib_doc/index.html

– Zhengyou Zhang’s web site: http://research.microsoft.com/~zhang/Calib/

**Step 1: data acquisition**

**Step 2: specify corner order**

### Step 3: corner extraction

### Step 3: corner extraction

**Step 4: minimize projection error**

**Step 4: camera calibration**

**Step 4: camera calibration**

**Step 5: refinement**

**Optimized parameters**

**Applications**

**How is calibration used?**

• Good for recovering intrinsic parameters; It is thus useful for many vision applications

• Since it requires a calibration pattern, it is often necessary to remove or replace the

pattern from the footage or utilize it in some ways…

**Example of calibration**

**Example of calibration**

**Example of calibration**

• Videos from GaTech

• DasTatoo, MakeOf

• P!NG, MakeOf

• Work, MakeOf

• LifeInPaints, MakeOf

**PhotoBook**

**MakeOf**

**PhotoBook**