Camera calibration
Digital Visual Effects Yung-Yu Chuang
with slides by Richard Szeliski, Steve Seitz,, Fred Pighin and Marc Pollefyes
Outline
• Camera projection models
• Camera calibration
• Nonlinear least square methods
• A camera calibration tool
• Applications
Camera projection
models
Pinhole camera
Pinhole camera model
(optical center) origin
principal point
P (X,Y,Z) p
(x,y)
Pinhole camera model
0 1 1
0 0
0 0
0
0 0
0
~
1 Z
Y X f
f
Z fY fX y
x
Z y fY
Z x fX
principal point
Pinhole camera model
0 1 1
0 0
0 0
1 0
0 0
0 1
1 0
0
0 0
0 0
~
1 Z
Y X f
f
Z fY fX y
x
principal point
Principal point offset
0 1 1
0 0
0 0
1 0
0 0
0 1
1 0
0 0
0
~ 1
0 0
Z Y X y
f
x f
Z fY fX y
x
I XK
x ~ 0
intrinsic matrix
principal point
only related to camera projection
Intrinsic matrix
• non-square pixels (digital video)
• skew
• radial distortion
1 0
0 0
0
0 0
y f
x f
K
1 0
0
0 0
0
y f
x s
fa K
Is this form of K good enough?
Distortion
• Radial distortion of the image
– Caused by imperfect lenses
– Deviations are most noticeable for rays that pass through the edge of the lens
No distortion Pin cushion Barrel
Camera rotation and translation
t
R
Z Y X
Z Y X
3 3
' ' '
1 1 0
0 0
0
~ 1
0 0
Z Y X y
f
x f
y x
t R
R t XK x ~
extrinsic matrix
Two kinds of parameters
• internal or intrinsic parameters such as focal length, optical center, aspect ratio:
what kind of camera?
• external or extrinsic (pose) parameters including rotation and translation:
where is the camera?
Other projection models
Orthographic projection
• Special case of perspective projection
– Distance from the COP to the PP is infinite
– Also called “parallel projection”: (x, y, z) (x, y)→
Image World
Other types of projections
• Scaled orthographic
– Also called “weak perspective”
• Affine projection
– Also called “paraperspective”
Illusion
Illusion
Fun with perspective
Perspective cues
Perspective cues
Fun with perspective
Ames room
Ames video BBC story
Forced perspective in LOTR
Camera calibration
Camera calibration
• Estimate both intrinsic and extrinsic parameters. Two main categories:
1. Photometric calibration: uses reference objects with known geometry
2. Self calibration: only assumes static scene, e.g.
structure from motion
Camera calibration approaches
1. linear regression (least squares) 2. nonlinear optimization
Chromaglyphs (HP research)
Camera calibration
Linear regression
R t X MXK
x ~
Linear regression
• Directly estimate 11 unknowns in the M matrix using known 3D points (Xi,Yi,Zi) and measured feature positions (ui,vi)
Linear regression
Linear regression
Linear regression
Solve for Projection Matrix M using least-square techniques
Normal equation
Given an overdetermined system
b Ax
b A
Ax
A
T
Tthe normal equation is that which minimizes the sum of the square differences between left and right sides
Linear regression
• Advantages:
– All specifics of the camera summarized in one matrix – Can predict where any world point will map to in the
image
• Disadvantages:
– Doesn’t tell us about particular parameters – Mixes up internal and external parameters
• pose specific: move the camera and everything breaks
– More unknowns than true degrees of freedom
Nonlinear optimization
• A probabilistic view of least square
• Feature measurement equations
• Probability of M given {(ui,vi)}
P
Optimal estimation
• Likelihood of M given {(ui,vi)}
• It is a least square problem (but not necessarily linear least square)
• How do we minimize L?
P L
Optimal estimation
• Non-linear regression (least squares), because the relations between ûi and ui are non-linear functions of M
• We can use Levenberg-Marquardt method to minimize it
R t X
K u
u
u ~ ˆ
known constant
We could have terms like in this
f cos
unknown parameters
Nonlinear least square
methods
Least square fitting
number of data points
number of parameters
Linear least square fitting
y
t
Linear least square fitting
y
t
t x x
t M t
y ( ) ( ; x )
0
1model parameters
Linear least square fitting
y
t
t x x
t M t
y ( ) ( ; x )
0
1model parameters
Linear least square fitting
y
t
t x x
t M t
y ( ) ( ; x )
0
1model parameters
)
; ( )
(
i ix
i
x y M t
f
residual prediction
Linear least square fitting
y
t
t x x
t M t
y ( ) ( ; x )
0
13 2 1
)
0;
( t x x t x t
M x
is linear, too.model parameters
)
; ( )
(
i ix
i
x y M t
f
residual prediction
Nonlinear least square fitting
t x t
x x e
e x t
M ( ;x) 3 1 4 2 model
x T
x x
x , , , ] [ 1 2 4 4
parameters x
)
; ( )
(x i i x
i y M t
f
x t x t
i x e x e
y 3 1 4 2
residuals
Function minimization
It is very hard to solve in general. Here, we only consider a simpler problem of finding local minimum.
Least square is related to function minimization.
Function minimization
Quadratic functions
Approximate the function with a quadratic function within
a small neighborhood
Quadratic functions
A is positive definite.
All eigenvalues are positive.
For all x, xTAx>0.
negative definite
A is indefinite A is singular
Function minimization
Why?
By definition, if is a local minimizer,
h F(x* h) F(x*)
x*
) h O(
) (x F' h
) F(x h)
F(x* * T * 2
is small enough
Function minimization
Function minimization
Descent methods
Descent direction
Steepest descent method
the decrease of F(x) per unit along h direction
→
hsd is a descent direction because hTsd F’(x) = -F’(x)2 <0
Line search
minimum is
) (
that so
Find
0 h)
F(x α
) (
' )
0 (
0 0
h x
F x h
x F
h) F(x
T α
α
) (x F'
h 0
Line search
Hh h
h h
T
T
0 )
(
' x0 h F
hT α
) (x F'
h 0
0 ) (
) (
) (
'
0 ''
0 '
0
Hh h
h h
h x
F x
F h
h x
F h
T T
T T
T
α
α α
Steepest descent method
isocontour gradient
Steepest descent method
It has good performance in the initial stage of the iterative process. Converge very slow with a linear rate.
Newton’s method
→
→
→
→
Newton’s method
• Another view
• Minimizer satisfies
Hh h
g h
x h
x
h
T T2 ) 1
( )
( )
( F F
E
0 )
(
' h
* E
0 )
(
' h g Hh E
g H
h
1Newton’s method
• It requires solving a linear system and H is not always positive definite.
• It has good performance in the final stage of the iterative process, where x is close to x*.
g H
h
1Gauss-Newton method
• Use the approximate Hessian
• No need for second derivative
• H is positive semi-definite
J J
H
THybrid method
This needs to calculate second-order derivative which might not be available.
Levenberg-Marquardt method
• LM can be thought of as a combination of steepest descent and the Newton method.
When the current solution is far from the correct one, the algorithm behaves like a
steepest descent method: slow, but guaranteed to converge. When the current solution is close to the correct solution, it becomes a Newton’s method.
Nonlinear least square
).
ˆ ( with ˆ ,
Here, minimal.
is distance
squared
that the so
vector parameter
best the
find try to
, ts measuremen of
set a
Given
p x
x x
p
x
f
T
Levenberg-Marquardt method
Levenberg-Marquardt method
• μ=0 → Newton’s method
• μ ∞ → → steepest descent method
• Strategy for choosing μ
– Start with some small μ
– If F is not reduced, keep trying larger μ until it does – If F is reduced, accept it and reduce μ for the next
iteration
g I)h
J
(J
T
Recap (the Rosenbrock function)
2 2 2
2) 100( )
1 ( )
,
(x y x y x
f
z
Global minimum at (1,1)
Steepest descent
g x
x
k1
k
Hh h
h h
T
T
1
xk
xk
x1
x2 xmin
g F x'
kx1
x2
g
1
xk
xk
xmin
In the plane of the steepest descent direction
Hh h
h h
T
T
1
xk xk
Regularized Least- Squares
Steepest descent (1000 iterations)
Gauss-Newton method
• With the approximate Hessian
• No need for second derivative
• H is positive semi-definite
g H
x
x
k1
k
1J J
H
Tx1
x2
g H - 1
1
xk
xk
xmin
Regularized Least- Squares
Newton’s method (48 evaluations)
Levenberg-Marquardt
• Blends steepest descent and Gauss-Newton
• At each step, solve for the descent direction h
• If μ large, , steepest descent
• If μ small, , Gauss-Newton
g I)h
J
(J
T
g h
g J)
(J
h
T 1Regularized Least- Squares
Levenberg-Marquardt (90 evaluations)
A popular calibration tool
Multi-plane calibration
Images courtesy Jean-Yves Bouguet, Intel Corp.
Advantage
• Only requires a plane
• Don’t have to know positions/orientations
• Good code available online!
– Intel’s OpenCV library: http://www.intel.com/research/mrl/research/opencv/
– Matlab version by Jean-Yves Bouget:
http://www.vision.caltech.edu/bouguetj/calib_doc/index.html
– Zhengyou Zhang’s web site: http://research.microsoft.com/~zhang/Calib/
Step 1: data acquisition
Step 2: specify corner order
Step 3: corner extraction
Step 3: corner extraction
Step 4: minimize projection error
Step 4: camera calibration
Step 4: camera calibration
Step 5: refinement
Optimized parameters
Applications
How is calibration used?
• Good for recovering intrinsic parameters; It is thus useful for many vision applications
• Since it requires a calibration pattern, it is often necessary to remove or replace the
pattern from the footage or utilize it in some ways…
Example of calibration
Example of calibration
Example of calibration
• Videos from GaTech
• DasTatoo, MakeOf
• P!NG, MakeOf
• Work, MakeOf
• LifeInPaints, MakeOf
PhotoBook
MakeOf
PhotoBook