Image stitching

(1)

Image stitching

Digital Visual Effects, Spring 2007 Yung-Yu Chuang

2007/4/3

with slides by Richard Szeliski, Steve Seitz, Matthew Brown and Vaclav Hlavac

(2)

Image stitching

• Stitching = alignment + blending geometrical

registration

photometric registration

(3)

Applications of image stitching

• Video stabilization

• Video summarization

• Video compression

• Video matting

• Panorama creation

(4)

Video summarization

(5)

Video compression

(6)

Object removal

input video

(7)

Object removal

remove foreground

(8)

Object removal

estimate background

(9)

Object removal

background estimation

(10)

Panorama creation

(11)

Why panorama?

• Are you getting the whole picture?

– Compact Camera FOV = 50 x 35°

(12)

Why panorama?

– Human FOV = 200 x 135°

(13)

Why panorama?

– Human FOV = 200 x 135°

– Panoramic Mosaic = 360 x 180°

(14)

Panorama examples

• Like HDR, it is a topic of computational

photography, seeking ways to build a better camera mostly in software.

• Most consumer cameras have a panorama mode

• Mars:

http://www.panoramas.dk/fullscreen3/f2_mars97.html

• Earth:

http://www.panoramas.dk/new-year-2006/taipei.html

(15)

A pencil of rays contains all views

real

camera synthetic

camera

Can generate any synthetic camera view

as long as it has the same center of projection!

(16)

Mosaic as an image reprojection

mosaic projection plane

• The images are reprojected onto a common plane

• The mosaic is formed on this plane

• Mosaic is a synthetic wide-angle camera

(17)

Changing camera center

• Does it still work? synthetic PP PP1

PP2

(18)

Planar scene (or far away)

• PP3 is a projection plane of both centers of projection, so we are OK!

• This is how big aerial photographs are made

PP1

PP3

PP2

(19)

Motion models

• Parametric models as the assumptions on the relation between two images.

(20)

2D Motion models

(21)

Motion models

Translation

2 unknowns

Affine

6 unknowns

Perspective

8 unknowns

3D rotation

3 unknowns

(22)

A case study: cylindrical panorama

• What if you want a 360° field of view?

mosaic projection cylinder

(23)

Cylindrical panoramas

• Steps

– Reproject each image onto a cylinder – Blend

– Output the resulting mosaic

(24)

Cylindrical panorama

1. Take pictures on a tripod (or handheld) 2. Warp to cylindrical coordinate

3. Compute pairwise alignments

4. Fix up the end-to-end alignment 5. Blending

6. Crop the result and import into a viewer

(25)

Taking pictures

Kaidan panoramic tripod head

(26)

Translation model

(27)

Cylindrical projection

(28)

Cylindrical projection

Adopted from http://www.cambridgeincolour.com/tutorials/image-projections.htm

(29)

Cylindrical projection

Adopted from http://www.cambridgeincolour.com/tutorials/image-projections.htm

(30)

Cylindrical projection

unwrapped cylinder

x y

f

θ

x

(31)

z Cylindrical projection

unwrapped cylinder

x y

θ

x y

f

(32)

Cylindrical projection

unwrapped cylinder

x y

z

x y

f

s=f

gives less distortion

(33)

f = 180 (pixels)

Cylindrical reprojection

f = 380 f = 280

Image 384x300

top-down view Focal length – the dirty secret…

(34)

A simple method for estimating f

Or, you can use other software, such as AutoStich, to help.

d f

w p

(35)

Input images

(36)

Cylindrical warping

(37)

Blending

• Why blending: parallax, lens distortion, scene motion, exposure difference

(38)

Blending

(39)

Blending

(40)

Blending

(41)

Assembling the panorama

• Stitch pairs together, blend, then crop

(42)

Problem: Drift

• Error accumulation

– small errors accumulate over time

(43)

Problem: Drift

• Solution

– add another copy of first image at the end

– there are a bunch of ways to solve this problem

• add displacement of (y₁ – y_n)/(n -1) to each image after the first

• compute a global warp: y’ = y + ax

• run a big optimization problem, incorporating this constraint

– best solution, but more complicated – known as “bundle adjustment”

(x₁,y₁)

• copy of first image

(x_n,y_n)

(44)

End-to-end alignment and crop

(45)

Viewer: panorama

++

++ ++

example: http://www.cs.washington.edu/education/courses/cse590ss/01wi/projects/project1/students/dougz/index.html

(46)

Viewer: texture mapped model

example: http://www.panoramas.dk/

(47)

Cylindrical panorama

1. Take pictures on a tripod (or handheld) 2. Warp to cylindrical coordinate

3. Compute pairwise alignments

4. Fix up the end-to-end alignment 5. Blending

6. Crop the result and import into a viewer

(48)

Determine pairwise alignment?

• Feature-based methods: only use feature points to estimate parameters

• We will study the “Recognising panorama”

paper published in ICCV 2003

• Run SIFT for each image, find feature matches.

(49)

Determine pairwise alignment

• p’=Mp, where M is a transformation matrix, p and p’ are feature matches

• It is possible to use more complicated models such as affine or perspective

• For example, assume M is a 2x2 matrix

• Find M with the least square error

( )

∑

= n −

i

p Mp

1

' 2

⎟⎟⎠

⎜⎜ ⎞

⎝

⎟⎟⎛

⎠

⎜⎜ ⎞

⎝

= ⎛

⎟⎟⎠

⎜⎜ ⎞

⎝

⎛

y x m

m

m m

y x

22 21

12 11

' '

(50)

Determine pairwise alignment

• Overdetermined system

⎟⎟⎠

⎜⎜ ⎞

⎝

⎟⎟⎛

⎠

⎜⎜ ⎞

⎝

= ⎛

⎟⎟⎠

⎜⎜ ⎞

⎝

⎛

y x m

m

m m

y x

22 21

12 11

' '

' 1 22

1 21

1

' 1 12

1 11

1

y m

x

x m

y m

x

= +

⎟⎟

⎠

⎞

⎜⎜

⎝

⎛

=

⎟⎟

⎟

⎠

⎞

⎜⎜

⎜

⎝

⎛

⎟⎟

⎠

⎞

⎜⎜

⎝

⎛

' ' ' 2

' 1

22 21 12 11 2

2

1 1

0 0

n n n

n n

n

y x x y x

m m m m

y x

M M M

M M

(51)

Normal equation

Given an overdetermined system

b Ax =

b A

Ax

A

^T

=

^T

the normal equation is that which minimizes the sum of the square differences between left and right sides

Why?

(52)

Normal equation

( )

²

)

( x = Ax − b E

⎥ ⎥

⎦

⎤

⎢ ⎢

⎣

⎡

=

⎥ ⎥

⎥

⎦

⎤

⎢ ⎢

⎢

⎣

⎡

⎥ ⎥

⎦

⎤

⎢ ⎢

⎣

⎡

n m

nm n

m

b b

x x

a a

: : : :

...

: :

...

₁

1

1 11

nxm, n equations, m variables

(53)

Normal equation

⎥ ⎥

⎦

⎤

⎢ ⎢

⎣

⎡

⎟⎟ −

⎠

⎜⎜ ⎞

⎝

⎛

⎟⎟ −

⎠

⎜⎜ ⎞

⎝

⎛

⎟⎟ −

⎠

⎜⎜ ⎞

⎝

⎛

=

⎥ ⎥

⎦

⎤

⎢ ⎢

⎣

⎡

−

⎥ ⎥

⎥

⎦

⎤

⎢ ⎢

⎢

⎣

⎡

=

−

∑

=

n m

j

j nj

i m

j

j ij m

j

j j

n i

m

j

j nj m

j

j ij m

j

j j

b x

a

b x

a

b x

a

b b b

x a

1 1

1 1 1

1

: : :

:

: :

b

Ax

( ) ∑ ∑

= =

⎥ ⎥

⎦

⎤

⎢ ⎢

⎣

⎡ ⎟⎟ −

⎠

⎜⎜ ⎞

⎝

= ⎛

−

=

ⁿ

i

i m

j

ij

x b

a E

1

2

1

)

2

( x Ax b

(54)

Normal equation

( ) ∑ ∑

= =

⎥ ⎥

⎦

⎤

⎢ ⎢

⎣

⎡ ⎟⎟ −

⎠

⎜⎜ ⎞

⎝

= ⎛

−

=

ⁿ

i

i m

j

ij

x b

a E

1

2

1

)

2

( x Ax b

∂ =

= ∂

1

0 x

E

∑

∑ ∑

= = =

−

=

ⁿ

i

i i n

i

j m

j

ij

i

a x a b

a

1

1 1

1

2

1

1 1

2

_i

n

i

i m

j

ij

x b a

∑ ∑ a

= =

⎥ ⎥

⎦

⎤

⎢ ⎢

⎣

⎡ ⎟⎟ −

⎠

⎜⎜ ⎞

⎝

⎛

) (

2 0 A Ax A b

x

T

−

∂ =

= ∂ E

b A

Ax

A

^T

=

^T

→

(55)

Normal equation

( )

( ) ( )

( ( ) ) ⁽ ⁾

( ) ⁽ ⁾

( ) ( ) ^A ^b ^x ^A ^b ^x ^b ^b

Ax A

x

b b b

A x

Ax b

Ax A

x

b Ax

b A

x

b Ax

T T T T

T T

T

T T

T

+

−

=

+

−

=

−

=

−

=

−

=

−

T T

T 2

b A Ax

x A

T

2 2 −

∂ =

∂E

(56)

Determine pairwise alignment?

• p’=Mp, where M is a transformation matrix, p and p’ are feature matches

• For translation model, it is easier.

• What if the match is false? Avoid impact of outliers.

( ) ( )

[ ]

∑

=

− +

+

− +

= ⁿ

i

i i

i

i x m y y

x m

E

1

' 2 2

' 2 1

1

0 m

E

∂

= ∂

(57)

RANSAC

• RANSAC = Random Sample Consensus

• an algorithm for robust fitting of models in the presence of many data outliers

• Compare to robust statistics

• Given N data points x_i, assume that mjority of them are generated from a model with

parameters Θ, try to recover Θ.

(58)

RANSAC algorithm

Run k times:

(1) draw n samples randomly

(2) fit parameters Θ with these n samples (3) for each of other N-n points, calculate

its distance to the fitted model, count the number of inlier points, c

Output Θ with the largest c

How many times?

How big?

Smaller is better

How to define?

Depends on the problem.

(59)

How to determine k

p: probability of real inliers

P: probability of success after k trials

k

p

n

P = 1 − ( 1 − )

n samples are all inliers a failure

failure after k trials

) 1

log(

) 1

log(

p

n

k P

−

= −

293 0.5

6

97 0.6

6

35 0.5

3

k p

n for P=0.99

(60)

Example: line fitting

(61)

Example: line fitting

n=2

(62)

Model fitting

(63)

Measure distances

(64)

Count inliers

c=3

(65)

Another trial

c=3

(66)

The best model

c=15

(67)

RANSAC for Homography

(68)

RANSAC for Homography

(69)

RANSAC for Homography

(70)

Applications of panorama in VFX

• Background plates

• Image-based lighting

(71)

Spiderman 2 (background plate)

(72)

Troy (image-based lighting)

http://www.cgnetworks.com/story_custom.php?story_id=2195&page=4