• 沒有找到結果。

Image stitching

N/A
N/A
Protected

Academic year: 2022

Share "Image stitching"

Copied!
74
0
0

加載中.... (立即查看全文)

全文

(1)

Image stitching

Digital Visual Effectsg Yung-Yu Chuang

with slides by Richard Szeliski, Steve Seitz, Matthew Brown and Vaclav Hlavac

(2)

Image stitching

• Stitching = alignment + blending

geometrical photometric registration registration

(3)

Applications of image stitching

• Video stabilization

Vid i i

• Video summarization

• Video compression

• Video matting

• Panorama creation

• Panorama creation

(4)

Video summarization

(5)

Video compression

(6)

Object removal

input video

(7)

Object removal

remove foreground

(8)

Object removal

estimate background

(9)

Object removal

background estimation

(10)

Panorama creation

(11)

Why panorama?

• Are you getting the whole picture?

C t C FOV 50 35°

– Compact Camera FOV = 50 x 35°

(12)

Why panorama?

• Are you getting the whole picture?

C t C FOV 50 35°

– Compact Camera FOV = 50 x 35°

– Human FOV = 200 x 135°

(13)

Why panorama?

• Are you getting the whole picture?

C t C FOV 50 35°

– Compact Camera FOV = 50 x 35°

– Human FOV = 200 x 135°

P i M i 360 180°

– Panoramic Mosaic = 360 x 180°

(14)

Panorama examples

• Like HDR, it is a topic of computational

photography seeking ways to build a better photography, seeking ways to build a better camera mostly in software.

M t h d

• Most consumer cameras have a panorama mode

• Mars:

http://www.panoramas.dk/fullscreen3/f2_mars97.html

• Earth:

http://www.panoramas.dk/new-year-2006/taipei.html http://www.360cities.net/

http://www.360cities.net/

(15)

What can be globally aligned?

• In image stitching, we seek for a matrix to

globally warp one image into another Are any globally warp one image into another. Are any two images of the same scene can be aligned this way?

this way?

– Images captured with the same center of j ti

projection

– A planar scene or far-away scene

(16)

A pencil of rays contains all views

real synthetic

camera y

camera

Can generate any synthetic camera viewg y y

as long as it has the same center of projection!

(17)

Mosaic as an image reprojection

mosaic projection plane

• The images are reprojected onto a common plane

• The mosaic is formed on this planeThe mosaic is formed on this plane

• Mosaic is a synthetic wide-angle camera

(18)

Changing camera center

• Does it still work? synthetic PP PP1

PP2

(19)

Planar scene (or a faraway one)

PP3 PP1

PP2

• PP3 is a projection plane of both centers of projection, so we are OK!

• This is how big aerial photographs are made

(20)

Motion models

• Parametric models as the assumptions on the relation between two images

relation between two images.

(21)

2D Motion models

(22)

Motion models

Translation Affine Perspective 3D rotation

2 unknowns 6 unknowns 8 unknowns 3 unknowns

(23)

A case study: cylindrical panorama

• What if you want a 360 field of view?

mosaic projection cylinder

(24)

Cylindrical panoramas

• Steps

– Reproject each image onto a cylinder – Blend

– Output the resulting mosaic

(25)

Cylindrical panorama

1. Take pictures on a tripod (or handheld) 2 W li d i l di

2. Warp to cylindrical coordinate 3. Compute pairwise alignments

4. Fix up the end-to-end alignment 5 Blending

5. Blending

6. Crop the result and import into a viewer

It is required to do radial distortion correction for better stitching results!

correction for better stitching results!

(26)

Taking pictures

K id i t i d h d

Kaidan panoramic tripod head

(27)

Translation model

Try to align this in PaintShop Pro

(28)

Where should the synthetic camera be

real synthetic

camera y

camera

• The projection plan of some camera

• Onto a cylinder

• Onto a cylinder

(29)

Cylindrical projection

Adopted from http://www.cambridgeincolour.com/tutorials/image-projections.htm

(30)

Cylindrical projection

(31)

Cylindrical projection

Adopted from http://www.cambridgeincolour.com/tutorials/image-projections.htm

(32)

Cylindrical projection

y

unwrapped cylinder

x

y

θ

x

f

θ

(33)

Cylindrical projection

y

unwrapped cylinder

x

y

z f

θ

f

x

(34)

Cylindrical projection

y

unwrapped cylinder

x

y

z f

s=f

gives less distortion

f

distortion

x

(35)

Cylindrical reprojection

top-down viewp Focal lengthFocal length – the dirty secret…– the dirty secret

f = 180 (pixels) f = 280 f = 380 Image 384x300

(36)

A simple method for estimating f

w

d f

p

f

Or, you can use other software, such as AutoStich, to help.

to help.

(37)

Input images

(38)

Cylindrical warping

(39)

Blending

• Why blending: parallax, lens distortion, scene motion exposure difference

motion, exposure difference

(40)

Blending

(41)

Blending

(42)

Blending

(43)

Assembling the panorama

• Stitch pairs together blend then crop

• Stitch pairs together, blend, then crop

(44)

Problem: Drift

• Error accumulation

– small errors accumulate over time

(45)

Problem: Drift

(x1,y1)

(xn,yn)

• Solution copy of first

• Solution

– add another copy of first image at the end

– there are a bunch of ways to solve this problem

image

– there are a bunch of ways to solve this problem

• add displacement of (y1 yn)/(n -1) to each image after the first

• compute a global warp: y’ = y + ax

• run a big optimization problem, incorporating this constraint

– best solution, but more complicated – known as “bundle adjustment”

(46)

End-to-end alignment and crop

(47)

Viewer: panorama

+ +

+ +

+ + +

+

+ +

example: http://www.cs.washington.edu/education/courses/cse590ss/01wi/projects/project1/students/dougz/index.html

(48)

Viewer: texture mapped model

example: http://www.panoramas.dk/

(49)

Cylindrical panorama

1. Take pictures on a tripod (or handheld) 2 W li d i l di

2. Warp to cylindrical coordinate 3. Compute pairwise alignments

4. Fix up the end-to-end alignment 5 Blending

5. Blending

6. Crop the result and import into a viewer

(50)

Determine pairwise alignment?

• Feature-based methods: only use feature points to estimate parameters

to estimate parameters

• We will study the “Recognising panorama”

paper published in ICCV 2003

• Run SIFT (or other feature algorithms) for each

• Run SIFT (or other feature algorithms) for each image, find feature matches.

(51)

Determine pairwise alignment

• p’=Mp, where M is a transformation matrix, p and p’ are feature matches

and p are feature matches

• It is possible to use more complicated models

h ffi ti

such as affine or perspective

• For example, assume M is a 2x2 matrix



 



 

 



 

y x m

m

m m

y x

22 21

12 11

' '

• Find M with the least square error

 

n 2

y m21 m22 y

 

i

p Mp

1

' 2

(52)

Determine pairwise alignment











 x' m11 m12 x x1m11y1m12x1'









 

 

y' m21 m22 y '

1 22

1 21

1m y m y

x  

• Overdetermined system

 

 0 0 '





















' 1 1 11

1 1

1 1

0 0

0 0

y x m

y x

y x





 













'

12 2 2

2 0 0 x

m m y

x

 





















' 22

21

0

0 n

n

n m x

m y

x

 







 0 0 xn yn yn'

(53)

Normal equation

Given an overdetermined system

b Ax

the normal equation is that which minimizes the sum of the square differences between left and sum of the square differences between left and right sides

b A

Ax

A

T

T

Why?

(54)

Normal equation

 

2

)

( xAxb

E ( )  

a

11

a

1

b

1

 

 

 

 

 

 

m

b

x a

a

: :

:

...

1

1 1

11

 

 

 

 

 

 

 

: : :

:

1

 

 

 

 

 

 

 

 

 

m

b

x :

: :







 a

n1

... a

nm

b

n

nxm n equations m variables

nxm, n equations, m variables

(55)

Normal equation

 

 

 

 

 

  

m a x

m a1jxj b1

 

 

 

 

 

 

 

 

 

 

  

j

j j j

j

j a x b

x b

a 1

1 1 1

1 1

: : :

 

 

  

 

 

 

 

 

 

 

 

 

  

m i

j

j ij i

m

j

j

ij x b a x b

a

1 1

:

b

Ax

 

 

 

 

 

 

 

 

 

 

m j

m j

b

1 1

: : :

 

 

 

  

 

 

 

 

  

n

m

j

j nj n

j

j

nj a x b

x b a

1 1

   

n m

a

ij

x

j

b

i

E

2

)

2

( xAx b   

 

 

  

 

i

i j

j

ij

x b

a E

1 1

)

( x Ax b

(56)

Normal equation

   

n

m

b

E

2

)

2

(  A b   

 

 

   

 

 

i

i j

j

ij

x b

a E

1 1

)

( x Ax b

  

 

 

0 x

E

2

i1

n

i m

j

ij

x b a

  a

 

 

  

 

x

1

 

n m n

1 1

i j

j

 

j



 

 

 

i

i i i

j j

ij

i

a x a b

a

1

1

1 1

1

2

2

) (

2

0   A

T

AxA

T

b

E

b A

Ax

A

T

T

)

( 2

0 A Ax A b

x

  A AxA b

(57)

Normal equation

Ax b

2

   

   Ax b Ax b

T T

T

   

x Ax

T

A

T

b b

T

Ax Ax b b

T T

 

b b b

A x

Ax b

Ax A

x

b Ax

b A

x

T T

T T

T

T

  

    A b x A b x b b

Ax A

x

T T

T T

T T

T

b A Ax

A

T

2

T

2 

 

E

x

(58)

Determine pairwise alignment

• p’=Mp, where M is a transformation matrix, p and p’ are feature matches

and p are feature matches

• For translation model, it is easier.

 

 

n

i

i i

i

i x m y y

x m

E

1

' 2 2

' 2 1

0 E

1

0 m

• What if the match is false? Avoid impact of outliers.

(59)

RANSAC

• RANSAC = Random Sample Consensus

A l i h f b fi i f d l i h

• An algorithm for robust fitting of models in the presence of many data outliers

• Compare to robust statistics

• Given N data points xi, assume that mjority of them are generated from a model with

them are generated from a model with parameters , try to recover .

(60)

RANSAC algorithm

Run k times:

(1) d l d l

How many times?

How big?

(1) draw n samples randomly

(2) fit parameters  with these n samples

How big?

Smaller is better

(3) for each of other N-n points, calculate

its distance to the fitted model count the its distance to the fitted model, count the number of inlier points, c

O t t  ith th l t Output  with the largest c

How to define?

Depends on the problem.

(61)

How to determine k

p: probability of real inliers

P: probability of success after k trials P: probability of success after k trials

k

p

n

P  1  ( 1  p ) P 1 ( 1 )

n samples are all inliers a failure

f il ft k t i l failure after k trials

) 1

l ( P

n p k

) 1

log(

) 1

log(

p

n

k P

 

3 0.5 35

6 0.6 97

for P=0.99

) 1

log( p

6 0.5 293

(62)

Example: line fitting

(63)

Example: line fitting

n=2 n=2

(64)

Model fitting

(65)

Measure distances

(66)

Count inliers

c=3 c=3

(67)

Another trial

c=3 c=3

(68)

The best model

c=15 c=15

(69)

RANSAC for Homography

(70)

RANSAC for Homography

(71)

RANSAC for Homography

(72)

Applications of panorama in VFX

• Background plates I b d li h i

• Image-based lighting

(73)

Troy (image-based lighting)

http://www.cgnetworks.com/story custom.php?story id=2195&page=4 http://www.cgnetworks.com/story_custom.php?story_id 2195&page 4

(74)

Spiderman 2 (background plate)

參考文獻

相關文件

• The scene with depth variations and the camera has movement... Planar scene (or a

• Richard Szeliski, Image Alignment and Stitching: A Tutorial, Foundations and Trends in Computer Graphics and Computer Vision, 2(1):1-104, December 2006. Szeliski

• Richard Szeliski, Image Alignment and Stitching: A Tutorial, Foundations and Trends in Computer Graphics and Computer Vision, 2(1):1-104, December 2006. Szeliski

• The scene with depth variations and the camera has movement... Planar scene (or a

It is required to do radial distortion correction for better stitching results. correction for better

It is required to do radial distortion correction for better stitching results. correction for better

Creating full view panoramic image mosaics and texture-mapped models, SIGGRAPH 1997, pp251-258. Lowe, Recognising Panoramas,

• We will show a case study on constructing cylindrical panorama using a direct method.... Warp to