### Image stitching

Digital Visual Effects
*Yung-Yu Chuang*

*with slides by Richard Szeliski, Steve Seitz, Matthew Brown and Vaclav Hlavac*

**Image stitching**

• Stitching = alignment + blending

geometrical registration

photometric registration

**Applications of image stitching**

• Video stabilization

• Video summarization

• Video compression

• Video matting

• Panorama creation

**Video summarization**

**Video compression** **Object removal**

input video

**Object removal**

remove foreground

**Object removal**

estimate background

**Object removal**

background estimation

**Panorama creation**

**Why panorama?**

• Are you getting the whole picture?

– Compact Camera FOV = 50 x 35°

**Why panorama?**

• Are you getting the whole picture?

– Compact Camera FOV = 50 x 35°

– Human FOV = 200 x 135°

**Why panorama?**

• Are you getting the whole picture?

– Compact Camera FOV = 50 x 35°

– Human FOV = 200 x 135°

– Panoramic Mosaic = 360 x 180°

**Panorama examples**

• Similar to HDR, it is a topic of computational photography, seeking ways to build a better camera using either hardware or software.

• Most consumer cameras have a panorama mode

• Mars:

http://www.panoramas.dk/fullscreen3/f2_mars97.html

• Earth:

http://www.panoramas.dk/new-year-2006/taipei.html http://www.360cities.net/

http://maps.google.com.tw/

**What can be globally aligned?**

• In image stitching, we seek for a matrix to globally warp one image into another. Are any two images of the same scene can be aligned this way?

– Images captured with the same center of projection

– A planar scene or far-away scene

**A pencil of rays contains all views**

real

camera synthetic

camera

Can generate any synthetic camera view

**as long as it has the same center of projection!**

**Mosaic as an image reprojection**

mosaic projection plane

• The images are reprojected onto a common plane

• The mosaic is formed on this plane

*• Mosaic is a synthetic wide-angle camera*

**Changing camera center**

• Does it still work? synthetic PP PP1

PP2

**What cannot **

• The scene with depth variations and the camera has movement

**Planar scene (or a faraway one)**

• PP3 is a projection plane of both centers of projection, so we are OK!

• This is how big aerial photographs are made

PP1

PP3

PP2

**Motion models**

• Parametric models as the assumptions on the relation between two images.

**2D Motion models**

**Motion models**

**Translation**

**2 unknowns**

**Affine**

**6 unknowns**

**Perspective**

**8 unknowns**

**3D rotation**

**3 unknowns**

**A case study: cylindrical panorama**

• What if you want a 360 field of view?

mosaic projection cylinder

**Cylindrical panoramas**

• Steps

– Reproject each image onto a cylinder – Blend

– Output the resulting mosaic

**applet**

• http://graphics.stanford.edu/courses/cs178/ap plets/projection.html

**Cylindrical panorama**

1. Take pictures on a tripod (or handheld) 2. Warp to cylindrical coordinate

3. Compute pairwise alignments 4. Fix up the end-to-end alignment 5. Blending

6. Crop the result and import into a viewer

It is required to do radial distortion correction for better stitching results!

**Taking pictures**

Kaidan panoramic tripod head

**Translation model** **Where should the synthetic camera be**

• The projection plane of some camera

• Onto a cylinder

real

camera synthetic

camera

**Cylindrical projection**

Adopted from http://www.cambridgeincolour.com/tutorials/image-projections.htm

**Cylindrical projection**

**Cylindrical projection**

Adopted from http://www.cambridgeincolour.com/tutorials/image-projections.htm

**Cylindrical projection**

unwrapped cylinder

*x*

*y*

*f*

θ

*x*

*z* **Cylindrical projection**

unwrapped cylinder

*x* *y*

θ

*x* *y*

*f*

**Cylindrical projection**

unwrapped cylinder

*x* *y*

*z*

*x* *y*

*f*

*s=f *

gives less
distortion
**f = 180 (pixels)**

**Cylindrical reprojection**

**f = 380**
**f = 280**

**Image 384x300**

top-down view Focal length – the dirty secret…

**A simple method for estimating f**

Or, you can use other software, such as AutoStich, to help.

*d*
*f*
*w*

*p*

**Input images** **Cylindrical warping**

**Blending**

• Why blending: parallax, lens distortion, scene motion, exposure difference

**Blending**

**Blending** **Blending**

**Gradient-domain stitching** **Gradient-domain stitching**

**Panorama weaving** **Assembling the panorama**

• Stitch pairs together, blend, then crop

**Problem: Drift**

• Error accumulation

– small errors accumulate over time

**Problem: Drift**

• Solution

– add another copy of first image at the end – there are a bunch of ways to solve this problem

• add displacement of (y_{1}–y_{n})/(n -1) to each image after
the first

• compute a global warp: y’ = y + ax

• run a big optimization problem, incorporating this constraint

– best solution, but more complicated – known as “bundle adjustment”

(x_{1},y_{1})

• copy of first
image
(x_{n},y_{n})

**End-to-end alignment and crop** **Rectangling panoramas**

**video**

**Rectangling panoramas** **Rectangling panoramas**

**Viewer: panorama**

**+**

**+**

**+**
**+**

example: http://www.cs.washington.edu/education/courses/cse590ss/01wi/projects/project1/students/dougz/index.html

**Viewer: texture mapped model**

example: http://www.panoramas.dk/

**Cylindrical panorama**

1. Take pictures on a tripod (or handheld) 2. Warp to cylindrical coordinate

3. Compute pairwise alignments 4. Fix up the end-to-end alignment 5. Blending

6. Crop the result and import into a viewer

**Determine pairwise alignment?**

• Feature-based methods: only use feature points to estimate parameters

• We will study the “Recognising panorama”

paper published in ICCV 2003

• Run SIFT (or other feature algorithms) for each image, find feature matches.

**Determine pairwise alignment**

• p’=Mp, where M is a transformation matrix, p and p’ are feature matches

• It is possible to use more complicated models such as affine or perspective

• For example, assume M is a 2x2 matrix

• Find M with the least square error

###

###

*n*

*i*

*p*
*Mp*

1

' 2

*y*
*x*
*m*
*m*

*m*
*m*
*y*

*x*

22 21

12 11

' '

**Determine pairwise alignment**

• Overdetermined system

*y*
*x*
*m*
*m*

*m*
*m*
*y*

*x*

22 21

12 11

' '

' 1 22 1 21 1

' 1 12 1 11 1

*y*
*m*
*y*
*m*
*x*

*x*
*m*
*y*
*m*
*x*

' ' ' 2

' 1 ' 1

22 21 12 11 2

2

1 1 1 1

0 0

0 0

0 0 0 0

0 0

*n*
*n*
*n*

*n*
*n*
*n*

*y*
*x*
*x*
*y*
*x*

*m*
*m*
*m*
*m*

*y*
*x*
*y*
*x*

*y*
*x*

*y*
*x*
*y*
*x*

**Normal equation**

Given an overdetermined system

**b** **Ax**

**b** **A** **Ax**

**A**

^{T}

###

^{T}

the normal equation is that which minimizes the sum of the square differences between left and right sides

Why?

**Normal equation **

###

^{2}

### )

### ( **x** **Ax** **b** *E*

###

###

###

###

###

###

###

###

###

###

###

###

###

###

###

###

###

###

###

###

###

###

###

###

###

###

###

###

###

*n*
*m*

*nm*
*n*

*m*

*b* *b*

*x* *x*

*a* *a*

*a* *a*

### : : : :

### ...

### : :

### : :

### : :

### ...

_{1}

1

1

1 11

*nxm, n equations, m variables*

**Normal equation**

###

###

###

###

###

###

*n*
*m*

*j*

*j*
*nj*

*i*
*m*

*j*

*j*
*ij*
*m*

*j*

*j*
*j*

*n*
*i*

*m*

*j*

*j*
*nj*
*m*

*j*

*j*
*ij*
*m*

*j*

*j*
*j*

*b*
*x*
*a*

*b*
*x*
*a*

*b*
*x*
*a*

*b*
*b*
*b*

*x*
*a*

*x*
*a*

*x*
*a*

1 1

1 1

1 1

1 1 1

1

: :

: :

:
:
**b**

**Ax**

###

^{n}

*i*

*i*
*m*

*j*

*j*

*ij**x* *b*

*a*
*E*

1

2

1

) 2

(**x** **Ax** **b**

**Normal equation**

###

^{n}

*i*

*i*
*m*

*j*

*j*

*ij**x* *b*

*a*
*E*

1

2

1

) 2

(**x** **Ax** **b**

1

0 *x*

*E*

###

###

###

###

^{n}*i*

*i*
*i*
*n*

*i*

*j*
*m*

*j*
*ij*

*i*

*a* *x* *a* *b*

*a*

1 1

1 1

1

### 2

### 2

1

1 1

### 2

_{i}*n*

*i*

*i*
*m*

*j*

*j*

*ij*

*x* *b* *a*

### *a*

###

###

###

###

###

###

###

###

###

### ) (

### 2

### 0 **A** **Ax** **A** **b**

**x**

**T**

**T**

###

###

### *E*

**b** **A** **Ax**

**A**

^{T}

###

^{T}

**→**

**Normal equation**

###

**Ax**

**b**

###

^{2}

**Normal equation**

###

###

## ^{} ^{}

### ^{} ^{}

###

^{A}

^{b}

^{x}

^{A}

^{b}

^{x}

^{b}

^{b}**Ax**
**A**
**x**

**b**
**b**
**b**
**A**
**x**
**Ax**
**b**
**Ax**
**A**
**x**

**b**
**Ax**
**b**

**A**
**x**

**b**
**Ax**
**b**

**Ax**

**b**
**Ax**
**b**

**Ax**
**b**
**Ax**

**T** **T**
**T** **T**

**T**
**T**

**T**

**T**
**T**

**T**
**T**

**T**
**T**

**T**
**T**
**T**

*T* *T*
*T*
2

**b**
**A**
**Ax**
**x** **A**

**T**

**T** 2

2

*E*

**Determine pairwise alignment**

• p’=Mp, where M is a transformation matrix, p and p’ are feature matches

• For translation model, it is easier.

• What if the match is false? Avoid impact of outliers.

###

##

###

^{n}

*i*

*i*
*i*
*i*

*i* *x* *m* *y* *y*

*x*
*m*
*E*

1

' 2 2

' 2 1

1

0 *m*

*E*

**RANSAC**

• RANSAC = Random Sample Consensus

• An algorithm for robust fitting of models in the presence of many data outliers

• Compare to robust statistics

*• Given N data points x** _{i}*, assume that majority of
them are generated from a model with

parameters , try to recover .

**RANSAC algorithm**

*Run k times:*

*(1) draw n samples randomly*

*(2) fit parameters with these n samples*
*(3) for each of other N-n points, calculate *

its distance to the fitted model, count the
*number of inlier points, c*

*Output with the largest c*

How many times?

How big?

Smaller is better

How to define?

Depends on the problem.

**How to determine k**

*p: probability of real inliers*

*P: probability of success after k trials*
*k*

*p*

*n*

*P* 1 ( 1 )

n samples are all inliers a failure

failure after k trials

### ) 1

### log(

### ) 1 log(

*p*

*n*

*k* *P*

###

###

*n* *p* *k*

3 0.5 35

6 0.6 97

6 0.5 293
*for P=0.99*

**Example: line fitting** **Example: line fitting**

*n=2*

**Model fitting** **Measure distances**

**Count inliers**

*c=3*

**Another trial**

*c=3*

**The best model**

*c=15*

**RANSAC for Homography**

**RANSAC for Homography** **RANSAC for Homography**

**Applications of panorama in VFX**

• Background plates

• Image-based lighting

**Troy (image-based lighting)**

http://www.cgnetworks.com/story_custom.php?story_id=2195&page=4

**Spiderman 2 (background plate)** **Reference**

• Richard Szeliski, Image Alignment and Stitching: A Tutorial,
*Foundations and Trends in Computer Graphics and Computer *
*Vision, 2(1):1-104, December 2006. *

• R. Szeliski and H.-Y. Shum. Creating full view panoramic image mosaics and texture-mapped models, SIGGRAPH 1997, pp251-258.

• M. Brown, D. G. Lowe, Recognising Panoramas, ICCV 2003.