Image stitching

Image stitching

Digital Visual Effects, Spring 2005 Yung-Yu Chuang

2005/3/30

with slides by Richard Szeliski, Steve Seitz, Matthew Brown and Vaclav Hlavac

Announcements

• Project #1 was due yesterday.

• Project #2 handout will be available on the web later tomorrow.

• I will set up a webpage for artifact voting soon.

Outline

• Image stitching

• Motion models

• Direct methods

• Feature-based methods

• Applications

• Project #2

Image stitching

• Stitching = alignment + blending

geometrical registration

photometric registration

Applications of image stitching

• Video stabilization

• Video summarization

• Video compression

• Video matting

• Panorama creation

Video summarization

Video compression Video matting

input video

Video matting

remove foreground

Video matting

estimate background

Video matting

background estimation

Video matting

alpha matte

Panorama creation Why panorama?

• Are you getting the whole picture?

– Compact Camera FOV = 50 x 35̓

Why panorama?

• Are you getting the whole picture?

– Compact Camera FOV = 50 x 35̓ – Human FOV = 200 x 135̓

Why panorama?

• Are you getting the whole picture?

– Compact Camera FOV = 50 x 35̓ – Human FOV = 200 x 135̓ – Panoramic Mosaic = 360 x 180̓

Panorama examples

• Panorama mode in consumer cameras

• Mars:

http://www.panoramas.dk/fullscreen3/f2_mars97.html

• Earth:

http://earthobservatory.nasa.gov/Newsroom/BlueMarble/

2D motion models

• translation: x’ = x + t x = (x,y)

• rotation: x’ = R x + t

• similarity: x’ = s R x + t

• affine: x’ = A x + t

• perspective: x’ # H x x = (x,y,1) (x is a homogeneous coordinate)

• These all form a nested group (closed under composition w/ inv.)

2D image transformations A pencil of rays contains all views

real

camera synthetic

camera

Can generate any synthetic camera view

as long as it has the same center of projection!

Mosaic as Image Reprojection

mosaic projection plane

• The images are reprojected onto a common plane

• The mosaic is formed on this plane

• Mosaic is a synthetic wide-angle camera

Changing camera center

• Does it still work? synthetic PP PP1

PP2

Planar scene (or far away)

• PP3 is a projection plane of both centers of projection, so we are OK!

• This is how big areal photographs are made

PP1

PP3

PP2

3D motion models

u

(X,Y,Z) f u

3D motion models

• Rotational

• Cylindrical

Direct methods

• Select a motion model and estimate parameters

• Direct methods use pixel-to-pixel matching

• We have covered this last time actually.

• We will show a case study on constructing cylindrical panorama using a direct method.

A case study: cylindrical panorama

1. Take pictures on a tripod (or handheld) 2. Warp to cylindrical coordinate

3. Compute pairwise alignments using the hierarchical Lucas-Kanade algorithm 4. Fix up the end-to-end alignment 5. Blending

6. Crop the result and import into a viewer

Taking pictures

Kaidan panoramic tripod head

Translation model A case study: cylindrical panorama

• What if you want a 360q field of view?

mosaic projection cylinder

– Map 3D point (X,Y,Z) onto cylinder

Cylindrical projection

X Y

Z

unit cylinder

unwrapped cylinder

– Convert to cylindrical coordinates

cylindrical image

– Convert to cylindrical image coordinates

f = 180 (pixels)

Cylindrical reprojection

f = 380 f = 280

Image 384x300

top-down view Focal length – the dirty secret…

A simple method for estimating f

We will discuss more accurate methods next time

Distortion

• Radial distortion of the image

– Caused by imperfect lenses

– Deviations are most noticeable for rays that pass through the edge of the lens

No distortion Pin cushion Barrel

Radial correction

• Correct for “bending” in wide field of view lenses

Input images

Cylindrical warping Alignment

• a rotation of the camera is a translation of the cylinder!

»»

»

¼ º

««

«

¬ ª





»¼

« º

¬ ª

»»

»

¼ º

««

«

¬ ª

¦

¦

¦

¦

¦

¦

y

x y

y

x x

y

x y

y

x x y

y

x x y

y

x x

y x I y x J I

y x I y x J I v

u I I

I

I I I

, , ,

2 ,

, ,

2

) , ( ) , (

) , ( ) , (

LucasKanadeStep

void LucasKanadeStep(CByteImage& img1, CByteImage& img2, float t[2]) { // Transform the image

Translation(img2, img2t, t);

// Compute the gradients and summed error by comparing img1 and img2t double A[2][2], b[2];

for (int y = 1; y < height-1; y++) { // ignore borders for (int x = 1; x < width-1; x++) {

// If both have full alphas, then compute and accumulate the error double e = img2t.Pixel(x, y, k) - img1.Pixel (x, y, k);

// Accumulate the matrix entries

double gx = 0.5*(img2t.Pixel(x+1, y, k) - img2t.Pixel(x-1, y, k));

double gy = 0.5*(img2t.Pixel(x, y+1, k) - img2t.Pixel(x, y-1, k));

A[0][0] += gx*gx; A[0][1] += gx*gy;

A[1][0] += gx*gy; A[1][1] += gy*gy;

b[0] += e*gx; b[1] += e*gy;

} }

LucasKanadeStep (cont.)

// Solve for the update At=b and update the vector double det = 1.0 / (A[0][0]*A[1][1] - A[1][0]*A[1][0]);

t[0] += (A[1][1]*b[0] - A[1][0]*b[1]) * det;

t[1] += (A[0][0]*b[1] - A[1][0]*b[0]) * det;

}

PyramidLucasKanade

void PyramidalLucasKanade(CByteImage& img1, CByteImage& img2, float t[2], int nLevels, int nLucasKanadeSteps)

{

CBytePyramid p1(img1); // Form the two pyramids CBytePyramid p2(img2);

// Process in a coarse-to-fine hierarchy for (int l = nLevels-1; l >= 0; l--) {

t[0] /= (1 << l); // scale the t vector t[1] /= (1 << l);

CByteImage& i1 = p1[l];

CByteImage& i2 = p2[l];

for (int k = 0; k < nLucasKanadeSteps; k++) LucasKanadeStep(i1, i2, t);

t[0] *= (1 << l); // restore the full scaling t[1] *= (1 << l);

} }

Gaussian pyramid

Blending

• Why blending: parallax, lens distortion, scene motion, exposure difference

Blending

Blending Blending

Assembling the panorama

• Stitch pairs together, blend, then crop

Problem: Drift

• Error accumulation

– small errors accumulate over time

Problem: Drift

• Solution

– add another copy of first image at the end – there are a bunch of ways to solve this problem

• add displacement of (y₁– y_n)/(n -1) to each image after the first

• compute a global warp: y’ = y + ax

• run a big optimization problem, incorporating this constraint

– best solution, but more complicated – known as “bundle adjustment”

(x₁,y₁)

• copy of first image (xn,yn)

End-to-end alignment and crop

Viewer: panorama

++

++

++ ++

example: http://www.cs.washington.edu/education/courses/cse590ss/01wi/projects/project1/students/dougz/index.html

Viewer: texture mapped model

example:http://www.panoramas.dk/

Feature-based methods

• Only use feature points to estimate parameters

• We will study the “Recognising panorama”

paper published in ICCV 2003

RANSAC

• RANSAC = Random Sample Consensus

• an algorithm for robust fitting of models in the presence of many data outliers

• Compare to robust statistics

• Given N data points x_i, assume that mjority of them are generated from a model with

parameters 4, try to recover 4.

RANSAC algorithm

Run k times:

(1) draw n samples randomly

(2) fit parameters 4 with these n samples (3) for each of other N-n points, calculate

its distance to the fitted model, count the number of inlier points, c

Output 4 with the largest c

How many times?

How big?

Smaller is better

How to define?

Depends on the problem.

How to determine k

p: probability of real inliers

P: probability of success after k trials k

p

P 1  ( 1  )

n samples are all inliers a failure

failure after k trials

) 1

log(

) 1 log(

p

k P





293 0.5 6

97 0.6 6

35 0.5 3

k p n forP=0.99

Example: line fitting

n=2

Model fitting Measure distances

Count inliers

c=3

Another trial

c=3

The best model

c=15

Recognising Panoramas

• 1D Rotations (T)

– Ordering Ÿ matching images

Recognising Panoramas

• 1D Rotations (T)

– Ordering Ÿ matching images

Recognising Panoramas

• 1D Rotations (T)

– Ordering Ÿ matching images

Recognising Panoramas

• 1D Rotations (T)

– Ordering Ÿ matching images

• 2D Rotations (q, f)

– Ordering Ÿ matching images

Recognising Panoramas

• 1D Rotations (T)

– Ordering Ÿ matching images

• 2D Rotations (q, f)

– Ordering Ÿ matching images

Recognising Panoramas

• 1D Rotations (T)

– Ordering Ÿ matching images

• 2D Rotations (q, f)

– Ordering Ÿ matching images

Recognising Panoramas

Overview

• SIFT Feature Matching

• Image Matching

• Bundle Adjustment

• Multi-band Blending

Nearest Neighbour Matching

• Find k-NN for each feature

– k | number of overlapping images (we use k = 4)

• Use k-d tree

– k-d tree recursively bi-partitions data at mean in the dimension of maximum variance

– Approximate nearest neighbours found in O(nlogn)

Overview

• SIFT Feature Matching

• Image Matching

– For each image, use RANSAC to select inlier features from 6 images with most feature matches

• Bundle Adjustment

• Multi-band Blending

RANSAC for Homography

RANSAC for Homography RANSAC for Homography

Probabilistic model for verification

• Compare probability that this set of RANSAC inliers/outliers was generated by a

correct/false image match

• Choosing values for p₁, p₀ and p_min

Finding the panoramas

Finding the panoramas Finding the panoramas

Finding the panoramas ^Overview

• SIFT Feature Matching

• Image Matching

• Bundle Adjustment

• Multi-band Blending

• Parameterise each camera by rotation and focal length

• This gives pairwise homographies

Homography for Rotation Error function

• Sum of squared projection errors

– n = #images

– I(i) = set of image matches to image i

– F(i, j) = set of feature matches between images i,j – r_ij^k= residual of k^thfeature match between images

i,j

• Robust error function

Overview

• SIFT Feature Matching

• Image Matching

• Bundle Adjustment

• Multi-band Blending

Multi-band Blending

• Burt & Adelson 1983

– Blend frequency bands over range v O

Low frequency (O > 2 pixels)

High frequency (O < 2 pixels)

2-band Blending Linear Blending

2-band Blending Results

Direct vs feature-based

• Direct methods use all information and can be very accurate, but they depend on the fragile

“brightness constancy” assumption

• Iterative approaches require initialization

• Not robust to illumination change and noise images

• In early days, direct method is better.

• Feature based methods are now more robust and potentially faster

• Even better, it can recognize panorama without initialization

Applications of panorama in VFX

• Background plates

• Image-based lighting

Spiderman 2 (background plate) Troy (image-based lighting)

http://www.cgnetworks.com/story_custom.php?story_id=2195&page=4

Project #2 Image stitching

• Assigned: 3/30

• Due: 11:59pm 4/19

• Work in pairs

Reference software

• Autostitch

http://www.cs.ubc.ca/~mbrown/autostitch/autostitch.html

• Many others are available online.

Bells & whistles

• Full SIFT implementation

• Recognizing panorama

• Bundle adjustment

• Handle dynamic objects

• Better blending techniques

Artifacts

• Take your own pictures and generate a stitched image, be creative.

• http://www.cs.washington.edu/education/courses/cse590ss/01wi/projec ts/project1/students/allen/index.html

• Common focal point

• Rotate your camera to increase vertical FOV

• Tripod

• Fixed exposure?

Tips for taking pictures

Submission

• You have to turn in your complete source, the executable, a html report and an artifact.

• Report page contains:

description of the project, what do you learn, algorithm, implementation details, results, bells and whistles…

• Artifacts must be made using your own program.

artifacts voting on forum.

Reference

• Richard Szeliski, Image Alignment and Stitching, unpublished draft, 2005.

• R. Szeliski and H.-Y. Shum. Creating full view panoramic image mosaics and texture-mapped models, SIGGRAPH 1997, pp251-258.

• M. Brown, D. G. Lowe, Recognising Panoramas, ICCV 2003.