# Image warping

(1)

## Image warping/morphing

Digital Visual Effectsg Yung-Yu Chuang

with slides by Richard Szeliski, Steve Seitz, Tom Funkhouser and Alexei Efros

(2)

(3)

B

A

(4)

(5)

### What is an image

• We can think of an image as a function, f: R2R:

f( ) i th i t it t iti ( ) – f(x, y) gives the intensity at position (x, y) – defined over a rectangle, with a finite range:

f [ b] [ d]  [0 1]

• f: [a,b]x[c,d]  [0,1]

x f

y

• A color image ( , )

( , ) ( , )

r x y

f x y g x y

 

( , ) ( , )

( , )

f y g y

b x y

(6)

### A digital image

• We usually operate on digital (discrete) images:

S l th 2D l id – Sample the 2D space on a regular grid

– Quantize each sample (round to nearest integer)

f l h

• If our samples are D apart, we can write this as:

f[i ,j] = Quantize{ f(i D, j D) }

• The image can now be represented as a matrix of integer values

o tege values

(7)

### Image warping

image filtering: change range of image ( ) h(f( ))

g(x) = h(f(x))

f

h

g

h(y)=0.5y+0.5

x

h

x

image warping: change domain of image

f

g(x) = f(h(x))

h(y)=2y

f

h

g

x x

(8)

### Image warping

image filtering: change range of image ( ) h(f( ))

f g

g(x) = h(f(x))

h(y)=0.5y+0.5 h

f g

image warping: change domain of image

f

g(x) = f(h(x))

h([x,y])=[x,y/2]

h

f

g

(9)

### Parametric (global) warping

Examples of parametric warps:

Examples of parametric warps:

translation rotation aspect

affine perspective

cylindrical

(10)

### Parametric (global) warping

T T

p = (x,y) p’ = (x’,y’)

• Transformation T is a coordinate-changing machine: p’ = T(p)

p (x,y) p (x ,y )

machine: p T(p)

• What does it mean that T is global?

Is the same for any point p – Is the same for any point p

– can be described by just a few numbers (parameters)

R t T t i ’ M*

• Represent T as a matrix: p’ = M*p

(11)

### Scaling

• Scaling a coordinate means multiplying each of its components by a scalar

its components by a scalar

• Uniform scaling means this scalar is the same for all components:

for all components:

f g

 2

(12)

### Scaling

• Non-uniform scaling: different scalars per

component:

component:

x  2, y  0.5

(13)

### Scaling

• Scaling operation:

### x '  axbyy ' 

• Or, in matrix form:

###  y 0 by

scaling matrix S scaling matrix S What’s inverse of S?

(14)

### 2-D Rotation

• This is easy to capture in matrix form:

###  y sin  cos  y

R

• Even though sin() and cos() are nonlinear to ,

i li bi ti f d

R

– x’ is a linear combination of x and y – y’ is a linear combination of x and y

Wh i h i f i ?

• What is the inverse transformation?

– Rotation by –

T

– For rotation matrices, det(R) = 1 so R11 RT

(15)

### 2x2 Matrices

• What types of transformations can be represented with a 2x2 matrix?

represented with a 2x2 matrix?

2D Identity?

### ' '

2D Scale around (0,0)?

y

x

y x

y

y

(16)

### 2x2 Matrices

• What types of transformations can be represented with a 2x2 matrix?

represented with a 2x2 matrix?

2D Rotate around (0,0)?

* i

*

'   i

y x

y

y x

x

* cos

* sin

'

* sin

* cos

'

y x y

x

cos sin

sin cos

' '

2D Shear?

y x

sh y

y sh

x x

y

x

* '

*

'

x

y x

sh

y y

y

(17)

### 2x2 Matrices

• What types of transformations can be represented with a 2x2 matrix?

represented with a 2x2 matrix?

'

y y

x x

'

'

### ' '

2D Mirror over (0,0)?

y y

x x

'

'

y

y

(18)

### All 2D Linear Transformations

• Linear transformations are combinations of …

Scale – Scale,

– Rotation, – Shear andShear, and – Mirror

• Properties of linear transformations:

• Properties of linear transformations:

– Origin maps to origin – Lines map to linesLines map to lines

– Parallel lines remain parallel – Ratios are preserved

###  x'abx

p

– Closed under composition

(19)

### 2x2 Matrices

• What types of transformations can not be represented with a 2x2 matrix?

represented with a 2x2 matrix?

2D Translation?

t '

y x

t y y

t x x

'

' NO!

Only linear 2D transformations

b t d ith 2 2 t i

can be represented with a 2x2 matrix

(20)

### Translation

• Example of translation

Homogeneous Coordinates

1 1

1 0 0

1 0

0 1

1 ' '

y x y

x

t y

t x y

x t

t y

x













1 0 0 1 1 1

tx = 2 tyy = 1

(21)

### Affine Transformations

• Affine transformations are combinations of …

Li f i d

– Linear transformations, and – Translations

• Properties of affine transformations:

– Origin does not necessarily map to origin – Lines map to lines

– Parallel lines remain parallel – Ratios are preserved

– Closed under composition

### ' '

– Models change of basis

(22)

### Projective Transformations

• Projective transformations …

Affine transformations and – Affine transformations, and – Projective warps

P ti f j ti t f ti

• Properties of projective transformations:

– Origin does not necessarily map to origin l

– Lines map to lines

– Parallel lines do not necessarily remain parallel – Ratios are not preserved

– Closed under composition

###   x'abcx

– Models change of basis

(23)

### Image warping

• Given a coordinate transform x’ = T(x) and a source image I(x) how do we compute a

source image I(x), how do we compute a transformed image I’(x’) = I(T(x))?

T(x)

I(x) I’(x’)

x x’

(x)

I(x) I (x )

(24)

### Forward warping

• Send each pixel I(x) to its corresponding location x’ T(x) in I’(x’)

location x’ = T(x) in I’(x’)

T(x)

I(x) I’(x’)

x x’

(x)

I(x) I (x )

(25)

### Forward warping

fwarp(I, I’, T) {

{

for (y=0; y<I.height; y++)

for (x=0; x<I width; x++) { for (x=0; x<I.width; x++) {

(x’,y’)=T(x,y);

I’(x’ y’) I(x y);

I’(x’,y’)=I(x,y);

} }

} I I’

T x

x’

(26)

### Forward warping

• Send each pixel I(x) to its corresponding location x’ T(x) in I’(x’)

location x’ = T(x) in I’(x’)

• What if pixel lands “between” two pixels?p p

• Will be there holes?

h(x)

f(x) g(x’)

x x’

(x)

f(x) g(x )

(27)

### Forward warping

fwarp(I, I’, T) {

{

for (y=0; y<I.height; y++)

for (x=0; x<I width; x++) { for (x=0; x<I.width; x++) {

(x’,y’)=T(x,y);

Splatting(I’ x’ y’ I(x y) kernel);

Splatting(I’,x’,y’,I(x,y),kernel);

} }

} I I’

T x

x’

(28)

### Inverse warping

• Get each pixel I’(x’) from its corresponding location x T-1(x’) in I(x)

location x = T 1(x’) in I(x)

T-1(x’)

I(x) I’(x’)

x x’

(x )

I(x) I (x )

(29)

### Inverse warping

iwarp(I, I’, T) {

{

for (y=0; y<I’.height; y++)

for (x=0; x<I’ width; x++) { for (x=0; x<I’.width; x++) {

(x,y)=T-1(x’,y’);

I’(x’ y’) I(x y);

I’(x’,y’)=I(x,y);

} }

} I T-1 I’

x

x’

(30)

### Inverse warping

• Get each pixel I’(x’) from its corresponding location x T-1(x’) in I(x)

location x = T 1(x’) in I(x)

• What if pixel comes from “between” two pixels?

• Answer: resample color value from

• Answer: resample color value from

interpolated (prefiltered) source image

f(x) g(x’)

x x’

f(x) g(x )

(31)

### Inverse warping

iwarp(I, I’, T) {

{

for (y=0; y<I’.height; y++)

for (x=0; x<I’ width; x++) { for (x=0; x<I’.width; x++) {

(x,y)=T-1(x’,y’);

I’(x’ y’) Reconstruct(I x y kernel);

I’(x’,y’)=Reconstruct(I,x,y,kernel);

} }

} I T-1 I’

x

x’

(32)

band limited

(33)

### Reconstruction

The reconstructed function is obtained by interpolating The reconstructed function is obtained by interpolating among the samples in some manner

(34)

### Reconstruction

• Reconstruction generates an approximation to the original function Error is called aliasing

the original function. Error is called aliasing.

l l

sampling reconstruction

sample value

l i i sample position

(35)

### Reconstruction

• Computed weighted sum of pixel neighborhood;

output is weighted average of input where output is weighted average of input, where

weights are normalized values of filter kernel k

k(q )q

color=0;

###  

i i

i i i

q k

q q p k

) (

) (

color=0;

weights=0;

for all q’s dist < width width

d

q

d = dist(p, q);

w = kernel(d);

p

d color += w*q.color;

weights += w;

p Color = color/weights;

q

p.Color = color/weights;

(36)

### Reconstruction (interpolation)

• Possible reconstruction filters (kernels):

t i hb – nearest neighbor – bilinear

bi bi – bicubic

– sinc (optimal reconstruction)

(37)

### Bilinear interpolation (triangle filter)

• A simple method for resampling images

(38)

### Non-parametric image warping

• Specify a more detailed warp function

S li h i l fl ( i l i )

• Splines, meshes, optical flow (per-pixel motion)

(39)

### Non-parametric image warping

• Mappings implied by correspondences

I i

• Inverse warping

? P’

(40)

### Non-parametric image warping

' '

'

' w A w B w C PABC C

w B

w A

w

P   

Barycentric coordinate

C w B

w A

w

PABC

P P’

(41)

3 1

2 1

3 3 2

2 1

1

t t

t

A t A

t A

t P

3 1

2

1tt t

(42)

### Non-parametric image warping

' '

'

' w A w B w C PABC C

w B

w A

w

P   

Barycentric coordinate

C w B

w A

w

PABC

(43)

) 2

(r e r

Gaussian P K1

## 

kXi (P')Xi

) log(

)

(rr2 r

thin plate

spline

K i

(44)

### Demo

• http://www.colonize.com/warp/warp04-2.php W i i f l i f i id

• Warping is a useful operation for mosaics, video matching, view interpolation and so on.

(45)

(46)

### Image morphing

• The goal is to synthesize a fluid transformation from one image to another

from one image to another.

• Cross dissolving is a common transition between cuts, but it is not good for morphing because of the ghosting effects.

i g #1 di l i g image #2

image #1 dissolving image #2

(47)

### Artifacts of cross-dissolving

http://www.salavon.com/

(48)

### Image morphing

• Why ghosting?

M hi i di l i

• Morphing = warping + cross-dissolving

shape

(geometric)

color

(photometric) (geometric) (photometric)

(49)

### Image morphing

cross-dissolving

image #1 image #2

morphing

warp warp

(50)

(51)

average faces

(52)

### Image morphing

create a morphing sequence: for each time t

1 C t i t di t i fi ld (b 1. Create an intermediate warping field (by

interpolation)

2 Warp both images towards it 2. Warp both images towards it

3. Cross-dissolve the colors in the newly warped images

images

t=0 t=0.33 t=1

(53)

### An ideal example (in 2004)

t=0 morphingt=0.25t=0.75t=0.5 t=1

(54)

### An ideal example

middle face (t=0.5)

t=0 t=1

(55)

### Warp specification (mesh warping)

• How can we specify the warp?

1 S if di li t l i t

1. Specify corresponding spline control points

interpolate to a complete warping function

easy to implement but less expressive easy to implement, but less expressive

(56)

### Warp specification

• How can we specify the warp

2 S if di i t 2. Specify corresponding points

• interpolate to a complete warping function

(57)

### Solution: convert to mesh warping

1 D fi t i l h th i t 1. Define a triangular mesh over the points

– Same mesh in both images!

N h i l i l d

– Now we have triangle-to-triangle correspondences

2. Warp each triangle separately from source to destination – How do we warp a triangle?

– 3 points = affine warp!

– Just like texture mapping

(58)

### Warp specification (field warping)

• How can we specify the warp?

3 S if di t

3. Specify corresponding vectors

• interpolate to a complete warping function

• The Beier & Neely Algorithm

• The Beier & Neely Algorithm

(59)

### Beier&Neely (SIGGRAPH 1992)

• Single line-pair PQ to P’Q’:

(60)

### Algorithm (single line-pair)

• For each X in the destination image:

1 Fi d th di

1. Find the corresponding u,v

2. Find X’ in the source image for that u,v

3 d ti ti I (X) I (X’)

3. destinationImage(X) = sourceImage(X’)

• Examples:

Affine transformation Affine transformation

(61)

### Multiple Lines

X X

Di Xi' Xi D

length = length of the line segment, dist = distance to line segment

The influence of a, p, b. The same as the average of Xi

(62)

(63)

(64)

### Comparison to mesh morphing

• Pros: more expressive

C d d l

• Cons: speed and control

(65)

### Warp interpolation

• How do we create an intermediate warp at time t?

time t?

– linear interpolation for line end-points

B t li t ti 180 d ill b 0 – But, a line rotating 180 degrees will become 0

length in the middle

One solution is to interpolate line mid point and – One solution is to interpolate line mid-point and

orientation angle

t=0

t=1 t=1

(66)

(67)

### Animated sequences

• Specify keyframes and interpolate the lines for the inbetween frames

the inbetween frames

• Require a lot of tweaking

(68)

### Results

Michael Jackson’s MTV “Black or White”

(69)

(70)

(71)

### References

• Thaddeus Beier, Shawn Neely, Feature-Based Image Metamorphosis, SIGGRAPH 1992, pp35-42.

• Detlef Ruprecht, Heinrich Muller, Image Warping with Scattered Data Interpolation, IEEE Computer Graphics and Applications, March 1995 pp37-43

March 1995, pp37 43.

• Seung-Yong Lee, Kyung-Yong Chwa, Sung Yong Shin, Image Metamorphosis Using Snakes and Free-Form Deformations, SIGGRAPH 1995

SIGGRAPH 1995.

• Seungyong Lee, Wolberg, G., Sung Yong Shin, Polymorph: morphing among multiple images, IEEE Computer Graphics and Applications, g p g , p p pp , Vol. 18, No. 1, 1998, pp58-71.

• Peinsheng Gao, Thomas Sederberg, A work minimization approach to image morphing The Visual Computer 1998 pp390 400

to image morphing, The Visual Computer, 1998, pp390-400.

• George Wolberg, Image morphing: a survey, The Visual Computer, 1998, pp360-372.

Updating...

## References

Related subjects :