## Image warping/morphing

Digital Visual Effectsg
*Yung-Yu Chuang*

*with slides by Richard Szeliski, Steve Seitz, Tom Funkhouser and Alexei Efros*

**Image warping**

**Image warping**

**Image formation**

B

A

**Sampling and quantization**

**What is an image**

**• We can think of an image as a function, f: R**^{2}R:

*f( ) i* ** th i t** **it** t iti *( ) *
* – f(x, y) gives the intensity at position (x, y) *
– defined over a rectangle, with a finite range:

*f [ b] [ d] [0 1]*

*• f: [a,b]x[c,d] [0,1]*

x f

y

• A color image ^{( ,} ^{)}

( , ) ( , )

*r x y*

*f* *x y* *g x y*

( , ) ( , )

( , )

*f* *y* *g* *y*

*b x y*

**A digital image**

**• We usually operate on digital (discrete) images:**

**S** **l th 2D ** l id
**– Sample the 2D space on a regular grid**

**– Quantize each sample (round to nearest integer)**

f l h

• If our samples are D apart, we can write this as:

*f[i ,j] = Quantize{ f(i D, j D) }*

• The image can now be represented as a matrix of integer values

o tege values

**Image warping**

**image filtering: change range of image***( ) h(f( ))*

*g(x) = h(f(x))*

*f*

*h*

*g*

*h(y)=0.5y+0.5*

*x*

*h*

*x*

**image warping: change domain of image**

*f*

*g(x) = f(h(x))*

*h(y)=2y*

*f*

*h*

*g*

*x* *x*

**Image warping**

**image filtering: change range of image***( ) h(f( ))*

*f* *g*

*g(x) = h(f(x))*

*h(y)=0.5y+0.5*
*h*

*f* *g*

**image warping: change domain of image**

*f*

*g(x) = f(h(x))*

*h([x,y])=[x,y/2]*

*h*

*f*

*g*

**Parametric (global) warping**

Examples of parametric warps:

Examples of parametric warps:

translation rotation aspect

affine perspective

cylindrical

**Parametric (global) warping**

*T*
*T*

**p = (x,y)** **p’ = (x’,y’)**

• Transformation T is a coordinate-changing
*machine: p’ = T(p)*

**p (x,y)** **p** (x ,y )

*machine: p T(p)*

*• What does it mean that T is global?*

Is the same for any point p – Is the same for any point p

– can be described by just a few numbers (parameters)

R *t T* **t i ’ M***

**• Represent T as a matrix: p’ = M*p**

###

###

###

###

###

###

###

### *x* *x*

### ' **M**

### '

###

###

###

### *y'* *y*

**Scaling**

*• Scaling a coordinate means multiplying each of *
its components by a scalar

its components by a scalar

*• Uniform scaling means this scalar is the same *
for all components:

for all components:

**f****g**

###

###

###

### *x* ' 2 *x*

###

*x*

###

###

###

###

###

###

###

###

*y* *y* ' 2

###

###

###

### *y*

2

**Scaling**

*• Non-uniform scaling: different scalars per *

component:

### '

component:

###

###

###

###

###

###

###

###

###

###

###

###

###

###

###

###

### ' ' *y* *g* *x*

*y* *f* *x*

###

###

###

### *x* ' 2 *x*

###

###

###

### *y* *y*

###

###

###

###

###

###

###

###

*y* *y* ' 0 . 5

x 2, y 0.5

**Scaling**

• Scaling operation:

*x* ' *ax* *by* *y* '

• Or, in matrix form:

###

###

###

###

###

###

###

###

###

###

###

###

###

###

###

###

###

###

*y* *x* *b*

*a* *y*

*x*

### 0

### 0 '

### '

###

###

###

###

###

### *y* 0 *b* *y*

*scaling matrix S*
*scaling matrix S*
What’s inverse of S?

**2-D Rotation**

• This is easy to capture in matrix form:

### _{} _{} _{}

###

###

###

### ^{} _{} ^{} ^{} _{} ^{} ^{} _{} ^{}

###

###

###

###

###

###

###

*y* *x* *y*

*x*

###

###

###

###

### cos sin

### sin cos

### '

### ' _{} _{} _{} _{} _{}

### *y* sin cos *y*

**R**

• Even though sin() and cos() are nonlinear to ,

**’****i ****li****bi ti f d **

**R**

**– x’ is a linear combination of x and y****– y’ is a linear combination of x and y**

Wh i h i f i ?

• What is the inverse transformation?

– Rotation by –

*T*

– For rotation matrices, det(R) = 1 so **R**^{1}1 **R**^{T}

**2x2 Matrices**

• What types of transformations can be represented with a 2x2 matrix?

represented with a 2x2 matrix?

2D Identity?

### ' *x* ' 1 0 *x*

*y* *y*

*x* *x*

### '

### '

###

###

###

###

###

###

*y* *x* *y*

*x*

### 1 0

### 0 1

### ' '

2D Scale around (0,0)?

**x** **s**

**x**

**s**

**x** ' * ' **s** 0 **y**

**x**

**s**

**y**

**s** **y**

**s**

**y**

**x** **s**

**x**

**s**

**x**

**x**

**y**

**x**

### * '

###

###

###

###

###

###

###

###

###

###

###

###

###

**y** **x** **s**

**y**

**x**

**s**

**s** **y**

**s**

**y**

**x**

**x**

**y****x**

### 0

### 0 '

### '

**y**

###

^{y}###

**2x2 Matrices**

• What types of transformations can be represented with a 2x2 matrix?

represented with a 2x2 matrix?

2D Rotate around (0,0)?

* i

*

' _{} _{} _{} _{} _{i} _{} _{}_{} _{}

*y*
*x*

*y*

*y*
*x*

*x*

* cos

* sin

'

* sin

* cos

'

*y*
*x*
*y*

*x*

cos sin

sin cos

' '

2D Shear?

**y****x**

**sh****y**

**y****sh**

**x****x**

**y**

**x**

* '

*

'

###

###

###

###

###

###

###

###

###

###

###

###

###

**y** **x** **sh**

**y**

**x**

**sh**

**sh** **y**

**sh**

**y**

**x**

**x**

_{x}### 1 1

### ' '

**y****x**

**sh**

**y**_{y}

### **y** **sh**

**y**

**sh**

_{y}### 1 **y**

**y**

**2x2 Matrices**

• What types of transformations can be represented with a 2x2 matrix?

represented with a 2x2 matrix?

2D Mirror about Y axis?

'

### *x* ' 1 0 *x*

*y*
*y*

*x*
*x*

'

'

###

###

###

###

###

###

###

*y* *x* *y*

*x*

### 1 0

### 0 1

### ' '

2D Mirror over (0,0)?

*y*
*y*

*x*
*x*

'

'

###

###

###

###

###

###

###

###

*y* *x* *y*

*x*

### 1 0

### 0 1

### ' '

*y*

*y*

### *y* *y*

**All 2D Linear Transformations**

• Linear transformations are combinations of …

Scale – Scale,

– Rotation, – Shear andShear, and – Mirror

• Properties of linear transformations:

• Properties of linear transformations:

– Origin maps to origin – Lines map to linesLines map to lines

– Parallel lines remain parallel – Ratios are preserved

###

###

###

###

###

### *x'* *a* *b* *x*

p

– Closed under composition

###

###

###

###

###

###

###

###

###

###

###

###

*y* *x* *d*

*c*

*b* *a*

*y* *x* ' '

###

###

###

###

###

### *y* *c* *d* *y*

**2x2 Matrices**

• What types of transformations can not be represented with a 2x2 matrix?

represented with a 2x2 matrix?

2D Translation?

* t*
'

**y****x**

**t****y****y**

**t****x****x**

'

' NO!

Only linear 2D transformations

b t d ith 2 2 t i

can be represented with a 2x2 matrix

**Translation**

• Example of translation

Homogeneous Coordinates

1 1

1 0 0

1 0

0 1

1 ' '

*y*
*x*
*y*

*x*

*t*
*y*

*t*
*x*
*y*

*x*
*t*

*t*
*y*

*x*

1 0 0 1 1 1

t_{x} = 2
t_{y}_{y} = 1

**Affine Transformations**

• Affine transformations are combinations of …

Li f i d

– Linear transformations, and – Translations

• Properties of affine transformations:

– Origin does not necessarily map to origin – Lines map to lines

– Parallel lines remain parallel – Ratios are preserved

– Closed under composition

###

###

###

###

###

### *x*

*f* *d*

*c* *b*

*a* *x*

### ' '

– Models change of basis

###

###

###

###

###

###

###

###

###

###

###

###

### *w*

*y* *f*

*e* *d*

*w* *y*

### 1 0

### 0 '

###

###

###

###

###

###

**Projective Transformations**

• Projective transformations …

Affine transformations and – Affine transformations, and – Projective warps

P ti f j ti t f ti

• Properties of projective transformations:

– Origin does not necessarily map to origin l

– Lines map to lines

– Parallel lines do not necessarily remain parallel – Ratios are not preserved

– Closed under composition

###

###

###

###

###

### *x'* *a* *b* *c* *x*

– Models change of basis

###

###

###

###

###

###

###

###

###

###

###

###

### *w*

*y* *i*

*h* *g*

*f* *e*

*d* *w*

*y* ' '

###

###

###

###

###

### *w* *g* *h* *i* *w*

**Image warping**

**• Given a coordinate transform x’ = T(x) and a ****source image I(x) how do we compute a **

**source image I(x), how do we compute a ****transformed image I’(x’) = I(T(x))?**

**T(x)**

**I(x)****I’(x’)**

**x****x’**

**(x)**

**I(x)****I (x )**

**Forward warping**

**• Send each pixel I(x) to its corresponding ****location x’ T(x) in I’(x’)**

**location x’ = T(x) in I’(x’)**

**T(x)**

**I(x)****I’(x’)**

**x****x’**

**(x)**

**I(x)****I (x )**

**Forward warping**

**fwarp(I, I’, T)**
**{**

**{**

**for (y=0; y<I.height; y++)**

**for (x=0; x<I width; x++) {**
**for (x=0; x<I.width; x++) {**

**(x’,y’)=T(x,y);**

**I’(x’ y’) I(x y);**

**I’(x’,y’)=I(x,y);**

**}**
**}**

**}** *I* *I’*

**T**
**x**

**x’**

**Forward warping**

**• Send each pixel I(x) to its corresponding ****location x’ T(x) in I’(x’)**

**location x’ = T(x) in I’(x’)**

• What if pixel lands “between” two pixels?p p

• Will be there holes?

• Answer: add “contribution” to several pixels

• Answer: add contribution to several pixels,
*normalize later (splatting)*

**h(x)**

**f(x)****g(x’)**

**x****x’**

**(x)**

**f(x)****g(x )**

**Forward warping**

**fwarp(I, I’, T)**
**{**

**{**

**for (y=0; y<I.height; y++)**

**for (x=0; x<I width; x++) {**
**for (x=0; x<I.width; x++) {**

**(x’,y’)=T(x,y);**

**Splatting(I’ x’ y’ I(x y) kernel);**

**Splatting(I’,x’,y’,I(x,y),kernel);**

**}**
**}**

**}** *I* *I’*

**T**
**x**

**x’**

**Inverse warping**

**• Get each pixel I’(x’) from its corresponding ****location x T**^{-1}**(x’) in I(x)**

**location x = T**^{1}**(x’) in I(x)**

**T**^{-1}**(x’)**

**I(x)****I’(x’)**

**x****x’**

**(x )**

**I(x)****I (x )**

**Inverse warping**

**iwarp(I, I’, T)**
**{**

**{**

**for (y=0; y<I’.height; y++)**

**for (x=0; x<I’ width; x++) {**
**for (x=0; x<I’.width; x++) {**

**(x,y)=T**^{-1}**(x’,y’);**

**I’(x’ y’) I(x y);**

**I’(x’,y’)=I(x,y);**

**}**
**}**

**}** *I* **T**^{-1}*I’*

**x**

**x’**

**Inverse warping**

**• Get each pixel I’(x’) from its corresponding ****location x T**^{-1}**(x’) in I(x)**

**location x = T**^{1}**(x’) in I(x)**

• What if pixel comes from “between” two pixels?

*• Answer: resample color value from*

*• Answer: resample color value from *

*interpolated (prefiltered) source image*

**f(x)****g(x’)**

**x****x’**

**f(x)****g(x )**

**Inverse warping**

**iwarp(I, I’, T)**
**{**

**{**

**for (y=0; y<I’.height; y++)**

**for (x=0; x<I’ width; x++) {**
**for (x=0; x<I’.width; x++) {**

**(x,y)=T**^{-1}**(x’,y’);**

**I’(x’ y’) Reconstruct(I x y kernel);**

**I’(x’,y’)=Reconstruct(I,x,y,kernel);**

**}**
**}**

**}** *I* **T**^{-1}*I’*

**x**

**x’**

**Sampling**

*band limited*

**Reconstruction**

*The reconstructed function is obtained by interpolating *
*The reconstructed function is obtained by interpolating *
*among the samples in some manner*

**Reconstruction**

• Reconstruction generates an approximation to the original function Error is called aliasing

the original function. Error is called aliasing.

*l l*

sampling reconstruction

*sample value*

*l i i*
*sample position*

**Reconstruction**

• Computed weighted sum of pixel neighborhood;

output is weighted average of input where output is weighted average of input, where

weights are normalized values of filter kernel k

###

^{k}^{(}

^{q}^{)}

^{q}**color=0;**

###

*i* *i*

*i* *i* *i*

*q*
*k*

*q*
*q*
*p* *k*

) (

) (

**color=0;**

**weights=0;**

**for all q’s dist < width**
*width*

*d*

**q**

**d = dist(p, q);**

**w = kernel(d);**

*p*

*d* **color += w*q.color;**

**weights += w;**

**p Color = color/weights;**

*q*

**p.Color = color/weights;**

**Reconstruction (interpolation)**

• Possible reconstruction filters (kernels):

t i hb – nearest neighbor – bilinear

bi bi – bicubic

– sinc (optimal reconstruction)

**Bilinear interpolation (triangle filter)**

• A simple method for resampling images

**Non-parametric image warping**

• Specify a more detailed warp function

S li h i l fl ( i l i )

• Splines, meshes, optical flow (per-pixel motion)

**Non-parametric image warping**

• Mappings implied by correspondences

I i

• Inverse warping

**?** **P’**

**Non-parametric image warping**

' '

'

' *w* *A* *w* *B* *w* *C*
*P* * _{A}*

**

_{B}

_{C}*C*

*w*
*B*

*w*
*A*

*w*

*P*

*Barycentric coordinate*

*C*
*w*
*B*

*w*
*A*

*w*

*P* * _{A}*

**

_{B}

_{C}**P****P’**

**Barycentric coordinates**

3 1

2 1

3 3 2

2 1

1

*t*
*t*

*t*

*A*
*t*
*A*

*t*
*A*

*t*
*P*

3 1

2

1 *t* *t*
*t*

**Non-parametric image warping**

' '

'

' *w* *A* *w* *B* *w* *C*
*P* * _{A}*

**

_{B}

_{C}*C*

*w*
*B*

*w*
*A*

*w*

*P*

*Barycentric coordinate*

*C*
*w*
*B*

*w*
*A*

*w*

*P* * _{A}*

**

_{B}

_{C}**Non-parametric image warping**

) 2

(*r* *e* ^{}^{r}

###

^{}

Gaussian ^{}^{P}^{} _{K}^{1}

##

^{k}

^{X}

^{i}^{(}

^{P}^{'}

^{)}

^{}

^{X}

^{i}*radial basis function*

) log(

)

(*r* *r*^{2} *r*

thin plate

###

spline

*K* *i*

**Demo**

• http://www.colonize.com/warp/warp04-2.php W i i f l i f i id

• Warping is a useful operation for mosaics, video matching, view interpolation and so on.

**Image morphing**

**Image morphing**

**Image morphing**

• The goal is to synthesize a fluid transformation from one image to another

from one image to another.

• Cross dissolving is a common transition between cuts, but it is not good for morphing because of the ghosting effects.

i g #1 di l i g image #2

image #1 dissolving image #2

**Artifacts of cross-dissolving**

http://www.salavon.com/

**Image morphing**

• Why ghosting?

M hi i di l i

• Morphing = warping + cross-dissolving

shape

(geometric)

color

(photometric) (geometric) (photometric)

**Image morphing**

cross-dissolving

image #1 image #2

morphing

warp warp

**Morphing sequence**

**Face averaging by morphing**

average faces

**Image morphing**

create a morphing sequence: for each time t

1 C t i t di t i fi ld (b 1. Create an intermediate warping field (by

interpolation)

2 Warp both images towards it 2. Warp both images towards it

3. Cross-dissolve the colors in the newly warped images

images

*t=0* *t=0.33* *t=1*

**An ideal example (in 2004)**

t=0 morphingt=0.25t=0.75t=0.5 t=1

**An ideal example**

middle face (t=0.5)

t=0 t=1

**Warp specification (mesh warping)**

• How can we specify the warp?

1 S if di *li t l i t*

*1. Specify corresponding spline control points*

*interpolate to a complete warping function*

easy to implement but less expressive easy to implement, but less expressive

**Warp specification**

• How can we specify the warp

2 S if di *i t*
*2. Specify corresponding points*

*• interpolate to a complete warping function*

**Solution: convert to mesh warping**

1 D fi t i l h th i t 1. Define a triangular mesh over the points

– Same mesh in both images!

N h i l i l d

– Now we have triangle-to-triangle correspondences

2. Warp each triangle separately from source to destination – How do we warp a triangle?

– 3 points = affine warp!

– Just like texture mapping

**Warp specification (field warping)**

• How can we specify the warp?

3 S if di *t*

*3. Specify corresponding vectors*

*• interpolate to a complete warping function*

• The Beier & Neely Algorithm

• The Beier & Neely Algorithm

**Beier&Neely (SIGGRAPH 1992)**

• Single line-pair PQ to P’Q’:

**Algorithm (single line-pair)**

• For each X in the destination image:

1 Fi d th di

1. Find the corresponding u,v

2. Find X’ in the source image for that u,v

3 d ti ti I (X) I (X’)

3. destinationImage(X) = sourceImage(X’)

• Examples:

Affine transformation Affine transformation

**Multiple Lines**

*X*
*X*

*D*_{i}*X*_{i}^{'} *X*_{i}*D*

*length* = length of the line segment,
*dist* = distance to line segment

*The influence of a, p, b. The same as the average of X*_{i}’

**Full Algorithm**

**Resulting warp**

**Comparison to mesh morphing**

• Pros: more expressive

C d d l

• Cons: speed and control

**Warp interpolation**

• How do we create an intermediate warp at time t?

time t?

– linear interpolation for line end-points

B t li t ti 180 d ill b 0 – But, a line rotating 180 degrees will become 0

length in the middle

One solution is to interpolate line mid point and – One solution is to interpolate line mid-point and

orientation angle

t=0

t=1 t=1

**Animation**

**Animated sequences**

• Specify keyframes and interpolate the lines for the inbetween frames

the inbetween frames

• Require a lot of tweaking

**Results**

*Michael Jackson’s MTV “Black or White”*

**Multi-source morphing**

**Multi-source morphing**

**References**

• Thaddeus Beier, Shawn Neely, Feature-Based Image Metamorphosis, SIGGRAPH 1992, pp35-42.

• Detlef Ruprecht, Heinrich Muller, Image Warping with Scattered Data Interpolation, IEEE Computer Graphics and Applications, March 1995 pp37-43

March 1995, pp37 43.

• Seung-Yong Lee, Kyung-Yong Chwa, Sung Yong Shin, Image Metamorphosis Using Snakes and Free-Form Deformations, SIGGRAPH 1995

SIGGRAPH 1995.

• Seungyong Lee, Wolberg, G., Sung Yong Shin, Polymorph: morphing among multiple images, IEEE Computer Graphics and Applications, g p g , p p pp , Vol. 18, No. 1, 1998, pp58-71.

• Peinsheng Gao, Thomas Sederberg, A work minimization approach to image morphing The Visual Computer 1998 pp390 400

to image morphing, The Visual Computer, 1998, pp390-400.

• George Wolberg, Image morphing: a survey, The Visual Computer, 1998, pp360-372.