Image warping/morphing
Digital Visual Effectsg Yung-Yu Chuang
with slides by Richard Szeliski, Steve Seitz, Tom Funkhouser and Alexei Efros
Image warping
Image warping
Image formation
B
A
Sampling and quantization
What is an image
• We can think of an image as a function, f: R2R:
f( ) i th i t it t iti ( ) – f(x, y) gives the intensity at position (x, y) – defined over a rectangle, with a finite range:
f [ b] [ d] [0 1]
• f: [a,b]x[c,d] [0,1]
x f
y
• A color image ( , )
( , ) ( , )
r x y
f x y g x y
( , ) ( , )
( , )
f y g y
b x y
A digital image
• We usually operate on digital (discrete) images:
S l th 2D l id – Sample the 2D space on a regular grid
– Quantize each sample (round to nearest integer)
f l h
• If our samples are D apart, we can write this as:
f[i ,j] = Quantize{ f(i D, j D) }
• The image can now be represented as a matrix of integer values
o tege values
Image warping
image filtering: change range of image ( ) h(f( ))
g(x) = h(f(x))
f
h
g
h(y)=0.5y+0.5
x
h
x
image warping: change domain of image
f
g(x) = f(h(x))
h(y)=2y
f
h
g
x x
Image warping
image filtering: change range of image ( ) h(f( ))
f g
g(x) = h(f(x))
h(y)=0.5y+0.5 h
f g
image warping: change domain of image
f
g(x) = f(h(x))
h([x,y])=[x,y/2]
h
f
g
Parametric (global) warping
Examples of parametric warps:
Examples of parametric warps:
translation rotation aspect
affine perspective
cylindrical
Parametric (global) warping
T T
p = (x,y) p’ = (x’,y’)
• Transformation T is a coordinate-changing machine: p’ = T(p)
p (x,y) p (x ,y )
machine: p T(p)
• What does it mean that T is global?
Is the same for any point p – Is the same for any point p
– can be described by just a few numbers (parameters)
R t T t i ’ M*
• Represent T as a matrix: p’ = M*p
x x
' M
'
y' y
Scaling
• Scaling a coordinate means multiplying each of its components by a scalar
its components by a scalar
• Uniform scaling means this scalar is the same for all components:
for all components:
f g
x ' 2 x
x
y y ' 2
y
2
Scaling
• Non-uniform scaling: different scalars per
component:
'
component:
' ' y g x
y f x
x ' 2 x
y y
y y ' 0 . 5
x 2, y 0.5
Scaling
• Scaling operation:
x ' ax by y '
• Or, in matrix form:
y x b
a y
x
0
0 '
'
y 0 b y
scaling matrix S scaling matrix S What’s inverse of S?
2-D Rotation
• This is easy to capture in matrix form:
y x y
x
cos sin
sin cos
'
'
y sin cos y
R
• Even though sin() and cos() are nonlinear to ,
’ i li bi ti f d
R
– x’ is a linear combination of x and y – y’ is a linear combination of x and y
Wh i h i f i ?
• What is the inverse transformation?
– Rotation by –
T
– For rotation matrices, det(R) = 1 so R11 RT
2x2 Matrices
• What types of transformations can be represented with a 2x2 matrix?
represented with a 2x2 matrix?
2D Identity?
' x ' 1 0 x
y y
x x
'
'
y x y
x
1 0
0 1
' '
2D Scale around (0,0)?
x s
x ' * ' s 0 y
s y
x s
x
y
x
* '
y x s
s y
x
y x
0
0 '
'
y
y
2x2 Matrices
• What types of transformations can be represented with a 2x2 matrix?
represented with a 2x2 matrix?
2D Rotate around (0,0)?
* i
*
' i
y x
y
y x
x
* cos
* sin
'
* sin
* cos
'
y x y
x
cos sin
sin cos
' '
2D Shear?
y x
sh y
y sh
x x
y
x
* '
*
'
y x sh
sh y
x
x1 1
' '
y x
sh
y y
y sh
y1 y
2x2 Matrices
• What types of transformations can be represented with a 2x2 matrix?
represented with a 2x2 matrix?
2D Mirror about Y axis?
'
x ' 1 0 x
y y
x x
'
'
y x y
x
1 0
0 1
' '
2D Mirror over (0,0)?
y y
x x
'
'
y x y
x
1 0
0 1
' '
y
y
y y
All 2D Linear Transformations
• Linear transformations are combinations of …
Scale – Scale,
– Rotation, – Shear andShear, and – Mirror
• Properties of linear transformations:
• Properties of linear transformations:
– Origin maps to origin – Lines map to linesLines map to lines
– Parallel lines remain parallel – Ratios are preserved
x' a b x
p
– Closed under composition
y x d
c
b a
y x ' '
y c d y
2x2 Matrices
• What types of transformations can not be represented with a 2x2 matrix?
represented with a 2x2 matrix?
2D Translation?
t '
y x
t y y
t x x
'
' NO!
Only linear 2D transformations
b t d ith 2 2 t i
can be represented with a 2x2 matrix
Translation
• Example of translation
Homogeneous Coordinates
1 1
1 0 0
1 0
0 1
1 ' '
y x y
x
t y
t x y
x t
t y
x
1 0 0 1 1 1
tx = 2 tyy = 1
Affine Transformations
• Affine transformations are combinations of …
Li f i d
– Linear transformations, and – Translations
• Properties of affine transformations:
– Origin does not necessarily map to origin – Lines map to lines
– Parallel lines remain parallel – Ratios are preserved
– Closed under composition
x
f d
c b
a x
' '
– Models change of basis
w
y f
e d
w y
1 0
0 '
Projective Transformations
• Projective transformations …
Affine transformations and – Affine transformations, and – Projective warps
P ti f j ti t f ti
• Properties of projective transformations:
– Origin does not necessarily map to origin l
– Lines map to lines
– Parallel lines do not necessarily remain parallel – Ratios are not preserved
– Closed under composition
x' a b c x
– Models change of basis
w
y i
h g
f e
d w
y ' '
w g h i w
Image warping
• Given a coordinate transform x’ = T(x) and a source image I(x) how do we compute a
source image I(x), how do we compute a transformed image I’(x’) = I(T(x))?
T(x)
I(x) I’(x’)
x x’
(x)
I(x) I (x )
Forward warping
• Send each pixel I(x) to its corresponding location x’ T(x) in I’(x’)
location x’ = T(x) in I’(x’)
T(x)
I(x) I’(x’)
x x’
(x)
I(x) I (x )
Forward warping
fwarp(I, I’, T) {
{
for (y=0; y<I.height; y++)
for (x=0; x<I width; x++) { for (x=0; x<I.width; x++) {
(x’,y’)=T(x,y);
I’(x’ y’) I(x y);
I’(x’,y’)=I(x,y);
} }
} I I’
T x
x’
Forward warping
• Send each pixel I(x) to its corresponding location x’ T(x) in I’(x’)
location x’ = T(x) in I’(x’)
• What if pixel lands “between” two pixels?p p
• Will be there holes?
• Answer: add “contribution” to several pixels
• Answer: add contribution to several pixels, normalize later (splatting)
h(x)
f(x) g(x’)
x x’
(x)
f(x) g(x )
Forward warping
fwarp(I, I’, T) {
{
for (y=0; y<I.height; y++)
for (x=0; x<I width; x++) { for (x=0; x<I.width; x++) {
(x’,y’)=T(x,y);
Splatting(I’ x’ y’ I(x y) kernel);
Splatting(I’,x’,y’,I(x,y),kernel);
} }
} I I’
T x
x’
Inverse warping
• Get each pixel I’(x’) from its corresponding location x T-1(x’) in I(x)
location x = T 1(x’) in I(x)
T-1(x’)
I(x) I’(x’)
x x’
(x )
I(x) I (x )
Inverse warping
iwarp(I, I’, T) {
{
for (y=0; y<I’.height; y++)
for (x=0; x<I’ width; x++) { for (x=0; x<I’.width; x++) {
(x,y)=T-1(x’,y’);
I’(x’ y’) I(x y);
I’(x’,y’)=I(x,y);
} }
} I T-1 I’
x
x’
Inverse warping
• Get each pixel I’(x’) from its corresponding location x T-1(x’) in I(x)
location x = T 1(x’) in I(x)
• What if pixel comes from “between” two pixels?
• Answer: resample color value from
• Answer: resample color value from
interpolated (prefiltered) source image
f(x) g(x’)
x x’
f(x) g(x )
Inverse warping
iwarp(I, I’, T) {
{
for (y=0; y<I’.height; y++)
for (x=0; x<I’ width; x++) { for (x=0; x<I’.width; x++) {
(x,y)=T-1(x’,y’);
I’(x’ y’) Reconstruct(I x y kernel);
I’(x’,y’)=Reconstruct(I,x,y,kernel);
} }
} I T-1 I’
x
x’
Sampling
band limited
Reconstruction
The reconstructed function is obtained by interpolating The reconstructed function is obtained by interpolating among the samples in some manner
Reconstruction
• Reconstruction generates an approximation to the original function Error is called aliasing
the original function. Error is called aliasing.
l l
sampling reconstruction
sample value
l i i sample position
Reconstruction
• Computed weighted sum of pixel neighborhood;
output is weighted average of input where output is weighted average of input, where
weights are normalized values of filter kernel k
k(q )qcolor=0;
i i
i i i
q k
q q p k
) (
) (
color=0;
weights=0;
for all q’s dist < width width
d
q
d = dist(p, q);
w = kernel(d);
p
d color += w*q.color;
weights += w;
p Color = color/weights;
q
p.Color = color/weights;
Reconstruction (interpolation)
• Possible reconstruction filters (kernels):
t i hb – nearest neighbor – bilinear
bi bi – bicubic
– sinc (optimal reconstruction)
Bilinear interpolation (triangle filter)
• A simple method for resampling images
Non-parametric image warping
• Specify a more detailed warp function
S li h i l fl ( i l i )
• Splines, meshes, optical flow (per-pixel motion)
Non-parametric image warping
• Mappings implied by correspondences
I i
• Inverse warping
? P’
Non-parametric image warping
' '
'
' w A w B w C P A B C C
w B
w A
w
P
Barycentric coordinate
C w B
w A
w
P A B C
P P’
Barycentric coordinates
3 1
2 1
3 3 2
2 1
1
t t
t
A t A
t A
t P
3 1
2
1 t t t
Non-parametric image warping
' '
'
' w A w B w C P A B C C
w B
w A
w
P
Barycentric coordinate
C w B
w A
w
P A B C
Non-parametric image warping
) 2
(r e r
Gaussian P K1
kXi (P')Xiradial basis function
) log(
)
(r r2 r
thin plate
spline
K i
Demo
• http://www.colonize.com/warp/warp04-2.php W i i f l i f i id
• Warping is a useful operation for mosaics, video matching, view interpolation and so on.
Image morphing
Image morphing
Image morphing
• The goal is to synthesize a fluid transformation from one image to another
from one image to another.
• Cross dissolving is a common transition between cuts, but it is not good for morphing because of the ghosting effects.
i g #1 di l i g image #2
image #1 dissolving image #2
Artifacts of cross-dissolving
http://www.salavon.com/
Image morphing
• Why ghosting?
M hi i di l i
• Morphing = warping + cross-dissolving
shape
(geometric)
color
(photometric) (geometric) (photometric)
Image morphing
cross-dissolving
image #1 image #2
morphing
warp warp
Morphing sequence
Face averaging by morphing
average faces
Image morphing
create a morphing sequence: for each time t
1 C t i t di t i fi ld (b 1. Create an intermediate warping field (by
interpolation)
2 Warp both images towards it 2. Warp both images towards it
3. Cross-dissolve the colors in the newly warped images
images
t=0 t=0.33 t=1
An ideal example (in 2004)
t=0 morphingt=0.25t=0.75t=0.5 t=1
An ideal example
middle face (t=0.5)
t=0 t=1
Warp specification (mesh warping)
• How can we specify the warp?
1 S if di li t l i t
1. Specify corresponding spline control points
interpolate to a complete warping function
easy to implement but less expressive easy to implement, but less expressive
Warp specification
• How can we specify the warp
2 S if di i t 2. Specify corresponding points
• interpolate to a complete warping function
Solution: convert to mesh warping
1 D fi t i l h th i t 1. Define a triangular mesh over the points
– Same mesh in both images!
N h i l i l d
– Now we have triangle-to-triangle correspondences
2. Warp each triangle separately from source to destination – How do we warp a triangle?
– 3 points = affine warp!
– Just like texture mapping
Warp specification (field warping)
• How can we specify the warp?
3 S if di t
3. Specify corresponding vectors
• interpolate to a complete warping function
• The Beier & Neely Algorithm
• The Beier & Neely Algorithm
Beier&Neely (SIGGRAPH 1992)
• Single line-pair PQ to P’Q’:
Algorithm (single line-pair)
• For each X in the destination image:
1 Fi d th di
1. Find the corresponding u,v
2. Find X’ in the source image for that u,v
3 d ti ti I (X) I (X’)
3. destinationImage(X) = sourceImage(X’)
• Examples:
Affine transformation Affine transformation
Multiple Lines
X X
Di Xi' Xi D
length = length of the line segment, dist = distance to line segment
The influence of a, p, b. The same as the average of Xi’
Full Algorithm
Resulting warp
Comparison to mesh morphing
• Pros: more expressive
C d d l
• Cons: speed and control
Warp interpolation
• How do we create an intermediate warp at time t?
time t?
– linear interpolation for line end-points
B t li t ti 180 d ill b 0 – But, a line rotating 180 degrees will become 0
length in the middle
One solution is to interpolate line mid point and – One solution is to interpolate line mid-point and
orientation angle
t=0
t=1 t=1
Animation
Animated sequences
• Specify keyframes and interpolate the lines for the inbetween frames
the inbetween frames
• Require a lot of tweaking
Results
Michael Jackson’s MTV “Black or White”
Multi-source morphing
Multi-source morphing
References
• Thaddeus Beier, Shawn Neely, Feature-Based Image Metamorphosis, SIGGRAPH 1992, pp35-42.
• Detlef Ruprecht, Heinrich Muller, Image Warping with Scattered Data Interpolation, IEEE Computer Graphics and Applications, March 1995 pp37-43
March 1995, pp37 43.
• Seung-Yong Lee, Kyung-Yong Chwa, Sung Yong Shin, Image Metamorphosis Using Snakes and Free-Form Deformations, SIGGRAPH 1995
SIGGRAPH 1995.
• Seungyong Lee, Wolberg, G., Sung Yong Shin, Polymorph: morphing among multiple images, IEEE Computer Graphics and Applications, g p g , p p pp , Vol. 18, No. 1, 1998, pp58-71.
• Peinsheng Gao, Thomas Sederberg, A work minimization approach to image morphing The Visual Computer 1998 pp390 400
to image morphing, The Visual Computer, 1998, pp390-400.
• George Wolberg, Image morphing: a survey, The Visual Computer, 1998, pp360-372.
