Image warping/morphing
Digital Visual Effects Yung-Yu Chuang
with slides by Richard Szeliski, Steve Seitz, Tom Funkhouser and Alexei Efros
Image warping
Image formation
A
B
Sampling and quantization
What is an image
• We can think of an image as a function, f: R2R:
– f(x, y) gives the intensity at position (x, y) – defined over a rectangle, with a finite range:
• f: [a,b]x[c,d] [0,1]
• A color image ( , )
( , ) ( , )
( , ) r x y
f x y g x y
b x y
x y
f
A digital image
• We usually operate on digital (discrete) images:
– Sample the 2D space on a regular grid
– Quantize each sample (round to nearest integer)
• If our samples are D apart, we can write this as:
f[i ,j] = Quantize{ f(i D, j D) }
• The image can now be represented as a matrix of integer values
Image warping
image filtering: change range of image g(x) = h(f(x))
f
x
h
g
x
f
x
h
g
x
image warping: change domain of image g(x) = f(h(x))
h(y)=0.5y+0.5
h(y)=2y
Image warping
h
h
f
f
g g
image filtering: change range of image g(x) = h(f(x))
image warping: change domain of image g(x) = f(h(x))
h(y)=0.5y+0.5
h([x,y])=[x,y/2]
Parametric (global) warping
translation rotation aspect
affine perspective
cylindrical
Examples of parametric warps:
Parametric (global) warping
• Transformation T is a coordinate-changing machine: p’ = T(p)
• What does it mean that T is global?
– Is the same for any point p
– can be described by just a few numbers (parameters)
• Represent T as a matrix: p’ = M*p
T
p = (x,y) p’ = (x’,y’)
y x y
x M
'
'
Scaling
• Scaling a coordinate means multiplying each of its components by a scalar
• Uniform scaling means this scalar is the same for all components:
2
f g
y x y
x
2 2 '
'
y
x
• Non-uniform scaling: different scalars per component:
Scaling
x 2, y 0.5
y x y
x
5 . 0
2 '
'
' ' y g x
y
f x
Scaling
• Scaling operation:
• Or, in matrix form:
by y
ax x
' '
y x b
a y
x
0
0 '
'
scaling matrix S What’s inverse of S?
2-D Rotation
• This is easy to capture in matrix form:
• Even though sin() and cos() are nonlinear to ,
– x’ is a linear combination of x and y – y’ is a linear combination of x and y
• What is the inverse transformation?
– Rotation by –
– For rotation matrices, det(R) = 1 so
y x y
x
cos sin
sin cos
' '
RT
R1
R
2x2 Matrices
• What types of transformations can be represented with a 2x2 matrix?
2D Identity?
y y x x
' '
x y
x y
1
0 0
' 1 '
2D Scale around (0,0)?
y s
y
x s
x
y
x
* '
* '
y x s
s y
x
y x
0
0 '
'
2x2 Matrices
• What types of transformations can be represented with a 2x2 matrix?
2D Rotate around (0,0)?
y x
y
y x
x
* cos
* sin
'
* sin
* cos
'
y x y
x
cos sin
sin cos
' '
2D Shear?
y x
sh y
y sh
x x
y
x
* '
*
'
y x sh
sh y
x
y
x
1 1
'
'
2x2 Matrices
• What types of transformations can be represented with a 2x2 matrix?
2D Mirror about Y axis?
y
y x
x ''
x y
x y
1 0 1 0 ' '
2D Mirror over (0,0)?
y
y x
x ''
x y
x y
1
0 1 0
' '
All 2D Linear Transformations
• Linear transformations are combinations of …
– Scale,
– Rotation, – Shear, and – Mirror
• Properties of linear transformations:
– Origin maps to origin – Lines map to lines
– Parallel lines remain parallel – Ratios are preserved
– Closed under composition
y x d
c
b a
y
x
'
'
2x2 Matrices
• What types of transformations can not be represented with a 2x2 matrix?
2D Translation?
y x
t y y
t x x
' '
Only linear 2D transformations
can be represented with a 2x2 matrix
NO!
Translation
• Example of translation
1 1
1 0 0
1 0
0 1
1 ' '
y x y
x
t y
t x y
x t
t y
x
tx = 2 ty = 1
Homogeneous Coordinates
Affine Transformations
• Affine transformations are combinations of …
– Linear transformations, and – Translations
• Properties of affine transformations:
– Origin does not necessarily map to origin – Lines map to lines
– Parallel lines remain parallel – Ratios are preserved
– Closed under composition – Models change of basis
w x y f
e
d a b c w
x y
1 0
0
' '
Projective Transformations
• Projective transformations …
– Affine transformations, and – Projective warps
• Properties of projective transformations:
– Origin does not necessarily map to origin – Lines map to lines
– Parallel lines do not necessarily remain parallel – Ratios are not preserved
– Closed under composition – Models change of basis
w x y i
h g
f e
d a b c w
x y
'
' '
Image warping
• Given a coordinate transform x’ = T(x) and a source image I(x), how do we compute a
transformed image I’(x’) = I(T(x))?
I(x) I’(x’
)
x x’
T(x)
Forward warping
• Send each pixel I(x) to its corresponding location x’ = T(x) in I’(x’)
I(x) I’(x’
)
x x’
T(x)
Forward warping
fwarp(I, I’, T) {
for (y=0; y<I.height; y++)
for (x=0; x<I.width; x++) { (x’,y’)=T(x,y);
I’(x’,y’)=I(x,y);
}
} I I’
x
x’
T
Forward warping
• Send each pixel I(x) to its corresponding location x’ = T(x) in I’(x’)
f(x) g(x’)
x x’
h(x)
• What if pixel lands “between” two pixels?
• Will be there holes?
• Answer: add “contribution” to several pixels, normalize later (splatting)
Forward warping
fwarp(I, I’, T) {
for (y=0; y<I.height; y++)
for (x=0; x<I.width; x++) { (x’,y’)=T(x,y);
Splatting(I’,x’,y’,I(x,y),kernel);
}
} I I’
x
x’
T
Inverse warping
• Get each pixel I’(x’) from its corresponding location x = T-1(x’) in I(x)
I(x) I’(x’
)
x x’
T-1(x’)
Inverse warping
iwarp(I, I’, T) {
for (y=0; y<I’.height; y++)
for (x=0; x<I’.width; x++) { (x,y)=T-1(x’,y’);
I’(x’,y’)=I(x,y);
}
} I I’
x
x’
T-1
Inverse warping
• Get each pixel I’(x’) from its corresponding location x = T-1(x’) in I(x)
• What if pixel comes from “between” two pixels?
• Answer: resample color value from
interpolated (prefiltered) source image
f(x) g(x’)
x x’
Inverse warping
iwarp(I, I’, T) {
for (y=0; y<I’.height; y++)
for (x=0; x<I’.width; x++) { (x,y)=T-1(x’,y’);
I’(x’,y’)=Reconstruct(I,x,y,kernel);
}
} I I’
x
x’
T-1
Sampling
band limited
Reconstruction
The reconstructed function is obtained by interpolating among the samples in some manner
Reconstruction
• Reconstruction generates an approximation to the original function. Error is called aliasing.
sample position
sample value
sampling reconstruction
Reconstruction
• Computed weighted sum of pixel neighborhood;
output is weighted average of input, where
weights are normalized values of filter kernel k
width d
color=0;
weights=0;
for all q’s dist < width d = dist(p, q);
w = kernel(d);
color += w*q.color;
weights += w;
p.Color = color/weights;
p
q
i i
i i i
q k
q q p k
) (
) (
Reconstruction (interpolation)
• Possible reconstruction filters (kernels):
– nearest neighbor – bilinear
– bicubic
– sinc (optimal reconstruction)
Bilinear interpolation (triangle filter)
• A simple method for resampling images
Non-parametric image warping
• Specify a more detailed warp function
• Splines, meshes, optical flow (per-pixel motion)
Non-parametric image warping
• Mappings implied by correspondences
• Inverse warping
? P’
Non-parametric image warping
P’
' '
'
' w A w B w C P A B C
Barycentric coordinate
C w B
w A
w
P A B C
P
Barycentric coordinates
3
1
2 1
3 3 2
2 1
1
t t
t
A t A
t A
t
P
Non-parametric image warping
' '
'
' w A w B w C P A B C
Barycentric coordinate
C w B
w A
w
P A B C
Non-parametric image warping
radial basis function
) 2
(r e r
) log(
)
(r r2 r
Gaussian thin plate spline
i
i
X P X
K k
P 1 i ( ')
Demo
• http://www.colonize.com/warp/warp04-2.php
• Warping is a useful operation for mosaics, video matching, view interpolation and so on.
Image morphing
Image morphing
• The goal is to synthesize a fluid transformation from one image to another.
image #1 dissolving image #2
• Cross dissolving is a common transition between cuts, but it is not good for morphing because of the ghosting effects.
Artifacts of cross-dissolving
http://www.salavon.com/
Image morphing
• Why ghosting?
• Morphing = warping + cross-dissolving
shape
(geometric)
color
(photometric)
morphing
cross-dissolving
Image morphing
image #1 image #2
warp warp
Morphing sequence
Face averaging by morphing
average faces
Image morphing
create a morphing sequence: for each time t
1. Create an intermediate warping field (by interpolation)
2. Warp both images towards it
3. Cross-dissolve the colors in the newly warped images
t=0 t=0.33 t=1
An ideal example (in 2004)
t=0 morphingt=0.25t=0.75t=0.5 t=1
An ideal example
middle face (t=0.5)
t=0 t=1
Warp specification (mesh warping)
• How can we specify the warp?
1. Specify corresponding spline control points
interpolate to a complete warping function
easy to implement, but less expressive
Warp specification
• How can we specify the warp
2. Specify corresponding points
• interpolate to a complete warping function
Solution: convert to mesh warping
1. Define a triangular mesh over the points – Same mesh in both images!
– Now we have triangle-to-triangle correspondences
2. Warp each triangle separately from source to destination – How do we warp a triangle?
– 3 points = affine warp!
– Just like texture mapping
Warp specification (field warping)
• How can we specify the warp?
3. Specify corresponding vectors
• interpolate to a complete warping function
• The Beier & Neely Algorithm
Beier&Neely (SIGGRAPH 1992)
• Single line-pair PQ to P’Q’:
Algorithm (single line-pair)
• For each X in the destination image:
1. Find the corresponding u,v
2. Find X’ in the source image for that u,v 3. destinationImage(X) = sourceImage(X’)
• Examples:
Affine transformation
Multiple Lines
length = length of the line segment, dist = distance to line segment
The influence of a, p, b. The same as the average of Xi’
i i
i X X
D '
Full Algorithm
Resulting warp
Comparison to mesh morphing
• Pros: more expressive
• Cons: speed and control
Warp interpolation
• How do we create an intermediate warp at time t?
– linear interpolation for line end-points
– But, a line rotating 180 degrees will become 0 length in the middle
– One solution is to interpolate line mid-point and orientation angle
t=0
t=1
Animation
Animated sequences
• Specify keyframes and interpolate the lines for the inbetween frames
• Require a lot of tweaking
Results
Michael Jackson’s MTV “Black or White”
Multi-source morphing
Multi-source morphing
References
• Thaddeus Beier, Shawn Neely, Feature-Based Image Metamorphosis, SIGGRAPH 1992, pp35-42.
• Detlef Ruprecht, Heinrich Muller, Image Warping with Scattered Data Interpolation, IEEE Computer Graphics and Applications, March 1995, pp37-43.
• Seung-Yong Lee, Kyung-Yong Chwa, Sung Yong Shin, Image Metamorphosis Using Snakes and Free-Form Deformations, SIGGRAPH 1995.
• Seungyong Lee, Wolberg, G., Sung Yong Shin, Polymorph: morphing among multiple images, IEEE Computer Graphics and Applications, Vol. 18, No. 1, 1998, pp58-71.
• Peinsheng Gao, Thomas Sederberg, A work minimization approach to image morphing, The Visual Computer, 1998, pp390-400.
• George Wolberg, Image morphing: a survey, The Visual Computer, 1998, pp360-372.