Image warping/morphing
Digital Visual Effects Yung-Yu Chuang
with slides by Richard Szeliski, Steve Seitz, Tom Funkhouser, Jia-Bing Huang and Alexei Efros
Image warping
Image formation
A
B
Sampling and quantization
What is an image
• We can think of an image as a function, f: R2R:
– f(x, y) gives the intensity at position (x, y) – defined over a rectangle, with a finite range:
• f: [a,b]x[c,d] [0,1]
• A color image ( , )
( , ) ( , )
( , ) r x y
f x y g x y
b x y
x y
f
A digital image
• We usually operate on digital (discrete) images:
– Sample the 2D space on a regular grid
– Quantize each sample (round to nearest integer)
• If our samples are D apart, we can write this as:
f[i ,j] = Quantize{ f(i D, j D) }
• The image can now be represented as a matrix of integer values
Image warping
image filtering: change range of image g(x) = h(f(x))
f
x
h
g
x
f
x
h
g
x
image warping: change domain of image g(x) = f(h(x))
h(y)=0.5y+0.5
h(y)=2y
Image warping
h
h
f
f
g g
image filtering: change range of image g(x) = h(f(x))
image warping: change domain of image g(x) = f(h(x))
h(y)=0.5y+0.5
h([x,y])=[x,y/2]
Parametric (global) warping
translation rotation aspect
affine perspective
cylindrical
Examples of parametric warps:
Parametric (global) warping
• Transformation T is a coordinate-changing machine: p’ = T(p)
• What does it mean that T is global?
– Is the same for any point p
– can be described by just a few numbers (parameters)
• Represent T as a matrix: p’ = M*p
T
p = (x,y) p’ = (x’,y’)
y x y
x M
' '
Scaling
• Scaling a coordinate means multiplying each of its components by a scalar
• Uniform scaling means this scalar is the same for all components:
2
f g
y x y
x
2 2 '
'
y x
• Non-uniform scaling: different scalars per component:
Scaling
x 2, y 0.5
y x y
x
5 . 0
2 '
'
' ' y g x y
f x
Scaling
• Scaling operation:
• Or, in matrix form:
by y
ax x
' '
y x b a y
x
0 0 '
'
scaling matrix S What’s inverse of S?
2-D Rotation
• This is easy to capture in matrix form:
• Even though sin() and cos() are nonlinear to ,
– x’ is a linear combination of x and y – y’ is a linear combination of x and y
• What is the inverse transformation?
– Rotation by –
– For rotation matrices, det(R) = 1 so
y x y
x
cos sin
sin cos
' '
RT
R1 R
2x2 Matrices
• What types of transformations can be represented with a 2x2 matrix?
2D Identity?
y y
x x
''
y x y
x 01 10 ''
2D Scale around (0,0)?
y s y
x s x
y
x
* '
* '
y x s s y
x
y x
0 0 '
'
2x2 Matrices
• What types of transformations can be represented with a 2x2 matrix?
2D Rotate around (0,0)?
y x
y
y x
x
* cos
* sin '
* sin
* cos '
y x y
x
cos sin
sin cos
' '
2D Shear?
y x sh y
y sh x x
y x
* '
*
'
y x sh
sh y
x
y x
1 1
' '
2x2 Matrices
• What types of transformations can be represented with a 2x2 matrix?
2D Mirror about Y axis?
y y
x x
''
y x y
x 01 10 ''
2D Mirror over (0,0)?
y y
x x
''
y x y
x 01 01 ''
All 2D Linear Transformations
• Linear transformations are combinations of …
– Scale, – Rotation, – Shear, and – Mirror
• Properties of linear transformations:
– Origin maps to origin – Lines map to lines
– Parallel lines remain parallel – Ratios are preserved
– Closed under composition
y x d c
b a y
x ' '
2x2 Matrices
• What types of transformations can not be represented with a 2x2 matrix?
2D Translation?
y x
t y y
t x x
' '
Only linear 2D transformations can be represented with a 2x2 matrix
NO!
Translation
• Example of translation
1 1 1 0 0
1 0
0 1
1 ' '
y x y
x
t y
t x y x t t y
x
tx= 2 ty= 1
Homogeneous Coordinates
Affine Transformations
• Affine transformations are combinations of …
– Linear transformations, and – Translations
• Properties of affine transformations:
– Origin does not necessarily map to origin – Lines map to lines
– Parallel lines remain parallel – Ratios are preserved
– Closed under composition – Models change of basis
w y x f e d
c b a w
y x
1 0 0 ' '
Projective Transformations
• Projective transformations …
– Affine transformations, and – Projective warps
• Properties of projective transformations:
– Origin does not necessarily map to origin – Lines map to lines
– Parallel lines do not necessarily remain parallel – Ratios are not preserved
– Closed under composition – Models change of basis
w y x i h g
f e d
c b a w
y x ' ' '
Image warping
• Given a coordinate transform x’ = T(x) and a source image I(x), how do we compute a transformed image I’(x’) = I(T(x))?
I(x) I’(x’)
x x’
T(x)
Forward warping
• Send each pixel I(x) to its corresponding location x’ = T(x) in I’(x’)
I(x) I’(x’)
x x’
T(x)
Forward warping
fwarp(I, I’, T) {
for (y=0; y<I.height; y++) for (x=0; x<I.width; x++) {
(x’,y’)=T(x,y);
I’(x’,y’)=I(x,y);
}
} I I’
x
x’
T
Forward warping
Some destination may not be covered
Many source pixels could map to the same destination
Forward warping
• Send each pixel I(x) to its corresponding location x’ = T(x) in I’(x’)
f(x) g(x’)
x x’
h(x)
• What if pixel lands “between” two pixels?
• Will be there holes?
• Answer: add “contribution” to several pixels, normalize later (splatting)
Forward warping
fwarp(I, I’, T) {
for (y=0; y<I.height; y++) for (x=0; x<I.width; x++) {
(x’,y’)=T(x,y);
Splatting(I’,x’,y’,I(x,y),kernel);
}
} I I’
x
x’
T
Inverse warping
• Get each pixel I’(x’) from its corresponding location x = T-1(x’) in I(x)
I(x) I’(x’)
x x’
T-1(x’)
Inverse warping
iwarp(I, I’, T) {
for (y'=0; y’<I’.height; y’++) for (x’=0; x’<I’.width; x’++) {
(x,y)=T-1(x’,y’);
I’(x’,y’)=I(x,y);
}
} I I’
x
x’
T-1
Inverse warping
• Get each pixel I’(x’) from its corresponding location x = T-1(x’) in I(x)
• What if pixel comes from “between” two pixels?
• Answer: resample color value from interpolated (prefiltered) source image
f(x) g(x’)
x x’
Inverse warping
iwarp(I, I’, T) {
for (y’=0; y’<I’.height; y’++) for (x’=0; x’<I’.width; x’++) {
(x,y)=T-1(x’,y’);
I’(x’,y’)=Reconstruct(I,x,y,kernel);
}
} I I’
x
x’
T-1
Inverse warping
• No hole, but must resample
• What value should you take for non-integer coordinate? Closest one?
Inverse warping
• It could cause aliasing
Reconstruction
• Reconstruction generates an approximation to the original function. Error is called aliasing.
sample position sample value
sampling reconstruction
Reconstruction
• Computed weighted sum of pixel neighborhood;
output is weighted average of input, where weights are normalized values of filter kernel k
width d
color=0;
weights=0;
for all q’s dist < width d = dist(p, q);
w = kernel(d);
color += w*q.color;
weights += w;
p.Color = color/weights;
p
q
i i
i i i
q k
q q p k
) (
) (
Triangle filter Gaussian filter
Sampling
band limited
Reconstruction
The reconstructed function is obtained by interpolating among the samples in some manner
Reconstruction (interpolation)
• Possible reconstruction filters (kernels):
– nearest neighbor – bilinear
– bicubic
– sinc (optimal reconstruction)
Bilinear interpolation (triangle filter)
• A simple method for resampling images
Non-parametric image warping
• Specify a more detailed warp function
• Splines, meshes, optical flow (per-pixel motion)
Non-parametric image warping
• Mappings implied by correspondences
• Inverse warping
? P’
Non-parametric image warping
P’
' '
'
' w A w B w C P A B C
Barycentric coordinate
C w B w A w
P A B C
P
Barycentric coordinates
3 1
2 1
3 3 2 2 1 1
t t t
A t A t A t P
Non-parametric image warping
' '
'
' w A w B w C P A B C
Barycentric coordinate
C w B w A w
P A B C
Non-parametric image warping
radial basis function
) 2
(r e r
) log(
)
(r r2 r
Gaussian thin plate spline
i
i
X P X
K k
P 1 i( ')
Image warping
• Warping is a useful operation for mosaics,
retargeting, video matching, view interpolation and so on.
An application of image warping:
face beautification
Data-driven facial beautification Facial beautification
Facial beautification Facial beautification
Training set
• Face images
– 92 young Caucasian female – 33 young Caucasian male
Feature extraction
Feature extraction
• Extract 84 feature points by BTSM
• Delaunay triangulation -> 234D distance vector (normalized by the square root of face area)
BTSM
scatter plot forall training faces
234D vector
Beautification engine
Support vector regression (SVR)
• Similar concept to SVM, but for regression
• RBF kernels
• fb(v)
Beautification process
• Given the normalized distance vector v, generate a nearby vector v’ so that
fb(v’) > fb(v)
• Two options
– KNN-based – SVR-based
KNN-based beautification
4.3
4.5
5.1
3.1
5.3 4.6
v v'
SVR-based beautification
• Directly use fb to seek v’
• Use standard no-derivative direction set method for minimization
• Features were reduced to 35D by PCA
SVR-based beautification
• Problems: it sometimes yields distance vectors corresponding to invalid human face
• Solution: add log-likelihood term (LP)
• LP is approximated by modeling face space as a multivariate Gaussian distribution
u’s projection in PCA space
’s i-th component
i-th eigenvalue
PCA
λ1
λ2
Embedding and warping Distance embedding
• Convert modified distance vector v’ to a new face landmark
• A graph drawing problem referred to as a stress minimization problem, solved by LM algorithm for non-linear minimization
1 if i and j belong to different facial features 10 otherwise
Distance embedding
• Post processing to enforce similarity transform for features on eyes by minimizing
SVR K=5
K=3 Original
Results (in training set) User study
Results (not in training set) By parts
full
mouth eyes
Different degrees
50% 100%
Facial collage
Results
• video
Image morphing
Image morphing
• The goal is to synthesize a fluid transformation from one image to another.
image #1 dissolving image #2
• Cross dissolving is a common transition between cuts, but it is not good for morphing because of the ghosting effects.
Artifacts of cross-dissolving
http://www.salavon.com/
Image morphing
• Why ghosting?
• Morphing = warping + cross-dissolving
shape (geometric)
color (photometric)
morphing cross-dissolving
Image morphing
image #1 image #2
warp warp
Morphing sequence Face averaging by morphing
average faces
Image morphing
create a morphing sequence: for each time t
1. Create an intermediate warping field (by interpolation)
2. Warp both images towards it
3. Cross-dissolve the colors in the newly warped images
t=0 t=0.33 t=1
An ideal example (in 2004)
t=0 t=0.25t=0.75t=0.5 t=1
morphing
An ideal example (in 2004)
t=0 t=1
An ideal example
middle face (t=0.5)
t=0 t=1
Warp specification (mesh warping)
• How can we specify the warp?
1. Specify corresponding spline control points
interpolate to a complete warping function
easy to implement, but less expressive
Warp specification
• How can we specify the warp
2. Specify corresponding points
• interpolate to a complete warping function
Solution: convert to mesh warping
1. Define a triangular mesh over the points – Same mesh in both images!
– Now we have triangle-to-triangle correspondences 2. Warp each triangle separately from source to destination
– How do we warp a triangle?
– 3 points = affine warp!
– Just like texture mapping
Warp specification (field warping)
• How can we specify the warp?
3. Specify corresponding vectors
• interpolate to a complete warping function
• The Beier & Neely Algorithm
Beier&Neely (SIGGRAPH 1992)
• Single line-pair PQ to P’Q’:
Algorithm (single line-pair)
• For each X in the destination image:
1. Find the corresponding u,v
2. Find X’ in the source image for that u,v 3. destinationImage(X) = sourceImage(X’)
• Examples:
Affine transformation
Multiple Lines
length = length of the line segment, dist= distance to line segment
The influence of a, p, b. The same as the average of Xi’
i i
i X X
D '
Full Algorithm
Resulting warp Comparison to mesh morphing
• Pros: more expressive
• Cons: speed and control
Warp interpolation
• How do we create an intermediate warp at time t?
– linear interpolation for line end-points
– But, a line rotating 180 degrees will become 0 length in the middle
– One solution is to interpolate line mid-point and orientation angle
t=0
t=1
Animation
Animated sequences
• Specify keyframes and interpolate the lines for the inbetween frames
• Require a lot of tweaking
Results
Michael Jackson’s MTV “Black or White”
Multi-source morphing Multi-source morphing
Miss Korea
Picasa recognizes them as the same personAlign and mean face Morphing sequence
• video
Woman in arts References
• Thaddeus Beier, Shawn Neely, Feature-Based Image Metamorphosis, SIGGRAPH 1992, pp35-42.
• Detlef Ruprecht, Heinrich Muller, Image Warping with Scattered Data Interpolation, IEEE Computer Graphics and Applications, March 1995, pp37-43.
• Seung-Yong Lee, Kyung-Yong Chwa, Sung Yong Shin, Image Metamorphosis Using Snakes and Free-Form Deformations, SIGGRAPH 1995.
• Seungyong Lee, Wolberg, G., Sung Yong Shin, Polymorph: morphing among multiple images, IEEE Computer Graphics and Applications, Vol. 18, No. 1, 1998, pp58-71.
• Peinsheng Gao, Thomas Sederberg, A work minimization approach to image morphing, The Visual Computer, 1998, pp390-400.
• George Wolberg, Image morphing: a survey, The Visual Computer, 1998, pp360-372.
• Data-Driven Enhancement of Facial Attractiveness, SIGGRAPH 2008
