3D Face models from single images • Image-based faces • Relighting for faces (8)3D acquisition for faces (9)Cyberware scanners face &amp

(1)

Making faces

Digital Visual Effects, Spring 2006 Yung-Yu Chuang

2005/6/14

with slides by Richard Szeliski, Steve Seitz and Alex Efros

(2)

Announcements

• We will send out your current grades by the end of this week by emails.

• TA evaluation

• Project #3 artifacts winners

(3)

Honorable mention

汪澤先胡婷婷裴振宇

翁憲政洪韶憶

(4)

Third place

楊宗碩林柏劭胡仲榮

(5)

Second place

林立峯姜任遠

(6)

First place

梁家愷鐘志遠

(7)

Outline

• 3D acquisition for faces

• Statistical methods (with application to face super-resolution)

• 3D Face models from single images

• Image-based faces

• Relighting for faces

(8)

3D acquisition for faces

(9)

Cyberware scanners

face & head scanner whole body scanner

(10)

Making facial expressions from photos

• Similar to Façade, use a generic face model and view-dependent texture mapping

• Procedure

1. Take multiple photographs of a person 2. Establish corresponding feature points

3. Recover 3D points and camera parameters 4. Deform the generic face model to fit points 5. Extract textures from photos

(11)

Reconstruct a 3D model

input photographs

generic 3D face model

pose estimation

more features

deformed model

(12)

Mesh deformation

generic model displacement deformed model

– Compute displacement of feature points – Apply scattered data interpolation

(13)

Texture extraction

• The color at each point is a weighted combination of the colors in the photos

• Texture can be:

– view-independent – view-dependent

• Considerations for weighting

– occlusion – smoothness

– positional certainty – view similarity

(14)

Texture extraction

(15)

Texture extraction

(16)

view-independent view-dependent

Texture extraction

(17)

Model reconstruction

Use images to adapt a generic face model.

(18)

Creating new expressions

• In addition to global blending we can use:

– Regional blending – Painterly interface

(19)

Creating new expressions

Applying a global blend

+ =

/2 /2

New expressions are created with 3D morphing:

(20)

Applying a region-based blend

x

+

x

=

Creating new expressions

(21)

Using a painterly interface

+ + +

=

Creating new expressions

(22)

Drunken smile

(23)

Animating between expressions

Morphing over time creates animation:

“neutral” “joy”

(24)

Video

(25)

Spacetime faces

(26)

video projectors color cameras

black & white cameras

Spacetime faces

(27)

time

(28)

time

Face surface

(29)

stereo

time

(30)

stereo active stereo

time

(31)

spacetime stereo

stereo active stereo

time

(32)

time

time=1

Spacetime Stereo

Face surface

surface motion

(33)

time

time=2

Spacetime Stereo

Face surface

surface motion

(34)

time

time=3

Spacetime Stereo

Face surface

surface motion

(35)

time

time=4

Spacetime Stereo

Face surface

surface motion

(36)

time

time=5

Spacetime Stereo

Face surface

surface motion

(37)

time

time

Spacetime Stereo

surface motion

Better

• spatial resolution

• temporal stableness

(38)

Spacetime stereo matching

(39)

Video

(40)

Fitting

(41)

FaceIK

(42)

Animation

(43)

3D face applications: The one

(44)

3D face applications: Gladiator

extra 3M

(45)

3D face applications: Spiderman 2

(46)

Statistical methods

(47)

Statistical methods

f(z)+ε

z y

parameters

observed signal )

| ( max

* P z y

z =

z

) (

) ( )

| max (

y P

z P z

y P

=

z

) ( )

| (

min L y z L z

z

+

=

Example:

super-resolution de-noising

de-blocking Inpainting

…

(48)

Statistical methods

) ( )

| (

min

* L y z L z

z =

z

+

2

)

2

(

σ

z f

data

y −

evidence

a-priori

knowledge

f(z)+ε

z y

parameters

observed

signal

(49)

Statistical methods

There are approximately 10²⁴⁰ possible 10×10 gray-level images. Even human being has not seen them all yet. There must be a strong

statistical bias.

Takeo Kanade

Approximately 8X10¹¹blocks per day per person.

(50)

Generic priors

“Smooth images are good images.”

∑

=

x

x V

z

L ( ) ρ ( ( ))

)

2

( d = d

Gaussian MRF

ρ

T d

T T

d d

>

≤

⎩ ⎨

⎧

−

= +

) (

) 2

(

₂

ρ

2

Huber MRF

(51)

Generic priors

(52)

Example-based priors

“Existing images are good images.”

six 200×200 Images ⇒ 2,000,000 pairs

(53)

Example-based priors

L(z)

(54)

Example-based priors

low-resolution high-resolution

(55)

Model-based priors

“Face images are good images when working on face images …”

Parametric

model Z=WX+μ L(X)

) ( )

| (

min

* L y z L z

z =

z

+

⎩ ⎨

⎧

+

=

+ +

=

μ

*

) (

)

| (

min

*

WX z

X L WX

y L

X

x

⇒

(56)

PCA

• Principal Components Analysis (PCA):

approximating a high-dimensional data set with a lower-dimensional subspace

Original axes Original axes

**

** **

**

***

*

**

****** **

**

Data points Data points

First principal component First principal component Second principal component

Second principal component

(57)

PCA on faces: “eigenfaces”

Average Average

faceface

First principal component First principal component

Other Other components components

For all except average, For all except average,

“gray“gray”” = 0,= 0,

“white“white”” > 0,> 0,

““blackblack”” < 0< 0

(58)

Model-based priors

“Face images are good images when working on face images …”

Parametric

model Z=WX+μ L(X)

) ( )

| (

min

* L y z L z

z =

z

+

⎩ ⎨

⎧

+

=

+ +

=

μ

*

) (

)

| (

min

*

WX z

X L WX

y L

X

x

⇒

(59)

Super-resolution

(a) (b) (c) (d) (e) (f)

(a) Input low 24×32 (b) Our results (c) Cubic B-Spline

(d) Freeman et al. (e) Baker et al. (f) Original high 96×128

(60)

Face models from single images

(61)

Morphable model of 3D faces

• Start with a catalogue of 200 aligned 3D Cyberware scans

• Build a model of average shape and texture, and principal variations using PCA

(62)

Morphable model

(63)

Morphable model of 3D faces

• Adding some variations

(64)

Reconstruction from single image

Rendering must be similar to

the input if we guess right

(65)

Reconstruction from single image

shape and texture priors are learnt from database ρis the set of parameters for shading including camera pose, lighting and so on

prior

(66)

Modifying a single image

(67)

Animating from a single image

(68)

Video

(69)

Morphable model for human body

(70)

Image-based faces

(lip sync.)

(71)

Video rewrite (analysis)

(72)

Video rewrite (synthesis)

(73)

Results

• Video database

– 8 minutes of Ellen – 2 minutes of JFK

• Only half usable

• Head rotation

training video Read my lips.

I never met Forest Gump.

(74)

Morphable speech model

(75)

Preprocessing

(76)

Prototypes (PCA+k-mean clustering)

We find I_i and C_i for each prototype image.

(77)

Morphable model

I ^αβ

synthesis analysis

(78)

Morphable model

synthesis analysis

(79)

Synthesis

(80)

Results

(81)

Results

(82)

Relighting faces

(83)

Light is additive

lamp #1 lamp #2

(84)

Light stage 1.0

(85)

Light stage 1.0

64x32 lighting directions

(86)

Input images

(87)

Reflectance function

occlusion flare

(88)

Relighting

(89)

Results

(90)

Changing viewpoints

(91)

Results

(92)

Spiderman 2

real synthetic

(93)

Light stage 3

(94)

Application: The Matrix Reloaded

(95)

Application: The Matrix Reloaded

(96)

References

• F. Pighin, J. Hecker, D. Lischinski, D. H. Salesin, and R. Szeliski.

Synthesizing realistic facial expressions from photographs.

SIGGRAPH 1998, pp75-84.

• Li Zhang, Noah Snavely, Brian Curless, Steven M. Seitz, Spacetime Faces: High Resolution Capture for Modeling and Animation,

SIGGRAPH 2004.

• Blanz, V. and Vetter, T., A Morphable Model for the Synthesis of 3D Faces, SIGGRAPH 1999, pp187-194.

• Paul Debevec, Tim Hawkins, Chris Tchou, Haarm-Pieter Duiker, Westley Sarokin, Mark Sagar, Acquiring the Reflectance Field of a Human Face, SIGGRAPH 2000.

• Christoph Bregler, Malcolm Slaney, Michele Covell, Video Rewrite:

Driving Visual Speeach with Audio, SIGGRAPH 1997.

• Tony Ezzat, Gadi Geiger, Tomaso Poggio, Trainable Videorealistic Speech Animation, SIGGRAPH 2002.