• 沒有找到結果。

Image-based lighting

N/A
N/A
Protected

Academic year: 2022

Share "Image-based lighting"

Copied!
115
0
0

加載中.... (立即查看全文)

全文

(1)

Faces and Image-Based Lighting

Digital Visual Effects, Spring 2007 Yung-Yu Chuang

2007/6/12

with slides by Richard Szeliski, Steve Seitz, Alex Efros, Li-Yi Wei and Paul Debevec

(2)

Announcements

• TA evaluation

• Final project:

– Demo on 6/27 (Wednesday) 13:30pm in this room – Reports and videos due on 6/28 (Thursday) 11:59pm

(3)

Outline

• Image-based lighting

• 3D acquisition for faces

• Statistical methods (with application to face super-resolution)

• 3D Face models from single images

• Image-based faces

• Relighting for faces

(4)

Image-based lighting

(5)

Rendering

• Rendering is a function of geometry, reflectance, lighting and viewing.

• To synthesize CGI into real scene, we have to match the above four factors.

• Viewing can be obtained from calibration or structure from motion.

• Geometry can be captured using 3D photography or made by hands.

• How to capture lighting and reflectance?

(6)

Reflectance

The Bidirectional Reflection Distribution Function

Given an incoming ray and outgoing ray

what proportion of the incoming light is reflected along outgoing ray?

Answer given by the BRDF:

surface normal

(7)

Rendering equation

p

= ) ω p,

(

o

L

o

L

e

( p , ω

o

)

i i

i i

o

, ω ) ( p, ω ) cos θ ω

ω p,

2

( L

i

d

s

+ ρ

) ω p,

(

o

L

o

5D light field

ω

i

ω

o

) ω , p

(

i

L

i

(8)

Complex illumination

= ) ω p,

(

o

L

o

L

e

( p , ω

o

)

i i

i i

o

, ω ) ( p, ω ) cos θ ω

ω p,

2

f ( L

i

d

s

+

= ) ω p,

(

o

B

2

f ( p, ω

o

, ω

i

) L

d

( p, ω

i

) cos θ

i

d ω

i

s

reflectance lighting

(9)

Point lights

Classically, rendering is performed assuming point light sources

directional source

(10)

Environment maps

Miller and Hoffman, 1984

(11)

Capturing reflectance

(12)

Acquiring the Light Probe

(13)

HDRI Sky Probe

(14)

Clipped Sky + Sun Source

(15)

Lit by sun only

(16)

Lit by sky only

(17)

Lit by sun and sky

(18)

Illuminating a Small Scene

(19)
(20)

Real Scene Example

• Goal: place synthetic objects on table

(21)

Light Probe / Calibration Grid

(22)

real scene real scene

Modeling the Scene

light-based model light-based model

(23)

The Light-Based Room Model

(24)

Rendering into the Scene

• Background Plate

(25)

Rendering into the scene

• Objects and Local Scene matched to Scene

(26)

Differential rendering

• Local scene w/o objects, illuminated by model

(27)

Differential rendering

= =

- -

(28)

Differential Rendering

• Final Result

(29)

Environment map from single image?

(30)

Eye as light probe! (Nayar et al)

(31)

Cornea is an ellipsoid

(32)

Results

(33)

Application in “The Matrix Reloaded”

(34)

3D acquisition for faces

(35)

Cyberware scanners

face & head scanner whole body scanner

(36)

Making facial expressions from photos

• Similar to Façade, use a generic face model and view-dependent texture mapping

• Procedure

1. Take multiple photographs of a person 2. Establish corresponding feature points

3. Recover 3D points and camera parameters 4. Deform the generic face model to fit points 5. Extract textures from photos

(37)

Reconstruct a 3D model

input photographs

generic 3D face model

pose estimation

more features

deformed model

(38)

Mesh deformation

generic model displacement deformed model

– Compute displacement of feature points – Apply scattered data interpolation

(39)

Texture extraction

• The color at each point is a weighted combination of the colors in the photos

• Texture can be:

– view-independent – view-dependent

• Considerations for weighting

– occlusion – smoothness

– positional certainty – view similarity

(40)

Texture extraction

(41)

Texture extraction

(42)

view-independent view-dependent

Texture extraction

(43)

Model reconstruction

Use images to adapt a generic face model.

(44)

Creating new expressions

• In addition to global blending we can use:

– Regional blending – Painterly interface

(45)

Creating new expressions

Applying a global blend

+ =

/2 /2

New expressions are created with 3D morphing:

(46)

Applying a region-based blend

x

+

x

=

Creating new expressions

(47)

Using a painterly interface

+ + +

=

Creating new expressions

(48)

Drunken smile

(49)

Animating between expressions

Morphing over time creates animation:

“neutral” “joy”

(50)

Video

(51)

Spacetime faces

(52)

video projectors color cameras

black & white cameras

Spacetime faces

(53)

time

(54)

time

Face surface

(55)

stereo

time

(56)

stereo active stereo

time

(57)

spacetime stereo

stereo active stereo

time

(58)

time

time

Spacetime Stereo

surface motion

Better

• spatial resolution

• temporal stableness

(59)

Spacetime stereo matching

(60)

Video

(61)

Fitting

(62)

FaceIK

(63)

Animation

(64)

3D face applications: The one

(65)

3D face applications: Gladiator

extra 3M

(66)

Statistical methods

(67)

Statistical methods

f(z)+ε

z y

para- meters

observed signal )

| ( max

* P z y

z =

z

) (

) ( )

| max (

y P

z P z

y P

=

z

) ( )

| (

min L y z L z

z

+

=

Example:

super-resolution de-noising

de-blocking Inpainting

(68)

Statistical methods

) ( )

| (

min

* L y z L z

z =

z

+

2

)

2

(

σ

z f

data y −

evidence

a-priori knowledge

f(z)+ε

z y

para- meters

observed

signal

(69)

Statistical methods

There are approximately 10240 possible 10×10 gray-level images. Even human being has not seen them all yet. There must be a strong

statistical bias.

Takeo Kanade

Approximately 8X1011 blocks per day per person.

(70)

Generic priors

“Smooth images are good images.”

=

x

x V

z

L ( ) ρ ( ( ))

)

2

( d = d Gaussian MRF ρ

T d

T d

T d

T T

d d

>

⎩ ⎨

= +

) (

) 2

(

2

ρ

2

Huber MRF

(71)

Generic priors

(72)

Example-based priors

“Existing images are good images.”

six 200×200

Images ⇒

2,000,000

pairs

(73)

Example-based priors

L(z)

(74)

Example-based priors

low-resolution high-resolution

(75)

Model-based priors

“Face images are good images when working on face images …”

Parametric

model Z=WX+μ L(X)

) ( )

| (

min

* L y z L z

z =

z

+

⎩ ⎨

+

=

+ +

=

μ

μ

*

*

) (

)

| (

min

*

WX z

X L WX

y L

X

x

(76)

PCA

• Principal Components Analysis (PCA):

approximating a high-dimensional data set with a lower-dimensional subspace

Original axes Original axes

**

**

**

**

**

**

** **

**

**

***

*

**

**

**

**

**

****** **

**

**

**

Data points Data points

First principal component First principal component Second principal component

Second principal component

(77)

PCA on faces: “eigenfaces”

Average Average

faceface

First principal component First principal component

Other Other components components

For all except average, For all except average,

“graygray” = 0,= 0,

“whitewhite” > 0,> 0,

blackblack < 0< 0

(78)

Model-based priors

“Face images are good images when working on face images …”

Parametric

model Z=WX+μ L(X)

) ( )

| (

min

* L y z L z

z =

z

+

⎩ ⎨

+

=

+ +

=

μ

μ

*

*

) (

)

| (

min

*

WX z

X L WX

y L

X

x

(79)

Super-resolution

(a) (b) (c) (d) (e) (f)

(a) Input low 24×32 (b) Our results (c) Cubic B-Spline

(d) Freeman et al. (e) Baker et al. (f) Original high 96×128

(80)

Face models from single images

(81)

Morphable model of 3D faces

• Start with a catalogue of 200 aligned 3D Cyberware scans

Build a model of average shape and texture, and principal variations using PCA

(82)

Morphable model

shape examplars texture examplars

(83)

Morphable model of 3D faces

• Adding some variations

(84)

Reconstruction from single image

Rendering must be similar to

the input if we guess right

(85)

Reconstruction from single image

shape and texture priors are learnt from database ρis the set of parameters for shading including camera pose, lighting and so on

prior

(86)

Modifying a single image

(87)

Animating from a single image

(88)

Video

(89)

Morphable model for human body

(90)

Image-based faces

(lip sync.)

(91)

Video rewrite (analysis)

(92)

Video rewrite (synthesis)

(93)

Results

• Video database

– 2 minutes of JFK

• Only half usable

• Head rotation

training video Read my lips.

I never met Forest Gump.

(94)

Morphable speech model

(95)

Preprocessing

(96)

Prototypes (PCA+k-mean clustering)

We find Ii and Ci for each prototype image.

(97)

Morphable model

I αβ

synthesis

analysis

(98)

Morphable model

synthesis

analysis

(99)

Synthesis

(100)

Results

(101)

Relighting faces

(102)

Light is additive

lamp #1 lamp #2

(103)

Light stage 1.0

(104)

Light stage 1.0

64x32 lighting directions

(105)

Input images

(106)

Reflectance function

occlusion flare

(107)

Relighting

(108)

Results

(109)

Changing viewpoints

(110)

Results

(111)

3D face applications: Spiderman 2

(112)

Spiderman 2

real synthetic

(113)

Application: The Matrix Reloaded

(114)

Application: The Matrix Reloaded

(115)

References

• Paul Debevec, Rendering Synthetic Objects into Real Scenes:

Bridging Traditional and Image-based Graphics with Global Illumination and High Dynamic Range Photography,

SIGGRAPH 1998.

• F. Pighin, J. Hecker, D. Lischinski, D. H. Salesin, and R.

Szeliski. Synthesizing realistic facial expressions from photographs. SIGGRAPH 1998, pp75-84.

• Li Zhang, Noah Snavely, Brian Curless, Steven M. Seitz,

Spacetime Faces: High Resolution Capture for Modeling and Animation, SIGGRAPH 2004.

• Blanz, V. and Vetter, T., A Morphable Model for the Synthesis of 3D Faces, SIGGRAPH 1999, pp187-194.

• Paul Debevec, Tim Hawkins, Chris Tchou, Haarm-Pieter Duiker, Westley Sarokin, Mark Sagar, Acquiring the

Reflectance Field of a Human Face, SIGGRAPH 2000.

• Christoph Bregler, Malcolm Slaney, Michele Covell, Video

Rewrite: Driving Visual Speeach with Audio, SIGGRAPH 1997.

• Tony Ezzat, Gadi Geiger, Tomaso Poggio, Trainable Videorealistic Speech Animation, SIGGRAPH 2002.

參考文獻

相關文件

• Build a model of average shape and texture, and principal variations using PCA..

for Real - - Time Rendering in Dynamic, Time Rendering in Dynamic, Low Low - - Frequency Lighting Environments Frequency Lighting Environments. Peter-Pike Sloan, Microsoft

Efficient Image - - Based Methods Based Methods for Rendering Soft Shadows.. for Rendering

Use images to adapt a generic face model Use images to adapt a generic face model. Creating

‹ Based on the coded rules, facial features in an input image Based on the coded rules, facial features in an input image are extracted first, and face candidates are identified.

These include new curriculum and standards frameworks, a range of professional development programmes, autonomy for schools to select and manage staff, flexibility in the use

• Li-Yi Wei, Marc Levoy, Fast Texture Synthesis Using Tree-Structured Vector Quantization, SIGGRAPH 2000. Jacobs, Nuria Oliver, Brian Curless,

– Camera view plane is parallel to back of volume – Camera up is normal to volume bottom. – Volume bottom