Image-Based Lighting & Textures

Image-Based Lighting & Textures

Digital Visual Effects, Spring 2006 Yung-Yu Chuang

2005/6/7

with slides by Alex Efros, Li-Yi Wei and Paul Debevec

Announcements

• Winners for project #2

• Voting for project #3

• Final project:

– Checkpoint this Sunday, send me team, topic and brief progress update

– demo on 6/30 (Friday) 10:00am in this room – Report due on 7/3 (Monday) 11:59pm

Outline

• Image-based lighting

• Texture synthesis

• Acceleration by multi-resolution and TSVQ

• Patch-based texture synthesis

• Image analogies

Image-based lighting

Rendering

• Rendering is a function of geometry, reflectance, lighting and viewing.

• To synthesize CGI into real scene, we have to match the above four factors.

• Viewing can be obtained from calibration or structure from motion.

• Geometry can be captured using 3D photography or made by hands.

• How to capture lighting and reflectance?

Reflectance

• The Bidirectional Reflection Distribution Function

– Given an incoming ray and outgoing ray

what proportion of the incoming light is reflected along outgoing ray?

Answer given by the BRDF:

surface normal

Capturing reflectance

Application in “The Matrix Reloaded”

Rendering equation

p

= ) ω p, (

L

L

( p , ω

)

i i

i i

o

, ω ) ( p, ω ) cos θ ω

ω p,

2

( L

d

∫

+ ρ

) ω p, (

L

5D light field

ω

ω

) ω , p

(

L

Point lights

Classically, rendering is performed assuming point light sources

directional source

Environment maps

Miller and Hoffman, 1984

Complex illumination

= ) ω p, (

L

L

( p , ω

)

i i

i i

o

, ω ) ( p, ω ) cos θ ω

ω p,

2

f ( L

d

∫

+

= ) ω p, (

B

f ( p, ω

, ω

) L

( p, ω

) cos θ

d ω

∫

• Basis Functions are pieces of signal that can be used to produce approximations to a function

∫ ^×

×

×

∫

∫

c 1

= c 2

= c 3

=

Basis functions

• We can then use these coefficients to reconstruct an approximation to the original signal

1 ×

c =

=

=

2 × c

3 × c

Basis functions

• We can then use these coefficients to reconstruct an approximation to the original signal

( ) ⁼

∑

x B

c

N i

i i 1

Basis functions

Orthogonal basis functions

• Orthogonal Basis Functions

– These are families of functions with special properties

– Intuitively, it’s like functions don’t overlap each other’s footprint

• A bit like the way a Fourier transform breaks a functions into component sine waves

( ) ( )

⎩ ⎨

⎧

≠

= =

∫ ^B

^{x B}

^{x dx 1} _{0 i} ^{i j} _j

Basis functions

• Transform data to a space in which we can capture the essence of the data better

• Here, we use spherical harmonics, similar to

Fourier transform in spherical domain

Real spherical harmonics

• A system of signed, orthogonal functions over the sphere

• Represented in spherical coordinates by the function

where l is the band and m is the index within the band

( ) ( ) ( )

( ) ( )

( )

⎪ ⎩

⎪ ⎨

⎧

=

<

>

−

=

0 0 0 ,

cos

, cos

sin 2

, cos

cos 2

,

0

0

m

m m P

K

P m

K

P m

K y

l l

m l m

l

m l m

l m

l

θ

θ ϕ

θ ϕ

ϕ

θ

Real spherical harmonics

Reading SH diagrams

– +

Not this direction

This

direction

Reading SH diagrams

– +

Not this direction

This

direction

The SH functions

=

0

y

−1

=

y

= y

1

y

y

0

y

2

y

2

y

The SH functions

Spherical harmonics

Spherical harmonics

-1

-2 0 1 2

0

1

2

( , ) Y lm θ ϕ

y z x

xy yz 3 z

− 1 zx x

− y

1 m

l

SH projection

• First we define a strict order for SH functions

• Project a spherical function into a vector of SH coefficients

( ) ( )

∫

=

S

i

i f s y s ds

c

( ) ^{l m}

l

i = + 1 +

SH reconstruction

• To reconstruct the approximation to a function

• We truncate the infinite series of SH functions to give a low frequency approximation

( ) ∑ ( )

=

=

2

0

~ ^N

i

i

i y s c

s

f

Examples of reconstruction

An example

• Take a function comprised of two area light sources

– SH project them into 4 bands = 16 coefficients

⎥ ⎥

⎥ ⎥

⎦

⎤

⎢ ⎢

⎢ ⎢

⎣

⎡

−

− −

− −

238 0

0 425

0 642 0 001 0 317 0 837 0 940 0 0 417 0 0 278 0 679 0 930 0 908

329 0 1

. ,

,

. , . , . , . ,

. , , . , , . ,

. , . , . ,

. ,

.

Low frequency light source

• We reconstruct the signal

– Using only these coefficients to find a low frequency approximation to the original light source

SH lighting for diffuse objects

• An Efficient Representation for Irradiance

Environment Maps, Ravi Ramamoorthi and Pat Hanrahan, SIGGRAPH 2001

• Assumptions

– Diffuse surfaces

– Distant illumination

– No shadowing, interreflection

irradiance is a function of surface normal

= ) ( p,ω

B

f ( p, ω

, ω

) L

( p, ω

) cos θ

d ω

∫

) n ( ) ( E p ρ

=

n)

B(p,

Diffuse reflection

B = ρ E

radiosity

(image intensity)

reflectance

(albedo/texture) irradiance

(incoming light)

= ×

quake light map

Irradiance environment maps

Illumination Environment Map Irradiance Environment Map

L n

( )( )

Ω

∫

⋅

= L ω n ω d ω

n

E ) (

Spherical harmonic expansion

Expand lighting (L), irradiance (E) in basis functions

0

( , )

( , )

l m l

L θ φ

L Y θ φ

= =−

= ∑ ∑

0

( , )

( , )

l m l

E θ φ

E Y θ φ

= =−

= ∑ ∑

= .67 + .36 + …

Analytic irradiance formula

Lambertian surface acts like low-pass filter

lm l lm

E = A L ^A

π

2 / 3 π

π / 4 0

( )

2 1

2 2

( 1) !

2 ( 2)( 1) 2 !

l

l l l

A l l even

l l

π ^{− − ⎡ ⎤}

= ⎢ ⎥

+ − ⎢⎣ ⎥⎦

0 1 2 l

cosine term

9 parameter approximation

Exact image Order 0

1 term

RMS error = 25 %

-1

-2 0 1 2

0 1 2

( , ) Y

θ ϕ

y z x

xy yz 3z² −1 zx x² − y²

l

m

9 Parameter Approximation

Exact image Order 1

4 terms

RMS Error = 8%

-1

-2 0 1 2

0 1 2

( , ) Y

θ ϕ

y z x

xy yz 3z² −1 zx x² − y²

l

m

9 Parameter Approximation

Exact image Order 2

9 terms

RMS Error = 1%

For any illumination, average error < 3% [Basri Jacobs 01]

-1

-2 0 1 2

0 1 2

( , ) Y

θ ϕ

y z x

xy yz 3z² −1 zx x² − y²

l

m

Comparison

Incident illumination

300x300

Irradiance map Texture: 256x256

Hemispherical Integration 2Hrs

Irradiance map Texture: 256x256 Spherical Harmonic

Coefficients 1sec

Time 300 300 256 256∝ × × × Time 9 256 256∝ × ×

Complex geometry

Assume no shadowing: Simply use surface normal

y

Natural illumination

For diffuse objects, rendering with natural illumination can be done quickly

directional source natural illumination

HDRI Sky Probe

Clipped Sky + Sun Source

Lit by sun only

Lit by sky only

Lit by sun and sky

Acquiring the Light Probe

Illuminating a Small Scene

Real Scene Example

• Goal: place synthetic objects on table

Light Probe / Calibration Grid

real scene real scene

Modeling the Scene

light-based model light-based model

The Light-Based Room Model

Rendering into the Scene

• Background Plate

Rendering into the scene

• Objects and Local Scene matched to Scene

Differential rendering

• Local scene w/o objects, illuminated by model

Differential rendering

= =

- -

Differential Rendering

• Final Result

Environment map from single image?

Eye as light probe! (Nayar et al)

Cornea is an ellipsoid

Results

Application in “The Matrix Reloaded”

Texture synthesis

Texture synthesis

• Given a finite sample of some texture, the goal is to synthesize other samples from that same texture.

– The sample needs to be "large enough"

synthesis

generated image input image

The challenge

• How to capture the essence of texture?

• Need to model the whole spectrum: from

repeated to stochastic texture

N

Markov property

• P(f

|f

,f

,f

,…f

)= P(f

|f

,f

,…f

)

ff_i Ff_i-1

Ff₀

S

f_i Ff_i-2

… Ff_i-n …

• P(f

|f

)= P(f

|f

)

Motivation from language

• [Shannon’48] proposed a way to generate English-looking text using N-grams:

– Assume a generalized Markov model

– Use a large text to compute probability

distributions of each letter given N-1 previous letters

• precompute or sample randomly

– Starting from a seed repeatedly sample this Markov chain to generate new letters

– One can use whole words instead of letters too.

Mark V. Shaney (Bell Labs)

• Results (using alt.singles corpus):

– "One morning I shot an elephant in my arms and kissed him.”

– "I spent an interesting evening recently with a grain of salt"

• Notice how well local structure is preserved!

– Now let’s try this in 2D...

– Assuming Markov property, what is conditional probability distribution of p, given the neighbourhood window?

– Instead of constructing a model, let’s directly search the input image for all such neighbourhoods to produce a

histogram for p

– To synthesize p, just pick one match at random

SAMPLE

p

Ideally

Infinite sample image

generated image

finite sample image

Generated image

p

– However, since our sample image is finite, an exact neighbourhood match might not be present

– So we find the best match using SSD error (weighted by a

Gaussian to emphasize local structure), and take all samples within some distance from that match

– Using Gaussian-weighted SSD is very important

SAMPLE

In reality

Neighborhood size matters

More results

Increasing window size

french canvas rafia weave

More results

wood

More results

brick wall

Failure cases

Growing garbage Verbatim copying

Inpainting

• Growing is in “onion peeling” order

– within each “layer”, pixels with most neighbors are synthesized first

Inpainting

Inpainting

Results

Summary of the basic algorithm

• Exhaustively search neighborhoods

Neighborhood

• Neighborhood size determines the quality &

cost

3×3 5×5 7×7

9×9 11×11 41×41

423 s 528 s 739 s

1020 s 1445 s 24350 s

Summary

• Advantages:

– conceptually simple

– models a wide range of real-world textures – naturally does hole-filling

• Disadvantages:

– it’s slow

– it’s a heuristic

Acceleration by Wei & Levoy

• Multi-resolution

• Tree-structure

Multi-resolution pyramid

High resolution Low resolution

Multi-resolution algorithm

Benefits

• Better image quality & faster computation

1 level

5×5 3 levels

5×5 1 level

11×11

Results

Random Oriented

Regular Semi-regular

Failures

• Non-planar structures

• Global information

Acceleration

• Computation bottleneck: neighborhood search

Nearest point search

• Treat neighborhoods as high dimensional points

1 2 3 4 5 6 7 8 9 10 11 12

Neighborhood

1 2 3 4 5 6 7 8 9 10 11 12

High dimensional point/vector

Tree-Structured Vector Quantization

Timing

• Time complexity : O(log N) instead of O(N)

– 2 orders of magnitude speedup for non-trivial images

1941 secs 503 secs 12 secs

Efros 99 Full searching TSVQ

Results

Input Exhaustive: 360 s TSVQ: 7.5 s

p p

Patch-based methods

• Observation: neighbor pixels are highly correlated

Input image

non-parametric sampling

B B

Idea:

Idea: unit of synthesis = block unit of synthesis = block

• Exactly the same but now we want P(B|N(B))

• Much faster: synthesize all pixels in a block at once

Synthesizing a block

Algorithm

– Pick size of block and size of overlap – Synthesize blocks in raster order

– Search input texture for block that satisfies overlap constraints (above and left)

– Paste new block into resulting texture

• blending

• use dynamic programming to compute minimal error boundary cut

Input texture

B1 B2

Random placement of blocks

block

B1 B2

Neighboring blocks constrained by overlap

B1 B2

Minimal error boundary cut

min. error boundary

Minimal error boundary

overlapping blocks vertical boundary

_ _ ^{2 2} _{= =}

overlap error

Results

Results

Failure cases

GraphCut textures

GraphCut textures

Photomontage

Photomontage

Image Analogies

Coherence search

Image Analogies Implementation

Image Analogies Implementation

Image Analogies Implementation

Balance between approximate and

coherence searches

Learn to blur

Super-resolution

Colorization

Artistic filters

:

B B’

::

:

B B’

::

Texture by numbers

Texture by numbers

References

• Paul Debevec, Rendering Synthetic Objects into Real Scenes: Bridging Traditional and Image-based Graphics with Global Illumination and High Dynamic Range

Photography, SIGGRAPH 1998.

• Alexei A. Efros, Thomas K. Leung, Texture Synthesis by Non-parametric Sampling, ICCV 1999.

• Li-Yi Wei, Marc Levoy, Fast Texture Synthesis Using Tree-Structured Vector Quantization, SIGGRAPH 2000.

• Aaron Hertzmann, Charles E. Jacobs, Nuria Oliver, Brian Curless, David H. Salesin, Image Analogies, SIGGRAPH 2001.

• Alexei A. Efros, William T. Freeman, Image Quilting for Texture Synthesis and Transfer, SIGGRAPH 2001.