### Image-Based Lighting & Textures

### Digital Visual Effects, Spring 2006 *Yung-Yu Chuang*

### 2005/6/7

*with slides by Alex Efros, Li-Yi Wei and Paul Debevec*

**Announcements**

### • Winners for project #2

### • Voting for project #3

### • Final project:

– Checkpoint this Sunday, send me team, topic and brief progress update

– demo on 6/30 (Friday) 10:00am in this room – Report due on 7/3 (Monday) 11:59pm

**Outline**

### • Image-based lighting

### • Texture synthesis

### • Acceleration by multi-resolution and TSVQ

### • Patch-based texture synthesis

### • Image analogies

**Image-based lighting**

**Rendering**

### • Rendering is a function of geometry, reflectance, lighting and viewing.

### • To synthesize CGI into real scene, we have to match the above four factors.

*• Viewing can be obtained from calibration or * *structure from motion.*

*• Geometry can be captured using 3D * *photography or made by hands.*

### • How to capture lighting and reflectance?

**Reflectance**

• The Bidirectional Reflection Distribution Function

– Given an incoming ray and outgoing ray

what proportion of the incoming light is reflected along outgoing ray?

Answer given by the BRDF:

surface normal

**Capturing reflectance**

**Application in “The Matrix Reloaded”**

**Rendering equation**

### p

### = ) ω p, (

_{o}

*L*

*o*

*L*

_{e}### ( p , ω

_{o}

### )

i i

i i

o

### , ω ) ( p, ω ) cos θ ω

### ω p,

2

### ( *L*

_{i}*d*

### ∫

*s*

### + ρ

### ) ω p, (

_{o}

*L*

*o*

*5D light field*

### ω

i### ω

o### ) ω , p

### (

_{i}

*L*

*i*

**Point lights**

### Classically, rendering is performed assuming point light sources

### directional source

**Environment maps**

Miller and Hoffman, 1984

**Complex illumination**

### = ) ω p, (

_{o}

*L*

*o*

*L*

_{e}### ( p , ω

_{o}

### )

i i

i i

o

### , ω ) ( p, ω ) cos θ ω

### ω p,

2

*f* ( *L*

_{i}*d*

### ∫

*s*

### +

### = ) ω p, (

_{o}

*B*

_{2}

*f* ( p, ω

_{o}

### , ω

_{i}

### ) *L*

_{d}### ( p, ω

_{i}

### ) cos θ

_{i}

*d* ω

_{i}

### ∫

*s*

• Basis Functions are pieces of signal that can be used to produce approximations to a function

### ∫ ^{×}

### ×

### ×

### ∫

### ∫

*c* 1

### = *c* 2

### = *c* 3

### =

**Basis functions**

• We can then use these coefficients to reconstruct an approximation to the original signal

### 1 ×

*c* =

### =

### =

### 2 × *c*

### 3 × *c*

**Basis functions**

• We can then use these coefficients to reconstruct an approximation to the original signal

### ( ) ^{=}

### ∑

=*x* *B*

*c*

*N*
*i*

*i*
*i*
1

**Basis functions**

**Orthogonal basis functions**

### • Orthogonal Basis Functions

– These are families of functions with special properties

– Intuitively, it’s like functions don’t overlap each other’s footprint

• A bit like the way a Fourier transform breaks a functions into component sine waves

### ( ) ( )

### ⎩ ⎨

### ⎧

### ≠

### = =

### ∫ ^{B}

^{B}

^{i}^{x} ^{B}

^{x}

^{B}

^{j}^{x} ^{dx} ^{1} _{0} _{i} ^{i} ^{j} _{j}

^{x}

^{dx}

_{i}

^{i}

^{j}

_{j}

**Basis functions**

### • Transform data to a space in which we can capture the essence of the data better

### • Here, we use spherical harmonics, similar to

### Fourier transform in spherical domain

**Real spherical harmonics**

### • A system of signed, orthogonal functions over the sphere

### • Represented in spherical coordinates by the function

*where l is the band and m is the index within the band*

### ( ) ( ) ( )

### ( ) ( )

### ( )

### ⎪ ⎩

### ⎪ ⎨

### ⎧

### =

### <

### >

### −

### =

^{−}

### 0 0 0 ,

### cos

### , cos

### sin 2

### , cos

### cos 2

### ,

0

0

*m*

*m* *m* *P*

*K*

*P* *m*

*K*

*P* *m*

*K* *y*

*l*
*l*

*m*
*l*
*m*

*l*

*m*
*l*
*m*

*l*
*m*

*l*

### θ

### θ ϕ

### θ ϕ

### ϕ

### θ

**Real spherical harmonics**

**Reading SH diagrams**

**–** **+**

### Not this direction

### This

### direction

**Reading SH diagrams**

**–** **+**

### Not this direction

### This

### direction

**The SH functions**

### =

0

*y*

0
−1

### =

*y*

1 ### = *y*

_{1}

^{1}

1

*y*

2 *y*

_{2}

^{2}

0

*y*

2
1
2

*y*

−
2
2

*y*

−
**The SH functions**

**Spherical harmonics**

**Spherical harmonics**

**-1**

**-2** **0** **1** **2**

**0**

**1**

**2**

### ( , ) *Y* *lm* θ ϕ

*y* *z* *x*

*xy* *yz* 3 *z*

^{2}

### − 1 *zx* *x*

^{2}

### − *y*

^{2}

### 1 *m*

*l*

**SH projection**

**• First we define a strict order for SH functions**

**• Project a spherical function into a vector of** **SH coefficients**

### ( ) ( )

### ∫

### =

*S*

*i*

*i* *f* *s* *y* *s* *ds*

*c*

### ( ) ^{l} ^{m}

^{l}

^{m}

*l*

*i* = + 1 +

**SH reconstruction**

### • To reconstruct the approximation to a function

### • We truncate the infinite series of SH functions **to give a low frequency approximation**

### ( ) ∑ ( )

### =

### =

2

### 0

### ~ ^{N}

^{N}

*i*

*i*

*i* *y* *s* *c*

*s*

*f*

**Examples of reconstruction**

**An example**

### • Take a function comprised of two area light sources

– SH project them into 4 bands = 16 coefficients

### ⎥ ⎥

### ⎥ ⎥

### ⎦

### ⎤

### ⎢ ⎢

### ⎢ ⎢

### ⎣

### ⎡

### −

### − −

### − −

### 238 0

### 0 425

### 0 642 0 001 0 317 0 837 0 940 0 0 417 0 0 278 0 679 0 930 0 908

### 329 0 1

*.* *,*

*,*

*.* *,* *.* *,* *.* *,* *.* *,*

*.* *,* *,* *.* *,* *,* *.* *,*

*.* *,* *.* *,* *.* *,*

*.* *,*

*.*

**Low frequency light source**

### • We reconstruct the signal

– Using only these coefficients to find a low frequency approximation to the original light source

**SH lighting for diffuse objects**

### • *An Efficient Representation for Irradiance *

*Environment Maps, Ravi Ramamoorthi and Pat * Hanrahan, SIGGRAPH 2001

### • Assumptions

– Diffuse surfaces

– Distant illumination

– No shadowing, interreflection

irradiance is a function of surface normal

### = ) ( *p,ω*

_{o}*B*

_{2}

*f* ( p, ω

_{o}

### , ω

_{i}

### ) *L*

_{d}### ( p, ω

_{i}

### ) cos θ

_{i}

*d* ω

_{i}

### ∫

*s*

### ) n ( ) *( E* *p* ρ

### =

*n)*

*B(p,*

**Diffuse reflection**

*B* = ρ *E*

### radiosity

### (image intensity)

### reflectance

### (albedo/texture) irradiance

### (incoming light)

### = ×

### quake light map

**Irradiance environment maps**

Illumination Environment Map Irradiance Environment Map

L *n*

### ( )( )

Ω

### ∫

### ⋅

### = *L* ω *n* ω *d* ω

*n*

*E )* (

**Spherical harmonic expansion**

### Expand lighting (L), irradiance (E) in basis functions

0

### ( , )

^{l}

_{lm lm}### ( , )

*l* *m* *l*

*L* θ φ

^{∞}

^{+}

*L Y* θ φ

= =−

### = ∑ ∑

0

### ( , )

^{l}

_{lm lm}### ( , )

*l* *m* *l*

*E* θ φ

^{∞}

^{+}

*E Y* θ φ

= =−

### = ∑ ∑

**= .67** **+ .36** **+ …**

**Analytic irradiance formula**

### Lambertian surface acts like low-pass filter

*lm* *l lm*

*E* = *A L* ^{A}

^{A}

^{l}### π

### 2 / 3 π

### π / 4 0

### ( )

2 1

2 2

( 1) !

2 ( 2)( 1) 2 !

*l*

*l* *l* *l*

*A* *l* *l even*

*l* *l*

π ^{−} ^{−} ^{⎡} ^{⎤}

= ⎢ ⎥

+ − ⎢⎣ ⎥⎦

### 0 1 2 *l*

cosine term

**9 parameter approximation**

### Exact image Order 0

### 1 term

**RMS error = 25 %**

**-1**

**-2** **0** **1** **2**

**0**
**1**
**2**

### ( , ) *Y*

*lm*

### θ ϕ

*y* *z* *x*

*xy* *yz* 3*z*^{2} −1 *zx* *x*^{2} − *y*^{2}

*l*

*m*

**9 Parameter Approximation**

### Exact image Order 1

### 4 terms

**RMS Error = 8%**

**-1**

**-2** **0** **1** **2**

**0**
**1**
**2**

### ( , ) *Y*

*lm*

### θ ϕ

*y* *z* *x*

*xy* *yz* 3*z*^{2} −1 *zx* *x*^{2} − *y*^{2}

*l*

*m*

**9 Parameter Approximation**

### Exact image Order 2

### 9 terms

**RMS Error = 1%**

### For any illumination, average error < 3% [Basri Jacobs 01]

**-1**

**-2** **0** **1** **2**

**0**
**1**
**2**

### ( , ) *Y*

*lm*

### θ ϕ

*y* *z* *x*

*xy* *yz* 3*z*^{2} −1 *zx* *x*^{2} − *y*^{2}

*l*

*m*

**Comparison**

Incident illumination

300x300

Irradiance map Texture: 256x256

Hemispherical Integration 2Hrs

Irradiance map Texture: 256x256 Spherical Harmonic

Coefficients 1sec

Time 300 300 256 256∝ × × × Time 9 256 256∝ × ×

**Complex geometry**

### Assume no shadowing: Simply use surface normal

*y*

**Natural illumination**

### For diffuse objects, rendering with natural illumination can be done quickly

### directional source natural illumination

**HDRI Sky Probe**

**Clipped Sky + Sun Source**

**Lit by sun only**

**Lit by sky only**

**Lit by sun and sky**

**Acquiring the Light Probe**

**Illuminating a Small Scene**

**Real Scene Example**

### • Goal: place synthetic objects on table

**Light Probe / Calibration Grid**

real scene real scene

**Modeling the Scene**

light-based model light-based model

**The Light-Based Room Model**

**The Light-Based Room Model**

**Rendering into the Scene**

### • Background Plate

**Rendering into the scene**

### • Objects and Local Scene matched to Scene

**Differential rendering**

### • Local scene w/o objects, illuminated by model

**Differential rendering**

### = =

### - -

**Differential Rendering**

### • Final Result

**Environment map from single image?**

**Eye as light probe! (Nayar et al)**

**Cornea is an ellipsoid**

**Results**

**Application in “The Matrix Reloaded”**

**Texture synthesis**

**Texture synthesis**

### • Given a finite sample of some texture, the goal is to synthesize other samples from that same texture.

– The sample needs to be "large enough"

### synthesis

generated image input image

**The challenge**

### • How to capture the essence of texture?

### • Need to model the whole spectrum: from

### repeated to stochastic texture

*N*

_{i}**Markov property**

### • P(f

_{i}

### |f

_{i-1}

### ,f

_{i-2}

### ,f

_{i-3}

### ,…f

_{0}

### )= P(f

_{i}

### |f

_{i-1}

### ,f

_{i-2}

### ,…f

_{i-n}

### )

**ff**_{i}
**Ff**_{i-1}

**Ff**_{0}

*S*

*f*_{i}**Ff**_{i-2}

**…** **Ff**_{i-n} **…**

### • P(f

_{i}

### |f

_{S-{i}}

### )= P(f

_{i}

### |f

_{Ni}

### )

**Motivation from language**

### • [Shannon’48] proposed a way to generate English-looking text using N-grams:

– Assume a generalized Markov model

– Use a large text to compute probability

distributions of each letter given N-1 previous letters

• precompute or sample randomly

– Starting from a seed repeatedly sample this Markov chain to generate new letters

– One can use whole words instead of letters too.

**Mark V. Shaney (Bell Labs)**

### • Results (using alt.singles corpus):

– *"One morning I shot an elephant in my arms and *
*kissed him.”*

– *"I spent an interesting evening recently with a *
*grain of salt"*

### • Notice how well local structure is preserved!

– Now let’s try this in 2D...

– Assuming Markov property, what is conditional probability distribution of p, given the neighbourhood window?

– Instead of constructing a model, let’s directly search the input image for all such neighbourhoods to produce a

histogram for p

– To synthesize p, just pick one match at random

### SAMPLE

**p**

**Ideally**

**Infinite sample image**

**generated image**

**finite sample ****image**

**Generated image**

**p**

– However, since our sample image is finite, an exact neighbourhood match might not be present

– So we find the best match using SSD error (weighted by a

Gaussian to emphasize local structure), and take all samples within some distance from that match

– *Using Gaussian-weighted SSD is very important*

### SAMPLE

**In reality**

**Neighborhood size matters**

**More results**

Increasing window size

french canvas rafia weave

**More results**

wood

**More results**

brick wall

**Failure cases**

**Growing garbage** **Verbatim copying**

**Inpainting**

• Growing is in “onion peeling” order

– within each “layer”, pixels with most neighbors are synthesized first

**Inpainting**

**Inpainting**

**Results**

**Summary of the basic algorithm**

### • Exhaustively search neighborhoods

**Neighborhood**

### • Neighborhood size determines the quality &

### cost

### 3×3 5×5 7×7

### 9×9 11×11 41×41

423 s 528 s 739 s

1020 s 1445 s 24350 s

**Summary**

### • Advantages:

– conceptually simple

– models a wide range of real-world textures – naturally does hole-filling

### • Disadvantages:

– it’s slow

– it’s a heuristic

**Acceleration by Wei & Levoy**

### • Multi-resolution

### • Tree-structure

**Multi-resolution pyramid**

### High resolution Low resolution

**Multi-resolution algorithm**

**Benefits**

### • Better image quality & faster computation

### 1 level

### 5×5 3 levels

### 5×5 1 level

### 11×11

**Results**

Random Oriented

Regular Semi-regular

**Failures**

### • Non-planar structures

### • Global information

**Acceleration **

### • Computation bottleneck: neighborhood search

**Nearest point search**

### • Treat neighborhoods as high dimensional points

1 2 3 4 5 6 7 8 9 10 11 12

### Neighborhood

1 2 3 4 5 6 7 8 9 10 11 12

### High dimensional point/vector

**Tree-Structured Vector Quantization**

**Timing**

### • Time complexity : O(log N) instead of O(N)

– 2 orders of magnitude speedup for non-trivial images

### 1941 secs 503 secs 12 secs

### Efros 99 Full searching TSVQ

**Results**

### Input Exhaustive: 360 s TSVQ: 7.5 s

**p** **p**

**Patch-based methods**

• Observation: neighbor pixels are highly correlated

**Input image**

**non-parametric**
**sampling**

**B** **B**

**Idea:**

**Idea:** **unit of synthesis = block** **unit of synthesis = block**

• **Exactly the same but now we want P(B|N(B))**

• Much faster: synthesize all pixels in a block at once

**Synthesizing a block**

**Algorithm**

– Pick size of block and size of overlap – Synthesize blocks in raster order

– Search input texture for block that satisfies overlap constraints (above and left)

– Paste new block into resulting texture

• blending

• use dynamic programming to compute minimal error boundary cut

**Input texture**

**B1** **B2**

**Random placement **
**of blocks **

block

**B1** **B2**

**Neighboring blocks**
**constrained by overlap**

**B1** **B2**

**Minimal error**
**boundary cut**

**min. error boundary**

**Minimal error boundary**

**overlapping blocks** **vertical boundary**

### _ _ ^{2} ^{2} _{=} _{=}

_{=}

_{=}

**overlap error**

**Results**

**Results**

**Failure cases**

**GraphCut textures**

**GraphCut textures**

**Photomontage**

**Photomontage**

**Image Analogies**

**Coherence search**

**Image Analogies Implementation**

**Image Analogies Implementation**

**Image Analogies Implementation**

**Balance between approximate and **

**coherence searches**

**Learn to blur**

**Super-resolution**

**Colorization**

**Artistic filters**

**:**

**B** **B’**

**::**

**:**

**B** **B’**

**::**

**Texture by numbers**

**Texture by numbers**

**References**

• Paul Debevec, Rendering Synthetic Objects into Real Scenes: Bridging Traditional and Image-based Graphics with Global Illumination and High Dynamic Range

Photography, SIGGRAPH 1998.

• Alexei A. Efros, Thomas K. Leung, Texture Synthesis by Non-parametric Sampling, ICCV 1999.

• Li-Yi Wei, Marc Levoy, Fast Texture Synthesis Using Tree-Structured Vector Quantization, SIGGRAPH 2000.

• Aaron Hertzmann, Charles E. Jacobs, Nuria Oliver, Brian Curless, David H. Salesin, Image Analogies, SIGGRAPH 2001.

• Alexei A. Efros, William T. Freeman, Image Quilting for Texture Synthesis and Transfer, SIGGRAPH 2001.