Non-linear Wavelet Lighting Approximation

(1)

All-Frequency Shadows Using

Non-linear Wavelet Lighting Approximation

Pat Hanrahan Stanford Ravi Ramamoorthi

Columbia SIGGRAPH 2003 Ren Ng

Stanford

Light on Stone River (Goldsworthy)

From Frank Gehry Architecture, Ragheb ed. 2001

Lighting Design Existing Fast Shadow Techniques

We know how to render very hard and very soft shadows

Sloan, Kautz, Snyder 2002

Shadows from smooth lighting (precomputed radiance transfer)

Sen, Cammarano, Hanrahan, 2003

Shadows from point-lights (shadow maps, volumes)

(2)

Soft Lighting

Teapot in Grace Cathedral

All-Frequency Lighting

Teapot in Grace Cathedral

Formulation

∫

Ω

⋅

= ω ω ω ω ω ω

ω L V f d

B ( x ,

₀

) ( x , ) ( x , ) ( x , ,

₀

)( n ( x ))

Geometry relighting: assume diffuse

)) ( )(

, ( ) , ( ) ,

( x ω = V x ω f x ω → ω

₀

ω ⋅ n x T

Image relighting: fix view point

)) ( ))(

( ,

( ) , ( ) ,

( x ω = V x ω f x ω → ω

₀

x ω ⋅ n x T

∑

=

j

j j i

i

T L

B ( x ) ( x , ω ) ( ω ) TL

B =

T could include global effects

Relighting as Matrix-Vector Multiply

1 2 3

N

P P P

P

⎡ ⎤⎢ ⎥

⎢ ⎥⎢ ⎥

⎢ ⎥⎣ ⎦ M

11 12 1

1

21 22 2

2

31 32 3

1 2

M M M

N

N N NM

T T T

T T T L

⎡ ⎤

⎢ ⎥ ⎡ ⎤

⎢ ⎥ ⎢ ⎥

=⎢ ⎥ ⎢ ⎥

⎢ ⎥ ⎢ ⎥⎣ ⎦

⎢ ⎥

⎣ ⎦

L L

L M

M M O M

L

(3)

Relighting as Matrix-Vector Multiply

1 2 3

N

P P P

P

⎡ ⎤⎢ ⎥

⎢ ⎥⎢ ⎥

⎢ ⎥⎣ ⎦ M

11 12 1

1

21 22 2

2

31 32 3

1 2

M M M

N

N N NM

T T T

T T T L

⎡ ⎤

⎢ ⎥ ⎡ ⎤

⎢ ⎥ ⎢ ⎥

=⎢ ⎥ ⎢ ⎥

⎢ ⎥ ⎢ ⎥⎣ ⎦

⎢ ⎥

⎣ ⎦

L L

L M

M M O M

L

Input Lighting (Cubemap Vector) Output Image

(Pixel Vector)

Transport Matrix

Ray-Tracing Matrix Columns

11 12 1

21 22 2

31 32 3

1 2

M M M

N N NM

T T T

⎡ ⎤

⎢ ⎥

⎣ ⎦

L L L

M M O M

L

Ray-Tracing Matrix Columns

11 12 1

21 22 2

31 32 3

1 2

M M M

N N NM

T T T

⎡ ⎤

⎢ ⎥

⎣ ⎦

L L L

M M O M

L

11 12 1

21 22 2

31 32 3

1 2

M M M

N N NM

T T T

⎡ ⎤

⎢ ⎥

⎣ ⎦

L L L

M M O M

L

11 12 1

21 22 2

31 32 3

1 2

M M M

N N NM

T T T

⎡ ⎤

⎢ ⎥

⎣ ⎦

L L L

M M O M

L

Only works for image relighting

Light-Transport Matrix Rows

11 12 1

21 22 2

31 32 3

1 2

M M M

N N NM

T T T

⎡ ⎤

⎢ ⎥

⎣ ⎦

L L L

M M O M

L

(4)

Light-Transport Matrix Rows

11 12 1

21 22 2

31 32 3

1 2

M M M

N N NM

T T T

⎡ ⎤

⎢ ⎥

⎣ ⎦

L L L

M M O M

L

Light-Transport Matrix Rows

11 12 1

21 22 2

31 32 3

1 2

M M M

N N NM

T T T

⎡ ⎤

⎢ ⎥

⎣ ⎦

L L L

M M O M

L

Rasterizing Matrix Rows

Pre-computing rows

Rasterize visibility hemicubes with graphics hardware

Read back pixels and weight by reflection function

Matrix Multiplication is Enormous

Dimension

512 x 512 pixel images

6 x 64 x 64 cubemap environments Full matrix-vector multiplication is intractable

On the order of 10¹⁰operations per frame

(5)

Sparse Matrix-Vector Multiplication

Choose data representations with mostly zeroes Vector: Use non-linear wavelet approximation

on lighting

Matrix: Wavelet-encode transport rows

11 12 1

1

21 22 2

2

31 32 3

1 2

M M M

N

N N NM

T T T

T T T L

⎡ ⎤

⎢ ⎥ ⎡ ⎤

⎢ ⎥ ⎢ ⎥

⎢ ⎥ ⎢ ⎥⎣ ⎦

⎢ ⎥

⎣ ⎦

L L

L M

M M O M

L

Non-linear Wavelet Light Approximation

Wavelet Transform

1 2 3 4 5 6

N

L L L L L L

L

⎡ ⎤ ⎢ ⎥

⎢ ⎥ ⎢ ⎥

⎣ ⎦ M

2

6

0 0 0 0

0 L

L

⎡ ⎤ ⎢ ⎥

⎢ ⎥ ⎢ ⎥

⎣ ⎦ M

Non-linear Wavelet Light Approximation

Non-linear Approximation

Retain 0.1% – 1% terms

Why Non-linear Approximation?

Linear

Use a fixedset of approximating functions

Precomputed radiance transfer uses 25 - 100 of the lowest frequency spherical harmonics Non-linear

Use a dynamicset of approximating functions (depends on each frame’s lighting)

In our case: choose 10’s - 100’s from a basis of 24,576 wavelets

Idea: Compress lighting by considering input data

(6)

Why Wavelets?

Wavelets provide dual space / frequency locality

Large wavelets capture low frequency, area lighting

Small wavelets capture high frequency, compact features

In contrast

Spherical harmonics

Perform poorly on compact lights

Pixel basis

Perform poorly on large area lights

Choosing Non-linear Coefficients

Three methods of prioritizing

1. Magnitude of wavelet coefficient

Optimal for approximating lighting

2. Transport-weighted

Biases lights that generate bright images

3. Area-weighted

Biases large lights

Sparse Matrix-Vector Multiplication

on lighting

11 12 1

1

21 22 2

2

31 32 3

1 2

M M M

N

N N NM

T T T

T T T L

⎡ ⎤

⎢ ⎥ ⎡ ⎤

⎢ ⎥ ⎢ ⎥

⎢ ⎥ ⎢ ⎥⎣ ⎦

⎢ ⎥

⎣ ⎦

L L

L M

M M O M

L

on lighting

11 12 1

1

21 22 2

2

31 32 3

1 2

M M M

N

N N NM

T T T

T T T L

⎡ ⎤

⎢ ⎥ ⎡ ⎤

⎢ ⎥ ⎢ ⎥

⎢ ⎥ ⎢ ⎥⎣ ⎦

⎢ ⎥

⎣ ⎦

L L

L M

M M O M

L

11 12 13 14 1

21 22 23 24 2

31 32 24 34 3

41 42 43 44 4

51 52 53 54 5

61 62 63 64 6

7

1 2 3 4

M M M M M M M

N N N N NM

T T T T T

T

T T T T T

⎡ ⎤

⎢ ⎥

⎣ ⎦

L L L L L L

M M M M O

L

Matrix Row Wavelet Encoding

(7)

Matrix Row Wavelet Encoding

11 12 13 14 1

21 22 23 24 2

31 32 24 34 3

41 42 43 44 4

51 52 53 54 5

61 62 63 64 6

7

1 2 3 4

M M M M M M M

N N N N NM

T T T T T

T

T T T T T

⎡ ⎤

⎢ ⎥

⎣ ⎦

L L L L L L

M M M M O

L

Extract Row

Matrix Row Wavelet Encoding

11 12 13 14 1

21 22 23 24 2

31 32 24 34 3

41 42 43 44 4

51 52 53 54 5

61 62 63 64 6

7

1 2 3 4

M M M M M M M

N N N N NM

T T T T T

T

T T T T T

⎡ ⎤

⎢ ⎥

⎣ ⎦

L L L L L L

M M M M O

L

Matrix Row Wavelet Encoding

11 12 13 14 1

21 22 23 24 2

31 32 24 34 3

41 42 43 44 4

51 52 53 54 5

61 62 63 64 6

7

1 2 3 4

M M M M M M M

N N N N NM

T T T T T

T

T T T T T

⎡ ⎤

⎢ ⎥

⎣ ⎦

L L L L L L

M M M M O

L

Matrix Row Wavelet Encoding

11 12 13 14 1

21 22 23 24 2

31 32 24 34 3

41 42 43 44 4

51 52 53 54 5

61 62 63 64 6

7

1 2 3 4

M M M M M M M

N N N N NM

T T T T T

T

T T T T T

⎡ ⎤

⎢ ⎥

⎣ ⎦

L L L L L L

M M M M O

L

(8)

Matrix Row Wavelet Encoding

11 12 13 14 1

21 22 23 24 2

31 32 24 34 3

41 42 43 44 4

51 52 53 54 5

61 62 63 64 6

7

1 2 3 4

M M M M M M M

N N N N NM

T T T T T

T

T T T T T

⎡ ⎤

⎢ ⎥

⎣ ⎦

L L L L L L

M M M M O

L

Matrix Row Wavelet Encoding

11 12 13 14 1

21 22 23 24 2

31 32 24 34 3

41 42 43 44 4

51 52 53 54 5

61 62 63 64 6

7

1 2 3 4

M M M M M M M

N N N N NM

T T T T T

T

T T T T T

⎡ ⎤

⎢ ⎥

⎣ ⎦

L L L L L L

M M M M O

L

Store Back in Matrix

’

0 0

’

0 Matrix Row Wavelet Encoding

11 12 13 14 1

21 22 23 24 2

31 32 24 34 3

41 42 43 44 4

51 52 53 54 5

61 62 63 64 6

7

1 2 3 4

M M M M M M M

N N N N NM

T T T T T

T

T T T T T

⎡ ⎤

⎢ ⎥

⎣ ⎦

L L L L L L

M M M M O

L

’

0 0

’

0

Only 3% – 30% are non-zero

Total Compression

Lighting vector compression

Highly lossy

Compress to 0.1% – 1%

Matrix compression

Essentially lossless encoding

Represent with 3% – 30% non-zero terms

Total compression in sparse matrix-vector multiply

3 – 4 orders of magnitude less work than full multiplication

(9)

Error Analysis

Compare to linear spherical harmonic approximation

[Sloan, Kautz, Snyder, 2002]

Measure approximation quality in two ways:

Error in lighting

Error in output image pixels

Main point (for detailed natural lighting)

Non-linear wavelets converge exponentially faster than linear harmonics

Lighting Error

Error in Lighting: St Peter’s Basilica

Approximation Terms Relative L2Error (%)

Sph. Harmonics

Non-linear Wavelets

Output Image Comparison

Top: Linear Spherical Harmonic Approximation Bottom: Non-linear Wavelet Approximation

25 200 2,000 20,000

(10)

Error in Output Image: Plant Scene

Sph. Harmonics

Non-linear Wavelets

Results

SIGGRAPH 2003 video

Summary

A viable representation for all-frequency lighting

Sparse non-linear wavelet approximation

100 – 1000 times information reduction

Efficient relighting from detailed environment maps

Triple Product Wavelet Integrals For All-Frequency Relighting

Pat Hanrahan Stanford Ravi Ramamoorthi

Columbia SIGGRAPH 2004 Ren Ng

Stanford

(11)

Precomputed Relighting Techniques

Low-Frequency Lighting, Dynamic View [Sloan et al 2002, 2003]

All-Frequency Lighting, Fixed View

[Ng et al 2003]

Formulation

∫

Ω

⋅

= ω ω ρ ω ω ω ω

ω L V d

B ( x ,

₀

) ( x , ) ( x , ) ( x , ,

₀

)( n ( x ))

Simplifications:

1. Incorporate cosine term into BRDF

2. Assume the lighting is distant, so L depends only on ω

3. Uniform BRDF over the surface

4. Expressed in the global frame; BRDF has to be rotated

∫

Ω

= ω ω ρ ω ω ω

ω L V d

B ( x ,

₀

) ( ) ( x , ) ~ ( x , ,

₀

)

∫

Ω

= L ω V ω ρ ω d ω B ( ) ( ) ~ ( )

For a fixed point,

Double product

∑ ^Ψ

=

i i

L

i

L ( ω ) ( ω )

∫

Ω

= L ω V ω ρ ω d ω B ( ) ( ) ~ ( )

∑ ^Ψ

=

j j

T

j

T ( ω ) ( ω ) )

( ω T

Let

C

_ij

= ∫

Ω

Ψ

_i

( ω ) Ψ

_j

( ω ) d ω

L T ⋅

=

Ψ Ψ

=

⎟⎟ ⎠

⎜⎜ ⎞

⎝

⎛ Ψ

⎟ ⎠

⎜ ⎞

⎝

⎛ Ψ

=

∑

∑∑

∑∑ ∫

∫ ∑ ∑

Ω Ω

i i i

i j

ij j i

i j

ij j i

j

i j

i j i

j j j i

i i

T L T

L C

T L

d T

L

d T

L B

δ ω ω ω

ω ω ω

) ( ) (

) ( )

(

Double Product Integral Relighting

Changing Surface Position Changing Camera Viewpoint

Lighting

Transport

(12)

Problem Characterization

6D Precomputation Space

Distant Lighting (2D)

Surface (2D)

View (2D)

With ~ 100 samples per dimension

Total of ~ 10¹²samples

Factorization Approach

6D Transport

~ 10¹²samples

~ 10⁸samples ~ 10⁸samples

4D Visibility 4D BRDF

*

=

Triple Product Integral Relighting Triple Product Integrals

(13)

Basis Requirements

1. Need few non-zero “tripling” coefficients

2. Need sparse basis coefficients

1. Number Non-Zero Tripling Coeffs

O ( N log N )

Haar Wavelets

O ( N

^{5 / 2}

)

Sph. Harmonics

O ( N

²

)

Fourier Series

O ( N )

Pixels

O ( N

³

)

General (e.g. PCA)

Number Non-Zero Basis Choice

Basis Requirements

1. Need few non-zero “tripling” coefficients

2. Need sparse basis coefficients

2. Sparsity in Light Approximation

Pixels

Wavelets

(14)

2. Sparsity in Visibility and BRDF

Visibility

~ 40% sparse in pixels

1 – 5 % sparse in wavelets

BRDF with diffuse component

~ 50% sparse in pixels

0.1 – 1% sparse in wavelets

Summary of Basis Analysis

Choose wavelet basis because

1. Few non-zero tripling coefficients

2. Sparse basis coefficients

Haar Tripling Coefficient Analysis

* *

Visual Review of 2D Haar Basis

(15)

*

=

1. Non-Overlapping Haar Multiplication

*

1. Zero Triple Product Integral

*

=

2. Co-Square Haar Multiplication

*

2. Non-Zero Triple Product Integrals

(16)

=

3. Overlapping Haar Multiplication

*

3. More Non-Zero Triple Integrals

*

Haar Tripling Coefficient Theorem

The integral of three Haar wavelets is non-zero iff

All three are the scaling function

All three are co-square and different

Two are identical, and the third overlaps at a coarser level

Theorem Consequences

1. Prove O(N log N) Haar sparsity

2. Derive O(N log N) triple product integral algorithm

Dynamic programming eliminates log N term

Final complexity is linear in number of retained basis coefficients

(17)

High-Level Algorithm: Precomputation

for each vertex

compute the visibility cubemap wavelet encode

store

for each viewing direction compute BRDF for that view nonlinear wavelet approximation store

for each vertex

compute the visibility cubemap wavelet encode

store

for each viewing direction compute BRDF for that view nonlinear wavelet approximation store

for each frame

wavelet encode lighting, L for each vertex in mesh

look up visibility, V look up BRDF for view, ρ integrate product of L, V, ρ set color of vertex

draw colored mesh

High-Level Algorithm: Relighting

for each frame

wavelet encode lighting, L for each vertex in mesh

look up visibility, V look up BRDF for view, ρ integrate product of L, V, ρ set color of vertex

draw colored mesh

Relit Images Changing Lighting, Fixed View

(18)

Changing View, Fixed Lighting Dynamic Lighting and View

Results

SIGGRAPH 2004 video

Summary

All-frequency relighting, changing view

Factor into visibility and BRDF

Fast relight in Haar basis

Triple product analysis and algorithms

Analysis of several bases

Haar tripling theorem

Efficient triple product integral estimation

(19)

Efficient Wavelet Rotation for Environment Map Rendering

Rui Wang Ren Ng David Luebke Greg Humphreys EGSR 2006

Triple product

) ( ω L

x

ωo

x

The output color of vertex x given view direction ωBRDF : Represents lighting reflective ratio of x at giveno

light direction ω to view direction ωo. Lighting : Represents the radiance of direction ω.

Visibility : Decides whether light is visible in direction ω.

) ( ω V

x

⁽ ⁾

~

^,

ω ρ

^x^ω^o

Triple product

∫

Ω

=

ω ω ρ

^ω

ω ω

ω L V d

B^x^, ^o ^x( ) ^x( )~^x^, ^o( )

ωo

x

The output radiance is the integral of these functions over all directions!

BRDF Lighting

Global frame

(20)

BRDF Lighting

Local frame

Rotation

Because we can’t rotate wavelet basis easily, some functions must be sampled with different rotations in advance and use lookup at runtime.

Spherical harmonics does not have this problem because it can be rotated easily.