SURE-based Optimization for Adaptive Sampling and Reconstruction

SURE-based Optimization for Adaptive Sampling and

Reconstruction

Tzu-Mao Li, Yu-Ting Wu, Yung-Yu Chuang

National Taiwan University

Monte Carlo ray tracing

� ( _{�, �} ) ₌ ∫

Ω

❑

� ( _{�, � , �} ) _{�� ≈} ¹

� ∑

�=1

�

� (� , � ,�

)

Depth of field Glossy reflection Global illumination

Soft shadow Participating media

Monte Carlo ray tracing

Motion blur

68 samples per pixel

~890 seconds

8192 samples per pixel

~1 day 4 hours

Noise of Monte Carlo

Avg. 64 samples per pixel

~890 seconds

8192 samples per pixel

~1 day 4 hours

Noise of Monte Carlo

• Image space methods

– Mitchell [1987, 1991], Bala et al. [2003], Overbeck et al. [2009], Chen et al. [2011], Rousselle et al. [2011], Sen and Darabi [2012]

– Handle general effects, usually more computationally efficient and use less memory

Previous work

Previous work

• Image space methods

– Mitchell [1987, 1991], Bala et al. [2003], Overbeck et al. [2009], Chen et al. [2011], Rousselle et al. [2011], Sen and Darabi [2012]

– Handle general effects, usually more computationally efficient and use less memory

• Multidimensional methods

– Hachisuka et al. [2008], Soler et al. [2009], Egan et al.

[2009, 2011], Lehtinen et al. [2011, 2012]

– May eliminate the noise using fewer samples, but often limited to a few effects

• Image space methods

– Mitchell [1987, 1991], Bala et al. [2003], Overbeck et al. [2009], Chen et al. [2011], Rousselle et al. [2011], Sen and Darabi [2012]

– Handle general effects, usually more computationally efficient and use less memory

• Multidimensional methods

– Hachisuka et al. [2008], Soler et al. [2009], Egan et al.

[2009, 2011], Lehtinen et al. [2011, 2012]

– May eliminate the noise using fewer samples, but often limited to a few effects

Previous work

+ Ours

GEM [Rousselle et al. 2011]

GEM [Rousselle et al. 2011]

Renderer

Mean Var

Color

GEM [Rousselle et al. 2011]

Renderer

Mean Var

Color Filterbank

GEM [Rousselle et al. 2011]

Renderer

Mean Var

Color Filterbank

GEM [Rousselle et al. 2011]

Renderer

Mean Var

Color Filterbank

GEM [Rousselle et al. 2011]

Renderer

Mean Var

Color Filterbank

GEM [Rousselle et al. 2011]

Renderer

Mean Var

Color Filterbank

Error Estimator

GEM [Rousselle et al. 2011]

Renderer

Mean Var

Color Filterbank

Error Estimator Filter

Scale

GEM [Rousselle et al. 2011]

Renderer

Mean Var

Color Filterbank

Error Estimator Filter

Scale Sample

Density

GEM [Rousselle et al. 2011]

Renderer

Mean Var

Color Filterbank

Error Estimator Filter

Scale Sample

Density Finish

?

No

Yes

GEM [Rousselle et al. 2011]

Renderer

Mean Var

Color Filterbank

Error Estimator Filter

Scale Sample

Density Finish

?

No

Yes

Isotropic

Filters Only!

Isotropic filters cannot adapt to scene features well!

GEM [Rousselle et al. 2011]

Bilateral filters

[Tomasi and Manduchi 1998]

• Preventing Gaussian filters from blurring image

edges.

Bilateral filters

[Tomasi and Manduchi 1998]

• Preventing Gaussian filters from blurring image edges.

�

=exp ⁡(− ‖ ^�

− �

‖

2 �

)

Position ^Noisy

Gaussian

Bilateral filters

[Tomasi and Manduchi 1998]

• Preventing Gaussian filters from blurring image edges.

�

=exp ⁡(− ‖ ^�

− �

‖

2 �

) exp ⁡(− ‖ ^�

− �

‖

2 �

)

Position ^Noisy

Bilateral

Color

Cross bilateral filters

[Eisemann and Durand 2004]

•  

Position Color Scene Features

Estimating filter error

Estimating filter error

Estimating filter error

�

Mean

Estimating filter error

�

_Variance

�

Mean

Estimating filter error

�

Converged

Variance Mean

�

�

Estimating filter error

Converged

Variance

�

Mean

�

�

Converged

Variance Mean

Estimating filter error

�(�

)

Filtered

�

�

�

Estimating filter error

Want to estimate:

�[( � ( ^�

) ^{− �}

)

]

Estimating filter error

Want to estimate:

�[( � ( ^�

) ^{− �}

)

]

Don’t know!

Estimating filter error

Want to estimate:

�[( � ( ^�

) ^{− �}

)

]= � [���� ( � (� ))]

Stein’s Unbiased Risk Estimator!

Estimating filter error

Want to estimate:

�[( � ( ^�

) ^{− �}

)

]= � [���� ( � (� ))]

Estimating filter error

Want to estimate:

Analytically derived for cross bilateral filters.

�[( � ( ^�

) ^{− �}

)

]= � [���� ( � (� ))]

Our framework

Renderer

SURE

MSE Estimator Filter

Scale Sample

Density Finish

?

No

Yes

Mean Var

Color

Our framework

Renderer

(Cross Bilateral) Filterbank

SURE

MSE Estimator Filter

Scale Sample

Density Finish

?

No

Yes

Mean Var

Color

Our framework

Renderer

Mean Var

Color (Cross

Bilateral) Filterbank

SURE

MSE Estimator Filter

Scale Sample

Density Finish

?

No

Yes

Scene Features

Var Mean

Variance of the error estimator

• has its own variance!

•  

Variance of the error estimator

• has its own variance!

• We need to filter the estimated error.

– We use a fixed size cross bilateral filter to filter the estimated error.

•  

Before Filter After Filter

Results

Results Monte Carlo – 82 spp (60sec)

Results Our – 40 spp (60 sec)

Results

• Compare with GEM [Rousselle et al. 2011] and

RPF [Sen and Darabi 2012]

Results

• Compare with GEM [Rousselle et al. 2011] and RPF [Sen and Darabi 2012]

– GEM selects from a discrete filter set similar to our algorithm, but is limited to isotropic filters.

Results

• Compare with GEM [Rousselle et al. 2011] and RPF [Sen and Darabi 2012]

– GEM selects from a discrete filter set similar to our algorithm, but is limited to isotropic filters.

– RPF is an advanced cross bilateral filter operating on Monte Carlo samples.

Comparison to GEM

GEM

Average 52 spp

~60 sec

Our

Average 40 spp

~60 sec

Reference 4096 spp

~3000 sec

Comparison to GEM

Reference 4096 spp

~3000 sec

Comparison to GEM

GEM

Average 40 spp

~140 sec

Our

Average 27 spp

~140 sec

Reference 4096 spp

~13000 sec

Comparison to GEM

Reference 4096 spp

~13000 sec

Comparison to RPF

RPF 8 spp

~370 sec

Our 8 spp

~40 sec

Reference 4096 spp

~5000 sec

Reference 4096 spp

~5000 sec RPF

8 spp

~370 sec

Comparison to RPF

Our

Avg 289 spp

~370 sec

Comparison to RPF

RPF 16 spp

~1700 sec

Our 16 spp

~270 sec

Reference 8192 spp

~100000 sec

RPF 16 spp

~1700 sec

Comparison to RPF

Our

Avg 133 spp

~1700 sec

Reference 8192 spp

~100000 sec

Gaussian filters

GEM Our

Filtered Image

Filter Selection Map

Non-local means filters

Single NLM filter Reference

Non-local means filters

Reference

Combining all NLM filters

Discussion

• Our method inherits most of the advantages and disadvantages of the image space methods.

– Advantages:

• It does not assume any specific effects

• The computation and memory complexity only relate to the number of pixels

Discussion

• Our method inherits most of the advantages and disadvantages of the image space methods.

– Advantages:

• It does not assume any specific effects

• The computation and memory complexity only relate to the number of pixels

– Disadvantages:

• Oversmoothing

• Not considering high dimensional information

Discussion

– Disadvantages:

• Oversmoothing

• Not considering high dimensional information

Image-space Methods Multidimensional Methods

Image from

[Hachisuka et al. 2008]

Conclusion and future work

• We have presented a SURE-based

Adaptive Sampling and Reconstruction framework.

– Allow to use almost arbitrary filters.

• Future Work

– Implementation on GPUs for interactive applications

– Animation rendering

– More rendering applications using SURE

Conclusion and future work

• Rousselle et al. [2012] introduces an advanced non-local means filter designed for rendering task.

• Incorporate SURE and scene features buffer into

their method?

Acknowledgements

• Anonymous reviewers

• Funding agencies: NSC and NTU

• Model creators: Pedro Caparros, RenderHere,  Unity Development, Tippy, Skipper25, jotijoti,  ShareCG.com, dogbite1066, Tiago Crisostomo,  Marko Dabrovic and Crytek GmbH, Marko 

Dabrovic and Mihovil Odak, Andrew Kensler.