A Wide Range of Options

Bilateral Filters

Digital Visual Effects, Spring 2008 Yung-Yu Chuang

2008/5/27

with slides by Fredo Durand, Ramesh Raskar, Sylvain Paris, Soonmin Bae

Announcements

• Final project proposal

• Project #3 artifacts voting

Bilateral filtering

[Ben Weiss, Siggraph 2006]

[Ben Weiss, Siggraph 2006]

Input

Input Log(Intensity) Log(Intensity) Bilateral Smoothing Bilateral Smoothing Gaussian

Gaussian Smoothing Smoothing

Image Denoising

A Wide Range of Options

• Diffusion, Bayesian, Wavelets…

– All have their pros and cons.

• Bilateral filter

– not always the best result [Buades 05] but often good – easy to understand, adapt and set up

Noisy input

Basic denoising

Median 5x5

Basic denoising

Noisy input Bilateral filter 7x7 window

Tone Mapping

[Durand 02]

Tone Mapping

[Durand 02]

Tone Mapping

[Durand 02]

Tone Mapping

[Durand 02]

output

Photographic Style Transfer

[Bae 06]

Photographic Style Transfer

[Bae 06]

Photographic Style Transfer

[Bae 06]

Photographic Style Transfer

[Bae 06]

output

Cartoon Rendition

[Winnemöller 06]

Cartoon Rendition

[Winnemöller 06]

Cartoon Rendition

[Winnemöller 06]

Cartoon Rendition

[Winnemöller 06]

output

6 papers at

SIGGRAPH’07 6 papers at

SIGGRAPH’07

Gaussian Blur

input

per-pixel multiplication

output

*

input

box average

Gaussian blur

normalized

Equation of Gaussian Blur

( )

∑

−

=

S

I G

I GB

q

q

p

|| p q ||

]

[

Same idea: weighted average of pixels.

unrelated pixels unrelated

pixels

uncertain pixels uncertain

pixels

related pixels

Gaussian Profile

pixel position pixel

weight

⎟⎟⎠

⎜⎜ ⎞

⎝

⎛−

= ²₂

exp 2 2

) 1

( σ π σ

σ

x x G

size of the window

Spatial Parameter

( )

∑

−

=

S

I G

I GB

q

q

p

|| p q ||

]

[

small σ large σ

input

How to set σ

• Depends on the application.

• Common strategy: proportional to image size

– e.g. 2% of the image diagonal

– property: independent of image resolution

Properties of Gaussian Blur

• Weights independent of spatial location

– linear convolution

– well-known operation

– efficient computation (recursive algorithm, FFT…)

Properties of Gaussian Blur

• Does smooth images

• But smoothes too much:

edges are blurred.

– Only spatial distance matters – No edge term

input

output

( )

∑

−

=

S

I G

I GB

q

q

p

|| p q ||

]

[

space

Blur Comes from Averaging across Edges

*

*

input output

Bilateral Filter No Averaging across Edges

*

*

*

input output

The kernel shape depends on the image content.

[Aurich 95, Smith 97, Tomasi 98]

not new new new

Bilateral Filter Definition

( ) ( )

∑

−

−

=

S

I I

I G

W G I

BF

q

q q

p p

p

1 || p q || | |

]

[

Same idea: weighted average of pixels.

Illustration a 1D Image

• 1D image = line of pixels

• Better visualized as a plot

pixel intensity

pixel position

space

Gaussian Blur and Bilateral Filter

Gaussian blur

Bilateral filter

[Aurich 95, Smith 97, Tomasi 98]

space p

q

∑

∈

−

=

S

I G

I GB

q

q

p ||p q ||

]

[ _σ

q q

p p

Bilateral Filter on a Height Field

output input

( ) ( )

∑

−

−

=

S

I I

I G

W G I

BF

q

q q

p p

p 1 || p q || | |

]

[ σs σr

p p

reproduced from [Durand 02]

Space and Range Parameters

• space σ_s : spatial extent of the kernel, size of the considered neighborhood.

( ) ( )

∑

−

−

=

S

I I

I G

W G I

BF

q

q q

p p

p

1 || p q || | |

]

[

Influence of Pixels

p p

Only pixels close in space and in range are considered.

space

range

σ_s= 2

σ = 6

σ_r = 0.1 σ_r = 0.25

σ_r = ∞

(Gaussian blur)

input

Exploring the Parameter Space Exploring the Parameter Space

σ_s= 2

σ_s= 6

σ_s= 18

σ_r = 0.1 σ_r = 0.25

σ_r = ∞

(Gaussian blur)

input

Varying the Range Parameter Varying the Range Parameter

input

σ

= 0.1

σ

= 0.25

σ

= ∞

(Gaussian blur)

σ_s= 2

σ = 6

σ_r = 0.1 σ_r = 0.25

σ_r = ∞

(Gaussian blur)

input

Varying the Space Parameter Varying the Space Parameter

input

σ

= 2

σ

= 6

σ

= 18

How to Set the Parameters

Depends on the application. For instance:

• space parameter: proportional to image size

– e.g., 2% of image diagonal

• range parameter: proportional to edge amplitude

– e.g., mean or median of image gradients

• independent of resolution and exposure

Iterating the Bilateral Filter

• Generate more piecewise-flat images

• Often not needed in computational photo, but

] [

) 1

(n

BF I

I

=

input

1 iteration

2 iterations

4 iterations

Advantages of Bilateral Filter

• Easy to understand

– Weighted mean of nearby pixels

• Easy to adapt

– Distance between pixel values

• Easy to set up

– Non-iterative

Hard to Compute

• Nonlinear

• Complex, spatially varying kernels

– Cannot be precomputed, no FFT…

( )

( )

∑

−

−

=

S

I I

I G

W G I

BF

q

q q

p p

p 1 || p q || | |

]

[ σs σr

But Bilateral Filter is Nonlinear

• Slow but some accelerations exist:

– [Elad 02]: Gauss-Seidel iterations

• Only for many iterations

– [Durand 02, Weiss 06]: fast approximation

• No formal understanding of accuracy versus speed

• [Weiss 06]: Only box function as spatial kernel

A Fast Approximation of the Bilateral Filter

using a Signal Processing Approach

Sylvain Paris and Frédo Durand

Definition of Bilateral Filter

• [Smith 97, Tomasi 98]

• Smoothes an image and preserves edges

• Weighted average of neighbors

• Weights

– Gaussian on space distance – Gaussian on range distance – sum to 1

space range

Input Result

Contributions

• Link with

linear filtering

•

Fast

accurate

Intuition on 1D Signal

BF

p

Intuition on 1D Signal

Weighted Average of Neighbors

• Near and similar pixels have influence.

weights applied to pixels

p

Link with Linear Filtering

1. Handling the Division

sum of weights

Handling the division with a

projective space

Formalization: Handling the Division

• Normalizing factor as homogeneous coordinate

• Multiply both sides by

• Normalizing factor as homogeneous coordinate

• Multiply both sides by

Formalization: Handling the Division

• Similar to homogeneous coordinates in projective space

• Division delayed until the end

• Next step: Adding a dimension to make a convolution appear

with W_q=1

p

Link with Linear Filtering

2. Introducing a Convolution

q

space: 1D Gaussian

× range: 1D Gaussian combination: 2D Gaussian space: 1D Gaussian

× range: 1D Gaussian combination: 2D Gaussian

p

Link with Linear Filtering

2. Introducing a Convolution

q

space: 1D Gaussian

× range: 1D Gaussian combination: 2D Gaussian space: 1D Gaussian

× range: 1D Gaussian combination: 2D Gaussian

space x range

Corresponds to a 3D Gaussian on a 2D image.

Link with Linear Filtering

2. Introducing a Convolution

black = zero sum all values

space-range Gaussian

result of the convolution

Link with Linear Filtering

2. Introducing a Convolution

Link with Linear Filtering

2. Introducing a Convolution

result of the convolution

higher dimensional functions

Gaussian convolution

division

slicing

w i w

Reformulation: Summary

1. Convolution in higher dimension

• expensive but well understood (linear, FFT, etc)

2. Division and slicing

higher dimensional functions

Gaussian convolution

division

slicing

Low-pass filter Low-pass filter

Almost only low freq.

High freq.

negligible Almost only

low freq.

High freq.

negligible

w i w

higher dimensional functions

Gaussian convolution

w i w

D O W N S A M P L E

Almost no information loss Almost no information loss

Fast Convolution by Downsampling

• Downsampling cuts frequencies above Nyquist limit

– Less data to process – But induces error

• Evaluation of the approximation

– Precision versus running time – Visual accuracy

Accuracy versus Running Time

• Finer sampling increases accuracy.

• More precise than previous work.

ampling

PSNR as function of Running Time

Digital photograph 1200 × 1600

input exact BF our result prev. work

1200 × 1600

• Comparison with previous work [Durand 02]

– running time = 1s for both techniques

0

difference 0.1

with exact computation (intensities in [0:1])

Visual Results

Conclusions

Practical gain

• Interactive running time

Theoretical gain

• Link with linear filters

higher dimension Ö “better” computation

higher dimension Ö “better” computation

Two-scale Tone Management for Photographic Look

Soonmin Bae, Sylvain Paris, and Frédo Durand MIT CSAIL

Ansel Adams

An Amateur Photographer

A Variety of Looks

Goals

• Control over photographic look

• Transfer “look” from a model photo

For example, we want

with the look of

Aspects of Photographic Look

• Subject choice

• Framing and composition Î Specified by input photos

• Tone distribution and contrast ÎModified based on model photos

Input

Tonal Aspects of Look

Ansel Adams Kenro Izu

Tonal aspects of Look - Global Contrast

Tonal aspects of Look - Local Contrast

Variable amount of texture Texture everywhere

Ansel Adams Kenro Izu

Overview

Model

Overview

Local contrast Global contrast

Result

• Separate global and local contrast

Input Image

Split

Global contrast

Input Image

Careful

combination

Post- process

Overview

Split

Global contrast

Input Image

Result

Careful

combination

Post- process

Overview

Local contrast

Split Global vs. Local Contrast

• Naïve decomposition: low vs. high frequency

– Problem: introduce blur & halos

Halo Blur

Bilateral Filter

• Edge-preserving smoothing [Tomasi 98]

• We build upon tone mapping [Durand 02]

After bilateral filtering Residual after filtering

Global contrast Local contrast

Bilateral Filter

• Edge-preserving smoothing [Tomasi 98]

• We build upon tone mapping [Durand 02]

BASE layer DETAIL layer

Global contrast

Input Image

Result

Careful

combination

Post- process Bilateral

Filter

Local contrast

Global contrast

Input Image

Careful

combination

Post- process Bilateral

Filter

Global Contrast

• Intensity remapping of base layer

Input base Input intensity After remapping Remapped

intensity

Global Contrast (Model Transfer)

• Histogram matching

– Remapping function given input and model histogram Model

base

Local contrast Global contrast

Input Image

Result

Careful

combination

Post- process Bilateral

Filter

Intensity matching

Global contrast

Input Image

Careful

combination

Post- process Bilateral

Filter

Intensity matching

Local Contrast: Detail Layer

• Uniform control:

– Multiply all values in the detail layer

Input Base + 3 × Detail

The amount of local contrast is not uniform

Smooth region

Local Contrast Variation

• We define “textureness”: amount of local contrast

– at each pixel based on surrounding region

Smooth region

Ö Low textureness

Textured region

Ö High textureness

“Textureness”: 1D Example

Textured region

Ö High textureness Smooth region

Ö Small high-frequency Textured region

Ö Large high-frequency Previous work:

Low pass of |H|

[Li 05, Su 05]

Low pass of |H|

[Li 05, Su 05]

Textureness

Input Textureness

Textureness Transfer

Step 1:

Histogram transfer

Hist. transfer Input

Input textureness textureness

Desired Desired textureness textureness Model

Model textureness textureness

Local contrast Global contrast

Input Image

Result

Careful

combination

Post- process Bilateral

Filter

Intensity matching

Textureness matching

Global contrast

Input Image

Careful

combination

Post- process Bilateral

Filter

Intensity matching

Textureness

A Non Perfect Result

• Decoupled and large modifications (up to 6x)

ÎLimited defects may appear

input (HDR)

result after

global and local adjustments

Intensity Remapping

• Some intensities may be outside displayable range.

Î Compress histogram to fit visible range.

Preserving Details

1. In the gradient domain:

– Compare gradient amplitudes of input and current – Prevent extreme reduction & extreme increase

2. Solve the Poisson equation.

corrected result remapped

intensities initial

result

Effect of Detail Preservation

uncorrected result corrected result

Local contrast Global contrast

Input Image

Result

Post- process Bilateral

Filter

Intensity matching

Textureness matching

Constrained Poisson

Global contrast

Input Image

Bilateral Filter

Intensity matching

Textureness

Constrained Poisson

Post- process

Additional Effects

• Soft focus (high frequency manipulation)

• Film grain (texture synthesis [Heeger 95])

• Color toning (chrominance = f (luminance))

before effects

after effects model

Intensity matching

Bilateral Filter

Global contrast

Input Image

Textureness

Constrained Poisson

Soft focus Toning

Grain

Intensity matching

Bilateral Filter

Local contrast Global contrast

Input Image

Result

Textureness matching

Constrained Poisson

Soft focus Toning

Grain

Recap

Results

User provides input and model photographs.

Î Our system automatically produces the result.

Running times:

– 6 seconds for 1 MPixel or less

Input Model

Result

Input

Result

Input Model

Result

Input

Input ModelModel

Snapshot

Snapshot, Alfred Stieglitz, Alfred Stieglitz

Comparison with Naïve Histogram Matching

Input Input

Our Result Our Result

Model Model

Clearing Winter Storm, Ansel Adams

Histogram Matching Histogram Matching

Comparison with Naïve Histogram Matching

Local contrast too low

Color Images

• Lab color space: modify only luminance

Input

Input OutputOutput

Limitations

• Noise and JPEG artifacts

– amplified defects

• Can lead to unexpected

results if the image content is too different from the model

– Portraits, in particular, can suffer

Conclusions

• Transfer “look” from a model photo

• Two-scale tone management

– Global and local contrast

– New edge-preserving textureness

Video Enhancement Using

Per Pixel Exposures (Bennett, 06)

From this video:

ASTA: Adaptive Spatio-S TemporalT

Accumulation Filter

Joint bilateral filtering

Flash / No-Flash Photo Improvement (Petschnigg04) (Eisemann04)

Merge best features: warm, cozy candle light (no-flash) low-noise, detailed flash image

Overview

Remove noise + details from image A,

Keep as image A Lighting

--- No-flash

Basic approach of both flash/noflash papers

Petschnigg:

• Flash

Petschnigg:

• No Flash,

Petschnigg:

• Result

Our Approach

Registration Registration

Our Approach

Decomposition

Decomposition

Color / Intensity:

= *

Our Approach

Decomposition

Our Approach

Decoupling

Decoupling

• Lighting : Large-scale variation

• Texture : Small-scale variation

Texture Lighting

• Lighting : Large-scale variation

• Texture : Small-scale variation

Lighting

Large-scale Layer

• Bilateral filter – edge preserving filter

Smith and Brady 1997; Tomasi and Manducci 1998; Durand et al. 2002

Large-scale Layer

• Bilateral filter

Cross Bilateral Filter

• Similar to joint bilateral filter by Petschnigg et al.

• When no-flash image is too noisy

• Borrow similarity from flash image

¾ edge stopping from flash image

• See detail in paper

Detail Layer

Intensity Large-scale

/ =

Detail

Recombination: Large scale * Detail = Intensity

Recombination

* =

Recombination

Intensity Result

Color Flash

* ~ ~

Result

Recombination: Intensity * Color = Original

shadows

Our Approach

Our Approach

Shadow

Detection/Treatment

Results

No-flash

Joint bilateral upsampling

Joint bilateral upsampling

Joint bilateral upsampling

Joint bilateral upsampling

Joint bilateral upsampling

Joint bilateral upsampling

Joint bilateral upsampling

Joint bilateral upsampling

References

• Patrick Perez, Michel Gangnet, Andrew Blake, Poisson Image Editing, SIGGRAPH 2003.

• Dani Lischinski, Zeev Farbman, Matt Uytendaelle and Richard

Szeliski. Interactive Local Adjustment of Tonal Values. SIGGRAPH 2006.

• Carsten Rother, Andrew Blake, Vladimir Kolmogorov, GrabCut - Interactive Foreground Extraction Using Iterated Graph Cuts, SIGGRAPH 2004.

• Aseem Agarwala, Mira Dontcheva, Maneesh Agrawala, Steven

Drucker, Alex Colburn, Brian Curless, David H. Salesin, Michael F.

Cohen, Interactive Digital Photomontage, SIGGRAPH 2004.

• Sylvain Paris and Fredo Durand. A Fast Approximation of the Bilateral Filter using a Signal Processing Approach. ECCV 2006.

• Soonmin Bae, Sylvain Paris and Fredo Durand. Two-scale Tone Management for Photographic Look. SIGGRAPH 2006.