### Bilateral Filters

Digital Visual Effects, Spring 2008
*Yung-Yu Chuang*

2008/5/27

*with slides by Fredo Durand, Ramesh Raskar, Sylvain Paris, Soonmin Bae*

**Announcements**

• Final project proposal

• Project #3 artifacts voting

**Bilateral filtering**

[Ben Weiss, Siggraph 2006]

[Ben Weiss, Siggraph 2006]

Input

Input Log(Intensity) Log(Intensity) Bilateral Smoothing Bilateral Smoothing Gaussian

Gaussian Smoothing Smoothing

**Image Denoising**

**A Wide Range of Options**

### • Diffusion, Bayesian, Wavelets…

– All have their pros and cons.

### • Bilateral filter

– not always the best result [Buades 05] but often good – easy to understand, adapt and set up

Noisy input

**Basic denoising**

Median 5x5

**Basic denoising**

Noisy input Bilateral filter 7x7 window

**Tone Mapping**

[Durand 02]

**Tone Mapping**

[Durand 02]

**Tone Mapping**

[Durand 02]

**Tone Mapping**

[Durand 02]

output

**Photographic Style Transfer**

[Bae 06]

**Photographic Style Transfer**

[Bae 06]

**Photographic Style Transfer**

[Bae 06]

**Photographic Style Transfer**

[Bae 06]

output

**Cartoon Rendition**

[Winnemöller 06]

**Cartoon Rendition**

[Winnemöller 06]

**Cartoon Rendition**

[Winnemöller 06]

**Cartoon Rendition**

[Winnemöller 06]

output

**6 papers at**

**SIGGRAPH’07** **6 papers at**

**SIGGRAPH’07**

**Gaussian Blur**

input

per-pixel multiplication

output

### *

**input**

**box average**

**Gaussian blur**

normalized

**Equation of Gaussian Blur**

### ( )

### ∑

∈### −

### =

*S*

*I* *G*

*I* *GB*

**q**

**q**

**p**

### || **p** **q** ||

### ]

### [

_{σ}

**Same idea: weighted average of pixels.**

unrelated pixels unrelated

pixels

uncertain pixels uncertain

pixels

related pixels

**Gaussian Profile**

pixel position pixel

weight

⎟⎟⎠

⎜⎜ ⎞

⎝

⎛−

= ^{2}_{2}

exp 2 2

) 1

( σ π σ

σ

*x* *x*
*G*

size of the window

**Spatial Parameter**

### ( )

### ∑

∈### −

### =

*S*

*I* *G*

*I* *GB*

**q**

**q**

**p**

### || **p** **q** ||

### ]

### [

_{σ}

small σ large σ

input

**How to set ** σ

• Depends on the application.

• Common strategy: proportional to image size

– e.g. 2% of the image diagonal

– property: independent of image resolution

**Properties of Gaussian Blur**

• Weights independent of spatial location

– linear convolution

– well-known operation

– efficient computation (recursive algorithm, FFT…)

**Properties of Gaussian Blur**

• Does smooth images

• But smoothes too much:

**edges are blurred.**

– Only spatial distance matters – No edge term

input

output

### ( )

### ∑

∈### −

### =

*S*

*I* *G*

*I* *GB*

**q**

**q**

**p**

### || **p** **q** ||

### ]

### [

_{σ}

space

**Blur Comes from Averaging across Edges**

### *

### *

input output

**Bilateral Filter No Averaging across Edges**

### *

### *

### *

input output

The kernel shape depends on the image content.

[Aurich 95, Smith 97, Tomasi 98]

not new new new

**Bilateral Filter Definition**

### ( ) ( )

### ∑

∈### −

### −

### =

*S*

*I* *I*

*I* *G*

*W* *G* *I*

*BF*

**q**

**q**
**q**

**p**
**p**

**p**

### 1 || **p** **q** || | |

### ]

### [

σs σ r**Same idea: weighted average of pixels.**

**Illustration a 1D Image**

• 1D image = line of pixels

• Better visualized as a plot

**pixel**
**intensity**

**pixel position**

space

**Gaussian Blur and Bilateral Filter**

Gaussian blur

Bilateral filter

[Aurich 95, Smith 97, Tomasi 98]

space
**p**

**q**

### ∑

( )∈

−

=

*S*

*I*
*G*

*I*
*GB*

**q**

**q**

**p** ||**p** **q** ||

]

[ _{σ}

**q** **q**

**p** **p**

**Bilateral Filter on a Height Field**

output input

### ( ) ( )

### ∑

∈−

−

=

*S*

*I*
*I*

*I*
*G*

*W* *G*
*I*

*BF*

**q**

**q**
**q**

**p**
**p**

**p** 1 || **p** **q** || | |

]

[ σs σr

**p** **p**

*reproduced*
*from [Durand 02]*

**Space and Range Parameters**

• space σ_{s} : spatial extent of the kernel, size of
the considered neighborhood.

### ( ) ( )

### ∑

∈### −

### −

### =

*S*

*I* *I*

*I* *G*

*W* *G* *I*

*BF*

**q**

**q**
**q**

**p**
**p**

**p**

### 1 || **p** **q** || | |

### ]

### [

σs σr**Influence of Pixels**

**p** **p**

Only pixels close in space and in range are considered.

space

range

σ_{s}= 2

σ = 6

σ_{r} = 0.1 σ_{r} = 0.25

σ_{r} = ∞

(Gaussian blur)

input

**Exploring the Parameter Space**
**Exploring the Parameter Space**

σ_{s}= 2

σ_{s}= 6

σ_{s}= 18

σ_{r} = 0.1 σ_{r} = 0.25

σ_{r} = ∞

(Gaussian blur)

input

**Varying the Range Parameter**
**Varying the Range Parameter**

**input**

### σ

_{r}**= 0.1**

### σ

_{r}**= 0.25**

### σ

_{r}**= ∞**

**(Gaussian blur)**

σ_{s}= 2

σ = 6

σ_{r} = 0.1 σ_{r} = 0.25

σ_{r} = ∞

(Gaussian blur)

input

**Varying the Space Parameter**
**Varying the Space Parameter**

**input**

### σ

_{s}**= 2**

### σ

_{s}**= 6**

### σ

_{s}**= 18**

**How to Set the Parameters**

Depends on the application. For instance:

• space parameter: proportional to image size

– e.g., 2% of image diagonal

• range parameter: proportional to edge amplitude

– e.g., mean or median of image gradients

• independent of resolution and exposure

**Iterating the Bilateral Filter**

• Generate more piecewise-flat images

• Often not needed in computational photo, but

### ] [

_{(}

_{)}

) 1

(*n*

*BF* *I*

*n*

*I*

_{+}

### =

**input**

**1 iteration**

**2 iterations**

**4 iterations**

**Advantages of Bilateral Filter**

• Easy to understand

– Weighted mean of nearby pixels

• Easy to adapt

– Distance between pixel values

• Easy to set up

– Non-iterative

**Hard to Compute**

• Nonlinear

• Complex, spatially varying kernels

– Cannot be precomputed, no FFT…

( )

### ( )

### ∑

∈−

−

=

*S*

*I*
*I*

*I*
*G*

*W* *G*
*I*

*BF*

**q**

**q**
**q**

**p**
**p**

**p** 1 || **p** **q** || | |

]

[ σs σr

**But Bilateral Filter is Nonlinear**

• Slow but some accelerations exist:

– [Elad 02]: Gauss-Seidel iterations

**• Only for many iterations**

– [Durand 02, Weiss 06]: fast approximation

**• No formal understanding of accuracy versus speed**

**• [Weiss 06]: Only box function as spatial kernel**

### A Fast Approximation of the Bilateral Filter

### using a Signal Processing Approach

Sylvain Paris and Frédo Durand

**Definition of Bilateral Filter**

• [Smith 97, Tomasi 98]

• Smoothes an image and preserves edges

• Weighted average of neighbors

• Weights

*– Gaussian on space distance*
*– Gaussian on range distance*
– sum to 1

space range

Input Result

**Contributions**

• Link with

**linear filtering**

•

**Fast **

and **accurate**

approximation
**Intuition on 1D Signal**

BF

**p**

**Intuition on 1D Signal**

**Weighted Average of Neighbors**

• Near and similar pixels have influence.

weights applied to pixels

**p**

**Link with Linear Filtering**

**1. Handling the Division**

sum of weights

Handling the division with a

**projective space**

^{.}

**Formalization: Handling the Division**

• Normalizing factor as homogeneous coordinate

• Multiply both sides by

• Normalizing factor as homogeneous coordinate

• Multiply both sides by

**Formalization: Handling the Division**

• Similar to homogeneous coordinates in projective space

• Division delayed until the end

• Next step: Adding a dimension to make a convolution appear

*with W*** _{q}**=1

**p**

**Link with Linear Filtering**

**2. Introducing a Convolution**

**q**

space: 1D Gaussian

× range: 1D Gaussian combination: 2D Gaussian space: 1D Gaussian

× range: 1D Gaussian combination: 2D Gaussian

**p**

**Link with Linear Filtering**

**2. Introducing a Convolution**

**q**

space: 1D Gaussian

× range: 1D Gaussian combination: 2D Gaussian space: 1D Gaussian

× range: 1D Gaussian combination: 2D Gaussian

space x range

Corresponds to a 3D Gaussian on a 2D image.

**Link with Linear Filtering**

**2. Introducing a Convolution**

black = zero sum all values

space-range Gaussian

result of the convolution

**Link with Linear Filtering**

**2. Introducing a Convolution**

**Link with Linear Filtering**

**2. Introducing a Convolution**

result of the convolution

higher dimensional functions

Gaussian convolution

division

slicing

*w i* *w*

**Reformulation: Summary**

1. Convolution in higher dimension

• expensive but well understood (linear, FFT, etc)

2. Division and slicing

higher dimensional functions

Gaussian convolution

division

slicing

Low-pass filter Low-pass filter

Almost only low freq.

High freq.

negligible Almost only

low freq.

High freq.

negligible

*w i* *w*

higher dimensional functions

Gaussian convolution

*w i* *w*

**D O W N S A M P L E**

Almost no information loss Almost no information loss

**Fast Convolution by Downsampling**

### • Downsampling cuts frequencies above Nyquist limit

– Less data to process – But induces error

### • Evaluation of the approximation

– Precision versus running time – Visual accuracy

**Accuracy versus Running Time**

• Finer sampling increases accuracy.

• More precise than previous work.

ampling

**PSNR as function of Running Time**

Digital photograph 1200 × 1600

input exact BF our result prev. work

1200 × 1600

• Comparison with previous work [Durand 02]

– running time = 1s for both techniques

0

difference 0.1

with exact computation (intensities in [0:1])

**Visual Results**

**Conclusions**

**Practical gain**

• Interactive running time

**Theoretical gain**

• Link with linear filters

### higher dimension Ö “better” computation

### higher dimension Ö “better” computation

**Two-scale Tone Management ** **for Photographic Look**

Soonmin Bae, Sylvain Paris, and Frédo Durand MIT CSAIL

**Ansel Adams**

**An Amateur Photographer**

**A Variety of Looks**

**Goals**

• Control over photographic look

• Transfer “look” from a model photo

For example, we want

with the look of

**Aspects of Photographic Look**

• Subject choice

• Framing and composition Î Specified by input photos

• Tone distribution and contrast ÎModified based on model photos

Input

**Tonal Aspects of Look**

Ansel Adams Kenro Izu

**Tonal aspects of Look - Global Contrast**

**Tonal aspects of Look - Local Contrast**

**Variable amount of texture** **Texture everywhere**

Ansel Adams Kenro Izu

**Overview**

**Model**

**Overview**

**Local contrast****Global contrast**

**Result**

• Separate global and local contrast

**Input****Image**

**Split**

**Global contrast**

**Input****Image**

**Careful**

**combination**

**Post-**
**process**

**Overview**

**Split**

**Global contrast**

**Input****Image**

**Result**

**Careful**

**combination**

**Post-**
**process**

**Overview**

**Local contrast**

**Split Global vs. Local Contrast**

• Naïve decomposition: low vs. high frequency

– Problem: introduce blur & halos

Halo Blur

**Bilateral Filter**

• Edge-preserving smoothing [Tomasi 98]

• We build upon tone mapping [Durand 02]

**After bilateral filtering** **Residual after filtering**

**Global contrast****Local contrast**

**Bilateral Filter**

• Edge-preserving smoothing [Tomasi 98]

• We build upon tone mapping [Durand 02]

**BASE layer****DETAIL layer**

**Global contrast**

**Input****Image**

**Result**

**Careful**

**combination**

**Post-**
**process**
**Bilateral**

**Filter**

**Local contrast**

**Global contrast**

**Global contrast**

**Input****Image**

**Careful**

**combination**

**Post-**
**process**
**Bilateral**

**Filter**

**Global Contrast**

• Intensity remapping of base layer

Input base Input intensity After remapping Remapped

intensity

**Global Contrast (Model Transfer)**

• Histogram matching

– Remapping function given input and model histogram Model

base

**Local contrast****Global contrast**

**Input****Image**

**Result**

**Careful**

**combination**

**Post-**
**process**
**Bilateral**

**Filter**

**Intensity**
**matching**

**Global contrast**

**Input****Image**

**Careful**

**combination**

**Post-**
**process**
**Bilateral**

**Filter**

**Intensity**
**matching**

**Local Contrast: Detail Layer**

• Uniform control:

– Multiply all values in the detail layer

Input Base + 3 × Detail

**The amount of local contrast ** **is not uniform**

Smooth region

**Local Contrast Variation**

• We define “textureness”: amount of local contrast

– at each pixel based on surrounding region

Smooth region

Ö Low textureness

Textured region

Ö High textureness

**“Textureness”: 1D Example**

Textured region

Ö High textureness Smooth region

Ö Small high-frequency Textured region

Ö Large high-frequency Previous work:

Low pass of |H|

[Li 05, Su 05]

Low pass of |H|

[Li 05, Su 05]

**Textureness**

Input Textureness

**Textureness Transfer**

Step 1:

Histogram transfer

**Hist. transfer**
**Input **

**Input **
**textureness**
**textureness**

**Desired **
**Desired **
**textureness**
**textureness**
**Model **

**Model **
**textureness**
**textureness**

**Local contrast****Global contrast**

**Input****Image**

**Result**

**Careful**

**combination**

**Post-**
**process**
**Bilateral**

**Filter**

**Intensity**
**matching**

**Textureness**
**matching**

**Global contrast**

**Input****Image**

**Careful**

**combination**

**Post-**
**process**
**Bilateral**

**Filter**

**Intensity**
**matching**

**Textureness**

**A Non Perfect Result**

• Decoupled and large modifications (up to 6x)

ÎLimited defects may appear

input (HDR)

result after

global and local adjustments

**Intensity Remapping**

• Some intensities may be outside displayable range.

Î Compress histogram to fit visible range.

**Preserving Details**

**1. In the gradient domain:**

– Compare gradient amplitudes of input and current – Prevent extreme reduction & extreme increase

**2. Solve the Poisson equation.**

corrected result remapped

intensities initial

result

**Effect of Detail Preservation**

uncorrected result corrected result

**Local contrast****Global contrast**

**Input****Image**

**Result**

**Post-**
**process**
**Bilateral**

**Filter**

**Intensity**
**matching**

**Textureness**
**matching**

**Constrained**
**Poisson**

**Global contrast**

**Input****Image**

**Bilateral**
**Filter**

**Intensity**
**matching**

**Textureness**

**Constrained**
**Poisson**

**Post-**
**process**

**Additional Effects**

**• Soft focus (**high frequency manipulation)

**• Film grain (**texture synthesis [Heeger 95])

**• Color toning (***chrominance = f (luminance)*)

before effects

after effects model

**Intensity**
**matching**

**Bilateral**
**Filter**

**Global contrast**

**Input****Image**

**Textureness**

**Constrained**
**Poisson**

**Soft focus**
**Toning**

**Grain**

**Intensity**
**matching**

**Bilateral**
**Filter**

**Local contrast****Global contrast**

**Input****Image**

**Result**

**Textureness**
**matching**

**Constrained**
**Poisson**

**Soft focus**
**Toning**

**Grain**

**Recap**

**Results**

User provides input and model photographs.

Î **Our system automatically produces the result.**

Running times:

– 6 seconds for 1 MPixel or less

**Input** **Model**

**Result**

**Input**

**Result**

**Input** **Model**

**Result**

**Input**

**Input** **Model****Model**

*Snapshot*

*Snapshot, Alfred Stieglitz*, Alfred Stieglitz

**Comparison with Naïve Histogram Matching**

**Input**
**Input**

**Our Result**
**Our Result**

**Model**
**Model**

**Clearing Winter Storm, Ansel****Adams**

**Histogram Matching**
**Histogram Matching**

**Comparison with Naïve Histogram Matching**

Local contrast too low

**Color Images**

**• Lab color space: modify only luminance**

**Input**

**Input** **Output****Output**

**Limitations**

• Noise and JPEG artifacts

– amplified defects

• Can lead to unexpected

results if the image content is too different from the model

– Portraits, in particular, can suffer

**Conclusions**

• Transfer “look” from a model photo

• Two-scale tone management

– Global and local contrast

– New edge-preserving textureness

**Video Enhancement Using **

**Per Pixel Exposures (Bennett, 06)**

From this video:

**ASTA: Adaptive ****Spatio-****S****Temporal****T**

**Accumulation Filter**

**Joint bilateral filtering**

**Flash / No-Flash Photo Improvement** **(Petschnigg04) (Eisemann04)**

Merge best features: warm, cozy candle light (no-flash) low-noise, detailed flash image

**Overview**

Remove noise + details from image A,

Keep as image A Lighting

--- No-flash

Basic approach of both flash/noflash papers

**Petschnigg:**

• Flash

**Petschnigg:**

• No Flash,

**Petschnigg:**

• Result

**Our Approach**

Registration Registration

**Our Approach**

Decomposition

**Decomposition**

### Color / Intensity:

### = *

**Our Approach**

Decomposition

**Our Approach**

Decoupling

**Decoupling**

• Lighting : Large-scale variation

• Texture : Small-scale variation

Texture Lighting

• Lighting : Large-scale variation

• Texture : Small-scale variation

Lighting

**Large-scale Layer**

### • Bilateral filter – edge preserving filter

Smith and Brady 1997; Tomasi and Manducci 1998; Durand et al. 2002

**Large-scale Layer**

### • Bilateral filter

**Cross Bilateral Filter**

• Similar to joint bilateral filter by Petschnigg et al.

• When no-flash image is too noisy

• Borrow similarity from flash image

### ¾ edge stopping from flash image

• See detail in paper

**Detail Layer**

Intensity Large-scale

**/** =

Detail

### Recombination: Large scale * Detail = Intensity

**Recombination**

***** =

**Recombination**

Intensity Result

Color Flash

### * ~ ~

Result

### Recombination: Intensity * Color = Original

### shadows

**Our Approach**

**Our Approach**

Shadow

Detection/Treatment

**Results**

No-flash

**Joint bilateral upsampling**

**Joint bilateral upsampling**

**Joint bilateral upsampling**

**Joint bilateral upsampling**

**Joint bilateral upsampling**

**Joint bilateral upsampling**

**Joint bilateral upsampling**

**Joint bilateral upsampling**

**References**

• Patrick Perez, Michel Gangnet, Andrew Blake, Poisson Image Editing, SIGGRAPH 2003.

• Dani Lischinski, Zeev Farbman, Matt Uytendaelle and Richard

Szeliski. Interactive Local Adjustment of Tonal Values. SIGGRAPH 2006.

• Carsten Rother, Andrew Blake, Vladimir Kolmogorov, GrabCut - Interactive Foreground Extraction Using Iterated Graph Cuts, SIGGRAPH 2004.

• Aseem Agarwala, Mira Dontcheva, Maneesh Agrawala, Steven

Drucker, Alex Colburn, Brian Curless, David H. Salesin, Michael F.

Cohen, Interactive Digital Photomontage, SIGGRAPH 2004.

• Sylvain Paris and Fredo Durand. A Fast Approximation of the Bilateral Filter using a Signal Processing Approach. ECCV 2006.

• Soonmin Bae, Sylvain Paris and Fredo Durand. Two-scale Tone Management for Photographic Look. SIGGRAPH 2006.