Bilateral filtering

Bilateral Filters

Digital Visual Effectsg Yung-Yu Chuang

with slides by Fredo Durand, Ramesh Raskar, Sylvain Paris, Soonmin Bae

Bilateral filtering

Input

Input Log(Intensity) Log(Intensity) Bilateral Smoothing Bilateral Smoothing Gaussian

Gaussian Smoothing Smoothing

[Ben Weiss, Siggraph 2006]

[Ben Weiss, Siggraph 2006]

Image Denoising g g

noisy image naïve denoising Gaussian blur

better denoising edge preserving filter Gaussian blur edge-preserving filter

Smoothing an image without blurring its edges Smoothing an image without blurring its edges.

A Wide Range of Options g p

• Diffusion, Bayesian, Wavelets… Diffusion, Bayesian, Wavelets…

– All have their pros and cons.

Bil t l filt

• Bilateral filter

– not always the best result [Buades 05] but often good – easy to understand, adapt and set up

Basic denoising

Noisy input Median 5x5

Basic denoising

Noisy input Bilateral filter 7x7 window

Tone Mapping

[D d 02]

Tone Mapping

[D d 02]

[Durand 02]

[Durand 02]

HDR input

Tone Mapping

[D d 02]

Tone Mapping

[D d 02]

[Durand 02]

[Durand 02]

output

Photographic Style Transfer

[B 06]

Photographic Style Transfer

[B 06]

[Bae 06]

[Bae 06]

input

Photographic Style Transfer

[B 06]

Photographic Style Transfer

[B 06]

[Bae 06]

[Bae 06]

output

Cartoon Rendition

[Winnemöller 06]

Cartoon Rendition

[Winnemöller 06]

[Winnemöller 06]

[Winnemöller 06]

input

Cartoon Rendition

[Winnemöller 06]

Cartoon Rendition

[Winnemöller 06]

[Winnemöller 06]

[Winnemöller 06]

6 papers at 6 papers at p p SIGGRAPH’07

p p

SIGGRAPH’07 SIGGRAPH 07 SIGGRAPH 07

output

Gaussian Blur

per-pixel multiplication

input

*

average average

input

box average Gaussian blur

Equation of Gaussian Blur Equation of Gaussian Blur

Same idea: weighted average of pixels.

 

 ^

 G I

I

GB [ ]

|| p q ||

normalized

S q

normalized Gaussian function 1

0

Gaussian Profile

Gaussian Profile

pixel



 2 ² exp

2 )

(   

 x G

weight

pixel position unrelated

pixels unrelated

pixels

uncertain pixels uncertain

pixels

related pixels

Spatial Parameter Spatial Parameter

 

 ^{ }







S

I G

I GB

q

q

p

|| p q ||

]

[

size of the window

S q

small large

small  large 

limited smoothing strong smoothing

How to set 

• Depends on the application.

• Common strategy: proportional to image size

– e.g. 2% of the image diagonal

– property: independent of image resolution

Properties of Gaussian Blur

• Weights independent of spatial location

– linear convolution

– well-known operation

– efficient computation (recursive algorithm, FFT…)

Properties of Gaussian Blur

• Does smooth images B h h

input

• But smoothes too much:

edges are blurred.

– Only spatial distance matters – No edge term

output

 







S

I G

I GB

q

q

p

|| p q ||

]

[

space

S

q space

Blur Comes from Averaging across Edges

*

input output

*

*

S G i k l h

Same Gaussian kernel everywhere.

Bilateral Filter No Averaging across Edges

*

input output

*

*

Th k l h d d th i t t

The kernel shape depends on the image content.

Bilateral Filter Definition Bilateral Filter Definition

not new new

new

Same idea: weighted average of pixels.

   





S

I I I G W G

I BF

q

q q p

p 1 ||p q|| | |

]

[ s r

i ht i ht

li ti

W_{p q}S

space weight range weight I

normalization factor

Illustration a 1D Image

• 1D image = line of pixelsg p

• Better visualized as a plot

pixel intensity

pixel position pixel position

Gaussian Blur and Bilateral Filter

Gaussian blur

p

q ^{GB[I]p }



space space

q



S q

q

p 

Bilateral filter

[Aurich 95 Smith 97 Tomasi 98]

p

   

^{G G I I I}

I

BF 1 || || | |

] [ [Aurich 95, Smith 97, Tomasi 98]

rap

space range normalization

   









S

I I I G W G

I BF

q

q q p p

p 1 ||p q|| | |

]

[ s r

ange

q

space

Bilateral Filter on a Height Field

 

 



 G G I I I

I W

BF _p 1 ||p q|| | _{p q} | _q

]

[



W_{p q}S ^{p q q}

p s r

output input

reproduced from [Durand 02]

Space and Range Parameters

1

_    









S

I I I G W G

I BF

q

q q p p

p 1 ||p q|| | |

]

[ s r

• space  : spatial extent of the kernel size of

p q

• space _s : spatial extent of the kernel, size of the considered neighborhood.

• range _r : “minimum” amplitude of an edge

Influence of Pixels

Only pixels close in space and in range are considered.

space

range range

p p

 = 0 1  = 0 25

_r ^{= }

(G i bl )

Exploring the Parameter Space Exploring the Parameter Space

_r = 0.1 _r = 0.25 (Gaussian blur)

_s = 2

input

_s = 6

_s = 18

 = 0 1  = 0 25

_r ^{= }

(G i bl )

Varying the Range Parameter Varying the Range Parameter

_r = 0.1 _r = 0.25 (Gaussian blur)

_s = 2

input

_s = 6

_s = 18

input



= 0.1



= 0.25 

= 

(G i bl )

(Gaussian blur)

 = 0 1  = 0 25

_r = 

(G i bl )

Varying the Space Parameter Varying the Space Parameter

_r = 0.1 _r = 0.25 (Gaussian blur)

_s = 2

input

_s = 6

_s = 18

input



= 2 

= 6



= 18 How to Set the Parameters

Depends on the application. For instance:

• space parameter: proportional to image size

– e.g., 2% of image diagonal

• range parameter: proportional to edge amplitude

– e.g., mean or median of image gradients

• independent of resolution and exposure

Iterating the Bilateral Filter

] [I BF

I

 BF [ I

] I



• Generate more piecewise-flat images

• Often not needed in computational photo, but could be useful for applications such as NPR.pp

input

1 iteration 2 iterations

4 iterations

Advantages of Bilateral Filter

• Easy to understand

W i ht d f b i l – Weighted mean of nearby pixels

• Easy to adapt

– Distance between pixel values

• Easy to set upEasy to set up

– Non-iterative

Hard to Compute

• Nonlinear _{  }

 









S

I I I G W G

I BF

q

q q p p

p 1 ||p q|| | |

]

[ s r

• Complex, spatially varying kernels

Cannot be precomputed no FFT

S p q

– Cannot be precomputed, no FFT…

• Brute-force implementation is slow > 10min

But Bilateral Filter is Nonlinear

• Slow but some accelerations exist:

– [Elad 02]: Gauss-Seidel iterations

• Only for many iterations

– [Durand 02, Weiss 06]: fast approximation

• No formal understanding of accuracy versus speed

• [Weiss 06]: Only box function as spatial kernel

A Fast Approximation f th Bil t l Filt of the Bilateral Filter using a Signal Processing using a Signal Processing

Approach Approach

Sylvain Paris and Frédo Durand

Computer Science and Artificial Intelligence Laboratory Massachusetts Institute of Technology

Definition of Bilateral Filter Definition of Bilateral Filter

• [Smith 97, Tomasi 98]

I t R lt

• Smoothes an image and preserves edges

Input Result

• Weighted average of neighbors

• Weights

– Gaussian on space distance – Gaussian on range distance – sum to 1

space range

space range

Contributions

Li k ith

linear filtering

• Link with

linear filtering

•

Fast

accurate

Intuition on 1D Signal

BF

Basic idea

1D Gaussians

   



 G G I I I

I W

BF _p 1 q;p, _{s q}; _p, _{r q} ]

[



W qS

q p

q p

p

Basic idea

1D Gaussians

   



 G G I I I

I W

BF _p 1 q;p, _{s q}; _p, _{r q} ]

[



W qS

q p

q p

p

 

1



]

[I G I I I

BF

2D Gaussians

 



  



' ' ,

' , r s, , ,

; , ]

[

S I

I I

I I W G

I BF

q

q

q

q p

q p

p q p  

a special 2D i 2D image

Intuition on 1D Signal

BF

Intuition on 1D Signal

Weighted Average of Neighbors

p

Weighted Average of Neighbors

weights p

weights applied to pixels

• Near and similar pixels have influence.

• Far pixels have no influence.

• Pixels with different value have no influence.

Link with Linear Filtering

1 Handling the Division

p

1. Handling the Division

p

sum of weights

Handling the division with a

projective space

Formalization: Handling the Division g

• Normalizing factor as homogeneous coordinate

• Normalizing factor as homogeneous coordinate

• Multiply both sides by

Formalization: Handling the Division

with W_q=1

• Similar to homogeneous coordinates in projective space in projective space

• Division delayed until the end

• Division delayed until the end

• Next step: Adding a dimension to make a

• Next step: Adding a dimension to make a convolution appear

Link with Linear Filtering

2 Introducing a Convolution space: 1D Gaussian

 range: 1D Gaussian

p

2. Introducing a Convolution  range: 1D Gaussian combination: 2D Gaussian

p q

space range

Link with Linear Filtering

2 Introducing a Convolution space: 1D Gaussian

range: 1D Gaussian

p

2. Introducing a Convolution ^range: 1D Gaussian combination: 2D Gaussian

p q

space x range

Corresponds to a 3D Gaussian on a 2D image.

Link with Linear Filtering

2 Introducing a Convolution 2. Introducing a Convolution

black = zero sum all values

G i

space-range Gaussian

sum all values multiplied by kernel  convolution

Link with Linear Filtering

2 Introducing a Convolution 2. Introducing a Convolution

result of the convolution result of the convolution

G i

space-range Gaussian

Link with Linear Filtering

2 Introducing a Convolution 2. Introducing a Convolution

result of the convolution result of the convolution

G i

space-range Gaussian

higher dimensional functions

w i w

w i w

Gaussian convolution

division

slicing

Reformulation: Summaryy

1. Convolution in higher dimension

• expensive but well understood (linear, FFT, etc)

2. Division and slicing 2. Division and slicing

• nonlinear but simple and pixel-wise

Exact reformulation

Exact reformulation

higher dimensional functions

w i w

w i w

Gaussian convolution

Low-pass filter

division

slicing

higher dimensional functions

w i w

w i w

D O W N S A M P L E

Gaussian convolution

U P S A M P L E

division

slicing

Fast Convolution by Downsampling

• Downsampling cuts frequencies

• Downsampling cuts frequencies above Nyquist limit

Less data to process – Less data to process – But induces error

• Evaluation of the approximation

– Precision versus running time – Visual accuracy

Accuracy versus Running Time

• Finer sampling increases accuracy.

• More precise than previous work.p p

PSNR as function of Running Time

Digital photograph photograph 1200  1600

Straightforward Straightforward implementation is over 10 minutes.

Visual Results

• Comparison with previous work [Durand 02]

– running time = 1s for both techniques

i t t BF lt k

1200  1600 – running time = 1s for both techniques

input exact BF our result prev. work

difference 0.1

difference with exact computation (intensities in [0:1]) 0

Conclusions

hi h di i  “b tt ” t ti higher dimension  “better” computation

Practical gain Theoretical gain

• Interactive running time

• Visually similar results

• Link with linear filters

• Separation linear/nonlinear

• Visually similar results

• Simple to code (100 lines)

• Separation linear/nonlinear

• Signal processing framework

Two-scale Tone Management Two-scale Tone Management

for Photographic Look g p

Soonmin Bae, Sylvain Paris, and Frédo Durand MIT CSAIL

MIT CSAIL

Ansel Adams

Ansel Adams, Clearing Winter Storm

An Amateur Photographer A Variety of Looks

Goals

• Control over photographic look

• Transfer “look” from a model photop

For example For example,

we want

with the look of

Aspects of Photographic Look

• Subject choice

• Framing and compositiong p

 Specified by input photos

Input

• Tone distribution and contrast

Input

Modified based on model photos

Model

Tonal Aspects of Look

Ansel Adams Kenro Izu

Tonal aspects of Look - Global Contrast

Ansel Adams Kenro Izu

High Global Contrast Low Global Contrast

Tonal aspects of Look - Local Contrast

Ansel Adams Kenro Izu

Variable amount of texture Texture everywhere

Overview

Model

Input Image Result

Input Image Result

• Transfer look between photographsTransfer look between photographs

– Tonal aspects

Overview

Global contrast

Result Input

Image

Local contrast l b l d l l

• Separate global and local contrast

Global contrast Overview

Split Careful

combination

P t

Input Image

Post- process

Local contrast

Result

Global contrast Overview

Split ^Carefulcombination

P t

Input Image

Post- process

Local contrast

Result

Split Global vs. Local Contrast

• Naïve decomposition: low vs. high frequencyp g q y

– Problem: introduce blur & halos

Halo Blur

Low frequency High frequency

Low frequency High frequency

Global contrast Local contrast

Bilateral Filter

• Edge-preserving smoothing [Tomasi 98]g p g g [ ]

• We build upon tone mapping [Durand 02]

After bilateral filtering Residual after filtering After bilateral filtering Residual after filtering

Global contrast Local contrast

Bilateral Filter

• Edge-preserving smoothing [Tomasi 98]g p g g [ ]

• We build upon tone mapping [Durand 02]

BASE layer DETAIL layer

After bilateral filtering Residual after filtering After bilateral filtering Residual after filtering

Global contrast Local contrast

Global contrast

Bil t l Careful

combination

P t Bilateral

Filter

Input Image

Post- process

Local contrast

Result

Global contrast

Bil t l Careful

combination

P t Bilateral

Filter

Input Image

Post- process

Local contrast

Result

Global Contrast

• Intensity remapping of base layer

Remapped intensity

Input base Input intensity After remapping

Global Contrast (Model Transfer) ( )

• Histogram matching

• Histogram matching

– Remapping function given input and model histogram input and model histogram Model

base

Input Output

Input base

Output base

Global contrast

Intensity Intensity matching

Bil t l Careful

combination

P t Bilateral

Filter

Input Image

Post- process

Local contrast

Result

Global contrast

Intensity Intensity matching

Bil t l Careful

combination

P t Bilateral

Filter

Input Image

Post- process

Local contrast

Result

Local Contrast: Detail Layer

• Uniform control:Uniform control:

– Multiply all values in the detail layer

I t B 3 D t il

Input Base + 3  Detail

The amount of local contrast is not uniform

is not uniform

Smooth region

Textured region Textured region

Local Contrast Variation

• We define “textureness”: amount of local contrast

– at each pixel based on surrounding region

Smooth region

 Low textureness

Textured region Textured region

 High textureness

“Textureness”: 1D Example

Textured region

Smooth region Textured region

 Large high-frequency Previous work:

Low pass of |H|

Low pass of |H|

Textured region

 Hi h t t g

 Small high-frequency [Li 05, Su 05][Li 05, Su 05]

 High textureness

Smooth region

L t t

Input signal

 Low textureness

High frequency H Amplitude |H| Edge-preserving Input signal High frequency H Amplitude |H| Edge preserving

filter

Textureness

Input Textureness

Input Textureness

Textureness Transfer

Step 1:

f

Model Model Histogram transfer texturenesstextureness

Hist transfer Input

Input t t t t

Desired Desired t t t t Hist. transfer

textureness

textureness texturenesstextureness

x 0.5 Step 2:

x 2.7 Scaling detail layer

(per pixel) to match desired textureness

x 4.3

Input detail Output detail desired textureness

Global contrast

Intensity Intensity matching

Bil t l Careful

combination

P t Bilateral

Filter

Input Image

Post- process

Textureness matching

Local contrast

Result

Global contrast

Intensity Intensity matching

Bil t l Careful

combination

P t Bilateral

Filter

Input Image

Post- process

Textureness matching

Local contrast

Result

A Non Perfect Result

• Decoupled and large modifications (up to 6x)

Limited defects may appear

input (HDR)

result after

global and local adjustments

Intensity Remapping

S i i i b id di l bl

• Some intensities may be outside displayable range.

 Compress histogram to fit visible range.

corrected result remapped

intensities initial

result

Preserving Details

1. In the gradient domain:

C di t lit d f i t d t

– Compare gradient amplitudes of input and current – Prevent extreme reduction & extreme increase

l h

2. Solve the Poisson equation.

corrected result remapped

intensities initial

result

Effect of Detail Preservation

uncorrected result corrected result

Global contrast

Intensity Intensity matching

Bil t l

P t Bilateral

Filter

Constrained Poisson

Input Image

Post- process

Textureness matching

Local contrast

Result

Global contrast

Intensity Intensity matching

Bil t l Bilateral Filter

Constrained Poisson

Input P Image

Post- process

Textureness matching

Local contrast

Result

Additional Effects

• Soft focus (high frequency manipulation)

• Film grain (texture synthesis [Heeger 95])

• Film grain (texture synthesis [Heeger 95])

• Color toning (chrominance = f (luminance))

before after

before

effects effects

Intensity

Global contrast

Intensity matching

Bil t l Bilateral Filter

Constrained Poisson

S ft f

Input Image

Soft focus Toning

Grain Textureness

matching

Local contrast

Result

Intensity

Global contrast Recap

Intensity matching

Bil t l Bilateral Filter

Constrained Poisson

S ft f

Input Image

Soft focus Toning

Grain Textureness

matching

Local contrast

Result

Results

U id i d d l h h

User provides input and model photographs.

 Our system automatically produces the result.

Running times:

Running times:

– 6 seconds for 1 MPixel or less 23 seconds for 4 MPixels – 23 seconds for 4 MPixels

 multi-grid Poisson solver and fast bilateral filter [Paris 06]]

Input Model

Result Input Result

Input Model

Result Comparison with Naïve Histogram Matching

Input

Input ModelModel

Snapshot

Snapshot, Alfred Stieglitz, Alfred Stieglitz

p

p SnapshotSnapshot, Alfred Stieglitz, Alfred Stieglitz

Our result Our result Naïve Histogram Matching

Naïve Histogram Matching Local contrast, sharpness unfaithful

Comparison with Naïve Histogram Matching

Input Input

Model Model

Clearing Winter Storm, Ansel Adams

p p

Our Result Our Result Histogram Matching

Histogram Matching Local contrast too low

Color Images

• Lab color space: modify only luminance

Input

Input OutputOutput

Limitations

• Noise and JPEG artifacts

– amplified defectsamplified defects

• Can lead to unexpected

• Can lead to unexpected

results if the image content is too different from the model too different from the model

– Portraits, in particular, can suffer

suffer

Conclusions

T f “l k” f d l h

• Transfer “look” from a model photo

• Two-scale tone management

– Global and local contrastGlobal and local contrast

– New edge-preserving textureness – Constrained Poisson reconstructionConstrained Poisson reconstruction – Additional effects

Video Enhancement Using

Per Pixel Exposures (Bennett 06) Per Pixel Exposures (Bennett, 06)

From this video:

ASTA: Adaptive SS ti SSpatio- T

Temporal

A l ti Filt Accumulation Filter

Joint bilateral filtering

Flash / No-Flash Photo Improvement (Petschnigg04) (Eisemann04)

(Petschnigg04) (Eisemann04)

Merge best features: warm, cozy candle light (no-flash)g , y g ( ) low-noise, detailed flash image

Overview

Basic approach of both flash/noflash papers

Remove noise + details

Basic approach of both flash/noflash papers

Remove noise + details from image A,

Keep as image A Lighting

No-flash ---

Obtain noise-free details

o as

Obtain noise free details from image B,

Discard Image B Lighting Result

Petschnigg:

• Flash

Petschnigg:

• No Flash,

Petschnigg:

• Result

Our Approach

Registration Registration

Our Approach

Decomposition

Decomposition

Color / Intensity:

= *

original intensity color

Our Approach

Decomposition

Our Approach

Decoupling

Decoupling

• Lighting : Large-scale variation

T S ll l i i

• Texture : Small-scale variation

Texture Lighting

Lighting

Large-scale Layer

• Bilateral filter – edge preserving filter

• Bilateral filter – edge preserving filter

Smith and Brady 1997; Tomasi and Manducci 1998; Durand et al. 2002

I t O t t

Input Output

Large-scale Layer

• Bilateral filter

Cross Bilateral Filter

• Similar to joint bilateral filter by Petschnigg et al

al.

• When no-flash image is too noisyg y

• Borrow similarity from flash image

 edge stopping from flash image

Bilateral Cross Bilateral

Detail Layer

/ =

/ =

Intensity Large-scale Detail

Recombination: Large scale * Detail = Intensity Recombination: Large scale * Detail = Intensity

Recombination

* =

* =

Large-scale Detail Intensity

No-flash Flash Result

Recombination: Large scale * Detail = Intensity

Recombination: Large scale * Detail = Intensity

Recombination

shadows shadows

** ~ ~

Intensity Color Result

Result Flash

Recombination: Intensity * Color = Original

Our Approach

Our Approach

Shadow Detection/Treatment Detection/Treatment

Results

No-flash

Result Flash

Joint bilateral upsampling Joint bilateral upsampling

Joint bilateral upsampling Joint bilateral upsampling

Joint bilateral upsampling Joint bilateral upsampling

Joint bilateral upsampling Joint bilateral upsampling