Bilateral filtering

(1)

Bilateral Filters

Digital Visual Effects, Spring 2008 Yung-Yu Chuang

2008/5/27

with slides by Fredo Durand, Ramesh Raskar, Sylvain Paris, Soonmin Bae

Announcements

• Final project proposal

• Project #3 artifacts voting

Bilateral filtering

[Ben Weiss, Siggraph 2006]

Input

Input Log(IntensityLog(Intensity) ) Bilateral Smoothing Bilateral Smoothing Gaussian

Gaussian Smoothing Smoothing

Image Denoising

noisy image naïve denoising Gaussian blur

better denoising edge-preserving filter

Smoothing an image without blurring its edges.

(2)

A Wide Range of Options

• Diffusion, Bayesian, Wavelets…

– All have their pros and cons.

• Bilateral filter

– not always the best result [Buades 05] but often good – easy to understand, adapt and set up

Noisy input

Basic denoising

Median 5x5

Basic denoising

Noisy input Bilateral filter 7x7 window

Tone Mapping

[Durand 02]

Tone Mapping

[Durand 02]

HDR input

(3)

Tone Mapping

[Durand 02]

Tone Mapping

[Durand 02]

output

Photographic Style Transfer

[Bae 06]

input

[Bae 06]

output input

Cartoon Rendition

[Winnemöller 06]

(4)

[Winnemöller 06]

output

6 papers at SIGGRAPH’07

Gaussian Blur

average

input

per-pixel multiplication

output

*

input box average

(5)

Gaussian blur

normalized Gaussian function

Equation of Gaussian Blur

( )

∑

∈

−

=

S

I G

I GB

q

p

|| p q ||

]

[

_σ

Same idea: weighted average of pixels.

0 1

unrelated pixels unrelated

pixels

uncertain pixels uncertain

pixels

related pixels

Gaussian Profile

pixel position pixel

weight

⎟⎟⎠

⎜⎜ ⎞

⎝

⎛−

= ²₂

exp 2 2 ) 1

( σ π σ

σ

x x G

size of the window

Spatial Parameter

( )

∑

∈

−

=

S

I G

I GB

q

p

|| p q ||

]

[

_σ

small σ large σ

input

limited smoothing strong smoothing

(6)

How to set σ

• Depends on the application.

• Common strategy: proportional to image size

– e.g. 2% of the image diagonal

– property: independent of image resolution

Properties of Gaussian Blur

• Weights independent of spatial location

– linear convolution

– well-known operation

– efficient computation (recursive algorithm, FFT…)

Properties of Gaussian Blur

• Does smooth images

• But smoothes too much:

edges are blurred.

– Only spatial distance matters – No edge term

input

output

( )

∑

∈

−

=

S

I G

I GB

q

p

|| p q ||

]

[

_σ

space

Blur Comes from Averaging across Edges

*

input output

Same Gaussian kernel everywhere.

(7)

Bilateral Filter No Averaging across Edges

*

input output

The kernel shape depends on the image content.

[Aurich 95, Smith 97, Tomasi 98]

space weight not new

range weight I

new

normalization factor

new

Bilateral Filter Definition

( ) ( )

∑

∈

−

=

S

I I I G W G

I BF

q

q q p p

p 1 || p q || | |

]

[ σs σr

Same idea: weighted average of pixels.

Illustration a 1D Image

• 1D image = line of pixels

• Better visualized as a plot

pixel intensity

pixel position

space

Gaussian Blur and Bilateral Filter

space range normalization

Gaussian blur

( ) ( )

∑∈

−

=

S

I I I G W G

I BF

q

q q p p

p 1 ||p q|| | |

]

[ σs σr

Bilateral filter

[Aurich 95, Smith 97, Tomasi 98]

space

range

p

p q

q

( )

∑

∈

−

=

S

I G

I GB

q

p ||p q||

]

[ _σ

(8)

q q

p p

Bilateral Filter on a Height Field

output input

( )

∑

∈

−

=

S

I I I G W G

I BF

q

q q p p

p 1 ||p q|| | |

]

[ σs σr

p p

reproduced from [Durand 02]

Space and Range Parameters

• space σ_s: spatial extent of the kernel, size of the considered neighborhood.

• range σ_r: “minimum” amplitude of an edge

( ) ( )

∑

∈

−

=

S

I I I G W G

I BF

q

q q p p

p 1 ||p q|| | |

]

[ σs σr

Influence of Pixels

p p

Only pixels close in space and in range are considered.

space

range

σ_s= 2

σ_s= 6

σ_s= 18

σ_r= 0.1 σ_r= 0.25

σ_r= ∞

(Gaussian blur)

input

Exploring the Parameter Space Exploring the Parameter Space

(9)

σ_s= 2

σ_s= 6

σ_s= 18

σ_r= 0.1 σ_r= 0.25

σ_r= ∞

(Gaussian blur)

input

Varying the Range Parameter Varying the Range Parameter

input

σ

_r

= 0.1 σ

_r

= 0.25

(10)

σ

_r

= ∞

(Gaussian blur)

σ_s= 2

σ_s= 6

σ_s= 18

σ_r= 0.1 σ_r= 0.25

σ_r= ∞

(Gaussian blur)

input

Varying the Space Parameter Varying the Space Parameter

input

σ

_s

= 2

(11)

σ

_s

= 6 σ

_s

= 18

How to Set the Parameters

Depends on the application. For instance:

• space parameter: proportional to image size

– e.g., 2% of image diagonal

• range parameter: proportional to edge amplitude

– e.g., mean or median of image gradients

• independent of resolution and exposure

Iterating the Bilateral Filter

• Generate more piecewise-flat images

• Often not needed in computational photo, but could be useful for applications such as NPR.

] [

₍ ₎

) 1

(n

BF I

n

I

₊

=

(12)

input 1 iteration

2 iterations 4 iterations

(13)

Advantages of Bilateral Filter

• Easy to understand

– Weighted mean of nearby pixels

• Easy to adapt

– Distance between pixel values

• Easy to set up

– Non-iterative

Hard to Compute

• Nonlinear

• Complex, spatially varying kernels

– Cannot be precomputed, no FFT…

• Brute-force implementation is slow > 10min

( )

∑∈

−

=

S

I I I G W G

I BF

q

q q p p

p 1 ||p q|| | |

]

[ σs σr

But Bilateral Filter is Nonlinear

• Slow but some accelerations exist:

– [Elad 02]: Gauss-Seidel iterations

• Only for many iterations

– [Durand 02, Weiss 06]: fast approximation

• No formal understanding of accuracy versus speed

• [Weiss 06]: Only box function as spatial kernel

A Fast Approximation of the Bilateral Filter using a Signal Processing

Approach

Sylvain Paris and Frédo Durand

Computer Science and Artificial Intelligence Laboratory Massachusetts Institute of Technology

(14)

Definition of Bilateral Filter

• [Smith 97, Tomasi 98]

• Smoothes an image and preserves edges

• Weighted average of neighbors

• Weights

– Gaussian on space distance – Gaussian on range distance – sum to 1

space range

Input Result

Contributions

• Link with

linear filtering

•

Fast

and

accurate

approximation

Intuition on 1D Signal

BF

p

Intuition on 1D Signal

Weighted Average of Neighbors

• Near and similar pixels have influence.

• Far pixels have no influence.

• Pixels with different value have no influence.

weights applied to pixels

(15)

p

Link with Linear Filtering

1. Handling the Division

sum of weights

Handling the division with a

projective space

.

Formalization: Handling the Division

• Normalizing factor as homogeneous coordinate

• Multiply both sides by

• Normalizing factor as homogeneous coordinate

• Multiply both sides by

Formalization: Handling the Division

• Similar to homogeneous coordinates in projective space

• Division delayed until the end

• Next step: Adding a dimension to make a convolution appear

with W_q=1

space range

p

Link with Linear Filtering

2. Introducing a Convolution

q

space: 1D Gaussian

× range: 1D Gaussian combination: 2D Gaussian space: 1D Gaussian

× range: 1D Gaussian combination: 2D Gaussian

(16)

p

Link with Linear Filtering

q

space: 1D Gaussian

×range: 1D Gaussian combination: 2D Gaussian space: 1D Gaussian

×range: 1D Gaussian combination: 2D Gaussian

space x range

Corresponds to a 3D Gaussian on a 2D image.

Link with Linear Filtering

space-range Gaussian black = zero

sum all values

sum all values multiplied by kernel Ö convolution

space-range Gaussian

result of the convolution

Link with Linear Filtering

space-range Gaussian

result of the convolution

(17)

higher dimensional functions

Gaussian convolution

division

slicing

w i w

Reformulation: Summary

1. Convolution in higher dimension

• expensive but well understood (linear, FFT, etc)

2. Division and slicing

• nonlinear but simple and pixel-wise

Exact reformulation Exact reformulation

division

slicing

Low-pass filter Low-pass filter

Almost only low freq.

High freq.

negligible Almost only

low freq.

High freq.

negligible

w i w

division

slicing

w i w

D O W N S A M P L E

U P S A M P L E

Almost no information loss Almost no information loss

(18)

Fast Convolution by Downsampling

• Downsampling cuts frequencies above Nyquist limit

– Less data to process – But induces error

• Evaluation of the approximation

– Precision versus running time – Visual accuracy

Accuracy versus Running Time

• Finer sampling increases accuracy.

• More precise than previous work.

finer sampling

PSNR as function of Running Time

Digital photograph 1200 × 1600

Straightforward implementation is over 10 minutes.

input exact BF our result prev. work

1200 × 1600

• Comparison with previous work [Durand 02]

– running time = 1s for both techniques

0

difference 0.1

with exact computation (intensities in [0:1])

Visual Results Conclusions

Practical gain

• Interactive running time

• Visually similar results

• Simple to code (100 lines)

Theoretical gain

• Link with linear filters

• Separation linear/nonlinear

• Signal processing framework

higher dimension Ö “better” computation

(19)

Two-scale Tone Management for Photographic Look

Soonmin Bae, Sylvain Paris, and Frédo Durand MIT CSAIL

Ansel Adams

Ansel Adams, Clearing Winter Storm

An Amateur Photographer A Variety of Looks

(20)

Goals

• Control over photographic look

• Transfer “look” from a model photo

For example,

we want

with the look of

Aspects of Photographic Look

• Subject choice

• Framing and composition Î Specified by input photos

• Tone distribution and contrast ÎModified based on model photos

Input

Model

Tonal Aspects of Look

Ansel Adams Kenro Izu

Tonal aspects of Look - Global Contrast

High Global Contrast Low Global Contrast

(21)

Tonal aspects of Look - Local Contrast

Variable amount of texture Texture everywhere

Overview

Input Image Result

Model

• Transfer look between photographs

– Tonal aspects

Overview

Local contrast Global contrast

Result

• Separate global and local contrast

Input Image

Split

Input Image

Result

Careful combination

Post- process

Overview

(22)

Split

Global contrast

Input Image

Result

Overview

Local contrast

Split Global vs. Local Contrast

• Naïve decomposition: low vs. high frequency

– Problem: introduce blur & halos

Low frequency High frequency

Halo Blur

Global contrast Local contrast

Bilateral Filter

• Edge-preserving smoothing [Tomasi 98]

• We build upon tone mapping [Durand 02]

After bilateral filtering Residual after filtering

Bilateral Filter

• Edge-preserving smoothing [Tomasi 98]

• We build upon tone mapping [Durand 02]

After bilateral filtering Residual after filtering

BASE layer DETAIL layer

(23)

Global contrast

Input Image

Result

Post- process Bilateral

Filter

Local contrast Local contrast

Global contrast

Input Image

Result

Filter

Global Contrast

• Intensity remapping of base layer

Input base Input intensity After remapping Remapped

intensity

Global Contrast (Model Transfer)

• Histogram matching

– Remapping function given input and model histogram Model

base

Input base

Output base

(24)

Input Image

Result

Filter

Intensity matching

Local contrast

Global contrast

Input Image

Result

Filter

Local Contrast: Detail Layer

• Uniform control:

– Multiply all values in the detail layer

Input Base + 3 × Detail

The amount of local contrast is not uniform

Smooth region

Textured region

(25)

Local Contrast Variation

• We define “textureness”: amount of local contrast

– at each pixel based on surrounding region

Smooth region Ö Low textureness

Textured region Ö High textureness

Input signal

“Textureness”: 1D Example

Smooth region Ö Low textureness Textured region Ö High textureness

High frequency H Amplitude |H| Edge-preserving filter

Smooth region

Ö Small high-frequency Textured region

Ö Large high-frequency Previous work:

Low pass of |H|

[Li 05, Su 05]

Low pass of |H|

[Li 05, Su 05]

Textureness

Input Textureness

Textureness Transfer

Step 1:

Histogram transfer

Hist. transfer Input

Input textureness textureness

Desired Desired textureness textureness Model

Model textureness textureness

x 0.5 x 2.7

x 4.3

Input detail Output detail Step 2:

Scaling detail layer (per pixel) to match

desired textureness

(26)

Input Image

Result

Filter

Textureness matching

Input Image

Result

Filter

A Non Perfect Result

• Decoupled and large modifications (up to 6x) ÎLimited defects may appear

input (HDR)

result after

global and local adjustments

Intensity Remapping

• Some intensities may be outside displayable range.

Î Compress histogram to fit visible range.

corrected result remapped

intensities initial

result

(27)

Preserving Details

1. In the gradient domain:

– Compare gradient amplitudes of input and current – Prevent extreme reduction & extreme increase

2. Solve the Poisson equation.

corrected result remapped

intensities initial

result

Effect of Detail Preservation

uncorrected result corrected result

Input Image

Result

Filter

Constrained Poisson

Input Image

Result

Bilateral Filter

(28)

Additional Effects

• Soft focus (high frequency manipulation)

• Film grain (texture synthesis [Heeger 95])

• Color toning (chrominance = f (luminance))

before effects

after effects model

Input Image

Result

Soft focus Toning

Grain

Input Image

Result

Soft focus Toning

Grain

Recap

Results

User provides input and model photographs.

Î Our system automatically produces the result.

Running times:

– 6 seconds for 1 MPixel or less – 23 seconds for 4 MPixels

multi-grid Poisson solver and fast bilateral filter [Paris 06]

(29)

Input Model

Result Input Result

Input Model

Result

Input Input

Our result Our result NaïNaïve Histogram Matchingve Histogram Matching

Model Model

Snapshot

Snapshot, Alfred Stieglitz, Alfred Stieglitz

Comparison with Naïve Histogram Matching

Local contrast, sharpness unfaithful

(30)

Input Input

Our Result Our Result

Model Model

Clearing Winter Storm, Ansel Adams

Histogram Matching Histogram Matching

Comparison with Naïve Histogram Matching

Local contrast too low

Color Images

• Lab color space: modify only luminance

Input

Input OutputOutput

Limitations

• Noise and JPEG artifacts

– amplified defects

• Can lead to unexpected

results if the image content is too different from the model

– Portraits, in particular, can suffer

Conclusions

• Transfer “look” from a model photo

• Two-scale tone management

– Global and local contrast

– New edge-preserving textureness – Constrained Poisson reconstruction – Additional effects

(31)

Video Enhancement Using

Per Pixel Exposures (Bennett, 06)

From this video:

ASTA: Adaptive Spatio-S T

Temporal

Accumulation Filter

Joint bilateral filtering

Flash / No-Flash Photo Improvement (Petschnigg04) (Eisemann04)

Merge best features: warm, cozy candle light (no-flash) low-noise, detailed flash image

Overview

Remove noise + details from image A,

Keep as image A Lighting ---

Obtain noise-free details from image B,

Discard Image B Lighting Result No-flash Basic approach of both flash/noflash papers

(32)

Petschnigg:

• Flash

Petschnigg:

• No Flash,

Petschnigg:

• Result

Our Approach

Registration Registration

(33)

Our Approach

Decomposition

Color / Intensity:

original

= *

intensity color

Our Approach

Decomposition

Our Approach

Decoupling

(34)

Decoupling

• Lighting : Large-scale variation

• Texture : Small-scale variation

Texture Lighting

• Lighting : Large-scale variation

• Texture : Small-scale variation

Lighting

Large-scale Layer

• Bilateral filter – edge preserving filter

Smith and Brady 1997; Tomasi and Manducci 1998; Durand et al. 2002

Input Output

Large-scale Layer

• Bilateral filter

Cross Bilateral Filter

• Similar to joint bilateral filter by Petschnigg et al.

• When no-flash image is too noisy

• Borrow similarity from flash image

¾ edge stopping from flash image

• See detail in paper

Bilateral Cross Bilateral

(35)

Detail Layer

Intensity Large-scale

/ =

Detail

Recombination: Large scale * Detail = Intensity

Recombination

Large-scale No-flash

Detail Flash

*

Intensity Result

=

Recombination: Large scale * Detail = Intensity

Recombination

Intensity Result

Color Flash

* ~ ~

Result

Recombination: Intensity * Color = Original shadows

Our Approach

(36)

Our Approach

Shadow Detection/Treatment

Results

Result No-flash

Flash

Joint bilateral upsampling Joint bilateral upsampling

(37)

Joint bilateral upsampling Joint bilateral upsampling

(38)

Joint bilateral upsampling Joint bilateral upsampling

References

• Patrick Perez, Michel Gangnet, Andrew Blake, Poisson Image Editing, SIGGRAPH 2003.

• Dani Lischinski, Zeev Farbman, Matt Uytendaelle and Richard Szeliski. Interactive Local Adjustment of Tonal Values. SIGGRAPH 2006.

• Carsten Rother, Andrew Blake, Vladimir Kolmogorov, GrabCut - Interactive Foreground Extraction Using Iterated Graph Cuts, SIGGRAPH 2004.

• Aseem Agarwala, Mira Dontcheva, Maneesh Agrawala, Steven Drucker, Alex Colburn, Brian Curless, David H. Salesin, Michael F.

Cohen, Interactive Digital Photomontage, SIGGRAPH 2004.

• Sylvain Paris and Fredo Durand. A Fast Approximation of the Bilateral Filter using a Signal Processing Approach. ECCV 2006.

• Soonmin Bae, Sylvain Paris and Fredo Durand. Two-scale Tone Management for Photographic Look. SIGGRAPH 2006.