Bilateral Filters

(1)

Bilateral Filters

Digital Visual Effects Yung-Yu Chuang

with slides by Fredo Durand, Ramesh Raskar, Sylvain Paris, Soonmin Bae

(2)

Bilateral filtering

[Ben Weiss, Siggraph 2006]

Input

Input Log(Intensity) Log(Intensity) Bilateral Smoothing Bilateral Smoothing Gaussian

Gaussian Smoothing Smoothing

(3)

Image Denoising

noisy image naïve denoising Gaussian blur

better denoising edge-preserving filter

Smoothing an image without blurring its edges.

(4)

A Wide Range of Options

• Diffusion, Bayesian, Wavelets…

– All have their pros and cons.

• Bilateral filter

– not always the best result [Buades 05] but often good – easy to understand, adapt and set up

(5)

Noisy input

Basic denoising

Median 5x5

(6)

Basic denoising

Noisy input Bilateral filter 7x7 window

(7)

Tone Mapping

[Durand 02]

Tone Mapping

[Durand 02]

HDR input

(8)

Tone Mapping

[Durand 02]

Tone Mapping

[Durand 02]

output

(9)

Photographic Style Transfer

[Bae 06]

input

(10)

[Bae 06]

output

(11)

input

Cartoon Rendition

[Winnemöller 06]

(12)

[Winnemöller 06]

output

6 papers at

SIGGRAPH’07 6 papers at

SIGGRAPH’07

(13)

Gaussian Blur

average

input

per-pixel multiplication

output

*

(14)

input

(15)

box average

(16)

Gaussian blur

(17)

normalized Gaussian function

Equation of Gaussian Blur

 









S

I G

I GB

q

p

|| p q ||

]

[

_

Same idea: weighted average of pixels.

0 1

(18)

unrelated pixels unrelated

pixels

uncertain pixels uncertain

pixels

related pixels

Gaussian Profile

pixel position pixel

weight



 





 ²₂

exp 2 2

) 1

(   



x x G

(19)

size of the window

Spatial Parameter

 









S

I G

I GB

q

p

|| p q ||

]

[

_

small  large 

input

limited smoothing strong smoothing

(20)

How to set 

• Depends on the application.

• Common strategy: proportional to image size

– e.g. 2% of the image diagonal

– property: independent of image resolution

(21)

Properties of Gaussian Blur

• Weights independent of spatial location

– linear convolution

– well-known operation

– efficient computation (recursive algorithm, FFT…)

(22)

Properties of Gaussian Blur

• Does smooth images

• But smoothes too much:

edges are blurred.

– Only spatial distance matters – No edge term

input

output

 









S

I G

I GB

q

p

|| p q ||

]

[

_

space

(23)

Blur Comes from Averaging across Edges

*

input output

Same Gaussian kernel everywhere.

(24)

Bilateral Filter No Averaging across Edges

*

input output

The kernel shape depends on the image content.

[Aurich 95, Smith 97, Tomasi 98]

(25)

space weight

not new

range weight

I

new

normalization factor

new

Bilateral Filter Definition

   









S

I I

I G

W G I

BF

q

q q

p p

p

1 || p q || | |

]

[

s r

Same idea: weighted average of pixels.

(26)

Illustration a 1D Image

• 1D image = line of pixels

• Better visualized as a plot

pixel intensity

pixel position

(27)

space

Gaussian Blur and Bilateral Filter

space range normalization

Gaussian blur

 









S

I I

I G

W G I

BF

q

q q

p p

p 1 ||p q || | |

]

[ s r

Bilateral filter

[Aurich 95, Smith 97, Tomasi 98]

space

range

p

p q

q

 









S

I G

I GB

q

p || p q ||

]

[ _

(28)

q q

p p

Bilateral Filter on a Height Field

output input

   









S

I I

I G

W G I

BF

q

q q

p p

p 1 || p q || | |

]

[ s r

p p

reproduced from [Durand 02]

(29)

Space and Range Parameters

• space _s: spatial extent of the kernel, size of the considered neighborhood.

• range _r : “minimum” amplitude of an edge

   









S

I I

I G

W G I

BF

q

q q

p p

p

1 || p q || | |

]

[

s r

(30)

Influence of Pixels

p p

Only pixels close in space and in range are considered.

space

range

(31)

_s= 2

_s= 6

_s= 18

_r= 0.1 _r= 0.25 _r= 

(Gaussian blur)

input

Exploring the Parameter Space Exploring the Parameter Space

(32)

_s= 2

_s= 6

_s= 18

_r= 0.1 _r= 0.25 _r= 

(Gaussian blur)

input

Varying the Range Parameter Varying the Range Parameter

(33)

input

(34)



_r

= 0.1

(35)



_r

= 0.25

(36)



_r

= 

(Gaussian blur)

(37)

_s= 2

_s= 6

_s= 18

_r= 0.1 _r= 0.25 _r= 

(Gaussian blur)

input

Varying the Space Parameter Varying the Space Parameter

(38)

input

(39)



_s

= 2

(40)



_s

= 6

(41)



_s

= 18

(42)

How to Set the Parameters

Depends on the application. For instance:

• space parameter: proportional to image size

– e.g., 2% of image diagonal

• range parameter: proportional to edge amplitude

– e.g., mean or median of image gradients

• independent of resolution and exposure

(43)

Iterating the Bilateral Filter

• Generate more piecewise-flat images

• Often not needed in computational photo, but could be useful for applications such as NPR.

] [

₍ ₎

) 1

(n

BF I

n

I

_



(44)

input

(45)

1 iteration

(46)

2 iterations

(47)

4 iterations

(48)

Advantages of Bilateral Filter

• Easy to understand

– Weighted mean of nearby pixels

• Easy to adapt

– Distance between pixel values

• Easy to set up

– Non-iterative

(49)

Hard to Compute

• Nonlinear

• Complex, spatially varying kernels

– Cannot be precomputed, no FFT…

• Brute-force implementation is slow > 10min

 









S

I I

I G

W G I

BF

q

q q

p p

p 1 || p q || | |

]

[ s r

(50)

But Bilateral Filter is Nonlinear

• Slow but some accelerations exist:

– [Elad 02]: Gauss-Seidel iterations

• Only for many iterations

– [Durand 02, Weiss 06]: fast approximation

• No formal understanding of accuracy versus speed

• [Weiss 06]: Only box function as spatial kernel

(51)

A Fast Approximation of the Bilateral Filter

using a Signal Processing Approach

Sylvain Paris and Frédo Durand

Computer Science and Artificial Intelligence Laboratory Massachusetts Institute of Technology

(52)

Definition of Bilateral Filter

• [Smith 97, Tomasi 98]

• Smoothes an image and preserves edges

• Weighted average of neighbors

• Weights

– Gaussian on space distance – Gaussian on range distance – sum to 1

space range

Input Result

(53)

Contributions

• Link with

linear filtering

•

Fast

^and

accurate

approximation

(54)

Intuition on 1D Signal

BF

(55)

Basic idea

   







S

I I

I G W G

I BF

q

q p

p 1 q;p, _s ; , _r

]

[

 

1D Gaussians

(56)

Basic idea

   







S

I I

I G W G

I BF

q

q p

p 1 q;p, _s ; , _r

]

[

 

1D Gaussians

 





  



' '

,

' , r

s, ,

,

; 1 ,

] [

S I

I I

W G I

BF

q

q q

q p

p q p

 

2D Gaussians a special

2D image

(57)

Intuition on 1D Signal

BF

(58)

p

Intuition on 1D Signal

Weighted Average of Neighbors

• Near and similar pixels have influence.

• Far pixels have no influence.

• Pixels with different value have no influence.

weights applied to pixels

(59)

p

Link with Linear Filtering

1. Handling the Division

sum of weights

Handling the division with a

projective space

^.

(60)

Formalization: Handling the Division

• Normalizing factor as homogeneous coordinate

• Multiply both sides by

• Normalizing factor as homogeneous coordinate

• Multiply both sides by

(61)

Formalization: Handling the Division

• Similar to homogeneous coordinates in projective space

• Division delayed until the end

• Next step: Adding a dimension to make a convolution appear

with W_q=1

(62)

space range p

Link with Linear Filtering

2. Introducing a Convolution

q

space: 1D Gaussian  range: 1D Gaussian combination: 2D Gaussian space: 1D Gaussian  range: 1D Gaussian combination: 2D Gaussian

(63)

p

Link with Linear Filtering

q

space: 1D Gaussian

 range: 1D Gaussian combination: 2D Gaussian space: 1D Gaussian

 range: 1D Gaussian combination: 2D Gaussian

space x range

Corresponds to a 3D Gaussian on a 2D image.

(64)

Link with Linear Filtering

space-range Gaussian black = zero

sum all values

sum all values multiplied by kernel  convolution

(65)

space-range Gaussian

result of the convolution

Link with Linear Filtering

(66)

Link with Linear Filtering

space-range Gaussian

result of the convolution

(67)

higher dimensional functions

Gaussian convolution

division

slicing

w i w

(68)

Reformulation: Summary

1. Convolution in higher dimension

• expensive but well understood (linear, FFT, etc)

2. Division and slicing

• nonlinear but simple and pixel-wise

Exact reformulation

(69)

division

slicing

Low-pass filter Low-pass filter

Almost only low freq.

High freq.

negligible Almost only

low freq.

High freq.

negligible

w i w

(70)

division

slicing

w i w

D O W N S A M P L E

U P S A M P L E

Almost no information loss Almost no information loss

(71)

Fast Convolution by Downsampling

• Downsampling cuts frequencies above Nyquist limit

– Less data to process – But induces error

• Evaluation of the approximation

– Precision versus running time – Visual accuracy

(72)

Accuracy versus Running Time

• Finer sampling increases accuracy.

• More precise than previous work.

finer sampling

PSNR as function of Running Time

Digital photograph 1200  1600

Straightforward implementation is over 10 minutes.

(73)

input exact BF our result prev. work

1200  1600

• Comparison with previous work [Durand 02]

– running time = 1s for both techniques

0

difference 0.1

with exact computation (intensities in [0:1])

Visual Results

(74)

Conclusions

Practical gain

• Interactive running time

• Visually similar results

• Simple to code (100 lines)

Theoretical gain

• Link with linear filters

• Separation linear/nonlinear

• Signal processing framework

higher dimension  “better” computation

(75)

Two-scale Tone Management for Photographic Look

Soonmin Bae, Sylvain Paris, and Frédo Durand MIT CSAIL

(76)

Ansel Adams

Ansel Adams, Clearing Winter Storm

(77)

An Amateur Photographer

(78)

A Variety of Looks

(79)

Goals

• Control over photographic look

• Transfer “look” from a model photo For example,

we want

with the look of

(80)

Aspects of Photographic Look

• Subject choice

• Framing and composition

 Specified by input photos

• Tone distribution and contrast

Modified based on model photos

Input

Model

(81)

Tonal Aspects of Look

Ansel Adams Kenro Izu

(82)

Tonal aspects of Look - Global Contrast

High Global Contrast Low Global Contrast

(83)

Tonal aspects of Look - Local Contrast

Variable amount of texture Texture everywhere

(84)

Overview

Input Image Result

Model

• Transfer look between photographs

– Tonal aspects

(85)

Overview

Local contrast Global contrast

Result

• Separate global and local contrast

Input Image

(86)

Split

Input Image

Result

Careful

combination

Post- process

Overview

(87)

Split

Global contrast

Input Image

Result

Careful

combination

Overview

Local contrast

(88)

Split Global vs. Local Contrast

• Naïve decomposition: low vs. high frequency

– Problem: introduce blur & halos

Low frequency High frequency

Halo Blur

Global contrast Local contrast

(89)

Bilateral Filter

• Edge-preserving smoothing [Tomasi 98]

• We build upon tone mapping [Durand 02]

After bilateral filtering Residual after filtering

(90)

Bilateral Filter

• Edge-preserving smoothing [Tomasi 98]

• We build upon tone mapping [Durand 02]

After bilateral filtering Residual after filtering

BASE layer DETAIL layer

(91)

Global contrast

Input Image

Result

Careful

combination

Post- process Bilateral

Filter

Local contrast

(92)

Local contrast

Global contrast

Input Image

Result

Careful

combination

Filter

(93)

Global Contrast

• Intensity remapping of base layer

Input base Input intensity After remapping Remapped

intensity

(94)

Global Contrast (Model Transfer)

• Histogram matching

– Remapping function given input and model histogram Model

base

Input base

Output base

(95)

Input Image

Result

Careful

combination

Filter

Intensity matching

(96)

Local contrast

Global contrast

Input Image

Result

Careful

combination

Filter

(97)

Local Contrast: Detail Layer

• Uniform control:

– Multiply all values in the detail layer

Input Base + 3  Detail

(98)

The amount of local contrast is not uniform

Smooth region

Textured region

(99)

Local Contrast Variation

• We define “textureness”: amount of local contrast

– at each pixel based on surrounding region

Smooth region

 Low textureness

Textured region

 High textureness

(100)

Input signal

“Textureness”: 1D Example

Smooth region

 Low textureness Textured region

 High textureness

High frequency H Amplitude |H| Edge-preserving filter

Smooth region

 Small high-frequency Textured region

 Large high-frequency Previous work:

Low pass of |H|

[Li 05, Su 05]

Low pass of |H|

[Li 05, Su 05]

(101)

Textureness

Input Textureness

(102)

Textureness Transfer

Step 1:

Histogram transfer

Hist. transfer

Input Input textureness textureness

Desired Desired textureness textureness Model

Model textureness textureness

x 0.5 x 2.7

x 4.3

Input detail Output detail Step 2:

Scaling detail layer (per pixel) to match desired textureness

(103)

Input Image

Result

Careful

combination

Filter

Textureness matching

(104)

Input Image

Result

Careful

combination

Filter

(105)

A Non Perfect Result

• Decoupled and large modifications (up to 6x)

Limited defects may appear

input (HDR)

result after

global and local adjustments

(106)

Intensity Remapping

• Some intensities may be outside displayable range.

 Compress histogram to fit visible range.

corrected result remapped

intensities initial

result

(107)

Preserving Details

1. In the gradient domain:

– Compare gradient amplitudes of input and current – Prevent extreme reduction & extreme increase

2. Solve the Poisson equation.

corrected result remapped

intensities initial

result

(108)

Effect of Detail Preservation

uncorrected result corrected result

(109)

Input Image

Result

Filter

Constrained Poisson

(110)

Input Image

Result

Bilateral Filter

(111)

Additional Effects

• Soft focus (high frequency manipulation)

• Film grain (texture synthesis [Heeger 95])

• Color toning (chrominance = f (luminance))

before effects

after effects model

(112)

Input Image

Result

Soft focus Toning

Grain

(113)

Input Image

Result

Soft focus Toning

Grain

Recap

(114)

Results

User provides input and model photographs.

 Our system automatically produces the result.

Running times:

– 6 seconds for 1 MPixel or less – 23 seconds for 4 MPixels

 multi-grid Poisson solver and fast bilateral filter ^[Paris

06]

(115)

Input Model

Result

(116)

Input

Result

(117)

Input Model

Result

(118)

Input Input

Our result Our result Naïve Histogram Matching

Naïve Histogram Matching

Model Model

Snapshot

Snapshot, Alfred Stieglitz, Alfred Stieglitz

Comparison with Naïve Histogram Matching

Local contrast, sharpness unfaithful

(119)

Input Input

Our Result Our Result

Model Model

Clearing Winter Storm, Ansel Adams

Histogram Matching Histogram Matching

Comparison with Naïve Histogram Matching

Local contrast too low

(120)

Color Images

• Lab color space: modify only luminance

Input

Input OutputOutput

(121)

Limitations

• Noise and JPEG artifacts

– amplified defects

• Can lead to unexpected

results if the image content is too different from the model

– Portraits, in particular, can suffer

(122)

Conclusions

• Transfer “look” from a model photo

• Two-scale tone management

– Global and local contrast

– New edge-preserving textureness – Constrained Poisson reconstruction – Additional effects

(123)

Video Enhancement Using

Per Pixel Exposures (Bennett, 06)

From this video:

ASTA: Adaptive Spatio-S TemporalT

Accumulation Filter

(124)

Joint bilateral filtering

(125)

Flash / No-Flash Photo Improvement (Petschnigg04) (Eisemann04)

Merge best features: warm, cozy candle light (no-flash) low-noise, detailed flash image

(126)

Overview

Remove noise + details from image A,

Keep as image A Lighting ---

Obtain noise-free details from image B,

Discard Image B Lighting Result No-flash

Basic approach of both flash/noflash papers

(127)

Petschnigg:

• Flash

(128)

Petschnigg:

• No Flash,

(129)

Petschnigg:

• Result

(130)

Our Approach

Registration Registration

(131)

Our Approach

Decomposition

(132)

Decomposition

Color / Intensity:

original

= *

intensity color

(133)

Our Approach

Decomposition

(134)

Our Approach

Decoupling

(135)

Decoupling

• Lighting : Large-scale variation

• Texture : Small-scale variation

Texture Lighting

Lighting

(136)

Large-scale Layer

• Bilateral filter – edge preserving filter

Smith and Brady 1997; Tomasi and Manducci 1998; Durand et al. 2002

Input Output

(137)

Large-scale Layer

• Bilateral filter

(138)

Cross Bilateral Filter

• Similar to joint bilateral filter by Petschnigg et al.

• When no-flash image is too noisy

• Borrow similarity from flash image

 edge stopping from flash image

Bilateral Cross Bilateral

(139)

Detail Layer

Intensity Large-scale

/ =

Detail

Recombination: Large scale * Detail = Intensity

(140)

Recombination

Large-scale No-flash

Detail Flash

*

Intensity Result

=

Recombination: Large scale * Detail = Intensity

(141)

Recombination

Intensity Result

Color Flash

* ~ ~

Result

Recombination: Intensity * Color = Original

shadows

(142)

Our Approach

(143)

Our Approach

Shadow

Detection/Treatment

(144)

Results

Result No-flash

Flash

(145)

Joint bilateral upsampling

(146)

Joint bilateral upsampling

(147)

Joint bilateral upsampling

(148)

Joint bilateral upsampling

(149)

Joint bilateral upsampling

(150)

Joint bilateral upsampling

(151)

Joint bilateral upsampling

(152)