A Wide Range of Options

151  Download (0)

Full text

(1)

Bilateral Filters

Digital Visual Effects, Spring 2008 Yung-Yu Chuang

2008/5/27

with slides by Fredo Durand, Ramesh Raskar, Sylvain Paris, Soonmin Bae

(2)

Announcements

• Final project proposal

• Project #3 artifacts voting

(3)

Bilateral filtering

[Ben Weiss, Siggraph 2006]

[Ben Weiss, Siggraph 2006]

Input

Input Log(Intensity) Log(Intensity) Bilateral Smoothing Bilateral Smoothing Gaussian

Gaussian Smoothing Smoothing

(4)

Image Denoising

(5)

A Wide Range of Options

• Diffusion, Bayesian, Wavelets…

– All have their pros and cons.

• Bilateral filter

– not always the best result [Buades 05] but often good – easy to understand, adapt and set up

(6)

Noisy input

Basic denoising

Median 5x5

(7)

Basic denoising

Noisy input Bilateral filter 7x7 window

(8)

Tone Mapping

[Durand 02]

Tone Mapping

[Durand 02]

(9)

Tone Mapping

[Durand 02]

Tone Mapping

[Durand 02]

output

(10)

Photographic Style Transfer

[Bae 06]

Photographic Style Transfer

[Bae 06]

(11)

Photographic Style Transfer

[Bae 06]

Photographic Style Transfer

[Bae 06]

output

(12)

Cartoon Rendition

[Winnemöller 06]

Cartoon Rendition

[Winnemöller 06]

(13)

Cartoon Rendition

[Winnemöller 06]

Cartoon Rendition

[Winnemöller 06]

output

6 papers at

SIGGRAPH’07 6 papers at

SIGGRAPH’07

(14)

Gaussian Blur

input

per-pixel multiplication

output

*

(15)

input

(16)

box average

(17)

Gaussian blur

(18)

normalized

Equation of Gaussian Blur

( )

=

S

I G

I GB

q

q

p

|| p q ||

]

[

σ

Same idea: weighted average of pixels.

(19)

unrelated pixels unrelated

pixels

uncertain pixels uncertain

pixels

related pixels

Gaussian Profile

pixel position pixel

weight

⎟⎟⎠

⎜⎜ ⎞

⎛−

= 22

exp 2 2

) 1

( σ π σ

σ

x x G

(20)

size of the window

Spatial Parameter

( )

=

S

I G

I GB

q

q

p

|| p q ||

]

[

σ

small σ large σ

input

(21)

How to set σ

• Depends on the application.

• Common strategy: proportional to image size

e.g. 2% of the image diagonal

property: independent of image resolution

(22)

Properties of Gaussian Blur

• Weights independent of spatial location

linear convolution

well-known operation

efficient computation (recursive algorithm, FFT…)

(23)

Properties of Gaussian Blur

• Does smooth images

• But smoothes too much:

edges are blurred.

Only spatial distance matters No edge term

input

output

( )

=

S

I G

I GB

q

q

p

|| p q ||

]

[

σ

space

(24)

Blur Comes from Averaging across Edges

*

*

input output

(25)

Bilateral Filter No Averaging across Edges

*

*

*

input output

The kernel shape depends on the image content.

[Aurich 95, Smith 97, Tomasi 98]

(26)

not new new new

Bilateral Filter Definition

( ) ( )

=

S

I I

I G

W G I

BF

q

q q

p p

p

1 || p q || | |

]

[

σs σ r

Same idea: weighted average of pixels.

(27)

Illustration a 1D Image

• 1D image = line of pixels

• Better visualized as a plot

pixel intensity

pixel position

(28)

space

Gaussian Blur and Bilateral Filter

Gaussian blur

Bilateral filter

[Aurich 95, Smith 97, Tomasi 98]

space p

q

( )

=

S

I G

I GB

q

q

p ||p q ||

]

[ σ

(29)

q q

p p

Bilateral Filter on a Height Field

output input

( ) ( )

=

S

I I

I G

W G I

BF

q

q q

p p

p 1 || p q || | |

]

[ σs σr

p p

reproduced from [Durand 02]

(30)

Space and Range Parameters

• space σs : spatial extent of the kernel, size of the considered neighborhood.

( ) ( )

=

S

I I

I G

W G I

BF

q

q q

p p

p

1 || p q || | |

]

[

σs σr

(31)

Influence of Pixels

p p

Only pixels close in space and in range are considered.

space

range

(32)

σs= 2

σ = 6

σr = 0.1 σr = 0.25

σr = ∞

(Gaussian blur)

input

Exploring the Parameter Space Exploring the Parameter Space

(33)

σs= 2

σs= 6

σs= 18

σr = 0.1 σr = 0.25

σr = ∞

(Gaussian blur)

input

Varying the Range Parameter Varying the Range Parameter

(34)

input

(35)

σ

r

= 0.1

(36)

σ

r

= 0.25

(37)

σ

r

= ∞

(Gaussian blur)

(38)

σs= 2

σ = 6

σr = 0.1 σr = 0.25

σr = ∞

(Gaussian blur)

input

Varying the Space Parameter Varying the Space Parameter

(39)

input

(40)

σ

s

= 2

(41)

σ

s

= 6

(42)

σ

s

= 18

(43)

How to Set the Parameters

Depends on the application. For instance:

space parameter: proportional to image size

e.g., 2% of image diagonal

range parameter: proportional to edge amplitude

e.g., mean or median of image gradients

independent of resolution and exposure

(44)

Iterating the Bilateral Filter

• Generate more piecewise-flat images

• Often not needed in computational photo, but

] [

( )

) 1

(n

BF I

n

I

+

=

(45)

input

(46)

1 iteration

(47)

2 iterations

(48)

4 iterations

(49)

Advantages of Bilateral Filter

• Easy to understand

– Weighted mean of nearby pixels

• Easy to adapt

– Distance between pixel values

• Easy to set up

– Non-iterative

(50)

Hard to Compute

• Nonlinear

• Complex, spatially varying kernels

Cannot be precomputed, no FFT…

( )

( )

=

S

I I

I G

W G I

BF

q

q q

p p

p 1 || p q || | |

]

[ σs σr

(51)

But Bilateral Filter is Nonlinear

• Slow but some accelerations exist:

– [Elad 02]: Gauss-Seidel iterations

• Only for many iterations

– [Durand 02, Weiss 06]: fast approximation

• No formal understanding of accuracy versus speed

• [Weiss 06]: Only box function as spatial kernel

(52)

A Fast Approximation of the Bilateral Filter

using a Signal Processing Approach

Sylvain Paris and Frédo Durand

(53)

Definition of Bilateral Filter

• [Smith 97, Tomasi 98]

• Smoothes an image and preserves edges

• Weighted average of neighbors

• Weights

– Gaussian on space distance – Gaussian on range distance – sum to 1

space range

Input Result

(54)

Contributions

• Link with

linear filtering

Fast

and

accurate

approximation

(55)

Intuition on 1D Signal

BF

(56)

p

Intuition on 1D Signal

Weighted Average of Neighbors

• Near and similar pixels have influence.

weights applied to pixels

(57)

p

Link with Linear Filtering

1. Handling the Division

sum of weights

Handling the division with a

projective space

.

(58)

Formalization: Handling the Division

• Normalizing factor as homogeneous coordinate

• Multiply both sides by

• Normalizing factor as homogeneous coordinate

• Multiply both sides by

(59)

Formalization: Handling the Division

• Similar to homogeneous coordinates in projective space

• Division delayed until the end

• Next step: Adding a dimension to make a convolution appear

with Wq=1

(60)

p

Link with Linear Filtering

2. Introducing a Convolution

q

space: 1D Gaussian

× range: 1D Gaussian combination: 2D Gaussian space: 1D Gaussian

× range: 1D Gaussian combination: 2D Gaussian

(61)

p

Link with Linear Filtering

2. Introducing a Convolution

q

space: 1D Gaussian

× range: 1D Gaussian combination: 2D Gaussian space: 1D Gaussian

× range: 1D Gaussian combination: 2D Gaussian

space x range

Corresponds to a 3D Gaussian on a 2D image.

(62)

Link with Linear Filtering

2. Introducing a Convolution

black = zero sum all values

(63)

space-range Gaussian

result of the convolution

Link with Linear Filtering

2. Introducing a Convolution

(64)

Link with Linear Filtering

2. Introducing a Convolution

result of the convolution

(65)

higher dimensional functions

Gaussian convolution

division

slicing

w i w

(66)

Reformulation: Summary

1. Convolution in higher dimension

• expensive but well understood (linear, FFT, etc)

2. Division and slicing

(67)

higher dimensional functions

Gaussian convolution

division

slicing

Low-pass filter Low-pass filter

Almost only low freq.

High freq.

negligible Almost only

low freq.

High freq.

negligible

w i w

(68)

higher dimensional functions

Gaussian convolution

w i w

D O W N S A M P L E

Almost no information loss Almost no information loss

(69)

Fast Convolution by Downsampling

• Downsampling cuts frequencies above Nyquist limit

– Less data to process – But induces error

• Evaluation of the approximation

– Precision versus running time – Visual accuracy

(70)

Accuracy versus Running Time

• Finer sampling increases accuracy.

• More precise than previous work.

ampling

PSNR as function of Running Time

Digital photograph 1200 × 1600

(71)

input exact BF our result prev. work

1200 × 1600

• Comparison with previous work [Durand 02]

– running time = 1s for both techniques

0

difference 0.1

with exact computation (intensities in [0:1])

Visual Results

(72)

Conclusions

Practical gain

• Interactive running time

Theoretical gain

• Link with linear filters

higher dimension Ö “better” computation

higher dimension Ö “better” computation

(73)

Two-scale Tone Management for Photographic Look

Soonmin Bae, Sylvain Paris, and Frédo Durand MIT CSAIL

(74)

Ansel Adams

(75)

An Amateur Photographer

(76)

A Variety of Looks

(77)

Goals

• Control over photographic look

• Transfer “look” from a model photo

For example, we want

with the look of

(78)

Aspects of Photographic Look

• Subject choice

• Framing and composition Î Specified by input photos

• Tone distribution and contrast ÎModified based on model photos

Input

(79)

Tonal Aspects of Look

Ansel Adams Kenro Izu

(80)

Tonal aspects of Look - Global Contrast

(81)

Tonal aspects of Look - Local Contrast

Variable amount of texture Texture everywhere

Ansel Adams Kenro Izu

(82)

Overview

Model

(83)

Overview

Local contrast Global contrast

Result

• Separate global and local contrast

Input Image

(84)

Split

Global contrast

Input Image

Careful

combination

Post- process

Overview

(85)

Split

Global contrast

Input Image

Result

Careful

combination

Post- process

Overview

Local contrast

(86)

Split Global vs. Local Contrast

• Naïve decomposition: low vs. high frequency

– Problem: introduce blur & halos

Halo Blur

(87)

Bilateral Filter

• Edge-preserving smoothing [Tomasi 98]

• We build upon tone mapping [Durand 02]

After bilateral filtering Residual after filtering

Global contrast Local contrast

(88)

Bilateral Filter

• Edge-preserving smoothing [Tomasi 98]

• We build upon tone mapping [Durand 02]

BASE layer DETAIL layer

(89)

Global contrast

Input Image

Result

Careful

combination

Post- process Bilateral

Filter

Local contrast

(90)

Global contrast

Input Image

Careful

combination

Post- process Bilateral

Filter

(91)

Global Contrast

• Intensity remapping of base layer

Input base Input intensity After remapping Remapped

intensity

(92)

Global Contrast (Model Transfer)

• Histogram matching

– Remapping function given input and model histogram Model

base

(93)

Local contrast Global contrast

Input Image

Result

Careful

combination

Post- process Bilateral

Filter

Intensity matching

(94)

Global contrast

Input Image

Careful

combination

Post- process Bilateral

Filter

Intensity matching

(95)

Local Contrast: Detail Layer

• Uniform control:

– Multiply all values in the detail layer

Input Base + 3 × Detail

(96)

The amount of local contrast is not uniform

Smooth region

(97)

Local Contrast Variation

• We define “textureness”: amount of local contrast

– at each pixel based on surrounding region

Smooth region

Ö Low textureness

Textured region

Ö High textureness

(98)

“Textureness”: 1D Example

Textured region

Ö High textureness Smooth region

Ö Small high-frequency Textured region

Ö Large high-frequency Previous work:

Low pass of |H|

[Li 05, Su 05]

Low pass of |H|

[Li 05, Su 05]

(99)

Textureness

Input Textureness

(100)

Textureness Transfer

Step 1:

Histogram transfer

Hist. transfer Input

Input textureness textureness

Desired Desired textureness textureness Model

Model textureness textureness

(101)

Local contrast Global contrast

Input Image

Result

Careful

combination

Post- process Bilateral

Filter

Intensity matching

Textureness matching

(102)

Global contrast

Input Image

Careful

combination

Post- process Bilateral

Filter

Intensity matching

Textureness

(103)

A Non Perfect Result

• Decoupled and large modifications (up to 6x)

ÎLimited defects may appear

input (HDR)

result after

global and local adjustments

(104)

Intensity Remapping

• Some intensities may be outside displayable range.

Î Compress histogram to fit visible range.

(105)

Preserving Details

1. In the gradient domain:

– Compare gradient amplitudes of input and current – Prevent extreme reduction & extreme increase

2. Solve the Poisson equation.

corrected result remapped

intensities initial

result

(106)

Effect of Detail Preservation

uncorrected result corrected result

(107)

Local contrast Global contrast

Input Image

Result

Post- process Bilateral

Filter

Intensity matching

Textureness matching

Constrained Poisson

(108)

Global contrast

Input Image

Bilateral Filter

Intensity matching

Textureness

Constrained Poisson

Post- process

(109)

Additional Effects

• Soft focus (high frequency manipulation)

• Film grain (texture synthesis [Heeger 95])

• Color toning (chrominance = f (luminance))

before effects

after effects model

(110)

Intensity matching

Bilateral Filter

Global contrast

Input Image

Textureness

Constrained Poisson

Soft focus Toning

Grain

(111)

Intensity matching

Bilateral Filter

Local contrast Global contrast

Input Image

Result

Textureness matching

Constrained Poisson

Soft focus Toning

Grain

Recap

(112)

Results

User provides input and model photographs.

Î Our system automatically produces the result.

Running times:

– 6 seconds for 1 MPixel or less

(113)

Input Model

Result

(114)

Input

Result

(115)

Input Model

Result

(116)

Input

Input ModelModel

Snapshot

Snapshot, Alfred Stieglitz, Alfred Stieglitz

Comparison with Naïve Histogram Matching

(117)

Input Input

Our Result Our Result

Model Model

Clearing Winter Storm, Ansel Adams

Histogram Matching Histogram Matching

Comparison with Naïve Histogram Matching

Local contrast too low

(118)

Color Images

• Lab color space: modify only luminance

Input

Input OutputOutput

(119)

Limitations

• Noise and JPEG artifacts

– amplified defects

• Can lead to unexpected

results if the image content is too different from the model

– Portraits, in particular, can suffer

(120)

Conclusions

• Transfer “look” from a model photo

• Two-scale tone management

– Global and local contrast

– New edge-preserving textureness

(121)

Video Enhancement Using

Per Pixel Exposures (Bennett, 06)

From this video:

ASTA: Adaptive Spatio-S TemporalT

Accumulation Filter

(122)

Joint bilateral filtering

(123)

Flash / No-Flash Photo Improvement (Petschnigg04) (Eisemann04)

Merge best features: warm, cozy candle light (no-flash) low-noise, detailed flash image

(124)

Overview

Remove noise + details from image A,

Keep as image A Lighting

--- No-flash

Basic approach of both flash/noflash papers

(125)

Petschnigg:

• Flash

(126)

Petschnigg:

• No Flash,

(127)

Petschnigg:

• Result

(128)

Our Approach

Registration Registration

(129)

Our Approach

Decomposition

(130)

Decomposition

Color / Intensity:

= *

(131)

Our Approach

Decomposition

(132)

Our Approach

Decoupling

(133)

Decoupling

• Lighting : Large-scale variation

• Texture : Small-scale variation

Texture Lighting

• Lighting : Large-scale variation

• Texture : Small-scale variation

Lighting

(134)

Large-scale Layer

• Bilateral filter – edge preserving filter

Smith and Brady 1997; Tomasi and Manducci 1998; Durand et al. 2002

(135)

Large-scale Layer

• Bilateral filter

(136)

Cross Bilateral Filter

• Similar to joint bilateral filter by Petschnigg et al.

• When no-flash image is too noisy

• Borrow similarity from flash image

¾ edge stopping from flash image

• See detail in paper

(137)

Detail Layer

Intensity Large-scale

/ =

Detail

Recombination: Large scale * Detail = Intensity

(138)

Recombination

* =

(139)

Recombination

Intensity Result

Color Flash

* ~ ~

Result

Recombination: Intensity * Color = Original

shadows

(140)

Our Approach

(141)

Our Approach

Shadow

Detection/Treatment

(142)

Results

No-flash

(143)

Joint bilateral upsampling

(144)

Joint bilateral upsampling

(145)

Joint bilateral upsampling

(146)

Joint bilateral upsampling

(147)

Joint bilateral upsampling

(148)

Joint bilateral upsampling

(149)

Joint bilateral upsampling

(150)

Joint bilateral upsampling

(151)

References

• Patrick Perez, Michel Gangnet, Andrew Blake, Poisson Image Editing, SIGGRAPH 2003.

• Dani Lischinski, Zeev Farbman, Matt Uytendaelle and Richard

Szeliski. Interactive Local Adjustment of Tonal Values. SIGGRAPH 2006.

• Carsten Rother, Andrew Blake, Vladimir Kolmogorov, GrabCut - Interactive Foreground Extraction Using Iterated Graph Cuts, SIGGRAPH 2004.

• Aseem Agarwala, Mira Dontcheva, Maneesh Agrawala, Steven

Drucker, Alex Colburn, Brian Curless, David H. Salesin, Michael F.

Cohen, Interactive Digital Photomontage, SIGGRAPH 2004.

• Sylvain Paris and Fredo Durand. A Fast Approximation of the Bilateral Filter using a Signal Processing Approach. ECCV 2006.

• Soonmin Bae, Sylvain Paris and Fredo Durand. Two-scale Tone Management for Photographic Look. SIGGRAPH 2006.

Figure

Updating...

References

Related subjects :