Bilateral Filters
Digital Visual Effectsg Yung-Yu Chuang
with slides by Fredo Durand, Ramesh Raskar, Sylvain Paris, Soonmin Bae
Bilateral filtering
Input
Input Log(Intensity) Log(Intensity) Bilateral Smoothing Bilateral Smoothing Gaussian
Gaussian Smoothing Smoothing
[Ben Weiss, Siggraph 2006]
[Ben Weiss, Siggraph 2006]
Image Denoising g g
noisy image naïve denoising Gaussian blur
better denoising edge preserving filter Gaussian blur edge-preserving filter
Smoothing an image without blurring its edges Smoothing an image without blurring its edges.
A Wide Range of Options g p
• Diffusion, Bayesian, Wavelets… Diffusion, Bayesian, Wavelets…
– All have their pros and cons.
Bil t l filt
• Bilateral filter
– not always the best result [Buades 05] but often good – easy to understand, adapt and set up
Basic denoising
Noisy input Median 5x5
Basic denoising
Noisy input Bilateral filter 7x7 window
Tone Mapping
[D d 02]
Tone Mapping
[D d 02]
[Durand 02]
[Durand 02]
HDR input
Tone Mapping
[D d 02]
Tone Mapping
[D d 02]
[Durand 02]
[Durand 02]
output
Photographic Style Transfer
[B 06]
Photographic Style Transfer
[B 06]
[Bae 06]
[Bae 06]
input
Photographic Style Transfer
[B 06]
Photographic Style Transfer
[B 06]
[Bae 06]
[Bae 06]
output
Cartoon Rendition
[Winnemöller 06]
Cartoon Rendition
[Winnemöller 06]
[Winnemöller 06]
[Winnemöller 06]
input
Cartoon Rendition
[Winnemöller 06]
Cartoon Rendition
[Winnemöller 06]
[Winnemöller 06]
[Winnemöller 06]
6 papers at 6 papers at p p
SIGGRAPH’07 p p
SIGGRAPH’07 SIGGRAPH 07 SIGGRAPH 07
output
Gaussian Blur
per-pixel multiplication
input
*
outputaverage average
input
box average
Gaussian blur
Equation of Gaussian Blur Equation of Gaussian Blur
Same idea: weighted average of pixels.
G I
I
GB [ ]
p || p q ||
qnormalized
S q
normalized Gaussian function 1
0
Gaussian Profile
G (x) 1 exp x2 Gaussian Profile
pixel
2 2 2 exp
)
(
x G
weight
pixel position unrelated
pixels unrelated
pixels
uncertain pixels uncertain
pixels
related pixels
Spatial Parameter Spatial Parameter
S
I G
I GB
q
q
p
|| p q ||
]
[
inputsize of the window
S q
small large
small large
limited smoothing strong smoothing
How to set
• Depends on the application.
• Common strategy: proportional to image size
– e.g. 2% of the image diagonal
– property: independent of image resolution
Properties of Gaussian Blur
• Weights independent of spatial location
– linear convolution
– well-known operation
– efficient computation (recursive algorithm, FFT…)
Properties of Gaussian Blur
• Does smooth images
B h h
input
• But smoothes too much:
edges are blurred.
– Only spatial distance matters – No edge term
output
S
I G
I GB
q
q
p
|| p q ||
]
[
space
S
q space
Blur Comes from Averaging across Edges
*
input output
*
*
S G i k l h
Same Gaussian kernel everywhere.
Bilateral Filter No Averaging across Edges
[Aurich 95 Smith 97 Tomasi 98][Aurich 95, Smith 97, Tomasi 98]*
input output
*
*
Th k l h d d th i t t
The kernel shape depends on the image content.
Bilateral Filter Definition Bilateral Filter Definition
not new new
new
Same idea: weighted average of pixels.
S
I I
I G
W G I
BF
q
q q
p
p
1 || p q || | |
]
[
s ri ht i ht
li ti
W
p qSspace weight range weight I
normalization factor
Illustration a 1D Image
• 1D image = line of pixelsg p
• Better visualized as a plot
pixel intensity
pixel position pixel position
Gaussian Blur and Bilateral Filter
Gaussian blur
p
q GB[I]p
G ||p q || Iqspace space
q
S q
q
p
Bilateral filter
[Aurich 95 Smith 97 Tomasi 98]
p
G G I I II
BF 1 || || | |
] [
[Aurich 95, Smith 97, Tomasi 98]
rap
space range normalization
S
I I
I G
W G I
BF
q
q q
p p
p 1 || p q || | |
]
[ s r
ange
q
space
Bilateral Filter on a Height Field
G G I I I
I W
BF p 1 || p q || | p q | q
]
[
s rWp qS p q q
p s r
output input
reproduced from [Durand 02]
Space and Range Parameters
1
S
I I
I G
W G I
BF
q
q q
p p
p
1 || p q || | |
]
[
s r• space : spatial extent of the kernel size of
p q
• space s : spatial extent of the kernel, size of the considered neighborhood.
• range r : “minimum” amplitude of an edge
Influence of Pixels
Only pixels close in space and in range are considered.
space
range range
p
p
= 0 1 = 0 25
r =
(G i bl )
Exploring the Parameter Space Exploring the Parameter Space
r = 0.1 r = 0.25 (Gaussian blur)
s = 2
input
s = 6
s = 18
= 0 1 = 0 25
r =
(G i bl )
Varying the Range Parameter Varying the Range Parameter
r = 0.1 r = 0.25 (Gaussian blur)
s = 2
input
s = 6
s = 18
input
r= 0.1
r= 0.25
r=
(G i bl )
(Gaussian blur)
= 0 1 = 0 25
r =
(G i bl )
Varying the Space Parameter Varying the Space Parameter
r = 0.1 r = 0.25 (Gaussian blur)
s = 2
input
s = 6
s = 18
input
s= 2
s= 6
s= 18
How to Set the Parameters
Depends on the application. For instance:
• space parameter: proportional to image size
– e.g., 2% of image diagonal
• range parameter: proportional to edge amplitude
– e.g., mean or median of image gradients
• independent of resolution and exposure
Iterating the Bilateral Filter
] [I
BF
I
(n 1) BF [ I
(n)] I
• Generate more piecewise-flat images
• Often not needed in computational photo, but could be useful for applications such as NPR.pp
input
1 iteration
2 iterations
4 iterations
Advantages of Bilateral Filter
• Easy to understand
W i ht d f b i l – Weighted mean of nearby pixels
• Easy to adapt
– Distance between pixel values
• Easy to set upEasy to set up
– Non-iterative
Hard to Compute
• Nonlinear
S
I I
I G
W G I
BF
q
q q
p p
p 1 || p q || | |
]
[ s r
• Complex, spatially varying kernels
Cannot be precomputed no FFT
S p q
– Cannot be precomputed, no FFT…
• Brute-force implementation is slow > 10min
But Bilateral Filter is Nonlinear
• Slow but some accelerations exist:
– [Elad 02]: Gauss-Seidel iterations
• Only for many iterations
– [Durand 02, Weiss 06]: fast approximation
• No formal understanding of accuracy versus speed
• [Weiss 06]: Only box function as spatial kernel
A Fast Approximation f th Bil t l Filt
of the Bilateral Filter
using a Signal Processing using a Signal Processing
Approach Approach
Sylvain Paris and Frédo Durand
Computer Science and Artificial Intelligence Laboratory Massachusetts Institute of Technology
Definition of Bilateral Filter Definition of Bilateral Filter
• [Smith 97, Tomasi 98]
I t R lt
• Smoothes an image and preserves edges
Input Result
• Weighted average of neighbors
• Weights
– Gaussian on space distance – Gaussian on range distance – sum to 1
space range
space range
Contributions
Li k ith
linear filtering
• Link with
linear filtering
•
Fast
andaccurate
approximationIntuition on 1D Signal
BF
Basic idea
1D Gaussians
G G I I I
I W
BF p 1 q;p, s q ; p , r q ]
[
Wp qS q p q
p
Basic idea
1D Gaussians
G G I I I
I W
BF p 1 q;p, s q ; p , r q ]
[
Wp qS q p q
p
1
]
[I G I I I
BF
2D Gaussians
' '
,
' , r
s , ,
,
; ,
] [
S I
I I
I I
W G I
BF
q
q
q
q p
q p
p q p
a special 2D i
2D image
Intuition on 1D Signal
BF
Intuition on 1D Signal
Weighted Average of Neighbors
p
Weighted Average of Neighbors
weights p
weights applied to pixels
• Near and similar pixels have influence.
• Far pixels have no influence.
• Pixels with different value have no influence.
Link with Linear Filtering
1 Handling the Division
p
1. Handling the Division
p
sum of weights
Handling the division with a
projective space
.Formalization: Handling the Division g
• Normalizing factor as homogeneous coordinate
• Normalizing factor as homogeneous coordinate
• Multiply both sides by
Formalization: Handling the Division
with Wq=1
• Similar to homogeneous coordinates in projective space
in projective space
• Division delayed until the end
• Division delayed until the end
• Next step: Adding a dimension to make a
• Next step: Adding a dimension to make a convolution appear
Link with Linear Filtering
2 Introducing a Convolution space: 1D Gaussian
range: 1D Gaussian
p
2. Introducing a Convolution range: 1D Gaussian combination: 2D Gaussian
p
q
space range
Link with Linear Filtering
2 Introducing a Convolution space: 1D Gaussian
range: 1D Gaussian
p
2. Introducing a Convolution range: 1D Gaussian combination: 2D Gaussian
p
q
space x range
Corresponds to a 3D Gaussian on a 2D image.
Link with Linear Filtering
2 Introducing a Convolution 2. Introducing a Convolution
black = zero sum all values
G i
space-range Gaussian
sum all values multiplied by kernel convolution
Link with Linear Filtering
2 Introducing a Convolution 2. Introducing a Convolution
result of the convolution result of the convolution
G i
space-range Gaussian
Link with Linear Filtering
2 Introducing a Convolution 2. Introducing a Convolution
result of the convolution result of the convolution
G i
space-range Gaussian
higher dimensional functions
w i w
w i w
Gaussian convolution
division
slicing
Reformulation: Summaryy
1. Convolution in higher dimension
• expensive but well understood (linear, FFT, etc)
2. Division and slicing 2. Division and slicing
• nonlinear but simple and pixel-wise
Exact reformulation
Exact reformulation
higher dimensional functions
w i w
w i w
Gaussian convolution
Low-pass filter
division
slicing
higher dimensional functions
w i w
w i w
D O W N S A M P L E
Gaussian convolution
U P S A M P L E
division
slicing
Fast Convolution by Downsampling
• Downsampling cuts frequencies
• Downsampling cuts frequencies above Nyquist limit
Less data to process – Less data to process – But induces error
• Evaluation of the approximation
– Precision versus running time – Visual accuracy
Accuracy versus Running Time
• Finer sampling increases accuracy.
• More precise than previous work.p p
PSNR as function of Running Time
Digital photograph photograph 1200 1600
Straightforward Straightforward implementation is over 10 minutes.
Visual Results
• Comparison with previous work [Durand 02]
– running time = 1s for both techniques
i t t BF lt k
1200 1600 – running time = 1s for both techniques
input exact BF our result prev. work
difference 0.1
difference with exact computation (intensities in [0:1]) 0
Conclusions
hi h di i “b tt ” t ti
higher dimension “better” computation
Practical gain Theoretical gain
• Interactive running time
• Visually similar results
• Link with linear filters
• Separation linear/nonlinear
• Visually similar results
• Simple to code (100 lines)
• Separation linear/nonlinear
• Signal processing framework
Two-scale Tone Management Two-scale Tone Management
for Photographic Look g p
Soonmin Bae, Sylvain Paris, and Frédo Durand MIT CSAIL
MIT CSAIL
Ansel Adams
Ansel Adams, Clearing Winter Storm
An Amateur Photographer
A Variety of Looks
Goals
• Control over photographic look
• Transfer “look” from a model photop
For example For example, we want
with the look of
Aspects of Photographic Look
• Subject choice
• Framing and compositiong p
Specified by input photos
Input
• Tone distribution and contrast
Input
Modified based on model photos
Model
Tonal Aspects of Look
Ansel Adams Kenro Izu
Tonal aspects of Look - Global Contrast
Ansel Adams Kenro Izu
High Global Contrast Low Global Contrast
Tonal aspects of Look - Local Contrast
Ansel Adams Kenro Izu
Variable amount of texture Texture everywhere
Overview
Model
Input Image Result
Input Image Result
• Transfer look between photographsTransfer look between photographs
– Tonal aspects
Overview
Global contrast
Result Input
Image
Local contrast l b l d l l
• Separate global and local contrast
Global contrast
Overview
Split Careful
combination
P t
Input Image
Post- process
Local contrast
Result
Global contrast
Overview
Split Careful
combination
P t
Input Image
Post- process
Local contrast
Result
Split Global vs. Local Contrast
• Naïve decomposition: low vs. high frequencyp g q y
– Problem: introduce blur & halos
Halo Blur
Low frequency High frequency
Low frequency High frequency
Global contrast Local contrast
Bilateral Filter
• Edge-preserving smoothing [Tomasi 98]g p g g [ ]
• We build upon tone mapping [Durand 02]
After bilateral filtering Residual after filtering After bilateral filtering Residual after filtering
Global contrast Local contrast
Bilateral Filter
• Edge-preserving smoothing [Tomasi 98]g p g g [ ]
• We build upon tone mapping [Durand 02]
BASE layer DETAIL layer
After bilateral filtering Residual after filtering After bilateral filtering Residual after filtering
Global contrast Local contrast
Global contrast
Bil t l Careful
combination
P t Bilateral
Filter
Input Image
Post- process
Local contrast
Result
Global contrast
Bil t l Careful
combination
P t Bilateral
Filter
Input Image
Post- process
Local contrast
Result
Global Contrast
• Intensity remapping of base layer
Remapped intensity
Input base Input intensity After remapping
Global Contrast (Model Transfer) ( )
• Histogram matching
• Histogram matching
– Remapping function given input and model histogram input and model histogram Model
base
Input Output
Input base
Output base
Global contrast
Intensity Intensity matching
Bil t l Careful
combination
P t Bilateral
Filter
Input Image
Post- process
Local contrast
Result
Global contrast
Intensity Intensity matching
Bil t l Careful
combination
P t Bilateral
Filter
Input Image
Post- process
Local contrast
Result
Local Contrast: Detail Layer
• Uniform control:Uniform control:
– Multiply all values in the detail layer
I t B 3 D t il
Input Base + 3 Detail
The amount of local contrast is not uniform
is not uniform
Smooth region
Textured region Textured region
Local Contrast Variation
• We define “textureness”: amount of local contrast
– at each pixel based on surrounding region
Smooth region
Low textureness
Textured region Textured region
High textureness
“ Textureness”: 1D Example
Textured region
Smooth region Textured region
Large high-frequency Previous work:
Low pass of |H|
Low pass of |H|
Textured region
Hi h t t g
Small high-frequency [Li 05, Su 05][Li 05, Su 05]
High textureness
Smooth region L t t
Input signal
Low textureness
High frequency H Amplitude |H| Edge-preserving Input signal High frequency H Amplitude |H| Edge preserving
filter
Textureness
Input Textureness
Input Textureness
Textureness Transfer
Step 1:
f
Model Model Histogram transfer texturenesstextureness
Hist transfer Input
Input t t
t t
Desired Desired t t
t t Hist. transfer
textureness
textureness texturenesstextureness
x 0.5
Step 2:
x 2.7
Scaling detail layer (per pixel) to match
desired textureness
x 4.3
Input detail Output detail desired textureness
Global contrast
Intensity Intensity matching
Bil t l Careful
combination
P t Bilateral
Filter
Input Image
Post- process
Textureness matching
Local contrast
Result
Global contrast
Intensity Intensity matching
Bil t l Careful
combination
P t Bilateral
Filter
Input Image
Post- process
Textureness matching
Local contrast
Result
A Non Perfect Result
• Decoupled and large modifications (up to 6x)
Limited defects may appear
input (HDR)
result after
global and local adjustments
Intensity Remapping
S i i i b id di l bl
• Some intensities may be outside displayable range.
Compress histogram to fit visible range.
corrected result remapped
intensities initial
result
Preserving Details
1. In the gradient domain:
C di t lit d f i t d t
– Compare gradient amplitudes of input and current – Prevent extreme reduction & extreme increase
l h
2. Solve the Poisson equation.
corrected result remapped
intensities initial
result
Effect of Detail Preservation
uncorrected result corrected result
Global contrast
Intensity Intensity matching
Bil t l
P t Bilateral
Filter
Constrained Poisson
Input Image
Post- process
Textureness matching
Local contrast
Result
Global contrast
Intensity Intensity matching
Bil t l Bilateral Filter
Constrained Poisson
Input P Image
Post- process
Textureness matching
Local contrast
Result
Additional Effects
model• Soft focus (high frequency manipulation)
• Film grain (texture synthesis [Heeger 95])
• Film grain (texture synthesis [Heeger 95])
• Color toning (chrominance = f (luminance))
before after
before
effects effects
Intensity
Global contrast
Intensity matching
Bil t l Bilateral Filter
Constrained Poisson
S ft f
Input Image
Soft focus Toning
Grain Textureness
matching
Local contrast
Result
Intensity
Global contrast
Recap
Intensity matching
Bil t l Bilateral Filter
Constrained Poisson
S ft f
Input Image
Soft focus Toning
Grain Textureness
matching
Local contrast
Result
Results
U id i d d l h h
User provides input and model photographs.
Our system automatically produces the result.
Running times:
Running times:
– 6 seconds for 1 MPixel or less 23 seconds for 4 MPixels
– 23 seconds for 4 MPixels
multi-grid Poisson solver and fast bilateral filter [Paris
06]]
Input Model
Result
Input
Result
Input Model
Result
Comparison with Naïve Histogram Matching
Input
Input ModelModel
Snapshot
Snapshot, Alfred Stieglitz, Alfred Stieglitz
p
p SnapshotSnapshot, Alfred Stieglitz, Alfred Stieglitz
Our result Our result Naïve Histogram Matching
Naïve Histogram Matching Local contrast, sharpness unfaithful
Comparison with Naïve Histogram Matching
Input Input
Model Model
Clearing Winter Storm, Ansel Adams
p p
Our Result Our Result Histogram Matching
Histogram Matching
Local contrast too low
Color Images
• Lab color space: modify only luminance
Input
Input OutputOutput
Limitations
• Noise and JPEG artifacts
– amplified defectsamplified defects
• Can lead to unexpected
• Can lead to unexpected
results if the image content is too different from the model too different from the model
– Portraits, in particular, can suffer
suffer
Conclusions
T f “l k” f d l h
• Transfer “look” from a model photo
• Two-scale tone management
– Global and local contrastGlobal and local contrast
– New edge-preserving textureness – Constrained Poisson reconstructionConstrained Poisson reconstruction – Additional effects
Video Enhancement Using
Per Pixel Exposures (Bennett 06) Per Pixel Exposures (Bennett, 06)
From this video:
ASTA: Adaptive SS ti
SSpatio- T
Temporal
A l ti Filt Accumulation Filter
Joint bilateral filtering
Flash / No-Flash Photo Improvement (Petschnigg04) (Eisemann04)
(Petschnigg04) (Eisemann04)
Merge best features: warm, cozy candle light (no-flash)g , y g ( ) low-noise, detailed flash image
Overview
Basic approach of both flash/noflash papers
Remove noise + details
Basic approach of both flash/noflash papers
Remove noise + details from image A,
Keep as image A Lighting
No-flash ---
Obtain noise-free details
o as
Obtain noise free details from image B,
Discard Image B Lighting Result
Petschnigg:
• Flash
Petschnigg:
• No Flash,
Petschnigg:
• Result
Our Approach
Registration Registration
Our Approach
Decomposition
Decomposition
Color / Intensity:
= *
original intensity color
Our Approach
Decomposition
Our Approach
Decoupling
Decoupling
• Lighting : Large-scale variation
T S ll l i i
• Texture : Small-scale variation
Texture Lighting
Lighting
Large-scale Layer
• Bilateral filter – edge preserving filter
• Bilateral filter – edge preserving filter
Smith and Brady 1997; Tomasi and Manducci 1998; Durand et al. 2002
I t O t t
Input Output
Large-scale Layer
• Bilateral filter
Cross Bilateral Filter
• Similar to joint bilateral filter by Petschnigg et al
al.
• When no-flash image is too noisyg y
• Borrow similarity from flash image
edge stopping from flash image
Bilateral Cross Bilateral
Detail Layer
/ =
/ =
Intensity Large-scale Detail
Recombination: Large scale * Detail = Intensity
Recombination: Large scale * Detail = Intensity
Recombination
* =
* =
Large-scale Detail Intensity
No-flash Flash Result
Recombination: Large scale * Detail = Intensity
Recombination: Large scale * Detail = Intensity
Recombination
shadows shadows
** ~ ~
Intensity Color Result
Result Flash
Recombination: Intensity * Color = Original
Our Approach
Our Approach
Shadow
Detection/Treatment Detection/Treatment
Results
No-flash
Result Flash