### Computational Photography

Digital Visual Effects, Spring 2007
*Yung-Yu Chuang*

2007/5/22

*with slides by Fredo Durand, Ramesh Raskar, Sylvain Paris, Soonmin Bae*

**Computational photography**

**wikipedia:**

**Computational photography refers broadly to **
computational imaging techniques that enhance
or extend the capabilities of digital photography.

The output of these techniques is an ordinary photograph, but one that could not have been taken by a traditional camera.

**What is computational photography**

• Convergence of image processing, computer vision, computer graphics and photography

• Digital photography:

– Simply mimics traditional sensors and recording by digital technology

– Involves only simple image processing

• Computational photography

– More elaborate image manipulation, more computation

– New types of media (panorama, 3D, etc.)

– Camera design that take computation into account

**Computational photography**

• One of the most exciting fields.

• Symposium on Computational Photography and Video, 2005

• Full-semester courses in MIT, CMU, Stanford, GaTech, University of Delaware

• A new book by Raskar and Tumblin is coming out in SIGGRAPH 2007.

**Siggraph 2006 Papers (16/86=18.6%)**

Hybrid Images Drag-and-Drop Pasting

Two-scale Tone Management for Photographic Look Interactive Local Adjustment of Tonal Values Image-Based Material Editing

Flash Matting

Natural Video Matting using Camera Arrays Removing Camera Shake From a Single Photograph Coded Exposure Photography: Motion Deblurring Photo Tourism: Exploring Photo Collections in 3D AutoCollage

Photographing Long Scenes With Multi-Viewpoint Panoramas Projection Defocus Analysis for Scene Capture and Image Display Multiview Radial Catadioptric Imaging for Scene Capture Light Field Microscopy

Fast Separation of Direct and Global Components of a Scene Using High Frequency Illumination

**Siggraph 2007 Papers (23/108=21.3%)**

Image Deblurring with Blurred/Noisy Image Pairs Photo Clip Art

Scene Completion Using Millions of Photographs

Soft Scissors: An Interactive Tool for Realtime High Quality Matting Seam Carving for Content-Aware Image Resizing

Detail-Preserving Shape Deformation in Image Editing Veiling Glare in High Dynamic Range Imaging

Do HDR Displays Support LDR content? A Psychophysical Evaluation Ldr2hdr: On-the-fly Reverse Tone Mapping of Legacy Video and Photographs Rendering for an Interactive 360-Degree Light Field Display

Multiscale Shape and Detail Enhancement from Multi-light Image Collections Post-Production Facial Performance Relighting Using Reflectance Transfer Active Refocusing of Images and Videos

Multi-aperture Photography

Dappled Photography: Mask-Enhanced Cameras for Heterodyned Light Fields and Coded Aperture Refocusing

Image and Depth from a Conventional Camera with a Coded Aperture Capturing and Viewing Gigapixel Images

Efficient Gradient-Domain Compositing Using Quadtrees Image Upsampling via Imposed Edges Statistics Joint Bilateral Upsampling

Factored Time-Lapse Video Computational Time-Lapse Video

Real-Time Edge-Aware Image Processing With the Bilateral Grid

**Scope**

• We can’t yet set its precise definition. The following are scopes of what researchers are exploring in this field.

– Record a richer visual experience

– Overcome long-standing limitations of conventional cameras

– Enable new classes of visual signal – Enable synthesis impossible photos

**Scope**

• Image formation

• Color and color perception

• Demosaicing

**Scope**

• Panoramic imaging

• Image and video registration

• Spatial warping operations

**Scope**

• High Dynamic Range Imaging

• Bilateral filtering and HDR display

• Matting

**Scope**

• Active flash methods

• Lens technology

• Depth and defocus

No-flash

Flash

our result

**Removing Photography Artifacts using Gradient **
**Projection and Flash-Exposure Sampling**

**Continuous flash**

Flash = 0.0

Flash = 0.3 Flash = 0.7 Flash = 1.4

Flash = 1.0

**Flash matting**

**Depth Edge Detection and Stylized **

**Rendering Using a Multi-Flash Camera** **Motion-Based Motion Deblurring**

**Removing Camera Shake from a **

**Single Photograph** Motion Deblurring using Fluttered Shutter

**Scope**

• Future cameras

• Plenoptic function and light fields

**Scope**

• Gradient image manipulation

**Scope**

• Taking great pictures

Art Wolfe Ansel Adams

**Scope**

• Non-parametric image synthesis, inpainting,

analogies

**Scope**

Motion analysis

**Image Inpainting**

**Object Removal by **

**Exemplar-Based Inpainting**

**Image Completion with ** **Structure Propagation**

**Lazy snapping** **Lazy snapping**

### • Pre-segmentation

### • Boundary Editing

**Grab Cut - Interactive Foreground **

**Extraction using Iterated Graph Cuts** **Image Tools**

• Gradient domain operations,

– Tone mapping, fusion and matting

• Graph cuts,

– Segmentation and mosaicing

• Bilateral and Trilateral filters,

– Denoising, image enhancement

**Gradient domain operators**

**Intensity Gradient in 1D**

*I(x)*
1

10^{5}

G(x) 1

10^{5}

Intensity Gradient

Gradient at x,

*G(x) = I(x+1)- I(x)*

Forward Difference

**Reconstruction from Gradients**

*I(x)*
1

10^{5}

Intensity

G(x) 1

10^{5}

Gradient

### ? ?

*For n intensity values, about n gradients *

**Reconstruction from Gradients**

*I(x)*
1

10^{5}

Intensity

G(x) 1

10^{5}

Gradient

1D Integration

*I(x) = I(x-1) + G(x)*
Cumulative sum

### ?

**1D case with constraints**

Seamlessly paste onto

Just add a linear function so that the boundary condition is respected

**Discrete 1D example: minimization**

• Copy to

• Min ((f_{2}-f_{1})-1)^{2}

• Min ((f_{3}-f_{2})-(-1))^{2}

• Min ((f_{4}-f_{3})-2)^{2}

• Min ((f_{5}-f_{4})-(-1))^{2}

• Min ((f6-f_{5})-(-1))^{2}

0 1 2 3 4 5 6

0

1 2 3 4 5 6 7 -1 -1 -1

+2 +1

0 1 2 3 4 5 6

0

1 2 3 4 5 6 7

? ? ? ?

### With

### f

_{1}

### =6

### f

_{6}

### =1

**1D example: minimization**

• Copy to

• Min ((f_{2}-6)-1)^{2 } ==> f_{2}^{2}+49-14f_{2}

• Min ((f_{3}-f_{2})-(-1))^{2} ==> f_{3}^{2}+f_{2}^{2}+1-2f_{3}f_{2} +2f_{3}-2f_{2}

• Min ((f_{4}-f_{3})-2)^{2} ==> f_{4}^{2}+f_{3}^{2}+4-2f_{3}f_{4} -4f_{4}+4f_{3}

• Min ((f_{5}-f_{4})-(-1))^{2} ==> f_{5}^{2}+f_{4}^{2}+1-2f_{5}f_{4} +2f_{5}-2f_{4}

• Min ((1-f_{5})-(-1))^{2} ==> f_{5}^{2}+4-4f_{5}

0 1 2 3 4 5 6

0

1 2 3 4 5 6 7 -1 -1 -1

+2 +1

0 1 2 3 4 5 6

0

1 2 3 4 5 6 7

? ? ? ?

**1D example: big quadratic**

• Copy to

• Min (f_{2}^{2}+49-14f_{2}

+ f_{3}^{2}+f_{2}^{2}+1-2f_{3}f_{2}+2f_{3}-2f_{2}
+ f_{4}^{2}+f_{3}^{2}+4-2f_{3}f_{4}-4f_{4}+4f_{3}
+ f_{5}^{2}+f_{4}^{2}+1-2f_{5}f_{4}+2f_{5}-2f_{4}
+ f_{5}^{2}+4-4f_{5})

Denote it Q

0 1 2 3 4 5 6

0

1 2 3 4 5 6 7 -1 -1 -1

+2 +1

0 1 2 3 4 5 6

0

1 2 3 4 5 6 7

? ? ? ?

**1D example: derivatives**

• Copy to

0 1 2 3 4 5 6

0

1 2 3 4 5 6 7 -1 -1 -1

+2 +1

0 1 2 3 4 5 6

0

1 2 3 4 5 6 7

? ? ? ?

**Min (f**_{2}^{2}**+49-14f**_{2}

**+ f**_{3}^{2}**+f**_{2}^{2}**+1-2f**_{3}**f**_{2}**+2f**_{3}**-2f**_{2}**+ f**_{4}^{2}**+f**_{3}^{2}**+4-2f**_{3}**f**_{4}**-4f**_{4}**+4f**_{3}**+ f**_{5}^{2}**+f**_{4}^{2}**+1-2f**_{5}**f**_{4}**+2f**_{5}**-2f**_{4}**+ f**_{5}^{2}**+4-4f**_{5}**) **

**Denote it Q**

**1D example: set derivatives to zero**

• Copy to

0 1 2 3 4 5 6

0

1 2 3 4 5 6 7 -1 -1 -1

+2 +1

0 1 2 3 4 5 6

0

1 2 3 4 5 6 7

? ? ? ?

==>

**1D example**

• Copy to

0 1 2 3 4 5 6

0

1 2 3 4 5 6 7 -1 -1 -1

+2 +1

0 1 2 3 4 5 6

0

1 2 3 4 5 6 7

**1D example: remarks**

• Copy to

• Matrix is sparse

• Matrix is symmetric

• Everything is a multiple of 2

– because square and derivative of square

• Matrix is a convolution (kernel -2 4 -2)

• Matrix is independent of gradient field. Only RHS is

• Matrix is a second derivative

0 1 2 3 4 5 6

0

1 23 4567 -1 -1 -1

+2 +1

0 1 2 3 4 5 6

0

123 4 567

**Grad X**

**Grad Y**

**Intensity Gradient in 2D**

Gradient at x,y as Forward Differences
*G*_{x}*(x,y) = I(x+1 , y)- I(x,y)*
*G*_{y}*(x,y) = I(x , y+1)- I(x,y)*
*G(x,y)* = *(G*_{x}*, G*_{y}*)*

**Grad X**

**Grad Y**

Integration2D

**Image Intensity Gradients in 2D**

### Sanity Check:

### Recovering Original Image

Solve Poisson Equation, 2D linear system

**Grad X**

**Grad Y**

**New Grad X**

**New Grad Y**

Integration2D

**Intensity Gradient Manipulation**

Gradient Processing

A Common Pipeline

Modify Gradients

**2D case with constraints**

• *Given vector field v (pasted gradient), find the *
*value of f in unknown region that optimize: *

Pasted gradient Mask

Background unknown region

**Poisson image editing**

**Problems with direct cloning**

From Perez et al. 2003

**Solution: clone gradient** **Result**

**Reduce big gradients**

• Dynamic range compression

• Fattal et al. 2002

**Seamless Image Stitching in the Gradient Domain**

• **Anat Levin, Assaf Zomet, Shmuel Peleg, and **
**Yair Weiss**

http://www.cs.huji.ac.il/~alevin/papers/eccv04-blending.pdf http://eprints.pascal-network.org/archive/00001062/01/tips05- blending.pdf

• Various strategies (optimal cut, feathering)

**Gradient tone mapping**

• Fattal et al. Siggraph 2002

Slide from Siggraph 2005 by Raskar (Graphs by Fattal et al.)

**Gradient attenuation**

From Fattal et al.

**Fattal et al. Gradient tone mapping** **Poisson Matting**

• Sun et al. Siggraph 2004

• Assume gradient of F & B is negligible

• Plus various image-editing tools to refine matte

**Interactive Local Adjustment of ** **Tonal Values**

Dani Lischinski, Zeev Farbman The Hebrew University

Matt Uyttendaele, Richard Szeliski Microsoft Research

**Background (1)**

Dodging

Burning brushes

### Darkroom

### Camera shutter ---Æ Photograph

⎧⎨

### Tool

⎩### Only!

### But, …

### It is tedious, time-consuming and painstaking!

**Background (2)**

• A large arsenal of adjustment tools

• Hard to master these tools

– To learn, use

• Tedious and time-consuming

– Professional ability, experienced skill – Too many layer masks

• Incapable in some requirements [Adobe Photoshop CS2, 2005]

**Background (2)**

[Adobe Photoshop CS2, 2005]

Result Layer mask

Original image

**Related Work: Tone Mapping Operators**

• Global operators

[Ward Larson et al. 1997; Reinhard et al. 2002; Drago et al. 2003]

– Usually fast

• Local operators

[Fattal et al. 2002; Reinhard et al. 2002;

Li et al. 2005] …

– Better at preserving local contrasts – Introduce visual artifacts sometimes

**Limitations of Tone Mapping Operators**

• Lack of direct local control

– Can’t directly manipulate a particular region

• Not guaranteed to converge to a subjectively satisfactory result

– Involves several trial-and-error iterations – Change the entire image each iteration

**Algorithm Overview**

1.Load a digital negative, a camera RAW file, an HDR radiance map, or an ordinary image

2.Indicate regions in the image that require adjusting

3.Experiment with the available adjustment

parameters until a satisfactory result is obtained in the desired regions

4.Iterate 2 and 3 until a satisfactory image

**An Example**

**Region Selection: Strokes and Brushes**

• Basic brush

• Luminance brush

weight=1, for the selected pixels in the brush;

weight=0, else

**Region Selection: Luminance Brush**

**Region Selection: Strokes and Brushes**

• Basic brush

• Luminance brush

• Lumachrome brush (chromaticity) –

• Over-exposure brush

• Under-exposure brush

**Constraint Propagation**

User strokes Adjusted exposure

**Image-guided Energy Minimization**

Data term + smoothing term

**Image-guided Energy Minimization**

data term + smoothing term

: log-luminance channel : sensitivity factor

: a small zero-division constant : a balance factor

Default:

**Standard Finite Differences** **Fast Approximate Solution**

Solved iteratively by [Saad 2003]

preconditioned conjugate gradients (PCG)

**Interactive Local Adjustment of ** **Tonal Value**

arg min ( )( ( ) ( ))2 ( , )

*f*

*f* = ^{⎧}⎨ *w* *f* −*g* +λ *h* ∇ ∇*f* *L* ^{⎫}⎬

⎩

### ∑ ∑

⎭**X** **X**

**x** **x** **x**

*f*

**Results**

**Graph cut**

**Graph cut**

• Interactive image segmentation using graph cut

• Binary label: foreground vs. background

• User labels some pixels

– similar to trimap, usually sparser

• Exploit

– Statistics of known Fg & Bg – Smoothness of label

• Turn into discrete graph optimization

– Graph cut (min cut / max flow)

**F**

**B**
**F**
**F** **F**
**F** **B**

**B**
**B**

**Energy function**

• Labeling: one value per pixel, F or B

• Energy(labeling) = data + smoothness

– Very general situation – Will be minimized

• Data: for each pixel

– Probability that this color belongs to F (resp. B) – Similar in spirit to Bayesian matting

• Smoothness (aka regularization):

per neighboring pixel pair

– Penalty for having different label – Penalty is downweighted if the two

pixel colors are very different – Similar in spirit to bilateral filter

**One labeling**
**(ok, not best)**

**Data**

**Smoothness**

**Data term**

• A.k.a regional term

(because integrated over full region)

• D(L)=

### Σ

_{i}-log h[L

_{i}](C

_{i})

• *Where i is a pixel *

L_{i} *is the label at i (F or B), *
C_{i} is the pixel value

h[L_{i}] is the histogram of the observed Fg
(resp Bg)

• Note the minus sign

**Hard constraints**

• The user has provided some labels

• The quick and dirty way to include

constraints into optimization is to replace the data term by a huge penalty if not respected.

• D(L_i)=0 if respected

• D(L_i) = K if not respected

– e.g. K=- #pixels

**Smoothness term**

• a.k.a boundary term, a.k.a. regularization

• S(L)=

### Σ

**{j, i} in N**B(C

_{i},C

_{j}) δ(L

_{i}-L

_{j})

• Where i,j are neighbors

– e.g. 8-neighborhood

(but I show 4 for simplicity)

• δ(L_{i}-L_{j}) is 0 if L_{i}=L_{j}, 1 otherwise

• B(C_{i},C_{j}) is high when C_{i} and C_{j} are similar, low if
there is a discontinuity between those two pixels

– e.g. exp(-||C_{i}-Cj||^{2}/2σ^{2})
– where σ can be a constant

or the local variance

• Note positive sign

**Optimization**

• E(L)=D(L)+λ S(L)

• λ is a black-magic constant

• Find the labeling that minimizes E

• In this case, how many possibilities?

– 2^{9} (512)

– We can try them all!

– What about megapixel images?

**Labeling as a graph problem**

• Each pixel = node

• Add two nodes F & B

• Labeling: link each pixel to either F or B F

B

**Desired result**

**Data term**

• Put one edge between each pixel and F & G

• Weight of edge = minus data term

– Don’t forget huge weight for hard constraints – Careful with sign

B F

**Smoothness term**

• Add an edge between each neighbor pair

• Weight = smoothness term

B F

**Min cut**

• Energy optimization equivalent to min cut

• Cut: remove edges to disconnect F from B

• Minimum: minimize sum of cut edge weight

B

F **cut**

**Min cut <=> labeling**

• In order to be a cut:

– For each pixel, either the F or G edge has to be cut

• In order to be minimal

– Only one edge label per pixel can be cut (otherwise could be added)

B

F **cut**

**Computing a multiway cut**

• With 2 labels: classical min-cut problem

– Solvable by standard flow algorithms

• polynomial time in theory, nearly linear in practice

– More than 2 terminals: NP-hard

*[Dahlhaus et al., STOC ‘92]*

• Efficient approximation algorithms exist

– Within a factor of 2 of optimal

– Computes local minimum in a strong sense

• even very large moves will not improve the energy

– Yuri Boykov, Olga Veksler and Ramin Zabih, Fast Approximate Energy Minimization via Graph Cuts, International Conference on Computer Vision, September 1999.

**Move examples**

Starting point

Red-blue swap move

Green expansion move

**GrabCut** **GrabCut**

**Interactive Foreground Extraction **
**Interactive Foreground Extraction **

**using Iterated Graph Cuts**
**using Iterated Graph Cuts**

Carsten

Carsten RotherRother Vladimir Kolmogorov Vladimir Kolmogorov

Andrew Blake Andrew Blake

**Microsoft Research Cambridge**
**Microsoft Research Cambridge--UKUK**

Agrawala et al, Digital Photomontage, Siggraph 2004

Source images Brush strokes Computed labeling

Composite

Brush strokes Computed labeling

**Graph Cuts for Segmentation and Mosaicing**

### Interactive Digital Photomontage

• Extended depth of field

### Interactive Digital Photomontage

• Relighting

### Interactive Digital Photomontage

**Bilateral filtering**

[Ben Weiss, Siggraph 2006]

[Ben Weiss, Siggraph 2006]

Input

Input Log(IntensityLog(Intensity) ) Bilateral Smoothing Bilateral Smoothing Gaussian

Gaussian Smoothing Smoothing

**Image Denoising**

noisy image naïve denoising

Gaussian blur better denoising edge-preserving filter

Smoothing an image without blurring its edges.

**A Wide Range of Options**

### • Diffusion, Bayesian, Wavelets…

– All have their pros and cons.

### • Bilateral filter

– not always the best result [Buades 05] but often good – easy to understand, adapt and set up

**Start with Gaussian filtering**

• Here, input is a step function + noise

output input

=

*J* *f* ^{⊗}

*I*

**Start with Gaussian filtering**

• Spatial Gaussian f

output input

=

*J*

*f*

^{⊗}

^{I}**Start with Gaussian filtering**

• Output is blurred

output input

*J*

=

^{f}^{⊗}

^{I}**Gaussian filter as weighted average**

• Weight of ξ depends on distance to x

)
,
(*x* ξ

*f* *I*(ξ)

output input

=
)
*(x*

*J*

### ∑

ξ

*x* *x*

### ξ

**The problem of edges**

• Here, “pollutes” our estimate J(x)

• It is too different

*x*

) (

### ξ

*I*

### ) *(x* *I*

)
,
(*x*ξ

*f* *I*(ξ)

=
)
*(x*

*J*

### ∑

ξ

output input

**Principle of Bilateral filtering**

• [Tomasi and Manduchi 1998]

• Penalty g on the intensity difference

=
)
*(x*

*J*

### ∑

*f*(

*x*,ξ)

*g*(

*I*(ξ)−

*I*(

*x*))

*I*(ξ) ) ξ

( 1

*x*
*k*

*x* ^{I} ^{(x} ^{)}

^{I}

^{(x}

### ) ( ξ *I*

output input

**Bilateral filtering**

• [Tomasi and Manduchi 1998]

• Spatial Gaussian f

=
)
*(x*

*J*

### ∑ *f* ( *x* , ξ )

^{g}^{(}

^{I}^{(}

^{ξ}

^{)}

^{−}

^{I}^{(}

^{x}^{))}

^{I}^{(}

^{ξ}

^{)}

) ξ

( 1

*x*
*k*

*x*

output input

**Bilateral filtering**

• [Tomasi and Manduchi 1998]

• Spatial Gaussian f

• Gaussian g on the intensity difference

=
)
*(x*

*J*

### ∑

*f*(

*x*,ξ)

*g*(

*I*(ξ)−

*I*(

*x*))

*I*(ξ) ) ξ

( 1

*x*
*k*

*x*

output input output input

**Normalization factor**

• [Tomasi and Manduchi 1998]

• k(x)=

=
)
*(x*

*J*

### ∑

*I*(ξ)

) ξ

( 1

*x*
*k*

*x*

)
,
(*x* ξ

*f* *g*(*I*(ξ)−*I*(*x*))

### ∑

ξ)
,
(*x* ξ

*f* *g*(*I*(ξ)−*I*(*x*))

output input

**Bilateral filtering is non-linear**

• [Tomasi and Manduchi 1998]

• The weights are different for each output pixel

=
)
*(x*

*J*

### ∑

*f*(

*x*,ξ)

*g*(

*I*(ξ)−

*I*(

*x*))

*I*(ξ) ) ξ

( 1

*x*
*k*

*x* *x*

**Many Applications based on Bilateral Filter**

Tone Mapping [Durand 02]

Virtual Video Exposure [Bennett 05]

### And many others…

Flash / No-Flash [Eisemann 04, Petschnigg 04]

[Petschnigg 04]

Tone Management [Bae 06]

**Advantages of Bilateral Filter**

• Easy to understand

– Weighted mean of nearby pixels

• Easy to adapt

– Distance between pixel values

• Easy to set up

– Non-iterative

**But Bilateral Filter is Nonlinear**

• Slow but some accelerations exist:

– [Elad 02]: Gauss-Seidel iterations

**• Only for many iterations**

– [Durand 02, Weiss 06]: fast approximation

**• No formal understanding of accuracy versus speed**

**• [Weiss 06]: Only box function as spatial kernel**

### A Fast Approximation of the Bilateral Filter using a Signal Processing

### Approach

Sylvain Paris and Frédo Durand

Computer Science and Artificial Intelligence Laboratory Massachusetts Institute of Technology

**Definition of Bilateral Filter**

• [Smith 97, Tomasi 98]

• Smoothes an image and preserves edges

• Weighted average of neighbors

• Weights

*– Gaussian on space distance*
*– Gaussian on range distance*
– sum to 1

space range

Input Result

**Contributions**

• Link with

**linear filtering**

•

**Fast **

and **accurate**

approximation
**Intuition on 1D Signal**

BF

**p**

**Intuition on 1D Signal**

**Weighted Average of Neighbors**

• Near and similar pixels have influence.

• Far pixels have no influence.

• Pixels with different value have no influence.

weights applied to pixels

**p**

**Link with Linear Filtering**

**1. Handling the Division**

sum of weights

Handling the division with a

**projective space**

.
**Formalization: Handling the Division**

• Normalizing factor as homogeneous coordinate

• Multiply both sides by

• Normalizing factor as homogeneous coordinate

• Multiply both sides by

**Formalization: Handling the Division**

• Similar to homogeneous coordinates in projective space

• Division delayed until the end

• Next step: Adding a dimension to make a convolution appear

*with W*** _{q}**=1

space range
**p**

**Link with Linear Filtering**

**2. Introducing a Convolution**

**q**

space: 1D Gaussian

× range: 1D Gaussian combination: 2D Gaussian space: 1D Gaussian

× range: 1D Gaussian combination: 2D Gaussian

**p**

**Link with Linear Filtering**

**2. Introducing a Convolution**

**q**

space: 1D Gaussian

×range: 1D Gaussian combination: 2D Gaussian space: 1D Gaussian

×range: 1D Gaussian combination: 2D Gaussian

space x range

Corresponds to a 3D Gaussian on a 2D image.

**Link with Linear Filtering**

**2. Introducing a Convolution**

space-range Gaussian black = zero

sum all values

sum all values multiplied by kernel Ö convolution

space-range Gaussian

result of the convolution

**Link with Linear Filtering**

**2. Introducing a Convolution**

**Link with Linear Filtering**

**2. Introducing a Convolution**

space-range Gaussian

result of the convolution

higher dimensional functions

Gaussian convolution

division

slicing

*w i* *w*

**Reformulation: Summary**

1. Convolution in higher dimension

• expensive but well understood (linear, FFT, etc)

2. Division and slicing

• nonlinear but simple and pixel-wise

**Exact reformulation** **Exact reformulation**

higher dimensional functions

Gaussian convolution

division

slicing

Low-pass filter Low-pass filter

Almost only low freq.

High freq.

negligible Almost only

low freq.

High freq.

negligible

*w i* *w*

higher dimensional functions

Gaussian convolution

division

slicing

*w i* *w*

**D O W N S A M P L E**

**U P S A M P L E**

Almost no information loss Almost no information loss

**Fast Convolution by Downsampling**

• Downsampling cuts frequencies above Nyquist limit

– Less data to process – But induces error

• Evaluation of the approximation

– Precision versus running time – Visual accuracy

**Accuracy versus Running Time**

• Finer sampling increases accuracy.

• More precise than previous work.

finer sampling

**PSNR as function of Running Time**

Digital photograph 1200 × 1600

Straightforward implementation is over 10 minutes.

input exact BF our result prev. work

1200 × 1600

• Comparison with previous work [Durand 02]

– running time = 1s for both techniques

0

difference 0.1

with exact computation (intensities in [0:1])

**Visual Results**

input exact BF our result prev. work

1200 × 1600

• Comparison with previous work [Durand 02]

– running time = 1s for both techniques

0

difference 0.1

with exact computation (intensities in [0:1])

**Visual Results**

input our result exact BF prev. work

1200 × 1600

• Comparison with previous work [Durand 02]

– running time = 1s for both techniques

0

difference 0.1

with exact computation (intensities in [0:1])

**Visual Results**

input exact BF our result

difference with exact computation (intensities in [0:1])

prev. work

1200 × 1600

• Comparison with previous work [Durand 02]

– running time = 1s for both techniques

0 0.1

**Visual Results**

input prev. work our result exact BF

1200 × 1600

• Comparison with previous work [Durand 02]

– running time = 1s for both techniques

0

difference 0.1

with exact computation (intensities in [0:1])

**Visual Results**

**Discussion**

• Higher dimension Ö advantageous formulation

– akin to Level Sets with topology – our approach: isolate nonlinearities

– dimension increase largely offset by downsampling

• Space-range domain already appeared

– [Sochen 98, Barash 02]: image as an embedded manifold – new in our approach: image as a dense function

**Conclusions**

**Practical gain**

• Interactive running time

• Visually similar results

• Simple to code (100 lines)

**Theoretical gain**

• Link with linear filters

• Separation linear/nonlinear

• Signal processing framework

### higher dimension Ö “better” computation higher dimension Ö “better” computation

**Two-scale Tone Management ** **for Photographic Look**

Soonmin Bae, Sylvain Paris, and Frédo Durand MIT CSAIL

**Ansel Adams**

*Ansel Adams, Clearing Winter Storm*

**An Amateur Photographer** **A Variety of Looks**

**Goals**

• Control over photographic look

• Transfer “look” from a model photo

For example,

we want

with the look of

**Aspects of Photographic Look**

• Subject choice

• Framing and composition Î Specified by input photos

• Tone distribution and contrast ÎModified based on model photos

Input

Model

**Tonal Aspects of Look**

Ansel Adams Kenro Izu

**Tonal aspects of Look - Global Contrast**

Ansel Adams Kenro Izu

**High Global Contrast** **Low Global Contrast**

**Tonal aspects of Look - Local Contrast**

**Variable amount of texture** **Texture everywhere**

Ansel Adams Kenro Izu

**Overview**

**Input Image****Result**

**Model**

• Transfer look between photographs

– Tonal aspects

**Overview**

**Local contrast****Global contrast**

**Result**

• Separate global and local contrast

**Input****Image**

**Split**

**Local contrast****Global contrast**

**Input****Image**

**Result**

**Careful**
**combination**

**Post-**
**process**

**Overview**

**Split**

**Global contrast**

**Input****Image**

**Result**

**Careful**
**combination**

**Post-**
**process**

**Overview**

**Local contrast**

**Split Global vs. Local Contrast**

• Naïve decomposition: low vs. high frequency

– Problem: introduce blur & halos

**Low frequency** **High frequency**

Halo Blur

**Global contrast****Local contrast**

**Bilateral Filter**

• Edge-preserving smoothing [Tomasi 98]

• We build upon tone mapping [Durand 02]

**After bilateral filtering** **Residual after filtering**

**Global contrast****Local contrast**

**Bilateral Filter**

• Edge-preserving smoothing [Tomasi 98]

• We build upon tone mapping [Durand 02]

**After bilateral filtering** **Residual after filtering**

**BASE layer****DETAIL layer**

**Global contrast****Local contrast**

**Global contrast**

**Input****Image**

**Result**

**Careful**
**combination**

**Post-**
**process**
**Bilateral**

**Filter**

**Local contrast****Local contrast**

**Global contrast**

**Global contrast**

**Input****Image**

**Result**

**Careful**
**combination**

**Post-**
**process**
**Bilateral**

**Filter**

**Global Contrast**

• Intensity remapping of base layer

Input base Input intensity After remapping Remapped

intensity

**Global Contrast (Model Transfer)**

• Histogram matching

– Remapping function given input and model histogram Model

base

Input base

Output base

**Local contrast****Global contrast**

**Input****Image**

**Result**

**Careful**
**combination**

**Post-**
**process**
**Bilateral**

**Filter**

**Intensity**
**matching**

**Local contrast**

**Local contrast**

**Global contrast**

**Input****Image**

**Result**

**Careful**
**combination**

**Post-**
**process**
**Bilateral**

**Filter**

**Intensity**
**matching**

**Local Contrast: Detail Layer**

• Uniform control:

– Multiply all values in the detail layer

Input Base + 3 × Detail

**The amount of local contrast ** **is not uniform**

Smooth region

Textured region

**Local Contrast Variation**

• We define “textureness”: amount of local contrast

– at each pixel based on surrounding region

Smooth region Ö Low textureness

Textured region Ö High textureness

Input signal

**“Textureness”: 1D Example**

Smooth region Ö Low textureness Textured region Ö High textureness

High frequency H Amplitude |H| Edge-preserving filter

Smooth region

Ö Small high-frequency Textured region

Ö Large high-frequency Previous work:

Low pass of |H|

[Li 05, Su 05]

Low pass of |H|

[Li 05, Su 05]

**Textureness**

Input Textureness

**Textureness Transfer**

Step 1:

Histogram transfer

**Hist. transfer**
**Input **

**Input **
**textureness**
**textureness**

**Desired **
**Desired **
**textureness**
**textureness**
**Model **

**Model **
**textureness**
**textureness**

x 0.5 x 2.7

x 4.3

Input detail Output detail Step 2:

Scaling detail layer (per pixel) to match

desired textureness

**Local contrast****Global contrast**

**Input****Image**

**Result**

**Careful**
**combination**

**Post-**
**process**
**Bilateral**

**Filter**

**Intensity**
**matching**

**Textureness**
**matching**

**Local contrast****Global contrast**

**Input****Image**

**Result**

**Careful**
**combination**

**Post-**
**process**
**Bilateral**

**Filter**

**Intensity**
**matching**

**Textureness**
**matching**

**A Non Perfect Result**

• Decoupled and large modifications (up to 6x) ÎLimited defects may appear

input (HDR)

result after

global and local adjustments

**Intensity Remapping**

• Some intensities may be outside displayable range.

Î Compress histogram to fit visible range.

corrected result remapped

intensities initial

result

**Preserving Details**

**1. In the gradient domain:**

– Compare gradient amplitudes of input and current – Prevent extreme reduction & extreme increase

**2. Solve the Poisson equation.**

corrected result remapped

intensities initial

result

**Effect of Detail Preservation**

uncorrected result corrected result

**Local contrast****Global contrast**

**Input****Image**

**Result**

**Post-**
**process**
**Bilateral**

**Filter**

**Intensity**
**matching**

**Textureness**
**matching**

**Constrained**
**Poisson**

**Local contrast****Global contrast**

**Input****Image**

**Result**

**Bilateral**
**Filter**

**Intensity**
**matching**

**Textureness**
**matching**

**Constrained**
**Poisson**

**Post-**
**process**

**Additional Effects**

**• Soft focus (**high frequency manipulation)

**• Film grain (**texture synthesis [Heeger 95])

**• Color toning (***chrominance = f (luminance)*)

before effects

after effects model

**Intensity**
**matching**

**Bilateral**
**Filter**

**Local contrast****Global contrast**

**Input****Image**

**Result**

**Textureness**
**matching**

**Constrained**
**Poisson**

**Soft focus**
**Toning**

**Grain**

**Intensity**
**matching**

**Bilateral**
**Filter**

**Local contrast****Global contrast**

**Input****Image**

**Result**

**Textureness**
**matching**

**Constrained**
**Poisson**

**Soft focus**
**Toning**

**Grain**

**Recap**

**Results**

User provides input and model photographs.

Î **Our system automatically produces the result.**

Running times:

– 6 seconds for 1 MPixel or less – 23 seconds for 4 MPixels

multi-grid Poisson solver and fast bilateral filter [Paris 06]

**Input** **Model**

**Result** **Input** **Result**

**Input** **Model** **Result**

**Input**
**Input**

**Our result**
**Our result**
**Na**

**Naï****ïve Histogram Matching****ve Histogram Matching**

**Model**
**Model**

*Snapshot*

*Snapshot, Alfred Stieglitz*, Alfred Stieglitz

**Comparison with Naïve Histogram Matching**

Local contrast, sharpness unfaithful

**Input**
**Input**

**Our Result**
**Our Result**

**Model**
**Model**

**Clearing Winter Storm, Ansel****Adams**

**Histogram Matching**
**Histogram Matching**

**Comparison with Naïve Histogram Matching**

Local contrast too low

**Color Images**

**• Lab color space: modify only luminance**

**Input**

**Input** **Output****Output**

**Limitations**

• Noise and JPEG artifacts

– amplified defects

• Can lead to unexpected

results if the image content is too different from the model

– Portraits, in particular, can suffer

**Conclusions**

• Transfer “look” from a model photo

• Two-scale tone management

– Global and local contrast

– New edge-preserving textureness – Constrained Poisson reconstruction – Additional effects

**References**

• Patrick Perez, Michel Gangnet, Andrew Blake, Poisson Image Editing, SIGGRAPH 2003.

• Dani Lischinski, Zeev Farbman, Matt Uytendaelle and Richard Szeliski. Interactive Local Adjustment of Tonal Values. SIGGRAPH 2006.

• Carsten Rother, Andrew Blake, Vladimir Kolmogorov, GrabCut - Interactive Foreground Extraction Using Iterated Graph Cuts, SIGGRAPH 2004.

• Aseem Agarwala, Mira Dontcheva, Maneesh Agrawala, Steven Drucker, Alex Colburn, Brian Curless, David H. Salesin, Michael F.

Cohen, Interactive Digital Photomontage, SIGGRAPH 2004.

• Sylvain Paris and Fredo Durand. A Fast Approximation of the Bilateral Filter using a Signal Processing Approach. ECCV 2006.

• Soonmin Bae, Sylvain Paris and Fredo Durand. Two-scale Tone Management for Photographic Look. SIGGRAPH 2006.