Final resultFinal result

(1)

Computational Photography (I)

Digital Visual Effects, Spring 2008 Yung-Yu Chuang

2008/5/20

with slides by Fredo Durand, Ramesh Raskar, Sylvain Paris, Soonmin Bae, Amit Agrawal, Ramesh Raskar

Computational photography

wikipedia:

Computational photography refers broadly to computational imaging techniques that enhance or extend the capabilities of digital photography.

The output of these techniques is an ordinary photograph, but one that could not have been taken by a traditional camera.

What is computational photography

• Convergence of image processing, computer vision, computer graphics and photography

• Digital photography:

– Simply mimics traditional sensors and recording by digital technology

– Involves only simple image processing

• Computational photography

– More elaborate image manipulation, more computation

– New types of media (panorama, 3D, etc.)

– Camera design that take computation into account

Computational photography

• One of the most exciting fields.

• Symposium on Computational Photography and Video, 2005

• Full-semester courses in MIT, CMU, Stanford, GaTech, University of Delaware

• A new book by Raskar and Tumblin in SIGGRAPH 2007.

(2)

Siggraph 2006 Papers (16/86=18.6%)

Hybrid Images Drag-and-Drop Pasting

Two-scale Tone Management for Photographic Look Interactive Local Adjustment of Tonal Values Image-Based Material Editing

Flash Matting

Natural Video Matting using Camera Arrays Removing Camera Shake From a Single Photograph Coded Exposure Photography: Motion Deblurring Photo Tourism: Exploring Photo Collections in 3D AutoCollage

Photographing Long Scenes With Multi-Viewpoint Panoramas Projection Defocus Analysis for Scene Capture and Image Display Multiview Radial Catadioptric Imaging for Scene Capture Light Field Microscopy

Fast Separation of Direct and Global Components of a Scene Using High Frequency Illumination

Siggraph 2007 Papers (23/108=21.3%)

Image Deblurring with Blurred/Noisy Image Pairs Photo Clip Art

Scene Completion Using Millions of Photographs

Soft Scissors: An Interactive Tool for Realtime High Quality Matting Seam Carving for Content-Aware Image Resizing

Detail-Preserving Shape Deformation in Image Editing Veiling Glare in High Dynamic Range Imaging

Do HDR Displays Support LDR content? A Psychophysical Evaluation Ldr2hdr: On-the-fly Reverse Tone Mapping of Legacy Video and Photographs Rendering for an Interactive 360-Degree Light Field Display

Multiscale Shape and Detail Enhancement from Multi-light Image Collections Post-Production Facial Performance Relighting Using Reflectance Transfer Active Refocusing of Images and Videos

Multi-aperture Photography

Dappled Photography: Mask-Enhanced Cameras for Heterodyned Light Fields and Coded Aperture Refocusing

Image and Depth from a Conventional Camera with a Coded Aperture Capturing and Viewing Gigapixel Images

Efficient Gradient-Domain Compositing Using Quadtrees Image Upsampling via Imposed Edges Statistics Joint Bilateral Upsampling

Factored Time-Lapse Video Computational Time-Lapse Video

Real-Time Edge-Aware Image Processing With the Bilateral Grid

Scope

• We can’t yet set its precise definition. The following are scopes of what researchers are exploring in this field.

– Record a richer visual experience

– Overcome long-standing limitations of conventional cameras

– Enable new classes of visual signal – Enable synthesis impossible photos

Scope

• Image formation

• Color and color perception

• Demosaicing

(3)

Scope

• Panoramic imaging

• Image and video registration

• Spatial warping operations

Scope

• High Dynamic Range Imaging

• Bilateral filtering and HDR display

• Matting

Scope

• Active flash methods

• Lens technology

• Depth and defocus

No-flash

Flash

our result

Removing Photography Artifacts using Gradient Projection and Flash-Exposure Sampling

(4)

Continuous flash

Flash = 0.0

Flash = 0.3 Flash = 0.7 Flash = 1.4

Flash = 1.0

Flash matting

Depth Edge Detection and Stylized

Rendering Using a Multi-Flash Camera Motion-Based Motion Deblurring

(5)

Removing Camera Shake from a

Single Photograph Motion Deblurring using Fluttered Shutter

Scope

• Future cameras

• Plenoptic function and light fields

Scope

• Gradient image manipulation

(6)

Scope

• Taking great pictures

Art Wolfe Ansel Adams

Scope

• Non-parametric image synthesis, inpainting,

analogies

Scope

Motion analysis

Image Inpainting

(7)

Object Removal by

Exemplar-Based Inpainting

Image Completion with Structure Propagation

Lazy snapping Grab Cut - Interactive Foreground

Extraction using Iterated Graph Cuts

(8)

Image Tools

• Graph cuts,

– Segmentation and mosaicing

• Gradient domain operations,

– Tone mapping, fusion and matting

• Bilateral and Trilateral filters,

– Denoising, image enhancement

Graph cut

• Interactive image segmentation using graph cut

• Binary label: foreground vs. background

• User labels some pixels

– similar to trimap, usually sparser

• Exploit

– Statistics of known Fg & Bg – Smoothness of label

• Turn into discrete graph optimization

– Graph cut (min cut / max flow)

F

B F F F F B

B B

Energy function

• Labeling: one value per pixel, F or B

• Energy(labeling) = data + smoothness

– Very general situation – Will be minimized

• Data: for each pixel

– Probability that this color belongs to F (resp. B) – Similar in spirit to Bayesian matting

• Smoothness (aka regularization):

per neighboring pixel pair

– Penalty for having different label – Penalty is downweighted if the two

pixel colors are very different – Similar in spirit to bilateral filter

One labeling (ok, not best)

Data

Smoothness

(9)

Data term

• A.k.a regional term

(because integrated over full region)

• D(L)=

Σ

_i -log h[L_i](C_i)

• Where i is a pixel

L_i is the label at i (F or B), C_i is the pixel value

h[L_i] is the histogram of the observed Fg (resp Bg)

• Note the minus sign

Hard constraints

• The user has provided some labels

• The quick and dirty way to include

constraints into optimization is to replace the data term by a huge penalty if not respected.

• D(L_i)=0 if respected

• D(L_i)=K if not respected

– e.g. K=- #pixels

Smoothness term

• a.k.a boundary term, a.k.a. regularization

• S(L)=

Σ

{j, i} in N B(C_i,C_j) δ(L_i-L_j)

• Where i,j are neighbors

– e.g. 8-neighborhood

(but I show 4 for simplicity)

• δ(L_i-L_j) is 0 if L_i=L_j, 1 otherwise

• B(C_i,C_j) is high when C_i and C_j are similar, low if there is a discontinuity between those two pixels

– e.g. exp(-||C_i-C_j||²/2σ²) – where σ can be a constant

or the local variance

• Note positive sign

Optimization

• E(L)=D(L)+λ S(L)

• λ is a black-magic constant

• Find the labeling that minimizes E

• In this case, how many possibilities?

– 2⁹(512)

– We can try them all!

– What about megapixel images?

(10)

Labeling as a graph problem

• Each pixel = node

• Add two nodes F & B

• Labeling: link each pixel to either F or B F

B

Desired result

Data term

• Put one edge between each pixel and F & G

• Weight of edge = minus data term

– Don’t forget huge weight for hard constraints – Careful with sign

B F

Smoothness term

• Add an edge between each neighbor pair

• Weight = smoothness term

B F

Min cut

• Energy optimization equivalent to min cut

• Cut: remove edges to disconnect F from B

• Minimum: minimize sum of cut edge weight

B

F cut

(11)

Min cut <=> labeling

• In order to be a cut:

– For each pixel, either the F or G edge has to be cut

• In order to be minimal

– Only one edge label per pixel can be cut (otherwise could be added)

B

F cut

Computing a multiway cut

• With 2 labels: classical min-cut problem

– Solvable by standard flow algorithms

• polynomial time in theory, nearly linear in practice

– More than 2 terminals: NP-hard

[Dahlhaus et al., STOC ‘92]

• Efficient approximation algorithms exist

– Within a factor of 2 of optimal

– Computes local minimum in a strong sense

• even very large moves will not improve the energy

– Yuri Boykov, Olga Veksler and Ramin Zabih, Fast Approximate Energy Minimization via Graph Cuts, International Conference on Computer Vision, September 1999.

Move examples

Starting point

Red-blue swap move

Green expansion move

GrabCut GrabCut

Interactive Foreground Extraction Interactive Foreground Extraction

using Iterated Graph Cuts using Iterated Graph Cuts

Carsten

Carsten RotherRother Vladimir Kolmogorov Vladimir Kolmogorov

Andrew Blake Andrew Blake

Microsoft Research Cambridge Microsoft Research Cambridge--UKUK

(12)

Demo

• video

Interactive Digital Photomontage

Aseem Agarwala, Mira Dontcheva, Maneesh Agrawala, Steven Drucker, Alex Colburn, Brian Curless, David Salesin, Michael Cohen, “Interactive Digital Photomontage”, SIGGRAPH 2004

• Combining multiple photos

• Find seams using graph cuts

• Combine gradients and integrate

(13)

(14)

actual photomontage

set of originals perceived

Source images Brush strokes Computed labeling

Composite

Brush strokes Computed labeling

Interactive Digital Photomontage

• Extended depth of field

(15)

Interactive Digital Photomontage

• Relighting

Interactive Digital Photomontage

Gradient-domain fusion Poisson image editing

Demo

• video

(16)

Gradient domain operators

Gradient Domain Manipulations

Estimation of Gradients Manipulation of

Gradients

Non-Integrable Gradient Fields

Reconstruction from Gradients

Images/Videos/

Meshes/Surfaces Images/Videos/

Meshes/Surfaces

Grad X

Grad Y

Integration2D

Image Intensity Gradients in 2D

Solve Poisson Equation, 2D linear system

Grad X

Grad Y

New Grad X

New Grad Y

Integration2D

Intensity Gradient Manipulation

Gradient Processing

A Common Pipeline

1. Gradient manipulation

2. Reconstruction from gradients

(17)

Example Applications

Removing Glass Reflections

Seamless Image Stitching

Image Editing

Changing Local Illumination

High Dynamic Range Compression

Original PhotoshopGrey Color2Gray

Color to Gray Conversion

Edge Suppression under Significant Illumination Variations

Fusion of day and night images

(18)

Grad X

Grad Y

New Grad X

New Grad Y

Integration2D

Intensity Gradient Manipulation

Gradient Processing

A Common Pipeline

Intensity Gradient in 1D

I(x) 1

10⁵

G(x) 1

10⁵

Intensity Gradient

Gradient at x,

G(x) = I(x+1)- I(x)

Forward Difference

Reconstruction from Gradients

I(x) 1

10⁵

Intensity

G(x) 1

10⁵

Gradient

? ?

For n intensity values, about n gradients

Reconstruction from Gradients

I(x) 1

10⁵

Intensity

G(x) 1

10⁵

Gradient

1D Integration

I(x) = I(x-1) + G(x) Cumulative sum

?

(19)

1D case with constraints

Seamlessly paste onto

Just add a linear function so that the boundary condition is respected

Discrete 1D example: minimization

• Copy to

• Min ((f₂-f₁)-1)²

• Min ((f₃-f₂)-(-1))²

• Min ((f₄-f₃)-2)²

• Min ((f₅-f₄)-(-1))²

• Min ((f6-f₅)-(-1))²

0 1 2 3 4 5 6

0

1 2 3 4 5 6 7 -1 -1 -1

+2 +1

0 1 2 3 4 5 6

0

1 2 3 4 5 6 7

? ? ? ?

With f

₁

=6 f

₆

=1

1D example: minimization

• Copy to

• Min ((f₂-6)-1)² ==> f₂²+49-14f₂

• Min ((f₃-f₂)-(-1))² ==> f₃²+f₂²+1-2f₃f₂ +2f₃-2f₂

• Min ((f₄-f₃)-2)² ==> f₄²+f₃²+4-2f₃f₄ -4f₄+4f₃

• Min ((f₅-f₄)-(-1))² ==> f₅²+f₄²+1-2f₅f₄ +2f₅-2f₄

• Min ((1-f₅)-(-1))² ==> f₅²+4-4f₅

0 1 2 3 4 5 6

0

1 2 3 4 5 6 7 -1 -1 -1

+2 +1

0 1 2 3 4 5 6

0

1 2 3 4 5 6 7

? ? ? ?

1D example: big quadratic

• Copy to

• Min (f₂²+49-14f₂

+ f₃²+f₂²+1-2f₃f₂+2f₃-2f₂ + f₄²+f₃²+4-2f₃f₄-4f₄+4f₃ + f₅²+f₄²+1-2f₅f₄+2f₅-2f₄ + f₅²+4-4f₅)

Denote it Q

0 1 2 3 4 5 6

0

1 2 3 4 5 6 7 -1 -1 -1

+2 +1

0 1 2 3 4 5 6

0

1 2 3 4 5 6 7

? ? ? ?

(20)

1D example: derivatives

• Copy to

0 1 2 3 4 5 6

0

1 2 3 4 5 6 7 -1 -1 -1

+2 +1

0 1 2 3 4 5 6

0

1 2 3 4 5 6 7

? ? ? ?

Min (f₂²+49-14f₂

+ f₃²+f₂²+1-2f₃f₂+2f₃-2f₂ + f₄²+f₃²+4-2f₃f₄-4f₄+4f₃ + f₅²+f₄²+1-2f₅f₄+2f₅-2f₄ + f₅²+4-4f₅)

Denote it Q

1D example: set derivatives to zero

• Copy to

0 1 2 3 4 5 6

0

1 2 3 4 5 6 7 -1 -1 -1

+2 +1

0 1 2 3 4 5 6

0

1 2 3 4 5 6 7

? ? ? ?

==>

1D example

• Copy to

0 1 2 3 4 5 6

0

1 2 3 4 5 6 7 -1 -1 -1

+2 +1

0 1 2 3 4 5 6

0

1 2 3 4 5 6 7

1D example: remarks

• Copy to

• Matrix is sparse

• Matrix is symmetric

• Everything is a multiple of 2

– because square and derivative of square

• Matrix is a convolution (kernel -2 4 -2)

• Matrix is independent of gradient field. Only RHS is

• Matrix is a second derivative

0 1 2 3 4 5 6

0

1 23 4567 -1 -1 -1

+2 +1

0 1 2 3 4 5 6

0

123 4 567

(21)

Basics

• Images as scalar fields

– R²-> R

Gradients

• Vector field (gradient field)

– Derivative of a scalar field

• Direction

– Maximum rate of change of scalar field

• Magnitude

– Rate of change

Gradient Field

• Components of gradient

– Partial derivatives of scalar field

} ,

{ y

I x I I

∂

= ∂ ) ∇

, ( x y I

) , , ( x y t

I { , , }

t I y I x I I

∂

= ∂

∇

Example

Image

I(x,y) I_x I_y

Gradient at x,y as Forward Differences G_x(x,y) = I(x+1 , y)- I(x,y) G_y(x,y) = I(x , y+1)- I(x,y) G(x,y) = (G_x, G_y)

(22)

I_x

I_y

Integration2D

Reconstruction from Gradients

Sanity Check:

Recovering Original Image

Solve Poisson Equation 2D linear system

Same

Reconstruction from Gradients

Given G(x,y) = (G_x, G_y) How to compute I(x,y) for the image ? For n ²image pixels, 2 n ² gradients !

G_x

G_y

Integration2D

2D Integration is non-trivial

df/dx f(x)

f(x,y)

x y

Reconstruction depends on chosen path

Reconstruction from Gradient Field G

(23)

Poisson Equation

x G x

) G ,G div(G

I

_x _y ^x ^y

∂ + ∂

∂

= ∂

=

∇

²

Second order PDE

Boundary Conditions

• Dirichlet: Function values at boundary are known

• Neumann: Derivative normal to boundary = 0

Ω

∂

Ω

∂

∈

∀

=

•

∇I(x,y) n(x,y) 0, (x,y) Ω

∂

∈

∀

= ( , ) ( , ) )

,

(x y I₀ x y x y I

n

Numerical Solution

• Discretize Laplacian

∇

2 _⎥^⎥^⎥

⎦

⎤

⎢⎢

⎢

⎣

⎡

− 0 1 0

1 4 1

0 1 0

) ,

2

(

y x u ) ,G div(G

I =

_x _y

=

∇

) , ( ) , 1 ( ) , 1 ( ) 1 , ( ) 1 , ( ) , (

4I x y +I x y+ +I x y− +I x+ y +I x− y =h²u x y

−

h = grid size

Linear System

⎥⎥

⎦

⎤

⎢⎢

⎣

⎡

=

⎥⎥

⎦

⎤

⎢⎢

⎣

⎡

+ +

−

. . . . . . .

) , (

. . . . . .

. .

) , 1 (

. . .

) 1 , (

) , (

) 1 , (

. . .

) , 1 (

. .

y x u

y x I

x,y

[. 1 . . . 1 −4 1 . . . 1 .] H

H x,y-1

A x b

x-1,y H

W

) , ( ) , 1 ( ) , 1 ( ) 1 , ( ) 1 , ( ) , (

4I x y +I x y+ +I x y− +I x+ y +I x− y =u x y

−

(24)

Sparse Linear system

⎥⎥

⎥

⎦

⎤

⎢⎢

⎢

⎣

⎡

−

1 4 1 1

1 1

4 1 1

1 1

4 1 1

1 1

4 1 1

1 1

4 1

1 1

4 1

A matrix

Solving Linear System

• Image size N*N

• Size of A ~ N² by N²

• Impractical to form and store A

• Direct Solvers

• Basis Functions

• Multigrid

• Conjugate Gradients

Approximate Solution for Large Scale Problems

• Resolution is increasing in digital cameras

• Stitching, Alignment requires solving large linear system

xR _xG _xB

yR _yG _yB

[Perez et al. 03]

Gradient-domain compositing

Sources

xComposite

yComposite

(25)

I

_{i, j}

– I

_{i+1, j}

=

_x

C ^omposite Gradient-domain compositing

I

_{i, j}

– I

_{i, j+1}

=

_y

C ^omposite

A x = b

Scalability problem

50 Megapixel Panorama 10 X 10 MP X 50% overlap =

A x = b

Scalability problem

50 million element vectors!

A x = b

Approximate Solution

• Reduce size of linear system

• Handle high resolution images

• Part of Photoshop CS3

Aseem Agarwala. "Efficient gradient-domain compositing using quadtrees," ACM Transactions on Graphics (Proceedings of SIGGRAPH 2007)

(26)

The key insight

_

=

Initial Solution x₀ Desired solution x

Difference x_δ

Quadtree decomposition

• Maximally subdivide quadtree along seams

• Variables placed at node corners

• Restricted quadtree

• Bi-linear interpolation reconstructs full solution

• Square nodes

(27)

y

m variables

Reduced space

x

n variables

m << n

y

m variables

Reduced space

x

n variables

x = Sy

x = Sy x = Sy

(28)

x = Sy x = Sy

Performance

Quadtree [Agarwala 07]

Hierarchical basis preconditioning [Szeliski 90]

Locally-adapted hierarchical basis preconditioning [Szeliski 06]

1790 1252 1044 227

620 435 362 96

27 16

24 13

1 10 100 1000 10000

Rainier Beynac St. Emilion Plane

Memory (MB)

Cut-and-paste

(29)

Cut-and-paste

Grad X

Grad Y

New Grad X

New Grad Y

Integration2D

Intensity Gradient Manipulation

Gradient Processing

A Common Pipeline

Gradient Domain Manipulations: Overview

(A) Per pixel

(B) Corresponding gradients in two images

(C) Corresponding gradients in multiple images

(D) Combining gradients along seams

Gradient Domain Manipulations: Overview

(A) Per pixel

– Non-linear operations (HDR compression, local illumination change) – Set to zero (shadow removal, intrinsic images, texture de-emphasis) – Poisson Matting

(B) Corresponding gradients in two images – Vector operations (gradient projection)

• Combining flash/no-flash images, Reflection removal – Projection Tensors

• Reflection removal, Shadow removal – Max operator

• Day/Night fusion, Visible/IR fusion, Extending DoF – Binary, choose from first or second, copying

• Image editing, seamless cloning

(30)

Gradient Domain Manipulations

(C) Corresponding gradients in multiple images – Median operator

• Specularity reduction

• Intrinsic images – Max operation

• Extended DOF

(D) Combining gradients along seams – Weighted averaging

– Optimal seam using graph cut

• Image stitching, Mosaics, Panoramas, Image fusion

• A usual pipeline: Graph cut to find seams + gradient domain fusion

A. Per Pixel Manipulations

• Non-linear operations

– HDR compression, local illumination change

• Set to zero

– Shadow removal, intrinsic images, texture de-emphasis

• Poisson Matting

High Dynamic Range Imaging

Images from Raanan Fattal

Gradient Domain Compression

HDR

Image L Log L

Gradient Attenuation Function G

Multiply Integration^2D

Gradients L_x,L_y

(31)

Local Illumination Change

Original gradient field:

Original Image: f

f *

∇

Modified gradient field: v

Perez et al. Poisson Image editing, SIGGRAPH 2003

Illumination Invariant Image

G. D. Finlayson, S.D. Hordley & M.S. Drew, Removing Shadows From Images, ECCV 2002 Original Image Illumination invariant image

• Assumptions

– Sensor response = delta functions R, G, B in wavelength spectrum – Illumination restricted to Outdoor Illumination

Shadow Removal Using Illumination Invariant Image

G. D. Finlayson, S.D. Hordley & M.S. Drew, Removing Shadows From Images, ECCV 2002 Original Image

Illumination invariant image

Shadow Edge Locations

Edge Map

Integrate

Illumination invariant image

Detected Shadow Edges Original

Image

Invariant Image

Shadow Removed

G. D. Finlayson, S.D. Hordley & M.S. Drew, Removing Shadows From Images, ECCV 2002

(32)

Intrinsic Image

• Photo = Illumination Image * Intrinsic Image

• Retinex [Land & McCann 1971, Horn 1974]

– Illumination is smoothly varying

– Reflectance, piece-wise constant, has strong edges

– Keep strong image gradients, integrate to obtain reflectance

low-frequency

attenuate more high-frequency attenuate less

Poisson Matting

Trimap: User specified

Foreground F Background B

Alpha

Jian Sun, Jiaya Jia, Chi-Keung Tang, Heung-Yeung Shum, Poisson Matting, SIGGRAPH 2004

Poisson Matting

Approximate: Assume F and B are smooth

Poisson Equation

F and B in tri-map using nearest pixels

Poisson Matting

• Steps

– Approximate F and B in trimap – Solve for ,

– Refine F and B using – Iterate

(33)

Gradient Domain Manipulations: Overview

(A) Per pixel

(B) Corresponding gradients in two images (C) Corresponding gradients in multiple images

Ambient Flash

Self-Reflections and Flash Hotspot

Hands Face

Tripod

Result Ambient

Flash

Reflection Layer Reflection Layer

Hands Face

Tripod

Intensity Gradient Vectors in Flash and Ambient Images

Samegradient

vector direction Flash Gradient Vector Ambient Gradient Vector

Ambient Flash

No reflections

(34)

Reflection Ambient Gradient Vector

Different gradient vector direction

With reflections

Flash Gradient Vector Residual

Gradient Vector

Intensity Gradient Vector Projection

Result Gradient Vector

Result Residual

Reflection Ambient Gradient Vector

Flash Gradient Vector

Flash

Projection =

Result Residual = Reflection Layer Ambient

Flash

Ambient

Checkerboard outside glass window Reflections on

glass window

(35)

removed

2D Integration

Flash

Ambient

X

Y X

Y

Forward Differences

Intensity Gradient

Vector Projection Intensity Gradient

Vector Projection

Result X

Result Y

Result

2D Integration Gradient

Difference

Residual X

Residual Y

Reflection Layer Result

Checkerboard

Image Fusion for Context Enhancement

and Video Surrealism

Adrian Adrian IlieIlie

UNC Chapel Hill UNC Chapel Hill

Ramesh Raskar Ramesh Raskar

Mitsubishi Electric Mitsubishi Electric

Research Labs, Research Labs,

(MERL) (MERL)

Jingyi JingyiYuYu

MITMIT

Dark Bldgs

Reflections on bldgs

Unknown shapes

(36)

‘Well-lit’ Bldgs

Reflections in bldgs windows

Tree, Street shapes

Background is captured from day-time scene using the same fixed camera Night Image

Day Image

Context Enhanced Image

Mask is automatically computed from scene contrast

But, Simple Pixel Blending Creates Ugly Artifacts

(37)

Pixel Blending Pixel Blending

Our Method:

Integration of blended Gradients

Nighttime image Nighttime image

Daytime image

Daytime image Gradient fieldGradient field Importance

Importance image W image W

Final resultFinal result

Gradient field Gradient field

Mixed gradient field Mixed gradient field G

G₁₁ GG₁₁

G

G₂₂ GG₂₂

xx ^Y^Y

I I₁₁

I₂

G

Gxx GG^Y^Y

Reconstruction from Gradient Field

• Problem: minimize error |∇I’ – G|

• Estimate I’ so that G = ∇ I’

• Poisson equation

∇²I’ = div G

• Full multigrid solver

I’I’ GG^X^X

GG^Y^Y

(38)

Poisson Image Editing:

Inserting Objects

Poisson Image Editing:

Inserting Objects

– Precise selection: tedious and unsatisfactory – Alpha-Matting: powerful but involved

– Seamless cloning: loose selection but no seams?

– Precise selection: tedious and unsatisfactory – Alpha-Matting: powerful but involved

– Seamless cloning: loose selection but no seams?

Smooth Correction: Copying Gradients

g f*

Conceal Conceal

Copy Background gradients (user strokes)

Compose: Copy gradients from Source Images to Target Image

Target Image Source Images

(39)

Transparent Cloning Transparent Cloning

Largest variation from source and destination at each point

Compose (transparent) Compose (transparent)

Gradient Domain Manipulations: Overview

(A) Per pixel

Intrinsic images: Median of Gradient operator

• I = L * R

• L = illumination image

• R = reflectance image

(40)

Intrinsic images

– Use multiple images under different illumination – Assumption

• Illumination image gradients = Laplacian PDF

• Under Laplacian PDF, Median = ML estimator

– At each pixel, take Median of gradients across images

– Integrate to remove shadows

Yair Weiss, “Deriving intrinsic images from image sequences”, ICCV 2001 Result = Illumination Image * (Label in Intrinsic Image) Shadow free Intrinsic Image

Specularity Reduction in Active Illumination

Point Specularity

Line Specularity Area Specularity

Multiple images with same viewpoint, varying illumination

How do we remove highlights? Specularity Reduced

Image

(41)

Gradient Domain Manipulations: Overview

(A) Per pixel

Seamless Image Stitching

Anat Levin, Assaf Zomet, Shmuel Peleg and Yair Weiss, “Seamless Image Stitching in the Gradient Domain”, ECCV 2004