Image Intensity Gradients in 2D g y

Gradient domain operations

Digital Visual Effects g Yung-Yu Chuang

with slides by Fredo Durand, Ramesh Raskar, Amit Agrawal

Gradient Domain Manipulations

Images/Videos/

Meshes/Surfaces

Estimation Manipulation of

Meshes/Surfaces

of Gradients p

Gradients

Non-Integrable Gradient Fields

Reconstruction from

Gradients

Images/Videos/

Meshes/Surfaces

Image Intensity Gradients in 2D g y

Grad X

Integration2D

Grad Y

Solve

Poisson Equation, 2D linear system 2D linear system

Intensity Gradient Manipulation y p

A Common Pipeline

Grad X New Grad X

Integration2D Gradient

Processing

Grad Y New Grad Y

1. Gradient manipulation

2 Reconstruction from gradients 2. Reconstruction from gradients

Example Applications

Removing Glass Reflections

Seamless Image Stitching

Image Editing

Changing Local Illumination Changing Local Illumination

Original PhotoshopGrey Color2Gray

Color to Gray Conversiony

High Dynamic Range Compression

Ed S i d Si ifi t Ill i ti V i ti Edge Suppression under Significant Illumination Variations

Fusion of day and night images

Intensity Gradient Manipulation y p

A Common Pipeline

Grad X New Grad X

Integration2D Gradient

Processing

Grad Y New Grad Y

Intensity Gradient in 1D

Intensity Gradient

10⁵ 10⁵

y

I(x) 1

G(x) 1

Gradient at x,

G(x) = I(x+1)- I(x) G(x) I(x+1) I(x)

Forward Difference

Reconstruction from Gradients

Intensity Gradient

10⁵

y

10⁵

?

? ? ?

I(x) 1

G(x) 1

For n intensity values, about n gradients y , g

Reconstruction from Gradients

Intensity Gradient

10⁵

y

10⁵

?

?

I(x) 1

G(x) 1

1D Integration

I(x) = I(x 1) + G(x) I(x) = I(x-1) + G(x)

Cumulative sum

1D case with constraints

Seamlessly paste onto

Seamlessly paste onto

Just add a linear function so that the boundary condition is respected

Discrete 1D example: minimization

• Copy to

5 6

1 -1 1

+2

+1 ⁵

6

2 3

4 +1 -1 -1

2 3 4

0 1

0 1 2 3 4 5 6 7 ⁰

1 2

0

1 2 3 4 5 6 7

? ? ? ?

Min ((f f ) 1)

• Min ((f

-f

)-1)

• Min ((f

-f

)-(-1))

• Min ((f

-f

)-2)

With

• Min ((f

-f

)-(-1))

With f

=6

((

) ( ))

• Min ((f

-f

)-(-1))

f

=1

1D example: minimization

• Copy to

5 6

1 -1 1

+2

+1 ⁵

6

2 3

4 +1 -1 -1

2 3 4

0 1

0 1 2 3 4 5 6 7 ⁰

1 2

0

1 2 3 4 5 6 7

? ? ? ?

• Min ((f

-6)-1)

==> f

+49-14f

Mi ((f f ) ( 1))

f

f

1 2f f 2f 2f

• Min ((f

-f

)-(-1))

==> f

+f

+1-2f

f

+2f

-2f

• Min ((f

-f

)-2)

==> f

+f

+4-2f

f

-4f

+4f

• Min ((f

-f

)-(-1))

==> f

+f

+1-2f

f

+2f

-2f

• Min ((1-f

)-(-1))

==> f

+4-4f

• Min ((1 f

) ( 1)) > f

+4 4f

1D example: big quadratic

• Copy to

5 6

1 -1 1

+2

+1 ⁵

6

2 3

4 +1 -1 -1

2 3 4

0 1

0 1 2 3 4 5 6 7 ⁰

1 2

0

1 2 3 4 5 6 7

? ? ? ?

• Min (f₂²+49-14f₂

+ f ²+f ²+1 2f f +2f 2f + f₃²+f₂²+1-2f₃f₂ +2f₃-2f₂ + f₄²+f₃²+4-2f₃f₄ -4f₄+4f₃ + f ²+f ²+1 2f f +2f 2f + f₅²+f₄²+1-2f₅f₄ +2f₅-2f₄ + f₅²+4-4f₅)

Denote it Q Denote it Q

1D example: derivatives

• Copy to

5 6

1 -1 1

+2

+1 ⁵

6

2 3

4 +1 -1 -1

2 3 4

0 1

0 1 2 3 4 5 6 7 ⁰

1 2

0

1 2 3 4 5 6 7

? ? ? ?

Min (f₂²+49-14f₂

+ f₃²+f₂²+1-2f₃f₂ +2f₃-2f₂ + f₄²+f₃²+4-2f₃f₄ -4f₄+4f₃ + f₅²+f₄²+1-2f₅f₄ +2f₅-2f₄ + f ²+4 4f )

+ f₅²+4-4f₅) Denote it Q

1D example: set derivatives to zero

• Copy to

5 6

1 -1 1

+2

+1 ⁵

6

2 3

4 +1 -1 -1

2 3 4

0 1

0 1 2 3 4 5 6 7 ⁰

1 2

0

1 2 3 4 5 6 7

? ? ? ?

==>

1D example

• Copy to

5 6

1 -1 1

+2

+1 ⁵

6

3

4 +1 -1 -1

3 4

1 2

1 2

0 0

1 2 3 4 5 6 7 ⁰ ₀ 1 2 3 4 5 6 7

1D example: remarks

• Copy to

2 3 4 5 6

-1 -1 -1

+2 +1

2 3 4 5 6

0 1 2

0

1 2 3 4 5 6 7 ₀

1 2

0 1 2 3 4 5 6 7

Matrix is sparse

• Matrix is sparse

• Matrix is symmetric

E thi i lti l f 2

• Everything is a multiple of 2

– because square and derivative of square

• Matrix is a convolution (kernel 2 4 2)

• Matrix is a convolution (kernel -2 4 -2)

• Matrix is independent of gradient field. Only RHS is Matrix is a second derivative

• Matrix is a second derivative

2D example: images

• Images as scalar fields

– R² -> R

Gradients

• Vector field (gradient field)

D i ti f l fi ld – Derivative of a scalar field

• Direction

– Maximum rate of change of scalar field

• Magnitude

– Rate of changeg

Gradient Field

• Components of gradient

– Partial derivatives of Partial derivatives of scalar field

I I 

 , }

{ y

I x

I I







  ) 

, ( y x

I y

) , ,

( x y t

I { , , }

t I y

I x

I I











  ) 

, ,

( y  x  y  t

Example

Image

I(x y)

I

I

I(x,y)

Gradient at x y as Forward Differences Gradient at x,y as Forward Differences

G_x(x,y) = I(x+1 , y)- I(x,y) G_y(x,y) = I(x , y+1)- I(x,y) G_y(x,y) I(x , y 1) I(x,y) G(x,y) = (G_x , G_y)

Reconstruction from Gradients

Sanity Check:

Recovering Original Image

I_x

Recovering Original Image

2D I_y

Integration2D

Solve

Poisson Equation 2D linear system 2D linear system

Same

Reconstruction from Gradients

Given G(x,y) = (G_x , G_y)

How to compute I(x,y) for the image ? For n ² image pixels 2 n ² gradients ! For n ² image pixels, 2 n ² gradients !

G_xx

G_y

Integration2D G_y

2D Integration is non-trivial

df/dx f(x)

df/dx f(x)

f(x,y) y

xx

Reconstruction depends on chosen path

Reconstruction from Gradient Field G

• Look for image I with gradient closest to G in th l t

the least squares sense.

• I minimizes the integral: ^g  ^F  ^{ I , G}  ^dxdy

2 2



 



 

 ^{, y}

 ^, 

_



 



 



 

 

 



 



 









G

y G I

x G I

I G

I

F   x    y 

G G I

I   



_Poisson

y G x

G y

I x

I



 



 



 





2 2

Poisson

equation

Solve ^{ I   I   G}

^{  G}

2 2

Solve

y x

y

x  



 



) 1 ,

( )

, ( )

, 1 (

) ,

( x y  G x  y  G x y  G x y  G

( , y )

( , y )

( , y )

( , y )

) , ( 4 )

1 ,

( )

1 ,

( )

, 1 (

) , 1

( x y I x y I x y I x y I x y

I        

 

 

 

 

 

 

 



 

 

 



 



 



 



I

 

 



 



 



 



 



     



Linear System

) , ( )

, 1 (

) , 1 (

) 1 ,

( )

1 ,

( )

, (

4I x y  I x y   I x y   I x  y  I x y  u x y

















 . .



























. . . .

. .

) , 1 (

. y x

I

H x,y-1

H













 















) , (

.

) , (

) 1 , (

.

y x u y

x I

y x I

x,y

^{. 1 . . . 1  4 1 . . . 1 .}

x-1,y H























 

. . .

. .

) 1 , (x y I

H W







 







 







 











. . . .

. .

) , 1 (

. y x

I

W







 . .

A x b

Sparse Linear system













1 1

4 1

1 1

4 1













1 1

4 1

1

1 1

4 1

1

















1 4

1 1

1 1

4 1

1



 1 1  4 1

A matrix

A matrix

Solving Linear System

• Image size N*N

• Size of A ~ N

by N

• Impractical to form and store A

• Direct Solvers B i F ti

• Basis Functions

• Multigrid

• Conjugate Gradients

Approximate Solution for Large Scale Problems

Problems

• Resolution is increasing in digital cameras

• Stitching, Alignment requires solving large

linear system

Scalability problem

10 X 10 MP X 50% overlap =

50 Megapixel Panorama

A x = b

Scalability problem

A x = b

A x b

50 million element vectors!

Approximate Solution

• Reduce size of linear system H dl hi h l i i

• Handle high resolution images

• Part of Photoshop CS3

• Part of Photoshop CS3

Aseem Agarwala. "Efficient gradient-domain compositing using quadtrees," ACM Transactions on Graphics (Proceedings of SIGGRAPH 2007)

The key insight

Desired solution x

_

Initial

Solution x

=

Solution x

Difference x

x

Quadtree decomposition

• Maximally subdivide quadtree along seams

• Variables placed at node corners

R i d d

• Restricted quadtree

• Bi-linear interpolation reconstructs full solution Square nodes

• Square nodes

Reduced space

y

m variables

x

n variables m variables

n variables

<<

m << n

Reduced space

y

m variables

x

n variables m variables

n variables

S

x = Sy

x = Sy

x = Sy

x = Sy

x = Sy

x = Sy

x = Sy

x = Sy

x = Sy

Performance

227 96

13 Plane

1044 227

362 24

St. Emilion

1252 1044

435 16

Beynac

1790 1252

620 27

Rainier

1790

1 10 100 1000 10000

Memory (MB)

Quadtree [Agarwala 07]

Hierarchical basis preconditioning [Szeliski 90]p g [ ]

Locally-adapted hierarchical basis preconditioning [Szeliski 06]

Cut-and-paste

Cut-and-paste

Intensity Gradient Manipulation y p

A Common Pipeline

Grad X New Grad X

Integration2D Gradient

Processing

Grad Y New Grad Y

Gradient Domain Manipulations: Overview

(A) Per pixel

(B) Corresponding gradients in two images

( ) p g g g

(C) Corresponding gradients in multiple images

(C) Corresponding gradients in multiple images

(D) Combining gradients along seams

Gradient Domain Manipulations: Overview

(A) Per pixel

– Non-linear operations (HDR compression, local illumination change) – Set to zero (shadow removal, intrinsic images, texture de-emphasis)

Poisson Matting – Poisson Matting

(B) Corresponding gradients in two images (B) Corresponding gradients in two images – Vector operations (gradient projection)

• Combining flash/no-flash images, Reflection removalCombining flash/no flash images, Reflection removal

– Projection Tensors

• Reflection removal, Shadow removal

– Max operator

• Day/Night fusion, Visible/IR fusion, Extending DoF

– Binary, choose from first or second, copying

• Image editing, seamless cloning

Gradient Domain Manipulations

(C) Corresponding gradients in multiple images – Median operatorp

• Specularity reduction

• Intrinsic images

– Max operation

• Extended DOF

(D) Combining gradients along seams – Weighted averagingg g g

– Optimal seam using graph cut

• Image stitching, Mosaics, Panoramas, Image fusion

• A usual pipeline: Graph cut to find seams + gradient domain fusion

• A usual pipeline: Graph cut to find seams + gradient domain fusion

A. Per Pixel Manipulations

• Non-linear operations

– HDR compression, local illumination changep , g

• Set to zero

– Shadow removal intrinsic images texture de-emphasisShadow removal, intrinsic images, texture de emphasis

P i M tti

• Poisson Matting

High Dynamic Range Imaging

Images from Raanan Fattal

Gradient Domain Compression

Gradients L L

HDR

L_x,L_y

Image L Log L Multiply Integration^2D

Gradient Attenuation Gradient Attenuation Function G

Local Illumination Change

Original Image: f

Original gradient field: ^{g g}

  f f

Modified gradient field: v

Perez et al. Poisson Image editing, SIGGRAPH 2003

Illumination Invariant Image

Original Image Illumination invariant image

• Assumptions

– Sensor response = delta functions R G B in wavelength spectrumSensor response = delta functions R, G, B in wavelength spectrum – Illumination restricted to Outdoor Illumination

G. D. Finlayson, S.D. Hordley & M.S. Drew, Removing Shadows From Images, ECCV 2002

Shadow Removal Using Illumination Invariant Image

Invariant Image

Original Image Edge Map

Integrate

Illumination invariant image

Shadow Edge Locations

G. D. Finlayson, S.D. Hordley & M.S. Drew, Removing Shadows From Images, ECCV 2002

Illumination invariant image

Illumination invariant image g

Original Image

Invariant Image

g Image

Detected Sh d

Detected Shadow

Edges

Shadow Removed

G. D. Finlayson, S.D. Hordley & M.S. Drew, Removing Shadows From Images, ECCV 2002

Intrinsic Image

• Photo = Illumination Image * Intrinsic Image

• Retinex [Land & McCann 1971 Horn 1974]

• Retinex [Land & McCann 1971, Horn 1974]

– Illumination is smoothly varying

– Reflectance, piece-wise constant, has strong edges

– Keep strong image gradients, integrate to obtain reflectance

low-frequency high-frequency

attenuate more g q y

attenuate less

Poisson Matting

Background B

Alpha

Trimap: User specified

Foreground F Trimap: User specified

Jian Sun, Jiaya Jia, Chi-Keung Tang, Heung-Yeung Shum, Poisson Matting, SIGGRAPH 2004

Poisson Matting

Approximate: Assume F and B are smooth Approximate: Assume F and B are smooth

Poisson Equation

F and B in tri map using Poisson Equation

F and B in tri-map using nearest pixels

Poisson Matting

• Steps

– Approximate F and B in trimap

– Solve for ,

– Refine F and B using

– Iterate

Gradient Domain Manipulations: Overview

(A) Per pixel

(B) Corresponding gradients in two images

( ) p g g g

(C) Corresponding gradients in multiple images

(C) Corresponding gradients in multiple images

(D) Combining gradients along seams

Photography Artifacts: Flash Hotspot

Ambient Flash

Flash Flash Hotspot

Underexposed Reflections

Reflections due to Flash Reflections due to Flash

Underexposed Reflections

Ambient Flash

Ambient Flash

Self-Reflections and Flash Hotspot

Face

Hands

Tripod

Result

Ambient Reflection LayerReflection Layer

Fl h

Flash ^Face

Hands

Tripod

Intensity Gradient Vectors in Flash and Ambient Images

Same gradient

vector direction Flash Gradient Vector

Ambient Gradient Vector

Ambient Flash

No reflections

Reflection Ambient Gradient Reflection Ambient Gradient

Vector

Different gradient d

Flash Gradient Vector

vector directions

Ambient Flash

With reflections

Intensity Gradient Vector Projection

Reflection Ambient Gradient Residual

Gradient

Reflection Ambient Gradient Vector

Flash Gradient Vector Gradient

Vector

Result Gradient Vector

Result Residual

Ambient Flash

Flash

Projection = Result

Residual = Reflection Layer

Ambient Flash

Ambient

Flash

Ambient

Checkerboard outside glass window Reflections on

glass window

X

Forward Differences

Gradient Difference

Residual X

Checkerboard

Difference

Residual Y

Checkerboard

removed

Flash

Y

Result X

Residual Y

2D

Flash

Intensity Gradient

2D Integration

Integration Vector Projection

Result Y

Result X

Result

Reflection Layer Result

Y

Layer

Ambient

Image Fusion for

Context Enhancement Context Enhancement

and Video Surrealism

Adrian Ilie Adrian Ilie Ramesh Raskar

Ramesh Raskar Adrian IlieAdrian Ilie Jingyi YuJingyi Yu

UNC Chapel Hill UNC Chapel Hill

Ramesh Raskar Ramesh Raskar

Mitsubishi Electric Mitsubishi Electric

Research Labs Research Labs

Jingyi Yu Jingyi Yu

MIT MIT Research Labs,

Research Labs, (MERL) (MERL)

Dark Bldgs

Reflections on bldgs

Unknown shapes

Background is captured from day-time scene using the same fixed camera Night Image

Context Enhanced Image

Day Image

Context Enhanced Image

Mask is automatically computed from Mask is automatically computed from

scene contrast

But, Simple Pixel Blending Creates Ugly Artifacts

Ugly Artifacts

Pixel Blending

Pixel Blending

solution:

Integration of g blended Gradients

Nighttime image

Nighttime image Gradient fieldGradient field Nighttime image

Nighttime image Gradient fieldGradient field G₁x G^Y₁ I₁

Mixed gradient field

Importance Importance

image W image W

g

Gx G^Y

I₂

result

G₂xx G^Y₂

Daytime image

Daytime image Gradient fieldGradient field Final

y g

y g

Poisson Image Editing Poisson Image Editing

– Precise selection: tedious and unsatisfactory Alpha Matting: powerful but involved

– Precise selection: tedious and unsatisfactory Alpha Matting: powerful but involved

– Alpha-Matting: powerful but involved

– Seamless cloning: loose selection but no seams?

– Alpha-Matting: powerful but involved

– Seamless cloning: loose selection but no seams?

Conceal Conceal

Copy Background gradients (user strokes)

Compose Compose p p

Target Image Source Images

Transparent Cloning Transparent Cloning

Largest variation from source and destination at each Largest variation from source and destination at each

point

Gradient Domain Manipulations: Overview

(A) Per pixel

(B) Corresponding gradients in two images

( ) p g g g

(C) Corresponding gradients in multiple images

(C) Corresponding gradients in multiple images

(D) Combining gradients along seams

Intrinsic images

• I = L * R

L ill i i i

• L = illumination image

• R = reflectance image

Intrinsic images

U lti l i d diff t ill i ti – Use multiple images under different illumination – Assumption

Ill i ti i g g di t L l i PDF

• Illumination image gradients = Laplacian PDF

• Under Laplacian PDF, Median = ML estimator

At each pixel take Median of gradients across – At each pixel, take Median of gradients across

images

– Integrate to remove shadowsIntegrate to remove shadows

Yair Weiss, “Deriving intrinsic images from image sequences”, ICCV 2001

Shadow free Intrinsic Image

Result = Illumination Image * (Label in Intrinsic Image)

Specularity Reduction in Active Illumination

Illumination

Point Specularity

Line Specularity Area Specularity

Multiple images with same viewpoint, varying illumination How do we remove highlights?

Specularity Reduced Image

Image

Gradient Domain Manipulations: Overview

(A) Per pixel

(B) Corresponding gradients in two images

( ) p g g g

(C) Corresponding gradients in multiple images

(C) Corresponding gradients in multiple images

(D) Combining gradients along seams

Seamless Image Stitching

Anat Levin, Assaf Zomet, Shmuel Peleg and Yair Weiss, “Seamless Image Stitching in the Gradient Domain”, ECCV 2004