Intensity Gradient Manipulation

Gradient domain operations

Digital Visual Effects Yung-Yu Chuang

with slides by Fredo Durand, Ramesh Raskar, Amit Agrawal

Gradient domain operators

Gradient Domain Manipulations

Estimation of Gradients Manipulation of

Gradients

Non-Integrable Gradient Fields

Reconstruction from

Gradients

Images/Videos/

Meshes/Surfaces Images/Videos/

Meshes/Surfaces

Grad X

Grad Y

Integration2D

Image Intensity Gradients in 2D

Solve

Poisson Equation, 2D linear system

Grad X

Grad Y

New Grad X

New Grad Y

Integration2D

Intensity Gradient Manipulation

Gradient Processing

A Common Pipeline

1. Gradient manipulation

2. Reconstruction from gradients

Example Applications

Removing Glass Reflections

Seamless Image Stitching

Image Editing

Changing Local Illumination

High Dynamic Range Compression

Original PhotoshopGrey Color2Gray

Color to Gray Conversion

Edge Suppression under Significant Illumination Variations

Fusion of day and night images

Grad X

Grad Y

New Grad X

New Grad Y

Integration2D

Intensity Gradient Manipulation

Gradient Processing

A Common Pipeline

Intensity Gradient in 1D

I(x) 1

10⁵

G(x) 1

10⁵

Intensity Gradient

Gradient at x,

G(x) = I(x+1)- I(x)

Forward Difference

Reconstruction from Gradients

I(x) 1

10⁵ Intensity

G(x) 1

10⁵ Gradient

? ?

For n intensity values, about n gradients

Reconstruction from Gradients

I(x) 1

10⁵ Intensity

G(x) 1

10⁵ Gradient

1D Integration

I(x) = I(x-1) + G(x) Cumulative sum

?

1D case with constraints

Seamlessly paste onto

Just add a linear function so that the boundary condition is respected

Discrete 1D example: minimization

• Copy to

• Min ((f₂-f₁)-1)²

• Min ((f₃-f₂)-(-1))²

• Min ((f₄-f₃)-2)²

• Min ((f₅-f₄)-(-1))²

• Min ((f⁶-f₅)-(-1))²

0 1 2 3 4 5 6

0

1 2 3 4 5 6 7 -1 -1 -1

+2 +1

0 1 2 3 4 5 6

0 1 2 3 4 5 6 7

? ? ? ?

With

f

=6

f

=1

1D example: minimization

• Copy to

• Min ((f₂-6)-1)² ==> f₂²+49-14f₂

• Min ((f₃-f₂)-(-1))² ==> f₃²+f₂²+1-2f₃f₂ +2f₃-2f₂

• Min ((f₄-f₃)-2)² ==> f₄²+f₃²+4-2f₃f₄ -4f₄+4f₃

• Min ((f₅-f₄)-(-1))² ==> f₅²+f₄²+1-2f₅f₄ +2f₅-2f₄

• Min ((1-f₅)-(-1))² ==> f₅²+4-4f₅

0 1 2 3 4 5 6

0

1 2 3 4 5 6 7 -1 -1 -1

+2 +1

0 1 2 3 4 5 6

0 1 2 3 4 5 6 7

? ? ? ?

1D example: big quadratic

• Copy to

• Min (f₂²+49-14f₂

+ f₃²+f₂²+1-2f₃f₂ +2f₃-2f₂ + f₄²+f₃²+4-2f₃f₄ -4f₄+4f₃ + f₅²+f₄²+1-2f₅f₄ +2f₅-2f₄ + f₅²+4-4f₅)

Denote it Q

0 1 2 3 4 5 6

0

1 2 3 4 5 6 7 -1 -1 -1

+2 +1

0 1 2 3 4 5 6

0 1 2 3 4 5 6 7

? ? ? ?

1D example: derivatives

• Copy to

0 1 2 3 4 5 6

0

1 2 3 4 5 6 7 -1 -1 -1

+2 +1

0 1 2 3 4 5 6

0 1 2 3 4 5 6 7

? ? ? ?

Min (f₂²+49-14f₂

+ f₃²+f₂²+1-2f₃f₂ +2f₃-2f₂ + f₄²+f₃²+4-2f₃f₄ -4f₄+4f₃ + f₅²+f₄²+1-2f₅f₄ +2f₅-2f₄ + f₅²+4-4f₅)

Denote it Q

1D example: set derivatives to zero

• Copy to

0 1 2 3 4 5 6

0

1 2 3 4 5 6 7 -1 -1 -1

+2 +1

0 1 2 3 4 5 6

0 1 2 3 4 5 6 7

? ? ? ?

==>

1D example

• Copy to

0 1

2 3 4 5 6

0 1 2 3 4 5 6 7 -1

-1 -1

+2 +1

0 1 2 3 4 5 6

0

1 2 3 4 5 6 7

1D example: remarks

• Copy to

• Matrix is sparse

• Matrix is symmetric

• Everything is a multiple of 2

– because square and derivative of square

• Matrix is a convolution (kernel -2 4 -2)

• Matrix is independent of gradient field. Only RHS is

• Matrix is a second derivative

0 1 2 3 4 5 6

0

1 2 3 4 5 6 7 -1 -1 -1

+2 +1

0 1 2 3 4 5 6

0

1 2 3 4 5 6 7

2D example: images

• Images as scalar fields

– R² -> R

Gradients

• Vector field (gradient field)

– Derivative of a scalar field

• Direction

– Maximum rate of change of scalar field

• Magnitude

– Rate of change

Example

Image

I(x,y) I_x I_y

Gradient at x,y as Forward Differences G_x(x,y) = I(x+1 , y)- I(x,y)

G_y(x,y) = I(x , y+1)- I(x,y) G(x,y) = (G_x , G_y)

I_x

I_y

Integration2D

Reconstruction from Gradients

Sanity Check:

Recovering Original Image

Solve

Poisson Equation 2D linear system

Same

Reconstruction from Gradients

Given G(x,y) = (G_x , G_y)

How to compute I(x,y) for the image ? For n ² image pixels, 2 n ² gradients !

G_x

G_y

Integration2D

2D Integration is non-trivial

df/dx f(x)

f(x,y)

x y

Reconstruction depends on chosen path

Reconstruction from Gradient Field G

• Look for image

I

G

• I

( ^, )

_÷÷

ø çç ö

è

æ -

¶ + ¶

÷ ø ç ö

è

æ -

¶

= ¶ -

Ñ

=

Ñ

G

y G I

x G I

I G

I F

( )

òò ^{F Ñ I , G dxdy}

y G x

G y

I x

I

¶ + ¶

¶

= ¶

¶ + ¶

¶

¶

2 2 2

2 Poisson

equation

Solve

y G x

G y

I x

I

¶ + ¶

¶

= ¶

¶ + ¶

¶

¶

2 2 2

2

) 1 ,

( )

, ( )

, 1 (

) ,

(x y - G x - y + G x y - G x y -

G_{x x y y}

) , ( 4 )

1 ,

( )

1 ,

( )

, 1 (

) , 1

( x y I x y I x y I x y I x y

I + + - + + + - -

ú ú ú ú

û ù

ê ê ê ê

ë é

= ú ú

ú ú

û ù

ê ê ê ê

ë é

ú ú ú ú

û ù

ê ê ê ê

ë é

.. 1 … 1 -4 1 … 1 ..

I

Linear System

úú úú úú úú úú úú úú úú úú úú úú

û ù

êê êê êê êê êê êê êê êê êê êê êê

ë é

=

úú úú úú úú úú úú úú úú úú úú úú

û ù

êê êê êê êê êê êê êê êê êê êê êê

ë é

+ + - -

. . . . . . .

) , (

. . . . . .

. .

) , 1 (

. . .

) 1 ,

(

) , (

) 1 ,

( . . .

) , 1 (

. .

y x u

y x

I y x I

y x I

y x I

y x

I

x,y

[^{. 1 . . . 1 -4 1 . . . 1 .}] H

x,y-1 H

A x b

x-1,y H

W

) , ( )

, 1 (

) , 1 (

) 1 ,

( )

1 ,

( )

, (

4I x y + I x y + + I x y - + I x+ y + I x - y = u x y -

Sparse Linear system

úú úú úú úú ú

û ù

êê êê êê êê ê

ë é

- -

- -

- -

-

1 4

1 1

1 4

1 1

1 1

4 1

1

1 1

4 1

1

1 1

4 1

1

1 1

4 1

1 1

4 1

A matrix

Solving Linear System

• Image size N*N

• Size of A ~ N² by N²

• Impractical to form and store A

• Direct Solvers

• Basis Functions

• Multigrid

• Conjugate Gradients

Grad X

Grad Y

New Grad X

New Grad Y

Integration2D

Intensity Gradient Manipulation

Gradient Processing

A Common Pipeline

Gradient Domain Manipulations: Overview

(A) Per pixel

(B) Corresponding gradients in two images

(C) Corresponding gradients in multiple images (D) Combining gradients along seams

A. Per Pixel Manipulations

• Non-linear operations

– HDR compression, local illumination change

• Set to zero

– Shadow removal, intrinsic images, texture de-emphasis

High Dynamic Range Imaging

Images from Raanan Fattal

Gradient Domain Compression

HDR

Image L Log L

Gradient Attenuation Function G

Multiply Integration^2D

Gradients L_x,L_y

Illumination Invariant Image

G. D. Finlayson, S.D. Hordley & M.S. Drew, Removing Shadows From Images, ECCV 2002

Original Image Illumination invariant image

• Assumptions

– Sensor response = delta functions R, G, B in wavelength spectrum – Illumination restricted to Outdoor Illumination

Shadow Removal Using Illumination Invariant Image

G. D. Finlayson, S.D. Hordley & M.S. Drew, Removing Shadows From Images, ECCV 2002

Original Image

Illumination invariant image

Shadow Edge Locations

Edge Map

Integrate

Gradient Domain Manipulations: Overview

(A) Per pixel

(B) Corresponding gradients in two images

(C) Corresponding gradients in multiple images (D) Combining gradients along seams

Photography Artifacts: Flash Hotspot

Ambient Flash

Flash Hotspot

Underexposed Reflections

Ambient Flash

Reflections due to Flash

Ambient Flash

Self-Reflections and Flash Hotspot

Hands Face

Tripod

Result Ambient

Flash

Reflection Layer

Hands Face

Tripod

Intensity Gradient Vectors in Flash and Ambient Images

Same gradient

vector direction Flash Gradient Vector

Ambient Gradient Vector

Ambient Flash

No reflections

Reflection Ambient Gradient Vector

Different gradient vector directions

With reflections

Ambient Flash

Flash Gradient Vector

Residual Gradient

Vector

Intensity Gradient Vector Projection

Result Gradient Vector

Result Residual

Reflection Ambient Gradient Vector

Flash Gradient Vector

Ambient Flash

Flash

Projection = Result

Residual = Reflection Layer Ambient

Flash

Ambient

Checkerboard outside glass window Reflections on

glass window

removed

2D Integration

Flash

Ambient

X

Y X

Y

Forward Differences

Intensity Gradient

Vector Projection

Result X

Result Y

Result

2D Integration Gradient

Difference

Residual X

Residual Y

Reflection Layer Result

Checkerboard

Checkerboard

Poisson Image Editing

– Precise selection: tedious and unsatisfactory – Alpha-Matting: powerful but involved

– Seamless cloning: loose selection but no seams?

Conceal

Copy Background gradients (user strokes)

Compose

Target Image Source Images

Transparent Cloning

Largest variation from source and destination at each point

Gradient Domain Manipulations: Overview

(A) Per pixel

(B) Corresponding gradients in two images

(C) Corresponding gradients in multiple images (D) Combining gradients along seams

Intrinsic images

• I = L * R

• L = illumination image

• R = reflectance image

Intrinsic images

– Use multiple images under different illumination – Assumption

• Illumination image gradients = Laplacian PDF

• Under Laplacian PDF, Median = ML estimator

– At each pixel, take Median of gradients across images

– Integrate to remove shadows

Yair Weiss, “Deriving intrinsic images from image sequences”, ICCV 2001

Result = Illumination Image * (Label in Intrinsic Image) Shadow free

Intrinsic Image

Gradient Domain Manipulations: Overview

(A) Per pixel

(B) Corresponding gradients in two images

(C) Corresponding gradients in multiple images (D) Combining gradients along seams

Seamless Image Stitching

Anat Levin, Assaf Zomet, Shmuel Peleg and Yair Weiss, “Seamless Image Stitching in the Gradient Domain”, ECCV 2004