# Image Intensity Gradients in 2D g y

## Full text

(1)

### Digital Visual Effects g Yung-Yu Chuang

with slides by Fredo Durand, Ramesh Raskar, Amit Agrawal

(2)

Images/Videos/

Meshes/Surfaces

Estimation Manipulation of

Meshes/Surfaces

Reconstruction from

Images/Videos/

Meshes/Surfaces

(3)

### Image Intensity Gradients in 2Dgy

Integration2D

Solve

Poisson Equation, 2D linear system 2D linear system

(4)

A Common Pipeline

Processing

(5)

### Example Applications

Removing Glass Reflections

Seamless Image Stitching

(6)

Image Editing

Changing Local Illumination Changing Local Illumination

(7)

Original PhotoshopGrey Color2Gray

Color to Gray Conversiony

High Dynamic Range Compression

(8)

Ed S i d Si ifi t Ill i ti V i ti Edge Suppression under Significant Illumination Variations

Fusion of day and night images

(9)

A Common Pipeline

Processing

(10)

105 105

### y

I(x) 1

G(x) 1

G(x) = I(x+1)- I(x) G(x) I(x+1) I(x)

Forward Difference

(11)

105

105

I(x) 1

G(x) 1

(12)

105

105

### ?

I(x) 1

G(x) 1

1D Integration

I(x) = I(x 1) + G(x) I(x) = I(x-1) + G(x)

Cumulative sum

(13)

### 1D case with constraints

Seamlessly paste onto

Seamlessly paste onto

Just add a linear function so that the boundary condition is respected

(14)

### • Copy to

5 6

1 -1 1

+2

+1 5

6

2 3

4 +1 -1 -1

2 3 4

0 1

0 1 2 3 4 5 6 7 0

1 2

0

1 2 3 4 5 6 7

? ? ? ?

2

2

1

2

3

2

2

4

3

2

5

4

2

1

5 4

6

5

2

6

(15)

### • Copy to

5 6

1 -1 1

+2

+1 5

6

2 3

4 +1 -1 -1

2 3 4

0 1

0 1 2 3 4 5 6 7 0

1 2

0

1 2 3 4 5 6 7

? ? ? ?

2

2

22

2

2

2

2

3

2

2

32

22

3

2

3

2

4

3

2

42

32

3

4

4

3

5

4

2

52

42

5

4

5

4

5

2

52

5

5

5

### +4 4f

5

(16)

• Copy to

5 6

1 -1 1

+2

+1 5

6

2 3

4 +1 -1 -1

2 3 4

0 1

0 1 2 3 4 5 6 7 0

1 2

0

1 2 3 4 5 6 7

? ? ? ?

• Min (f22+49-14f2

+ f 2+f 2+1 2f f +2f 2f + f32+f22+1-2f3f2 +2f3-2f2 + f42+f32+4-2f3f4 -4f4+4f3 + f 2+f 2+1 2f f +2f 2f + f52+f42+1-2f5f4 +2f5-2f4 + f52+4-4f5)

Denote it Q Denote it Q

(17)

### 1D example: derivatives

• Copy to

5 6

1 -1 1

+2

+1 5

6

2 3

4 +1 -1 -1

2 3 4

0 1

0 1 2 3 4 5 6 7 0

1 2

0

1 2 3 4 5 6 7

? ? ? ?

Min (f22+49-14f2

+ f32+f22+1-2f3f2 +2f3-2f2 + f42+f32+4-2f3f4 -4f4+4f3 + f52+f42+1-2f5f4 +2f5-2f4 + f 2+4 4f )

+ f52+4-4f5) Denote it Q

(18)

### 1D example: set derivatives to zero

• Copy to

5 6

1 -1 1

+2

+1 5

6

2 3

4 +1 -1 -1

2 3 4

0 1

0 1 2 3 4 5 6 7 0

1 2

0

1 2 3 4 5 6 7

? ? ? ?

==>

(19)

### • Copy to

5 6

1 -1 1

+2

+1 5

6

3

4 +1 -1 -1

3 4

1 2

1 2

0 0

1 2 3 4 5 6 7 0 0 1 2 3 4 5 6 7

(20)

### 1D example: remarks

• Copy to

2 3 4 5 6

-1 -1 -1

+2 +1

2 3 4 5 6

0 1 2

0

1 2 3 4 5 6 7 0

1 2

0 1 2 3 4 5 6 7

Matrix is sparse

• Matrix is sparse

• Matrix is symmetric

E thi i lti l f 2

• Everything is a multiple of 2

– because square and derivative of square

• Matrix is a convolution (kernel 2 4 2)

• Matrix is a convolution (kernel -2 4 -2)

• Matrix is independent of gradient field. Only RHS is Matrix is a second derivative

• Matrix is a second derivative

(21)

– R2 -> R

(22)

### • Vector field (gradient field)

D i ti f l fi ld – Derivative of a scalar field

### • Direction

– Maximum rate of change of scalar field

### • Magnitude

– Rate of changeg

(23)

– Partial derivatives of Partial derivatives of scalar field

(24)

Image

I(x y)

x

### I

y

I(x,y)

Gradient at x y as Forward Differences Gradient at x,y as Forward Differences

Gx(x,y) = I(x+1 , y)- I(x,y) Gy(x,y) = I(x , y+1)- I(x,y) Gy(x,y) I(x , y 1) I(x,y) G(x,y) = (Gx , Gy)

(25)

Ix

### Recovering Original Image

2D Iy

Integration2D

Solve

Poisson Equation 2D linear system 2D linear system

Same

(26)

Given G(x,y) = (Gx , Gy)

How to compute I(x,y) for the image ? For n 2 image pixels 2 n 2 gradients ! For n 2 image pixels, 2 n 2 gradients !

Gxx

Gy

Integration2D Gy

(27)

### xx

Reconstruction depends on chosen path

(28)

2 2

##  ,y

2 2

2

x

y

2 2

x y

2 2

(29)

x

y

2 2

2 2

xx

xx

yy

yy

### 

.. 1 … 1 -4 1 … 1 ..

I

(30)

### Linear System

) , ( )

, 1 (

) , 1 (

) 1 ,

( )

1 ,

( )

, (

4I x y I x y I x y I x y I x y u x y

. .

. . . .

. .

) , 1 (

. y x

I

H x,y-1

H

) , (

.

) , (

) 1 , (

.

y x u y

x I

y x I

x,y

. 1 . . . 1 4 1 . . . 1 .

x-1,y H

. . .

. .

) 1 , (x y I

H W

. . . .

. .

) , 1 (

. y x

I

W







 . .

(31)

1 1

4 1

1 1

4 1

1 1

4 1

1

1 1

4 1

1

1 4

1 1

1 1

4 1

1



 1 1 4 1

(32)

2

2

(33)

(34)

### Scalability problem

10 X 10 MP X 50% overlap =

50 Megapixel Panorama

(35)

### A xb

50 million element vectors!

(36)

### • Part of Photoshop CS3

Aseem Agarwala. "Efficient gradient-domain compositing using quadtrees," ACM Transactions on Graphics (Proceedings of SIGGRAPH 2007)

(37)

0

0

δ

### x

δ

(38)
(39)

(40)

• Maximally subdivide quadtree along seams

• Variables placed at node corners

R i d d

• Bi-linear interpolation reconstructs full solution Square nodes

• Square nodes

(41)

(42)

(43)

(44)

(45)

(46)

(47)

### Performance

227 96

13 Plane

1044 227

362 24

St. Emilion

1252 1044

435 16

Beynac

1790 1252

620 27

Rainier

1790

1 10 100 1000 10000

Memory (MB)

Hierarchical basis preconditioning [Szeliski 90]p g [ ]

Locally-adapted hierarchical basis preconditioning [Szeliski 06]

(48)

(49)

### Cut-and-paste

(50)

A Common Pipeline

Processing

(51)

### (D) Combining gradients along seams

(52)

(A) Per pixel

– Non-linear operations (HDR compression, local illumination change) – Set to zero (shadow removal, intrinsic images, texture de-emphasis)

Poisson Matting – Poisson Matting

(B) Corresponding gradients in two images (B) Corresponding gradients in two images – Vector operations (gradient projection)

• Combining flash/no-flash images, Reflection removalCombining flash/no flash images, Reflection removal

– Projection Tensors

– Max operator

• Day/Night fusion, Visible/IR fusion, Extending DoF

– Binary, choose from first or second, copying

• Image editing, seamless cloning

(53)

(C) Corresponding gradients in multiple images – Median operatorp

• Specularity reduction

• Intrinsic images

– Max operation

• Extended DOF

(D) Combining gradients along seams – Weighted averagingg g g

– Optimal seam using graph cut

• Image stitching, Mosaics, Panoramas, Image fusion

• A usual pipeline: Graph cut to find seams + gradient domain fusion

• A usual pipeline: Graph cut to find seams + gradient domain fusion

(54)

### A. Per Pixel Manipulations

• Non-linear operations

– HDR compression, local illumination changep , g

• Set to zero

– Shadow removal intrinsic images texture de-emphasisShadow removal, intrinsic images, texture de emphasis

P i M tti

• Poisson Matting

(55)

### High Dynamic Range Imaging

Images from Raanan Fattal

(56)

HDR

Lx,Ly

Image L Log L Multiply Integration2D

(57)

### Local Illumination Change

Original Image: f

###   ff

*

Perez et al. Poisson Image editing, SIGGRAPH 2003

(58)

### Illumination Invariant Image

Original Image Illumination invariant image

• Assumptions

– Sensor response = delta functions R G B in wavelength spectrumSensor response = delta functions R, G, B in wavelength spectrum – Illumination restricted to Outdoor Illumination

G. D. Finlayson, S.D. Hordley & M.S. Drew, Removing Shadows From Images, ECCV 2002

(59)

### Invariant Image

Original Image Edge Map

Integrate

Illumination invariant image

G. D. Finlayson, S.D. Hordley & M.S. Drew, Removing Shadows From Images, ECCV 2002

Illumination invariant image

(60)

### Illumination invariant imageg

Original Image

Invariant Image

g Image

Detected Sh d

Edges

G. D. Finlayson, S.D. Hordley & M.S. Drew, Removing Shadows From Images, ECCV 2002

(61)

### Intrinsic Image

• Photo = Illumination Image * Intrinsic Image

• Retinex [Land & McCann 1971 Horn 1974]

• Retinex [Land & McCann 1971, Horn 1974]

– Illumination is smoothly varying

– Reflectance, piece-wise constant, has strong edges

– Keep strong image gradients, integrate to obtain reflectance

low-frequency high-frequency

attenuate more g q y

attenuate less

(62)

### Poisson Matting

Background B

Alpha

Trimap: User specified

Foreground F Trimap: User specified

Jian Sun, Jiaya Jia, Chi-Keung Tang, Heung-Yeung Shum, Poisson Matting, SIGGRAPH 2004

(63)

### Approximate: Assume F and B are smooth Approximate: Assume F and B are smooth

Poisson Equation

F and B in tri map using Poisson Equation

F and B in tri-map using nearest pixels

(64)

### • Steps

– Approximate F and B in trimap

– Solve for ,

– Refine F and B using

– Iterate

(65)

(66)

### Photography Artifacts: Flash Hotspot

Ambient Flash

Flash Flash Hotspot

(67)

Underexposed Reflections

### Reflections due to FlashReflections due to Flash

Underexposed Reflections

Ambient Flash

(68)

Ambient Flash

### Self-Reflections and Flash Hotspot

Face

Hands

Tripod

(69)

Result

Ambient Reflection LayerReflection Layer

Fl h

Flash Face

Hands

Tripod

(70)

Intensity Gradient Vectors in Flash and Ambient Images

Ambient Flash

### No reflections

(71)

Vector

vector directions

Ambient Flash

### With reflections

(72)

Vector

Result Residual

Ambient Flash

(73)

Flash

Projection = Result

Residual = Reflection Layer

Ambient Flash

Ambient

(74)

Flash

Ambient

Checkerboard outside glass window Reflections on

glass window

(75)

X

Forward Differences

Residual X

Checkerboard

Difference

Residual Y

Checkerboard

removed

Flash

Y

Result X

Residual Y

2D

Flash

2D Integration

Integration Vector Projection

Result Y

Result X

Result

Reflection Layer Result

Y

Layer

Ambient

(76)

### and Video Surrealism

UNC Chapel Hill UNC Chapel Hill

Mitsubishi Electric Mitsubishi Electric

Research Labs Research Labs

Jingyi Yu Jingyi Yu

MIT MIT Research Labs,

Research Labs, (MERL) (MERL)

(77)
(78)

Dark Bldgs

Reflections on bldgs

Unknown shapes

(79)

Background is captured from day-time scene using the same fixed camera Night Image

Context Enhanced Image

Day Image

Context Enhanced Image

(80)

scene contrast

(81)

But, Simple Pixel Blending Creates Ugly Artifacts

Ugly Artifacts

(82)

Pixel Blending

(83)

Pixel Blending

solution:

(84)

Nighttime image

Importance Importance

image W image W

g

Gx GY

I2

result

G2xx GY2

Daytime image

y g

y g

(85)

### Poisson Image Editing Poisson Image Editing

– Precise selection: tedious and unsatisfactory Alpha Matting: powerful but involved

– Precise selection: tedious and unsatisfactory Alpha Matting: powerful but involved

– Alpha-Matting: powerful but involved

– Seamless cloning: loose selection but no seams?

– Alpha-Matting: powerful but involved

– Seamless cloning: loose selection but no seams?

(86)

(87)

### ComposeComposepp

Target Image Source Images

(88)

### Transparent CloningTransparent Cloning

Largest variation from source and destination at each Largest variation from source and destination at each

point

(89)

(90)

(91)

### Intrinsic images

U lti l i d diff t ill i ti – Use multiple images under different illumination – Assumption

Ill i ti i g g di t L l i PDF

• Illumination image gradients = Laplacian PDF

• Under Laplacian PDF, Median = ML estimator

At each pixel take Median of gradients across – At each pixel, take Median of gradients across

images

Yair Weiss, “Deriving intrinsic images from image sequences”, ICCV 2001

(92)

Result = Illumination Image * (Label in Intrinsic Image)

(93)

### Illumination

Point Specularity

Line Specularity Area Specularity

Multiple images with same viewpoint, varying illumination How do we remove highlights?

(94)

Specularity Reduced Image

Image

(95)

(96)

### Seamless Image Stitching

Anat Levin, Assaf Zomet, Shmuel Peleg and Yair Weiss, “Seamless Image Stitching in the Gradient Domain”, ECCV 2004

Updating...

## References

Related subjects :