• 沒有找到結果。

Matting and Compositing

N/A
N/A
Protected

Academic year: 2022

Share "Matting and Compositing"

Copied!
65
0
0

加載中.... (立即查看全文)

全文

(1)

Matting and Compositing

Digital Visual Effectsg ff Yung-Yu Chuang

Outline Outline

• Traditional matting and compositing

• Traditional matting and compositing

• The matting problemg p

• Bayesian matting and extensions

• Matting with less user inputs

Matting with multiple observations

• Matting with multiple observations

• Beyond the compositing equation*Beyond the compositing equation

• Conclusions

Outline Outline

• Traditional matting and compositing

• Traditional matting and compositing

• The matting problemg p

• Bayesian matting and extensions

• Matting with less user inputs

Matting with multiple observations

• Matting with multiple observations

• Beyond the compositing equation*Beyond the compositing equation

• Conclusions

Photomontage Photomontage

The Two Ways of Life, 1857, Oscar Gustav Rejlander y j Printed from the original 32 wet collodion negatives.

(2)

Photographic compositions Photographic compositions

Lang Ching-shan

Use of mattes for compositing Use of mattes for compositing

The Great Train Robbery (1903) matte shot

Use of mattes for compositing Use of mattes for compositing

The Great Train Robbery (1903) matte shot

Optical compositing Optical compositing

King Kong (1933) Stop-motion + optical compositing

(3)

Digital matting and compositing Digital matting and compositing

The lost world (1925) The lost world (1997)

Miniature, stop-motion Computer-generated images

Digital matting and composting Digital matting and composting

King Kong (1933) Jurassic Park III (2001)

Optical compositing

Blue-screen matting, digital composition, Optical compositing digital composition, digital matte painting

Smith Duff Catmull Porter

O d 1996

Oscar award, 1996

Titanic

M tti d C iti

Matting and Compositing

(4)

background replacement

background

M tti d C iti

editing

Matting and Compositing

Digital matting: bluescreen matting Digital matting: bluescreen matting

Forrest Gump (1994)

• The most common approach for films.

• Expensive, studio setup.p p

• Not a simple one-step process.

Color difference method (Ultimatte) Color difference method (Ultimatte)

C=F+B F

Blue-screen photograph

Spill suppression if B>G then B=G

Matte creation

=B-max(G,R)

p g p (G, )

demo with Paint Shop Pro (B=min(B,G))

Problems with color difference Problems with color difference

Background color is usually not perfect! (lighting, shadowing…)

(5)

Chroma-keying (Primatte)

Chroma keying (Primatte) Chroma-keying (Primatte) Chroma keying (Primatte)

demo

Outline Outline

• Traditional matting and compositing

• Traditional matting and compositing

• The matting problemg p

• Bayesian matting and extensions

• Matting with less user inputs

Matting with multiple observations

• Matting with multiple observations

• Beyond the compositing equation*Beyond the compositing equation

• Conclusions

F B

F B

foreground color alpha matte background plate

B F

C α (1α) C

compositeF

B F

C α  1( α)

compositing

C

C iti

p g

equation

B =0

Compositing

(6)

F B

F B

B F

C α (1α) C

compositeF

B F

C α  1( α)

compositing

C

C iti

p g

equation

B

C =1

Compositing

F B

F B

B F

C α (1α) C

compositeC F

B F

C α  1( α)

compositing

C iti

p g

equation

B =0.6

Compositing

F B

F B

C

observation

B F

C αF (1α)B C α  1( α)

compositing

M tti

p g

equation

Matting

F B

F B

C C αF (1α)B

Three approaches:

1 d # k C αF  1( α)B

compositing 1 reduce #unknowns

2 add observations

M tti

p g

equation 3 add priors

Matting

(7)

F BB

F BB

C CC ααFF  1((1αα))BB

difference

M tti ( d # k )

difference matting

Matting (reduce #unknowns)

F

F

C B C αF (1α)B

B F

C α  1( α)

blue screen

M tti ( d # k )

blue screen matting

Matting (reduce #unknowns)

F

F

B F

C α (1α) B

C C αF  1( α)B

B F

C α (1α)

M tti ( dd b ti )

triangulation

Matting (add observations)

F B

F B

B

C BG CC ααFF  1((1αα))BB unknown FG

Natural image matting M tti ( dd i )

rotoscoping Ruzon-Tomasi

unknown

Natural image matting Matting (add priors)

(8)

Outline Outline

• Traditional matting and compositing

• Traditional matting and compositing

• The matting problemg p

• Bayesian matting and extensions

• Matting with less user inputs

Matting with multiple observations

• Matting with multiple observations

• Beyond the compositing equation*Beyond the compositing equation

• Conclusions

para observed

f(z)+

z y

para- meters

observed signal )

| ( max

* P z y

z  maxP(z | y) Example: super-resolution

z z

) ( )

|

max P(y z P z

super-resolution de-blurring de-blocking

) max (

y P

z de-blocking

B i f k

) ( )

| (

maxL y z L z

z

Bayesian framework

para observed

f(z)+

z y

para- meters

observed signal )

( )

| ( max

* L y z L z

z  maxL(y| z) L(z)

z z

) 2

(z f

data y  a-priori

B i f k

2

evidence knowledge

Bayesian framework

posterior probability

likelihood priors

B i f k

Bayesian framework

(9)

P i

Priors B Bayesian matting i tti

repeat

1. fix alpha

2. fix F and B

O ti i ti

until converge

Optimization

(10)
(11)
(12)
(13)
(14)

D

Demo

input trimapalphainput trimapalpha

R lt Results

input composite

input composite

R lt Results

input imagetrimap

C i

Comparisons

(15)

Bayesian Ruzon-Tomasi

C i

Comparisons

Bayesian Ruzon-Tomasi

C i

Comparisons

Mishima Mishima

C i

Comparisons

Bayesian Bayesian

C i

Comparisons

(16)

input image

C i

Comparisons

Bayesian Mishima

C i

Comparisons

Bayesian Mishima

C i

Comparisons

i t input video

Vid tti

Video matting

(17)

i t input video

input key trimaps

Vid tti Video matting

i t input video

interpo- lated trimaps

Vid tti Video matting

inp t input video

interpo- lated t t trimaps

output alpha

Vid tti Video matting

Compo-

inp t Compo-

input site video

interpo- lated t t trimaps

output alpha

Vid tti

Video matting

(18)

optical flow

optical flow

(19)
(20)

S l it

Sample composite

(21)

G b tt

Garbage mattes G b Garbage mattes tt

B k d ti ti

Background estimation B k Background estimation d ti ti

(22)

Al h tt Alpha matte

without

b k d with

b k d

C i

background background

Comparison

(23)
(24)

C P(F)

B

(25)

Problems with Bayesian matting Problems with Bayesian matting

• It requires fine trimaps for good results

• It requires fine trimaps for good results

• It is tedious to generate fine trimaps

• Its performance rapidly degrades when foreground and background patterns foreground and background patterns become complex

Th i di t d l l t l t th

• There is no direct and local control to the resulting mattes

Outline Outline

• Traditional matting and compositing

• Traditional matting and compositing

• The matting problemg p

• Bayesian matting and extensions

• Matting with less user inputs

Matting with multiple observations

• Matting with multiple observations

• Beyond the compositing equation*Beyond the compositing equation

• Conclusions

Scribble based input Scribble-based input

trimapp scribble

Motivation

Motivation

(26)

LazySnapping LazySnapping

L S i

n-th mean foreground color

LazySnapping

L S i

LazySnapping L LazySnapping S i

(27)

Matting approaches Matting approaches

• Sampling approaches: solve for each

• Sampling approaches: solve for each alpha separately by utilizing local fg/bg samples, e.g. Ruzon/Tomasi, Knockout and Bayesian matting

Knockout and Bayesian matting.

• Propagation approaches: solve the whole matte together by optimizing, e g Poisson BP random walker e.g. Poisson, BP, random walker, closed-form and robust matting.

Poisson matting Poisson matting

Poisson matting

Poisson matting Robust matting Robust matting

• Jue Wang and Michael Cohen CVPR

• Jue Wang and Michael Cohen, CVPR 2007

(28)

Robust matting Robust matting

• Instead of fitting models a non-

• Instead of fitting models, a non- parametric approach is used

B i R b

Bayesian Robust

Robust matting Robust matting

• We must evaluate hypothesized

• We must evaluate hypothesized foreground/background pairs

Bj C

Fi distance ratio

F

Robust matting Robust matting

• To encourage pure fg/bg pixels add

• To encourage pure fg/bg pixels, add weights

B F1

C

F F2

Robust matting Robust matting

• Combine them together Pick up the

• Combine them together. Pick up the best 3 pairs and average them

confidence

(29)

Robust matting

Robust matting Robust matting Robust matting

matte confidence

matte confidence

Matte optimization Matte optimization

Solved by Random Walk Algorithm

Matte optimization Matte optimization

data constraints data constraints

neighborhood constraints

(30)

Demo (EZ Mask)

Demo (EZ Mask) Evaluation Evaluation

• 8 images collected in 3 different

• 8 images collected in 3 different ways

• Each has a “ground truth” matte

Evaluation Evaluation

• Mean square error is used as the

• Mean square error is used as the accuracy metric

• Try 8 trimaps with different accuracy for testing robustness

for testing robustness

• 7 methods are tested: Bayesian, y ,

Belief propagation, Poisson, Random Walk KnockOut2 Closed Form and Walk, KnockOut2, Closed-Form and Robust mattingg

(31)

Quantitative evaluation

Quantitative evaluation Subjective evaluation Subjective evaluation

Subjective evaluation

Subjective evaluation Ranks of these algorithms Ranks of these algorithms

Poisson

accuracy 6 9

robustness Poisson 6 8

Random walk

6.9 6.0

6.8 4.4 Knockout2

Bayesian

4.5 3 9

4.5 Bayesian 6 0

Belief Propagation Close form

3.9 3.3 2 6

6.0 3.1 Close-form 2 0

Robust matting

2.6 1.0

2.0 1.3

(32)

Summary Summary

• Propagation-based methods are more

• Propagation-based methods are more robust

• Sampling-based methods often

generate more accurate mattes than generate more accurate mattes than propagation-based ones with fine trimaps

Robust matting combines strengths

• Robust matting combines strengths of both

New evaluation (CVPR 2009) New evaluation (CVPR 2009)

• http://www alphamatting com/

• http://www.alphamatting.com/

Soft scissor Soft scissor

• Jue Wang et al SIGGRAPH 2007

• Jue Wang et. al., SIGGRAPH 2007

• Users interact in a similar way to y intelligent scissors

Flowchart

Flowchart

(33)

Flowchart

Flowchart Soft scissor Soft scissor

Demo (Power Mask)

Demo (Power Mask) Outline Outline

• Traditional matting and compositing

• Traditional matting and compositing

• The matting problemg p

• Bayesian matting and extensions

• Matting with less user inputs

Matting with multiple observations

• Matting with multiple observations

• Beyond the compositing equation*Beyond the compositing equation

• Conclusions

(34)

Matting with multiple observations Matting with multiple observations

• Invisible lightsg

– Polarized lights Infrared

– Infrared

• Thermo-key

• Depth Keying (ZCam)

• Flash matting

• Flash matting

I i ibl li ht (I f d) Invisible lights (Infared)

I i ibl li ht (I f d)

Invisible lights (Infared) I i ibl li ht (I f Invisible lights (Infared) d)

(35)

I i ibl li ht (I f d)

Invisible lights (Infared) I i ibl li ht (I f Invisible lights (Infared) d)

I i ibl li ht (I f d)

Invisible lights (Infared) I i ibl li ht (P l i d) Invisible lights (Polarized)

(36)

I i ibl li ht (P l i d)

Invisible lights (Polarized) Th Thermo-Key K

Th K

Thermo-Key ZC ZCam

(37)

D f tti

Defocus matting

video

M tti ith

video

Matting with camera arrays

flash no flash matte

flash no flash matte

Fl h tti

Flash matting

(38)

Background is much further than foreground and receives almost no flash light

Fl h tti Flash matting

Foreground flash matting equationg g q

Generate a trimap and directly apply Bayesian matting.

Fl h tti Flash matting

F d fl h tti

Foreground flash matting J i t B Joint Bayesian flash matting i fl h tti

(39)

J i t B i fl h tti Joint Bayesian flash matting

flash no flash

C i

Comparison

foreground flash matting

ioint Bayesian flash matting

C i

Comparison Fl h tti Flash matting

(40)

Outline Outline

• Traditional matting and compositing

• Traditional matting and compositing

• The matting problemg p

• Bayesian matting and extensions

• Matting with less user inputs

Matting with multiple observations

• Matting with multiple observations

• Beyond the compositing equation*Beyond the compositing equation

• Conclusions

Conclusions Conclusions

• Matting algorithms improves a lot in

• Matting algorithms improves a lot in these 10 years

d ll l

• In production, it is still always

preferable to shoot against uniform

p g

backgrounds

Algorithms for more complex

• Algorithms for more complex backgrounds

• Devices or algorithms for automatic matting

matting

Thanks for your attention!

Thanks for your attention! Shadow matting

and composting

(41)

target background

source scene blue screen image target background

target background

blue screen composite blue screen composite photograph

(42)

photograph blue screen composite

G t i

Geometric errors

photograph blue screen composite

Ph t t i

Photometric errors

(43)

S L

S L

C

S L

S L

C=L+(1-)S

shadow compositing

C equationp g

Sh d iti ti

Shadow compositing equation



C=L+(1-)S

shadow compositing

C equationp g

Sh d tti Shadow matting

S L

S L

C=L+(1-)S

shadow compositing

C equationp g

Sh d tti

Shadow matting

(44)

S  L

S  L

C=L+(1-)S

shadow compositing

C equationp g

Sh d tti Shadow matting

S L

S L

C=L+(1-)S

shadow compositing

C equationp g

Sh d iti

Shadow compositing

(45)
(46)

G t i

Geometric errors

target background source scene

target background source scene

R i t #1

Requirement #1

target background source scene

R i t #2

Requirement #2

(47)
(48)
(49)
(50)
(51)
(52)
(53)
(54)
(55)
(56)

E i t tti Environment matting

photograph blue screen matting

traditional compositing equation

B

C F T

B

C F T

background foreground

composite

g

environment compositing equation [Zongker’99]

R

A B

C F T

B

C F T

background foreground

composite

g

(57)

O(k) images

E i t tti [Z k ’99]

Environment matting [Zongker’99]

photograph Zongker et al.

P bl l di i Problem: color dispersion

photograph Zongker et al.

P bl l f

Problem: glossy surface

photograph Zongker et al.

P bl lti l i

Problem: multiple mappings

(58)

W W

C T

C T

weighting background weighting

function

g composite

A bit i hti f ti

M lti Arbitrary weighting function d l i t d G i

Multimodal oriented Gaussian

(59)

high accuracy photograph Zongker et al.high accuracy

algorithm

P bl l di i Problem: color dispersion

high accuracy photograph Zongker et al.high accuracy

algorithm

Gl f

Glossy surface

without

withoutwith photograph without

orientation without orientation

with orientation

without orientation orientation

O i t d G i Oriented Gaussian

high accuracy photograph Zongker et al.high accuracy

algorithm

P bl lti l i

Problem: multiple mappings

(60)

R W

A B

C F T

B

C F T

background weighting

foreground composite

bac g ou d weighting

function

3 3 1 3 4

3

3 observations

3 1 3 4

3 observations 11 variables

A RA, R

F

4 3

3 3 4

3

3 observations 7 variables 3 observations

A RA, R

F

(61)

4 1

3 1 4

3

3 observations 5 variables

3 observations

A RA, R A

A,

colorless

F

T( c c )

1 3

T( c

x

, c

y

)

1 1 1

3

3 observations

1 1

3 variables 3 observations

A RA, R A

cx cy

A,

colorless

cx, cy,

specularly

F refractive

Stimulus function Stimulus function

x

Stimulus function Stimulus function

x y

A

y

T( )

M(T , A)  T(cx, cy) (cx, cy)

T

Ideal plane in RGB cube Ideal plane in RGB cube Ideal plane in RGB cube Ideal plane in RGB cube

x

y

T(x y) T(x,y)

(62)

Calibrated manifold in RGB cube Calibrated manifold in RGB cube Calibrated manifold in RGB cube Calibrated manifold in RGB cube

x (cx,cy)

C

(cx ,cy)

C

T(x y)

y

T(x,y)

Estimate c c and Estimate c c and

W

Estimate c

x

,c

y

and Estimate c

x

,c

y

and

background if ld

W T

manifold

C

 

(cx ,cy)

C

= KC

C

KC

K K

Problem: noisy matte

Problem: noisy matte Edge-preserving filtering Edge-preserving filtering Edge preserving filtering Edge preserving filtering

with filtering

without filteringg g

(63)

Input image

Input image Difference thresholding Difference thresholding

Morphological operation

Morphological operation Feathering Feathering

(64)

Heuristics for specular highlights Heuristics for specular highlights

background W

manifold

W T

manifold

C

>1+

C

C

>1+

K

Heuristics for specular highlights Heuristics for specular highlights Heuristics for specular highlights Heuristics for specular highlights

cx cy

C=

T(c c ) C=

T(cx, cy)

Heuristics for specular highlights Heuristics for specular highlights Heuristics for specular highlights Heuristics for specular highlights

=

=

input estimation foreground input estimation foreground (highlights)

(65)

Composite with highlights Composite with highlights

matting method compositing

model method

model

color blue-screen

B F

C α (1α)

color blending

blue-screen Bayesian

shadow Shadow

matting

β)L (

βS

C 1

refraction

matting High accuracy

refraction reflection

High-accuracy env. matting

F WB C

參考文獻

相關文件

Figure 1.9 illustrates a fractional-N modulator that achieves high spectral efficiency by using a digital transmit filter to obtain smooth transitions in the modulation data, and a

• The band-pass filter at the frontend filters out out-of-band signals and

Change Window Type to Video Sequence Editor. Add

– Change Window Type to Video Scene Editor – Select Add → Images and select all images – Drag the strip to the “1st Frame” in Layer

Edit Video Layer (Right click and drag, left click to set). Drag the strip

To support Test Project design and development, a rigorous quality assurance and design process is in place (Competition Rules sections 10.6-10.7 refer.) Once approved by

CeBIT is the world's largest trade fair showcasing digital IT and CeBIT is the world's largest trade fair showcasing digital IT and5. telecommunications solutions for home and work

• In Shutter-speed priority mode, photographers sets the shutter speed and the camera deduces the aperture. • In Program mode, the camera