Matting and Compositing
Digital Visual Effectsg ff Yung-Yu Chuang
Outline Outline
• Traditional matting and compositing
• Traditional matting and compositing
• The matting problemg p
• Bayesian matting and extensions
• Matting with less user inputs
Matting with multiple observations
• Matting with multiple observations
• Beyond the compositing equation*Beyond the compositing equation
• Conclusions
Outline Outline
• Traditional matting and compositing
• Traditional matting and compositing
• The matting problemg p
• Bayesian matting and extensions
• Matting with less user inputs
Matting with multiple observations
• Matting with multiple observations
• Beyond the compositing equation*Beyond the compositing equation
• Conclusions
Photomontage Photomontage
The Two Ways of Life, 1857, Oscar Gustav Rejlander y j Printed from the original 32 wet collodion negatives.
Photographic compositions Photographic compositions
Lang Ching-shan
Use of mattes for compositing Use of mattes for compositing
The Great Train Robbery (1903) matte shot
Use of mattes for compositing Use of mattes for compositing
The Great Train Robbery (1903) matte shot
Optical compositing Optical compositing
King Kong (1933) Stop-motion + optical compositing
Digital matting and compositing Digital matting and compositing
The lost world (1925) The lost world (1997)
Miniature, stop-motion Computer-generated images
Digital matting and composting Digital matting and composting
King Kong (1933) Jurassic Park III (2001)
Optical compositing
Blue-screen matting, digital composition, Optical compositing digital composition, digital matte painting
Smith Duff Catmull Porter
O d 1996
Oscar award, 1996
Titanic
M tti d C iti
Matting and Compositing
background replacement
background
M tti d C iti
editing
Matting and Compositing
Digital matting: bluescreen matting Digital matting: bluescreen matting
Forrest Gump (1994)
• The most common approach for films.
• Expensive, studio setup.p p
• Not a simple one-step process.
Color difference method (Ultimatte) Color difference method (Ultimatte)
C=F+B F
Blue-screen photograph
Spill suppression if B>G then B=G
Matte creation
=B-max(G,R)
p g p (G, )
demo with Paint Shop Pro (B=min(B,G))
Problems with color difference Problems with color difference
Background color is usually not perfect! (lighting, shadowing…)
Chroma-keying (Primatte)
Chroma keying (Primatte) Chroma-keying (Primatte) Chroma keying (Primatte)
demo
Outline Outline
• Traditional matting and compositing
• Traditional matting and compositing
• The matting problemg p
• Bayesian matting and extensions
• Matting with less user inputs
Matting with multiple observations
• Matting with multiple observations
• Beyond the compositing equation*Beyond the compositing equation
• Conclusions
F B
F B
foreground color alpha matte background plate
B F
C α (1α) C
compositeF
B F
C α 1( α)
compositing
C
C iti
p g
equation
B =0
Compositing
F B
F B
B F
C α (1α) C
compositeF
B F
C α 1( α)
compositing
C
C iti
p g
equation
B
C =1
Compositing
F B
F B
B F
C α (1α) C
compositeC F
B F
C α 1( α)
compositing
C iti
p g
equation
B =0.6
Compositing
F B
F B
C
observation
B F
C αF (1α)B C α 1( α)
compositing
M tti
p g
equation
Matting
F B
F B
C C αF (1α)B
Three approaches:
1 d # k C αF 1( α)B
compositing 1 reduce #unknowns
2 add observations
M tti
p g
equation 3 add priors
Matting
F BB
F BB
C CC ααFF 1((1αα))BB
difference
M tti ( d # k )
difference matting
Matting (reduce #unknowns)
F
F
C B C αF (1α)B
B F
C α 1( α)
blue screen
M tti ( d # k )
blue screen matting
Matting (reduce #unknowns)
F
F
B F
C α (1α) B
C C αF 1( α)B
B F
C α (1α)
M tti ( dd b ti )
triangulation
Matting (add observations)
F B
F B
B
C BG CC ααFF 1((1αα))BB unknown FG
Natural image matting M tti ( dd i )
rotoscoping Ruzon-Tomasi
unknown
Natural image matting Matting (add priors)
Outline Outline
• Traditional matting and compositing
• Traditional matting and compositing
• The matting problemg p
• Bayesian matting and extensions
• Matting with less user inputs
Matting with multiple observations
• Matting with multiple observations
• Beyond the compositing equation*Beyond the compositing equation
• Conclusions
para observed
f(z)+
z y
para- meters
observed signal )
| ( max
* P z y
z maxP(z | y) Example: super-resolution
z z
) ( )
|
max P(y z P z
super-resolution de-blurring de-blocking
) max (
y P
z de-blocking
…
B i f k
) ( )
| (
maxL y z L z
z
Bayesian framework
para observed
f(z)+
z y
para- meters
observed signal )
( )
| ( max
* L y z L z
z maxL(y| z) L(z)
z z
) 2
(z f
data y a-priori
B i f k
2
evidence knowledge
Bayesian framework
posterior probability
likelihood priors
B i f k
Bayesian framework
P i
Priors B Bayesian matting i tti
repeat
1. fix alpha
2. fix F and B
O ti i ti
until converge
Optimization
D
Demo
input trimapalphainput trimapalpha
R lt Results
input composite
input composite
R lt Results
input imagetrimap
C i
Comparisons
Bayesian Ruzon-Tomasi
C i
Comparisons
Bayesian Ruzon-Tomasi
C i
Comparisons
Mishima Mishima
C i
Comparisons
Bayesian Bayesian
C i
Comparisons
input image
C i
Comparisons
Bayesian Mishima
C i
Comparisons
Bayesian Mishima
C i
Comparisons
i t input video
Vid tti
Video matting
i t input video
input key trimaps
Vid tti Video matting
i t input video
interpo- lated trimaps
Vid tti Video matting
inp t input video
interpo- lated t t trimaps
output alpha
Vid tti Video matting
Compo-
inp t Compo-
input site video
interpo- lated t t trimaps
output alpha
Vid tti
Video matting
optical flow
optical flow
S l it
Sample composite
G b tt
Garbage mattes G b Garbage mattes tt
B k d ti ti
Background estimation B k Background estimation d ti ti
Al h tt Alpha matte
without
b k d with
b k d
C i
background background
Comparison
C P(F)
B
Problems with Bayesian matting Problems with Bayesian matting
• It requires fine trimaps for good results
• It requires fine trimaps for good results
• It is tedious to generate fine trimaps
• Its performance rapidly degrades when foreground and background patterns foreground and background patterns become complex
Th i di t d l l t l t th
• There is no direct and local control to the resulting mattes
Outline Outline
• Traditional matting and compositing
• Traditional matting and compositing
• The matting problemg p
• Bayesian matting and extensions
• Matting with less user inputs
Matting with multiple observations
• Matting with multiple observations
• Beyond the compositing equation*Beyond the compositing equation
• Conclusions
Scribble based input Scribble-based input
trimapp scribble
Motivation
Motivation
LazySnapping LazySnapping
L S i
n-th mean foreground color
LazySnapping
L S i
LazySnapping L LazySnapping S i
Matting approaches Matting approaches
• Sampling approaches: solve for each
• Sampling approaches: solve for each alpha separately by utilizing local fg/bg samples, e.g. Ruzon/Tomasi, Knockout and Bayesian matting
Knockout and Bayesian matting.
• Propagation approaches: solve the whole matte together by optimizing, e g Poisson BP random walker e.g. Poisson, BP, random walker, closed-form and robust matting.
Poisson matting Poisson matting
Poisson matting
Poisson matting Robust matting Robust matting
• Jue Wang and Michael Cohen CVPR
• Jue Wang and Michael Cohen, CVPR 2007
Robust matting Robust matting
• Instead of fitting models a non-
• Instead of fitting models, a non- parametric approach is used
B i R b
Bayesian Robust
Robust matting Robust matting
• We must evaluate hypothesized
• We must evaluate hypothesized foreground/background pairs
Bj C
Fi distance ratio
F
Robust matting Robust matting
• To encourage pure fg/bg pixels add
• To encourage pure fg/bg pixels, add weights
B F1
C
F F2
Robust matting Robust matting
• Combine them together Pick up the
• Combine them together. Pick up the best 3 pairs and average them
confidence
Robust matting
Robust matting Robust matting Robust matting
matte confidence
matte confidence
Matte optimization Matte optimization
Solved by Random Walk Algorithm
Matte optimization Matte optimization
data constraints data constraints
neighborhood constraints
Demo (EZ Mask)
Demo (EZ Mask) Evaluation Evaluation
• 8 images collected in 3 different
• 8 images collected in 3 different ways
• Each has a “ground truth” matte
Evaluation Evaluation
• Mean square error is used as the
• Mean square error is used as the accuracy metric
• Try 8 trimaps with different accuracy for testing robustness
for testing robustness
• 7 methods are tested: Bayesian, y ,
Belief propagation, Poisson, Random Walk KnockOut2 Closed Form and Walk, KnockOut2, Closed-Form and Robust mattingg
Quantitative evaluation
Quantitative evaluation Subjective evaluation Subjective evaluation
Subjective evaluation
Subjective evaluation Ranks of these algorithms Ranks of these algorithms
Poisson
accuracy 6 9
robustness Poisson 6 8
Random walk
6.9 6.0
6.8 4.4 Knockout2
Bayesian
4.5 3 9
4.5 Bayesian 6 0
Belief Propagation Close form
3.9 3.3 2 6
6.0 3.1 Close-form 2 0
Robust matting
2.6 1.0
2.0 1.3
Summary Summary
• Propagation-based methods are more
• Propagation-based methods are more robust
• Sampling-based methods often
generate more accurate mattes than generate more accurate mattes than propagation-based ones with fine trimaps
Robust matting combines strengths
• Robust matting combines strengths of both
New evaluation (CVPR 2009) New evaluation (CVPR 2009)
• http://www alphamatting com/
• http://www.alphamatting.com/
Soft scissor Soft scissor
• Jue Wang et al SIGGRAPH 2007
• Jue Wang et. al., SIGGRAPH 2007
• Users interact in a similar way to y intelligent scissors
Flowchart
Flowchart
Flowchart
Flowchart Soft scissor Soft scissor
Demo (Power Mask)
Demo (Power Mask) Outline Outline
• Traditional matting and compositing
• Traditional matting and compositing
• The matting problemg p
• Bayesian matting and extensions
• Matting with less user inputs
Matting with multiple observations
• Matting with multiple observations
• Beyond the compositing equation*Beyond the compositing equation
• Conclusions
Matting with multiple observations Matting with multiple observations
• Invisible lightsg
– Polarized lights Infrared
– Infrared
• Thermo-key
• Depth Keying (ZCam)
• Flash matting
• Flash matting
I i ibl li ht (I f d) Invisible lights (Infared)
I i ibl li ht (I f d)
Invisible lights (Infared) I i ibl li ht (I f Invisible lights (Infared) d)
I i ibl li ht (I f d)
Invisible lights (Infared) I i ibl li ht (I f Invisible lights (Infared) d)
I i ibl li ht (I f d)
Invisible lights (Infared) I i ibl li ht (P l i d) Invisible lights (Polarized)
I i ibl li ht (P l i d)
Invisible lights (Polarized) Th Thermo-Key K
Th K
Thermo-Key ZC ZCam
D f tti
Defocus matting
videoM tti ith
video
Matting with camera arrays
flash no flash matte
flash no flash matte
Fl h tti
Flash matting
Background is much further than foreground and receives almost no flash light
Fl h tti Flash matting
Foreground flash matting equationg g q
Generate a trimap and directly apply Bayesian matting.
Fl h tti Flash matting
F d fl h tti
Foreground flash matting J i t B Joint Bayesian flash matting i fl h tti
J i t B i fl h tti Joint Bayesian flash matting
flash no flash
C i
Comparison
foreground flash matting
ioint Bayesian flash matting
C i
Comparison Fl h tti Flash matting
Outline Outline
• Traditional matting and compositing
• Traditional matting and compositing
• The matting problemg p
• Bayesian matting and extensions
• Matting with less user inputs
Matting with multiple observations
• Matting with multiple observations
• Beyond the compositing equation*Beyond the compositing equation
• Conclusions
Conclusions Conclusions
• Matting algorithms improves a lot in
• Matting algorithms improves a lot in these 10 years
d ll l
• In production, it is still always
preferable to shoot against uniform
p g
backgrounds
Algorithms for more complex
• Algorithms for more complex backgrounds
• Devices or algorithms for automatic matting
matting
Thanks for your attention!
Thanks for your attention! Shadow matting
and composting
target background
source scene blue screen image target background
target background
blue screen composite blue screen composite photograph
photograph blue screen composite
G t i
Geometric errors
photograph blue screen composite
Ph t t i
Photometric errors
S L
S L
C
S L
S L
C=L+(1-)S
shadow compositing
C equationp g
Sh d iti ti
Shadow compositing equation
C=L+(1-)S
shadow compositing
C equationp g
Sh d tti Shadow matting
S L
S L
C=L+(1-)S
shadow compositing
C equationp g
Sh d tti
Shadow matting
S L
S L
C=L+(1-)S
shadow compositing
C equationp g
Sh d tti Shadow matting
S L
S L
C=L+(1-)S
shadow compositing
C equationp g
Sh d iti
Shadow compositing
G t i
Geometric errors
target background source scene
target background source scene
R i t #1
Requirement #1
target background source scene
R i t #2
Requirement #2
E i t tti Environment matting
photograph blue screen matting
traditional compositing equation
B
C F T
B
C F T
background foreground
composite
g
environment compositing equation [Zongker’99]
R
A B
C F T
B
C F T
background foreground
composite
g
O(k) images
E i t tti [Z k ’99]
Environment matting [Zongker’99]
photograph Zongker et al.
P bl l di i Problem: color dispersion
photograph Zongker et al.
P bl l f
Problem: glossy surface
photograph Zongker et al.
P bl lti l i
Problem: multiple mappings
W W
C T
C T
weighting background weighting
function
g composite
A bit i hti f ti
M lti Arbitrary weighting function d l i t d G i
Multimodal oriented Gaussian
high accuracy photograph Zongker et al.high accuracy
algorithm
P bl l di i Problem: color dispersion
high accuracy photograph Zongker et al.high accuracy
algorithm
Gl f
Glossy surface
without
withoutwith photograph without
orientation without orientation
with orientation
without orientation orientation
O i t d G i Oriented Gaussian
high accuracy photograph Zongker et al.high accuracy
algorithm
P bl lti l i
Problem: multiple mappings
R W
A B
C F T
B
C F T
background weighting
foreground composite
bac g ou d weighting
function
3 3 1 3 4
3
3 observations
3 1 3 4
3 observations 11 variables
A RA, R
F
4 3
3 3 4
3
3 observations 7 variables 3 observations
A RA, R
F
4 1
3 1 4
3
3 observations 5 variables
3 observations
A RA, R A
A,
colorless
F
T( c c )
1 3
T( cx , c
y)
1 1 1
3
3 observations
1 1
3 variables 3 observations
A RA, R A
cx cy
A,
colorless
cx, cy,
specularly
F refractive
Stimulus function Stimulus function
x
Stimulus function Stimulus function
x y
A
y
T( )
M(T , A) T(cx, cy) (cx, cy)
T
Ideal plane in RGB cube Ideal plane in RGB cube Ideal plane in RGB cube Ideal plane in RGB cube
x
y
T(x y) T(x,y)
Calibrated manifold in RGB cube Calibrated manifold in RGB cube Calibrated manifold in RGB cube Calibrated manifold in RGB cube
x (cx,cy)
C
(cx ,cy)C
T(x y)
y
T(x,y)
Estimate c c and Estimate c c and
W
Estimate c
x,c
yand Estimate c
x,c
yand
background if ld
W T
manifold
C
(cx ,cy)C
= KC
C
KC
K K
Problem: noisy matte
Problem: noisy matte Edge-preserving filtering Edge-preserving filtering Edge preserving filtering Edge preserving filtering
with filtering
without filteringg g
Input image
Input image Difference thresholding Difference thresholding
Morphological operation
Morphological operation Feathering Feathering
Heuristics for specular highlights Heuristics for specular highlights
background W
manifold
W T
manifold
C
>1+
C
C
>1+
K
Heuristics for specular highlights Heuristics for specular highlights Heuristics for specular highlights Heuristics for specular highlights
cx cy
C=
T(c c ) C=
T(cx, cy)Heuristics for specular highlights Heuristics for specular highlights Heuristics for specular highlights Heuristics for specular highlights
=
=
input estimation foreground input estimation foreground (highlights)
Composite with highlights Composite with highlights
matting method compositing
model method
model
color blue-screen
B F
C α (1α)
color blending
blue-screen Bayesian
shadow Shadow
matting
β)L (
βS
C 1
refraction
matting High accuracy
refraction reflection
High-accuracy env. matting
F WB C