Matting and Compositing
Digital Visual Effects, Spring 2008 Yung-Yu Chuang
2008/4/29
Outline
• Traditional matting and compositing
• The matting problem
• Bayesian matting and extensions
• Matting with less user inputs
• Matting with multiple observations
• Beyond the compositing equation*
• Conclusions
Outline
• Traditional matting and compositing
• The matting problem
• Bayesian matting and extensions
• Matting with less user inputs
• Matting with multiple observations
• Beyond the compositing equation*
• Conclusions
Photomontage
The Two Ways of Life, 1857, Oscar Gustav Rejlander Printed from the original 32 wet collodion negatives.
Photographic compositions
Lang Ching-shan
Use of mattes for compositing
The Great Train Robbery (1903) matte shot
Use of mattes for compositing
The Great Train Robbery (1903) matte shot
Optical compositing
King Kong (1933) Stop-motion + optical compositing
Digital matting and compositing
The lost world (1925) The lost world (1997)
Miniature, stop-motion Computer-generated images
Digital matting and composting
King Kong (1933) Jurassic Park III (2001)
Optical compositing
Blue-screen matting, digital composition, digital matte painting
Titanic
Matting and Compositing
Matting and Compositing
background replacement
background editing
Digital matting: bluescreen matting
Forrest Gump (1994)
• The most common approach for films.
• Expensive, studio setup.
• Not a simple one-step process.
Color difference method (Ultimatte)
Blue-screen photograph
C=F+αB
Spill suppression if B>G then B=G
F
Matte creation α=B-max(G,R)
α
demo with Paint Shop Pro (B=min(B,G))
Problems with color difference
Background color is usually not perfect! (lighting, shadowing…)
Chroma-keying (Primatte)
Chroma-keying (Primatte)
demo
Outline
• Traditional matting and compositing
• The matting problem
• Bayesian matting and extensions
• Matting with less user inputs
• Matting with multiple observations
• Beyond the compositing equation*
• Conclusions
Compositing
B F
C = α + (1− α)
F α B
C
foreground color alpha matte background plate composite
compositing equation
B
F C
α=0
Compositing
B F
C = α + (1− α)
F α B
C
composite
compositing equation
B
F
C α=1
Compositing
B F
C = α + (1− α)
F α B
C
composite
compositing equation
B
C F
α=0.6
Matting
C
observation
B F
C = α + (1− α)
compositing equation
F α B
Matting
C C = αF + (1− α)B
compositing equation
F α B
Three approaches:
1 reduce #unknowns 2 add observations 3 add priors
Matting (reduce #unknowns)
C
F BB
B F
C = α + (1− α)
difference matting
α
Matting (reduce #unknowns)
C F
B F
C = α + (1− α) B
blue screen matting
α
Matting (add observations)
F
B F
C = α + (1− α)
triangulation
α
B F
C = α + (1− α) B
C
Natural image matting
B C
Matting (add priors)
F
B F
C = α + (1− α)
α B
rotoscoping Ruzon-Tomasi
FG BG
unknown
Outline
• Traditional matting and compositing
• The matting problem
• Bayesian matting and extensions
• Matting with less user inputs
• Matting with multiple observations
• Beyond the compositing equation*
• Conclusions
Bayesian framework
f(z)+ε
z y
para- meters
observed signal )
| ( max
* P z y
z = z
) (
) ( )
| max (
y P
z P z
y P
= z
) ( )
| (
max L y z L z
z +
=
Example:
super-resolution de-blurring
de-blocking
…
Bayesian framework
) ( )
| (
max
* L y z L z
z
z +
=
2
) 2
(
σ
z f
data y −
evidence
a-priori knowledge
f(z)+ε
z y
para- meters
observed signal
Bayesian framework
likelihood priors posterior probability
Priors
Bayesian matting
Optimization
repeat
until converge 1. fix alpha
2. fix F and B
Demo
input trimapalpha
Results
input composite
Results
Comparisons
input imagetrimap
Comparisons
Bayesian Ruzon-Tomasi
Comparisons
Bayesian Ruzon-Tomasi
Comparisons
Mishima
Comparisons
Bayesian
Comparisons
input image
Comparisons
Bayesian Mishima
Comparisons
Bayesian Mishima
input video
Video matting
input key
trimaps input video
Video matting
interpo- lated trimaps
input video
Video matting
interpo- lated trimaps
input video
output alpha
Video matting
Compo- site
interpo- lated trimaps
input video
output alpha
Video matting
optical flow
optical flow
Sample composite
Garbage mattes
Garbage mattes
Background estimation
Background estimation
Alpha matte
Comparison
without background
with
background
C
B P(F)
Problems with Bayesian matting
• It requires fine trimaps for good results
• It is tedious to generate fine trimaps
• Its performance rapidly degrades when foreground and background patterns
become complex
• There is no direct and local control to the resulted mattes
Outline
• Traditional matting and compositing
• The matting problem
• Bayesian matting and extensions
• Matting with less user inputs
• Matting with multiple observations
• Beyond the compositing equation*
• Conclusions
Motivation
LazySnapping
LazySnapping
LazySnapping
LazySnapping
Matting approaches
• Sampling approaches: solve for each alpha separately by utilizing local
fg/bg samples, e.g. Ruzon/Tomasi, Knockout and Bayesian matting.
• Propagation approaches: solve the
whole matte together by optimizing, e.g. Poisson, BP, random walker,
closed-form and robust matting.
Poisson matting
Poisson matting
Robust matting
• Jue Wang and Michael Cohen, CVPR 2007
Robust matting
• Instead of fitting models, a non- parametric approach is used
Bayesian Robust
Robust matting
• We must evaluate hypothesized foreground/background pairs
Bj
Fi C
distance ratio
Robust matting
• To encourage pure fg/bg pixels, add weights
B F1
C
F2
Robust matting
• Combine them together. Pick up the best 3 pairs and average them
confidence
Robust matting
Robust matting
matte confidence
Matte optimization
Solved by Random Walk Algorithm
Matte optimization
data constraints
neighborhood constraints
Demo (EZ Mask)
Evaluation
• 8 images collected in 3 different ways
• Each has a “ground truth” matte
Evaluation
• Mean square error is used as the accuracy metric
• Try 8 trimaps with different accuracy for testing robustness
• 7 methods are tested: Bayesian,
Belief propagation, Poisson, Random Walk, KnockOut2, Closed-Form and Robust matting
Quantitative evaluation
Subjective evaluation
Subjective evaluation
Ranks of these algorithms
Poisson
Random walk Knockout2
Bayesian
Belief Propagation Close-form
Robust matting
accuracy 6.9
6.0 4.5 3.9 3.3 2.6 1.0
robustness 6.8
4.4 4.5 6.0 3.1 2.0 1.3
Summary
• Propagation-based methods are more robust
• Sampling-based methods often
generate more accurate mattes than propagation-based ones with fine
trimaps
• Robust matting combines strengths of both
Soft scissor
• Jue Wang et. al., SIGGRAPH 2007
• Users interact in a similar way to intelligent scissors
Flowchart
Flowchart
Soft scissor
Demo (Power Mask)
Outline
• Traditional matting and compositing
• The matting problem
• Bayesian matting and extensions
• Matting with less user inputs
• Matting with multiple observations
• Beyond the compositing equation*
• Conclusions
Matting with multiple observations
• Invisible lights
– Polarized lights – Infrared
• Thermo-key
• Depth Keying (ZCam)
• Flash matting
Invisible lights (Infared)
Invisible lights (Infared)
Invisible lights (Infared)
Invisible lights (Infared)
Invisible lights (Infared)
Invisible lights (Infared)
Invisible lights (Polarized)
Invisible lights (Polarized)
Thermo-Key
Thermo-Key
ZCam
Defocus matting
video
Matting with camera arrays
video
Flash matting
flash no flash matte
Flash matting
Background is much further than foreground and receives almost no flash light
Flash matting
Foreground flash matting equation
Generate a trimap and directly apply Bayesian matting.
Foreground flash matting
Joint Bayesian flash matting
Joint Bayesian flash matting
Comparison
flash no flash
Comparison
foreground flash matting
ioint Bayesian flash matting
Flash matting
Outline
• Traditional matting and compositing
• The matting problem
• Bayesian matting and extensions
• Matting with less user inputs
• Matting with multiple observations
• Beyond the compositing equation*
• Conclusions
Conclusions
• Matting algorithms improves a lot in these 10 years
• In production, it is still always
preferable to shoot against uniform backgrounds
• Algorithms for more complex backgrounds
• Devices or algorithms for automatic matting