Graph Cut
Digital Visual Effects g Yung-Yu Chuang
with slides by Fredo Durand, Ramesh Raskar
Graph cut
Graph cut
• Interactive image segmentation using graph cut
Bi l b l f d b k d
• Binary label: foreground vs. background
• User labels some pixels
– similar to trimap, usually sparser
• Exploit
F F B
p
– Statistics of known Fg & Bg – Smoothness of label
F F
F F B
B Smoothness of label
• Turn into discrete graph optimization
Graph cut (min cut / max flow)
B F B – Graph cut (min cut / max flow)
Energy function
• Labeling: one value per pixel, F or B
• Energy(labeling) = data + smoothness
• Energy(labeling) data + smoothness
– Very general situation
– Will be minimized One labeling
• Data: for each pixel
– Probability that this color belongs to F (resp. B)
g (ok, not best)
– Similar in spirit to Bayesian matting
• Smoothness (aka regularization):
i hb i i l i Data
per neighboring pixel pair
– Penalty for having different label – Penalty is downweighted if the two
Data
– Penalty is downweighted if the two pixel colors are very different – Similar in spirit to bilateral filter
Smoothness Smoothness
Data term
• A.k.a regional term
(because integrated over full region) (because integrated over full region)
• D(L)= ( )
ii-log h[L g [
ii](C ](
ii) )
• Where i is a pixel
L
iis the label at i (F or B), L
iis the label at i (F or B), C
iis the pixel value
h[L
i] is the histogram of the observed Fg h[L
i] is the histogram of the observed Fg (resp Bg)
• Note the minus sign
• Note the minus sign
Hard constraints
• The user has provided some labels Th i k d di i l d
• The quick and dirty way to include
constraints into optimization is to replace the d t t b h lt if t t d data term by a huge penalty if not respected.
• D(L_i)=0 if respected
• D(L_i)=K if not respected
– e.g. K=- #pixelsg p
Smoothness term
• a.k.a boundary term, a.k.a. regularization S(L) B(C C ) (L L )
• S(L)=
{j, i} in NB(C
i,C
j) (L
i-L
j)
• Where i,j are neighbors
– e.g. 8-neighborhood
(but I show 4 for simplicity)
(L L ) is 0 if L L 1 otherwise
• (L
i-L
j) is 0 if L
i=L
j, 1 otherwise
• B(C
i,C
j) is high when C
iand C
jare similar, low if there is a discontinuity between those two pixels there is a discontinuity between those two pixels
– e.g. exp(-||Ci-Cj||2/22) where can be a constant – where can be a constant
or the local variance
• Note positive sign
• Note positive sign
Optimization
• E(L)=D(L)+ S(L)
i bl k i
• is a black-magic constant
• Find the labeling that minimizes E
• In this case, how many possibilities?
– 22 (512)9(512)
– We can try them all!
– What about megapixel images?What about megapixel images?
Labeling as a graph problem
• Each pixel = node Add d F & B
• Add two nodes F & B
• Labeling: link each pixel to either F or B
F
Desired resultB
Data term
• Put one edge between each pixel and F & G W i h f d i d
• Weight of edge = minus data term
– Don’t forget huge weight for hard constraints – Careful with sign
F
B
Smoothness term
• Add an edge between each neighbor pair W i h h
• Weight = smoothness term F
B
Min cut
• Energy optimization equivalent to min cut
C d di F f B
• Cut: remove edges to disconnect F from B
• Minimum: minimize sum of cut edge weight
F
cutB
Min cut <=> labeling
• In order to be a cut:
F h i l ith th F G d h t b t – For each pixel, either the F or G edge has to be cut
• In order to be minimal
– Only one edge label per pixel can be cut (otherwise could
F
cut(otherwise could be added)
B
Energy minimization via graph cuts
edge weight ) , , (x y d3 D
d3
d2
d1
Labels edge weight
Labels
(disparities) V(d1,d1)
Energy minimization via graph cuts
d
d d3
d1 d2
d1
• Graph Cost
– Matching cost between images – Neighborhood matching term
– Goal: figure out which labels are connected to g which pixels
Energy minimization via graph cuts
d
d d3
d1 d2
d1
• Graph Cut
– Delete enough edges so that
• each pixel is (transitively) connected to exactly one label node
label node
– Cost of a cut: sum of deleted edge weights Finding min cost cut equivalent to finding global – Finding min cost cut equivalent to finding global
minimum of energy function
Computing a multiway cut
• With 2 labels: classical min-cut problem
– Solvable by standard flow algorithms
• polynomial time in theory, nearly linear in practice
More than 2 terminals: NP hard – More than 2 terminals: NP-hard
[Dahlhaus et al., STOC ‘92]
• Efficient approximation algorithms exist
• Efficient approximation algorithms exist
– Within a factor of 2 of optimal
– Computes local minimum in a strong senseComputes local minimum in a strong sense
• even very large moves will not improve the energy
– Yuri Boykov, Olga Veksler and Ramin Zabih, Fast Approximate Energy Minimization via Graph Cuts International Conference on Computer Minimization via Graph Cuts, International Conference on Computer Vision, September 1999.
Move examples
Red-blue swap move
Starting point
Green expansion move
The swap move algorithm
1. Start with an arbitrary labeling
2. Cycle through every label pair (A,B) in some ordery g y p ( , )
2.1 Find the lowest E labeling within a single AB-swap 2.2 Go there if E is lower than the current labeling
3. If E did not decrease in the cycle, we’re done Otherwise, go to step 2
B B
O i i l h
A
AB b h
A
Original graph AB subgraph
(run min-cut on this graph)
The expansion move algorithm
1. Start with an arbitrary labeling
2 Cycle through every label A in some order 2. Cycle through every label A in some order
2.1 Find the lowest E labeling within a single A-expansion 2.2 Go there if it E is lower than the current labelingg
3. If E did not decrease in the cycle, we’re done Otherwise, go to step 2
GrabCut GrabCut
Interacti e Foregro nd E traction Interacti e Foregro nd E traction Interactive Foreground Extraction Interactive Foreground Extraction using Iterated Graph Cuts
using Iterated Graph Cuts Carsten Rother
Carsten Rother Carsten Rother Carsten Rother
Vladimir Kolmogorov Vladimir Kolmogorov Andrew Blake
Andrew Blake Andrew Blake Andrew Blake
Microsoft Research Cambridge Microsoft Research Cambridge--UK UK
Demo
• video
Interactive Digital Photomontage
• Combining multiple photos
• Combining multiple photos
• Find seams using graph cuts
• Combine gradients and integrate
• Combine gradients and integrate
Aseem Agarwala, Mira Dontcheva, Maneesh Agrawala, Steven Drucker, Alex Colburn, Brian Curless, David Salesin, Michael Cohen, “Interactive Digital Photomontage”, SIGGRAPH 2004
actual photomontage
set of originals perceived
Source images Brush strokes Computed labeling
Composite
Brush strokes Computed labeling
Brush strokes Computed labeling