Algorithm Design and Analysis
Slides credited from Hsueh-I Lu, Hsu-Chun Hsiao, & Michael Tsai
Midterm Feedback
• Mini-HW
• NTU COOL
• Helpful TAs
• Course recordings (access the channel here)
• Instant feedback
• Grade release
• Course recordings (two classes & last year)
• Homework hints
• TA hour changes
2
Outline
• DFS Applications
• Connected Components
• Strongly Connected Components
• Topological Sorting
• Minimal Spanning Trees (MST)
• Boruvka’s Algorithm
• Kruskal’s Algorithm
• Prim’s Algorithm
Depth-First Search
4
Textbook Chapter 22.3 – Depth-first search
Depth-First Search (DFS)
• Search as deep as possible and then backtrack until finding a new path
1
2 3
4 8
9 12 13
14
5 6
7
10 11
Connected Components
6
Connected Components Problem
• Input: a graph 𝐺 = 𝑉, 𝐸
• Output: a connected component of 𝐺
• a maximal subset 𝑈 of 𝑉 s.t. any two nodes in 𝑈 are connected in 𝐺
Connected Components
8
10 1
2
5
3 4
6
7
8 9
Time Complexity:
BFS and DSF both find the connected components with the same complexity
Problem Complexity
Strongly Connected Components
10
Textbook Chapter 22.5 – Strongly connected components
Strongly Connected Components
• Input: a directed graph 𝐺 = 𝑉, 𝐸
• Output: a connected component of 𝐺
• a maximal subset 𝑈 of 𝑉 s.t. any two nodes in 𝑈 are reachable in 𝐺
1
2
4
7
8
Why must the strongly connected components of a graph be disjoint?
Algorithm
• Step 1: Run DFS on 𝐺 to obtain the finish time 𝑣. 𝑓 for 𝑣 ∈ 𝑉.
• Step 2: Run DFS on the transpose of 𝐺 where the vertices 𝑉 are processed in the decreasing order of their finish time.
• Step 3: output the vertex partition by the second DFS
12
Transpose of A Graph
1
2
4
6
5
1
2
4
6
5
Example Illustration
14
1
3
2
6 5
4 1
2
4
5 3
6
Algorithm Correctness
• Proof by contradiction
• Assume that 𝑣, 𝑤 is an incoming edge to 𝐶.
• Since 𝐶 is a strongly connected component of 𝐺, there cannot be any path from any node of 𝐶 to 𝑣 in 𝐺.
• Therefore, the finish time of 𝑣 has to be larger than any node in 𝐶, including 𝑢. → Lemma
Let 𝐶 be the strongly connected component of 𝐺 (and 𝐺𝑇) that contains the node 𝑢 with the largest finish time 𝑢. 𝑓. Then 𝐶 cannot have any incoming edge from any node of 𝐺 not in 𝐶.
𝑢
G
C 𝑤
𝑣
Algorithm Correctness
• Practice to prove using induction
16
𝑢
G
C
𝑢
GT
C Theorem
By continuing the process from the vertex 𝑢∗ whose finish time 𝑢∗. 𝑓 is the largest excluding those in 𝐶, the algorithm returns the strongly connected components.
Example
1
3
2
6
4
Example
18
1
3
2
6 5
4
Time Complexity
• Step 1: Run DFS on 𝐺 to obtain the finish time 𝑣. 𝑓 for 𝑣 ∈ 𝑉.
• Step 2: Run DFS on the transpose of 𝐺 where the vertices 𝑉 are processed in the decreasing order of their finish time.
• Step 3: output the vertex partition by the second DFS
Time Complexity:
Problem Complexity
20
Topological Sort
Textbook Chapter 22.4 – Topological sort
Directed Graph
22
1
2
3
5 4
6 1
2
3
5 4
6
Directed Acyclic Graph (DAG)
• Definition
• a directed graph without any directed cycle
1
2
3
Topological Sort Problem
• Taking courses should follow the specific order
• How to find a course taking order?
24
計程 資料結構 演算法
計概 作業系統
計算機網路
微積分上 微積分下 機率
計組
Topological Sort Problem
• Input: a directed acyclic graph 𝐺 = (𝑉, 𝐸)
• Output: a linear order of 𝑉 s.t. all edges of 𝐺 going from lower-indexed nodes to higher-indexed nodes (左→右)
a b d
f c e
a
b
d
Algorithm
• Run DFS on the input DAG G.
• Output the nodes in decreasing order of their finish time.
26
DFS(G)
for each vertex u in G.V u.color = WHITE
u.pi = NIL time = 0
for each vertex u in G.V if u.color == WHITE
DFS-VISIT(G, u)
DFS-Visit(G, u) time = time + 1 u.d = time
u.color = GRAY
for each v in G.Adj[u] (outgoing) if v.color == WHITE
v.pi = u
DFS-VISIT(G, v) u.color = BLACK
time = time + 1
u.f = time // finish time
Example Illustration
a
b
d
f c
e
a b d
f c e
1
4 2
5
Example Illustration
28
a
b
d
f c
e
f b d
a c e
1
2
3 4
6
5
Time Complexity
• Run DFS on the input DAG G.
• Output the nodes in decreasing order of their finish time.
• As each vertex is finished, insert it onto the front of a linked list
• Return the linked list of vertices
DFS(G)
for each vertex u in G.V u.color = WHITE
u.pi = NIL
DFS-Visit(G, u) time = time + 1 u.d = time
u.color = GRAY
for each v in G.Adj[u]
if v.color == WHITE v.pi = u
Time Complexity:
Algorithm Correctness
• Proof
• →: suppose there is a back edge 𝑢, 𝑣
• 𝑣 is an ancestor of 𝑢 in DFS forest
• There is a path from 𝑣 to 𝑢 in 𝐺 and 𝑢, 𝑣 completes the cycle
• : suppose there is a cycle 𝑐
• Let 𝑣 be the first vertex in 𝑐 to be discovered and 𝑢 is a predecessor of 𝑣 in 𝑐
• Upon discovering 𝑣 the whole cycle from 𝑣 to 𝑢 is WHITE
• At time 𝑣. 𝑑, the vertices of 𝑐 form a path of white vertices from 𝑣 to 𝑢
• By the white-path theorem, vertex 𝑢 becomes a descendant of 𝑣 in the DFS forest
• Therefore, 𝑢, 𝑣 is a back edge
30
Lemma 22.11
A directed graph is acyclic a DFS yields no back edges.
White Path Theorem: In a DFS forest of 𝐺, 𝑣 is a descendant of 𝑢 in the forest at the time 𝑢. 𝑑 that the search discovers 𝑢, there is a path from 𝑢 to 𝑣 in 𝐺 consisting entirely of WHITE vertices
Algorithm Correctness
• Proof
• When 𝑢, 𝑣 is being explored, 𝑢 is GRAY and there are three cases for 𝑣:
• Case 1 – GRAY
• 𝑢, 𝑣 is a back edge (contradicting Lemma 22.11), so 𝑣 cannot be GRAY
• Case 2 – WHITE
• 𝑣 becomes descendant of 𝑢
• 𝑣 will be finished before 𝑢
Theorem 22.12
The algorithm produces a topological sort of the input DAG. That is, if 𝑢, 𝑣 is a directed edge (from 𝑢 to 𝑣) of 𝐺, then 𝑢. 𝑓 > 𝑣. 𝑓.
Problem Complexity
32
Discussion
• Since cycle detection becomes back edge detection (Lemma 22.11), DFS can be used to test whether a graph is a DAG
• Is there a topological order for cyclic graphs?
• Given a topological order, is there always a DFS traversal that produces such
an order?
Minimal Spanning Tree (MST)
34
Textbook Chapter 23 – Minimal Spanning Trees
Spanning Tree
• Definition
• a subgraph that is a tree and connects all vertices
• Exactly 𝑛 − 1 edges
• Acyclic
• There can be many spanning trees of a graph
• BFS and DFS also generate spanning trees
• BFS tree is typically “short and bushy”
• DFS tree is typically “long and stringy”
2
1 1
2 2
3
1
Minimal Spanning Tree Problem
• Input: a connected 𝑛-node 𝑚-edge graph 𝐺 with edge weights 𝑤
• Output: a spanning tree 𝑇 of 𝐺 with minimum 𝑤(𝑇)
36
2
1
1
1
2
2 2
3
1
WLOG: we may assume that all edge weights are distinct
Minimal Spanning Tree Problem
• Q: What if the graph is unweighted?
• Q: What if the graph contains edges with negative weights?
Trivial
Add a large constant to every edge; a MST remains the same
Uniqueness of MST
• Proof by contradiction
• Suppose there are two MSTs 𝐴 and 𝐵
• Let 𝑒 be the least-weight edge in 𝐴⋃𝐵 and 𝑒 is not in both
• WLOG, assume 𝑒 is in 𝐴
• Add 𝑒 to 𝐵; 𝑒 ⋃𝐵 contains a cycle 𝐶
• B includes at least one edge 𝑒′ that is not in 𝐴 but on 𝐶
• Replacing 𝑒′ with 𝑒 yields a MST with less cost
38
Theorem: MST is unique if all edge weights are distinct
If edge weights are not all distinct, then the (multi-)set of weights in MST is unique
Borůvka’s Algorithm
Inventor of MST
• Otakar Borůvka
• Czech scientist
• Introduced the problem
• Gave an 𝑂 𝑚 log 𝑛 time algorithm
• The original paper was written in Czech in 1926
• The purpose was to efficiently provide electric coverage of Bohemia
40
Borůvka’s Algorithm
• Repeat the below procedure until the resulting graph becomes a single node
• For each node 𝑢, mark its lightest incident edge
• From the marked edges form a forest 𝐹, add the edges of 𝐹 into the set of edges to be reported
• Contract each maximal subtree of 𝐹 into a single node
Borůvka’s Algorithm Illustration
42
2.1
1.3
2.3
1.2 2.2
3.1
2.4 3
1
1.5
1.4
2.6
2.7 2.5
3.2
5
3.3 4
4.1 5.1
Algorithm Correctness
• Proof via contradiction
• An MST 𝑇 of 𝐺 that does not contain 𝑢, 𝑣
• A cycle 𝐶 = 𝑇 ∪ 𝑢, 𝑣 contains an edge 𝑢, 𝑤 in 𝐶 that has larger weight than 𝑢, 𝑣
• 𝑇′ = 𝑇 ∪ 𝑢, 𝑣 \ 𝑢, 𝑤 must be a spanning tree of 𝐺 lighter than 𝑇
Claim: If 𝑢, 𝑣 is the lightest edge incident to 𝑢 in 𝐺, 𝑢, 𝑣 must belong
to any MST of 𝐺
Time Complexity
• The recurrence relation
• We check all edges in each phase
• After each contraction phase, the number of nodes is reduced by at least one half
• Time complexity:
44
Cycle Property
• Proof by contradiction
• Suppose 𝑒 is in the MST
• Removing 𝑒 disconnects the MST into two components T1 and T2
• There exists another edge 𝑒′ in 𝐶 that can reconnect T1 and T2
• Since 𝑤 𝑒’ < 𝑤(𝑒), the new tree has a lower weight
• Contradiction!
Let 𝐶 be any cycle in the graph 𝐺, and let 𝑒 be an edge with the maximum weight on 𝐶. Then the MST does not contain 𝑒.
• For simplicity, assume all edge weights are distinct
Cut Property
• Proof by contradiction
• Suppose 𝑒 is not in the current MST
• Adding 𝑒 creates a cycle in the MST
• There exists another edge 𝑒′ in 𝐶 that can break the cycle
• Since 𝑤 𝑒’ > 𝑤(𝑒), the new tree has a lower weight
• Contradiction!
46
Let 𝐶 be a cut in the graph, and let 𝑒 be the edge with the minimum cost in 𝐶. Then the MST contains 𝑒.
• Cut = a partition of the vertices
• For simplicity, assume all edge weights are distinct
Kruskal’s Algorithm
Textbook Chapter 23.2 – The algorithms of Kruskal and Prim
Kruskal’s Algorithm
• For each node 𝑢
• Make-set(𝑢): create a set consisting of 𝑢
• For each edge 𝑢, 𝑣 , taken in non-decreasing order by weights
• if Find-set(𝑢) ≠Find-set(𝑣) (i.e., 𝑢 and 𝑣 are not in the same set) then
• Output edge 𝑢, 𝑣
• Union(𝑢, 𝑣): union the sets containing 𝑢 and 𝑣 into a single set
48
Kruskal’s Algorithm Illustration
2.1
1.3 1.2
2.2
1
1.5
1.4 2.7
2.5 3.2
4.1 3.1 2.3
2.4 3
5 2.6 3.3 4
5.1
Kruskal’s Algorithm Correctness
50
The lightest edge incident to a vertex must be in the MST
Kruskal’s Algorithm Correctness
• Consider whether adding 𝑒 creates a cycle:
• If adding 𝑒 to 𝑇 creates a cycle 𝐶
• Then 𝑒 is the max weight edge in 𝐶
• The cycle property ensures that 𝑒 is not in the MST
• If adding 𝑒 = 𝑢, 𝑣 to 𝑇 does not create a cycle
• Before adding 𝑒, the current MST can be divided into two trees T1 and T2 such that 𝑢 in T1 and 𝑉 in T2
• 𝑒 is the minimum-cost edge on the cut of T1 and T2
• The cut property ensures that 𝑒 is in the MST
Kruskal’s Time Complexity
• Disjoint-set data structure with union-by-rank (Textbook Ch. 21)
• MAKE-SET:
• FIND-SET:
• UNION:
• The amortized cost of 𝑚 operations on 𝑛 elements (Exercise 21.4-4):
• Total complexity:
52 MST-KRUSKAL(G, w) // w = weights
A = empty // edge set of MST for v in G.V
MAKE-SET(v)
sort edges of G.E into non-decreasing order by weight w for (u, v) in G.E, taken in non-decreasing order by weight
if FIND-SET(u) ≠ FIND-SET(v) A = A ∪ {u, v}
UNION(u, v) return A
Prim’s Algorithm
Textbook Chapter 23.2 – The algorithms of Kruskal and Prim
Prim’s Algorithm
• Let 𝑇 consist of an arbitrary node
• For 𝑖 = 1 to 𝑛 − 1
• add the least-weighted edge incident to the current subtree 𝑇 that does not incur a cycle
54
Prim’s Algorithm Illustration
13
50 11
7
2
8 12
9
10 40 14
1 3 6
20
Prim’s Algorithm Illustration
56
13
50 11
7
2
8 12
9
10 40 14
1 3 6
20
Prim’s Algorithm Illustration
13
50 11
7
2
8 12
9
10 40 14
1 3 6
20
Prim’s Algorithm Illustration
58
13
50 11
7
2
8 12
9
10 40 14
1 3 6
20
Prim’s Algorithm Illustration
13
50 11
7
2
8 12
9
10 40 14
1 3 6
20
Prim’s Algorithm Illustration
60
13
50 11
7
2
8 12
9
10 40 14
1 3 6
20
Prim’s Algorithm Illustration
13
50 11
7
2
8 12
9
10 40 14
1 3 6
20
Prim’s Algorithm Illustration
62
13
50 11
7
2
8 12
9
10 40 14
1 3 6
20
Prim’s Algorithm Illustration
13
50 11
7
2
8 12
9
10 40 14
1 3 6
20
Prim’s Algorithm Correctness
64
The lightest edge incident to a vertex must be in the MST
Prim’s Time Complexity
• Binary min-heap (Textbook Ch. 6)
MST-PRIM(G, w, r) // w = weights, r = root for u in G.V
u.key = ∞ u.π = NIL r.key = 0 Q = G.V
while Q ≠ empty
u = EXTRACT-MIN(Q) for v in G.adj[u]
if v ∈ Q and w(u, v) < v.key v.π = u
v.key = w(u, v) // DECREASE-KEY
Prim’s Time Complexity
• Fibonacci heap (Textbook Ch. 19)
• BUILD-MIN-HEAP:
• EXTRACT-MIN: (amortized)
• DECREASE-KEY: (amortized)
• Total complexity:
66 MST-PRIM(G, w, r) // w = weights, r = root
for u in G.V u.key = ∞ u.π = NIL r.key = 0 Q = G.V
while Q ≠ empty
u = EXTRACT-MIN(Q) for v in G.adj[u]
if v ∈ Q and w(u, v) < v.key v.π = u
v.key = w(u, v) // DECREASE-KEY
Concluding Remarks
• Minimal Spanning Trees (MST)
• Boruvka’s Algorithm:
• Kruskal’s Algorithm:
• Prim’s Algorithm: with binary min-heap
• Prim’s Algorithm: with Fabonacci heap
Question?
Important announcement will be sent to
@ntu.edu.tw mailbox & post to the course website
Course Website: http://ada.miulab.tw Email: ada-ta@csie.ntu.edu.tw
