Algorithm Design and Analysis Dynamic Programming (1)

(1)

Algorithm Design and Analysis Dynamic Programming (1)

http://ada.miulab.tw

(2)

Announcement

• Mini-HW 3 released

• Due on 10/10 (Thu) 14:20

• Online submission

• Homework 1 released

• Due on 10/17 (Thur) 17:20 (2 weeks left)

• Writing: print out the A4 hard copy and submit to NTU COOL

• Programming: submit to Online Judge – http://ada-judge.csie.ntu.edu.tw

(3)

Outline

• Dynamic Programming

• DP #1: Rod Cutting

• DP #2: Stamp Problem

• DP #3: Sequence Alignment Problem

• Longest Common Subsequence (LCS) / Edit Distance

• Viterbi Algorithm

• Space Efficient Algorithm

• DP #4: Matrix-Chain Multiplication

• DP #5: Weighted Interval Scheduling

• DP #6: Knapsack Problem

• 0/1 Knapsack

• Unbounded Knapsack

• Multidimensional Knapsack

• Fractional Knapsack

(4)

動腦一下 – 囚犯問題

• 有100個死囚，隔天執行死刑，典獄長開恩給他們一個存活的機會。

• 當隔天執行死刑時，每人頭上戴一頂帽子(黑或白)排成一隊伍，在死刑執行前，由隊伍中最後的囚犯開始，每個人可以猜測自己頭上的帽子顏色(只允許說黑或白)，猜對則免除死刑，猜錯則執行死刑。

• 若這些囚犯可以前一天晚上先聚集討論方案，是否有好的方法可以使總共存活的囚犯數量期望值最高？

(5)

猜測規則

• 囚犯排成一排，每個人可以看到前面所有人的帽子，但看不到自己及後面囚犯的。

• 由最後一個囚犯開始猜測，依序往前。

• 每個囚犯皆可聽到之前所有囚犯的猜測內容。

……

Example: 奇數者猜測內容為前面一位的帽子顏色 → 存活期望值為75人有沒有更多人可以存活的好策略?

(6)

Algorithm Design Strategy

• Do not focus on “specific algorithms”

• But “some strategies” to “design” algorithms

• First Skill: Divide-and-Conquer (各個擊破/分治法)

• Second Skill: Dynamic Programming (動態規劃)

(7)

Dynamic Programming

Textbook Chapter 15 – Dynamic Programming

Textbook Chapter 15.3 – Elements of dynamic programming

(8)

What is Dynamic Programming?

• Dynamic programming, like the divide-and-conquer method, solves problems by combining the solutions to subproblems

• 用空間換取時間

• 讓走過的留下痕跡

• “Dynamic”: time-varying

• “Programming”: a tabular method

Dynamic Programming: planning over time

(9)

Algorithm Design Paradigms

• Divide-and-Conquer

• partition the problem into

independent or disjoint subproblems

• repeatedly solving the common subsubproblems

→ more work than necessary

• Dynamic Programming

• partition the problem into dependent or overlapping subproblems

• avoid recomputation

✓ Top-down with memoization

✓ Bottom-up method

(10)

Dynamic Programming Procedure

• Apply four steps

1. Characterize the structure of an optimal solution 2. Recursively define the value of an optimal solution

3. Compute the value of an optimal solution, typically in a bottom-up fashion 4. Construct an optimal solution from computed information

(11)

Rethink Fibonacci Sequence

• Fibonacci sequence (費波那契數列)

• Base case: F(0) = F(1) = 1

• Recursive case: F(n) = F(n-1) + F(n-2)

Fibonacci(n)

if n < 2 // base case return 1

// recursive case

return Fibonacci(n-1)+Fibonacci(n-2) F(5)

F(4) F(3)

F(3) F(2) F(2) F(1)

F(2) F(1) F(1) F(0) F(1) F(0)

F(1) F(0) Calling overlapping subproblems result in poor efficiency

✓F(3) was computed twice

✓F(2) was computed 3 times

(12)

Fibonacci Sequence

Top-Down with Memoization

• Solve the overlapping subproblems recursively with memoization

• Check the memo before making the calls

F(5)

F(4) F(3)

F(3) F(2)

F(2) F(1)

F(1) F(0)

備忘錄

n 0 1 2 3 4 5

F(n) 1 1 ?2 ?3 ?5 8?

Avoid recomputation of the same subproblems using memo

(13)

Fibonacci Sequence

Top-Down with Memoization

Memoized-Fibonacci(n)

// initialize memo (array a[]) a[0] = 1

a[1] = 1

for i = 2 to n a[i] = 0

return Memoized-Fibonacci-Aux(n, a) Memoized-Fibonacci-Aux(n, a)

if a[n] > 0 return a[n]

// save the result to avoid recomputation

a[n] = Memoized-Fibonacci-Aux(n-1, a) + Memoized-Fibonacci-Aux(n-2, a) return a[n]

(14)

Fibonacci Sequence

Bottom-Up Method

• Building up solutions to larger and larger subproblems

Bottom-Up-Fibonacci(n) if n < 2

return 1 a[0] = 1 a[1] = 1

for i = 2 … n

a[i] = a[i-1] + a[i-2]

return a[n]

F(5)

F(4)

F(3)

F(2)

F(1)

F(0) Avoid recomputation of the same subproblems

(15)

Optimization Problem

• Principle of Optimality

• Any subpolicy of an optimum policy must itself be an optimum policy with regard to the initial and terminal states of the subpolicy

• Two key properties of DP for optimization

• Overlapping subproblems

• Optimal substructure – an optimal solution can be constructed from optimal solutions to subproblems

✓ Reduce search space (ignore non-optimal solutions)

If the optimal substructure (principle of optimality) does not hold, then it is incorrect to use DP

(16)

Optimal Substructure Example

• Shortest Path Problem

• Input: a graph where the edges have positive costs

• Output: a path from S to T with the smallest cost

Taipei (T)

Tainan (S)

M

C_S→M C_M→T

C’_S→M < C_S→M?

The path costing C_S→M+ C_M→Tis the shortest path from S to T

→ The path with the cost C_S→Mmust be a shortest path from S to M

Proof by “Cut-and-Paste” argument (proof by contradiction):

Suppose that it exists a path with smaller cost C’_S→M, then we can

“cut” C and “paste” C’

(17)

DP#1: Rod Cutting

Textbook Chapter 15.1 – Rod Cutting

(18)

Rod Cutting Problem

• Input: a rod of length 𝑛 and a table of prices 𝑝

_𝑖

for 𝑖 = 1, … , 𝑛

• Output: the maximum revenue 𝑟

_𝑛

obtainable by cutting up the rod and selling the pieces

length 𝑖 (m) 1 2 3 4 5

price 𝑝_𝑖 1 5 8 9 10

4m

2m

(19)

Brute-Force Algorithm

• A rod with the length = 4

4m

3m 1m

2m 2m

1m 3m

2m 1m 1m

1m 2m 1m

1m 1m 1m

1m

→ 9

→ 8 + 1 = 9

→ 5 + 5 = 10

→ 1 + 8 = 9

→ 5 + 1 + 1 = 7

→ 1 + 5 + 1 = 7

→ 1 + 1 + 5 = 7

→ 1 + 1 + 1 + 1 = 4

length 𝑖 (m) 1 2 3 4 5

price 𝑝_𝑖 1 5 8 9 10

(20)

Brute-Force Algorithm

• A rod with the length = 𝑛

• For each integer position, we can choose “cut” or “not cut”

• There are 𝑛 – 1 positions for consideration

• The total number of cutting results is 2

^𝑛−1

= Θ 2

^𝑛−1

n

length 𝑖 (m) 1 2 3 4 5

price 𝑝_𝑖 1 5 8 9 10

(21)

Recursive Thinking

• We use a recursive function to solve the subproblems

• If we know the answer to the subproblem, can we get the answer to the original problem?

• Optimal substructure – an optimal solution can be constructed from optimal solutions to subproblems

𝑟_𝑛−𝑖 𝑟_𝑖

no cut

cut at the i-th position (from left to right)

𝑟_𝑛: the maximum

revenue obtainable for a rod of length 𝑛

(22)

Recursive Algorithms

• Version 1

• Version 2

• try to reduce the number of subproblems → focus on the left-most cut

no cut

cut at the i-th position (from left to right)

left-most value maximum value obtainable from the remaining part

𝑟_𝑛−𝑖 𝑝_𝑖

(23)

Recursive Procedure

• Focus on the left-most cut

• assume that we always cut from left to right → the first cut

optimal solution to subproblems

𝑟_𝑛−𝑖 𝑝_𝑖

𝑟_𝑛−1 𝑝₁

𝑟_𝑛−2 𝑝₂

: : optimal solution

(24)

Naïve Recursion Algorithm

• 𝑇 𝑛 = time for running

Cut-Rod(p, n)

Cut-Rod(p, n) // base case if n == 0

return 0

// recursive case q = -∞

for i = 1 to n

q = max(q, p[i] + Cut-Rod(p, n - i)) return q

(25)

Naïve Recursion Algorithm

• Rod cutting problem

Cut-Rod(p, n) // base case if n == 0

return 0

// recursive case q = -∞

for i = 1 to n

q = max(q, p[i] + Cut-Rod(p, n - i)) return q

CR(4)

CR(3) CR(0)

CR(2) CR(1) CR(1) CR(0)

CR(1) CR(0) CR(0)

CR(0)

Calling overlapping subproblems result in poor efficiency

CR(2) CR(1)

CR(0) CR(0)

(26)

Dynamic Programming

• Idea: use space for better time efficiency

• Rod cutting problem has overlapping subproblems and optimal substructures

→ can be solved by DP

• When the number of subproblems is polynomial, the time complexity is polynomial using DP

• DP algorithm

• Top-down: solve overlapping subproblems recursively with memoization

• Bottom-up: build up solutions to larger and larger subproblems

(27)

Dynamic Programming

• Top-Down with Memoization

• Solve recursively and memo the subsolutions (跳著填表)

• Suitable that not all subproblems should be solved

• Bottom-Up with Tabulation

• Fill the table from small to large

• Suitable that each small problem should be solved

f(0) f(1) f(2) … f(n) f(0) f(1) f(2) … f(n)

(28)

Algorithm for Rod Cutting Problem

Top-Down with Memoization

• 𝑇 𝑛 = time for running

Memoized-Cut-Rod(p, n)

// initialize memo (an array r[] to keep max revenue) r[0] = 0

for i = 1 to n

r[i] = -∞ // r[i] = max revenue for rod with length = i return Memorized-Cut-Rod-Aux(p, n, r)

Memoized-Cut-Rod-Aux(p, n, r) if r[n] >= 0

return r[n] // return the saved solution q = -∞

for i = 1 to n

q = max(q, p[i] + Memoized-Cut-Rod-Aux(p, n-i, r)) r[n] = q // update memo

return q

(29)

Algorithm for Rod Cutting Problem

Bottom-Up with Tabulation

• 𝑇 𝑛 = time for running

Bottom-Up-Cut-Rod(p, n)

Bottom-Up-Cut-Rod(p, n) r[0] = 0

for j = 1 to n // compute r[1], r[2], ... in order q = -∞

for i = 1 to j

q = max(q, p[i] + r[j - i]) r[j] = q

return r[n]

(30)

Rod Cutting Problem

• Input: a rod of length 𝑛 and a table of prices 𝑝

_𝑖

for 𝑖 = 1, … , 𝑛

• Output: the maximum revenue 𝑟

_𝑛

obtainable and the list of cut pieces

4m

2m

length 𝑖 (m) 1 2 3 4 5

price 𝑝_𝑖 1 5 8 9 10

(31)

Algorithm for Rod Cutting Problem

Bottom-Up with Tabulation

• Add an array to keep the cutting positions cut

Extended-Bottom-Up-Cut-Rod(p, n) r[0] = 0

for j = 1 to n //compute r[1], r[2], ... in order q = -∞

for i = 1 to j

if q < p[i] + r[j - i]

q = p[i] + r[j - i]

cut[j] = i // the best first cut for len j rod r[i] = q

return r[n], cut

Print-Cut-Rod-Solution(p, n)

(r, cut) = Extended-Bottom-up-Cut-Rod(p, n) while n > 0

print cut[n]

(32)

f(0) f(1) f(2) … f(n)

Dynamic Programming

• Top-Down with Memoization

• Better when some subproblems not be solved at all

• Solve only the required parts of subproblems

• Bottom-Up with Tabulation

• Better when all subproblems must be solved at least once

• Typically outperform top-down method by a constant factor

• No overhead for recursive calls

• Less overhead for maintaining the table f(0) f(1) f(2) … f(n) F(5)

F(4)

F(3)

F(2)

F(1)

F(0)

(33)

Informal Running Time Analysis

• Approach 1: approximate via (#subproblems) * (#choices for each subproblem)

• For rod cutting

• #subproblems = n

• #choices for each subproblem = O(n)

• → T(n) is about O(n²)

• Approach 2: approximate via subproblem graphs

(34)

Subproblem Graphs

• The size of the subproblem graph allows us to estimate the time complexity of the DP algorithm

• A graph illustrates the set of subproblems involved and how

subproblems depend on another 𝐺 = 𝑉, 𝐸 (E: edge, V: vertex)

• 𝑉 : #subproblems

• A subproblem is run only once

• |𝐸|: sum of #subsubproblems are needed for each subproblem

• Time complexity: linear to 𝑂( 𝐸 + 𝑉 )

Bottom-up: Reverse Topological Sort Top-down: Depth First Search

Graph Algorithm (taught later)

F(5)

F(4)

F(3)

F(2)

F(1)

F(0)

(35)

Dynamic Programming Procedure

1. Characterize the structure of an optimal solution

✓ Overlapping subproblems: revisit same subproblems

✓ Optimal substructure: an optimal solution to the problem contains within it optimal solutions to subproblems

2. Recursively define the value of an optimal solution

✓ Express the solution of the original problem in terms of optimal solutions for subproblems

3. Compute the value of an optimal solution

✓ typically in a bottom-up fashion

4. Construct an optimal solution from computed information

✓ Step 3 and 4 may be combined

(36)

Revisit DP for Rod Cutting Problem

1. Characterize the structure of an optimal solution 2. Recursively define the value of an optimal solution 3. Compute the value of an optimal solution

4. Construct an optimal solution from computed information

(37)

Step 1: Characterize an OPT Solution

• Step 1-Q1: What can be the subproblems?

• Step 1-Q2: Does it exhibit optimal structure? (an optimal solution can be represented by the optimal solutions to subproblems)

• Yes. → continue

• No. → go to Step 1-Q1 or there is no DP solution for this problem Rod Cutting Problem

Input: a rod of length 𝑛 and a table of prices 𝑝_𝑖 for 𝑖 = 1, … , 𝑛 Output: the maximum revenue 𝑟_𝑛 obtainable

(38)

Step 1: Characterize an OPT Solution

• Step 1-Q1: What can be the subproblems?

• Subproblems: Cut-Rod(0), Cut-Rod(1), …, Cut-Rod(n-1)

• Cut-Rod(i): rod cutting problem with length-i rod

• Goal: Cut-Rod(n)

• Suppose we know the optimal solution to Cut-Rod(i), there are i cases:

• Case 1: the first segment in the solution has length 1

• Case i: the first segment in the solution has length i

從solution中拿掉一段長度為1的鐵條, 剩下的部分是Cut-Rod(i-1)的最佳解從solution中拿掉一段長度為2的鐵條, 剩下的部分是Cut-Rod(i-2)的最佳解

從solution中拿掉一段長度為i的鐵條, 剩下的部分是Cut-Rod(0)的最佳解

Rod Cutting Problem

:

(39)

Step 1: Characterize an OPT Solution

• Step 1-Q2: Does it exhibit optimal structure? (an optimal solution can be represented by the optimal solutions to subproblems)

• Yes. Prove by contradiction.

(40)

Step 2: Recursively Define the Value of an OPT Solution

• Suppose we know the optimal solution to Cut-Rod(i), there are i cases:

:

• Case i: the first segment in the solution has length i

• Recursively define the value

從solution中拿掉一段長度為1的鐵條, 剩下的部分是Cut-Rod(i-1)的最佳解

從solution中拿掉一段長度為2的鐵條, 剩下的部分是Cut-Rod(i-2)的最佳解

從solution中拿掉一段長度為i的鐵條, 剩下的部分是Cut-Rod(0)的最佳解

(41)

Step 3: Compute Value of an OPT Solution

• Bottom-up method: solve smaller subproblems first

i 0 1 2 3 4 5 … n

r[i]

Bottom-Up-Cut-Rod(p, n) r[0] = 0

for i = 1 to j

q = max(q, p[i] + r[j - i]) r[j] = q

(42)

Step 4: Construct an OPT Solution by Backtracking

• Bottom-up method: solve smaller subproblems first

i 0 1 2 3 4 5 … n

r[i] 0

cut[i] 0 ¹ 1

2 5

3 8

2 10

length 𝑖 1 2 3 4 5

price 𝑝_𝑖 1 5 8 9 10

(43)

Step 4: Construct an OPT Solution by Backtracking

Cut-Rod(p, n) r[0] = 0

for i = 1 to j

if q < p[i] + r[j - i]

q = p[i] + r[j - i]

cut[j] = i // the best first cut for len j rod r[i] = q

return r[n], cut

Print-Cut-Rod-Solution(p, n) (r, cut) = Cut-Rod(p, n) while n > 0

print cut[n]

n = n – cut[n] // remove the first piece

(44)

DP#2: Stamp Problem

(45)

Stamp Problem

• Input: the postage 𝑛 and the stamps with values 𝑣

₁

, 𝑣

₂

, … , 𝑣

_𝑘

• Output: the minimum number of stamps to cover the postage

(46)

A Recursive Algorithm

• The optimal solution 𝑆

_𝑛

can be recursively defined as

Stamp(v, n) r_min = ∞

if n == 0 // base case return 0

for i = 1 to k // recursive case r[i] = Stamp(v, n - v[i])

if r[i] < r_min r_min = r[i]

return r_min + 1

(47)

Step 1: Characterize an OPT Solution

• Subproblems

• S(i): the min #stamps with postage i

• Goal: S(n)

• Optimal substructure: suppose we know the optimal solution to S(i), there are k cases:

• Case 1: there is a stamp with v₁ in OPT

• Case 2: there is a stamp with v₂ in OPT

:

• Case k: there is a stamp with v_k in OPT

Stamp Problem

Input: the postage 𝑛 and the stamps with values 𝑣₁, 𝑣₂, … , 𝑣_𝑘 Output: the minimum number of stamps to cover the postage

從solution中拿掉一張郵資為v₁的郵票, 剩下的部分是S(i-v[1])的最佳解從solution中拿掉一張郵資為v₂的郵票, 剩下的部分是S(i-v[2])的最佳解

從solution中拿掉一張郵資為v 的郵票, 剩下的部分是S(i-v[k])的最佳解

(48)

Step 2: Recursively Define the Value of an OPT Solution

• Suppose we know the optimal solution to S(i), there are k cases:

• Case 1: there is a stamp with v₁ in OPT

• Case 2: there is a stamp with v₂ in OPT

:

• Case k: there is a stamp with v_k in OPT

• Recursively define the value

從solution中拿掉一張郵資為v₁的郵票, 剩下的部分是S(i-v[1])的最佳解

從solution中拿掉一張郵資為v₂的郵票, 剩下的部分是S(i-v[2])的最佳解

從solution中拿掉一張郵資為v_k的郵票, 剩下的部分是S(i-v[k])的最佳解

Stamp Problem

(49)

Stamp Problem

Step 3: Compute Value of an OPT Solution

• Bottom-up method: solve smaller subproblems first

i 0 1 2 3 4 5 … n

S[i]

Stamp(v, n) S[0] = 0

for i = 1 to n // compute r[1], r[2], ... in order r_min = ∞

for j = 1 to k

if S[i - v[j]] < r_min r_min = 1 + S[i – v[j]]

S[i] = r_min

(50)

Step 4: Construct an OPT Solution by Backtracking

Stamp(v, n) S[0] = 0

for i = 1 to n r_min = ∞

for j = 1 to k

if S[i - v[j]] < r_min

r_min = 1 + S[i – v[j]]

B[i] = j // backtracking for stamp with v[j]

S[i] = r_min return S[n], B

Print-Stamp-Selection(v, n) (S, B) = Stamp(v, n)

while n > 0 print B[n]

n = n – v[B[n]]

(51)

DP#3: Sequence Alignment

Textbook Chapter 15.4 – Longest common subsequence Textbook Problem 15-5 – Edit distance

(52)

Monkey Speech Recognition

• 猴子們各自講話，經過語音辨識系統後，哪一支猴子發出最接近英文字”banana”的語音為優勝者

• How to evaluate the similarity between two sequences?

aeniqadikjaz

svkbrlvpnzanczyqza

banana

(53)

Longest Common Subsequence (LCS)

• Input: two sequences

• Output: longest common subsequence of two sequences

• The maximum-length sequence of characters that appear left-to-right (but not necessarily a continuous string) in both sequences

X = banana

Y = svkbrlvpnzanczyqza X → ---ba---n-an---a Y → svkbrlvpnzanczyqza X = banana

Y = aeniqadikjaz X → ba-n--an---a- Y → -aeniqadikjaz

The infinite monkey theorem: a monkey hitting keys at random

4 5

(54)

Edit Distance

• Input: two sequences

• Output: the minimum cost of transformation from X to Y

• Quantifier of the dissimilarity of two strings

X = banana

Y = svkbrlvpnzanczyqza X → ---ba---n-an---a Y → svkbrlvpnzanczyqza X = banana

Y = aeniqadikjaz X → ba-n--an---a- Y → -aeniqadikjaz

1 deletion, 7 insertions, 1 substitution 12 insertions, 1 substitution

9 13

(55)

Sequence Alignment Problem

• Input: two sequences

• Output: the minimal cost 𝑀

_𝑚,𝑛

for aligning two sequences

• Cost = #insertions × 𝐶_INS + #deletions × 𝐶_DEL + #substitutions × 𝐶_𝑝,𝑞

(56)

Step 1: Characterize an OPT Solution

• Subproblems

• SA(i, j): sequence alignment between prefix strings 𝑥₁, … , 𝑥_𝑖 and 𝑦₁, … , 𝑦_𝑗

• Goal: SA(m, n)

• Optimal substructure: suppose OPT is an optimal solution to SA(i, j), there are 3 cases:

• Case 1: 𝑥_𝑖 and 𝑦_𝑗 are aligned in OPT (match or substitution)

• OPT/{𝑥_𝑖, , 𝑦_𝑗} is an optimal solution of SA(i-1, j-1)

• Case 2: 𝑥_𝑖 is aligned with a gap in OPT (deletion)

• OPT is an optimal solution of SA(i-1, j)

• Case 3: 𝑦_𝑗 is aligned with a gap in OPT (insertion)

• OPT is an optimal solution of SA(i, j-1)

Sequence Alignment Problem Input: two sequences

Output: the minimal cost 𝑀_𝑚,𝑛 for aligning two sequences

(57)

Step 2: Recursively Define the Value of an OPT Solution

• Suppose OPT is an optimal solution to SA(i, j) , there are 3 cases:

• Case 1: 𝑥_𝑖 and 𝑦_𝑗 are aligned in OPT (match or substitution)

• OPT/{𝑥_𝑖, , 𝑦_𝑗} is an optimal solution of SA(i-1, j-1)

• Case 2: 𝑥_𝑖 is aligned with a gap in OPT (deletion)

• OPT is an optimal solution of SA(i-1, j)

• Case 3: 𝑦_𝑗 is aligned with a gap in OPT (insertion)

• OPT is an optimal solution of SA(i, j-1)

• Recursively define the value

(58)

Step 3: Compute Value of an OPT Solution

• Bottom-up method: solve smaller subproblems first

X\Y 0 1 2 3 4 5 … n

0 1 : m

(59)

Step 3: Compute Value of an OPT Solution

• Bottom-up method: solve smaller subproblems first

X\Y 0 1 2 3 4 5 6 7 8 9 10 11 12

0 0 4 8 12 16 20 24 28 32 36 40 44 48 1 4 7 11 15 19 23 27 31 35 39 43 47 51 2 8 4 8 12 16 20 23 27 31 35 39 43 47 3 12 8 12 8 12 16 20 24 28 32 36 40 44 4 16 12 15 12 15 19 16 20 24 28 32 36 40 5 20 16 19 15 19 22 20 23 27 31 35 39 43

a e n i q a d i k j a z

b a n a n

(60)

Step 3: Compute Value of an OPT Solution

• Bottom-up method: solve smaller subproblems first

Seq-Align(X, Y, C_DEL, C_INS, C_p,q) for j = 0 to n

M[0][j] = j * C_INS // |X|=0, cost=|Y|*penalty for i = 1 to m

M[i][0] = i * C_DEL // |Y|=0, cost=|X|*penalty for i = 1 to m

for j = 1 to n

M[i][j] = min(M[i-1][j-1]+C_xi,yi, M[i-1][j]+C_DEL, M[i][j-1]+C_INS) return M[m][n]

(61)

Step 4: Construct an OPT Solution by Backtracking

• Bottom-up method: solve smaller subproblems first

X\Y 0 1 2 3 4 5 6 7 8 9 10 11 12

0 0 4 8 12 16 20 24 28 32 36 40 44 48 1 4 7 11 15 19 23 27 31 35 39 43 47 51 2 8 4 8 12 16 20 23 27 31 35 39 43 47 3 12 8 12 8 12 16 20 24 28 32 36 40 44 4 16 12 15 12 15 19 16 20 24 28 32 36 40 5 20 16 19 15 19 22 20 23 27 31 35 39 43

a e n i q a d i k j a z

b a n a n

(62)

Step 4: Construct an OPT Solution by Backtracking

• Bottom-up method: solve smaller subproblems first

Find-Solution(M) if m = 0 or n = 0

return {}

v = min(M[m-1][n-1] + C_xm,yn, M[m-1][n] + C_DEL, M[m][n-1] + C_INS) if v = M[m-1][n] + C_DEL// ↑: deletion

return Find-Solution(m-1, n)

if v = M[m][n-1] + C_INS// ←: insertion return Find-Solution(m, n-1)

return {(m, n)} ∪ Find-Solution(m-1, n-1) // ↖: match/substitution

(63)

Step 4: Construct an OPT Solution by Backtracking

Seq-Align(X, Y, C_DEL, C_INS, C_p,q) for j = 0 to n

M[0][j] = j * C_INS // |X|=0, cost=|Y|*penalty for i = 1 to m

M[i][0] = i * C_DEL // |Y|=0, cost=|X|*penalty for i = 1 to m

for j = 1 to n

M[i][j] = min(M[i-1][j-1]+C_xi,yi, M[i-1][j]+C_DEL, M[i][j-1]+C_INS) return M[m][n]

Find-Solution(M) if m = 0 or n = 0

return {}

v = min(M[m-1][n-1] + C_xm,yn, M[m-1][n] + C_DEL, M[m][n-1] + C_INS) if v = M[m-1][n] + C_DEL// ↑: deletion

return Find-Solution(m-1, n)

if v = M[m][n-1] + C_INS// ←: insertion return Find-Solution(m, n-1)

(64)

Space Complexity

• Space complexity

• If only keeping the most recent two rows: Space-Seq-Align(X, Y)

X\Y 0 1 2 3 … j … n

i - 1 i

The optimal value can be computed, but the solution cannot be reconstructed

X\Y 0 1 2 3 4 5 … n

0 1 : m

(65)

Space-Efficient Solution

• Problem: find the min-cost alignment → find the shortest path

Divide-and-Conquer +

Dynamic Programming

a

e p p l

p e a

X\Y 0 1 2 3

0 0 4 8 12

1 4 7 11 15

2 8 4 8 12

3 12 8 12 8

4 16 12 15 12 5 20 16 19 15 a p e

p p l e a

→ distance = C_INS

↓ distance = C_DEL

↘ distance = C for edge (u, v) START

END

(66)

𝐹 2,3 = distance of the shortest path

Shortest Path in Graph

• Each edge has a length/cost

• 𝐹 𝑖, 𝑗 : length of the shortest path from 0,0 to 𝑖, 𝑗 (START → 𝑖, 𝑗 )

• 𝐵 𝑖, 𝑗 : length of the shortest path from 𝑖, 𝑗 to 𝑚, 𝑛 ( 𝑖, 𝑗 → END)

• 𝐹 𝑚, 𝑛 = 𝐵 0,0

i = 0

4 1 2 3

j = 0 1 2 3 4 5 6 7

5 𝐵 2,3 = distance of the

shortest path

(67)

Recursive Equation

• Each edge has a length/cost

• 𝐹 𝑖, 𝑗 : length of the shortest path from 0,0 to 𝑖, 𝑗 (START → 𝑖, 𝑗 )

• 𝐵 𝑖, 𝑗 : length of the shortest path from 𝑖, 𝑗 to 𝑚, 𝑛 ( 𝑖, 𝑗 → END)

• Forward formulation

• Backward formulation

i = 0

4 1 2 3

j = 0 1 2 3 4 5 6 7

5

(68)

Shortest Path Problem

• Observation 1: the length of the shortest path from 0,0 to 𝑚, 𝑛 that passes through 𝑖, 𝑗 is 𝐹 𝑖, 𝑗 + 𝐵 𝑖, 𝑗

𝐹 𝑖, 𝑗 : length of the shortest path from 0,0 to 𝑖, 𝑗 𝐵 𝑖, 𝑗 : length of the shortest path from 𝑖, 𝑗 to 𝑚, 𝑛

i = 0

4 1 2 3

j = 0 1 2 3 4 5 6 7

𝑭 𝒊, 𝒋

𝑩 𝒊, 𝒋

→ optimal substructure

(69)

Shortest Path Problem

• Observation 2: for any 𝑣 in {0, … , 𝑛}, there exists a 𝑢 s.t. the shortest path between (0,0) and 𝑚, 𝑛 goes through (𝑢, 𝑣)

→ the shortest path must go across a vertical cut 𝐹 𝑖, 𝑗 : length of the shortest path from 0,0 to 𝑖, 𝑗

𝐵 𝑖, 𝑗 : length of the shortest path from 𝑖, 𝑗 to 𝑚, 𝑛

i = 0

4 1 2 3

j = 0 1 2 3 4 5 6 7

(70)

Shortest Path Problem

• Observation 1+2:

i = 0

4 1 2 3

j = 0 ¹ 2 3 4 5 ⁶ 7

5

i = 0

4 1 2 3

j = 0 1 2 3 4 5 ⁶ 7

5

𝐹 𝑖, 𝑗 : length of the shortest path from 0,0 to 𝑖, 𝑗 𝐵 𝑖, 𝑗 : length of the shortest path from 𝑖, 𝑗 to 𝑚, 𝑛

(71)

Divide-and-Conquer Algorithm

• Goal: finds optimal solution

How to find the value of 𝑢^∗?

▪ Idea: utilize sequence alignment algo.

▪ Call Space-Seq-Align(X,Y[1:v]) to find 𝐹 0, 𝑣 , 𝐹 1, 𝑣 , … , 𝐹 𝑚, 𝑣

▪ Call Back-Space-Seq-Align(X,Y[v+1:n]) to find 𝐵 0, 𝑣 , 𝐵 1, 𝑣 , … , 𝐵 𝑚, 𝑣

▪ Let 𝑢 be the index minimizing 𝐹 𝑢, 𝑣 + 𝐵 𝑢, 𝑣

(72)

Divide-and-Conquer Algorithm

• Goal: finds optimal solution – DC-Align(X, Y)

1. Divide

2. Conquer

3. Combine

▪ Divide the sequence of size n into 2 subsequences

▪ Find 𝑢 to minimize 𝐹 𝑢, 𝑣 + 𝐵 𝑢, 𝑣

▪ Recursive case (𝑛 > 1)

▪ prefix

= DC-Align(X[1:u], Y[1:v])

▪ suffix

= DC-Align(X[u+1:m], Y[v+1:n])

▪ Base case (𝑛 = 1)

▪ Return Seq-Align(X, Y)

▪ Return prefix + suffix

▪ 𝑇 𝑚, 𝑛 = time for running ^DC-

Align(X, Y) with 𝑋 = 𝑚, 𝑌 = 𝑛

Space Complexity:

(73)

Time Complexity Analysis

• Theorem

• Proof

• There exists positive constants 𝑎, 𝑏 s.t. all

• Use induction to prove

Inductive hypothesis

when Practice to check the initial condition

(74)

Extension: 注音文 Recognition

• Given a graph 𝐺 = 𝑉, 𝐸 , each edge 𝑢, 𝑣 ∈ 𝐸 has an associated non-

negative probability 𝑝 𝑢, 𝑣 of traversing the edge 𝑢, 𝑣 and producing the corresponding character. Find the most probable path with the label 𝑠 =

𝜎

₁

, 𝜎

₂

, … , 𝜎

_𝑛

.

ㄨㄅㄒㄎㄕ

START

我烏為問

END

爸不

想續小

考看卡

書

試上

白鄉

Find the path from START to END with highest prob

(75)

Viterbi Algorithm

𝜎₁ 𝜎₂ … … 𝜎_𝑛

START END

produce 𝜎₁

produce 𝜎_𝑗

V: vocabulary size

(76)

To Be Continued…

(77)

Question?

Important announcement will be sent to

@ntu.edu.tw mailbox & post to the course website

Course Website: http://ada.miulab.tw Email: ada-ta@csie.ntu.edu.tw