Slides credited from Hsu-Chun Hsiao
▪
Mini-HW 5 released
▪ Due on 10/26 (Thu) 17:20
▪
Homework 1 due soon
▪
Homework 2
▪ Due on 11/09 (Thur) 17:20 (4 weeks)
▪
TA Recitation (next week)
▪ 10/26 (Thu) at R103
▪ Homework 1 QA
▪
Another course website you can get the videos sooner
▪ http://ada.miulab.tw
▪
Note: if you have questions about the homework, please find TAs
Frequently check the website for the updated information! 2
3
▪ Dynamic Programming
▪ DP #1: Rod Cutting
▪ DP #2: Stamp Problem
▪ DP #3: Matrix-Chain Multiplication
▪ DP #4: Sequence Alignment Problem
▪ Longest Common Subsequence (LCS) / Edit Distance
▪ Space Efficient Algorithm
▪ Viterbi Algorithm
▪ DP #5: Weighted Interval Scheduling
▪ DP #6: Knapsack Problem
▪ 0-1 Knapsack
▪ Unbounded Knapsack
▪ Multidimensional Knapsack
▪ Multi-Choice Knapsack
▪ Fractional Knapsack
4
▪
有100個死囚,隔天執行死刑,典獄長開恩給他們一個存活的機會。
▪
當隔天執行死刑時,每人頭上戴一頂帽子(黑或白)排成一隊伍,在
死刑執行前,由隊伍中最後的囚犯開始,每個人可以猜測自己頭上 的帽子顏色(只允許說黑或白),猜對則免除死刑,猜錯則執行死刑。
▪
若這些囚犯可以前一天晚上先聚集討論方案,是否有好的方法可以
使總共存活的囚犯數量期望值最高?
5
▪
囚犯排成一排,每個人可以看到前面所有人的帽子,但看不到自己 及後面囚犯的。
▪
由最後一個囚犯開始猜測,依序往前。
▪
每個囚犯皆可聽到之前所有囚犯的猜測內容。
6
……
Example: 奇數者猜測內 容為前面一位的帽子顏 色 存活期望值為75人 有沒有更多人可以存活的好策略?
▪
http://qstn.co/q/OOZK7sYQ
7
8
▪
Divide-and-Conquer
▪ partition the problem into independent or disjoint subproblems
▪ repeatedly solving the common subsubproblems
more work than necessary
▪
Dynamic Programming
▪ partition the problem into dependent or overlapping subproblems
▪ avoid recomputation
✓Top-down with memoization
✓Bottom-up method
9
1.
Characterize the structure of an optimal solution
✓
Overlapping subproblems: revisit same subproblems
✓
Optimal substructure: an optimal solution to the problem contains within it optimal solutions to subproblems
2.
Recursively define the value of an optimal solution
✓
Express the solution of the original problem in terms of optimal solutions for subproblems
3.
Compute the value of an optimal solution
✓
typically in a bottom-up fashion
4.
Construct an optimal solution from computed information
✓
Step 3 and 4 may be combined
10
Textbook Chapter 15.4 – Longest common subsequence Textbook Problem 15-5 – Edit distance
11
▪
猴子們各自講話,經過語音辨識系統後,哪一支猴子發出最接近英 文字”banana”的語音為優勝者
▪
How to evaluate the similarity between two sequences?
12
aeniqadikjaz
svkbrlvpnzanczyqza
banana
▪
Input: two sequences
▪
Output: longest common subsequence of two sequences
▪ The maximum-length sequence of characters that appear left-to-right (but not necessarily a continuous string) in both sequences
13
X = banana
Y = svkbrlvpnzanczyqza X → ---ba---n-an---a Y → svkbrlvpnzanczyqza X = banana
Y = aeniqadikjaz X → ba-n--an---a- Y → -aeniqadikjaz
4 5
▪
Input: two sequences
▪
Output: the minimum cost of transformation from X to Y
▪ Quantifier of the dissimilarity of two strings
14
X = banana
Y = svkbrlvpnzanczyqza X → ---ba---n-an---a Y → svkbrlvpnzanczyqza X = banana
Y = aeniqadikjaz X → ba-n--an---a- Y → -aeniqadikjaz
1 deletion, 7 insertions, 1 substitution 12 insertions, 1 substitution
9 13
▪
Input: two sequences
▪
Output: the minimal cost 𝑀
𝑚,𝑛for aligning two sequences
▪ Cost = #insertions × 𝐶INS + #deletions × 𝐶DEL + #substitutions × 𝐶𝑝,𝑞
15
▪ Subproblems
▪ SA(i, j): sequence alignment between prefix strings 𝑥1, … , 𝑥𝑖 and 𝑦1, … , 𝑦𝑗
▪ Goal: SA(m, n)
▪ Optimal substructure: suppose OPT is an optimal solution to SA(i, j), there are 3 cases:
▪ Case 1: 𝑥𝑖 and 𝑦𝑗 are aligned in OPT (match or substitution)
▪ OPT/{𝑥𝑖, , 𝑦𝑗} is an optimal solution of SA(i-1, j-1)
▪ Case 2: 𝑥𝑖 is aligned with a gap in OPT (deletion)
▪ OPT is an optimal solution of SA(i-1, j)
▪ Case 3: 𝑦𝑗 is aligned with a gap in OPT (insertion)
▪ OPT is an optimal solution of SA(i, j-1)
16
Sequence Alignment Problem Input: two sequences
Output: the minimal cost 𝑀𝑚,𝑛 for aligning two sequences
▪
Suppose OPT is an optimal solution to SA(i, j), there are 3 cases:
▪ Case 1: 𝑥𝑖 and 𝑦𝑗 are aligned in OPT (match or substitution)
▪ OPT/{𝑥𝑖, , 𝑦𝑗} is an optimal solution of SA(i-1, j-1)
▪ Case 2: 𝑥𝑖 is aligned with a gap in OPT (deletion)
▪ OPT is an optimal solution of SA(i-1, j)
▪ Case 3: 𝑦𝑗 is aligned with a gap in OPT (insertion)
▪ OPT is an optimal solution of SA(i, j-1)
▪
Recursively define the value
17
Sequence Alignment Problem Input: two sequences
Output: the minimal cost 𝑀𝑚,𝑛 for aligning two sequences
▪ Bottom-up method: solve smaller subproblems first
18
X\Y 0 1 2 3 4 5 … n
0 1 : m
Sequence Alignment Problem Input: two sequences
Output: the minimal cost 𝑀𝑚,𝑛 for aligning two sequences
▪ Bottom-up method: solve smaller subproblems first
19
X\Y 0 1 2 3 4 5 6 7 8 9 10 11 12
0 0 4 8 12 16 20 24 28 32 36 40 44 48
1 4 7 11 15 19 23 27 31 35 39 43 47 51
2 8 4 8 12 16 20 23 27 31 35 39 43 47
3 12 8 12 8 12 16 20 24 28 32 36 40 44
4 16 12 15 12 15 19 16 20 24 28 32 36 40 5 20 16 19 15 19 22 20 23 27 31 35 39 43 6 24 20 23 19 22 26 22 26 30 34 38 35 39
Sequence Alignment Problem Input: two sequences
Output: the minimal cost 𝑀𝑚,𝑛 for aligning two sequences
a e n i q a d i k j a z
b a n a n a
20
Seq-Align(X, Y, CDEL, CINS, Cp,q) for j = 0 to n
M[0][j] = j * CINS // |X|=0, cost=|Y|*penalty for i = 1 to m
M[i][0] = i * CDEL // |Y|=0, cost=|X|*penalty for i = 1 to m
for j = 1 to n
M[i][j] = min(M[i-1][j-1]+Cxi,yi, M[i-1][j]+CDEL, M[i][j-1]+CINS) return M[m][n]
▪ Bottom-up method: solve smaller subproblems first Sequence Alignment Problem
Input: two sequences
Output: the minimal cost 𝑀𝑚,𝑛 for aligning two sequences
21
▪ Bottom-up method: solve smaller subproblems first Sequence Alignment Problem
Input: two sequences
Output: the minimal cost 𝑀𝑚,𝑛 for aligning two sequences
X\Y 0 1 2 3 4 5 6 7 8 9 10 11 12
0 0 4 8 12 16 20 24 28 32 36 40 44 48
1 4 7 11 15 19 23 27 31 35 39 43 47 51
2 8 4 8 12 16 20 23 27 31 35 39 43 47
3 12 8 12 8 12 16 20 24 28 32 36 40 44
4 16 12 15 12 15 19 16 20 24 28 32 36 40 5 20 16 19 15 19 22 20 23 27 31 35 39 43 6 24 20 23 19 22 26 22 26 30 34 38 35 39
a e n i q a d i k j a z
b a n a n a
22
▪ Bottom-up method: solve smaller subproblems first Sequence Alignment Problem
Input: two sequences
Output: the minimal cost 𝑀𝑚,𝑛 for aligning two sequences
Find-Solution(M) if m = 0 or n = 0
return {}
v = min(M[m-1][n-1] + Cxm,yn, M[m-1][n] + CDEL, M[m][n-1] + CINS) if v = M[m-1][n] + CDEL // ↑: deletion
return Find-Solution(m-1, n)
if v = M[m][n-1] + CINS // ←: insertion return Find-Solution(m, n-1)
return {(m, n)} ∪ Find-Solution(m-1, n-1) // ↖: match/substitution
23
Find-Solution(M) if m = 0 or n = 0
return {}
v = min(M[m-1][n-1] + Cxm,yn, M[m-1][n] + CDEL, M[m][n-1] + CINS) if v = M[m-1][n] + CDEL // ↑: deletion
return Find-Solution(m-1, n)
if v = M[m][n-1] + CINS // ←: insertion return Find-Solution(m, n-1)
return {(m, n)} ∪ Find-Solution(m-1, n-1) // ↖: match/substitution Seq-Align(X, Y, CDEL, CINS, Cp,q)
for j = 0 to n
M[0][j] = j * CINS // |X|=0, cost=|Y|*penalty for i = 1 to m
M[i][0] = i * CDEL // |Y|=0, cost=|X|*penalty for i = 1 to m
for j = 1 to n
M[i][j] = min(M[i-1][j-1]+Cxi,yi, M[i-1][j]+CDEL, M[i][j-1]+CINS) return M[m][n]
▪
Space complexity
▪
If only keeping the most recent two rows: Space-Seq-Align(X, Y)
24
X\Y 0 1 2 3 … j … n
i - 1 i
The optimal value can be computed, but the solution cannot be reconstructed
X\Y 0 1 2 3 4 5 … n
0 1 : m
▪
Problem: find the min-cost alignment find the shortest path
25
Divide-and-Conquer +
Dynamic Programming
a
e p p l
p e
a
X\Y 0 1 2 3
0 0 4 8 12
1 4 7 11 15
2 8 4 8 12
3 12 8 12 8
4 16 12 15 12 5 20 16 19 15 a p e
p p l e a
→ distance = CINS
↓ distance = CDEL
↘ distance = Cu,v for edge (u, v) START
END
𝐹 2,3 = distance of the shortest path
▪
Each edge has a length/cost
▪
𝐹 𝑖, 𝑗 : length of the shortest path from 0,0 to 𝑖, 𝑗 (START 𝑖, 𝑗 )
▪
𝐵 𝑖, 𝑗 : length of the shortest path from 𝑖, 𝑗 to 𝑚, 𝑛 ( 𝑖, 𝑗 END)
▪
𝐹 𝑚, 𝑛 = 𝐵 0,0
26
i = 0
4 1 2 3
j = 0 1 2 3 4 5 6 7
5
𝐵 2,3 = distance of the shortest path
▪
Each edge has a length/cost
▪
𝐹 𝑖, 𝑗 : length of the shortest path from 0,0 to 𝑖, 𝑗 (START 𝑖, 𝑗 )
▪
𝐵 𝑖, 𝑗 : length of the shortest path from 𝑖, 𝑗 to 𝑚, 𝑛 ( 𝑖, 𝑗 END)
▪
Forward formulation
▪
Backward formulation
27
i = 0
4 1 2 3
j = 0 1 2 3 4 5 6 7
5
▪
Observation 1: the length of the shortest path from 0,0 to 𝑚, 𝑛 that passes through 𝑖, 𝑗 is 𝐹 𝑖, 𝑗 + 𝐵 𝑖, 𝑗
28
𝐹 𝑖, 𝑗 : length of the shortest path from 0,0 to 𝑖, 𝑗 𝐵 𝑖, 𝑗 : length of the shortest path from 𝑖, 𝑗 to 𝑚, 𝑛
i = 0
4 1 2 3
j = 0 1 2 3 4 5 6 7
5
𝐹 𝑖, 𝑗
𝐵 𝑖, 𝑗
optimal substructure
▪
Observation 2: for any 𝑣 in {0, … , 𝑛}, there exists a 𝑢 s.t. the shortest path between (0,0) and 𝑚, 𝑛 goes through (𝑢, 𝑣)
29
𝐹 𝑖, 𝑗 : length of the shortest path from 0,0 to 𝑖, 𝑗 𝐵 𝑖, 𝑗 : length of the shortest path from 𝑖, 𝑗 to 𝑚, 𝑛
i = 0
4 1 2 3
j = 0 1 2 3 4 5 6 7
5
the shortest path must go across a vertical cut
▪
Observation 1+2:
30
𝐹 𝑖, 𝑗 : length of the shortest path from 0,0 to 𝑖, 𝑗 𝐵 𝑖, 𝑗 : length of the shortest path from 𝑖, 𝑗 to 𝑚, 𝑛
i = 0
4 1 2 3
j = 0 1 2 3 4 5 6 7
5
i = 0
4 1 2 3
j = 0 1 2 3 4 5 6 7
5
▪
Goal: finds optimal solution
31
How to find the value of 𝑢∗?
▪
Idea: utilize sequence alignment algo.
▪ Call Space-Seq-Align(X,Y[1:v]) to find 𝐹 0, 𝑣 , 𝐹 1, 𝑣 , … , 𝐹 𝑚, 𝑣
▪ Call Back-Space-Seq-Align(X,Y[v+1:n]) to find 𝐵 0, 𝑣 , 𝐵 1, 𝑣 , … , 𝐵 𝑚, 𝑣
▪ Let 𝑢 be the index minimizing 𝐹 𝑢, 𝑣 + 𝐵 𝑢, 𝑣
▪
Goal: finds optimal solution – DC-Align(X, Y)
32
1. Divide
2. Conquer
3. Combine
▪ Divide the sequence of size n into 2 subsequences
▪ Find 𝑢 to minimize 𝐹 𝑢, 𝑣 + 𝐵 𝑢, 𝑣
▪ Recursive case (𝑛 > 1)
▪ prefix
= DC-Align(X[1:u], Y[1:v])
▪ suffix
= DC-Align(X[u+1:m], Y[v+1:n])
▪ Base case (𝑛 = 1)
▪ Return Seq-Align(X, Y)
▪ Return prefix + suffix
▪ 𝑇 𝑚, 𝑛 = time for running DC-Align(X, Y) with 𝑋 = 𝑚, 𝑌 = 𝑛
Space Complexity:
▪
Theorem
▪
Proof
▪ There exists positive constants 𝑎, 𝑏 s.t. all
▪ Use induction to prove
33
Inductive hypothesis
when
Practice to check the initial condition
▪
Given a graph 𝐺 = 𝑉, 𝐸 , each edge 𝑢, 𝑣 ∈ 𝐸 has an associated non- negative probability 𝑝 𝑢, 𝑣 of traversing the edge 𝑢, 𝑣 and producing the corresponding character. Find the most probable path with the
label 𝑠 = 𝜎
1, 𝜎
2, … , 𝜎
𝑛.
34
ㄨ ㄅ ㄒ ㄎ ㄕ
START
我 烏 為 問
END
爸 不
想 續 小
考 看 卡
書
試 上
白 鄉
Find the path from START to END with
highest prob
35
𝜎
1𝜎
2… … 𝜎
𝑛START END
produce 𝜎1
produce 𝜎𝑗
V: vocabulary size
Viterbi has been applied to many AI applications, e.g. speech recognition
36
▪
Input: 𝑛 job requests with start times 𝑠
𝑖, finish times 𝑓
𝑖▪
Output: the maximum number of compatible jobs
▪
The interval scheduling problem can be solved using an “early-finish-time- first” greedy algorithm in 𝑂(𝑛) time
37
“Greedy Algorithm”
Next topic!
time 1
2 3 4 5 6
job index
1 2 3 4 5 6 7 8 9
▪
Input: 𝑛 job requests with start times 𝑠
𝑖, finish times 𝑓
𝑖, and values 𝑣
𝑖▪
Output: the maximum total value obtainable from compatible jobs
time 38
1
3
3 4
3 1 1
2 3 4 5 6
job index
1 2 3 4 5 6 7 8 9
Assume that the requests are sorted in non-decreasing order (𝑓𝑖 ≤ 𝑓𝑗 when 𝑖 < 𝑗) 𝑝(𝑗) = largest index 𝑖 < 𝑗 s.t. jobs 𝑖 and 𝑗 are compatible
e.g. 𝑝 1 = 0, 𝑝 2 = 0, 𝑝 3 = 1, 𝑝 4 = 1, 𝑝 5 = 4, 𝑝 6 = 3
▪
Subproblems
▪ WIS(i): weighted interval scheduling for the first 𝑖 jobs
▪ Goal: WIS(n)
▪
Optimal substructure: suppose OPT is an optimal solution to WIS(i), there are 2 cases:
▪ Case 1: job 𝑖 in OPT
▪ OPT\{𝑖} is an optimal solution of WIS(p(i))
▪ Case 2: job 𝑖 not in OPT
▪ OPT is an optimal solution of WIS(i-1)
39
Weighted Interval Scheduling Problem
Input: 𝑛 jobs with 𝑠𝑖, 𝑓𝑖, 𝑣𝑖 , 𝑝(𝑗) = largest index 𝑖 < 𝑗 s.t. jobs 𝑖 and 𝑗 are compatible Output: the maximum total value obtainable from compatible
time 1
3
3 4
3 1 1
2 3 4 5 6
job index
2
1 3 4 5 6 7 8 9
4
1
▪
Optimal substructure: suppose OPT is an optimal solution to WIS(i), there are 2 cases:
▪ Case 1: job 𝑖 in OPT
▪ OPT\{𝑖} is an optimal solution of WIS(p(i))
▪ Case 2: job 𝑖 not in OPT
▪ OPT is an optimal solution of WIS(i-1)
▪
Recursively define the value
40
Weighted Interval Scheduling Problem
Input: 𝑛 jobs with 𝑠𝑖, 𝑓𝑖, 𝑣𝑖 , 𝑝(𝑗) = largest index 𝑖 < 𝑗 s.t. jobs 𝑖 and 𝑗 are compatible Output: the maximum total value obtainable from compatible
▪ Bottom-up method: solve smaller subproblems first
41
i 0 1 2 3 4 5 … n
M[i]
WIS(n, s, f, v, p) M[0] = 0
for i = 1 to n
M[i] = max(v[i] + M[p[i]], M[i - 1]) return M[n]
Weighted Interval Scheduling Problem
Input: 𝑛 jobs with 𝑠𝑖, 𝑓𝑖, 𝑣𝑖 , 𝑝(𝑗) = largest index 𝑖 < 𝑗 s.t. jobs 𝑖 and 𝑗 are compatible Output: the maximum total value obtainable from compatible
42
▪ Bottom-up method: solve smaller subproblems first Weighted Interval Scheduling Problem
Input: 𝑛 jobs with 𝑠𝑖, 𝑓𝑖, 𝑣𝑖 , 𝑝(𝑗) = largest index 𝑖 < 𝑗 s.t. jobs 𝑖 and 𝑗 are compatible Output: the maximum total value obtainable from compatible
i 0 1 2 3 4 5 6
M[i] 0 1 3 4 5 6 7
time 1
3
3 4
3 1 1
2 3 4 5 6 job index
1 2 3 4 5 6 7 8 9
43
WIS(n, s, f, v, p) M[0] = 0
for i = 1 to n
M[i] = max(v[i] + M[p[i]], M[i - 1]) return M[n]
Weighted Interval Scheduling Problem
Input: 𝑛 jobs with 𝑠𝑖, 𝑓𝑖, 𝑣𝑖 , 𝑝(𝑗) = largest index 𝑖 < 𝑗 s.t. jobs 𝑖 and 𝑗 are compatible Output: the maximum total value obtainable from compatible
Find-Solution(M, n) if n = 0
return {}
if v[n] + M[p[n]] > M[n-1] // case 1 return {n} ∪ Find-Solution(p[n]) return Find-Solution(n-1) // case 2
Textbook Exercise 16.2-2
44
▪
Input: 𝑛 items where 𝑖-th item has value 𝑣
𝑖and weighs 𝑤
𝑖(𝑣
𝑖and 𝑤
𝑖are positive integers)
▪
Output: the maximum value for the knapsack with capacity of 𝑊
▪
Variants of knapsack problem
▪ 0-1 Knapsack Problem: 每項物品只能拿一個
▪ Unbounded Knapsack Problem: 每項物品可以拿多個
▪ Multidimensional Knapsack Problem: 背包空間有限
▪ Multiple-Choice Knapsack Problem: 每一類物品最多拿一個
▪ Fractional Knapsack Problem: 物品可以只拿部分
45
▪
Input: 𝑛 items where 𝑖-th item has value 𝑣
𝑖and weighs 𝑤
𝑖(𝑣
𝑖and 𝑤
𝑖are positive integers)
▪
Output: the maximum value for the knapsack with capacity of 𝑊
▪
Variants of knapsack problem
▪ 0-1 Knapsack Problem: 每項物品只能拿一個
▪ Unbounded Knapsack Problem: 每項物品可以拿多個
▪ Multidimensional Knapsack Problem: 背包空間有限
▪ Multiple-Choice Knapsack Problem: 每一類物品最多拿一個
▪ Fractional Knapsack Problem: 物品可以只拿部分
46
▪
Subproblems
▪ ZO-KP(i, w): 0-1 knapsack problem within 𝑤 capacity for the first 𝑖 items
▪ Goal: ZO-KP(n, W)
▪
Optimal substructure: suppose OPT is an optimal solution to ZO-KP(i, w), there are 2 cases:
▪ Case 1: item 𝑖 in OPT
▪ OPT\{𝑖} is an optimal solution of ZO-KP(i - 1, w - wi)
▪ Case 2: item 𝑖 not in OPT
▪ OPT is an optimal solution of ZO-KP(i - 1, w)
47
0-1 Knapsack Problem
Input: 𝑛 items where 𝑖-th item has value 𝑣𝑖 and weighs 𝑤𝑖
Output: the max value within 𝑊 capacity, where each item is chosen at most once ZO-KP(i) ZO-KP(i, w)
consider the available capacity
▪
Optimal substructure: suppose OPT is an optimal solution to ZO-KP(i, w), there are 2 cases:
▪ Case 1: item 𝑖 in OPT
▪ OPT\{𝑖} is an optimal solution of ZO-KP(i - 1, w - wi)
▪ Case 2: item 𝑖 not in OPT
▪ OPT is an optimal solution of ZO-KP(i - 1, w)
▪
Recursively define the value
48
0-1 Knapsack Problem
Input: 𝑛 items where 𝑖-th item has value 𝑣𝑖 and weighs 𝑤𝑖
Output: the max value within 𝑊 capacity, where each item is chosen at most once
▪ Bottom-up method: solve smaller subproblems first
49
i\w 0 1 2 3 … w … W
0 1 2 i n
0-1 Knapsack Problem
Input: 𝑛 items where 𝑖-th item has value 𝑣𝑖 and weighs 𝑤𝑖
Output: the max value within 𝑊 capacity, where each item is chosen at most once
▪ Bottom-up method: solve smaller subproblems first
50
i\w 0 1 2 3 4 5
0 0 0 0 0 0 0
1 0 4 4 4 4 4
2 0 4 9 13 13 13
3 0 4 9 13 20 24
0-1 Knapsack Problem
Input: 𝑛 items where 𝑖-th item has value 𝑣𝑖 and weighs 𝑤𝑖
Output: the max value within 𝑊 capacity, where each item is chosen at most once
i wi vi
1 1 4
2 2 9
3 4 20
𝑊 = 5
▪ Bottom-up method: solve smaller subproblems first
51
ZO-KP(n, v, W) for w = 0 to W
M[0, w] = 0 for i = 1 to n
for w = 0 to W if(wi > w)
M[i, w] = M[i-1, w]
else
M[i, w] = max(vi + M[i-1, w-wi], M[i-1, w]) return M[n, W]
0-1 Knapsack Problem
Input: 𝑛 items where 𝑖-th item has value 𝑣𝑖 and weighs 𝑤𝑖
Output: the max value within 𝑊 capacity, where each item is chosen at most once
52
ZO-KP(n, v, W) for w = 0 to W
M[0, w] = 0 for i = 1 to n
for w = 0 to W if(wi > w)
M[i, w] = M[i-1, w]
else
M[i, w] = max(vi + M[i-1, w-wi], M[i-1, w]) return M[n, W]
Find-Solution(M, n, W) S = {}
w = W
for i = n to 1
if M[i, w] > M[i – 1, w] // case 1 w = w – wi
S = S ∪ {i}
return S
▪
Polynomial: polynomial in the length of the input (#bits for the input)
▪
Pseudo-polynomial: polynomial in the numeric value
▪
The time complexity of 0-1 knapsack problem is Θ 𝑛𝑊
▪ 𝑛: number of objects
▪ 𝑊: knapsack’s capacity (non-negative integer)
▪ polynomial in the numeric value
= pseudo-polynomial in input size
= exponential in the length of the input
▪
Note: the size of the representation of 𝑊 is log
2𝑊
53
= 2𝑚 = 𝑚
▪
Input: 𝑛 items where 𝑖-th item has value 𝑣
𝑖and weighs 𝑤
𝑖(𝑣
𝑖and 𝑤
𝑖are positive integers)
▪
Output: the maximum value for the knapsack with capacity of 𝑊
▪
Variants of knapsack problem
▪ 0-1 Knapsack Problem: 每項物品只能拿一個
▪ Unbounded Knapsack Problem: 每項物品可以拿多個
▪ Multidimensional Knapsack Problem: 背包空間有限
▪ Multiple-Choice Knapsack Problem: 每一類物品最多拿一個
▪ Fractional Knapsack Problem: 物品可以只拿部分
54
▪
Subproblems
▪ U-KP(i, w): unbounded knapsack problem with 𝑤 capacity for the first 𝑖 items
▪ Goal: U-KP(n, W)
55
Unbounded Knapsack Problem
Input: 𝑛 items where 𝑖-th item has value 𝑣𝑖 and weighs 𝑤𝑖, each has unlimited supplies Output: the max value within 𝑊 capacity
0-1 Knapsack Problem Unbounded Knapsack Problem
each item can be chosen at most once each item can be chosen multiple times a sequence of binary choices: whether
to choose item 𝑖
a sequence of 𝑖 choices: which one (from 1 to 𝑖) to choose
Time complexity = Θ 𝑛𝑊 Time complexity = Θ 𝑛2𝑊 Can we do better?
▪ Subproblems
▪ U-KP(w): unbounded knapsack problem with 𝑤 capacity
▪ Goal: U-KP(W)
▪ Optimal substructure: suppose OPT is an optimal solution to U-KP(w), there are 𝑛 cases:
▪ Case 1: item 1 in OPT
▪ Removing an item 1 from OPT is an optimal solution of U-KP(w – w1)
▪ Case 2: item 2 in OPT
▪ Removing an item 2 from OPT is an optimal solution of U-KP(w – w2) :
▪ Case 𝑛: item 𝑛 in OPT
▪ Removing an item 𝑛 from OPT is an optimal solution of U-KP(w - wn)
56
Unbounded Knapsack Problem
Input: 𝑛 items where 𝑖-th item has value 𝑣𝑖 and weighs 𝑤𝑖, each has unlimited supplies Output: the max value within 𝑊 capacity
▪
Optimal substructure: suppose OPT is an optimal solution to U-KP(w), there are 𝑛 cases:
▪ Case 𝑖: item 𝑖 in OPT
▪ Removing an item i from OPT is an optimal solution of U-KP(w – w1)
▪
Recursively define the value
57
Unbounded Knapsack Problem
Input: 𝑛 items where 𝑖-th item has value 𝑣𝑖 and weighs 𝑤𝑖, each has unlimited supplies Output: the max value within 𝑊 capacity
只考慮背包還裝的下的情形
▪ Bottom-up method: solve smaller subproblems first
58
Unbounded Knapsack Problem
Input: 𝑛 items where 𝑖-th item has value 𝑣𝑖 and weighs 𝑤𝑖, each has unlimited supplies Output: the max value within 𝑊 capacity
w 0 1 2 3 4 5 … W
M[w]
i wi vi
1 1 4
2 2 9
3 4 20
𝑊 = 5
▪ Bottom-up method: solve smaller subproblems first
59
Unbounded Knapsack Problem
Input: 𝑛 items where 𝑖-th item has value 𝑣𝑖 and weighs 𝑤𝑖, each has unlimited supplies Output: the max value within 𝑊 capacity
w 0 1 2 3 4 5
M[w] 0
i wi vi
1 1 4
2 2 9
3 4 17
𝑊 = 5
4 9 13 18 22
▪ Bottom-up method: solve smaller subproblems first
60
U-KP(v, W)
for w = 0 to W M[w] = 0 for w = 0 to W
for i = 1 to n if(wi <= w)
tmp = vi + M[w - wi] M[w] = max(M[w], tmp) return M[W]
Unbounded Knapsack Problem
Input: 𝑛 items where 𝑖-th item has value 𝑣𝑖 and weighs 𝑤𝑖, each has unlimited supplies Output: the max value within 𝑊 capacity
61
U-KP(v, W)
for w = 0 to W M[w] = 0
for w = 0 to W for i = 1 to n
if(wi <= w)
tmp = vi + M[w - wi] M[w] = max(M[w], tmp) return M[W]
Find-Solution(M, n, W) for i = 1 to n
C[i] = 0 // C[i] = # of item i in solution w = W
for i = i to n while w > 0
if(wi <= w && M[w] == (vi + M[w - wi])) w = w - wi
C[i] += 1 return C
▪
Input: 𝑛 items where 𝑖-th item has value 𝑣
𝑖and weighs 𝑤
𝑖(𝑣
𝑖and 𝑤
𝑖are positive integers)
▪
Output: the maximum value for the knapsack with capacity of 𝑊
▪
Variants of knapsack problem
▪ 0-1 Knapsack Problem: 每項物品只能拿一個
▪ Unbounded Knapsack Problem: 每項物品可以拿多個
▪ Multidimensional Knapsack Problem: 背包空間有限
▪ Multiple-Choice Knapsack Problem: 每一類物品最多拿一個
▪ Fractional Knapsack Problem: 物品可以只拿部分
62
▪
Subproblems
▪ M-KP(i, w, d): multidimensional knapsack problem with 𝑤 capacity and 𝑑 size for the first 𝑖 items
▪ Goal: M-KP(n, W, D)
▪
Optimal substructure: suppose OPT is an optimal solution to M-KP(i, w, d) , there are 2 cases:
▪ Case 1: item 𝑖 in OPT
▪ OPT\{𝑖} is an optimal solution of M-KP(i - 1, w - wi, d – di)
▪ Case 2: item 𝑖 not in OPT
▪ OPT is an optimal solution of M-KP(i - 1, w, d)
63
Multidimensional Knapsack Problem
Input: 𝑛 items where 𝑖-th item has value 𝑣𝑖, weighs 𝑤𝑖, and size 𝑑𝑖
Output: the max value within 𝑊 capacity and with the size of 𝑫, where each item is chosen at most once
▪
Optimal substructure: suppose OPT is an optimal solution to M-KP(i, w, d) , there are 2 cases:
▪ Case 1: item 𝑖 in OPT
▪ OPT\{𝑖} is an optimal solution of M-KP(i - 1, w - wi, d – di)
▪ Case 2: item 𝑖 not in OPT
▪ OPT is an optimal solution of M-KP(i - 1, w, d)
▪
Recursively define the value
64
Multidimensional Knapsack Problem
Input: 𝑛 items where 𝑖-th item has value 𝑣𝑖, weighs 𝑤𝑖, and size 𝑑𝑖
Output: the max value within 𝑊 capacity and with the size of 𝑫, where each item is chosen at most once
▪ Step 3: Compute Value of an OPT Solution
▪ Step 4: Construct an OPT Solution by Backtracking
▪ What is the time complexity?
65
Multidimensional Knapsack Problem
Input: 𝑛 items where 𝑖-th item has value 𝑣𝑖, weighs 𝑤𝑖, and size 𝑑𝑖
Output: the max value within 𝑊 capacity and with the size of 𝑫, where each item is chosen at most once
▪
Input: 𝑛 items where 𝑖-th item has value 𝑣
𝑖and weighs 𝑤
𝑖(𝑣
𝑖and 𝑤
𝑖are positive integers)
▪
Output: the maximum value for the knapsack with capacity of 𝑊
▪
Variants of knapsack problem
▪ 0-1 Knapsack Problem: 每項物品只能拿一個
▪ Unbounded Knapsack Problem: 每項物品可以拿多個
▪ Multidimensional Knapsack Problem: 背包空間有限
▪ Multiple-Choice Knapsack Problem: 每一類物品最多拿一個
▪ Fractional Knapsack Problem: 物品可以只拿部分
66
▪
Input: 𝑛 items
▪ 𝑣𝑖,𝑗: value of 𝑗-th item in the group 𝑖
▪ 𝑤𝑖,𝑗: weight of 𝑗-th item in the group 𝑖
▪ 𝑛𝑖: number of items in group 𝑖
▪ 𝑛: total number of items (σ 𝑛𝑖)
▪ 𝐺: total number of groups
▪
Output: the maximum value for the knapsack with capacity of 𝑊, where the item from each group can be selected at most once
67
group 1 group 2 group 3
▪
Subproblems
▪ MC-KP(w): 𝑤 capacity
▪ MC-KP(i, w): 𝑤 capacity for the first 𝑖 groups
▪ MC-KP(i, j, w): 𝑤 capacity for the first 𝑗 items from first 𝑖 groups
68
Multiple-Choice Knapsack Problem
Input: 𝑛 items with value 𝑣𝑖,𝑗 and weighs 𝑤𝑖,𝑗 (𝑛𝑖: #items in group 𝑖, 𝐺: #groups) Output: the max value within 𝑊 capacity, where each group is chosen at most once
Which one is more suitable for this problem?
the constraint is for groups
▪
Subproblems
▪ MC-KP(i, w): multi-choice knapsack problem with 𝑤 capacity for the first 𝑖 groups
▪ Goal: MC-KP(G, W)
▪
Optimal substructure: suppose OPT is an optimal solution to MC-KP(i, w), for the group 𝑖, there are 𝑛
𝑖+ 1 cases:
▪ Case 1: no item from 𝑖-th group in OPT
▪ OPT is an optimal solution of MC-KP(i - 1, w)
:
▪ Case 𝑗 + 1: 𝑗-th item from 𝑖-th group (itemi,j) in OPT
▪ OPT\itemi,j is an optimal solution of MC-KP(i - 1, w – wi,j)
69
Multiple-Choice Knapsack Problem
Input: 𝑛 items with value 𝑣𝑖,𝑗 and weighs 𝑤𝑖,𝑗 (𝑛𝑖: #items in group 𝑖, 𝐺: #groups) Output: the max value within 𝑊 capacity, where each group is chosen at most once
▪
Optimal substructure: suppose OPT is an optimal solution to MC-KP(i, w), for the group 𝑖, there are 𝑛
𝑖+ 1 cases:
▪ Case 1: no item from 𝑖-th group in OPT
▪ OPT is an optimal solution of MC-KP(i - 1, w)
▪ Case 𝑗 + 1: 𝑗-th item from 𝑖-th group (itemi,j) in OPT
▪ OPT\itemi,j is an optimal solution of MC-KP(i - 1, w – wi,j)
▪
Recursively define the value
70
Multiple-Choice Knapsack Problem
Input: 𝑛 items with value 𝑣𝑖,𝑗 and weighs 𝑤𝑖,𝑗 (𝑛𝑖: #items in group 𝑖, 𝐺: #groups) Output: the max value within 𝑊 capacity, where each group is chosen at most once
𝑛𝑖 + 1
▪ Bottom-up method: solve smaller subproblems first
71
i\w 0 1 2 3 … w … W
0 1 2 i n
Multiple-Choice Knapsack Problem
Input: 𝑛 items with value 𝑣𝑖,𝑗 and weighs 𝑤𝑖,𝑗 (𝑛𝑖: #items in group 𝑖, 𝐺: #groups) Output: the max value within 𝑊 capacity, where each group is chosen at most once
▪ Bottom-up method: solve smaller subproblems first
72
MC-KP(n, v, W) for w = 0 to W
M[0, w] = 0
for i = 1 to G // consider groups 1 to i for w = 0 to W // consider capacity = w
M[i, w] = M[i - 1, w]
for j = 1 to ni // check j-th item in group i if(vi,j + M[i - 1, w - wi,j] > M[i, w])
M[i, w] = vi,j + M[i - 1, w - wi,j] return M[G, W]
Multiple-Choice Knapsack Problem
Input: 𝑛 items with value 𝑣𝑖,𝑗 and weighs 𝑤𝑖,𝑗 (𝑛𝑖: #items in group 𝑖, 𝐺: #groups) Output: the max value within 𝑊 capacity, where each group is chosen at most once
73
MC-KP(n, v, W) for w = 0 to W
M[0, w] = 0
for i = 1 to G // consider groups 1 to i for w = 0 to W // consider capacity = w
M[i, w] = M[i - 1, w]
for j = 1 to ni // check items in group i if(vi,j + M[i - 1, w - wi,j] > M[i, w])
M[i, w] = vi,j + M[i - 1, w - wi,j] B[i, w] = j
return M[G, W], B[G, W]
Practice to write the pseudo code for Find-Solution()
▪
Input: 𝑛 items where 𝑖-th item has value 𝑣
𝑖and weighs 𝑤
𝑖(𝑣
𝑖and 𝑤
𝑖are positive integers)
▪
Output: the maximum value for the knapsack with capacity of 𝑊
▪
Variants of knapsack problem
▪ 0-1 Knapsack Problem: 每項物品只能拿一個
▪ Unbounded Knapsack Problem: 每項物品可以拿多個
▪ Multidimensional Knapsack Problem: 背包空間有限
▪ Multiple-Choice Knapsack Problem: 每一類物品最多拿一個
▪ Fractional Knapsack Problem: 物品可以只拿部分
74
▪
Input: 𝑛 items where 𝑖-th item has value 𝑣
𝑖and weighs 𝑤
𝑖(𝑣
𝑖and 𝑤
𝑖are positive integers)
▪
Output: the maximum value for the knapsack with capacity of 𝑊, where we can take any fraction of items
▪
Dynamic programming algorithm should work
▪
Choose maximal
𝑣𝑖𝑤𝑖
(類似CP值) first
75
“Greedy Algorithm”
Next topic!
Can we do better?
▪
“Dynamic Programming”: solve many subproblems in polynomial time for which a naïve approach would take exponential time
▪
When to use DP
▪ Whether subproblem solutions can combine into the original solution
▪ When subproblems are overlapping
▪ Whether the problem has optimal substructure
▪ Common for optimization problem
▪
Two ways to avoid recomputation
▪ Top-down with memoization
▪ Bottom-up method
▪
Complexity analysis
▪ Space for tabular filling
▪ Size of the subproblem graph 76
Course Website: http://ada17.csie.org Email: ada-ta@csie.ntu.edu.tw
77
Important announcement will be sent to @ntu.edu.tw mailbox
& post to the course website