Slides credited from Hsueh-I Lu, Hsu-Chun Hsiao, & Michael Tsai

(1)

(2)

▪

Mini-HW 6 Released

▪ Due on 11/09 (Thu) 17:20

▪

Homework 2

▪ Due on 11/09 (Thur) 17:20

▪

Midterm

▪ Time: 11/16 (Thur) 14:20-17:20

▪ Format: close book

▪ Location: R103 (please check the assigned seat before entering the room)

2

Frequently check the website for the updated information!

(3)

3

(4)

▪ Greedy Algorithms

▪ Greedy #1: Activity-Selection / Interval Scheduling

▪ Greedy #2: Coin Changing

▪ Greedy #3: Fractional Knapsack Problem

▪ Greedy #4: Breakpoint Selection

▪ Greedy #5: Huffman Codes

▪ Greedy #6: Task-Scheduling

▪ Greedy #7: Scheduling to Minimize Lateness

4

(5)

▪

Do not focus on “specific algorithms”

▪

But “some strategies” to “design” algorithms

▪

First Skill: Divide-and-Conquer (各個擊破)

▪

Second Skill: Dynamic Programming (動態規劃)

▪

Third Skill: Greedy (貪婪法則)

5

(6)

Textbook Chapter 16 – Greedy Algorithms

Textbook Chapter 16.2 – Elements of the greedy strategy

6

(7)

▪ always makes the choice that looks best at the moment

▪ makes a locally optimal choice in the hope that this choice will lead to a globally optimal solution

▪

not always yield optimal solution; may end up at local optimal

7

Greedy: move towards max gradient and hope it is global maximum local maximal

global maximal

local maximal

(8)

▪

Dynamic Programming

▪ has optimal substructure

▪ make an informed choice after getting optimal solutions to subproblems

▪ dependent or overlapping subproblems

▪

Greedy Algorithms

▪ has optimal substructure

▪ make a greedy choice before solving the subproblem

▪ no overlapping subproblems

✓Each round selects only one subproblem

✓The subproblem size decreases

8

Optimal Solution

Possible Case 1 Possible

Case 2

Possible Case k

max /min

Subproblem Solution Subproblem

Solution

Subproblem Solution

+ +

+

=

Optimal Solution

Greedy Choice

Subproblem Solution

= +

(9)

1. Cast the optimization problem as one in which we make a choice and remain one subproblem to solve

2. Demonstrate the optimal substructure

✓

Combining an optimal solution to the subproblem via greedy can arrive an optimal solution to the original problem

3. Prove that there is always an optimal solution to the original problem that makes the greedy choice

9

(10)

To yield an optimal solution, the problem should exhibit

1. Optimal Substructure : an optimal solution to the problem contains within it optimal solutions to subproblems

2. Greedy-Choice Property : making locally optimal (greedy) choices leads to a globally optimal solution

10

(11)

▪

Optimal Substructure : an optimal solution to the problem contains within it optimal solutions to subproblems

▪

Greedy-Choice Property : making locally optimal (greedy) choices leads to a globally optimal solution

▪ Show that it exists an optimal solution that “contains” the greedy choice using exchange argument

▪ For any optimal solution OPT, the greedy choice 𝑔 has two cases

▪ 𝑔 is in OPT: done

▪ 𝑔 not in OPT: modify OPT into OPT’ s.t. OPT’ contains 𝑔 and is at least as good as OPT

11

OPT OPT’

𝑔

✓ If OPT’ is better than OPT, the property is proved by contradiction

✓ If OPT’ is as good as OPT, then we showed that there exists an optimal solution containing 𝑔 by construction

(12)

Textbook Chapter 16.1 – An activity-selection problem

12

(13)

▪

Input: 𝑛 activities with start times 𝑠

_𝑖

and finish times 𝑓

_𝑖

(the activities are sorted in monotonically increasing order of finish time 𝑓

₁

≤ 𝑓

₂

≤ ⋯ ≤ 𝑓

_𝑛

)

▪

Output: the maximum number of compatible activities

▪

Without loss of generality: 𝑠

₁

< 𝑠

₂

< ⋯ < 𝑠

_𝑛

and 𝑓

₁

< 𝑓

₂

< ⋯ < 𝑓

_𝑛

▪ 大的包小的則不考慮大的  用小的取代大的一定不會變差

time 13

1 2 3 4 5 6

activity index

1 2 3 4 5 6 7 8 9

(14)

▪

Subproblems

▪ WIS(i): weighted interval scheduling for the first 𝑖 jobs

▪ Goal: WIS(n)

▪ Dynamic programming algorithm

14

i 0 1 2 3 4 5 … n

M[i]

Weighted Interval Scheduling Problem

Input: 𝑛 jobs with 𝑠_𝑖, 𝑓_𝑖, 𝑣_𝑖 , 𝑝(𝑗) = largest index 𝑖 < 𝑗 s.t. jobs 𝑖 and 𝑗 are compatible Output: the maximum total value obtainable from compatible

Set 𝑣_𝑖 = 1 for all 𝑖 to formulate it into the activity-selection problem

(15)

▪ Dynamic programming

▪ Optimal substructure is already proved

▪ Greedy algorithm

15

Activity-Selection Problem

Input: 𝑛 activities with 𝑠_𝑖, 𝑓_𝑖 , 𝑝(𝑗) = largest index 𝑖 < 𝑗 s.t. 𝑖 and 𝑗 are compatible Output: the maximum number of activities

select the 𝑖-th activity

Why does the 𝑖-th activity must appear in an OPT?

(16)

▪

Goal:

▪

Proof

▪ Assume there is an OPT solution for the first 𝑖 − 1 activities (𝑀_𝑖−1)

▪ 𝐴_𝑗 is the last activity in the OPT solution 

▪ Replacing 𝐴_𝑗 with 𝐴_𝑖 does not make the OPT worse

time 16

1 2 : i i - 1

:

activity index

1 2 3 4 5 6 7 8 9

(17)

17

Act-Select(n, s, f, v, p) M[0] = 0

for i = 1 to n if p[i] >= 0

M[i] = 1 + M[p[i]]

return M[n]

Find-Solution(M, n) if n = 0

return {}

return {n} ∪ Find-Solution(p[n])

Activity-Selection Problem

Input: 𝑛 activities with 𝑠_𝑖, 𝑓_𝑖 , 𝑝(𝑗) = largest index 𝑖 < 𝑗 s.t. 𝑖 and 𝑗 are compatible Output: the maximum number of activities

Select the last compatible one (←) = Select the first compatible one (→)

(18)

Textbook Exercise 16.1

18

(19)

▪

Input: 𝑛 dollars and unlimited coins with values 𝑣

_𝑖

(1, 5, 10, 50)

▪

Output: the minimum number of coins with the total value 𝑛

▪

Cashier’s algorithm: at each iteration, add the coin with the largest value no more than the current total

19

Does this algorithm return the OPT?

(20)

▪

Subproblems

▪ C(i): minimal number of coins for the total value 𝑖

▪ Goal: C(n)

20

Coin Changing Problem

Input: 𝑛 dollars and unlimited coins with values 𝑣_𝑖 (1, 5, 10, 50) Output: the minimum number of coins with the total value 𝑛

(21)

▪

Suppose OPT is an optimal solution to C(i), there are 4 cases:

▪ Case 1: coin 1 in OPT

▪ OPT\coin1 is an optimal solution of C(i – v₁)

▪ OPT\coin2 is an optimal solution of C(i – v₂)

▪ OPT\coin3 is an optimal solution of C(i – v₃)

▪ OPT\coin4 is an optimal solution of C(i – v₄)

21

(22)

▪

Greedy choice: select the coin with the largest value no more than the current total

▪

Proof via contradiction (use the case 10 ≤ 𝑖 < 50 for demo)

▪ Assume that there is no OPT including this greedy choice (choose 10)

 all OPT use 1, 5, 50 to pay 𝑖

▪ 50 cannot be used

▪ #coins with value 5 < 2  otherwise we can use a 10 to have a better output

▪ #coins with value 1 < 5  otherwise we can use a 5 to have a better output

▪ We cannot pay 𝑖 with the constraints (at most 5 + 4 = 9)

22

(23)

Textbook Exercise 16.2-2

23

(24)

▪

Input: 𝑛 items where 𝑖-th item has value 𝑣

_𝑖

and weighs 𝑤

_𝑖

(𝑣

_𝑖

and 𝑤

_𝑖

are positive integers)

▪

Output: the maximum value for the knapsack with capacity of 𝑊

▪

Variants of knapsack problem

▪ 0-1 Knapsack Problem: 每項物品只能拿一個

▪ Unbounded Knapsack Problem: 每項物品可以拿多個

▪ Multidimensional Knapsack Problem: 背包空間有限

▪ Multiple-Choice Knapsack Problem: 每一類物品最多拿一個

▪ Fractional Knapsack Problem: 物品可以只拿部分

24

(25)

▪

Input: 𝑛 items where 𝑖-th item has value 𝑣

_𝑖

and weighs 𝑤

_𝑖

(𝑣

_𝑖

and 𝑤

_𝑖

are positive integers)

▪

Output: the maximum value for the knapsack with capacity of 𝑊

▪

Variants of knapsack problem

▪ 0-1 Knapsack Problem: 每項物品只能拿一個

▪ Unbounded Knapsack Problem: 每項物品可以拿多個

▪ Multidimensional Knapsack Problem: 背包空間有限

▪ Multiple-Choice Knapsack Problem: 每一類物品最多拿一個

▪ Fractional Knapsack Problem: 物品可以只拿部分

25

(26)

▪

Input: 𝑛 items where 𝑖-th item has value 𝑣

_𝑖

and weighs 𝑤

_𝑖

(𝑣

_𝑖

and 𝑤

_𝑖

are positive integers)

▪

Output: the maximum value for the knapsack with capacity of 𝑊, where we can take any fraction of items

▪

Greedy algorithm: at each iteration, choose the item with the highest

^𝑣^𝑖

𝑤_𝑖

and continue when 𝑊 − 𝑤

_𝑖

> 0

26

(27)

▪

Subproblems

▪ F-KP(i, w): fractional knapsack problem within 𝑤 capacity for the first 𝑖 items

▪ Goal: F-KP(n, W)

27

Fractional Knapsack Problem

Input: 𝑛 items where 𝑖-th item has value 𝑣_𝑖 and weighs 𝑤_𝑖

Output: the max value within 𝑊 capacity, where we can take any fraction of items

(28)

▪

Suppose OPT is an optimal solution to F-KP(i, w), there are 2 cases:

▪ Case 1: full/partial item 𝑖 in OPT

▪ Remove 𝑤′ of item 𝑖 from OPT is an optimal solution of F-KP(i - 1, w – w^’)

▪ Case 2: item 𝑖 not in OPT

▪ OPT is an optimal solution of F-KP(i - 1, w)

28

(29)

▪

Greedy choice: select the item with the highest

^𝑣^𝑖

𝑤_𝑖

▪

Proof via contradiction (𝑗 = argmax

𝑖

𝑣_𝑖 𝑤_𝑖

)

▪ Assume that there is no OPT including this greedy choice

▪ If 𝑊 ≤ 𝑤_𝑗, we can replace all items in OPT with item 𝑗

▪ If 𝑊 > 𝑤_𝑗, we can replace any item weighting 𝑤_𝑗 in OPT with item 𝑗

▪ The total value must be equal or higher, because item 𝑗 has the highest ^𝑣^𝑖

𝑤_𝑖

29

Do other knapsack problems have this property?

(30)

30

(31)

▪

Input: a planned route with 𝑛 + 1 gas stations 𝑏

₀

, … , 𝑏

_𝑛

; the car can go at most 𝐶 after refueling at a breakpoint

▪

Output: a refueling schedule (𝑏

₀

𝑏

_𝑛

) that minimizes the number of stops

▪

Greedy algorithm: go as far as you can before refueling

31

1 2 3 4 5

Ideally: stop when out of gas

Actually: may not be able to find the gas station when out of gas

1 2 3 4 5 6

(32)

▪

Subproblems

▪ B(i): breakpoint selection problem from 𝑏_𝑖 to 𝑏_𝑛

▪ Goal: B(0)

32

Breakpoint Selection Problem

Input: 𝑛 + 1 breakpoints 𝑏₀, … , 𝑏_𝑛; gas storage is 𝐶

Output: a refueling schedule (𝑏₀𝑏_𝑛) that minimizes the number of stops

(33)

▪

Suppose OPT is an optimal solution to B(i) where 𝑗 is the largest index satisfying 𝑏

_𝑗

− 𝑏

_𝑖

≤ 𝐶, there are 𝑗 − 𝑖 cases

▪ Case 1: stop at 𝑏_𝑖+1

▪ OPT+{b_i+1} is an optimal solution of B(i + 1)

▪ Case 2: stop at 𝑏_𝑖+2

▪ OPT+{b_i+2} is an optimal solution of B(i + 2) :

▪ Case 𝑗 − 𝑖: stop at 𝑏_𝑗

▪ OPT+{b_j} is an optimal solution of B(j)

33

(34)

▪

Greedy choice: go as far as you can before refueling (select 𝑏

_𝑗

)

▪

Proof via contradiction

▪ Assume that there is no OPT including this greedy choice (after 𝑏_𝑖 then stop at 𝑏_𝑘, 𝑘 ≠ 𝑗)

▪ If 𝑘 > 𝑗, we cannot stop at 𝑏_𝑘 due to out of gas

▪ If 𝑘 < 𝑗, we can replace the stop at 𝑏_𝑘 with the stop at 𝑏_𝑗

▪ The total value must be equal or higher, because we refuel later (𝑏_𝑗 > 𝑏_𝑘)

34

(35)

35

BP-Select(C, b)

Sort(b) s.t. b[0] < b[1] < … < b[n]

p = 0 S = {0}

for i = 1 to n - 1

if b[i + 1] – b[p] > C if i == p

return “no solution”

A = A ∪ {i}

p = i return A

(36)

Textbook Chapter 16.3 – Huffman codes

36

(37)

▪

Code (編碼) is a system of rules to convert information—such as a letter, word, sound, image, or gesture—into another, sometimes shortened or secret, form or representation for communication through a channel or storage in a medium.

37

input message

decoded message

Encoder

encoded

Decoder

message

(38)

▪ Goal

▪

Enable communication and storage

▪

Detect or correct errors introduced during transmission

▪

Compress data: lossy or lossless

38

Snoopy

^Encoder 536E6F6F7079 ^Decoder

Snoopy

Encoder Decoder

(39)

39

▪

Goal: encode each symbol using an unique binary code (w/o ambiguity)

▪ How to represent symbols?

▪ How to ensure decode(encode(x))=x?

▪ How to minimize the number of bits?

(40)

40

▪

Goal: encode each symbol using an unique binary code (w/o ambiguity)

▪ How to represent symbols?

A G T C

0 1

0 1 0 1

10101101011010100101010010 T T C G G T T T G G G A T find a binary tree

(41)

▪

Fixed-length: use the same number of bits for encoding every symbol

▪ Ex. ASCII, Big5, UTF

▪

The length of this sequence is

▪

Variable-length: shorter codewords for more frequent symbols

▪

The length of this sequence is

41

Symbol A B C D E F

Frequency (K) 45 13 12 16 9 5

Fixed-length 000 001 010 011 100 101

Variable-length 0 101 100 111 1101 1100

E F

0 1

A B

0 1

C D

0 1

0

0 1

A

1

C D

1 0

E B

F 0

(42)

42

▪

Goal: encode each symbol using an unique binary code (w/o ambiguity)

▪ How to ensure decode(encode(x))=x?

use codes that are uniquely decodable

(43)

▪ Definition: a variable-length code where no codeword is a prefix of some other codeword

▪ Ambiguity: decode(1011100) can be ‘BF’ or ‘CDAA’

43

prefix codes are uniquely decodable

Symbol A B C D E F

Frequency (K) 45 13 12 16 9 5

Variable-length Prefix code 0 101 100 111 1101 1100

Not prefix code 0 101 10 111 1101 1100

(44)

44

▪

Goal: encode each symbol using an unique binary code (w/o ambiguity)

▪ How to minimize the number of bits?

more frequent symbols should use shorter codewords

(45)

45

shorter codewords longer codewords

(46)

46

▪

The weighted depth of a leaf = weight of a leaf (freq) × depth of a leaf

▪

Total length of codes = Total weighted depth of leaves

▪

Cost of the tree 𝑇

▪

Average bits per character

0 1

0

1

0 1

A:45

1

C:12 D:16

1 0

E:9 B:13

F:5 0

100

55

25 30

How to find the optimal prefix 14 code to minimize the cost?

(47)

▪

Input: 𝑛 positive integers 𝑤

₁

, 𝑤

₂

, … , 𝑤

_𝑛

indicating word frequency

▪

Output: a binary tree of 𝑛 leaves, whose weights form 𝑤

₁

, 𝑤

₂

, … , 𝑤

_𝑛

s.t.

the cost of the tree is minimized

47

(48)

▪

Subproblem: merge two characters into a new one whose weight is their sum

▪ PC(i): prefix code problem for 𝑖 leaves

▪ Goal: PC(n)

▪

Issues

▪ It is not the subproblem of the original problem

▪ The cost of two merged characters should be considered

48

Prefix Code Problem

Input: 𝑛 positive integers 𝑤₁, 𝑤₂, … , 𝑤_𝑛 indicating word frequency Output: a binary tree of 𝑛 leaves with minimal cost

PC(n)  PC(n - 1)

(49)

49

0 1

0

1

0 1

A:45

1

C:12 D:16

1 0

E:9 B:13

F:5 0

100

55

25 30

14

0 1

1

0 1

A:45

C:12 D:16

1 0

B:13

0 100

55

25 30

EF:14

(50)

▪

Suppose 𝑇’ is an optimal solution to PC(i, {w

_1…i-1

, z})

50

z

x y

▪

𝑇 is an optimal solution to

PC(i+1, {w

_1…i-1

, x, y})

(51)

▪

𝑇’

51

z

x y

▪

𝑇

(52)

▪

Optimal substructure: T is OPT if and only if T’ is OPT

52

T T’

The difference is

(53)

▪

Greedy choice: merge repeatedly until one tree left

▪ Select two trees 𝑥, 𝑦 with minimal frequency roots freq 𝑥 and freq 𝑦

▪ Merge into a single tree by adding root 𝑧 with the frequency freq 𝑥 + freq 𝑦

53

(54)

54

45

16 14

5 9

12 13

45

16 5 9 12 13

Initial set (store in a priority queue)

(55)

55

45

16 14

5 9

25

12 13

45

16 14

5 9

12 13

(56)

56

45

16

30

14

5 9

25

12 13

45

16 14

5 9

25

12 13

(57)

57

55 45

16

30

14

5 9

25

12 13

45

16

30

14

5 9

25

12 13

(58)

58

55 45

16

30

14

5 9

25

12 13

100

55 45

16

30

14

5 9

25

12 13

(59)

▪

Greedy choice: merge two nodes with min weights repeatedly

▪

Proof via contradiction

▪ Assume that there is no OPT including this greedy choice

▪ 𝑥 and 𝑦 are two symbols with lowest frequencies

▪ 𝑎 and 𝑏 are siblings with largest depths

▪ WLOG, assume freq 𝑎 ≤ freq 𝑏 and freq 𝑥 ≤ freq 𝑦

 freq 𝑥 ≤ freq 𝑎 and freq 𝑦 ≤ freq 𝑏

▪ Exchanging 𝑎 with 𝑥 and then 𝑏 with 𝑦 can make the tree equally or better

59

x

y

a b

OPT: T

(60)

60

x

y

a b

OPT: T

a

y

x b

T’

▪

Because T is OPT, T’ must be another optimal solution.

(61)

61

x

y

a b

OPT: T

a

y

x b

T’

a

b

x y

T’’

▪

Because T’ is OPT, T’’ must be another optimal solution.

Practice: prove the optimal tree must be a full tree

(62)

▪

Theorem: Huffman algorithm generates an optimal prefix code

▪

Proof

▪

Use induction to prove: Huffman codes are optimal for 𝑛 symbols

▪ 𝑛 = 2, trivial

▪ For a set 𝑆 with 𝑛 + 1 symbols,

1. Based on the greedy choice property, two symbols with minimum frequencies are siblings in T

2. Construct T’ by replacing these two symbols 𝑥 and 𝑦 with 𝑧 s.t. 𝑆′ = (𝑆\{𝑥, 𝑦}) ∪ 𝑧 and freq 𝑧 = freq 𝑥 + freq 𝑦

3. Assume T’ is the optimal tree for 𝑛 symbols by inductive hypothesis

4. Based on the optimal substructure property, we know that when T’ is optimal, T is optimal too (case 𝑛 + 1 holds)

62

This induction proof framework can be applied to prove its optimality using the optimal substructure and the greedy choice property.

(63)

63

Huffman(S) n = |S|

Q = Build-Priority-Queue(S) for i = 1 to n – 1

allocate a new node z

z.left = x = Extract-Min(Q) z.right = y = Extract-Min(Q) freq(z) = freq(x) + freq(y) Insert(Q, z)

return Extract-Min(Q) // return the prefix tree

(64)

▪ Huffman’s algorithm is optimal for a symbol-by-symbol coding with a known input probability distribution

▪ Huffman’s algorithm is sub-optimal when

▪

blending among symbols is allowed

▪

the probability distribution is unknown

▪

symbols are not independent

64

(65)

65

(66)

Course Website: http://ada17.csie.org Email: ada-ta@csie.ntu.edu.tw

66

Important announcement will be sent to @ntu.edu.tw mailbox

& post to the course website