Algorithm Design and Analysis Greedy Algorithms (1)

Greedy Algorithms (1)

Outline

• Greedy Algorithms

• Greedy #1: Activity-Selection / Interval Scheduling

• Greedy #2: Coin Changing

• Greedy #3: Fractional Knapsack Problem

• Greedy #4: Breakpoint Selection

• Greedy #5: Huffman Codes

• Greedy #6: Task-Scheduling

• Greedy #7: Scheduling to Minimize Lateness

Algorithm Design Strategy

• Do not focus on “specific algorithms”

• But “some strategies” to “design” algorithms

• First Skill: Divide-and-Conquer (各個擊破/分治)

• Second Skill: Dynamic Programming (動態規劃)

• Third Skill: Greedy (貪婪法則)

Greedy Algorithms

Textbook Chapter 16 – Greedy Algorithms

Textbook Chapter 16.2 – Elements of the greedy strategy

What is Greedy Algorithms?

• always makes the choice that looks best at the moment

• makes a locally optimal choice in the hope that this choice will lead to a globally optimal solution

• not always yield optimal solution; may end up at local optimal

local maximal

global maximal

local maximal

Algorithm Design Paradigms

• Dynamic Programming

• has optimal substructure

• make an informed choice after getting optimal solutions to subproblems

• dependent or overlapping subproblems

• Greedy Algorithms

• has optimal substructure

• make a greedy choice before solving the subproblem

• no overlapping subproblems

✓ Each round selects only one subproblem

✓ The subproblem size decreases

Optimal Solution

Possible Case 1 Possible

Case 2

max /min

Subproblem Solution Subproblem

Solution

+

= + ^Optimal

Solution

Greedy Choice

Subproblem Solution

= +

Greedy Procedure

1. Cast the optimization problem as one in which we make a choice and remain one subproblem to solve

2. Demonstrate the optimal substructure

✓ Combining an optimal solution to the subproblem via greedy can arrive an optimal solution to the original problem

3. Prove that there is always an optimal solution to the original problem that

makes the greedy choice

Greedy Algorithms

To yield an optimal solution, the problem should exhibit

1. Optimal Substructure : an optimal solution to the problem contains within its optimal solutions to subproblems

2. Greedy-Choice Property : making locally optimal (greedy) choices leads to

a globally optimal solution

Proof of Correctness Skills

• Optimal Substructure : an optimal solution to the problem contains within its optimal solutions to subproblems

• Greedy-Choice Property : making locally optimal (greedy) choices leads to a globally optimal solution

• Show that it exists an optimal solution that “contains” the greedy choice using exchange argument

• For any optimal solution OPT, the greedy choice 𝑔 has two cases

• 𝑔 is in OPT: done

• 𝑔 not in OPT: modify OPT into OPT’ s.t. OPT’ contains 𝑔 and is at least as good as OPT

OPT OPT’

✓ If OPT’ is better than OPT, the property is proved by contradiction

✓ If OPT’ is as good as OPT, then we showed that there exists an optimal solution containing 𝑔 by construction

Greedy #1: Activity-Selection / Interval Scheduling

Textbook Chapter 16.1 – An activity-selection problem

Activity-Selection/ Interval Scheduling

• Input: 𝑛 activities with start times 𝑠

and finish times 𝑓

(the activities are sorted in monotonically increasing order of finish time 𝑓

≤ 𝑓

≤ ⋯ ≤ 𝑓

)

• Output: the maximum number of compatible activities

• Without loss of generality: 𝑠

< 𝑠

< ⋯ < 𝑠

and 𝑓

< 𝑓

< ⋯ < 𝑓

• 大的包小的則不考慮大的 → 用小的取代大的一定不會變差

1 2 3 4 5 6

activity index

Weighted Interval Scheduling

• Subproblems

• WIS(i): weighted interval scheduling for the first 𝑖 jobs

• Goal: WIS(n)

• Dynamic programming algorithm

i 0 1 2 3 4 5 … n

M[i]

Weighted Interval Scheduling Problem

Input: 𝑛 jobs with 𝑠_𝑖, 𝑓_𝑖, 𝑣_𝑖 , 𝑝(𝑗) = largest index 𝑖 < 𝑗 s.t. jobs 𝑖 and 𝑗 are compatible Output: the maximum total value obtainable from compatible

Activity-Selection Problem

• Dynamic programming

• Optimal substructure is already proved

• Greedy algorithm

select the 𝑖-th activity

Why does the 𝑖-th activity must appear

in an OPT?

Activity-Selection Problem

Input: 𝑛 activities with 𝑠_𝑖, 𝑓_𝑖 , 𝑝(𝑗) = largest index 𝑖 < 𝑗 s.t. 𝑖 and 𝑗 are compatible Output: the maximum number of activities

Greedy-Choice Property

• Goal:

• Proof

• Assume there is an OPT solution for the first 𝑖 − 1 activities (𝑀_𝑖−1)

• 𝐴_𝑗 is the last activity in the OPT solution →

• Replacing 𝐴_𝑗 with 𝐴_𝑖 does not make the OPT worse

1 2 : i i - 1

activity index

Pseudo Code

Act-Select(n, s, f, v, p) M[0] = 0

for i = 1 to n

M[i] = 1 + M[p[i]]

return M[n]

Find-Solution(M, n) if n = 0

return {}

return {n} ∪ Find-Solution(p[n])

Activity-Selection Problem

Input: 𝑛 activities with 𝑠_𝑖, 𝑓_𝑖 , 𝑝(𝑗) = largest index 𝑖 < 𝑗 s.t. 𝑖 and 𝑗 are compatible Output: the maximum number of activities

Greedy #2: Coin Changing

Textbook Exercise 16.1

Coin Changing Problem

• Input: 𝑛 dollars and unlimited coins with values 𝑣

(1, 5, 10, 50)

• Output: the minimum number of coins with the total value 𝑛

• Cashier’s algorithm: at each iteration, add the coin with the largest value no more than the current total

Does this algorithm return the OPT?

Step 1: Cast Optimization Problem

• Subproblems

• C(i): minimal number of coins for the total value 𝑖

• Goal: C(n)

Coin Changing Problem

Input: 𝑛 dollars and unlimited coins with values 𝑣_𝑖 (1, 5, 10, 50) Output: the minimum number of coins with the total value 𝑛

Step 2: Prove Optimal Substructure

• Suppose OPT is an optimal solution to C(i) , there are 4 cases:

• Case 1: coin 1 in OPT

• OPT\coin1 is an optimal solution of C(i – v₁)

• Case 2: coin 2 in OPT

• OPT\coin2 is an optimal solution of C(i – v₂)

• Case 3: coin 3 in OPT

• OPT\coin3 is an optimal solution of C(i – v₃)

• Case 4: coin 4 in OPT

• OPT\coin4 is an optimal solution of C(i – v₄)

Coin Changing Problem

Input: 𝑛 dollars and unlimited coins with values 𝑣_𝑖 (1, 5, 10, 50) Output: the minimum number of coins with the total value 𝑛

Step 3: Prove Greedy-Choice Property

• Greedy choice: select the coin with the largest value no more than the current total

• Proof via contradiction (use the case 10 ≤ 𝑖 < 50 for demo)

• Assume that there is no OPT including this greedy choice (choose 10)

→ all OPT use 1, 5, 50 to pay 𝑖

• 50 cannot be used

• #coins with value 5 < 2 → otherwise we can use a 10 to have a better output

• #coins with value 1 < 5 → otherwise we can use a 5 to have a better output

• We cannot pay 𝑖 with the constraints (at most 5 + 4 = 9) Coin Changing Problem

Input: 𝑛 dollars and unlimited coins with values 𝑣_𝑖 (1, 5, 10, 50) Output: the minimum number of coins with the total value 𝑛

Greedy #3:

Fractional Knapsack Problem

Textbook Exercise 16.2-2

Knapsack Problem

• Input: 𝑛 items where 𝑖-th item has value 𝑣

and weighs 𝑤

(𝑣

and 𝑤

are positive integers)

• Output: the maximum value for the knapsack with capacity of 𝑊

• Variants of knapsack problem

• 0-1 Knapsack Problem: 每項物品只能拿一個

• Unbounded Knapsack Problem: 每項物品可以拿多個

• Multidimensional Knapsack Problem: 背包空間有限

• Multiple-Choice Knapsack Problem: 每一類物品最多拿一個

• Fractional Knapsack Problem: 物品可以只拿部分

Knapsack Problem

• Input: 𝑛 items where 𝑖-th item has value 𝑣

and weighs 𝑤

(𝑣

and 𝑤

are positive integers)

• Output: the maximum value for the knapsack with capacity of 𝑊

• Variants of knapsack problem

• 0-1 Knapsack Problem: 每項物品只能拿一個

• Unbounded Knapsack Problem: 每項物品可以拿多個

• Multidimensional Knapsack Problem: 背包空間有限

• Multiple-Choice Knapsack Problem: 每一類物品最多拿一個

• Fractional Knapsack Problem: 物品可以只拿部分

Fractional Knapsack Problem

• Input: 𝑛 items where 𝑖-th item has value 𝑣

and weighs 𝑤

(𝑣

and 𝑤

are positive integers)

• Output: the maximum value for the knapsack with capacity of 𝑊, where we can take any fraction of items

• Greedy algorithm: at each iteration, choose the item with the highest

𝑤_𝑖

and

continue when 𝑊 − 𝑤

> 0

Step 1: Cast Optimization Problem

• Subproblems

• F-KP(i, w): fractional knapsack problem within 𝑤 capacity for the first 𝑖 items

• Goal: F-KP(n, W)

Fractional Knapsack Problem

Input: 𝑛 items where 𝑖-th item has value 𝑣_𝑖 and weighs 𝑤_𝑖

Output: the max value within 𝑊 capacity, where we can take any fraction of items

Step 2: Prove Optimal Substructure

• Suppose OPT is an optimal solution to F-KP(i, w) , there are 2 cases:

• Case 1: full/partial item 𝑖 in OPT

• Remove 𝑤′ of item 𝑖 from OPT is an optimal solution of F-KP(i - 1, w – w^’)

• Case 2: item 𝑖 not in OPT

• OPT is an optimal solution of F-KP(i - 1, w) Fractional Knapsack Problem

Input: 𝑛 items where 𝑖-th item has value 𝑣_𝑖 and weighs 𝑤_𝑖

Output: the max value within 𝑊 capacity, where we can take any fraction of items

Step 3: Prove Greedy-Choice Property

• Greedy choice: select the item with the highest

𝑤_𝑖

• Proof via contradiction (𝑗 = argmax

𝑖

𝑣_𝑖 𝑤_𝑖

)

• Assume that there is no OPT including this greedy choice

• If 𝑊 ≤ 𝑤_𝑗, we can replace all items in OPT with item 𝑗

• If 𝑊 > 𝑤_𝑗, we can replace any item weighting 𝑤_𝑗 in OPT with item 𝑗

• The total value must be equal or higher, because item 𝑗 has the highest ^𝑣𝑖

𝑤_𝑖

Do other knapsack problems have this property?

Fractional Knapsack Problem

Input: 𝑛 items where 𝑖-th item has value 𝑣_𝑖 and weighs 𝑤_𝑖

Output: the max value within 𝑊 capacity, where we can take any fraction of items

Greedy #4: Breakpoint Selection

Breakpoint Selection Problem

• Input: a planned route with 𝑛 + 1 gas stations 𝑏

, … , 𝑏

; the car can go at most 𝐶 after refueling at a breakpoint

• Output: a refueling schedule (𝑏

→𝑏

) that minimizes the number of stops

• Greedy algorithm: go as far as you can before refueling

1 2 3 4 5

Ideally: stop when out of gas

Actually: may not be able to find the gas station when out of gas

1 2 3 4 5 6

Step 1: Cast Optimization Problem

• Subproblems

• B(i): breakpoint selection problem from 𝑏_𝑖 to 𝑏_𝑛

• Goal: B(0)

Breakpoint Selection Problem

Input: 𝑛 + 1 breakpoints 𝑏₀, … , 𝑏_𝑛; gas storage is 𝐶

Output: a refueling schedule (𝑏₀→𝑏_𝑛) that minimizes the number of stops

Step 2: Prove Optimal Substructure

• Suppose OPT is an optimal solution to B(i) where 𝑗 is the largest index satisfying 𝑏

− 𝑏

≤ 𝐶, there are 𝑗 − 𝑖 cases

• Case 1: stop at 𝑏_𝑖+1

• OPT+{b_i+1} is an optimal solution of B(i + 1)

• Case 2: stop at 𝑏_𝑖+2

• OPT+{b_i+2} is an optimal solution of B(i + 2) :

• Case 𝑗 − 𝑖: stop at 𝑏_𝑗

• OPT+{b_j} is an optimal solution of B(j) Breakpoint Selection Problem

Input: 𝑛 + 1 breakpoints 𝑏₀, … , 𝑏_𝑛; gas storage is 𝐶

Output: a refueling schedule (𝑏₀→𝑏_𝑛) that minimizes the number of stops

Step 3: Prove Greedy-Choice Property

• Greedy choice: go as far as you can before refueling (select 𝑏

)

• Proof via contradiction

• Assume that there is no OPT including this greedy choice (after 𝑏_𝑖 then stop at 𝑏_𝑘, 𝑘 ≠ 𝑗)

• If 𝑘 > 𝑗, we cannot stop at 𝑏_𝑘 due to out of gas

• If 𝑘 < 𝑗, we can replace the stop at 𝑏_𝑘 with the stop at 𝑏_𝑗

• The total value must be equal or higher, because we refuel later (𝑏_𝑗 > 𝑏_𝑘)

Breakpoint Selection Problem

Input: 𝑛 + 1 breakpoints 𝑏₀, … , 𝑏_𝑛; gas storage is 𝐶

Output: a refueling schedule (𝑏₀→𝑏_𝑛) that minimizes the number of stops

Pseudo Code

BP-Select(C, b)

Sort(b) s.t. b[0] < b[1] < … < b[n]

p = 0 S = {0}

for i = 1 to n - 1

if b[i + 1] – b[p] > C if i == p

return “no solution”

A = A ∪ {i}

p = i return A

Breakpoint Selection Problem

Input: 𝑛 + 1 breakpoints 𝑏₀, … , 𝑏_𝑛; gas storage is 𝐶

Output: a refueling schedule (𝑏₀→𝑏_𝑛) that minimizes the number of stops

Greedy #5: Huffman Codes

Textbook Chapter 16.3 – Huffman codes

Encoding & Decoding

• Code (編碼) is a system of rules to convert information—such as a letter, word, sound, image, or gesture—into another, sometimes shortened or secret, form or representation for communication through a channel or storage in a medium.

input message

decoded message Encoder encoded Decoder

message

Encoding & Decoding

• Goal

• Enable communication and storage

• Detect or correct errors introduced during transmission

• Compress data: lossy or lossless

Snoopy ^Encoder 536E6F6F7079 ^Decoder Snoopy

Encoder Decoder

Autoencoder

Lossless Data Compression

• Goal: encode each symbol using a unique binary code (w/o ambiguity)

• How to represent symbols?

• How to ensure decode(encode(x))=x?

• How to minimize the number of bits?

Lossless Data Compression

• Goal: encode each symbol using a unique binary code (w/o ambiguity)

• How to represent symbols?

• How to ensure decode(encode(x))=x?

• How to minimize the number of bits?

0 1

0 1 0 1

10101101011010100101010010 T T C G G T T T G G G A T

find a binary tree

Code

• Fixed-length: use the same number of bits for encoding every symbol

• Ex. ASCII, Big5, UTF

• The length of this sequence is

• Variable-length: shorter codewords for more frequent symbols

• The length of this sequence is

Fixed-length 000 001 010 011 100 101 Variable-length 0 101 100 111 1101 1100

E F

0 1

0 1

0 1

A B

0 1

C D

0 1

0

0 1

0 1

0 1

A

1

C D

1 0

E B

F 0

Lossless Data Compression

• Goal: encode each symbol using an unique binary code (w/o ambiguity)

• How to represent symbols?

• How to ensure decode(encode(x))=x?

• How to minimize the number of bits?

use codes that are uniquely decodable

Prefix Code

• Definition: a variable-length code where no codeword is a prefix of some other codeword

• Ambiguity: decode(1011100) can be ‘BF’ or ‘CDAA’

prefix codes are uniquely decodable

Symbol A B C D E F

Frequency (K) 45 13 12 16 9 5

Variable-length Prefix code 0 101 100 111 1101 1100 Not prefix code 0 101 10 111 1101 1100

Lossless Data Compression

• Goal: encode each symbol using an unique binary code (w/o ambiguity)

• How to represent symbols?

• How to ensure decode(encode(x))=x?

• How to minimize the number of bits?

more frequent symbols should use shorter codewords

Letter Frequency Distribution

shorter codewords longer codewords

Total Length of Codes

• The weighted depth of a leaf = weight of a leaf (freq) × depth of a leaf

• Total length of codes = Total weighted depth of leaves

• Cost of the tree 𝑇

• Average bits per character

0 1

0

1

0 1

A:45

1

C:12 D:16

1 0

B:13

0 100

55

25 30

14 How to find the optimal prefix

code to minimize the cost?

Prefix Code Problem

• Input: 𝑛 positive integers 𝑤

, 𝑤

, … , 𝑤

indicating word frequency

• Output: a binary tree of 𝑛 leaves, whose weights form 𝑤

, 𝑤

, … , 𝑤

s.t. the

cost of the tree is minimized

Step 1: Cast Optimization Problem

• Subproblem: merge two characters into a new one whose weight is their sum

• PC(i): prefix code problem for 𝑖 leaves

• Goal: PC(n)

• Issues

• It is not the subproblem of the original problem

• The cost of two merged characters should be considered

Prefix Code Problem

Input: 𝑛 positive integers 𝑤₁, 𝑤₂, … , 𝑤_𝑛 indicating word frequency Output: a binary tree of 𝑛 leaves with minimal cost

PC(n) → PC(n - 1)

Example

0 1

0

1

0 1

A:45

1

C:12 D:16

1 0

E:9 B:13

F:5 0

100

55

25 30

14

0 1

1

0 1

A:45

C:12 D:16

1 0

B:13

0 100

55

25 30

EF:14

Step 2: Prove Optimal Substructure

• Suppose 𝑇’ is an optimal solution to

PC(i, {w

, z}) • 𝑇 is an optimal solution to PC(i+1, {w

, x, y})

z Prefix Code Problem

Input: 𝑛 positive integers 𝑤₁, 𝑤₂, … , 𝑤_𝑛 indicating word frequency Output: a binary tree of 𝑛 leaves with minimal cost

Step 2: Prove Optimal Substructure

• 𝑇’ • 𝑇

z

x y

Step 2: Prove Optimal Substructure

• Optimal substructure: T’ is OPT if and only if T is OPT

T’ T

The difference is

Greedy Algorithm Design

• Greedy choice: merge repeatedly until one tree left

• Select two trees 𝑥, 𝑦 with minimal frequency roots freq 𝑥 and freq 𝑦

• Merge into a single tree by adding root 𝑧 with the frequency freq 𝑥 + freq 𝑦

Prefix Code Problem

Input: 𝑛 positive integers 𝑤₁, 𝑤₂, … , 𝑤_𝑛 indicating word frequency Output: a binary tree of 𝑛 leaves with minimal cost

Example

45

16 14

5 9

12 13

45

16 5 9 12 13

Initial set (store in a priority queue)

Example

45

16 14

5 9

25

12 13

45

16 14

5 9

12 13

Example

45

16

30

14

5 9

25

12 13

45

16 14

5 9

25

12 13

Example

55 45

16

30

14

5 9

25

12 13

45

16

30

14

5 9

25

12 13

Example

55 45

16

30

14

5 9

25

12 13

100

55 45

16

30

14

5 9

25

12 13

Step 3: Prove Greedy-Choice Property

• Greedy choice: merge two nodes with min weights repeatedly

• Proof via contradiction

• Assume that there is no OPT including this greedy choice

• 𝑥 and 𝑦 are two symbols with lowest frequencies

• 𝑎 and 𝑏 are siblings with largest depths

• WLOG, assume freq 𝑎 ≤ freq 𝑏 and freq 𝑥 ≤ freq 𝑦

→ freq 𝑥 ≤ freq 𝑎 and freq 𝑦 ≤ freq 𝑏

• Exchanging 𝑎 with 𝑥 and then 𝑏 with 𝑦 can make the tree equally or better

x

y

a b

OPT: T Prefix Code Problem

Input: 𝑛 positive integers 𝑤₁, 𝑤₂, … , 𝑤_𝑛 indicating word frequency Output: a binary tree of 𝑛 leaves with minimal cost

Step 3: Prove Greedy-Choice Property

x

y

a b

OPT: T

a

y

x b

T’

▪ Because T is OPT, T’ must be another optimal solution.

Step 3: Prove Greedy-Choice Property

x

y

a b

OPT: T

a

y

x b

T’

a

b

x y

T’’

▪ Because T’ is OPT, T’’ must be another optimal solution.

Correctness and Optimality

• Theorem: Huffman algorithm generates an optimal prefix code

• Proof

• Use induction to prove: Huffman codes are optimal for 𝑛 symbols

• 𝑛 = 2, trivial

• For a set 𝑆 with 𝑛 + 1 symbols,

1. Based on the greedy choice property, two symbols with minimum frequencies are siblings in T

2. Construct T’ by replacing these two symbols 𝑥 and 𝑦 with 𝑧 s.t. 𝑆′ = (𝑆\{𝑥, 𝑦}) ∪ 𝑧 and freq 𝑧 = freq 𝑥 + freq 𝑦

3. Assume T’ is the optimal tree for 𝑛 symbols by inductive hypothesis

4. Based on the optimal substructure property, we know that when T’ is optimal, T is optimal too (case 𝑛 + 1 holds)

Pseudo Code

Huffman(S) n = |S|

Q = Build-Priority-Queue(S) for i = 1 to n – 1

allocate a new node z

z.left = x = Extract-Min(Q) z.right = y = Extract-Min(Q) freq(z) = freq(x) + freq(y) Insert(Q, z)

Delete(Q, x) Delete(Q, y)

return Extract-Min(Q) // return the prefix tree Prefix Code Problem

Input: 𝑛 positive integers 𝑤₁, 𝑤₂, … , 𝑤_𝑛 indicating word frequency Output: a binary tree of 𝑛 leaves with minimal cost

Drawbacks of Huffman Codes

• Huffman’s algorithm is optimal for a symbol-by-symbol coding with a known input probability distribution

• Huffman’s algorithm is sub-optimal when

• blending among symbols is allowed

• the probability distribution is unknown

• symbols are not independent

To Be Continued…

Important announcement will be sent to

@ntu.edu.tw mailbox & post to the course website

Course Website: http://ada.miulab.tw