**Greedy Algorithms (1)**

**Outline**

### • Greedy Algorithms

### • Greedy #1: Activity-Selection / Interval Scheduling

### • Greedy #2: Coin Changing

### • Greedy #3: Fractional Knapsack Problem

### • Greedy #4: Breakpoint Selection

### • Greedy #5: Huffman Codes

### • Greedy #6: Task-Scheduling

### • Greedy #7: Scheduling to Minimize Lateness

**Algorithm Design Strategy**

### • Do not focus on “specific algorithms”

### • But “some strategies” to “design” algorithms

### • First Skill: Divide-and-Conquer (各個擊破/分治)

### • Second Skill: Dynamic Programming (動態規劃)

### • Third Skill: Greedy (貪婪法則)

**Greedy Algorithms**

Textbook Chapter 16 – Greedy Algorithms

Textbook Chapter 16.2 – Elements of the greedy strategy

**What is Greedy Algorithms?**

### • always makes the choice that looks best at the moment

### • makes a locally optimal choice in the hope that this choice will lead to a globally optimal solution

• not always yield optimal solution; may end up at local optimal

local maximal

global maximal

local maximal

**Algorithm Design Paradigms**

### • Dynamic Programming

**• has optimal substructure**

• make an informed choice after getting optimal solutions to subproblems

**• dependent or overlapping**
subproblems

### • Greedy Algorithms

**• has optimal substructure**

• make a greedy choice before solving the subproblem

**• no overlapping subproblems**

✓ Each round selects only one subproblem

✓ The subproblem size decreases

Optimal Solution

Possible Case 1 Possible

Case 2

max /min

Subproblem Solution Subproblem

Solution

+

= + ^{Optimal }

Solution

Greedy Choice

Subproblem Solution

= +

**Greedy Procedure**

### 1. Cast the optimization problem as one in which we make a choice and remain one subproblem to solve

### 2. Demonstrate the optimal substructure

✓ Combining an optimal solution to the subproblem via greedy can arrive an optimal solution to the original problem

### 3. Prove that there is always an optimal solution to the original problem that

### makes the greedy choice

**Greedy Algorithms**

### To yield an optimal solution, the problem should exhibit

### 1. Optimal Substructure : an optimal solution to the problem contains within its optimal solutions to subproblems

### 2. Greedy-Choice Property : making locally optimal (greedy) choices leads to

### a globally optimal solution

**Proof of Correctness Skills**

### • Optimal Substructure : an optimal solution to the problem contains within its optimal solutions to subproblems

### • Greedy-Choice Property : making locally optimal (greedy) choices leads to a globally optimal solution

• Show that it exists an optimal solution that “contains” the greedy choice using
**exchange argument**

• For any optimal solution OPT, the greedy choice 𝑔 has two cases

• 𝑔 is in OPT: done

• 𝑔 not in OPT: modify OPT into OPT’ s.t. OPT’ contains 𝑔 and is at least as good as OPT

OPT OPT’

✓ If OPT’ is better than OPT, the property is proved by contradiction

✓ If OPT’ is as good as OPT, then we showed that there exists an optimal solution containing 𝑔 by construction

**Greedy #1: Activity-Selection ** **/ Interval Scheduling**

Textbook Chapter 16.1 – An activity-selection problem

**Activity-Selection/ Interval Scheduling**

### • Input: 𝑛 activities with start times 𝑠

_{𝑖}

### and finish times 𝑓

_{𝑖}

### (the activities are sorted in monotonically increasing order of finish time 𝑓

_{1}

### ≤ 𝑓

_{2}

### ≤ ⋯ ≤ 𝑓

_{𝑛}

### )

### • Output: the maximum number of compatible activities

### • Without loss of generality: 𝑠

_{1}

### < 𝑠

_{2}

### < ⋯ < 𝑠

_{𝑛}

### and 𝑓

_{1}

### < 𝑓

_{2}

### < ⋯ < 𝑓

_{𝑛}

• 大的包小的則不考慮大的 → 用小的取代大的一定不會變差

1 2 3 4 5 6

activity index

**Weighted Interval Scheduling**

### • Subproblems

• WIS(i): weighted interval scheduling for the first 𝑖 jobs

• Goal: WIS(n)

### • Dynamic programming algorithm

**i** **0** **1** **2** **3** **4** **5** **…** **n**

M[i]

**Weighted Interval Scheduling Problem**

Input: 𝑛 jobs with 𝑠_{𝑖}, 𝑓_{𝑖}, 𝑣_{𝑖} , 𝑝(𝑗) = largest index 𝑖 < 𝑗 s.t. jobs 𝑖 and 𝑗 are compatible
Output: the maximum total value obtainable from compatible

**Activity-Selection Problem**

### • Dynamic programming

**• Optimal substructure is already proved**

### • Greedy algorithm

select the 𝑖-th activity

Why does the 𝑖-th activity must appear

in an OPT?

**Activity-Selection Problem**

Input: 𝑛 activities with 𝑠_{𝑖}, 𝑓_{𝑖} , 𝑝(𝑗) = largest index 𝑖 < 𝑗 s.t. 𝑖 and 𝑗 are compatible
Output: the maximum number of activities

**Greedy-Choice Property**

### • Goal:

### • Proof

• Assume there is an OPT solution for the first 𝑖 − 1 activities (𝑀_{𝑖−1})

• 𝐴_{𝑗} is the last activity in the OPT solution →

• Replacing 𝐴_{𝑗} with 𝐴_{𝑖} does not make the OPT worse

1
2
:
i
*i - 1*

activity index

**Pseudo Code**

Act-Select(n, s, f, v, p) M[0] = 0

for i = 1 to n

M[i] = 1 + M[p[i]]

return M[n]

Find-Solution(M, n) if n = 0

return {}

return {n} ∪ Find-Solution(p[n])

**Activity-Selection Problem**

Input: 𝑛 activities with 𝑠_{𝑖}, 𝑓_{𝑖} , 𝑝(𝑗) = largest index 𝑖 < 𝑗 s.t. 𝑖 and 𝑗 are compatible
Output: the maximum number of activities

**Greedy #2: Coin Changing**

Textbook Exercise 16.1

**Coin Changing Problem**

### • Input: 𝑛 dollars and unlimited coins with values 𝑣

_{𝑖}

### (1, 5, 10, 50)

### • Output: the minimum number of coins with the total value 𝑛

**• Cashier’s algorithm: at each iteration, add the coin with the largest value no ** more than the current total

Does this algorithm return the OPT?

**Step 1: Cast Optimization Problem**

### • Subproblems

• C(i): minimal number of coins for the total value 𝑖

• Goal: C(n)

**Coin Changing Problem**

Input: 𝑛 dollars and unlimited coins with values 𝑣_{𝑖} (1, 5, 10, 50)
Output: the minimum number of coins with the total value 𝑛

**Step 2: Prove Optimal Substructure**

### • Suppose OPT is an optimal solution to C(i) , there are 4 cases:

• Case 1: coin 1 in OPT

• OPT\coin1 is an optimal solution of C(i – v_{1})

• Case 2: coin 2 in OPT

• OPT\coin2 is an optimal solution of C(i – v_{2})

• Case 3: coin 3 in OPT

• OPT\coin3 is an optimal solution of C(i – v_{3})

• Case 4: coin 4 in OPT

• OPT\coin4 is an optimal solution of C(i – v_{4})

**Coin Changing Problem**

Input: 𝑛 dollars and unlimited coins with values 𝑣_{𝑖} (1, 5, 10, 50)
Output: the minimum number of coins with the total value 𝑛

**Step 3: Prove Greedy-Choice Property **

### • Greedy choice: select the coin with the largest value no more than the current total

### • Proof via contradiction (use the case 10 ≤ 𝑖 < 50 for demo)

• Assume that there is no OPT including this greedy choice (choose 10)

→ all OPT use 1, 5, 50 to pay 𝑖

• 50 cannot be used

• #coins with value 5 < 2 → otherwise we can use a 10 to have a better output

• #coins with value 1 < 5 → otherwise we can use a 5 to have a better output

• We cannot pay 𝑖 with the constraints (at most 5 + 4 = 9)
**Coin Changing Problem**

Input: 𝑛 dollars and unlimited coins with values 𝑣_{𝑖} (1, 5, 10, 50)
Output: the minimum number of coins with the total value 𝑛

**Greedy #3:**

**Fractional Knapsack Problem**

Textbook Exercise 16.2-2

**Knapsack Problem**

### • Input: 𝑛 items where 𝑖-th item has value 𝑣

_{𝑖}

### and weighs 𝑤

_{𝑖}

### (𝑣

_{𝑖}

### and 𝑤

_{𝑖}

### are positive integers)

### • Output: the maximum value for the knapsack with capacity of 𝑊

### • Variants of knapsack problem

• 0-1 Knapsack Problem: 每項物品只能拿一個

• Unbounded Knapsack Problem: 每項物品可以拿多個

• Multidimensional Knapsack Problem: 背包空間有限

• Multiple-Choice Knapsack Problem: 每一類物品最多拿一個

• Fractional Knapsack Problem: 物品可以只拿部分

**Knapsack Problem**

### • Input: 𝑛 items where 𝑖-th item has value 𝑣

_{𝑖}

### and weighs 𝑤

_{𝑖}

### (𝑣

_{𝑖}

### and 𝑤

_{𝑖}

### are positive integers)

### • Output: the maximum value for the knapsack with capacity of 𝑊

### • Variants of knapsack problem

• 0-1 Knapsack Problem: 每項物品只能拿一個

• Unbounded Knapsack Problem: 每項物品可以拿多個

• Multidimensional Knapsack Problem: 背包空間有限

• Multiple-Choice Knapsack Problem: 每一類物品最多拿一個

**• Fractional Knapsack Problem: 物品可以只拿部分**

**Fractional Knapsack Problem**

### • Input: 𝑛 items where 𝑖-th item has value 𝑣

_{𝑖}

### and weighs 𝑤

_{𝑖}

### (𝑣

_{𝑖}

### and 𝑤

_{𝑖}

### are positive integers)

### • Output: the maximum value for the knapsack with capacity of 𝑊, where we **can take any fraction of items**

### • Greedy algorithm: at each iteration, choose the item with the highest

^{𝑣}

^{𝑖}

𝑤_{𝑖}

### and

### continue when 𝑊 − 𝑤

_{𝑖}

### > 0

**Step 1: Cast Optimization Problem**

### • Subproblems

• F-KP(i, w): fractional knapsack problem within 𝑤 capacity for the first 𝑖 items

• Goal: F-KP(n, W)

**Fractional Knapsack Problem**

Input: 𝑛 items where 𝑖-th item has value 𝑣_{𝑖} and weighs 𝑤_{𝑖}

Output: the max value within 𝑊 capacity, where we can take any fraction of items

**Step 2: Prove Optimal Substructure**

### • Suppose OPT is an optimal solution to F-KP(i, w) , there are 2 cases:

• Case 1: full/partial item 𝑖 in OPT

• Remove 𝑤′ of item 𝑖 from OPT is an optimal solution of F-KP(i - 1, w – w^{’})

• Case 2: item 𝑖 not in OPT

• OPT is an optimal solution of F-KP(i - 1, w)
**Fractional Knapsack Problem**

Input: 𝑛 items where 𝑖-th item has value 𝑣_{𝑖} and weighs 𝑤_{𝑖}

Output: the max value within 𝑊 capacity, where we can take any fraction of items

**Step 3: Prove Greedy-Choice Property **

### • Greedy choice: select the item with the highest

^{𝑣}

^{𝑖}

𝑤_{𝑖}

### • Proof via contradiction (𝑗 = argmax

𝑖

𝑣_{𝑖}
𝑤_{𝑖}

### )

• Assume that there is no OPT including this greedy choice

• If 𝑊 ≤ 𝑤_{𝑗}, we can replace all items in OPT with item 𝑗

• If 𝑊 > 𝑤_{𝑗}, we can replace any item weighting 𝑤_{𝑗} in OPT with item 𝑗

• The total value must be equal or higher, because item 𝑗 has the highest ^{𝑣}^{𝑖}

𝑤_{𝑖}

Do other knapsack problems have this property?

**Fractional Knapsack Problem**

Input: 𝑛 items where 𝑖-th item has value 𝑣_{𝑖} and weighs 𝑤_{𝑖}

Output: the max value within 𝑊 capacity, where we can take any fraction of items

**Greedy #4: Breakpoint Selection**

**Breakpoint Selection Problem**

### • Input: a planned route with 𝑛 + 1 gas stations 𝑏

_{0}

### , … , 𝑏

_{𝑛}

### ; the car can go at most 𝐶 after refueling at a breakpoint

### • Output: a refueling schedule (𝑏

_{0}

### →𝑏

_{𝑛}

### ) that minimizes the number of stops

### • Greedy algorithm: go as far as you can before refueling

1 2 3 4 5

Ideally: stop when out of gas

Actually: may not be able to find the gas station when out of gas

1 2 3 4 5 6

**Step 1: Cast Optimization Problem**

### • Subproblems

• B(i): breakpoint selection problem from 𝑏_{𝑖} to 𝑏_{𝑛}

• Goal: B(0)

**Breakpoint Selection Problem**

Input: 𝑛 + 1 breakpoints 𝑏_{0}, … , 𝑏_{𝑛}; gas storage is 𝐶

Output: a refueling schedule (𝑏_{0}→𝑏_{𝑛}) that minimizes the number of stops

**Step 2: Prove Optimal Substructure**

### • Suppose OPT is an optimal solution to B(i) where 𝑗 is the largest index satisfying 𝑏

_{𝑗}

### − 𝑏

_{𝑖}

### ≤ 𝐶, there are 𝑗 − 𝑖 cases

• Case 1: stop at 𝑏_{𝑖+1}

*• OPT+{b** _{i+1}*} is an optimal solution of B(i + 1)

• Case 2: stop at 𝑏_{𝑖+2}

*• OPT+{b** _{i+2}*} is an optimal solution of B(i + 2)
:

• Case 𝑗 − 𝑖: stop at 𝑏_{𝑗}

*• OPT+{b** _{j}*} is an optimal solution of B(j)

**Breakpoint Selection Problem**

Input: 𝑛 + 1 breakpoints 𝑏_{0}, … , 𝑏_{𝑛}; gas storage is 𝐶

Output: a refueling schedule (𝑏_{0}→𝑏_{𝑛}) that minimizes the number of stops

**Step 3: Prove Greedy-Choice Property **

### • Greedy choice: go as far as you can before refueling (select 𝑏

_{𝑗}

### )

### • Proof via contradiction

• Assume that there is no OPT including this greedy choice (after 𝑏_{𝑖} then stop at 𝑏_{𝑘}, 𝑘 ≠ 𝑗)

• If 𝑘 > 𝑗, we cannot stop at 𝑏_{𝑘} due to out of gas

• If 𝑘 < 𝑗, we can replace the stop at 𝑏_{𝑘} with the stop at 𝑏_{𝑗}

• The total value must be equal or higher, because we refuel later (𝑏_{𝑗} > 𝑏_{𝑘})

**Breakpoint Selection Problem**

Input: 𝑛 + 1 breakpoints 𝑏_{0}, … , 𝑏_{𝑛}; gas storage is 𝐶

Output: a refueling schedule (𝑏_{0}→𝑏_{𝑛}) that minimizes the number of stops

**Pseudo Code**

BP-Select(C, b)

Sort(b) s.t. b[0] < b[1] < … < b[n]

p = 0 S = {0}

for i = 1 to n - 1

if b[i + 1] – b[p] > C if i == p

return “no solution”

A = A ∪ {i}

p = i return A

**Breakpoint Selection Problem**

Input: 𝑛 + 1 breakpoints 𝑏_{0}, … , 𝑏_{𝑛}; gas storage is 𝐶

Output: a refueling schedule (𝑏_{0}→𝑏_{𝑛}) that minimizes the number of stops

**Greedy #5: Huffman Codes**

Textbook Chapter 16.3 – Huffman codes

**Encoding & Decoding**

**• Code (編碼) is a system of rules to convert information—such as a letter, ** word, sound, image, or gesture—into another, sometimes shortened or secret, form or representation for communication through a channel or storage in a medium.

input message

decoded message Encoder encoded Decoder

message

**Encoding & Decoding**

### • Goal

• Enable communication and storage

• Detect or correct errors introduced during transmission

• Compress data: lossy or lossless

Snoopy ^{Encoder} 536E6F6F7079 ^{Decoder} Snoopy

Encoder Decoder

**Autoencoder**

**Lossless Data Compression**

### • Goal: encode each symbol using a unique binary code (w/o ambiguity)

• How to represent symbols?

• How to ensure decode(encode(x))=x?

• How to minimize the number of bits?

**Lossless Data Compression**

### • Goal: encode each symbol using a unique binary code (w/o ambiguity)

**• How to represent symbols?**

• How to ensure decode(encode(x))=x?

• How to minimize the number of bits?

0 1

0 1 0 1

**10101101011010100101010010**
**T T** **C** **G G** **T T T** **G G G** **A** **T**

find a binary tree

**Code**

**• Fixed-length: use the same number ** of bits for encoding every symbol

• Ex. ASCII, Big5, UTF

### • The length of this sequence is

**• Variable-length: shorter codewords** for more frequent symbols

### • The length of this sequence is

Fixed-length 000 001 010 011 100 101 Variable-length 0 101 100 111 1101 1100

E F

0 1

0 1

0 1

A B

0 1

C D

0 1

0

0 1

0 1

0 1

A

1

C D

1 0

E B

F 0

**Lossless Data Compression**

### • Goal: encode each symbol using an unique binary code (w/o ambiguity)

• How to represent symbols?

**• How to ensure decode(encode(x))=x?**

• How to minimize the number of bits?

use codes that are uniquely decodable

**Prefix Code**

• Definition: a variable-length code where no codeword is a prefix of some other codeword

• Ambiguity: decode(1011100) can be ‘BF’ or ‘CDAA’

prefix codes are uniquely decodable

**Symbol** **A** **B** **C** **D** **E** **F**

*Frequency (K)* *45* *13* *12* *16* *9* *5*

Variable-length Prefix code 0 101 100 111 1101 1100
Not prefix code 0 **101** **10** 111 1101 1100

**Lossless Data Compression**

### • Goal: encode each symbol using an unique binary code (w/o ambiguity)

• How to represent symbols?

• How to ensure decode(encode(x))=x?

**• How to minimize the number of bits?**

more frequent symbols should use shorter codewords

**Letter Frequency Distribution**

shorter codewords longer codewords

**Total Length of Codes**

### • The weighted depth of a leaf = weight of a leaf (freq) × depth of a leaf

### • Total length of codes = Total weighted depth of leaves

### • Cost of the tree 𝑇

### • Average bits per character

0 1

0

1

0 1

A:45

1

C:12 D:16

1 0

B:13

0 100

55

25 30

14
**How to find the optimal prefix **

**code to minimize the cost?**

**Prefix Code Problem**

### • Input: 𝑛 positive integers 𝑤

_{1}

### , 𝑤

_{2}

### , … , 𝑤

_{𝑛}

### indicating word frequency

### • Output: a binary tree of 𝑛 leaves, whose weights form 𝑤

_{1}

### , 𝑤

_{2}

### , … , 𝑤

_{𝑛}

### s.t. the

### cost of the tree is minimized

**Step 1: Cast Optimization Problem**

### • Subproblem: merge two characters into a new one whose weight is their sum

• PC(i): prefix code problem for 𝑖 leaves

• Goal: PC(n)

### • Issues

• It is not the subproblem of the original problem

• The cost of two merged characters should be considered

**Prefix Code Problem**

Input: 𝑛 positive integers 𝑤_{1}, 𝑤_{2}, … , 𝑤_{𝑛} indicating word frequency
Output: a binary tree of 𝑛 leaves with minimal cost

PC(n) → PC(n - 1)

**Example**

0 1

0

1

0 1

A:45

1

C:12 D:16

1 0

E:9 B:13

F:5 0

100

55

25 30

14

0 1

1

0 1

A:45

C:12 D:16

1 0

B:13

0 100

55

25 30

EF:14

**Step 2: Prove Optimal Substructure**

### • Suppose 𝑇’ is an optimal solution to

### PC(i, {w

_{1…i-1}

### , z}) • 𝑇 is an optimal solution to PC(i+1, {w

_{1…i-1}

### , x, y})

z
**Prefix Code Problem**

Input: 𝑛 positive integers 𝑤_{1}, 𝑤_{2}, … , 𝑤_{𝑛} indicating word frequency
Output: a binary tree of 𝑛 leaves with minimal cost

**Step 2: Prove Optimal Substructure**

### • 𝑇’ • 𝑇

z

x y

**Step 2: Prove Optimal Substructure**

### • Optimal substructure: T’ is OPT if and only if T is OPT

T’ T

The difference is

**Greedy Algorithm Design**

### • Greedy choice: merge repeatedly until one tree left

• Select two trees 𝑥, 𝑦 with minimal frequency roots freq 𝑥 and freq 𝑦

• Merge into a single tree by adding root 𝑧 with the frequency freq 𝑥 + freq 𝑦

**Prefix Code Problem**

Input: 𝑛 positive integers 𝑤_{1}, 𝑤_{2}, … , 𝑤_{𝑛} indicating word frequency
Output: a binary tree of 𝑛 leaves with minimal cost

**Example**

45

16 14

5 9

12 13

45

16 5 9 12 13

Initial set (store in a priority queue)

**Example**

45

16 14

5 9

25

12 13

45

16 14

5 9

12 13

**Example**

45

16

30

14

5 9

25

12 13

45

16 14

5 9

25

12 13

**Example**

55 45

16

30

14

5 9

25

12 13

45

16

30

14

5 9

25

12 13

**Example**

55 45

16

30

14

5 9

25

12 13

100

55 45

16

30

14

5 9

25

12 13

**Step 3: Prove Greedy-Choice Property **

### • Greedy choice: merge two nodes with min weights repeatedly

### • Proof via contradiction

• Assume that there is no OPT including this greedy choice

• 𝑥 and 𝑦 are two symbols with lowest frequencies

• 𝑎 and 𝑏 are siblings with largest depths

• WLOG, assume freq 𝑎 ≤ freq 𝑏 and freq 𝑥 ≤ freq 𝑦

→ freq 𝑥 ≤ freq 𝑎 and freq 𝑦 ≤ freq 𝑏

• Exchanging 𝑎 with 𝑥 and then 𝑏 with 𝑦 can make the tree equally or better

x

y

a b

OPT: T
**Prefix Code Problem**

_{1}, 𝑤_{2}, … , 𝑤_{𝑛} indicating word frequency
Output: a binary tree of 𝑛 leaves with minimal cost

**Step 3: Prove Greedy-Choice Property **

x

y

a b

OPT: T

a

y

x b

T’

▪ Because T is OPT, T’ must be another optimal solution.

**Step 3: Prove Greedy-Choice Property **

x

y

a b

OPT: T

a

y

x b

T’

a

b

x y

T’’

▪ Because T’ is OPT, T’’ must be another optimal solution.

**Correctness and Optimality**

### • Theorem: Huffman algorithm generates an optimal prefix code

### • Proof

• Use induction to prove: Huffman codes are optimal for 𝑛 symbols

• 𝑛 = 2, trivial

• For a set 𝑆 with 𝑛 + 1 symbols,

1. Based on the greedy choice property, two symbols with minimum frequencies are siblings in T

2. Construct T’ by replacing these two symbols 𝑥 and 𝑦 with 𝑧 s.t. 𝑆′ = (𝑆\{𝑥, 𝑦}) ∪ 𝑧 and freq 𝑧 = freq 𝑥 + freq 𝑦

3. Assume T’ is the optimal tree for 𝑛 symbols by inductive hypothesis

4. Based on the optimal substructure property, we know that when T’ is optimal, T is optimal too (case 𝑛 + 1 holds)

**Pseudo Code**

Huffman(S) n = |S|

Q = Build-Priority-Queue(S) for i = 1 to n – 1

allocate a new node z

z.left = x = Extract-Min(Q) z.right = y = Extract-Min(Q) freq(z) = freq(x) + freq(y) Insert(Q, z)

Delete(Q, x) Delete(Q, y)

return Extract-Min(Q) // return the prefix tree
**Prefix Code Problem**

_{1}, 𝑤_{2}, … , 𝑤_{𝑛} indicating word frequency
Output: a binary tree of 𝑛 leaves with minimal cost

**Drawbacks of Huffman Codes**

### • Huffman’s algorithm is optimal for a symbol-by-symbol coding with a known input probability distribution

### • Huffman’s algorithm is sub-optimal when

• blending among symbols is allowed

• the probability distribution is unknown

• symbols are not independent

**To Be Continued…**

Important announcement will be sent to

@ntu.edu.tw mailbox & post to the course website

Course Website: http://ada.miulab.tw