• 沒有找到結果。

# Algorithm Design and Analysis

N/A
N/A
Protected

Share "Algorithm Design and Analysis"

Copied!
73
0
0

(1)

(2)

### Algorithm Design Strategy

• Do not focus on “specific algorithms”

• But “some strategies” to “design” algorithms

• First Skill: Divide-and-Conquer (各個擊破/分治法)

(3)

### Outline

• Recurrence (遞迴)

• Divide-and-Conquer

• D&C #1: Tower of Hanoi (河內塔)

• D&C #2: Merge Sort

• D&C #3: Bitonic Champion

• D&C #4: Maximum Subarray

• Solving Recurrences

• Substitution Method

• Recursion-Tree Method

• Master Method

• D&C #5: Matrix Multiplication

• D&C #6: Selection Problem

• D&C #7: Closest Pair of Points Problem

Divide-and-Conquer 之神乎奇技

Divide-and-Conquer 首部曲

(4)

### What is Divide-and-Conquer?

• Solve a problem recursively

• Apply three steps at each level of the recursion

1. Divide the problem into a number of subproblems that are smaller instances of the same problem (比較小的同樣問題)

2. Conquer the subproblems by solving them recursively If the subproblem sizes are small enough

• then solve the subproblems

• else recursively solve itself

3. Combine the solutions to the subproblems into the solution for the original problem

base case recursive case

(5)

### Divide-and-Conquer Benefits

• Easy to solve difficult problems

• Thinking: solve easiest case + combine smaller solutions into the original solution

• Easy to find an efficient algorithm

• Better time complexity

• Suitable for parallel computing (multi-core systems)

• More efficient memory access

• Subprograms and their data can be put in cache in stead of accessing main memory

(6)

## Recurrence (遞迴)

(7)

### Recurrence Relation

• Definition

A recurrence is an equation or inequality that describes a function in terms of its value on smaller inputs.

• Example

Fibonacci sequence (費波那契數列)

• Base case: F(0) = F(1) = 1

• Recursive case: F(n) = F(n-1) + F(n-2)

n 0 1 2 3 4 5 6 7 8

2

1 1 3 5 8

13

21

(8)

(9)

### Recurrence Benefits

• Easy & Clear

• Define base case and recursive case

• Define a long sequence

Base case

Recursive case

F(0), F(1), F(2)………

unlimited sequence

a program for solving F(n)

Fibonacci(n) // recursive function: 程式中會呼叫自己的函數 if n < 2 // base case: termination condition

return 1

// recursive case: call itself for solving subproblems

important otherwise the program cannot stop

(10)

### Recurrence v.s. Non-Recurrence

Fibonacci(n) if n < 2

____return 1 a[0] <- 1

a[1] <- 1

for i = 2 … n

____a[i] = a[i-1] + a[i-2]

return a[n]

Fibonacci(n)

if n < 2 // base case ____return 1

// recursive case

return Fibonacci(n-1) + Fibonacci(n-2)

Recursive function

• Clear structure

• Poor efficiency

Non-recursive function

• Better efficiency

• Unclear structure

(11)

### Recurrence Benefits

• Easy & Clear

• Define base case and recursive case

• Define a long sequence

Hanoi(n) is not easy to solve.

✓ It is easy to solve when n is small

✓ we can find the relation between Hanoi(n) & Hanoi(n-1)

Base case

Recursive case

If a problem can be simplified into a base case and a recursive case, then we can find an algorithm that solves this problem.

Base case

Recursive case

F(0), F(1), F(2)………

unlimited sequence

a program for solving F(n)

(12)

## D&C #1: Tower of Hanoi

(13)

### Tower of Hanoi (河內塔)

• Problem: move n disks from A to C

• Rules

• Move one disk at a time

• Cannot place a larger disk onto a smaller disk

A B C

(14)

### Hanoi(1)

• Move 1 from A to C

Disk 1

→ 1 move in total Base case

Disk 1

(15)

### Hanoi(2)

• Move 1 from A to B

• Move 2 from A to C

• Move 1 from B to C

Disk 2 Disk 1

→ 3 moves in total

Disk 1 Disk 2

Disk 1

(16)

### Hanoi(3)

• How to move 3 disks?

• How many moves in total?

Disk 3 Disk 2 Disk 1

(17)

### Hanoi(n)

• How to move n disks?

• How many moves in total?

Disk n Disk n-1 Disk n-2

(18)

### Hanoi(n)

• To move n disks from A to C (for n > 1):

1. Move Disk 1~n-1 from A to B

Disk n Disk n-1 Disk n-2

(19)

### Hanoi(n)

• To move n disks from A to C (for n > 1):

1. Move Disk 1~n-1 from A to B

Disk n Disk n-1

Disk n-2

(20)

### Hanoi(n)

• To move n disks from A to C (for n > 1):

1. Move Disk 1~n-1 from A to B 2. Move Disk n from A to C

Disk n Disk n-1

Disk n-2

(21)

### Hanoi(n)

• To move n disks from A to C (for n > 1):

1. Move Disk 1~n-1 from A to B 2. Move Disk n from A to C

Disk n Disk n-1

Disk n-2

(22)

### Hanoi(n)

• To move n disks from A to C (for n > 1):

1. Move Disk 1~n-1 from A to B 2. Move Disk n from A to C

3. Move Disk 1~n-1 from B to C

Disk n-1 Disk n-2

Disk n

(23)

### Hanoi(n)

• To move n disks from A to C (for n > 1):

1. Move Disk 1~n-1 from A to B 2. Move Disk n from A to C

3. Move Disk 1~n-1 from B to C

Disk n-1 Disk n-2 Disk n

→ 2Hanoi(n-1) + 1 moves in total recursive case

(24)

### Pseudocode for Hanoi

• Call tree

Hanoi(n, src, dest, spare) if n==1 // base case

Move disk from src to dest else // recursive case

Hanoi(n-1, src, spare, dest) Move disk from src to dest Hanoi(n-1, spare, dest, src)

No need to combine the results in this case

Hanoi(3, A, C, B)

Hanoi(2, A, B, C) Hanoi(2, B, C, A)

(25)

### Algorithm Time Complexity

• 𝑇 𝑛 = #moves with n disks

• Base case: 𝑇 1 = 1

• Recursive case (𝑛 > 1): 𝑇 𝑛 = 2𝑇 𝑛 − 1 + 1

• We will learn how to derive 𝑇 𝑛 later

Hanoi(n, src, dest, spare) if n==1 // base case

Move disk from src to dest else // recursive case

Hanoi(n-1, src, spare, dest) Move disk from src to dest Hanoi(n-1, spare, dest, src)

(26)

### Further Questions

• Q1: Is 𝑂 2𝑛 tight for Hanoi? Can 𝑇 𝑛 < 2𝑛 − 1?

• Q2: What about more than 3 pegs?

• Q3: Double-color Hanoi problem

• Input: 2 interleaved-color towers

• Output: 2 same-color towers

(27)

## D&C #2: Merge Sort

Textbook Chapter 2.3.1 – The divide-and-conquer approach

(28)

### Sorting Problem

6

Input: unsorted list of size n

What are the base case and recursive case?

3 5 1 8 7 2 4

1 2 3 4 5 6 7 8

(29)

### Divide-and-Conquer

• Base case (n = 1)

• Directly output the list

• Recursive case (n > 1)

• Divide the list into two sub-lists

• Sort each sub-list recursively

• Merge the two sorted lists

1 3 5 6 2 4 7 8 2 sublists of size n/2

# of comparisons = Θ(𝑛)

How?

(30)

### Illustration for n = 10

6 3 5 1 8 9 7 2 10 4

6 3 5 1 8 9 7 2 10 4

6 3 5 1 8 9 7 2 10 4

6 3 5 1 8 9 7 2 10 4

6 3 9 7

(31)

6 3 5 1 8 9 7 2 10 4

6 3 5 1 8 9 7 2 10 4

6 3 5 1 8 9 7 2 10 4

6 3 5 1 8 9 7 2 10 4

6 3 9 7

6 3

3 5 6 1 8

6

3 5 8

1

7 9

2 7 9 4 10

10 9

4 7 2

9 10 8

7 6

5 3 4

1 2

(32)

### Pseudocode for Merge Sort

• Divide a list of size n into 2 sublists of size n/2

• Recursive case (𝑛 > 1)

• Sort 2 sublists recursively using merge sort

• Base case (𝑛 = 1)

• Return itself

• Merge 2 sorted sublists into one sorted list in linear time

MergeSort(A, p, r) // base case

if p == r ___return

// recursive case // divide

q = [(p+r-1)/2]

// conquer

MergeSort(A, p, q) MergeSort(A, q+1, r) // combine

Merge(A, p, q, r)

1. Divide

2. Conquer

3. Combine

(33)

### Time Complexity for Merge Sort

• Divide a list of size n into 2 sublists of size n/2

• Recursive case (𝑛 > 1)

• Sort 2 sublists recursively using merge sort

• Base case (𝑛 = 1)

• Return itself

• Merge 2 sorted sublists into one sorted list in linear time

MergeSort(A, p, r) // base case

if p == r ___return

// recursive case // divide

q = [(p+r-1)/2]

// conquer

MergeSort(A, p, q) MergeSort(A, q+1, r) // combine

Merge(A, p, q, r)

1. Divide

2. Conquer

3. Combine

𝑇 𝑛 = time for running MergeSort(A, p, r) with 𝑟– 𝑝 + 1 = 𝑛

(34)

### Time Complexity for Merge Sort

• Simplify recurrences

• Ignore floors and ceilings (boundary conditions)

• Assume base cases are constant (for small n)

2nd expansion 1st expansion

kth expansion

(35)

### Theorem 1

• Theorem

• Proof

• There exists positive constant 𝑎, 𝑏 s.t.

• Use induction to prove

• n = 1, trivial

• n > 1,

Inductive hypothesis

(36)

### 1. Substitution Method (取代法)

• Guess a bound and then prove by induction

### 2. Recursion-Tree Method (遞迴樹法)

• Expand the recurrence into a tree and sum up the cost

### 3. Master Method (套公式大法/大師法)

• Apply Master Theorem to a specific form of recurrences

(37)

## D&C #3: Bitonic Champion Problem

(38)

### Bitonic Champion Problem

The bitonic sequence means “increasing before the champion and decreasing after the champion” (冠軍之前遞增、冠軍之後遞減)

3 7 9 17 35 28 21 18 6 4

(39)

Why not Ω(n)?

Why?

(40)

### Bitonic Champion Problem Complexity

• When there are n inputs, any solution has n different outputs

• Any comparison-based algorithm needs Ω(log𝑛) time in the worst case

Ω(log 𝑛)

(41)

(42)

### Divide-and-Conquer

• Idea: divide A into two subproblems and then find the final champion based on the champions from two subproblems

Champion(i, j)

if i==j // base case return i

else // recursive case k = floor((i+j)/2) l = Champion(i, k) r = Champion(k+1, j) if A[l] > A[r]

return l

if A[l] < A[r]

return r

Output = Champion(1, n)

(43)

### Illustration for n = 10

3 7 9 17 35 28 21 18 6 4

3 7 9 17 35 28 21 18 6 4

3 7 9 17 35 28 21 18 6 4

3 7 9 17 35 28 21 18 6 4

3 7 28 21

(44)

### Proof of Correctness

• Practice by yourself!

Champion(i, j)

if i==j // base case return i

else // recursive case k = floor((i+j)/2) l = Champion(i, k) r = Champion(k+1, j) if A[l] > A[r]

return l

if A[l] < A[r]

return r

Output = Chamption(1, n)

Hint: use induction on (j – i) to prove Champion(i, j) can return the champion from A[i … j]

(45)

### Algorithm Time Complexity

• 𝑇 𝑛 = time for running Champion(i, j) with 𝑗 – 𝑖 + 1 = 𝑛

Divide a list of size n into 2 sublists of size n/2

Recursive case

Find champions from 2 sublists recursively

Base case

Return itself

Choose the final champion by a single comparison

1. Divide

2. Conquer

3. Combine

Champion(i, j)

if i==j // base case return i

else // recursive case k = floor((i+j)/2) l = Champion(i, k) r = Champion(k+1, j) if A[l] > A[r]

return l

if A[l] < A[r]

return r

Output = Chamption(1, n)

(46)

### Theorem 2

• Theorem

• Proof

• There exists positive constant 𝑎, 𝑏 s.t.

• Use induction to prove

• n = 1, trivial

• n > 1,

Inductive hypothesis

(47)

### Bitonic Champion Problem Complexity

Can we have a better algorithm by using the bitonic sequence property?

(48)

### Improved Algorithm

Champion-2(i, j)

if i==j // base case return i

else // recursive case k = floor((i+j)/2) if A[k] > A[k+1]

return Champion(i, k) if A[k] < A[k+1]

return Champion(k+1, j) Champion(i, j)

if i==j // base case return i

else // recursive case k = floor((i+j)/2) l = Champion(i, k) r = Champion(k+1, j) if A[l] > A[r]

return l

if A[l] < A[r]

return r

(49)

### Illustration for n = 10

3 7 9 17 35 28 21 18 6 4

3 7 9 17 35

17 35

35

(50)

### Correctness Proof

• Practice by yourself!

Champion-2(i, j)

if i==j // base case return i

else // recursive case k = floor((i+j)/2) if A[k] > A[k+1]

return Champion(i, k) if A[k] < A[k+1]

return Champion(k+1, j)

Output = Champion-2(1, n)

Two crucial observations:

• If 𝐴[1 … 𝑛] is bitonic, then so is 𝐴[𝑖, 𝑗] for any indices 𝑖 and 𝑗 with 1 ≤ 𝑖 ≤ 𝑗 ≤ 𝑛.

• For any indices 𝑖, 𝑗, and 𝑘 with 1 ≤ 𝑖 ≤ 𝑗 ≤ 𝑛, we know that 𝐴[𝑘] > 𝐴[𝑘 + 1] if and only if the maximum of 𝐴[𝑖 … 𝑗] lies in 𝐴[𝑖 … 𝑘].

(51)

### Algorithm Time Complexity

• 𝑇 𝑛 = time for running Champion-2(i, j) with 𝑗 – 𝑖 + 1 = 𝑛

Divide a list of size n into 2 sublists of size n/2

Recursive case

Find champions from 1 sublists recursively

Base case

Return itself

Return the champion

1. Divide

2. Conquer

3. Combine Champion-2(i, j)

if i==j // base case return i

else // recursive case k = floor((i+j)/2) if A[k] > A[k+1]

return Champion(i, k) if A[k] < A[k+1]

return Champion(k+1, j)

(52)

### Algorithm Time Complexity

The algorithm time complexity is 𝑂 log 𝑛

• each recursive call reduces the size of (j - i) into half

• there are 𝑂 log 𝑛 levels

• each level takes 𝑂 1

Champion-2(i, j)

if i==j // base case return i

else // recursive case k = floor((i+j)/2) if A[k] > A[k+1]

return Champion(i, k) if A[k] < A[k+1]

return Champion(k+1, j)

• 𝑇 𝑛 = time for running Champion-2(i, j) with 𝑗 – 𝑖 + 1 = 𝑛

(53)

### Theorem 3

• Theorem

• Proof Practice to prove by induction

(54)

(55)

## D&C #4: Maximum Subarray

Textbook Chapter 4.1 – The maximum-subarray problem

(56)

### Coding Efficiency

• How can we find the most efficient time interval for continuous coding?

5pm 6pm 7pm 8pm 9pm 10p m

11p m

12a m

1am 2am 3am

1 2 3 4

-4 -3 -2 -1 0

Coding power 戰鬥力 (K)

7pm-2:59am

Coding power= 8k

(57)

### Maximum Subarray Problem

3 7 9 17 5 28 21 18 6 4

-3 7 -9 17 -5 28 -21 18 -6 4

(58)

### O(n3) Brute Force Algorithm

MaxSubarray-1(i, j) for i = 1,…,n

for j = 1,…,n S[i][j] = - ∞ for i = 1,…,n

for j = i,i+1,…,n

S[i][j] = A[i] + A[i+1] + … + A[j]

return Champion(S)

(59)

### O(n2) Brute Force Algorithm

MaxSubarray-2(i, j) for i = 1,…,n

for j = 1,…,n S[i][j] = - ∞ R[0] = 0

for i = 1,…,n

R[i] = R[i-1] + A[i]

for i = 1,…,n

for j = i+1,i+2,…,n

S[i][j] = R[j] - R[i-1]

return Champion(S)

R[n] is the sum over A[1…n]

(60)

(61)

### Divide-and-Conquer

• Base case (n = 1)

• Return itself (maximum subarray)

• Recursive case (n > 1)

• Divide the array into two sub-arrays

• Find the maximum sub-array recursively

• Merge the results How?

(62)

### Where is the Solution?

• The maximum subarray for any input must be in one of following cases:

Case 1: left

Case 2: right

Case 3: cross the middle

Case 1: MaxSub(A, i, j) = MaxSub(A, i, k)

(63)

• Goal: find the maximum subarray that crosses the middle

• Observation

• The sum of 𝐴[𝑥 … 𝑘] must be the maximum among 𝐴[𝑖 … 𝑘] (left: 𝑖 ≤ 𝑘)

• The sum of 𝐴[𝑘 + 1 … 𝑦] must be the maximum among 𝐴[𝑘 + 1 … 𝑗] (right: 𝑗 > 𝑘)

• Solvable in linear time → Θ 𝑛

### Case 3: Cross the Middle

(1) Start from the middle to find the left maximum subarray

(2) Start from the middle to find the right maximum subarray

The solution of Case 3 is the combination of (1) and (2)

(64)

### Divide-and-Conquer Algorithm

MaxCrossSubarray(A, i, k, j) left_sum = -∞

sum=0

for p = k downto i sum = sum + A[p]

if sum > left_sum left_sum = sum max_left = p right_sum = -∞

sum=0

for q = k+1 to j sum = sum + A[q]

if sum > right_sum right_sum = sum max_right = q

(65)

Combine

Conquer Divide

### Divide-and-Conquer Algorithm

MaxSubarray(A, i, j)

if i == j // base case return (i, j, A[i]) else // recursive case

k = floor((i + j) / 2)

(l_low, l_high, l_sum) = MaxSubarray(A, i, k) (r_low, r_high, r_sum) = MaxSubarray(A, k+1, j)

(c_low, c_high, c_sum) = MaxCrossSubarray(A, i, k, j) if l_sum >= r_sum and l_sum >= c_sum // case 1

return (l_low, l_high, l_sum)

else if r_sum >= l_sum and r_sum >= c_sum // case 2 return (r_low, r_high, r_sum)

else // case 3

return (c_low, c_high, c_sum)

(66)

### Divide-and-Conquer Algorithm

MaxSubarray(A, i, j)

if i == j // base case return (i, j, A[i]) else // recursive case

k = floor((i + j) / 2)

(l_low, l_high, l_sum) = MaxSubarray(A, i, k) (r_low, r_high, r_sum) = MaxSubarray(A, k+1, j)

(c_low, c_high, c_sum) = MaxCrossSubarray(A, i, k, j) if l_sum >= r_sum and l_sum >= c_sum // case 1

return (l_low, l_high, l_sum)

else if r_sum >= l_sum and r_sum >= c_sum // case 2 return (r_low, r_high, r_sum)

else // case 3

return (c_low, c_high, c_sum)

(67)

### Algorithm Time Complexity

• Divide a list of size n into 2 subarrays of size n/2

• Recursive case (𝑛 > 1)

• find MaxSub for each subarrays

• Base case (𝑛 = 1)

• Return itself

• Find MaxCrossSub for the original list

• Pick the subarray with the maximum sum among 3 subarrays

1. Divide

2. Conquer

3. Combine

𝑇 𝑛 = time for running MaxSubarray(A, i, j) with 𝑗 – 𝑖 + 1 = 𝑛

(68)

### Theorem 1

• Theorem

• Proof

• There exists positive constant 𝑎, 𝑏 s.t.

• Use induction to prove

• n = 1, trivial

• n > 1,

Inductive hypothesis

(69)

### Theorem 1 (Simplified)

• Theorem

• Proof

• There exists positive constant 𝑎, 𝑏 s.t.

• Use induction to prove

• n = 1, trivial

• n > 1,

Inductive hypothesis

(70)

(71)

### Max Subarray Problem Complexity

Exercise 4.1-5 page 75 of textbook

Next topic!

(72)

(73)

## Question?

Important announcement will be sent to

@ntu.edu.tw mailbox & post to the course website

• A concrete problem is polynomial-time solvable if there exists an algorithm that solves any concrete instance of length n in time for some constant k. • Solvable = can produce

• In the worst case, what is the growth of the function the optimal algorithm of the problem takes Design Step.

Textbook Chapter 4.3 – The substitution method for solving recurrences Textbook Chapter 4.4 – The recursion-tree method for solving recurrences Textbook Chapter 4.5 – The master

• “Greedy”: always makes the choice that looks best at the moment in the hope that this choice will lead to a globally optimal solution. • When to

✓ Express the solution of the original problem in terms of optimal solutions for subproblems. Construct an optimal solution from

• Algorithmic design methods to solve problems efficiently (polynomial time).. • Divide

• makes a locally optimal choice in the hope that this choice will lead to a globally optimal solution.. • not always yield optimal solution; may end up at

Textbook Chapter 33.4 – Finding the closest pair of points.. Closest Pair of

✓ Combining an optimal solution to the subproblem via greedy can arrive an optimal solution to the original problem. Prove that there is always an optimal solution to the

✓ Combining an optimal solution to the subproblem via greedy can arrive an optimal solution to the original problem.. Prove that there is always an optimal solution to the

Textbook Chapter 4.3 – The substitution method for solving recurrences Textbook Chapter 4.4 – The recursion-tree method for solving recurrences Textbook Chapter 4.5 – The master

Calculate the amortized cost of each operation based on the potential function 4. Calculate total amortized cost based on

jobs

 This technique is solving a problem by reducing its instance to a smaller one, solving the latte r, and then extending the obtained solution to g et a solution to the

✓ Combining an optimal solution to the subproblem via greedy can arrive an optimal solution to the original problem.. Prove that there is always an optimal solution to the

 Combine: find closet pair with one point in each region, and return the best of three

✓ Express the solution of the original problem in terms of optimal solutions for subproblems. Construct an optimal solution from

✓ Express the solution of the original problem in terms of optimal solutions for subproblems.. Construct an optimal solution from

 “More Joel on Software : Further Thoughts on Diverse and Occasionally Related Matters That Will Prove of Interest to Software Developers, Designers, and Managers, and to Those

 Combining an optimal solution to the subproblem via greedy can arrive an optimal solution to the original problem. Prove that there is always an optimal solution to the

We will design a simple and effective iterative algorithm to implement “The parallel Tower of Hanoi problem”, and prove its moving is an optimal

The proposed algorithms use the optimal-searching technique of genetic algorithm (GA) to get an efficient scheduling solution in grid computing environment and adapt to

Based on the insertion of redundant wires and the analysis of the clock skew in a clock tree, an efficient OPE-aware algorithm is proposed to repair the zero-skew