Slides credited from Hsueh-I Lu, Hsu-Chun Hsiao, & Michael Tsai

Slides credited from Hsueh-I Lu, Hsu-Chun Hsiao, & Michael Tsai



Mini-HW 3 released

 Due on 10/12 (Thu) 17:20

 Print out the A4 hard copy and submit before the lecture finishes



Homework 1 released

 Due on 10/19 (Thur) 17:20 (2 weeks left)

 Writing: print out the A4 hard copy and submit before the lecture finishes

 Programming: submit to Online Judge – http://ada-judge.csie.org



Mid-term date changed

 Original: 11/09 (Thu)

 New: 11/16 (Thu)

3

 Recurrence (遞迴)

 Divide-and-Conquer

 D&C #1: Tower of Hanoi (河內塔)

 D&C #2: Merge Sort

 D&C #3: Bitonic Champion

 D&C #4: Maximum Subarray

 Solving Recurrences

 Substitution Method

 Recursion-Tree Method

 Master Method

Divide-and-Conquer 首部曲

 Solve a problem recursively

 Apply three steps at each level of the recursion

1.

Divide the problem into a number of subproblems that are smaller instances of the same problem (比較小的同樣問題)

2.

Conquer the subproblems by solving them recursively If the subproblem sizes are small enough

 then solve the subproblems

 else recursively solve itself

3.

Combine the solutions to the subproblems into the solution for the original problem

5

base case recursive case

Textbook Chapter 4.3 – The substitution method for solving recurrences

6



𝑇 𝑛 : running time for input size 𝑛



𝐷 𝑛 : time of Divide for input size 𝑛



𝐶 𝑛 : time of Combine for input size 𝑛



𝑎: number of subproblems



𝑛/𝑏: size of each subproblem

7

1. Substitution Method (取代法)



Guess a bound and then prove by induction

2. Recursion-Tree Method (遞迴樹法)



Expand the recurrence into a tree and sum up the cost 3. Master Method (套公式大法/大師法)



Apply Master Theorem to a specific form of recurrences

 Useful simplification tricks

Textbook Chapter 4.3 – The substitution method for solving recurrences

9



Time Complexity for Merge Sort



Theorem



Proof

 There exists positive constant 𝑎, 𝑏 s.t.

 Use induction to prove

 n = 1, trivial

 n > 1, Substitution Method (取代法)

 Guess the form of the solution

 Verify by mathematical induction (數學歸納法)



Prove it works for 𝑛 = 1



Prove that if it works for 𝑛 = 𝑚, then it works for 𝑛 = 𝑚 + 1

 It can work for all positive integer 𝑛

 Solve constants to show that the solution works

 Prove 𝑂 and Ω separately

11

1. Guess

2. Verify

3. Solve



Proof



There exists positive constants 𝑛₀, 𝑐 s.t. for all 𝑛 ≥ 𝑛₀,

 Use induction to find the constants 𝑛₀, 𝑐

 n = 1, trivial

 n > 1,

Inductive hypothesis

Guess Verify



Proof



There exists positive constants 𝑛₀, 𝑐 s.t. for all 𝑛 ≥ 𝑛₀,

 Use induction to find the constants 𝑛₀, 𝑐

 n = 1, trivial

 n > 1,

13

Inductive hypothesis

Tighter upper bound?

証不出來…

猜錯了？還是推導錯了？

沒猜錯 推導也沒錯 這是取代法的小盲點



Proof



There exists positive constants 𝑛₀, 𝑐₁, 𝑐₂ s.t. for all 𝑛 ≥ 𝑛₀,

 Use induction to find the constants 𝑛₀,𝑐₁, 𝑐₂

 n = 1, holds for

 n > 1,

Inductive hypothesis

Guess

Verify

Strengthen the inductive hypothesis by subtracting a low-order term

 Guess based on seen recurrences

 Use the recursion-tree method

 From loose bound to tight bound

 Strengthen the inductive hypothesis by subtracting a low- order term

 Change variables



E.g.,

1.

Change variable:

2.

Change variable again:

3.

Solve recurrence

15

Textbook Chapter 4.4 – The recursion-tree method for solving recurrences

16



Time Complexity for Merge Sort



Theorem



Proof

17

2^nd expansion 1^st expansion

k^th expansion

The expansion stops when 2^𝑘 = 𝑛

Recursion-Tree Method (遞迴樹法)

Expand the recurrence into a tree and sum up the cost

 Expand a recurrence into a tree

 Sum up the cost of all nodes as a good guess

 Verify the guess as in the substitution method

 Advantages



Promote intuition



Generate good guesses for the substitution method

1. Expand

2. Sumup

3. Verify

19

21

+

Textbook Chapter 4.5 – The master method for solving recurrences

23

divide a problem of size 𝑛 into 𝑎 subproblems each of size ^𝑛_𝑏 is solved in time 𝑇 ^𝑛_𝑏 recursively

The proof is in Ch. 4.6

Should follow this format

25

+

𝑎 𝑎





𝑎 ≥ 1, the number of subproblems



𝑏 > 1, the factor by which the subproblem size decreases



𝑓(𝑛) = work to divide/combine subproblems

 Compare 𝑓 𝑛 with 𝑛

1.

Case 1: 𝑓 𝑛 grows polynomially slower than 𝑛

2.

Case 2: 𝑓 𝑛 and 𝑛

grow at similar rates

27

𝑎 𝑎

𝑓 𝑛 grows polynomially slower than 𝑛^{log𝑏 𝑎}

29

𝑎 𝑎

𝑓 𝑛 and 𝑛^{log𝑏 𝑎} grow at similar rates

31

𝑎 𝑎

𝑓 𝑛 grows polynomially faster than 𝑛^{log𝑏 𝑎}

compare 𝑓 𝑛 with 𝑛^log𝑏𝑎 33

divide a problem of size 𝑛 into 𝑎 subproblems each of size ^𝑛_𝑏 is solved in time 𝑇 ^𝑛_𝑏 recursively

The proof is in Ch. 4.6

compare 𝑓 𝑛 with 𝑛^{log𝑏 𝑎}

 Master theorem can be extended to recurrences with floors and ceilings

 The proof is in the Ch. 4.6

35



Case 2



Case 1

37



Case 2

Textbook Chapter 4.2 – Strassen’s algorithm for matrix multiplication

39

41



Each entry takes 𝑛 multiplications



There are total 𝑛

entries

A B C



We can assume that 𝑛 = 2

for simplicity

 Otherwise, we can increase 𝑛 s.t. 𝑛 = 2 ^log2𝑛

 𝑛 may not be twice large as the original in this modification

43

A₁₁ A₁₂

A₂₁ A₂₂

B₁₁ B₁₂

B₂₁ B₂₂

C₁₁ C₁₂

C₂₁ C₂₂

Combine

Conquer Divide

MatrixMultiply(n, A, B) //base case

if n == 1 __ _return AB

//recursive case

Divide A and B into n/2 by n/2 submatrices

C₁₁ = MatrixMultiply(n/2,A₁₁,B₁₁) + MatrixMultiply(n/2,A₁₂,B₂₁) C₂₁ = MatrixMultiply(n/2,A₁₁,B₁₂) + MatrixMultiply(n/2,A₁₂,B₂₂) C₂₁ = MatrixMultiply(n/2,A₂₁,B₁₁) + MatrixMultiply(n/2,A₂₂,B₂₁) C₂₂ = MatrixMultiply(n/2,A₂₁,B₁₂) + MatrixMultiply(n/2,A₂₂,B₂₂) return C



Important theoretical breakthrough by Volker Strassen in 1969



Reduces the running time from Θ(𝑛

) to Θ(𝑛

) ≈ Θ(𝑛

)



The key idea is to reduce the number of recursive calls

 From 8 recursive calls to 7 recursive calls

 At the cost of extra addition and subtraction operations

45

4 multiplications 3 additions

1 multiplication 2 additions

Intuition:



𝐶 = 𝐴 × 𝐵

2 + 1×

1 + 1×

1 − 1×

1 − 1×

1 + 1×

2 + 1 − 1 + 1 + 2 + 1 −



Practice

47

Conquer

Divide

Strassen(n, A, B) // base case if n == 1 ___ return AB

// recursive case

Divide A and B into n/2 by n/2 submatrices M₁ = Strassen(n/2, A₁₁+A₂₂, B₁₁+B₂₂)

M₂ = Strassen(n/2, A₂₁+A₂₂, B₁₁) M₃ = Strassen(n/2, A₁₁, B₁₂-B₂₂) M₄ = Strassen(n/2, A₂₂, B₂₁-B₁₁) M₅ = Strassen(n/2, A₁₁+A₁₂, B₂₂) M₆ = Strassen(n/2, A₁₁-A₂₁, B₁₁+B₁₂) M₇ = Strassen(n/2, A₁₂-A₂₂, B₂₁+B₂₂) C = M + M - M + M

 𝑇 𝑛 = time for running Strassen(n,A,B)

 Disadvantages

1.

Larger constant factor than it in the naïve approach

2.

Less numerical stable than the naïve approach

 Larger errors accumulate in non-integer computation due to limited precision 3.

The submatrices at the levels of recursion consume space

4.

Faster algorithms exist for sparse matrices

 Advantages: find the crossover point and combine two subproblems

49



Each algorithm gives an upper bound

51

Textbook Chapter 9.3 – Selection in worst-case linear time

52

53

3 7 9 17 5 2 21 18 33 4



If the sorting problem can be solved in 𝑂 𝑓 𝑛 , so can the selection problem based on the algorithm design

 Step 1: sort A into increasing order

 Step 2: output 𝐴[𝑛 − 𝑘 + 1]

55

Can we make the upper bound better if we do not sort them?

57



Upper bounds in terms of #comparisons

 3n + o(n) by Schonhage, Paterson, and Pippenger (JCSS 1975).

 2.95n by Dor and Zwick (SODA 1995, SIAM Journal on Computing 1999).



Lower bounds in terms of #comparisons

 2n+o(n) by Bent and John (STOC 1985)

 (2+2^-80)n by Dor and Zwick (FOCS 1996, SIAM Journal on Discrete Math 2001).



Idea

 Select a pivot and divide the inputs into two subproblems

 If 𝑘 ≤ 𝑋_> , we find the 𝑘-th largest

 If 𝑘 > 𝑋_> , we find the 𝑘 − 𝑋_> -th largest pivot

a

59

61

small number  large number

Larger than MoM Smaller than MoM

MoM



Three cases

1. If 𝑘 ≤ 𝑋_> , then output the 𝑘-th largest number in 𝑋_>

2. If 𝑘 = 𝑋_> + 1, then output MoM

3. If 𝑘 > 𝑋_> + 1, then output the 𝑘 − 𝑋_> − 1 -th largest number in 𝑋_<



Practice to prove by induction

63

Smaller than MoM Larger than MoM MoM



Step (2): Determining MoM



Step (5): Selection in X

or X

65

Selection(X, k) // base case if |X| <= 4

__ sort X and return X[k]

// recursive case

Divide X into |X|/5 groups with size 5 M[i] = median from group i

MoM = Selection(M, |M|/2) for i = 1 … |X|

if X[i] > MoM

insert X[i] into X2 else

insert X[i] into X1 if |X2| == k – 1

return x

if |X2| > k – 1

return Selection(X2, k)

return Selection(X1, k - |X2| - 1)

delete delete

 𝑇 𝑛 = time for running Selection(X, k) with |X| = n

 Intuition

67



Theorem



Proof

 There exists positive constant 𝑎, 𝑏 s.t.

 Use induction to prove

 n = 1, 𝑎 > 𝑐

 n > 1,

Inductive

69

Textbook Chapter 33.4 – Finding the closest pair of points

70



Input: 𝑛 ≥ 2 points, where 𝑝

= 𝑥

, 𝑦

for 0 ≤ 𝑖 < 𝑛



Output: two points 𝑝

and 𝑝

that are closest

 “Closest”: smallest Euclidean distance

 Euclidean distance between 𝑝_𝑖 and 𝑝_𝑗:

71



Brute-force algorithm

 Check all pairs of points:

Θ 𝐶₂^𝑛 = Θ 𝑛²



1D:

 Sort all points

 Scan the sorted points to find the closest pair in one pass

 We only need to examine the adjacent points



2D:



Divide: divide points evenly along x-coordinate



Conquer: find closest pair in each region recursively



Combine: find closet pair with one point in each region, and return the best of three solutions

73

left-min = 10

right-min = 13 cross-min = 7



Algo 1: check all pairs that cross two regions  𝑛/2 × 𝑛/2 combinations



Algo 2: only consider points within 𝛿 of the cut, 𝛿 = min{l−min, r−min}

 Other pairs of points must have distance larger than 𝛿

left-min = 10

right-min = 13 cross-min = 7

𝛿 𝛿

縮小搜尋範圍!



Algo 1: check all pairs that cross two regions  𝑛/2 × 𝑛/2 combinations



Algo 2: only consider points within 𝛿 of the cut, 𝛿 = min{l−min, r−min}



Algo 3: only consider pairs within 𝛿 × 2𝛿 blocks

 Obs 1: every pair with smaller than 𝛿 distance must appear in a 𝛿 × 2𝛿 block

75

要是很倒霉，所有的 點都聚集在某個𝛿 ×

2𝛿區塊內怎麼辦

縮小搜尋範圍!



Algo 1: check all pairs that cross two regions  𝑛/2 × 𝑛/2 combinations



Algo 2: only consider points within 𝛿 of the cut, 𝛿 = min{l−min, r−min}



Algo 3: only consider pairs within 𝛿 × 2𝛿 blocks

 Obs 1: every pair with smaller than 𝛿 distance must appear in a 𝛿 × 2𝛿 block

 Obs 2: there are at most 8 points in a 𝛿 × 2𝛿 block

 Each 𝛿/2 × 𝛿/2 block contains at most 1 point, otherwise the distance returned from left/right region should be smaller than 𝛿



Algo 1: check all pairs that cross two regions  𝑛/2 × 𝑛/2 combinations



Algo 2: only consider points within 𝛿 of the cut, 𝛿 = min{l−min, r−min}



Algo 3: only consider pairs within 𝛿 × 2𝛿 blocks

 Obs 1: every pair with smaller than 𝛿 distance must appear in a 𝛿 × 2𝛿 block

 Obs 2: there are at most 8 points in a 𝛿 × 2𝛿 block

p_i 77

p_i+4

p_i+2 p_i+5

p_i+3

Find-closet-pair-across-regions 1. Sort the points by y-values within 𝛿 of the

cut (yellow region)

2. For the sorted point 𝑝_𝑖, compute the distance with 𝑝_𝑖+1, 𝑝_𝑖+2, …, 𝑝_𝑖+7

3. Return the smallest one

At most 7 distance calculations needed

Closest-Pair(P)

// termination condition (base case)

if |P| <= 3 brute-force finding closest pair and return it // Divide

find a vertical line L s.t. both planes_contain half of the points // Conquer (by recursion)

left-pair, left-min = Closest-Pair(points in the left) right-pair, right-min = Closest-Pair(points in the right) // Combine

delta = min{left-min, right-min}

remove points that are delta or more away from L // Obs 1 sort remaining points by y-coordinate into p₀, …, p_k

for point p_i:

____compute distances with p_i+1, p_i+2, …, p_i+7_// Obs 2 ____update delta if a closer pair is found

return the closest pair and its distance



Idea: do not sort inside the recursive case

79

Closest-Pair(P)

sort P by x- and y-coordinate and store in Px and Py // termination condition (base case)

if |P| <= 3 brute-force finding closest pair and return it // Divide

find a vertical line L s.t. both planes_contain half of the points // Conquer (by recursion)

left-pair, left-min = Closest-Pair(points in the left) right-pair, right-min = Closest-Pair(points in the right) // Combine

delta = min{left-min, right-min}

remove points that are delta or more away from L // Obs 1 for point p_i in sorted candidates

____compute distances with p_i+1, p_i+2, …, p_i+7_// Obs 2 ____update delta if a closer pair is found

return the closest pair and its distance

 𝑂(𝑛) algorithm



Taking advantage of randomization

 Chapter 13.7 of Algorithm Design by Kleinberg & Tardos

 Samir Khuller and Yossi Matias. 1995. A simple randomized sieve

algorithm for the closest-pair problem. Inf. Comput. 118, 1 (April 1995), 34-37.

 When to use D&C



Whether the problem with small inputs can be solved directly



Whether subproblem solutions can be combined into the original solution



Whether the overall complexity is better than naïve

 Note



Try different ways of dividing



D&C may be suboptimal due to repetitive computations



Example.

 D&C algo for Fibonacci:

 Bottom-up algo for Fibonacci:

81

1. Divide

2. Conquer

3. Combine

Fibonacci(n) if n < 2 ____return 1

a[0]=1 a[1]=1

for i = 2 … n

____a[i]=a[i-1]+a[i-2]

return a[n]

Our next topic: Dynamic Programming

“a technique for solving problems with overlapping subproblems”

Course Website: http://ada17.csie.org

82

Important announcement will be sent to @ntu.edu.tw mailbox

& post to the course website