Slides credited from Hsueh-I Lu, Hsu-Chun Hsiao, & Michael Tsai

### Mini-HW 3 released

Due on 10/12 (Thu) 17:20

Print out the A4 hard copy and submit before the lecture finishes

### Homework 1 released

Due on 10/19 (Thur) 17:20 (2 weeks left)

Writing: print out the A4 hard copy and submit before the lecture finishes

Programming: submit to Online Judge – http://ada-judge.csie.org

### Mid-term date changed

Original: 11/09 (Thu)

New: 11/16 (Thu)

**3**

Recurrence (遞迴)

Divide-and-Conquer

D&C #1: Tower of Hanoi (河內塔)

D&C #2: Merge Sort

D&C #3: Bitonic Champion

D&C #4: Maximum Subarray

Solving Recurrences

Substitution Method

Recursion-Tree Method

Master Method

Divide-and-Conquer 首部曲

### Solve a problem recursively

### Apply three steps at each level of the recursion

**1.**

**Divide the problem into a number of subproblems that are ** smaller instances of the same problem (比較小的同樣問題)

**2.**

**Conquer the subproblems by solving them recursively** *If the subproblem sizes are small enough*

then solve the subproblems

else recursively solve itself

**3.**

**Combine** the solutions to the subproblems into the solution for the original problem

**5**

base case recursive case

Textbook Chapter 4.3 – The substitution method for solving recurrences

**6**

### 𝑇 𝑛 : running time for input size 𝑛

### 𝐷 𝑛 : time of Divide for input size 𝑛

### 𝐶 𝑛 : time of Combine for input size 𝑛

### 𝑎: number of subproblems

### 𝑛/𝑏: size of each subproblem

**7**

**1.** **Substitution Method (取代法)**

### Guess a bound and then prove by induction

**2.** **Recursion-Tree Method (遞迴樹法)**

### Expand the recurrence into a tree and sum up the cost **3.** **Master Method (套公式大法/大師法)**

### Apply Master Theorem to a specific form of recurrences

### Useful simplification tricks

Textbook Chapter 4.3 – The substitution method for solving recurrences

**9**

### Time Complexity for Merge Sort

### Theorem

### Proof

There exists positive constant 𝑎, 𝑏 s.t.

Use induction to prove

n = 1, trivial

n > 1, **Substitution Method (取代法)**

### Guess the form of the solution

### Verify by mathematical induction (數學歸納法)

### Prove it works for 𝑛 = 1

### Prove that if it works for 𝑛 = 𝑚, then it works for 𝑛 = 𝑚 + 1

### It can work for all positive integer 𝑛

### Solve constants to show that the solution works

### Prove 𝑂 and Ω separately

**11**

1. Guess

2. Verify

3. Solve

### Proof

There exists positive constants 𝑛_{0}, 𝑐 s.t. for all 𝑛 ≥ 𝑛_{0},

Use induction to find the constants 𝑛_{0}, 𝑐

n = 1, trivial

n > 1,

Inductive hypothesis

Guess Verify

### Proof

There exists positive constants 𝑛_{0}, 𝑐 s.t. for all 𝑛 ≥ 𝑛_{0},

Use induction to find the constants 𝑛_{0}, 𝑐

n = 1, trivial

n > 1,

**13**

Inductive hypothesis

Tighter upper bound?

証不出來…

猜錯了？還是推導錯了？

沒猜錯 推導也沒錯 這是取代法的小盲點

### Proof

There exists positive constants 𝑛_{0}, 𝑐_{1}, 𝑐_{2} s.t. for all 𝑛 ≥ 𝑛_{0},

Use induction to find the constants 𝑛_{0},𝑐_{1}, 𝑐_{2}

n = 1, holds for

n > 1,

Inductive hypothesis

Guess

Verify

Strengthen the inductive hypothesis by subtracting a low-order term

### Guess based on seen recurrences

### Use the recursion-tree method

### From loose bound to tight bound

### Strengthen the inductive hypothesis by subtracting a low- order term

### Change variables

### E.g.,

1.

### Change variable:

2.

### Change variable again:

3.

### Solve recurrence

**15**

Textbook Chapter 4.4 – The recursion-tree method for solving recurrences

**16**

### Time Complexity for Merge Sort

### Theorem

### Proof

**17**

2^{nd} expansion
1^{st} expansion

k^{th} expansion

The expansion stops when 2^{𝑘} = 𝑛

**Recursion-Tree Method (遞迴樹法)**

Expand the recurrence into a tree and sum up the cost

### Expand a recurrence into a tree

### Sum up the cost of all nodes as a good guess

### Verify the guess as in the substitution method

### Advantages

### Promote intuition

### Generate good guesses for the substitution method

1. Expand

2. Sumup

3. Verify

**19**

**21**

**+**

Textbook Chapter 4.5 – The master method for solving recurrences

**23**

divide a problem of size 𝑛 into 𝑎 subproblems
each of size ^{𝑛}_{𝑏} is solved in time 𝑇 ^{𝑛}_{𝑏} recursively

The proof is in Ch. 4.6

Should follow this format

**25**

**+**

𝑎 𝑎

###

### 𝑎 ≥ 1, the number of subproblems

### 𝑏 > 1, the factor by which the subproblem size decreases

### 𝑓(𝑛) = work to divide/combine subproblems

### Compare 𝑓 𝑛 with 𝑛

^{log}

^{𝑏}

^{𝑎}

1.

### Case 1: 𝑓 𝑛 grows polynomially slower than 𝑛

^{log}

^{𝑏}

^{𝑎}

2.

### Case 2: 𝑓 𝑛 and 𝑛

^{log}

^{𝑏}

^{𝑎}

### grow at similar rates

**27**

𝑎 𝑎

𝑓 𝑛 grows polynomially slower than 𝑛^{log}^{𝑏} ^{𝑎}

**29**

𝑎 𝑎

𝑓 𝑛 and 𝑛^{log}^{𝑏} ^{𝑎} grow at similar rates

**31**

𝑎 𝑎

𝑓 𝑛 grows polynomially faster than 𝑛^{log}^{𝑏} ^{𝑎}

compare 𝑓 𝑛 with 𝑛^{log}^{𝑏}^{𝑎} **33**

divide a problem of size 𝑛 into 𝑎 subproblems
each of size ^{𝑛}_{𝑏} is solved in time 𝑇 ^{𝑛}_{𝑏} recursively

The proof is in Ch. 4.6

compare 𝑓 𝑛 with 𝑛^{log}^{𝑏} ^{𝑎}

### Master theorem can be extended to recurrences with floors and ceilings

### The proof is in the Ch. 4.6

**35**

### Case 2

### Case 1

**37**

### Case 2

Textbook Chapter 4.2 – Strassen’s algorithm for matrix multiplication

**39**

**41**

### Each entry takes 𝑛 multiplications

### There are total 𝑛

^{2}

### entries

*A* *B* *C*

### We can assume that 𝑛 = 2

^{𝑘}

### for simplicity

Otherwise, we can increase 𝑛 s.t. 𝑛 = 2 ^{log}^{2}^{𝑛}

𝑛 may not be twice large as the original in this modification

**43**

A_{11} A_{12}

A_{21} A_{22}

B_{11} B_{12}

B_{21} B_{22}

C_{11} C_{12}

C_{21} C_{22}

Combine

Conquer Divide

MatrixMultiply(n, A, B) //base case

if n == 1 __ _return AB

//recursive case

Divide A and B into n/2 by n/2 submatrices

C_{11} = MatrixMultiply(n/2,A_{11},B_{11}) + MatrixMultiply(n/2,A_{12},B_{21})
C_{21} = MatrixMultiply(n/2,A_{11},B_{12}) + MatrixMultiply(n/2,A_{12},B_{22})
C_{21} = MatrixMultiply(n/2,A_{21},B_{11}) + MatrixMultiply(n/2,A_{22},B_{21})
C_{22} = MatrixMultiply(n/2,A_{21},B_{12}) + MatrixMultiply(n/2,A_{22},B_{22})
return C

### Important theoretical breakthrough by Volker Strassen in 1969

### Reduces the running time from Θ(𝑛

^{3}

### ) to Θ(𝑛

^{𝑙𝑜𝑔}

^{2}

^{7}

### ) ≈ Θ(𝑛

^{2.807}

### )

### The key idea is to reduce the number of recursive calls

From 8 recursive calls to 7 recursive calls

At the cost of extra addition and subtraction operations

**45**

4 multiplications 3 additions

1 multiplication 2 additions

**Intuition:**

### 𝐶 = 𝐴 × 𝐵

**2 + 1×**

**1 + 1×**

**1 − 1×**

**1 − 1×**

**1 + 1×**

**2 + 1 −**
**1 +**
**1 +**
**2 + 1 −**

### Practice

**47**

Conquer

Divide

Strassen(n, A, B) // base case if n == 1 ___ return AB

// recursive case

Divide A and B into n/2 by n/2 submatrices
M_{1} = Strassen(n/2, A_{11}+A_{22}, B_{11}+B_{22})

M_{2} = Strassen(n/2, A_{21}+A_{22}, B_{11})
M_{3} = Strassen(n/2, A_{11}, B_{12}-B_{22})
M_{4} = Strassen(n/2, A_{22}, B_{21}-B_{11})
M_{5} = Strassen(n/2, A_{11}+A_{12}, B_{22})
M_{6} = Strassen(n/2, A_{11}-A_{21}, B_{11}+B_{12})
M_{7} = Strassen(n/2, A_{12}-A_{22}, B_{21}+B_{22})
C = M + M - M + M

𝑇 𝑛 = time for running Strassen(n,A,B)

### Disadvantages

1.

### Larger constant factor than it in the naïve approach

2.

### Less numerical stable than the naïve approach

Larger errors accumulate in non-integer computation due to limited precision 3.

### The submatrices at the levels of recursion consume space

4.

### Faster algorithms exist for sparse matrices

### Advantages: find the crossover point and combine two subproblems

**49**

### Each algorithm gives an upper bound

**51**

Textbook Chapter 9.3 – Selection in worst-case linear time

**52**

**53**

3 7 9 17 5 2 21 18 33 4

### If the sorting problem can be solved in 𝑂 𝑓 𝑛 , so can the selection problem based on the algorithm design

Step 1: sort A into increasing order

Step 2: output 𝐴[𝑛 − 𝑘 + 1]

**55**

Can we make the upper bound better if we do not sort them?

**57**

### Upper bounds in terms of #comparisons

*3n + o(n) by Schonhage, Paterson, and Pippenger (JCSS 1975).*

*2.95n by Dor and Zwick (SODA 1995, SIAM Journal on Computing 1999).*

### Lower bounds in terms of #comparisons

*2n+o(n) by Bent and John (STOC 1985)*

(2+2^{-80}*)n by Dor and Zwick (FOCS 1996, SIAM Journal on Discrete Math 2001).*

### Idea

Select a pivot and divide the inputs into two subproblems

If 𝑘 ≤ 𝑋_{>} , we find the 𝑘-th largest

If 𝑘 > 𝑋_{>} , we find the 𝑘 − 𝑋_{>} -th largest
pivot

a

**59**

**61**

small number large number

Larger than MoM Smaller than MoM

### MoM

### Three cases

1. If 𝑘 ≤ 𝑋_{>} , then output the 𝑘-th largest number in 𝑋_{>}

2. If 𝑘 = 𝑋_{>} + 1, then output MoM

3. If 𝑘 > 𝑋_{>} + 1, then output the 𝑘 − 𝑋_{>} − 1 -th largest number in 𝑋_{<}

### Practice to prove by induction

**63**

Smaller than MoM Larger than MoM MoM

### Step (2): Determining MoM

*Step (5): Selection in X*

_{<}

*or X*

_{>}

**65**

Selection(X, k) // base case if |X| <= 4

__ sort X and return X[k]

// recursive case

Divide X into |X|/5 groups with size 5 M[i] = median from group i

MoM = Selection(M, |M|/2) for i = 1 … |X|

if X[i] > MoM

insert X[i] into X2 else

insert X[i] into X1 if |X2| == k – 1

return x

if |X2| > k – 1

return Selection(X2, k)

return Selection(X1, k - |X2| - 1)

delete delete

𝑇 𝑛 = time for running Selection(X, k) with |X| = n

Intuition

**67**

### Theorem

### Proof

There exists positive constant 𝑎, 𝑏 s.t.

Use induction to prove

n = 1, 𝑎 > 𝑐

n > 1,

Inductive

**69**

Textbook Chapter 33.4 – Finding the closest pair of points

**70**

### Input: 𝑛 ≥ 2 points, where 𝑝

_{𝑖}

### = 𝑥

_{𝑖}

### , 𝑦

_{𝑖}

### for 0 ≤ 𝑖 < 𝑛

### Output: two points 𝑝

_{𝑖}

### and 𝑝

_{𝑗}

### that are closest

“Closest”: smallest Euclidean distance

Euclidean distance between 𝑝_{𝑖} and 𝑝_{𝑗}:

**71**

### Brute-force algorithm

Check all pairs of points:

Θ 𝐶_{2}^{𝑛} = Θ 𝑛^{2}

### 1D:

Sort all points

Scan the sorted points to find the closest pair in one pass

We only need to examine the adjacent points

### 2D:

### Divide: divide points evenly along x-coordinate

### Conquer: find closest pair in each region recursively

### Combine: find closet pair with one point in each region, and return the best of three solutions

**73**

left-min = 10

right-min = 13 cross-min = 7

### Algo 1: check all pairs that cross two regions 𝑛/2 × 𝑛/2 combinations

### Algo 2: only consider points within 𝛿 of the cut, 𝛿 = min{l−min, r−min}

Other pairs of points must have distance larger than 𝛿

left-min = 10

right-min = 13 cross-min = 7

𝛿 𝛿

### 縮小搜尋範圍!

### Algo 1: check all pairs that cross two regions 𝑛/2 × 𝑛/2 combinations

### Algo 2: only consider points within 𝛿 of the cut, 𝛿 = min{l−min, r−min}

### Algo 3: only consider pairs within 𝛿 × 2𝛿 blocks

Obs 1: every pair with smaller than 𝛿 distance must appear in a 𝛿 × 2𝛿 block

**75**

要是很倒霉，所有的 點都聚集在某個𝛿 ×

2𝛿區塊內怎麼辦

### 縮小搜尋範圍!

### Algo 1: check all pairs that cross two regions 𝑛/2 × 𝑛/2 combinations

### Algo 2: only consider points within 𝛿 of the cut, 𝛿 = min{l−min, r−min}

### Algo 3: only consider pairs within 𝛿 × 2𝛿 blocks

Obs 1: every pair with smaller than 𝛿 distance must appear in a 𝛿 × 2𝛿 block

Obs 2: there are at most 8 points in a 𝛿 × 2𝛿 block

Each 𝛿/2 × 𝛿/2 block contains at most 1 point, otherwise the distance returned from left/right region should be smaller than 𝛿

### Algo 1: check all pairs that cross two regions 𝑛/2 × 𝑛/2 combinations

### Algo 2: only consider points within 𝛿 of the cut, 𝛿 = min{l−min, r−min}

### Algo 3: only consider pairs within 𝛿 × 2𝛿 blocks

Obs 1: every pair with smaller than 𝛿 distance must appear in a 𝛿 × 2𝛿 block

Obs 2: there are at most 8 points in a 𝛿 × 2𝛿 block

p_{i} **77**

p_{i+4}

p_{i+2}
p_{i+5}

p_{i+3}

Find-closet-pair-across-regions 1. Sort the points by y-values within 𝛿 of the

cut (yellow region)

2. For the sorted point 𝑝_{𝑖}, compute the
distance with 𝑝_{𝑖+1}, 𝑝_{𝑖+2}, …, 𝑝_{𝑖+7}

3. Return the smallest one

### At most 7 distance calculations needed

Closest-Pair(P)

// termination condition (base case)

if |P| <= 3 brute-force finding closest pair and return it // Divide

find a vertical line L s.t. both planes_contain half of the points // Conquer (by recursion)

left-pair, left-min = Closest-Pair(points in the left) right-pair, right-min = Closest-Pair(points in the right) // Combine

delta = min{left-min, right-min}

remove points that are delta or more away from L // Obs 1
sort remaining points by y-coordinate into p_{0}, …, p_{k}

for point p_{i}:

____compute distances with p_{i+1}, p_{i+2}, …, p_{i+7}_// Obs 2
____update delta if a closer pair is found

return the closest pair and its distance

### Idea: do not sort inside the recursive case

**79**

Closest-Pair(P)

sort P by x- and y-coordinate and store in Px and Py // termination condition (base case)

if |P| <= 3 brute-force finding closest pair and return it // Divide

find a vertical line L s.t. both planes_contain half of the points // Conquer (by recursion)

left-pair, left-min = Closest-Pair(points in the left) right-pair, right-min = Closest-Pair(points in the right) // Combine

delta = min{left-min, right-min}

remove points that are delta or more away from L // Obs 1
for point p_{i} in sorted candidates

____compute distances with p_{i+1}, p_{i+2}, …, p_{i+7}_// Obs 2
____update delta if a closer pair is found

return the closest pair and its distance

### 𝑂(𝑛) algorithm

### Taking advantage of randomization

Chapter 13.7 of Algorithm Design by Kleinberg & Tardos

Samir Khuller and Yossi Matias. 1995. A simple randomized sieve

algorithm for the closest-pair problem. Inf. Comput. 118, 1 (April 1995), 34-37.

### When to use D&C

### Whether the problem with small inputs can be solved directly

### Whether subproblem solutions can be combined into the original solution

### Whether the overall complexity is better than naïve

### Note

### Try different ways of dividing

### D&C may be suboptimal due to repetitive computations

### Example.

D&C algo for Fibonacci:

Bottom-up algo for Fibonacci:

**81**

1. Divide

2. Conquer

3. Combine

Fibonacci(n) if n < 2 ____return 1

a[0]=1 a[1]=1

for i = 2 … n

____a[i]=a[i-1]+a[i-2]

return a[n]

**Our next topic: Dynamic Programming**

“a technique for solving problems with overlapping subproblems”

Course Website: http://ada17.csie.org

**82**

Important announcement will be sent to @ntu.edu.tw mailbox

& post to the course website