Michael Tsai 2017/3/28
Sorting
• Definition:
• Input: 𝑎", 𝑎$, … , 𝑎& a sequence of n numbers
• Output: 𝑎"', 𝑎$', … , 𝑎&' is a permutation (reordering) of the original sequence, such that 𝑎"' ≤ 𝑎$' ≤ ⋯ ≤ 𝑎&'
• In reality, 𝑎* is the key of a record (of multiple fields) (e.g., student ID)
• In a record, the data fields other than the key is called satellite data
• If satellite data is large in size, we will only sort the pointers pointing to the records. (avoiding moving the data)
Applications of Sorting
• Example 1: Looking for an item in a list
• Q: How do we look for an item in an unsorted list?
• A: We likely can only linearly traverse the list from the beginning.
• à𝑂(𝑛)
• Q: What if it is sorted?
• A: We can do binary search à𝑂 log 𝑛
• But, how much time do we need for sorting? (pre-‐processing)
Applications of Sorting
• Example 2:
• Compare to see if two lists are identical (list all different items)
• The two lists are n and m in length
• Q: What if they are unsorted?
• Compare the 1st item in list 1 with (m-‐1) items in list 2
• Compare the 2nd item in list 1 with (m-‐1) items in list 2
• …
• Compare the n-‐th item in list 1 with (m-‐1) items in list 2
• 𝑂 𝑛𝑚 time is needed
• Q: What if they are sorted?
• A: 𝑂(𝑛 + 𝑚)
• Again, do not forget we also need time for sorting. But, how much?
Categories of Sorting Algo.
• Internal Sort:
• Place all data in the memory
• External Sort:
• The data is too large to fit it entirely in the memory.
• Some need to be temporarily placed onto other (slower) storage, e.g., hard drive, flash disk, network storage, etc.
• In this lecture, we will only discuss internal sort.
• Storage is cheap nowadays. In most cases, only internal sort is needed.
Some terms related to sorting
• Stability:
• If 𝑎* = 𝑎5 (equal key value),then they maintain the same order before and after sorting.
• In-‐place:
• Directly sort the keys at their current memory locations.
Therefore, only O(1) additional space is needed for sorting.
• Adaptability:
• If part of the sequence is sorted, then the time complexity of the sorting algorithm reduces.
How fast can we sort?
• Assumption: compare and swap
• Compare: compare two items in the list
• Swap: Swap the locations of these two items
• How much time do we need in the worst case?
𝐾" ≤ 𝐾$
𝐾$ ≤ 𝐾7 𝐾" ≤ 𝐾7
𝐾" ≤ 𝐾7 𝐾$ ≤ 𝐾7
stop stop
stop
stop
stop stop
[1,2,3]
[1,3,2]
[1,2,3]
[1,2,3]
[1,3,2]
[2,3,1]
[2,1,3]
[2,1,3]
[3,1,2]
[3,1,2] [3,2,1]
Yes No
Decision tree for sorting
• Every node represents a comparison & swap
• Sorting is completed when reaching the leaf
• How many leaves?
• 𝑛! , since there are that many possible permutations
𝐾" ≤ 𝐾$
𝐾$ ≤ 𝐾7 𝐾" ≤ 𝐾7
𝐾" ≤ 𝐾7
stop stop
stop
stop
stop stop
[1,2,3]
[1,3,2]
[1,2,3]
[1,2,3]
[1,3,2]
[2,3,1]
[2,1,3]
[2,1,3]
[3,1,2] [3,2,1]
𝐾$ ≤ 𝐾7
How fast can we sort?
• 所以, worst case所需要花的時間, 為此binary tree的height.
• 如果decision tree height為h, 有l個leaves
• 𝑙 ≥ 𝑛!, we have a least n! outcomes (leaves)
• 𝑙 ≤ 2=, a binary tree (decision tree) of height h has at most 2=>"
leaves
• 2= ≥ 𝑙 ≥ 𝑛!
• ℎ ≥ log$𝑛!
• 𝑛! = 𝑛 𝑛 − 1 𝑛 − 2 … 3 ⋅ 2 ⋅ 1 ≥ &$
D E
• log$ 𝑛! ≥ log$ &$
D
E = &$ log$&$ = Ω 𝑛 log 𝑛
• Summary: Any “comparison-‐based” sorting algorithm has worst-‐case time complexity of Ω 𝑛 log 𝑛 .
Review: Selection Sort
• Select the smallest, move it to the first position.
• Select the second smallest, move it to the second position.
• ….
• The last item will automatically be placed at the last position.
ㄅ ㄆ 1
1 ㄆ 2 ㄅ
1 2 ㄆ ㄅ
Review: Selection Sort
• Selection sort does not change the execution of the algorithm due to the current conditions.
• Always going through the entire array in each iteration.
• Therefore, its best-‐case, worst-‐case, average-‐case running time are all 𝑂 𝑛$
• Not adaptive!
• In-‐place
Insertion Sort
• In each iteration, add one item to a sorted list of i item.
• Turning it into a sorted list of (i+1) item
2 3 6 5 1 4
2 3 6 5 1 4
2 3 6 5 1 4
2 3 5 6 1 4
2 3 5 6 1 4
1 2 3 5 6 4
1 2 3 5 6 4
1 2 3 4 5 6
Pseudo code
Insertion-Sort(A)
1 for j = 2 to A. length
2 key = A[j]
3 // Insert A[j] into the sorted sequence A[1 . . j − 1].
4 i = j − 1
5 while i > 0 and A[i] > key
6 A[i + 1] = A[i]
7 i = i − 1
8 A[i + 1] = key
1
Insertion Sort
• Q: How much time is needed?
• A: In the worst case, the item needs to be placed at the beginning for each and every iteration.
• (Spending time linear to the size of sorted part)
• ∑&>" 𝑖
" = & &>"
$ = 𝑂(𝑛$)
• Average-‐case complexity:𝑂(𝑛$). (Why?)
• Possible variation: (do those improve the time complexity?)
• 1. Use binary search to look for the location to insert.
• 2. Use linked list to store the items. Then moving takes only 𝑂(1)!
What’s good about insertion sort
• Simple (small constant in time complexity representation)
• Good choice when sorting a small list
• Stable
• In-‐place
• Adaptive
• Example: In 1,2,5,3,4 , only two inversions <5,3>, <5,4>.
• The running time for insertion sort: O(n+d), d is the number of inversions Best case: O(n) (No inversion, sorted)
• Online:
No need to know all the numbers to be sorted. Possible to sort and take input at the same time.
Merge Sort
• Use Divide-‐and-‐Conquer strategy
• Divide-‐and-‐Conquer:
• Divide: Split the big problem into small problems
• Conquer: Solve the small problems
• Combine: Combine the solutions to the small problems into the solution of the big problems.
• Merge sort:
• Divide: Split the n numbers into two sub-‐sequences of n/2 numbers
• Conquer: Sort the two sub-‐sequences (use recursive calls to delegate to the clones)
• Combine: Combine the two sorted sub-‐sequences into the one sorted sequence
Merge Sort
17
To see that the MERGE procedure runs in ‚.n/ time, where n D r ! p C 1, observe that each of lines 1–3 and 8–11 takes constant time, the for loops of lines 4–7 take ‚.n1 C n2/ D ‚.n/ time,7 and there are n iterations of the for loop of lines 12–17, each of which takes constant time.
We can now use the MERGE procedure as a subroutine in the merge sort al- gorithm. The procedure MERGE-SORT.A; p; r/ sorts the elements in the subar- ray AŒp : : r!. If p " r, the subarray has at most one element and is therefore already sorted. Otherwise, the divide step simply computes an index q that par- titions AŒp : : r! into two subarrays: AŒp : : q!, containing dn=2e elements, and AŒq C 1 : : r!, containing bn=2c elements.8
MERGE-SORT.A; p; r/
1 if p < r
2 q D b.p C r/=2c
3 MERGE-SORT.A; p; q/
4 MERGE-SORT.A; q C 1; r/
5 MERGE.A; p; q; r/
To sort the entire sequence A D hAŒ1!; AŒ2!; : : : ; AŒn!i, we make the initial call MERGE-SORT.A; 1; A:length/, where once again A:length D n. Figure 2.4 il- lustrates the operation of the procedure bottom-up when n is a power of 2. The algorithm consists of merging pairs of 1-item sequences to form sorted sequences of length 2, merging pairs of sequences of length 2 to form sorted sequences of length 4, and so on, until two sequences of length n=2 are merged to form the final sorted sequence of length n.
2.3.2 Analyzing divide-and-conquer algorithms
When an algorithm contains a recursive call to itself, we can often describe its running time by a recurrence equation or recurrence, which describes the overall running time on a problem of size n in terms of the running time on smaller inputs.
We can then use mathematical tools to solve the recurrence and provide bounds on the performance of the algorithm.
7We shall see in Chapter 3 how to formally interpret equations containing ‚-notation.
8The expression dxe denotes the least integer greater than or equal to x, and bxc denotes the greatest integer less than or equal to x. These notations are defined in Chapter 3. The easiest way to verify that setting q to b.p C r/=2c yields subarrays AŒp : : q! and AŒq C 1 : : r! of sizes dn=2e and bn=2c, respectively, is to examine the four cases that arise depending on whether each of p and r is odd or even.
Divide
Conquer x2 Combine
Merge Sort
void Mergesort(int A[], int temp, int left, int right) { int mid;
if (right > left) {
mid=(right+left)/2;
Mergesort(A,temp,left,mid);
Mergesort(A,temp,mid+1,right);
Merge(A,temp,left,mid+1,right);
} }
Divide
Conquer Combine
temp: temporarily storage
left,right: the left & right indices of the range to be sorted.
Merge Sort: Example
How to combine (merge)?
1 4 5 8 2 3 6 9
1 2 3 4 5 6 8 9
i j
• Running time: 𝑂 𝑛" + 𝑛$ = 𝑂(𝑛), 𝑛"和𝑛$ are the lengths of the two sub-‐sequences.
• A temporary storage of size O(n) is needed during the merge process
Original array
Temporary storage
Implementation: Merge
21
we just take the remaining input pile and place it face down onto the output pile.
Computationally, each basic step takes constant time, since we are comparing just the two top cards. Since we perform at most n basic steps, merging takes ‚.n/
time.
The following pseudocode implements the above idea, but with an additional twist that avoids having to check whether either pile is empty in each basic step.
We place on the bottom of each pile a sentinel card, which contains a special value that we use to simplify our code. Here, we use 1 as the sentinel value, so that whenever a card with 1 is exposed, it cannot be the smaller card unless both piles have their sentinel cards exposed. But once that happens, all the nonsentinel cards have already been placed onto the output pile. Since we know in advance that exactly r ! p C 1 cards will be placed onto the output pile, we can stop once we have performed that many basic steps.
MERGE.A; p; q; r/
1 n1 D q ! p C 1 2 n2 D r ! q
3 let LŒ1 : : n1 C 1! and RŒ1 : : n2 C 1! be new arrays 4 for i D 1 to n1
5 LŒi ! D AŒp C i ! 1!
6 for j D 1 to n2
7 RŒj ! D AŒq C j ! 8 LŒn1 C 1! D 1
9 RŒn2 C 1! D 1 10 i D 1
11 j D 1
12 for k D p to r 13 if LŒi! " RŒj !
14 AŒk! D LŒi!
15 i D i C 1
16 else AŒk! D RŒj !
17 j D j C 1
In detail, the MERGE procedure works as follows. Line 1 computes the length n1 of the subarray AŒp : : q!, and line 2 computes the length n2 of the subarray AŒq C 1 : : r!. We create arrays L and R (“left” and “right”), of lengths n1 C 1 and n2 C 1, respectively, in line 3; the extra position in each array will hold the sentinel. The for loop of lines 4–5 copies the subarray AŒp : : q! into LŒ1 : : n1!, and the for loop of lines 6–7 copies the subarray AŒq C 1 : : r! into RŒ1 : : n2!. Lines 8–9 put the sentinels at the ends of the arrays L and R. Lines 10–17, illus-
Merge sort
• Every item to be sorted is processed once per “pass”à𝑂(𝑛)
• How many passes is needed?
• The length of the sub-‐sequence doubles every pass, and finally it becomes the large sequence of n numbers
• Therefore, log$ 𝑛 passes.
• Total running time: 𝑂 𝑛 log$𝑛 = 𝑂(𝑛 log 𝑛)
• Worst-‐case, best-‐case, average-‐case: 𝑂 𝑛 log 𝑛 (Not adaptive)
• Not in-‐place: need additional storage for sorted sub-‐
sequences
• Additional space: O(n)
• Find a pivot(支點), manipulate the locations of the items so that:
• (1) all items to its left is smaller or equal (unsorted),
• (2) all items to its right is larger
• Recursively call itself to sort the left and right sub-‐sequences.
26 5 37 1 61 11 59 15 48 19
26 5 37 1 61 11 59 15 48 19
26 5 19 1 61 11 59 15 48 37
26 5 19 1 15 11 59 61 48 37
11 5 19 1 15 26 59 61 48 37
Pseudo Code
Combine: Because the subarrays are already sorted, no work is needed to combine them: the entire array AŒp : : r! is now sorted.
The following procedure implements quicksort:
QUICKSORT.A; p; r/
1 if p < r
2 q D PARTITION.A; p; r/
3 QUICKSORT.A; p; q ! 1/
4 QUICKSORT.A; q C 1; r/
To sort an entire array A, the initial call is QUICKSORT.A; 1; A:length/.
Partitioning the array
The key to the algorithm is the PARTITION procedure, which rearranges the subar- ray AŒp : : r! in place.
PARTITION.A; p; r/
1 x D AŒr!
2 i D p ! 1
3 for j D p to r ! 1 4 if AŒj ! " x
5 i D i C 1
6 exchange AŒi! with AŒj ! 7 exchange AŒi C 1! with AŒr!
8 return i C 1
Figure 7.1 shows how PARTITION works on an 8-element array. PARTITION always selects an element x D AŒr! as a pivot element around which to partition the subarray AŒp : : r!. As the procedure runs, it partitions the array into four (possibly empty) regions. At the start of each iteration of the for loop in lines 3–6, the regions satisfy certain properties, shown in Figure 7.2. We state these properties as a loop invariant:
At the beginning of each iteration of the loop of lines 3–6, for any array index k,
1. If p " k " i, then AŒk! " x.
2. If i C 1 " k " j ! 1, then AŒk! > x.
3. If k D r, then AŒk! D x.
Divide
Conquer x2
No Combine!
Quick Sort
11 5 19 1 15 26 59 61 48 37
1 5 11 19 15 26 59 61 48 37
1 5 11 19 15 26 59 61 48 37
1 5 11 15 19 26 59 61 48 37
1 5 11 15 19 26 59 61 48 37
1 5 11 15 19 26 48 37 59 61
1 5 11 15 19 26 37 48 59 61
1 5 11 15 19 26 37 48 59 61
Quick Sort: Worst & Best case
• But worst case running time is still O 𝑛$
• Q: Give an example which produces worst-‐case running time for the quick sort algorithm.
• In this case: running time is O(𝑛$)
• Best case?
• Pivot can split the sequence into two sub-‐sequences of equal size.
• Therefore, T(n)=2T(n/2)+O 𝑛
• T(n)=O(𝑛 log 𝑛)
Randomized Quick Sort
• Avoid worst case to happen frequently
• Randomly select a pivot (not always the leftmost key)
• Reduce the probability of the worst case
• However, worst case running time is still 𝑂 𝑛$
26 5 37 1 61 11 59 15 48 19
Randomly select a pivot
Swap in advance
Average running time
• Better if the selection of pivot can evenly split the sequence into two sub-‐sequences of equal size
• Why the average running time is close to the best-‐case one?
• 假設很糟的一個狀況: 每次都分成1:9
• 𝑇(𝑛) = 𝑇(9𝑛/10) + 𝑇(𝑛/10) + 𝑐𝑛
• = 𝑇 Q"&
"RR + 𝑇 "RRS& + ST&"R + 𝑇 "RRS& + 𝑇 "RR"& + T&"R + 𝑐𝑛
• = ⋯
Time needed for the “9/10 subsequence”
Time needed for the “1/10 subsequence”
Time needed for partitioning
Average running time
As long as the pivot can partition according to a particular ratio (even not close to 50%), we can still obtain 𝑂 𝑛 log 𝑛 running time!
Average running time
Case 1:
Worst case for the first level partition,
but best-‐case for second level.
Case 2:
Best case for the first level partition
Partition time for the first level:Θ(𝑛)
Partition time, second level:
Θ(𝑛 − 1)
Partition time for the first level:Θ(𝑛)
Θ 𝑛 + Θ 𝑛 − 1 = Θ(𝑛)
Same!
(Case 1 has larger constant)
The better-‐partitioned level would “absorb” the extra running time for worse-‐partitioned level.
比較四大金剛
• Insertion sort: quick with small input size n. (small constant)
• Quick sort: Best average performance (fairly small constant)
• Merge sort: Best worst-‐case performance
• Heap sort: Good worst-‐case performance, no additional space needed.
• Real-‐world strategy: a hybrid of insertion sort + others. Use input size n to determine the algorithm to use.
Worst Average Additional
Space?
Insertion sort 𝑂(𝑛$) 𝑂(𝑛$) O(1)
Merge sort 𝑂(𝑛 log𝑛) 𝑂(𝑛 log𝑛) O(n)
Quick sort O(𝑛$) O(𝑛 log 𝑛) O(1)
Heap sort 𝑂(𝑛 log𝑛) − O(1)
Not covered today!