Case Study 7: Sorting Things Out
Concepts illustrated by this case study
■ Benchmarking
■ Performance Analysis
■ Cost/Performance Analysis
■ Amortization of Overhead
■ Balanced Systems
The database field has a long history of using benchmarks to compare systems. In this question, you will explore one of the benchmarks introduced by Anon. et al.
[1985] (see Chapter 1): external, or disk-to-disk, sorting.
Sorting is an exciting benchmark for a number of reasons. First, sorting exer-cises a computer system across all its components, including disk, memory, and processors. Second, sorting at the highest possible performance requires a great deal of expertise about how the CPU caches, operating systems, and I/O subsys-tems work. Third, it is simple enough to be implemented by a student (see below!).
Depending on how much data you have, sorting can be done in one or multi-ple passes. Simply put, if you have enough memory to hold the entire dataset in memory, you can read the entire dataset into memory, sort it, and then write it out; this is called a “one-pass” sort.
If you do not have enough memory, you must sort the data in multiple passes.
There are many different approaches possible. One simple approach is to sort each chunk of the input file and write it to disk; this leaves (input file size)/(memory size) sorted files on disk. Then, you have to merge each sorted temporary file into a final sorted output. This is called a “two-pass” sort. More passes are needed in the unlikely case that you cannot merge all the streams in the second pass.
In this case study, you will analyze various aspects of sorting, determining its effectiveness and cost-effectiveness in different scenarios. You will also write your own version of an external sort, measuring its performance on real hard-ware.
D.28 [20/20/20] <D.4> We will start by configuring a system to complete a sort in the least possible time, with no limits on how much we can spend. To get peak band-width from the sort, we have to make sure all the paths through the system have sufficient bandwidth.
Assume for simplicity that the time to perform the in-memory sort of keys is lin-early proportional to the CPU rate and memory bandwidth of the given machine (e.g., sorting 1 MB of records on a machine with 1 MB/sec of memory bandwidth and a 1 MIPS processor will take 1 second). Assume further that you have care-fully written the I/O phases of the sort so as to achieve sequential bandwidth.
And, of course, realize that if you don’t have enough memory to hold all of the data at once that sort will take two passes.
One problem you may encounter in performing I/O is that systems often perform extra memory copies; for example, when the read() system call is invoked, data may first be read from disk into a system buffer and then subsequently copied into the specified user buffer. Hence, memory bandwidth during I/O can be an issue.
Finally, for simplicity, assume that there is no overlap of reading, sorting, or writ-ing. That is, when you are reading data from disk, that is all you are doing; when sorting, you are just using the CPU and memory bandwidth; when writing, you are just writing data to disk.
Your job in this task is to configure a system to extract peak performance when sorting 1 GB of data (i.e., roughly 10 million 100-byte records). Use the follow-ing table to make choices about which machine, memory, I/O interconnect, and disks to buy.
Note: Assume that you are buying a single-processor system and that you can have up to two I/O interconnects. However, the amount of memory and number of disks are up to you (assume there is no limit on disks per I/O interconnect).
CPU I/O interconnect
Slow 1 GIPS $200 Slow 80 MB/sec $50
Standard 2 GIPS $1000 Standard 160 MB/sec $100
Fast 4 GIPS $2000 Fast 320 MB/sec $400
Memory Disks
Slow 512 MB/sec $100/GB Slow 30 MB/sec $70
Standard 1 GB/sec $200/GB Standard 60 MB/sec $120
Fast 2 GB/sec $500/GB Fast 110 MB/sec $300
a. [20] <D.4> What is the total cost of your machine? (Break this down by part, including the cost of the CPU, amount of memory, number of disks, and I/O bus.)
b. [20] <D.4> How much time does it take to complete the sort of 1 GB worth of records? (Break this down into time spent doing reads from disk, writes to disk, and time spent sorting.)
c. [20] <D.4> What is the bottleneck in your system?
D.29 [25/25/25] <D.4> We will now examine cost-performance issues in sorting. After all, it is easy to buy a high-performing machine; it is much harder to buy a cost-effective one.
One place where this issue arises is with the PennySort competition (research.
microsoft.com/barc/SortBenchmark/). PennySort asks that you sort as many records as you can for a single penny. To compute this, you should assume that a system you buy will last for 3 years (94,608,000 seconds), and divide this by the total cost in pennies of the machine. The result is your time budget per penny.
Our task here will be a little simpler. Assume you have a fixed budget of $2000 (or less). What is the fastest sorting machine you can build? Use the same hard-ware table as in Exercise D.28 to configure the winning machine.
(Hint: You might want to write a little computer program to generate all the pos-sible configurations.)
a. [25] <D.4> What is the total cost of your machine? (Break this down by part, including the cost of the CPU, amount of memory, number of disks, and I/O bus.)
b. [25] <D.4> How does the reading, writing, and sorting time break down with this configuration?
c. [25] <D.4> What is the bottleneck in your system?
D.30 [20/20/20] <D.4, D.6> Getting good disk performance often requires amortiza-tion of overhead. The idea is simple: If you must incur an overhead of some kind, do as much useful work as possible after paying the cost and hence reduce its impact. This idea is quite general and can be applied to many areas of computer systems; with disks, it arises with the seek and rotational costs (overheads) that you must incur before transferring data. You can amortize an expensive seek and rotation by transferring a large amount of data.
In this exercise, we focus on how to amortize seek and rotational costs during the second pass of a two-pass sort. Assume that when the second pass begins, there are N sorted runs on the disk, each of a size that fits within main memory. Our task here is to read in a chunk from each sorted run and merge the results into a final sorted output. Note that a read from one run will incur a seek and rotation, as it is very likely that the last read was from a different run.
a. [20] <D.4, D.6> Assume that you have a disk that can transfer at 100 MB/sec, with an average seek cost of 7 ms, and a rotational rate of 10,000 RPM.
Assume further that every time you read from a run, you read 1 MB of data and that there are 100 runs each of size 1 GB. Also assume that writes (to the final sorted output) take place in large 1 GB chunks. How long will the merge phase take, assuming I/O is the dominant (i.e., only) cost?
b. [20] <D.4, D.6> Now assume that you change the read size from 1 MB to 10 MB. How is the total time to perform the second pass of the sort affected?
c. [20] <D.4, D.6> In both cases, assume that what we wish to maximize is disk efficiency. We compute disk efficiency as the ratio of the time spent transfer-ring data over the total time spent accessing the disk. What is the disk effi-ciency in each of the scenarios mentioned above?
D.31 [40] <D.2, D.4, D.6> In this exercise, you will write your own external sort. To generate the data set, we provide a tool generate that works as follows:
generate <filename> <size (in MB)>
By running generate, you create a file named filename of size size MB. The file consists of 100 byte keys, with 10-byte records (the part that must be sorted).
We also provide a tool called check that checks whether a given input file is sorted or not. It is run as follows:
check <filename>
The basic one-pass sort does the following: reads in the data, sorts the data, and then writes the data out. However, numerous optimizations are available to you:
overlapping reading and sorting, separating keys from the rest of the record for better cache behavior and hence faster sorting, overlapping sorting and writing, and so forth.
One important rule is that data must always start on disk (and not in the file system cache). The easiest way to ensure this is to unmount and remount the file system.
One goal: Beat the Datamation sort record. Currently, the record for sorting 1 million 100-byte records is 0.44 seconds, which was obtained on a cluster of 32 machines. If you are careful, you might be able to beat this on a single PC config-ured with a few disks.