Orders of Growth

1 1

1 2 1

1 3 3 1

1 4 6 4 1

. . .

e numbers at the edge of the triangle are all 1, and each number inside the triangle is the sum of the two numbers above it.³⁵ Write a procedure that computes elements of Pascal’s triangle by means of a recursive process.

Exercise 1.13: Prove that Fib(n) is the closest integer to ϕⁿ/√

5, where ϕ = (1 +√

5)/2. Hint: Let ψ = (1 −√ 5)/2.

Use induction and the definition of the Fibonacci numbers (seeSection 1.2.2) to prove that Fib(n)= (ϕⁿ − ψⁿ)/√

5.

1.2.3 Orders of Growth

e previous examples illustrate that processes can differ considerably in the rates at which they consume computational resources. One con-venient way to describe this difference is to use the notion of order of growthto obtain a gross measure of the resources required by a process as the inputs become larger.

35e elements of Pascal’s triangle are called the binomial coefficients, because the n^throw consists of the coefficients of the terms in the expansion of (x + y)ⁿ. is pat-tern for computing the coefficients appeared in Blaise Pascal’s 1653 seminal work on probability theory, Traité du triangle arithmétique. According toKnuth (1973), the same paern appears in the Szu-yuen Yü-chien (“e Precious Mirror of the Four Elements”), published by the Chinese mathematician Chu Shih-chieh in 1303, in the works of the twelh-century Persian poet and mathematician Omar Khayyam, and in the works of the twelh-century Hindu mathematician Bháscara Áchárya.

Let n be a parameter that measures the size of the problem, and let R(n)be the amount of resources the process requires for a problem of size n. In our previous examples we took n to be the number for which a given function is to be computed, but there are other possibilities. For instance, if our goal is to compute an approximation to the square root of a number, we might take n to be the number of digits accuracy required.

For matrix multiplication we might take n to be the number of rows in the matrices. In general there are a number of properties of the problem with respect to which it will be desirable to analyze a given process.

Similarly, R(n) might measure the number of internal storage registers used, the number of elementary machine operations performed, and so on. In computers that do only a fixed number of operations at a time, the time required will be proportional to the number of elementary machine operations performed.

We say that R(n) has order of growth Θ(f (n)), wrien R(n) = Θ(f (n)) (pronounced “theta of f (n)”), if there are positive constants k₁and k₂ independent of n such that k1f (n) ≤ R(n) ≤ k2f (n)for any sufficiently large value of n. (In other words, for large n, the value R(n) is sandwiched between k₁f (n)and k₂f (n).)

For instance, with the linear recursive process for computing facto-rial described inSection 1.2.1the number of steps grows proportionally to the input n. us, the steps required for this process grows as Θ(n).

We also saw that the space required grows as Θ(n). For the iterative factorial, the number of steps is still Θ(n) but the space is Θ(1)—that is, constant.³⁶e tree-recursive Fibonacci computation requires Θ(ϕⁿ)

36ese statements mask a great deal of oversimplification. For instance, if we count process steps as “machine operations” we are making the assumption that the number of machine operations needed to perform, say, a multiplication is independent of the size of the numbers to be multiplied, which is false if the numbers are sufficiently large.

Similar remarks hold for the estimates of space. Like the design and description of a process, the analysis of a process can be carried out at various levels of abstraction.

steps and space Θ(n), where ϕ is the golden ratio described inSection 1.2.2.

Orders of growth provide only a crude description of the behavior of a process. For example, a process requiring n² steps and a process requiring 1000n²steps and a process requiring 3n²+ 10n + 17 steps all have Θ(n²)order of growth. On the other hand, order of growth provides a useful indication of how we may expect the behavior of the process to change as we change the size of the problem. For a Θ(n) (linear) process, doubling the size will roughly double the amount of resources used. For an exponential process, each increment in problem size will multiply the resource utilization by a constant factor. In the remainder ofSection 1.2 we will examine two algorithms whose order of growth is logarithmic, so that doubling the problem size increases the resource requirement by a constant amount.

Exercise 1.14:Draw the tree illustrating the process gen-erated by thecount-changeprocedure ofSection 1.2.2in making change for 11 cents. What are the orders of growth of the space and number of steps used by this process as the amount to be changed increases?

Exercise 1.15:e sine of an angle (specified in radians) can be computed by making use of the approximation sin x ≈ x if x is sufficiently small, and the trigonometric identity

sin x = 3 sinx

3 − 4 sin³x 3

to reduce the size of the argument of sin. (For purposes of this exercise an angle is considered “sufficiently small” if its magnitude is not greater than 0.1 radians.) ese ideas are incorporated in the following procedures:

(define (cube x) (* x x x))

(define (p x) (- (* 3 x) (* 4 (cube x)))) (define (sine angle)

(if (not (> (abs angle) 0.1)) angle

(p (sine (/ angle 3.0)))))

a. How many times is the procedure^papplied when^(sine

12.15)is evaluated?

b. What is the order of growth in space and number of steps (as a function of a) used by the process generated by the^sineprocedure when^{(sine a)}is evaluated?

1.2.4 Exponentiation

Consider the problem of computing the exponential of a given number.

We would like a procedure that takes as arguments a base b and a posi-tive integer exponent n and computes bⁿ. One way to do this is via the recursive definition

bⁿ = b · bⁿ⁻¹, b⁰ = 1, which translates readily into the procedure

(define (expt b n) (if (= n 0)

1

(* b (expt b (- n 1)))))

is is a linear recursive process, which requires Θ(n) steps and Θ(n) space. Just as with factorial, we can readily formulate an equivalent lin-ear iteration:

(define (expt b n) (expt-iter b n 1))

(define (expt-iter b counter product) (if (= counter 0)

product (expt-iter b

(- counter 1) (* b product))))

is version requires Θ(n) steps and Θ(1) space.

We can compute exponentials in fewer steps by using successive squaring. For instance, rather than computing b⁸as

b· (b · (b · (b · (b · (b · (b · b)))))) , we can compute it using three multiplications:

b²= b · b, b⁴= b²· b², b⁸= b⁴· b⁴.

is method works fine for exponents that are powers of 2. We can also take advantage of successive squaring in computing exponentials in general if we use the rule

bⁿ = (b^n/2)² if n is even, bⁿ = b · bⁿ⁻¹ if n is odd.

We can express this method as a procedure:

(define (fast-expt b n) (cond ((= n 0) 1)

((even? n) (square (fast-expt b (/ n 2)))) (else (* b (fast-expt b (- n 1))))))

where the predicate to test whether an integer is even is defined in terms of the primitive procedure^remainderby

(define (even? n)

(= (remainder n 2) 0))

e process evolved by^fast-exptgrows logarithmically with n in both space and number of steps. To see this, observe that computing b²ⁿ us-ing^fast-exptrequires only one more multiplication than computing bⁿ. e size of the exponent we can compute therefore doubles (approx-imately) with every new multiplication we are allowed. us, the num-ber of multiplications required for an exponent of n grows about as fast as the logarithm of n to the base 2. e process has Θ(log n) growth.³⁷

e difference between Θ(log n) growth and Θ(n) growth becomes striking as n becomes large. For example,^fast-exptfor n = 1000 re-quires only 14 multiplications.³⁸ It is also possible to use the idea of successive squaring to devise an iterative algorithm that computes ex-ponentials with a logarithmic number of steps (seeExercise 1.16), al-though, as is oen the case with iterative algorithms, this is not wrien down so straightforwardly as the recursive algorithm.³⁹

Exercise 1.16:Design a procedure that evolves an itera-tive exponentiation process that uses successive squaring

37More precisely, the number of multiplications required is equal to 1 less than the log base 2 of n plus the number of ones in the binary representation of n. is total is always less than twice the log base 2 of n. e arbitrary constants k1and k2in the definition of order notation imply that, for a logarithmic process, the base to which logarithms are taken does not maer, so all such processes are described as Θ(log n).

38You may wonder why anyone would care about raising numbers to the 1000th power. SeeSection 1.2.6.

39is iterative algorithm is ancient. It appears in the Chandah-sutra by Áchárya Pingala, wrien before 200 .. SeeKnuth 1981, section 4.6.3, for a full discussion and analysis of this and other methods of exponentiation.

and uses a logarithmic number of steps, as does^fast-expt. (Hint: Using the observation that (b^n/2)² = (b²)^n/2, keep, along with the exponent n and the base b, an additional state variable a, and define the state transformation in such a way that the product abⁿis unchanged from state to state.

At the beginning of the process a is taken to be 1, and the answer is given by the value of a at the end of the process.

In general, the technique of defining an invariant quantity that remains unchanged from state to state is a powerful way to think about the design of iterative algorithms.) Exercise 1.17:e exponentiation algorithms in this sec-tion are based on performing exponentiasec-tion by means of repeated multiplication. In a similar way, one can perform integer multiplication by means of repeated addition. e following multiplication procedure (in which it is assumed that our language can only add, not multiply) is analogous to the^exptprocedure:

(define (* a b) (if (= b 0)

0

(+ a (* a (- b 1)))))

is algorithm takes a number of steps that is linear in^b. Now suppose we include, together with addition, opera-tions^double, which doubles an integer, and^halve, which divides an (even) integer by 2. Using these, design a mul-tiplication procedure analogous to^fast-expt that uses a logarithmic number of steps.

Exercise 1.18:Using the results ofExercise 1.16and Exer-cise 1.17, devise a procedure that generates an iterative pro-cess for multiplying two integers in terms of adding, dou-bling, and halving and uses a logarithmic number of steps.⁴⁰

Exercise 1.19:ere is a clever algorithm for computing the Fibonacci numbers in a logarithmic number of steps.

Recall the transformation of the state variables a and b in the^fib-iterprocess ofSection 1.2.2: a← a +b and b ← a.

Call this transformation T , and observe that applying T over and over again n times, starting with 1 and 0, produces the pair Fib(n + 1) and Fib(n). In other words, the Fibonacci numbers are produced by applying Tⁿ, the n^thpower of the transformationT , starting with the pair (1, 0). Now consider T to be the special case of p = 0 and q = 1 in a family of transformations Tpq, where Tpq transforms the pair (a, b) according to a ← bq + aq + ap and b ← bp + aq. Show that if we apply such a transformation Tpq twice, the effect is the same as using a single transformation Tp^′q^′ of the same form, and compute p^′and q^′in terms of p and q. is gives us an explicit way to square these transformations, and thus we can compute Tⁿusing successive squaring, as in the^fast-exptprocedure. Put this all together to com-plete the following procedure, which runs in a logarithmic number of steps:⁴¹

40is algorithm, which is sometimes known as the “Russian peasant method” of multiplication, is ancient. Examples of its use are found in the Rhind Papyrus, one of the two oldest mathematical documents in existence, wrien about 1700 .. (and copied from an even older document) by an Egyptian scribe named A’h-mose.

41is exercise was suggested to us by Joe Stoy, based on an example inKaldewaij

(define (fib n)

(fib-iter 1 0 0 1 n))

(define (fib-iter a b p q count) (cond ((= count 0) b)

((even? count) (fib-iter a

b

⟨^??⟩ ; compute p^′

⟨^??⟩ ; compute q^′ (/ count 2)))

(else (fib-iter (+ (* b q) (* a q) (* a p)) (+ (* b p) (* a q))

p q

(- count 1)))))