ELEMENTS OF THE THEORY OF COMPUTATION

(1)

ELEMENTS OF THE THEORY OF

COMPUTATION

Second Edition

Harry R. Lewis

Gordon McKay Professor of Computer Science Harvard University

and Dean of Harvard College Cambridge, Massachusetts

Christos H. Papadimitriou

C. Lester Hogan Professor of Electrical Engineering and Computer Science

University of California Berkeley, California

PRENTICE-HALL, Upper Saddle River, New Jersey 07458

(2)

Library of Congress Cataloging-in-PubJication Data

Lewis, Harry R.

Elements of the theory of computation I Harry R. Lewis and Christos H. Papadimitriou. - 2nd ed.

p. em.

Includes bibliological references and index.

ISBN: 0-13-26247&-8

I. Machine theory. 2. Formal languages. 3. Computational complexity. 4. Logic, Symbolic and mathematical.

I. Papadimitriou. Christos H. II. Title.

QA267.L49 1998

511.3--<1c21 97-13879

Publisher: Alan Apt

Development Editor: Sondra Chavez

EditorialJProduction Supervision: Barbara Kraemer Managing Editor: Bayani Mendoza DeLeon Editor-in-Chief: Marcia Horton

OP

Assistant Vice President of Production and Manufacturing: David W. Riccardi Art Director: Jayne Conte

Manufacturing Manager: Trudy Pisciotti Manufacturing Buyer: Donna Sullivan Editorial Assistant: Toni Holm

Simon & Schuster I A Viacom Company Upper Saddle River, New Jersey 07458

The author and publisher of this book have used their best efforts in preparing this book. These efforts include the development, research, and testing of the theories and programs to determine their effectiveness.

The author and publisher make no warranty of any kind, expressed or implied, with regard to these programs or the documentation contained in this book. The author and publisher shall not be liable in any event for incidental or consequential damages in connection with, or arising out of. the furnishing. performance. or use of these programs.

Printed in the United States of America 1 0 9 8 7 6 5 4 3 2

ISBN 0-13-262478-8

Prentice-Hall International (UK) Limited, London Prentice-Hall of Australia Pty. Limited, Sydney Prentice-Hall Canada Inc., Toronto

Prentice-Hall Hispanoamericana, S.A., Mexico Prentice-Hall of India Private Limited, New Delhi Prentice-Hall of Japan, Inc., Tokyo

Simon & Schuster Asia Pte. Ltd., Singapore Editora Prentice-Hall do Brasil, Ltda., Rio de Janeiro

(3)

To our daughters

(4)

(5)

NP

292 References 299

7 NP-completeness 301

7.1 Polynomial-time reductions 301 7.2 Cook's Theorem 309

7.3 More NP-complete problems 7.4 Coping with NP-completeness References 350

Index

317 333

353

(7)

Preface to the First Edition

This book is an introuuction, on the undergraduate level, to the classical and contemporary theory of computation. The topics covered are, in a few words, the theory of automata and formal languages, computability by Turing machines and recursive functions, uncomputability, computational complexity, and math- ematicallogic. The treatment is mathematical but the viewpoint is that of computer science; thus the chapter on context-free languages includes a discussion of parsing, and the chapters on logic establish the soundness and completeness of resolution theorem-proving.

In the undergraduate curriculum, exposure to this subject tends to come late, if at all, and collaterally with courses on the design and analysis of algorithms. It is our view that computer science students should be exposed to this material earlier -as sophomores or juniors- both because of the deeper insights it yields on specific topics in computer science, and because it serves to establish essential mathematical paradigms. But we have found teaching to a rigorous undergraduate course on the subject a difficult undertaking because of the mathematical maturity assumed by the more advanced textbooks. Our goal in writing this book has been to make the essentials of the subject accessible to a broad undergraduate audience in a way that is mathematically sound but presupposes no special mathematical experience.

The whole book represents about a year's worth of coursework. We have each taught a one-term course covering much of the material in Chapters 1 through 6, omitting on various occasions and in various combinations the sections of parsing, on recursive functions, and on particular unsolvable decision problems. Other selections are possible; for example, a course emphasizing computability and the foundations of mechanical logic might skip quickly over Chap- ters 1 through 3 and concentrate on Chapters 4, 6, 8, and 9. However, it is used, our fervent hope is that the book will contribute to the intellectual development

(8)

Preface of the next generation of computer scientists by introducing them at an early stage of their education to crisp and methodical thinking about computational problems.

We take this opportunity to thank all from whom we have learned, both teachers and students. Specific thanks go to Larry Denenberg and Aaron Temin for their proofreading of early drafts, and to Michael Kahl and Oded Shmueli for their assistance and advice as teaching assistants. In the spring of 1980 Al- bert Meyer taught a course at M.LT. from a draft of this book, and we thank him warmly for his criticisms and corrections. Of course, the blame for any re- maining errors rests with us alone. Renate D' Arcangelo typed and illustrated the manuscript with her characteristic but extraordinary perfectionism and rapidity.

(9)

Preface to the Second Edition

Much has changed in the fifteen years since the Elements of the Theory of Com- putation first appeared -and much has remained the same. Computer science is now a much more mature and established discipline, playing a role of ever in- creasing importance in a world of ubiquitous computing, globalized information, and galloping complexity -more reasons to keep in touch with its foundations.

The authors of the Elements are now themselves much more mature and busy -that is why this second edition has been so long in coming. We undertook it because we felt that a few things could be said better, a few made simpler -some even omitted altogether. More importantly, we wanted the book to reflect how the theory of computation, and its students, have evolved during these years. Although the theory of computation is now taught more widely in absolute terms, its relative position within the computer science curriculum, for example vis

a

^visthe subject of algorithms, has not been strengthened. In fact, the field of the design and analysis of algorithms is now so mature, that its elementary principles are arguably a part of a basic course on the theory of computation. Besides, undergraduates today, with their extensive and early computational experience, are much more aware of the applications of automata in compilers, for example, and more suspicious when simple models such as the Turing machine are presented as general computers. Evidently, the treatment of these subjects needs some updating.

Concretely, these are the major differences from the first edition:

o Rudiments of the design and analysis of algorithms are introduced informally already in Chapter 1 (in connection with closures), and algorithmic questions are pursued throughout the book. There are sections on algorithmic problems in connection with finite automata and context-free grammars in Chapters 2 and 3 (including state minimization and context- free recognition), algorithms for easy variants of NP-complete problems,

(10)

Preface and a section that reviews algorithmic techniques for "coping with

NP-

completeness" (special case algorithms, approximation algorithms, back- tracking and branch-and-bound, local improvement, and simulated anneal- ing algorithms).

o The treatment of Turing machines in Chapter 4 is more informal, and the simulation arguments are simpler and more quantitative. A random access Turing machine is introduced, helping bridge the gap between the clum- siness of Turing machines and the power of computers and programming languages.

o We included in Chapter 5 On undecidability some recursive function theory (up to Rice's Theorem). Grammars and recursive numerical functions are introduced and proved equivalent to Turing machines earlier, and the proofs are simpler. The undecidability of problems related to context-free grammars is proved by a simple and direct argument, without recourse to the Post correspondence problem. We kept the tiling problem, which we revisit in the NP-completeness chapter.

o Complexity is done in a rather novel way: In Chapter 6, we define no other time bounds besides the polynomial ones --thus P is the first complexity class and concept encountered. Diagonalization then shows that there are exponential problems not in P. Real-life problems are introduced side-by- side with their language representations (a distinction that is deliberately blurred), and their algorithmic questions are examined extensively.

o There is a separate NP-completeness chapter with a new, extensive, and, we think, pedagogically helpful suite of NP-completeness reductions, cul- minating with the equivalence problem for regular expressions --closing a full circle to the first subject of the book. As mentioned above, the book ends with a section on algorithmic techniques for "coping with

NP-

completeness."

o There are no logic chapters in the new edition. This was a difficult decision, made for two reasons: According to all evidence, these were the least read and taught chapters of the book; and there are now books that treat this subject better. However, there is extensive treatment of Boolean logic and its satisfiability problems in Chapter 6.

o Overall, proofs and exposition have been simplified and made more informal at some key points. In several occasions, as in the proof of the equivalence of context-free languages and pushdown automata, long technical proofs of inductive statements have become exercises. There are problems following each section.

As a result of these changes, there is now at least one more way of teaching out of the material of this book (besides the ones outlined in the first edition, and the ones that emerged from its use): A semester-length course aiming at the

(11)

Preface

coverage of the basics of both the theory of computation and algorithms may be based on a selection of material from Chapters 2 through 7.

We want to express our sincere thanks to all of our students and colleagues who provided feedback, ideas, errors, and corrections during these fifteen years- it is impossible to come up with a complete list. Special thanks to Martha Sideri for her help with the revision of Chapter 3. Also, many thanks to our editor, Alan Apt, and the people at Prentice-Hall-Barbara Kraemer, Sondra Chavez, and Bayani de Leon- who have been so patient and helpful.

In the present second printing. many errors have been corrected. We are in- debted to Carl Smith. Elaine Rich. Ray Miller. James Grimrnelmann. Rocio Guillen.

Paliath Narendran. Kuo-liang Chung Zhizhang Shen. Hua Ren. Charles Wells. Eric Thomson. Eric Fried. Jeremy Dawson. and especially Mikkel Nygaard Hansen. for pointing out errors to us. and to Barbara Taylor-Laino for making sure they were corrected.

Finally, we would appreciate receiving error reports or other comments.

preferably by electronic mail to the address elements(Olcs.berkeley.edu. Con- firmed errors, corrections. and other information about the book can also be obtained by writing to this address.

(12)

(13)

ELEMENTS OF THE THEORY OF

COMPUTATION

(14)

(15)

Introduction

Look around you. Computation happens everywhere, all the time, initiated by everybody, and affecting us all. Computation can happen because computer scientists over the past decades have discovered sophisticated methods for managing computer resources, enabling communication, translating programs, de- signing chips and databases, creating computers and programs that are faster, cheaper, easier to use, more secure.

As it is usually the case with all major disciplines, the practical successes of computer science build on its elegant and solid foundations. At the basis of physical sciences lie fundamental questions such aH what is the nature of matter? and what is the basis and origin of organic life? Computer science has its own set of fundamental questions: What is an algorithm? What can and what cannot be computed? When should an algorithm be considered practically feasible? For more than sixty years (starting even before the advent of the electronic computer) computer scientists have been pondering these questions, and coming up with ingenious answers that have deeply influenced computer science.

The purpose of this book is to introduce you to these fundamental ideas, models, and results that permeate computer science, the basic paradigms of our field. They are worth studying, for many reasons. First, much of modern computer science is based more or less explicitly on them -and much of the rest should ... Also, these ideas and models are powerful and beautiful, excellent examples of mathematical modeling that is elegant, productive, and of lasting value. Besides, they are so much a part of the history and the "collective sub- conscious" of our field, that it is hard to understand computer science without first being exposed to them.

It probably comes as no surprise that these ideas and models are mathemat- ical in nature. Although a computer is undeniably a physical object, it is also

1

(16)

2 Introduction

true that very little that is useful can be said of its physical aspects, such as its molecules and its shape; the most useful abstractions of a computer are clearly mathematical, and so the techniques needed to argue about them are necessarily likewise. Besides, practical computational tasks require the ironclad guarantees that only mathematics provides (we want our compilers to translate correctly, our application programs to eventually terminate, and so on). However, the mathematics employed in the theory of computation is rather different from the mathematics used in other applied disciplines. It is generally discrete, in that the emphasis is not on real numbers and continuous variables, but on finite sets and sequences. It is based on very few and elementary concepts, and draws its power and depth from the careful, patient, extensive, layer-by-Iayer manipulation of these concepts -just like the computer. In the first chapter you will be reminded of these elementary concepts and techniques (sets, relations, and induction, among others), and you will be introduced to the style in which they are used in the theory of computation.

The next two chapters, Chapters 2 and 3, describe certain restricted models of computation capable of performing very specialized string manipulation tasks, such as telling whether a given string, say the word punk, appears in a given text, such as the collective works of Shakespeare; or for testing whether a given string of parentheses is properly balanced --like

0

and

(())O,

but not )0. These restricted computational devices (called finite-state automata and pushdown automata, respectively) actually come up in practice as very useful and highly optimized components of more general systems such as circuits and compilers. Here they provide fine warm-up exercises in our quest for a formal, general definition of an algorithm. Furthermore, it is instructive to see how the power of these devices waxes and wanes (or, more often, is preserved) with the addition or removal of various features, most notably of non determinism, an in- triguing aspect of computation which is as central as it is (quite paradoxically) unrealistic.

In Chapter 4 we study general models of algorithms, of which the most basic is the Turing machine,t a rather simple extension of the string-manipulating devices of Chapters 2 and 3 which turns out to be, surprisingly, a general frame-

t

Named after Alan M. Turing (1912~1954), the brilliant English mathematician and philosopher whose seminal paper in 1936 marked the beginning of the theory of computation (and whose image, very appropriately, adorns the cover of this book). Turing also pioneered the fields of artificial intelligence and chess-playing by computer, as well as that of morphogenesis in biology, and was instrumental in breaking Enigma, the German naval code during World War II. For more on his fascinating life and times (and on his tragic end in the hands of official cruelty and bigotry) see the book Alan Turing: The Enigma, by Andrew Hodges, New York: Simon Schuster, 1983.

(17)

Introduction 3 work for describing arbitrary algorithms. In order to argue this point, known as the Church- Turing thesis, we introduce more and more elaborate models of com- putation (more powerful variants of the Turing machine, even a random access Turing machine and recursive definitions of numerical functions), and show that they are all precisely equivalent in power to the basic Turing machine model.

The following chapter deals with undecidability, the surprising property of certain natural and well-defined computational tasks to lie provably beyond the reach of algorithmic solution. For example, suppose that you are asked whether we can use tiles from a given finite list of basic shapes to tile the whole plane.

If the set of shapes contains a square, or even any triangle, then the answer is obviously "yes." But what if it consists of a few bizarre shapes, or if some of the shapes are mandatory, that is, they must be used at least once for the tiling to qualify? This is surely the kind of complicated question that you would like to have answered by a machine. In Chapter 5 we use the formalism of Turing machines to prove that this and many other problems cannot be solved

by computers at all.

Even when a computational task is amenable to solution by some algorithm, it may be the case that there is no reasonably fast, practically feasible algorithm that solves it. In the last two chapters of this book we show how real-life com- putational problems can be categorized in terms of their complexity: Certain problems can be solved within reasonable, polynomial time bounds, whereas others seem to require amounts of time that grow astronomically, exponentially.

In Chapter 7 we identify a class of common, practical, and notoriously difficult problems that are called NP-complete (the traveling salesman problem is only one of them). We establish that all these problems are equivalent in that, if one of them has an efficient algorithm, then all of them do. It is widely believed that all NP-complete problems are of inherently exponential complexity; whether this conjecture is actually true is the famous P -:j; NP problem, one of the most important and deep problems facing mathematicians and computer scientists today.

This book is very much about algorithms and their formal foundations.

However, as you are perhaps aware, the subject of algorithms, their analysis and their design, is considered in today's computer science curriculum quite separate from that of the theory of computation. In the present edition of this book we have tried to restore some of the unity of the subject. As a result, this book also provides a decent, if somewhat specialized and unconventional, introduction to the subject of algorithms. Algorithms and their analysis are introduced informally in Chapter 1, and are picked up again and again in the context of the restricted models of computation studied in Chapters 2 and 3, and of the natural computational problems that they Hpawn. This way, when general models of algorithms are sought later, the reader is in a better position to appreciate the scope of the quest, and to judge its success. Algorithms playa

(18)

4 I ntrod uction

major role in our exposition of complexity as well, because there is no better way to appreciate a complex problem than to contrast it with another, amenable to an efficient algorithm. The last chapter culminates in a Hection on coping with NP-completeness, where we present an array of algorithmic techniques that have been successfully used in attacking NP-complete problems (approximation algorithms, exhaustive algorithms, local search heuristics, and so on).

Computation is eSHential, powerful, beautiful, challenging, ever-expanding -and so is its theory. This book only tells the beginning of an exciting Htory.

It is a modest introduction to a few basic and carefully selected topics from the treasure chest of the theory of computation. We hope that it will motivate its readerH to seek out more; the references at the end of each chapter point to good places to start.

(19)

1 Sets, Relations, a nd La nguages

1.1 SETS

They say that mathematics is the language of science - it is certainly the language of the theory of computation, the scientific discipline we shall be studying in this book. And the language of mathematics deals with sets, and the com- plex ways in which they overlap, intersect, and in fact take part themselves in forming new sets.

A set is a collection of objects. For example, the collection of the four letters a, b, c, and d is a set, which we may name L; we write L

=

{a, b, c, d}. The objects comprising a set are called its elements or members. For example, b is an element of the set L; in symbols, bEL. Sometimes we simply say that b is in L, or that L contains b. On the other hand, z is not an element of L, and we write z ~ L.

In a set we do not distinguish repetitions of the elements. Thus the set {red, blue, red} is the same set as {red, blue}. Similarly, the order of the elements is immaterial; for example, {3, 1, 9}, {9, 3,1}, and {I, 3, 9} are the same set. To summarize: Two sets are equal (that is, the same) if and only if they have the same elements.

The elements of a set need not be related in any way (other than happening to be all members of the same set); for example, {3, red, {d, blue}} is a set with three clements, one of which is itself a set. A set may have only one element;

it is then called a singleton. For example, {I} is the set with 1 as its only element; thus {1} and 1 are quite different. There is also a set with no element at all. Naturally, there can be only one such set: it is called the empty set, and is denoted by

0.

Any set other than the empty set is said to be nonempty.

So far we have specified sets by simply listing all their elements, separated by commas and included in braces. Some sets cannot be written in this way, 5

(20)

6 Chapter 1: SETS. RELATIONS. AND LANGUAGES because they are infinite. For example, the set N of natural numbers is infinite;

we may suggest its elements by writing N = {D, 1, 2, ... }, using the three dots and your intuition in place of an infinitely long list. A set that is not infinite is finite.

Another way to specify a set is by referring to other sets and to properties that elements mayor may not have. Thus if I = {l, 3, 9} and G

=

{3,9}, G may be described as the set of elements of I that are greater than 2. We write this fact as follows.

G = {x : x E I and x is greater than 2}.

In general, if a set A has been defined and P is a property that elements of A mayor may not have, then we can define a new set

B = {x: x E A and x has property

Pl.

As another example, the set of odd natural numbers is 0= {x : x ^EN and x is not divisible by 2}.

A set A is a subset of a set B --in symbols, A ~ B- if each element of A is also an element of B. Thus 0 ~ N, since each odd natural number is a natural number. Note that any set is a subset of itself. If A is a subset of B but A is not the same as B, we say that A is a proper subset of B and write A C B. Also note that the empty set is a subset of every set. For if B is any set, then

0

^~B, since each element of

0

(of which there are none) is also an element of B.

To prove that two sets A and B are equal, we may prove that A ~ Band B ~ A. Every element of A must then be an element of B and vice versa, so that A and B have the same elements and A

=

B.

Two sets can be combined to form a third by various set operations, just as numbers are eombined by arithmetic operations such as addition. One set operation is union: the union of two sets is that set having as elements the objects that are elements of at least one of the two given sets, and possibly of both. We use the symbol U to denote union, so that

A U B = {x : x E A or x E B}.

For example,

{l, 3, 9}

u

{3, 5, 7} = {I, 3, 5, 7, 9}.

The intersection of two sets is the eollection of all elements the two sets have in common; that is,

An B = {x : x E A and x E B}.

(21)

1.1: Sets 7 For example,

{I, 3, 9}

n

{3, 5, 7} = {3}, and

{1,3,9}

n

{a,b,c,d} =

0.

Finally, the difference of two sets A and B, denoted by A - B, is the set of all elements of A that are not elements of B.

A - B = {x: x E A and x ~ B}.

For example,

{I, 3, 9} - {3, 5, 7} = {I, 9}.

Certain properties of the set operations follow easily from their definitions.

For example, if A, B, and C are sets, the following laws hold.

Idempotency Commutativity Associativity Distributivity Absorption DeMorgan's laws

AUA=A AnA=A AUB=BUA AnB=BnA

(A U B) U C = AU (B U C) (A

n

B)

n C

=

An

(B

n C)

(A U B)

n

C = (A

n

C) U (B

n

C) (A

n

B) U C = (A U C)

n

(B U C) (A U B)

nA =

^A

(AnB) UA = A

A - (B U C) = (A - B) n (A - C) A - (B n C) = (A - B) U (A - C)

Example 1.1.1: Let us prove the first of De Morgan's laws. Let L = A - (B U C)

and

R = (A - B)

n

(A - C);

we are to show that L

=

R. We do this by showing (a) L ~ R and (b) R ~ L.

(a) Let x be any element of L; then x E A, but x ~ B and x ~ C. Hence x is an element of both A - B and A - C, and is thus an element of R.

Therefore L ~ R.

( b) Let x E R; then x is an element of both A - B and A - C, and is therefore in A but in neither B nor C. Hence x E A but x ~ B U C, so x E L.

(22)

8 Chapter 1: SETS, RELATIONS, AND LANGUAGES Therefore R ~ L, and we have established that L = R.O

Two sets are disjoint if they have no element in common, that is, if their intersection is empty.

It is possible to form intersections and unions of more than two sets. If S is any collection of sets, we write

U

S for the set whose elements are the elements of all the sets in S. For example, if S = {{a, b}, {b, c}, {c, d}} then US = {a, b, c, d}; and if S = {{ n} : n EN}, that is, the collection of all the singleton sets with natural numbers as elements, then US = N. In general,

u

^S

⁼

^{{x : x}^E^Pfor some set PES}.

Similarly,

n

^S

⁼

^{{x : x}^E^Pfor each set PES}.

The collection of all subsets of a set A is itself a set, called the power set of A and denoted 2A. For example, the subsets of {c, d} are {c, d} itself, the singletons {c} and {d} and the empty set

0,

so

2{c,d} = {{c,d},{c},{d},0}.

A partition of a non empty set A is a subset II of 2A such that

0

is not an element of II and such that each element of A is in one and only one set in II.

That is, II is a partition of A if II is a set of subsets of A such that (1) each element of II is nonempty;

(2) distinct members of II are disjoint;

(3) UII = A.

For example, {{a, b}, {c}, {d}} is a partition of {a, b, c, d}, but {{b, c}, {c, d} } is not. The sets of even and odd natural numbers form a partition of N.

Problems for Section 1.1

1.1.1. Determine whether each of the following is true or false.

(a)

0

~

0

(b)

0

E

0

(c) 0 E {0}

(d)

0

~

{0}

(e) {a,b} E {a,b,c,{a,b}}

(f) {a, b} ~ {a, b, {a, b}}

(g) {a,b} ~ 2{a,b,{a,b}}

(h) {{a, b}} E 2{a,b,{a,b}}

(i) {a,b,{a,b}}-{a,b}={a,b}

(23)

1.2: Relations and Functions 9 1.1.2. '\'hat are these sets? Write them using braces, commas, and numerals only.

(a) ({1,3,5}U{3,1})n{3,5,7}

(b) U{ {3}, {3, 5},

n{

{5, 7}, {7, 9}}}

(c) ({1,2,5} - {5, 7,9})U ({5, 7,9} - {l,2,5}) (d) 2{7,8,9} _ 2{7,9}

(e) 2⁰

1.1.3. Prove each of the following.

(a) AU(BnC)=(AUB)n(AUC)

(b) An (B U C) = (A n B) U (A n C) (c) A n (A U B) = A

(d) AU(AnB)=A

(e) A - (B

n

C) = (A - B) U (A - C) 1.1.4. Let S = {a, b, c, d}.

(a) What partition of S has the fewest members? The most members?

(b) List all partitions of S with exactly two members.

liiJ RELATIONS AND FUNCTIONS

Mathematics deals with statements about objects and the relations between them. It is natural to say, for example, that "less than" is a relation between objects of a certain kind -namely, numbers- which holds between 4 and 7 but does not hold between 4 and 2, or between 4 and itself. But how can we express relations between objects in the only mathematical language we have available at this point -that is to say, the language of sets? We simply think of a relation as being itself a set. The objects that belong to the relation are, in essence, the combinations of individuals for which that relation holds in the intuitive sense.

So the less-than relation is the set of all pairs of numbers such that the first number is less than the second.

But we have moved a bit quickly. In a pair that belongs to a relation, we need to be able to distinguish the two parts of the pair, and we have not explained how to do so. We cannot write these pairs as sets, since {4, 7} is the same thing as {7, 4}. It is easiest to introduce a new device for grouping objects called an ordered pair.

t

We write the ordered pair of two objects a and b as (a, b); a and b are called the components of the ordered pair (a, b). The ordered pair (a, b) is not the same as the set {a,b}. First, the order matters: (a,b) is different from (b,a),

t

True fundamentalists would see the ordered pair (a, b) not as a new kind of object,

but as identical to {a, {a,b}}.

(24)

10 Chapter 1: SETS, RELATIONS, AND LANGUAGES whereas {a, b} = {b, a}. Second, the two components of an ordered pair need not be distinct; (7,7) is a valid ordered pair. Note that two ordered pairs (a, b) and (c, d) are equal only when a

=

^{c and}^b

=

^d.

The Cartesian product of two sets A and B, denoted by A x B, is the set of all ordered pairs (a, b) with a E A and b E B. For example,

{ 1, 3, 9} x {b, c, d} = {( 1, b), (1, c), (1, d), (3, b), (3 , c), (3, d), (9, b), (9 , c), (9, d)} . A binary relation on two sets A and B is a subset of Ax B. For example, {(1,b),(1,c),(3,d),(9,d)} is a binary relation on {1,3,9} and {b,c,d}. And {(i,j) : i,j EN and i

<

j} is the less-than relation; it is a subset of N x N ---often the two sets related by a binary relation are identical.

More generally, let n be any natural number. Then if aI, ... , an are any n objects, not necessarily distinct, (al"'" an) is an ordered tuple; for each i = 1, ...

,n,

ai is the ith component of (al, ... ,an). An ordered m-tuple (bl, ... ,bm), where m is a natural number, is the same as (al, ... ,an) ^{if and} only ifm = nand ai = bi , for i = 1, ... ,n. Thus (4,4), (4,4,4), ((4,4),4), and (4, (4,4)) are all distinct. Ordered 2-tuples are the same as the ordered pairs discussed above, and ordered 3-, 4-, 5-, and 6-tuples are called ordered triples, quadruples, quintuples, and sextuples, respectively. On the other hand, a sequence is an ordered n-tuple for some unspecified n (the length of the sequence). If AI, ... , An are any sets, then the n-fold Cartesian product Al x '" x An is the set of all ordered n-tuples (al, ... , an), with ai E Ai, for each i = 1, ... , n. In case all the Ai, are the same set A, the n-fold Cartesian product A x ... x A of A with itself is also written An. For example, N²is the set of ordered pairs of natural numbers. An n-ary relation on sets AI, ... , An is a subset of Al x ... x An; 1-, 2-, and 3-ary relations are called unary, binary, and ternary relations, respectively.

Another fundamental mathematical idea is that of a function. On the intuitive level, a function is an association of each object of one kind with a unique object of another kind: of persons with their ages, dogs with their owners, numbers with their successors, and so on. But by using the idea of a binary relation as a set of ordered pairs, we can replace this intuitive idea by a concrete definition. A function from a set A to a set B is a binary relation R on A and B with the following special property: for each element a E A, there is exactly one ordered pair in R with first component a. To illustrate the definition, let C be the set of cities in the United States and let 51 be the set of states; and let

RI = {(x,y) : x E C,y E 51, and x is a city in state y}, R2 = {(x,y): x E S,y E C, and y is a city in state x}.

(25)

1.2: Relations and Functions 11 Then Rl is a function, since each city is in one and only one state, but R2 is not a function, since some states have more than one city.t

In general, we use letters such as

I,

^g,and h for functions and we write I : A I-t B to indicate that I is a function from A to B. We call A the domain of

f.

If a is any element of A we write I(a) for that element b of B such that (a, b) E

I;

since

I

is a function, there is exactly one b E B with this property, so I(a) denotes a unique object. The object I(a) is called the image of a under

f.

To specify a function

I :

A I-t B, it suffices to specify

I

(a) for each a E A;

for example, to specify the function RI above, it suffices to specify, for each city, the state in which it is located. If I : A I-t B and AI is a subset of A, then we define J[A'l

=

{/(a) : a E A'} (that is, {b: b

=

I(a) for some a E A'}). We call J[A'l the image of AI under

f.

The range of

I

is the image of its domain.

Ordinarily, if the domain of a function is a Cartesian product, one set of parentheses is dropped. For example, if

I :

N x N I-t N is defined so that the image under

I

of an ordered pair (m, n) is the sum of m and n, we would write I(m, n)

=

m+n rather than I((m, n))

=

m+n, simply as a matter of notational convenience.

If I: Al x A2 ^{X ...}x An I-t B is a function, and I(al, ... ,an) = b, where ai ^E Ai for i = 1, ... , nand b ^E B, then we sometimes call al,··., an the arguments of I and b the corresponding value of I. Thus I may be specified by giving its value for each n-tuple of arguments.

Certain kinds of functions are of special interest. A function

I :

A I-t B is one-to-one if for any two distinct elements a, a' ^EA, I(a) ^::f:- I(a' ). ^For example, if C is the set of cities in the United States, 5 is the set of states, and g : 5 I-t C is specified by

g( s)

=

the capital of state s

for each s E 5, then g is one-to-one since no two states have the same capital.

A function

I :

A I-t B is onto B if each element of B is the image under

I

of some element of A. The function g just specified is not onto C, but the function RI defined above is onto 5 since each state contains at least one city. Finally a mapping

I :

A I-t B is a bijection between A and B if it is both one-to-one and onto B; for example, if Co is the set of capital cities, then the function g: 5 I-t Co specified, as before, by

g( s) = the capital of state s is a bijection between 5 and Co.

t

We consider Cambridge, Massachusetts, and Cambridge, Maryland, not the same city, but different cities that happen to have the same name.

(26)

12 Chapter 1: SETS, RELATIONS, AND LANGUAGES The inverse of a binary relation R t:;;; A x B, denoted R-¹t:;;; B x A, is simply the relation {(b, a) : (a, b) E R}. For example, the relation R2 defined above is the inverse of R_{1 •}Thus, the inverse of a function need not be a function. In the case of Rl its inverse fails to be a function since some states have more than one city;

that is, there are distinct cities Cl and C2 such that Rl (Cl) = Rl (C2). A function

I :

A f-t B may also fail to have an inverse if there is some element b E B such that I(a) ::f:- b for all a E A. If I: A f-t B is a bijection, however, neither of these eventualities can occur, and 1-1 is a function -indeed, a bijection between B and A. Moreover I-l(f(a)) = a for each a E A, and l(f-l(b)) = b for each bE B.

When a particularly simple bijection between two sets has been specified, it is sometimes possible to view an object in the domain and its image in the range as virtually indistinguishable: the one may be seen as a renaming or a way of rewriting the other. For example, singleton sets and ordered I-tuples are, strictly speaking, different, but not much harm is done if we occasionally blur the distinction, because of the obvious bijection

I

such that

I ( {

a}) = (a) for any singleton {a}. Such a bijection is called a natural isomorphism; of course this is not a formal definition since what is "natural" and what distinctions can be blurred depend on the context. Some slightly more complex examples should make the point more clearly.

Example 1.2.1: For any three sets A, B, and C, there is a natural isomorphism of Ax B x C to (A x B) x C, namely

I(a,b,c) = ((a,b),c) for any a E A, bE B, and c E

C.O

Example 1.2.2: For any sets A and B, there is a natural isomorphism ¢ from

that is, the set of all binary relations on A and B, to the set

{I : I

is a function from A to 2B}.

Namely, for any relation R t:;;; A x B, let ¢(R) be that function

I :

A f-t 2B such that

I(a) = {b: bE Band (a,b) E R}.

For example, if S is the set of states and R t:;;; S x S contains any ordered pair of states with a common border, then the naturally associated function

I :

S f-t 2⁵is specified by

I

(s) = {s' : s' E Sand s' shares a border with s}.

0

(27)

1.3: Special Types of Binary Relations 13 Example 1.2.3: Sometimes we regard the inverse of a function

I :

A f--t B as a function even when

I

is not a bijection. The idea is to regard 1-1 t::;; B x A as a function from B to 2A, using the natural isomorphism described under Example 1.2.2. Thus 1-1 (b) is, for any b E B, the set of all a E A such that I(a) = b.

For example, if Rl is as defined above -the function that assigns to each city the state in which it is located- then Rll (s), where s is a state, is the set of all cities in that state.

If Q and R are binary relations, then their composition Q ⁰R, or simply QR, is the relation {(a, b) : for some c, (a, c) E Q and (c, b) E R}. Note that the composition of two functions

I :

A f--t Band g : B f--t C is a function h from A to C such that h(a) = g(f(a)) for each a E A. For example, if

I

is the function that assigns to each dog its owner and g assigns to each person his or her age, then log assigns to each dog the age of its owner.<>

Problems for Sectior 1.2

1.2.1. Write each of the following explicitly.

(a)

{I}

x

{1,2}

x

{1,2,3}

(b)

0

^x

{1,2}

(c) 2{1,2} x {I, 2}

1.2.2. Let R = {(a, b), (a, c), (c, d), (a, a), (b, a)}. What is R 0 R, the composition of R with itself? What is R-¹, the inverse of R? Is R, R ⁰R, or R-¹a function?

1.2.3. Let

I :

A f--t Band g : B f--t C. Let h : A f--t C be their composition. In each of the following cases state necessary and sufficient conditions on

I

and g for h to be as specified.

(a) Onto.

(b) One-to-one.

(c) A bijection.

1.2.4. If A and B are any sets, we write BA for the set of all functions from A to B. Describe a natural isomorphism between {O, l}A and 2A.

liiJ SPECIAL TYPES OF BINARY RELATIONS

Binary relations will be found over and over again in these pages; it will be helpful to have convenient ways of representing them and some terminology for discussing their properties. A completely "random" binary relation has no significant internal structure; but many relations we shall encounter arise out

(28)

14 Chapter 1: SETS, RELATIONS, AND LANGUAGES of specific contexts and therefore have important regularities. For example, the relation that holds between two cities if they belong to the same state has certain "symmetries" and other properties that are worth noting, discussing, and exploiting.

In this section we study relattons that exhibit these and similar regularities.

We shall deal only with binary relations on a set and itself. Thus, let A be a set, and R t::;: A x A be a relation

bn

^A.The relation R can be represented by a directed graph. Each element of A is represented by a small circle-what we call a node of the directed graph~ and an arrow is drawn from a to b if and only if (a, b) E R. The arrows are the edges of the directed graph. For example, the relation R = {(a,b),(b,a), (a,d),(d,c),(c,c),(c,a)} is represented by the graph in Figure 1-1. Note in particular the loop from c to itself, corresponding to the pair (c, c) E R. From a node of a graph to another there is either no edge, or one edge ~we do not allow "parallel arrows."

a b

~--~

d

Figure 1-1

There is no formal distinction between binary relations on a set A and di- rected graphs with nodes from A. We use the term directed graph when we want to emphasize that the set on which the relation is defined is of no independent interest to us, outside the context of this particular relation. Directed graphs, as well as the undirected graphs soon to be introduced, are useful as models and abstractions of complex systems (traffic and communication networks, computational structures and processes, etc.). In Section 1.6, and in much more detail in Chapters 6 and 7, we shall discuss many interesting computational problems arising in connection with directed graphs.

For another example of a binary relation/directed graph, the less-than-or- equal-to relation::; defined on the natural numbers is illustrated in Figure 1-2.

Of course, the entire directed graph cannot be drawn, since it would be infinite.

A relation R t::;: A x A is reflexive if (a, a) E R for each a E A. The directed graph representing a reflexive relation has a loop from each node to itself. For example, the directed graph of Figure 1-2 represents a reflexive relation, but that of Figure 1-1 does not.

A relation R t::;: A x A is symmetric if (b, a) E R whenever (a, b) E R.

(29)

1.3: Special Types of Binary Relations 15

Figure 1-2

In the corresponding directed graph, whenever there is an arrow between two nodes, there are arrows between those nodes in both directions. For example, the directed graph of Figure 1-3 represents a symmetric relation. This directed graph might depict the relation of "friendship" among six people, since whenever x is a friend of y, y is also a friend of x. The relation of friendship is not reflexive, since we do not regard a person as his or her own friend. Of course, a relation could be both symmetric and reflexive; for example, {(a, b) : a and b are persons with the same father} is such a relation.

a

^f ~:

Figure 1-3

A symmetric relation without pairs of the form (a, a) is represented as an undirected graph, or simply a graph. Graphs are drawn without arrowheads, combining pairs of arrows going back' and forth between the same nodes. For example, the relation shown in Figure 1-3 could also be represented by the graph in Figure 1-4.

A relation R is antisymmetric if whenever (a, b) E R and a and bare distinct, then (b, a) ~ R. For example, let P be the set of all persons. Then

{(a,b) : a,b E P and a is the father of b}

(30)

16 Chapter 1: SETS, RELATIONS, AND LANGUAGES

ao-r ^IC

10--1

^d

Figure 1-4

is antisymmetric. A relation may be neither symmetric nor antisymmetric; for example, the relation

{( a, b) ; a, b E P and a is the brother of b}

and the relation represelrted in Figure 1-1 are neither.

A binary relation R is transitive if whenever (a, b) E Rand (b, c) E R, then (a, c) E R. The relation

{(a, b) ; a, b E P and a is an ancestor of b}

is transitive, since if a is an ancestor of band b is an ancestor of c, then a is an ancestor of c. So is the less-than-or-equal relation. In terms of the directed graph representation, transitivity is equivalent to the requirement that whenever there is a sequence of arrows leading from an element a to an element z, there is an arrow directly from a to z. For example, the relation illustrated in Figure 1-5 is transitive.

aK77(

b~C

Figure 1-5

A relation that is reflexive, symmetric, and transitive is called an equivalence relation. The representation of an equivalence relation by an undirected graph consists of a number of clusters; within each cluster, each pair of nodes is connected by a line (see Figure 1-6). The "clusters" of an equivalence relation are called its equivalence classes. We normally write [aJ for the equivalence class containing an element a, provided the equivalence relation R is under- stood by the context. That is, [aJ = {b ; (a, b) E R}, or, since R is symmetric, [aJ = {b ; (b, a) E R}. For example, the equivalence relation in Figure 1-6 has three equivalence classes, one with four elements, one with three elements, and one with one element.

(31)

Q

Figure 1-6

Theorem 1.3.1: Let R be an equivalence relation on a nonempty set A. Then the equivalence classes of R constitute a partition of A.

Proof: Let II = {raj : a E A}. We must show that the sets in II are non empty, disjoint, and together exhaust A. All equivalence classes are nonempty, since a E [aJ for all a E A, by reflexivity. To show that they are disjoint, consider any two distinct equivalence classes [aJ and [bJ, and suppose that [aJ

n

[bJ

f::. 0.

Thus there is an element c such that c E [aJ and c E [bJ. Hence (a, c) E Rand (c, b) E R; since R is transitive, (a, b) E R; and since R is symmetric, (b, a) E R.

But now take any element dE raj; then (d, a) E R and, by transitivity, (d, b) E R.

Hence d E [b], so that [aJ <:;;; [bJ. Likewise [bJ <:;;; raj. Therefore [aJ = [bJ. But this contradicts the assumption that [aJ and [bJ are distinct.

To see that

U

II = A, simply notice that each element a of A is in some set in II -namely, a E [aJ, by reflexivity . •

Thus starting from an equivalence relation R, we can always construct a corresponding partition II. For example, if

R = {( a, b) : a and b are persons and a and b have the same parents}, then the equivalence classes of R are all groups of siblings. Note that the con- struction of Theorem 1.3.1 can be reversed: from any partition, we can construct a corresponding equivalence relation. Namely, if II is a partition of A, then

R

= {(

a, b) : a and b belong in the same set of II}

is an equivalence relation. Thus there is a natural isomorphism between the set of equivalence relations on a set A and the set of partitions of A.

A relation that is reflexive, antisymmetric, and transitive is called a partial order. For example,

{ ( a, b) : a, b are persons and a is an ancestor of b}

(32)

18 Chapter 1: SETS, RELATIONS, AND LANGUAGES is a partial order (provided we consider eaeh person to be an ancestor of himself or herself). If R ~ A x A is a partial order, an element a E A is called minimal if the following is true: (b, a) E R only if a

=

^b. For example, in the ancestor relation defined above, Adam and Eve are the only minimal elements. A finite partial order must have at least one minimal element, but an infinite partial order need not have one.

A partial order R ~ A x A is a total order if, for all a, b E A, either ( a, b) E R or (b, a) E R. Thus the ancestor relation is not a total order since not any two people are ancestrally related (for example, siblings are not); but the less-than-or-equal-to relation on numbers is a total order. A total order cannot have two or more minimal elements.

A path in a binary relation R is a sequence (al, ... , an) for some n ;::: 1 such that (ai, ai+l) E R for i = 1, ... , n - 1; this path is said to be from al to an. The length of a path (al, ... , an) is n. The path (al, ... , an) is a cycle if the ai's are all distinct and also (an' al) E R.

Problems for Section 1.3

1.3.1. Let R

=

^{(a,^c),(c,e), (e, e), (e, b), (d, b), (d, d)}. Draw directed graphs representing each of the following.

(a) R (b) R-^I

(c) RUR-^I (d) Rn R-^I

1.3.2. Let Rand S be the binary relations on A

=

{I, ... , 7} with the graphical representations shown in the next page.

(a) Indicate whether each of Rand S is (i) symmetric, (ii) reflexive, and (iii) transitive.

(b) Repeat (a) for the relation R U S.

1.3.3. Draw directed graphs representing relations of the following types.

(a) Reflexive, transitive, and antisymmetric.

(b) Reflexive, transitive, and neither symmetric nor antisymmetric.

1.3.4. Let A be a nonempty set and let R ~ A x A be the empty set. Which properties does R have?

( a) Reflexi vi ty.

(b) Symmetry.

(c) Antisymmetry.

(d) Transitivity.

1.3.5. Let

f :

A f--t B. Show that the following relation R is an equivalence relation on A: (a, b) E R if and only if f(a)

=

f(b).

(33)

>---{3

®

1.3.6. Let R <:;:; A x A be a binary relation as defined below. In which cases is R a partial order'? a total order'?

(a) A = the positive integers; (a, b) E R if and only if b is divisible by a.

(b) A = N x N; ((a, b)(c, d)) E R if and only if a:::: cor b:::: d.

(c) A

=

N; (a, b) E R if and only if b

=

a or b

=

a

+

l.

(d) A is the set of all English words; ( a, b) E R if and only if a is no longer than b.

(e) A is the set of all English words; (a, b) E R if and only if a is the same as b or occurs more frequently than b in the present book.

(34)

20 Chapter 1: SETS, RELATIONS, AND LANGUAGES 1.3.7. Let Rl and R2 be any two partial orders on the same set.4. Show that

Rl

n

R2 is a partial order.

1.3.8. (a) Prove that if 5 is any collection of sets, then Rs = {(.4, B) : A, B E 5 and A ~ B} is a partial order.

(b) Let 5 = 2{1,2,3}. Draw a directed graph representing the partial order Rs defined in (a). Which are the minimal elements of Rs?

1.3.9. Under what circumstances does a directed graph represent a function?

1.3.10. Show that any function from a finite set to itself contains a cycle.

1.3.11. Let 5 be any set, and let P be the set of all partitions of 5. Let R be the binary relation on P such that (III, II2) E R if and only if for every 51 E III, there is an 52 E II2 such that 51 ~ 52; if (III, II2) E R we say that III refines II2. Show that R is a partial order on P. What elements of P are maximal and minimal? Suppose that P were an arbitrary collection of subsets of 2s , which need not be partitions of 5. Would R necessarily be a partial order?

G

FINITE AND INFINITE SETS

A basic property of a finite set is its size, that is, the number of elements it contains. Some facts about the sizes of finite sets are so obvious they hardly need proof. For example, if A ~ B, then the size of A is less than or equal to that of B; the size of A is zero if and only if A is the empty set.

However, an extension of the notion of "size" to infinite sets leads to dif- ficulties if we attempt to follow our intuition. Are there more multiples of 17 (0,17,34,51,68, ... ) than there are perfect squares (0,1,4,9,16, ... )? You are welcome to speculate on alternatives, but experience has shown that the only satisfactory convention is to regard these sets as having the same size.

We call two sets A and B equinumerous if there is a bijection

I :

A f-t

B. Recall that if there is a bijection

I :

A ^f-tB, then there is a bijection 1-1 : B ^f-tA; hence equinumerosity is a symmetrie relation. In fact, as is easily shown, it is an equivalence relation. For example, {8,red,{0,b}} and {1,2,3}

are equinumerous; let f(8) = 1, I(red) = 2, f( {0, b}) = 3. So are the multiples of17 and the perfect squares; a bijection is given by f(17n) = n²for each n E N.

In general, we call a set finite if, intuitively, it is equinumerous with {I, 2, ... , n} for some natural number n. (For n = 0, {I, ... , n} is the empty set, so (I) is finite, being equinumerous with itself.) If A and {1, ... ,n} are equinumer- ous, then we say that the cardinality of A (in symbols, IAI) is n. The cardinality of a finite set is thus the number of elements in it.

(35)

1.4: Finite and Infinite Sets 21 A set is infinite if it is not finite. For example, the set N of natural numbers is infinite; so are sets such as the set of integers, the set of reals, and the set of perfect squares. However, not all infinite sets are equinumerous.

A set is said to be countably infinite if it is equinumerous with N, and countable if it is finite or count ably infinite. A set that is not countable is uncountable. To show that a set A is count ably infinite we must exhibit a bijection / between A. and N; equivalently, we need only suggest a way in which A. can be enumerated as

and so on, since such an enumeration immediately suggests a bijection -just take /(0) = ao, /(1) = a1,'"

For example, we can show that the union of any finite number of countably infinite sets is count ably infinite. Let us only illustrate the proof for the case of three pairwise disjoint, count ably infinite sets; a similar argument works in general. Call the sets A, B, and C. The sets can be listed as above: A

=

{ao, a1, ... }, B

=

_{bo,b1 , . . . }, C

=

{co, C1, . . . }, Then their union can be listed as Au B

u

C = {ao, bo, ^Co,a1, b1 , C1, a2, ... }. This listing amounts to a way of

"visiting" all the elements in A U B

u

C by alternating between different sets, as illustrated in Figure 1-7. The technique of interweaving the enumeration of several sets is called "dovetailing" (for reasons that any carpenter can give after looking at Figure 1-7).

A

B

C

Figure 1-7

The same idea can be used to show that the union of a countably infinite collection of countably infinite sets is count ably infinite. For example, let us show that N x N is count ably infinite; note that N x N is the union of {O} x N, {I} x N, {2} x N, and so on, that is, the union of a count ably infinite collection of countably infinite sets. Dovetailing must here be more subtle than in the

ELEMENTS OF THE THEORY OF COMPUTATION

ELEMENTS OF THE THEORY OF

COMPUTATION

Second Edition

Harry R. Lewis

Gordon McKay Professor of Computer Science Harvard University

and Dean of Harvard College Cambridge, Massachusetts

Christos H. Papadimitriou

C. Lester Hogan Professor of Electrical Engineering and Computer Science

University of California Berkeley, California

PRENTICE-HALL, Upper Saddle River, New Jersey 07458

Contents

NP

Preface to the First Edition

Preface to the Second Edition

a

NP-

NP-

ELEMENTS OF THE THEORY OF

COMPUTATION

Introduction

0

(())O,

t

1 Sets, Relations, a nd La nguages

1.1 SETS

=

0.

=

Pl.

0

0

=

u

n

n

0.

n

n C

An

n C)

n

n

n

n

n

nA =

n

=

U

u

=

n

=

0,

0

0

0

0

0

0

{0}

n{

n

liiJ RELATIONS AND FUNCTIONS

t

t

=

=

<

,n,

I,

f.

I;

I

f.

I :

I

=

=

⁼

⁼

^f ~:

ao-r ^IC