Theory of Computation

1

Theory of Computation

2 Instructor: 顏嗣鈞 E-mail: [email protected] Web: http://www.ee.ntu.edu.tw/~yen Time: 2:20-5:10 PM, Tuesday Place: BL 112

Office hours: by appointment Class web page:

http://www.ee.ntu.edu.tw/~yen/courses/TOC-2006.htm

3 • :

Introduction to Automata Theory, Languages, and Computation John E. Hopcroft, Rajeev Motwani, Jeffrey D. Ullman, (2nd Ed. Addison-Wesley, 2001)

textbook

4

1st Edition

Introduction to

Automata Theory,

Languages, and

Computation

John E. Hopcroft, Jeffrey D. Ullman, (Addison-Wesley, 1979)

5

Grading

HW : 0-20%

Midterm exam.: 35-45%

Final exam.: 45-55%

6

Why Study Automata Theory?

Finite automata are a useful model for

important kinds of hardware and software:

• Software for designing and checking digital

circuits.

• Lexical analyzer of compilers.

• Finding words and patterns in large bodies of

text, e.g. in web pages.

• Verification of systems with finite number of

states, e.g. communication protocols.

7

Why Study Automata Theory? (2)

The study of Finite Automata and Formal

Languages are intimately connected.

Methods for specifying formal languages are

very important in many areas of CS, e.g.:

• Context Free Grammars are very useful when designing software that processes data with

recursive structure, like the parser in a compiler. • Regular Expressions are very useful for

specifying lexical aspects of programming languages and search patterns.

8

Why Study Automata Theory? (3)

Automata are essential for the study of

the limits of computation. Two issues:

• What can a computer do at all?

(

Decidability

)

• What can a computer do efficiently?

(

Intractability

)

9

Applications

Theoretical Computer Science

Automata Theory, Formal Languages, Computability, Complexity …

Co

m

pil

er

Pr og . la ng ua ge s Co m m . p ro to co ls

cir

cu

its

Pa tt er n re co gn itio n S up er vis or y co nt ro l Q ua nt um co m pu tin g Co m pu te r-A id ed Ve rif ica tio n

_...

10

Aims of the Course

•

To familiarize you with key Computer Science

concepts in central areas like

- Automata Theory - Formal Languages

- Models of Computation - Complexity Theory

•

To equip you with tools with wide applicability

in the fields of CS and EE, e.g. for

- Complier Construction - Text Processing

11

Fundamental Theme

• What are the capabilities and

limitations of computers and computer

programs?

– What can we do with

computers/programs?

– Are there things we cannot do with

computers/programs?

12

Studying the Theme

• How do we prove something CAN be done

by SOME program?

• How do we prove something CANNOT be

done by ANY program?

13

Example: The Halting Problem (1)

Consider the following program. Does it

terminate for all values of n  1?

while (n > 1) {

if even(n) {

n = n / 2;

} else {

n = n * 3 + 1;

}

}

14

Example: The Halting Problem (2)

Not as easy to answer as it might first seem.

Say we start with n = 7, for example:

7, 22, 11, 34, 17, 52, 26, 13, 40, 20, 10, 5,

16, 8, 4, 2, 1

In fact, for all numbers that have been tried

(a lot!), it does terminate . . .

15

Example: The Halting Problem (3)

Then the following important undecidability

result should perhaps not come as a total

surprise:

It is impossible to write a program that

decides if another, arbitrary, program

terminates (halts) or not.

What might be surprising is that it is possible

to prove such a result. This was first done

by the British mathematician Alan Turing.

16

Our focus

Automata

Computability

17

Topics

1.

Finite automata, Regular languages, Regular

grammars

: deterministic vs. nondeterministic,

one-way vs. two-way finite automata,

minimization, pumping lemma for regular sets,

closure properties.

2.

Pushdown automata, Context-free languages,

Context-free grammars

: deterministic vs.

nondeterministic, one-way vs. two-way PDAs,

reversal bounded PDAs, linear grammars, counter

machines, pumping lemma for CFLs, Chomsky

normal form, Greibach normal form, closure

18

Topics (cont’d)

3.

Linear bounded automata,

Context-sensitive languages, Context-Context-sensitive

grammars.

4.

Turing machines, Recursively

enumerable sets, Type 0 grammars

:

variants of Turing machines, halting

problem, undecidability, Post

correspondence problem, valid and

invalid computations of TMs.

19

Topics (cont’d)

5.

Basic recursive function theory

6.

Basic complexity theory

:

Various

resource bounded complexity classes,

including NLOGSPACE, P, NP,

PSPACE, EXPTIME, and many more.

reducibility, completeness.

7.

Advanced topics

:

Tree Automata,

quantum automata, probabilistic

automata, interactive proof systems,

oracle computations, cryptography.

20

Who should take this course?

21

Languages

The terms

language and word are used in

a strict technical sense in this course

:

A language is a set of words.

A word is a sequence (or string) of

symbols.

 (or ) denotes the empty word, the

sequence of zero symbols.

22

Symbols and Alphabets

• What is a symbol, then?

• Anything, but it has to come from an

alphabet which is a finite set.

• A common (and important) instance is

= {0, 1}.

 , the empty word, is never an symbol

of an alphabet.

23

Computation

24 CPU input memory output memory Program memory temporary memory

25 CPU input memory output memory Program memory temporary memory

3

)

(

x

x

f



compute

x



x

compute

x

2



x

Example:

26 CPU input memory output memory Program memory temporary memory

3

)

(

x

x

f



compute

x



x

compute

x

2



x

2



x

27 CPU input memory output memory Program memory temporary memory

f

(

x

)



x

3

compute

x



x

compute

x

2



x

2



x

4

2

*

2





z

8

2

*

)

(

x

 z



f

28 CPU input memory output memory Program memory temporary memory

f

(

x

)



x

3

compute

x



x

compute

x

2



x

2



x

4

2

*

2





z

8

2

*

)

(

x

 z



f

8

)

(

x



f

29

Automaton

CPU input memory output memory Program memory temporary memory Automaton

30

Different Kinds of Automata

Automata are distinguished by the temporary memory

• Finite Automata: no temporary memory

• Pushdown Automata: stack

31 input memory output memory temporary memory Finite Automaton

Finite Automaton

Example: Vending Machines

32 input memory output memory Stack Pushdown Automaton

Pushdown Automaton

Example: Compilers for Programming Languages (medium computing power)

33

input memory output memory Random Access Memory

Turing Machine

Turing Machine

Examples: Any Algorithm

34 Finite Automata Pushdown Automata Turing Machine

Power of Automata

Less power More power

Solve more

35

Mathematical Preliminaries

36

Mathematical Preliminaries

• Sets • Functions • Relations • Graphs • Proof Techniques

37

}

3

,

2

,

1

{



A

A set is a collection of elements

SETS

}

,

,

,

{

train

bus

bicycle

airplane

B



We write

A



1

B

ship



38

Set Representations

C = { a, b, c, d, e, f, g, h, i, j, k }

C = { a, b, …, k }

S = { 2, 4, 6, … }

S = { j : j > 0, and j = 2k for some k>0 }

S = { j : j is nonnegative and even }

finite set

infinite set

39

A = { 1, 2, 3, 4, 5 }

Universal Set: all possible elements

U = { 1 , … , 10 } 1 2 3 4 ₅

A

U

6 7 8 9 10

40

Set Operations

A = { 1, 2, 3 } B = { 2, 3, 4, 5} • Union A U B = { 1, 2, 3, 4, 5 } • Intersection A B = { 2, 3 } • Difference A - B = { 1 } B - A = { 4, 5 } U A B A-B

41

A

• Complement Universal set = {1, …, 7} A = { 1, 2, 3 } A = { 4, 5, 6, 7} 1 2 3 4 5 6 7

A

A = A

42 0 2 4 6 1 3 5 7 even

{ even integers } = { odd integers }

odd

43

DeMorgan’s Laws

A U B = A BU

44

Empty, Null Set:

= { } S U = S S = S - = S - S = U = Universal Set

45

Subset

A = { 1, 2, 3} B = { 1, 2, 3, 4, 5 } A BU Proper Subset: A BU A B

46

Disjoint Sets

A = { 1, 2, 3 } B = { 5, 6} A B = U

47

Set Cardinality

• For finite sets A = { 2, 5, 7 }

48

Powersets

A powerset is a set of sets

Powerset of S = the set of all the subsets of S

S = { a, b, c }

2S = { , {a}, {b}, {c}, {a, b}, {a, c}, {b, c}, {a, b, c} }

49

Cartesian Product

A = { 2, 4 } B = { 2, 3, 5 }

A X B = { (2, 2), (2, 3), (2, 5), ( 4, 2), (4, 3), (4, 5) }

|A X B| = |A| |B|

Generalizes to more than two sets

50

FUNCTIONS

domain 1 2 3 a b c range f : A -> B A B If A = domain

then f is a total function

otherwise f is a partial function

51

RELATIONS

R = {(x₁, y₁), (x₂, y₂), (x₃, y₃), …} x_i R y_i e. g. if R = ‘>’: 2 > 1, 3 > 2, 3 > 1

52

Equivalence Relations

• Reflexive: x R x • Symmetric: x R y y R x • Transitive: x R y and y R z x R z Example: R = ‘=‘ • x = x • x = y y = x • x = y and y = z x = z

53

Equivalence Classes

For equivalence relation R

equivalence class of x = {y : x R y} Example: R = { (1, 1), (2, 2), (1, 2), (2, 1), (3, 3), (4, 4), (3, 4), (4, 3) } Equivalence class of 1 = {1, 2} Equivalence class of 3 = {3, 4}

54

GRAPHS

A directed graph • Nodes (Vertices) V = { a, b, c, d, e } • Edges

E = { (a,b), (b,c), (b,e),(c,a), (c,e), (d,c), (e,b), (e,d) }

node edge a b c d e

55

Labeled Graph

a b c d e 1 ₃ 5 6 2 6 2

56

Walk

a b c d e

Walk is a sequence of adjacent edges

57

Path

a b c d e

Path is a walk where no edge is repeated Simple path: no node is repeated

58

Cycle

a b c d e 1 2 3

Cycle: a walk from a node (base) to itself

Simple cycle: only the base node is repeated

59

Euler Tour

a b c d e 1 2 3 4 5 6 7 8 _base

60

Hamiltonian Cycle

a b c d e 1 2 3 4 5 _base

61

Trees

root leaf parent child

62 root leaf Level 0 Level 1 Level 2 Level 3 Height 3

63

64

PROOF TECHNIQUES

• Proof by induction

65

Induction

We have statements P₁, P₂, P₃, …

If we know

• for some b that P₁, P₂, …, P_b are true • for any k >= b that

P₁, P₂, …, P_k imply P_k+1

Then

66

Proof by Induction

• Inductive basis

Find P₁, P₂, …, P_b which are true

• Inductive hypothesis

Let’s assume P₁, P₂, …, P_k are true, for any k >= b

• Inductive step

67

Example

Theorem: A binary tree of height n has at most 2n leaves.

Proof by induction:

let L(i) be the number of leaves at level i

L(0) = 1 L(1) = 2 L(2) = 4 L(3) = 8

68

We want to show: L(i) <= 2i

• Inductive basis

L(0) = 1 (the root node)

• Inductive hypothesis

Let’s assume L(i) <= 2i for all i = 0, 1, …, k

• Induction step

69

Induction Step

From Inductive hypothesis: L(k) <= 2k

Level k k+1

70 L(k) <= 2k Level k k+1 L(k+1) <= 2 * L(k) <= 2 * 2k = 2k+1

Induction Step

71

Remark

Recursion is another thing

Example of recursive function:

f(n) = f(n-1) + f(n-2)

72

Proof by Contradiction

We want to prove that a statement P is true • we assume that P is false

• then we arrive at an incorrect conclusion

73

Example

Theorem: is not rational

Proof:

Assume by contradiction that it is rational

= n/m

n and m have no common factors

We will show that this is impossible

2

74

= n/m 2 m2 = n2

Therefore, n2 is even n is even

n = 2 k

2 m2 = 4k2 m2 = 2k2 m is even

m = 2 p Thus, m and n have common factor 2

Contradiction!