Algorithm Design and Analysis NP Completeness (1)
Yun-Nung (Vivian) Chen
http://ada.miulab.tw
Outline
• Decision Problems v.s. Optimization Problems
• Complexity Classes
• P v.s. NP
• NP, NP-Complete, NP-Hard
• Polynomial-Time Reduction
Algorithm Design & Analysis
• Design Strategy
• Divide-and-Conquer
• Dynamic Programming
• Greedy Algorithms
• Graph Algorithms
• Analysis
• Amortized Analysis
• NP-Completeness
3
Polynomial Time Algorithms
• For an input with size n, the worst-case running time is for some constant k
• Problems that are solvable by polynomial-time algorithms as being tractable, easy, or efficient
• Problems that require superpolynomial time as being intractable, or hard, or inefficient
Four Color Problem
• Use total four colors s.t. the neighboring parts have different colors
5
Four Color Problem (after 100 yrs)
• Finally proven (with the help of computers) by Kenneth Appel and Wolfgang Haken in 1976
• Their algorithm runs in O(n2) time
• First major theorem proved by a computer
• Open problems remain...
• Linear time algorithms to find a solution
• Concise, human-checkable, mathematical proofs
Planar k-Colorability
• Given a planar graph G (e.g., a map), can we color the vertices with k colors such that no adjacent vertices have the same color?
• k = 1?
• k = 2?
• k = 3?
• k ≥ 4?
7
How hard is it when k = 3?
Can we know its level of difficulty before solving it?
Planar k-Colorability
Decision Problems v.s.
Optimization Problems
9
Decision Problems
• Definition: the answer is simply “yes” or “no” (or “1” or “0”)
• MST: Given a graph 𝐺 = 𝑉, 𝐸 and a bound 𝐾, is there a spanning tree with a cost at most 𝐾?
• KNAPSACK: Given a knapsack of capacity 𝐶, a set of objects with weights and values, and a target value 𝑉, is there a way to fill the knapsack with at least 𝑉 value?
Optimization Problems
• Definition: each feasible solution has an associated value, and we wish to find a feasible solution with the best value (maximum or minimum)
• MST-OPT: Given a graph 𝐺 = 𝑉, 𝐸 , find the minimum spanning tree of 𝐺
• KNAPSACK-OPT: Given a knapsack of capacity 𝐶 and a set of objects with weights and values, fill the knapsack so as to maximize the total value
11
Which is Easier? Why?
How to convert an optimization problem to a related decision problem?
Imposing a (lower or upper) bound on the value to be optimized
Difficulty Levels
• Every optimization problem has a decision version that is no harder than the optimization problem.
• Using Aopt to solve Adec
• check if the optimal value ≤ k, constant overhead
• Using Adec to solve Aopt
• apply binary search on the value range, logarithmic overhead
13
Aopt: given a graph, find the length of the shortest path
Adec: given a graph, determine whether there is a path ≤ k
P v.s. NP
Textbook Chapter 34 – NP-Completeness
Algorithm Design
• Algorithmic design methods to solve problems efficiently (polynomial time)
• Divide and conquer
• Dynamic programming
• Greedy
• “Hard” problems without known efficient algorithms
• Hamilton, knapsack, etc.
15
Complexity Classes
• Can we decide whether a problem is “too hard to solve” before investing our time in solving it?
• Idea: decide which complexity classes the problem belongs to via reduction
• 已知問題A很難。若能證明問題B至少跟A一樣難,那麼問題B也很難。
To Solve v.s. Not to Solve
• Algorithm design
• Design algorithms to solve computational problems
• Mostly concerned with upper bounds on resources
17
• Complexity theory
• Classify problems based on their difficulty and identify relationships between classes
• Mostly concerned with lower bounds on resources
upper bound
Problem
Problem B
Problem A
Problem B is no easier than A
lower bound
Complexity Classes
• A complexity class is “a set of problems of related resource-based complexity”
• Resource = time, memory, communication, ...
• Focus: decision problems and the resource of time
P
• The class P consists of all the problems that can be solved in polynomial time.
• Sorting
• Exact string matching
• Primes
• …
• Polynomial time algorithm
• For inputs of size n, their worst-case running time is for some constant k
19
NP
• NP consists of the problems that can be solved in non-deterministically polynomial time.
• NP consists of the problems that can be “verified” in polynomial time.
• P consists of the problems that can be solved in (deterministically) polynomial time.
Non-Deterministic
Deterministic Algorithm
21
initial
configuration
Non-Deterministic Algorithm
initial
configuration
Non-Deterministic Bubble Sort
23
This is not a randomized algorithm.
Non-Deterministic-Bubble-Sort(n) for i = 1 to n
for j = 1 to n – 1
if A[j] < A[i+1] then
Either exchange A[j] and A[i+1] or do nothing
Vertex Cover Problem (路燈問題)
• Input: a graph G
• Output: a smallest vertex subset of G that covers all edges of G.
• Known to be NP-complete
Illustration
25
Vertex Cover (Decision Version)
• Input: a Graph G and an integer k.
• Output: Does G contain a vertex cover of size no more than k?
• Original problem → optimization problem
• 原先的路燈問題是要算出放路燈的方法
• Yes/No → decision problem
• 問k盞路燈夠不夠照亮整個公園
Non-Deterministic Algorithm
27
Non-Deterministic-Vertex-Cover(G, k) set S = {}
for each vertex x of G
non-deterministically insert x to S if |S| > k
output no
if S is not a vertex cover output no
output yes
Algorithm Correctness
• If the correct answer is yes, then there is a computation path of the algorithm that leads to yes.
• 至少有一條路是對的
• If the correct answer is no, then all computation paths of the algorithm lead to no.
Non-Deterministic-Vertex-Cover(G, k) set S = {}
for each vertex x of G
non-deterministically insert x to S if |S| > k
output no
if S is not a vertex cover output no
output yes
Non-Deterministic Problem Solving
29
initial
configuration
correct answer
Non-Deterministic Polynomial
polynomial
“solved” in non-deterministic polynomial time
P ⊆ NP or NP ⊆ P?
• P ⊆ NP
• A problem solvable in polynomial time is verifiable in polynomial time as well
• Any NP problem can be solved in (deterministically) exponential time?
• Yes
• Any NP problem can be solved in (deterministically) polynomial time?
• Open problem
31
Why?
US$1,000,000 Per Problem
• http://www.claymath.org/millennium-problems
Millennium Problems
• Yang–Mills and Mass Gap
• Riemann Hypothesis
• P vs NP Problem
• Navier–Stokes Equation
• Hodge Conjecture
• Poincaré Conjecture (solved by Grigori Perelman)
• Birch and Swinnerton-Dyer Conjecture
33
Grigori Perelman
Fields Medal (2006), declined Millennium Prize (2010), declined
Vinay Deolalikar
• Aug 2010 claimed a proof of P is not equal to NP.
If P = NP
• problems that are verifiable → solvable
35
• public-key cryptography will be broken
Widespread belief in P ≠ NP
“If P = NP, then the world would be a profoundly different place than we usually assume it to be. There would be no special value in “creative leaps,” no fundamental gap between solving a problem and recognizing the solution once it's found. Everyone who could appreciate a symphony would be Mozart; everyone who could follow a step-by-step argument would be Gauss...” – Scott Aaronson, MIT
Travelling Salesman (2012)
NP, NP-Complete, NP-Hard
37
NP-Hardness
• A problem is NP-hard if it is as least as hard as all NP problems.
• In other words, a problem X is NP-hard if the following condition holds:
• If X can be solved in (deterministic) polynomial time, then all NP problems can be solved in (deterministic) polynomial time.
NP-Completeness (NPC)
• A problem is NP-complete if
• it is NP-hard and
• it is in NP.
• In other words, an NP-complete problem is one of the “hardest” problems in the class NP.
• In other words, an NP-complete problem is a hardness representative problem of the class NP.
• Hardest in NP → solving one NPC can solve all NP problems (“complete”)
• It is wildly believed that NPC problems have no polynomial-time solution
→ good reference point to judge whether a problem is in P
• We can decide if a problem is “too hard to solve” by showing it is as hard as an NPC problem
• We then focus on designing approximate algorithms or solving special cases
39
Complexity Classes
• Class P: class of problems that can be solved in
• Class NP: class of problems that can be verified in
• Class NP-hard: class of problems that are “at least as hard as all NP problems”
• Class NP-complete: class of problems in both NP and NP-hard
More Complexity Classes
41
undecidable: no algorithm; e.g.
halting problem
https://www.youtube.com/watch?v=wGLQiHXHWNk
An Undecidable Problem – Halting Problem
• Halting problem is to determine whether a program 𝑝 halts on input 𝑥
• Proof for undecidable via a counterexample
• Suppose ℎ can determine whether a program 𝑝 halts on input 𝑥
• ℎ(𝑝, 𝑥) = return (p halts on input x)
• Define g(p) = if h(p,p) is 0 then return 0 else HANG
• → g(g) = if h(g,g) is 0 then return 0 else HANG
• Both cases contradict the assumption:
1.g halts on g: then h(g,g)=1, which would make g(g) hang
2.g does not halt on g: then h(g,g)=0, which would make g(g) halt
43
Complexity Classes
• Which one is in P?
Shortest Simple Path Longest Simple Path
Euler Tour Hamitonian Cycle
LCS with 2 Input Sequences LCS with Arbitrary Input Sequences Degree-Constrained Spanning Tree Minimal Spanning Tree
Candy Crush is NP-Hard
45
Sudoku is NPC
46
Minesweeper Consistency is NPC
• Minesweeper Consistency: Given a state of what purports to be a Minesweeper games, is it logically consistent?
47
Polynomial-Time Reduction
Textbook Chapter 34.3 – NP-completeness and reducibility
First NP-Complete Problem – SAT (Satisfiability)
• Input: a Boolean formula with variables
• Output: whether there is a truth assignment for the variables that satisfies the input Boolean formula
49
• Stephan A. Cook [FOCS 1971] proved that
▪ SAT can be solved in non-deterministic polynomial time → SAT ∈ NP
▪ If SAT can be solved in deterministic polynomial time, then so can any NP problems → SAT ∈ NP-hard
Reduction
• Problem A can be reduced (in polynomial time) to Problem B
= Problem B can be reduced (in polynomial time) from Problem A
• We can find an algorithm that solves Problem B to help solve Problem A
• If problem B has a polynomial-time algorithm, then so does problem A What is the complexity of Algorithm for A?
Algorithm for B
? Instance 𝛽 of B Answer for 𝛽 ? Answer for 𝛼 Instance 𝛼 of A
Algorithm for A
Reduction
• A reduction is an algorithm for transforming a problem instance into another
• Definition
• Reduction from A to B implies A is not harder than B
• A ≤p B if A can be reduced to B in polynomial time
• Applications
• Designing algorithms: given algorithm for B, we can also solve A
• Classifying problems: establish relative difficulty between A and B
• Proving limits: if A is hard, then so is B
51
Algorithm to decide B Reduction
Algorithm
Instance 𝛽 of B Yes
Instance 𝛼 of A
Algorithm to decide A No
This is why we need it for proving NP-completeness!
Questions
• If A is an NP-hard problem and B can be reduced from A, then B is an NP- hard problem?
• If A is an NP-complete problem and B can be reduced from A, then B is an NP-complete problem?
• If A is an NP-complete problem and B can be reduced from A, then B is an NP-hard problem?
Problem Difficulty
• Q: Which one is harder?
• A: They have equal difficulty.
• Proof:
• PARTITION ≤p KNAPSACK
• KNAPSACK ≤p PARTITION
53
KNAPSACK: Given a set 𝑎1, … , 𝑎𝑛 of non-negative integers, and an integer 𝐾, decide if there is a subset 𝑃 ⊆
1, 𝑛 such that σ𝑖∈𝑃 𝑎𝑖 = 𝐾.
PARTITION: Given a set of 𝑛 non-negative integers
𝑎1, … , 𝑎𝑛 , decide if there is a subset 𝑃 ⊆ 1, 𝑛 such that σ𝑖∈𝑃 𝑎𝑖 = σ𝑖∉𝑃𝑎𝑖.
Polynomial-time reducible?
Polynomial-time reducible?
Polynomial Time Reduction
• PARTITION ≤p KNAPSACK
• If we can solve KNAPSACK, how can we use that to solve PARTITION?
• KNAPSACK ≤p PARTITION
KNAPSACK: Given a set 𝑎1, … , 𝑎𝑛 of non-negative integers, and an integer 𝐾, decide if there is a subset 𝑃 ⊆
1, 𝑛 such that σ𝑖∈𝑃 𝑎𝑖 = 𝐾.
PARTITION: Given a set of 𝑛 non-negative integers
𝑎1, … , 𝑎𝑛 , decide if there is a subset 𝑃 ⊆ 1, 𝑛 such that σ𝑖∈𝑃 𝑎𝑖 = σ𝑖∉𝑃𝑎𝑖.
Polynomial-time reducible?
Polynomial-time reducible?
PARTITION ≤ p KNAPSACK
• If we can solve KNAPSACK, how can we use that to solve PARTITION?
• Polynomial-time reduction
• Set
55
KNAPSACK: Given a set 𝑎1, … , 𝑎𝑛 of non-negative integers, and an integer 𝐾, decide if there is a subset 𝑃 ⊆ 1, 𝑛 such that σ𝑖∈𝑃 𝑎𝑖 = 𝐾.
PARTITION: Given a set of 𝑛 non-negative integers 𝑎1, … , 𝑎𝑛 , decide if there is a
subset 𝑃 ⊆ 1, 𝑛 such that σ𝑖∈𝑃 𝑎𝑖 = σ𝑖∉𝑃𝑎𝑖.
5 6 7 8 5 6 7 8
p-time reduction
PARTITION instance KNAPSACK instance with
PARTITION ≤ p KNAPSACK
• If we can solve KNAPSACK, how can we use that to solve PARTITION?
• Polynomial-time reduction
• Set
• Correctness proof: KNAPSACK returns yes if and only if an equal-size partition
5 6 7 8 5 6 7 8
p-time reduction
PARTITION instance KNAPSACK instance with
Algorithm to decide KNAPSACK
P-time Reduction
Instance 𝛽 of KNAPSACK Instance 𝛼 of Yes
PARTITION
Algorithm to decide PARTITION No
KNAPSACK ≤ p PARTITION
• If we can solve PARTITION, how can we use that to solve KNAPSACK?
• Polynomial-time reduction
• Set
• Add ,
57
5 6 7 8 5 6 7 8
p-time reduction
PARTITION instance 48 52 KNAPSACK instance with
KNAPSACK: Given a set 𝑎1, … , 𝑎𝑛 of non-negative integers, and an integer 𝐾, decide if there is a subset 𝑃 ⊆ 1, 𝑛 such that σ𝑖∈𝑃 𝑎𝑖 = 𝐾.
PARTITION: Given a set of 𝑛 non-negative integers 𝑎1, … , 𝑎𝑛 , decide if there is a
subset 𝑃 ⊆ 1, 𝑛 such that σ𝑖∈𝑃 𝑎𝑖 = σ𝑖∉𝑃𝑎𝑖.
KNAPSACK ≤ p PARTITION
• If we can solve PARTITION, how can we use that to solve KNAPSACK?
• Polynomial-time reduction
• Set
• Add ,
• Correctness proof: PARTITION returns yes if and only if there is a subset
Algorithm to decide PARTITION
P-time Reduction
Instance 𝛽 of PARTITION Instance 𝛼 of Yes
KNAPSACK
Algorithm to decide KNAPSACK No
5 6 7 8 5 6 7 8
p-time reduction
PARTITION instance 48 52 KNAPSACK instance with
KNAPSACK ≤ p PARTITION
• Polynomial-time reduction
• Set
• Add ,
• Correctness proof: PARTITION returns yes if and only if there is a subset 𝑃 ⊆ 1, 𝑛 such that σ𝑖∈𝑃 𝑎𝑖 = 𝐾
• “if” direction
59
𝑎1 𝑎2 𝑎3 𝑎4 𝑎1 𝑎4 𝑎5 𝑎2 𝑎3 𝑎6
PARTITION returns yes!
KNAPSACK ≤ p PARTITION
• Polynomial-time reduction
• Set
• Add ,
• Correctness proof: PARTITION returns yes if and only if there is a subset 𝑃 ⊆ 1, 𝑛 such that σ𝑖∈𝑃 𝑎𝑖 = 𝐾
• “only if” direction
• Because , if PARTITION returns yes, each set has 4𝐻 + 𝐾
• 𝑎1, … , 𝑎𝑛 must be divided into 2𝐻 − 𝐾 and 𝐾
𝑎6
𝑎2 𝑎3 𝑎5 𝑎1 𝑎4 𝑎1 𝑎4 𝑎2 𝑎3
Reduction for Proving Limits
• Definition
• Reduction from A to B implies A is not harder than B
• A ≤p B if A can be reduced to B in polynomial time
• NP-completeness proofs
• Goal: prove that B is NP-hard
• Known: A is NP-complete/NP-hard
• Approach: construct a polynomial-time reduction algorithm to convert 𝛼 to 𝛽
• Correctness: if we can solve B, then A can be solved → A ≤p B
• B is no easier than A → A is NP-hard, so B is NP-hard
61
If the reduction is not p-time, does this argument hold?
Algorithm to decide B Reduction
Algorithm
Instance 𝛽 of B Yes
Instance 𝛼 of A
Algorithm to decide A No
Proving NP-Completeness
Formal Language Framework
• Focus on decision problems
• A language L over σ is any set of strings made up of symbols from σ
• Every language L over σ is a subset of σ∗
• An algorithm A accepts a string if
• The language accepted by an algorithm A is the set of strings
• An algorithm A rejects a string x if
63
The formal-language framework allows us to express concisely the relation between decision problems and algorithms that solve them.
Proving NP-Completeness
• NP-Complete (NPC): class of decision problems in both NP and NP-hard
• In other words, a decision problem L is NP-complete if 1.L ∈ NP
2.L ∈ NP-hard (that is, L’ ≤p L for every L’ ∈ NP)
L1 L2 L3 :
L
≤p How to prove L is NP-hard ?
L1 L2 L3 :
≤p
L known
NPC problem
≤p
held by definition
Goal: prove polynomial- time reduction
Polynomial-Time Reducible
• If are languages s.t. , then L2 ∈ P implies L1 ∈ P.
65
A2 Transform
function f 𝑥 𝑓 𝑥
A1
P v.s. NP
• If one proves that SAT can be solved by a polynomial-time algorithm, then NP = P.
• If somebody proves that SAT cannot be solved by any polynomial-
time algorithm, then NP ≠ P.
Circuit Satisfiability Problem
• Given a Boolean combinational circuit composed of AND, OR, and NOT gates, is it satisfiable?
• Satisfiable: there exists an assignment s.t. outputs = 1
67
Satisfiable Unsatisfiable
CIRCUIT-SAT
• CIRCUIT-SAT can be solved in non-deterministic polynomial time
→ ∈ NP
• If CIRCUIT-SAT can be solved in deterministic polynomial time, then so can any NP problems
→ ∈ NP-hard
• (proof in textbook 34.3)
• CIRCUIT-SAT is NP-complete
CIRCUIT-SAT = {<C>: C is a satisfiable Boolean combinational circuit}
Karp’s NP-Complete Problems
1. CNF-SAT
2. 0-1 INTEGER PROGRAMMING 3. CLIQUE
4. SET PACKING 5. VERTEX COVER 6. SET COVERING
7. FEEDBACK ARC SET 8. FEEDBACK NODE SET
9. DIRECTED HAMILTONIAN CIRCUIT
10.UNDIRECTED HAMILTONIAN CIRCUIT 11.3-SAT
69
12.CHROMATIC NUMBER 13.CLIQUE COVER
14.EXACT COVER
15.3-dimensional MATCHING 16.STEINER TREE
17.HITTING SET 18.KNAPSACK
19.JOB SEQUENCING 20.PARTITION
21.MAX-CUT
Karp’s NP-Complete Problems
Formula Satisfiability Problem (SAT)
• Given a Boolean formula Φ with variables, is there a variable assignment satisfying Φ
• ∧ (AND), ∨ (OR), ¬ (NOT), → (implication), ↔ (if and only if)
• Satisfiable: Φ is evaluated to 1
71
SAT
• Is SAT ∈ NP-Complete?
• To prove that SAT is NP-Complete, we show that
• SAT ∈ NP
• SAT ∈ NP-hard (CIRCUIT-SAT ≤p SAT)
1) CIRCUIT-SAT is a known NPC problem
2) Construct a reduction f transforming every CIRCUIT-SAT instance to an SAT instance 3) Prove that x ∈ CIRCUIT-SAT iff f(x) ∈ SAT
4) Prove that f is a polynomial time transformation
SAT = {Φ | Φ is a Boolean formula with a satisfying assignment }
SAT ∈ NP
• Polynomial-time verification: replaces each variable in the formula with the corresponding value in the certificate and then evaluates the expression
73 initial
configuration
polynomial
SAT ∈ NP-Hard
1)CIRCUIT-SAT is a known NPC problem
2)Construct a reduction f transforming every CIRCUIT-SAT instance to an SAT instance
• Assign a variable to each wire in circuit C
• Represent the operation of each gate using a formula, e.g.
• Φ = AND the output variable and the operations of all gates
SAT ∈ NP-Hard
• Prove that x ∈ CIRCUIT-SAT ↔ f(x) ∈ SAT
• x ∈ CIRCUIT-SAT → f(x) ∈ SAT
• f(x) ∈ SAT → x ∈ CIRCUIT-SAT
• f is a polynomial time transformation
75
CIRCUIT-SAT ≤p SAT → SAT ∈ NP-hard
3-CNF-SAT Problem
• 3-CNF-SAT: Satisfiability of Boolean formulas in 3-conjunctive normal form (3- CNF)
• 3-CNF = AND of clauses, each of which is the OR of exactly 3 distinct literals
• A literal is an occurrence of a variable or its negation, e.g., x1 or ¬x1
→ satisfiable
3-CNF-SAT
• Is 3-CNF-SAT ∈ NP-Complete?
• To prove that 3-CNF-SAT is NP-Complete, we show that
• 3-CNF-SAT ∈ NP
• 3-CNF-SAT ∈ NP-hard (SAT ≤p 3-CNF-SAT)
1) SAT is a known NPC problem
2) Construct a reduction f transforming every SAT instance to an 3-CNF-SAT instance 3) Prove that x ∈ SAT iff f(x) ∈ 3-CNF-SAT
4) Prove that f is a polynomial time transformation
77
3-CNF-SAT = {Φ | Φ is a Boolean formula in 3-conjunctive normal form (3-CNF) with a satisfying assignment }
We focus on the reduction construction from now on, but remember that a full proof requires showing that all other conditions are true as well
SAT ≤ p 3-CNF-SAT
a)Construct a binary parser tree for an input formula Φ and introduce a variable yi for the output of each internal node
SAT ≤ p 3-CNF-SAT
b)Rewrite Φ as the AND of the root variable and clauses describing the operation of each node
79
SAT ≤ p 3-CNF-SAT
c)Convert each clause Φi’ to CNF
• Construct a truth table for each clause Φi’
• Construct the disjunctive normal form for ¬Φi’
• Apply DeMorgan’s Law to get the CNF formula Φi’’
𝒚𝟏 𝒚𝟐 𝒚𝟐 Φ1’ ¬Φ1’
1 1 1 0 1
1 1 0 1 0
1 0 1 0 1
1 0 0 0 1
0 1 1 1 0
0 1 0 0 1
𝒚𝟏 𝒚𝟐 𝒚𝟐 Φ1’
1 1 1 0
1 1 0 1
1 0 1 0
1 0 0 0
0 1 1 1
0 1 0 0
SAT ≤ p 3-CNF-SAT
d)Construct Φ’’’ in which each clause Ci exactly 3 distinct literals
• 3 distinct literals:
• 2 distinct literals:
• 1 literal only:
• Φ’’’ is satisfiable iff Φ is satisfiable
• All transformation can be done in polynomial time
• → 3-CNF-SAT is NP-Complete
81
Concluding Remarks
• Proving NP-Completeness: L ∈ NPC iff L ∈ NP and L ∈ NP-hard
• Step-by-step approach for proving L in NPC:
• Prove L ∈ NP
• Prove L ∈ NP-hard
• Select a known NPC problem C
• Construct a reduction f transforming every instance of C to an instance of L
• Prove that
• Prove that f is a polynomial time transformation L ∈ NP
P ≠ NP
P = NP
Question?
Important announcement will be sent to
@ntu.edu.tw mailbox & post to the course website
Course Website: http://ada.miulab.tw Email: ada-ta@csie.ntu.edu.tw