Algorithm Design and Analysis Amortized Analysis

Algorithm Design and Analysis Amortized Analysis

Yun-Nung (Vivian) Chen

http://ada.miulab.tw

Outline

• Amortized analysis

• #1: Stack Operations

• Aggregate method

• Accounting method

• Potential method

• #2: Binary Counter

• Aggregate method

• Accounting method

• Potential method

Algorithm Design & Analysis

• Design Strategy

• Divide-and-Conquer

• Dynamic Programming

• Greedy Algorithms

• Graph Algorithms

• Analysis

• Amortized Analysis

Amortized Analysis

Textbook Chapter 17 – Amortized Analysis

Data-Structure Operations

• A data structure comes with operations that organize the stored data

• Different operations may have different costs

• The same operation may have different costs

stack

PUSH POP

MULTIPOP

cost

operations

Worst Case Time Complexity

stack

PUSH POP

MULTIPOP

cost

operations worst-case

Cost of stack operations P^USH(S, x) = O(1)

P^OP(S) = O(1)

M^ULTIPOP(S, k) = O(min(|S|, k))

Worst Case Time Complexity

• n-th operation takes M^ULTIPOP(S, n) = O(n) time in the worst case

• n operations take O(n²) time

Stack Operations

Suppose that we apply a sequence of n operations on a data structure. What is the time complexity of the procedure?

Can this be an over-estimate?

What if only a few operations take O(n) time and the rest of them take O(1) time?

The worst-case bound is not tight because this

expensive Multipop operation cannot occur so frequently!

Amortized Analysis

• Goal: obtain an accurate worst-case bound in executing a sequence of operations on a given data structure

• An upper bound for any sequence of n operations

• Comparison: types of running-time analysis

Type Description

Worst case Running time guarantee for any input of size n

Average case Expected running time for a random input of size n Probabilistic Expected running time of a randomized algorithm

Amortized Worst-case running time for a sequence of n operations

3 Methods for Amortized Analysis

Aggregate method (聚集法)

• Determine an upper bound 𝑇(𝑛) on the cost over any sequence of 𝑛 operations

• The average cost per operation is then 𝑇(𝑛)/𝑛

• All operations have the same amortized cost

Accounting method (記帳法)

• Each operation is assigned an amortized cost (may differ from the actual cost)

• Each object of the data structure is associated with a credit

• Need to ensure that every object has sufficient credit at any time

Potential method (位能法)

• Similar to accounting method; each operation is assigned an amortized cost

• The data structure as a whole maintains a credit (i.e., potential)

• Need to ensure that the potential level is nonnegative at any time

Stack Operations

Textbook Chapter 17.1 – Aggregate analysis

Textbook Chapter 17.2 – The accounting method Textbook Chapter 17.3 – The potential method

Stack Operations

• Implementation with an array or a linked list

Operation Type Cost

PUSH(S, x): inset an element x into S P^OP(S): pop the top element from S

M^ULTIPOP(S, k): pop top k elements from S at once

stack

PUSH POP

MULTIPOP MULTIPOP(S, k)

while not STACK-EMPTY(S) and k > 0 POP(S)

k = k - 1

Stack Operations

Suppose that we apply a sequence of n operations on a data structure. What is the time complexity of the procedure?

Aggregate Method (聚集法)

• Approach:

1. Determine an upper bound 𝑇(𝑛) on the cost of any sequence of 𝑛 operations 2. Calculate the amortized cost per operation as 𝑇(𝑛)/𝑛

3. All operations have the same amortized cost

cost

operations 𝑇(𝑛) Amortized cost of each op =

… …

Aggregate Method for Stack

• The number of each operation type

• These n_pop + n_multipop operations together take at most

• Total cost for n operations:

• Amortized cost per operation:

Operation Type #Operations

P^USH(S, x): inset an element x into S n_push P^OP(S): pop the top element from S n_pop MULTIPOP(S, k): pop top k elements from S at once n_multipop

n

Key idea: #pop elements ≤ #push operations/elements

Another Thinking

• Once the push operation is taken, we prepare the additional cost for the future usage of multipop

Key idea: #pop elements ≤ #push operations/elements

Accounting Method (記帳法)

• Idea: save credits from the operations that take less cost for future use of

operations that take more cost (針對使用花費較低的operations時先存錢未 雨綢繆, 供未來花費較高的operations使用)

• Approach:

1. Each operation is assigned a valid amortized cost

• If amortized cost > actual cost, the difference becomes credit (存)

• Credit is deposited in an object of the data structure

• If amortized cost < actual cost, then withdraw (提) stored credits

2. Validity check: ensure that every object has sufficient credit for any sequence of n operations

3. Calculate total amortized cost based on individual ones

Accounting Method (記帳法)

• Validity check: ensure that every object has sufficient credit for any times of n operations (不能有赤字)

• c_i: the actual cost of the i-th operation

• ĉ_i: the amortized cost of the i-th operation

→ For all sequences of n operations, we require

▪ Aggregate Method

▪ Each type of operations have its actual cost

▪ Compute amortized cost using T(n)

▪ Accounting Method

▪ Each type of operations can have a different amortized cost

▪ Assign valid amortized costs first and then compute T(n)

Accounting Method for Stack

1. Assign the amortized cost

2. Show that for each object s.t.

• P^USH: the pushed element is deposited $1 credit

• P^OP and M^ULTIPOP: use the credit stored with the popped element

• There is always enough credit to pay for each operation

3. Each amortized cost is O(1) → total amortized cost is O(n)

Operation Type Actual Cost Amortized Cost

PUSH(S, x) 1 2

POP(S) 1 0

MULTIPOP(S, k) min(|S|, k) 0

Potential Method (位能法)

• Idea: represent the prepaid work as “potential,” which can be released to pay for future operations (the potential is associated with the whole data

structure rather than specific objects)

• Approach:

1. Select a potential function that takes the current data structure state as input and outputs a “potential level”

2. Validity check: ensure that the potential level is nonnegative

3. Calculate the amortized cost of each operation based on the potential function 4. Calculate total amortized cost based on individual ones

▪ Accounting Method

▪ Each object within the data structure has its credit

▪ Potential Method

▪ The data structure has credits

Potential Method (位能法)

• Potential function Φ maps any state of the data structure to a real number

• D₀: the initial state of data structure

• D_i: the state of data structure after i-th operation

• c_i: the actual cost of i-th operation

• ĉ_i: the amortized cost of i-th operation, defined as

Potential Method (位能法)

• Total amortized cost

• To obtain an upper bound on the actual cost

• Define a potential function such that

• Usually we set

Potential Method for Stack

1. Define Φ 𝐷_𝑖 to be the number of elements in the stack after the i-th operation

2. Validity check:

• The stack is initially empty →

• The number of elements in the stack is always ≥ 0 →

3. Compute amortized cost of each operation:

• P^USH(S, X):

• P^OP(S):

• M^ULTIPOP(S, k):

4. All operations have O(1) amortized cost → total amortized cost is O(n)

Practice: justify why it is zero

c_i: the actual cost of i-th operation ĉ_i: the amortized cost of i-th operation

Fibonacci Heap

Prim’s Time Complexity

• Fibonacci heap (Textbook Ch. 19)

• BUILD-MIN-HEAP:

• EXTRACT-MIN: (amortized)

• DECREASE-KEY: (amortized)

• Total complexity:

MST-PRIM(G, w, r) // w = weights, r = root for u in G.V

u.key = ∞ u.π = NIL r.key = 0 Q = G.V

while Q ≠ empty

u = EXTRACT-MIN(Q) for v in G.adj[u]

if v ∈ Q and w(u, v) < v.key v.π = u

v.key = w(u, v) // DECREASE-KEY

Dijkstra’s Time Complexity

• Fabonacci heap (Textbook Ch. 19)

• BUILD-MIN-HEAP:

• EXTRACT-MIN: (amortized)

• DECREASE-KEY: (amortized)

• Total complexity:

DIJKSTRA(G, w, s)

INITIALIZATION(G, s) S = empty

Q = G.v // INSERT while Q ≠ empty

u = EXTRACT-MIN(Q) S = S∪{u}

for v in G.adj[u]

RELAX(u, v, w)

INITIALIZATION(G, s) for v in G.V

v.d = ∞ v.π = NIL s.d = 0

RELAX(u, v, w)

if v.d > u.d + w(u, v) // DECREASE-KEY

v.d = u.d + w(u, v) v.π = u

Binary Counter

Textbook Chapter 17.1 – Aggregate analysis

Textbook Chapter 17.2 – The accounting method Textbook Chapter 17.3 – The potential method

Binary Counter

• Implementation with a k-bit array

• Each operation takes O(log n) time in the worst case

• n operations take O(n log n) time

INCREMENT(A) i = 0

while i < A.length and A[i] == 1 A[i] = 0

i = i + 1

if i < A.length A[i] = 1

Binary Counter

Suppose that a counter is initially zero. We increment the counter n times. How many bits are altered throughout the process?

0 1 10 11 100 101 110 111 1000

1001 1010 1011 1100 1101 1110 1111 10000 increment

Aggregate Method for Binary Counter

Counter

Value A[3] A[2] A[1] A[0] Total Cost of First n Operations

0 0 0 0 0 0

1 0 0 0 1 1

2 0 0 1 0 3

3 0 0 1 1 4

4 0 1 0 0 7

5 0 1 0 1 8

6 0 1 1 0 10

7 0 1 1 1 11

8 1 0 0 0 15

flip every increment flip every 2 increments

flip every 4 increments flip every 8 increments

Aggregate Method for Binary Counter

• Total #bits flipping in n increment operations:

• Total cost of the sequence:

• Amortized cost per operation:

Accounting Method for Binary Counter

1. Assign the amortized cost

2. Validity check:

• Each bit 0 to bit 1, we save additional $1 in the bit 1

• When bit 1 becomes to bit 0, we spend the saved cost

3. Each increment

• Change many 1s to 0s → free

• Change exactly a 0 to 1 → O(1)

• Each amortized cost is O(1) → total amortized cost is O(n)

Operation Actual Cost Amortized Cost

bit 0 → bit 1 1 2 (存$1到bit 1)

bit 1 → bit 0 1 0 (用掉存在bit 1裡面的$1)

increment #flipped bits 2 for setting a bit to 1

Accounting Method for Binary Counter

Counter

Value A[3] A[2] A[1] A[0] Total Cost of First n

Operations

0 0 0 0 0 0

1 0 0 0 1 1

2 0 0 1 0 3

3 0 0 1 1 4

4 0 1 0 0 7

5 0 1 0 1 8

6 0 1 1 0 10

7 0 1 1 1 11

8 1 0 0 0 15

Potential Method for Binary Counter

1. Define Φ 𝐷_𝑖 to be the number of 1s in the counter after the i-th operation

2. Validity check:

• The counter is initially zero →

• The number of 1’s cannot be negative →

3. Compute amortized cost of each INCREMENT:

• Let LSB₀(i) be the number of continuous 1s in the suffix

• For example, LSB₀(01011011) = 2, and LSB₀(01011111) = 5

4. All operations have O(1) amortized cost → total amortized cost is O(n)

c_i: the actual cost of i-th operation ĉ_i: the amortized cost of i-th operation

Concluding Remarks

Aggregate method (聚集法)

• Determine an upper bound 𝑇(𝑛) on the cost over any sequence of 𝑛 operations

• The average cost per operation is then 𝑇(𝑛)/𝑛

• All operations have the same amortized cost

Accounting method (記帳法)

• Each operation is assigned an amortized cost (may differ from the actual cost)

• Each object of the data structure is associated with a credit

• Need to ensure that every object has sufficient credit at any time

Potential method (位能法)

• Similar to accounting method; each operation is assigned an amortized cost

• The data structure as a whole maintains a credit (i.e., potential)

• Need to ensure that the potential level is nonnegative at any time

Question?

Important announcement will be sent to

@ntu.edu.tw mailbox & post to the course website

Course Website: http://ada.miulab.tw Email: ada-ta@csie.ntu.edu.tw