Power and inverse power methods

師大

Power and inverse power methods

Tsung-Ming Huang

Department of Mathematics National Taiwan Normal University, Taiwan

February 15, 2011

Outline

1 Power method

2 Inverse power method

師大

Definition 1

1 An eigenvalue whose geometric multiplicity is less than its algebraic multiplicity isdefective.

2 The matrix A ∈ C^n×n has acomplete systemof eigenvectors if it has n linearly independent eigenvectors.

Let A be a nondefective matrix and (λ_i, x_i)for i = 1, · · · , n be a complete set of eigenpairs of A. That is {x₁, · · · , xn} is linearly independent. Hence, for any u0 6= 0, ∃ α1, · · · , αnsuch that

u₀ = α₁x₁+ · · · + α_nx_n. Now A^kxi= λ^k_ixi, so that

A^ku₀= α₁λ^k₁x₁+ · · · + α_nλ^k_nx_n. (1) If |λ₁| > |λ_i| for i ≥ 2 and α₁ 6= 0, then

1

λ^k₁A^ku₀ = α₁x₁+ (λ₂ λ1

)^kα₂x₂+ · · · + α_n(λ_n λ1

)^kx_n→ α₁x₁as k → ∞.

Algorithm 1 (Power Method with 2-norm) Choose an initial u 6= 0 with kuk2 = 1.

Iterate until convergence

Computev = Au; k = kvk2; u := v/k Theorem 2

The sequence defined by Algorithm 1 is satisfied

i→∞lim k_i = |λ₁|

i→∞lim εⁱui= x1

kx₁k α1

|α₁|, where ε = |λ₁| λ₁

師大

Proof: It is obvious that

us = A^su0/kA^su0k, ks= kA^su0k/kA^s−1u0k. (2) This follows from λ1−sA^su0 −→ α₁x1that

|λ₁|^−skA^su0k −→ |α₁|kx₁k

|λ₁|^−s+1kA^s−1u0k −→ |α₁|kx₁k and then

|λ₁|⁻¹kA^su₀k/kA^s−1u₀k = |λ₁|⁻¹k_s−→ 1.

From (1) follows now for s → ∞

ε^su_s = ε^s A^su₀ kA^su₀k =

α₁x₁+Pn i=2α_i

λi

λ1

s

x_i kα₁x1+Pn

i=2αi

λi

λ1

s

xik

→ α₁x₁

kα₁x1k = x₁ kx₁k

α₁

|α₁|.

Algorithm 2 (Power Method with Linear Function) Choose an initial u 6= 0.

Iterate until convergence

Computev = Au; k = `(v); u := v/k

where`(v), e.g. e₁(v)ore_n(v), is a linear functional.

Theorem 3

Suppose `(x<sub>1</sub>) 6= 0and `(v_i) 6= 0, i = 1, 2, . . . ,then

i→∞lim ki = λ1 i→∞lim u_i = x₁

`(x₁).

師大

Proof: As above we show that

ui = Aⁱu0/`(A<sup>i</sup>u0), ki= `(Aⁱu0)/`(Aⁱ⁻¹u0).

From (1) we get for i → ∞

λ₁⁻ⁱ`(A<sup>i</sup>u<sub>0</sub>) −→ α<sub>1</sub>`(x₁), λ₁⁻ⁱ⁺¹`(A<sup>i−1</sup>u<sub>0</sub>) −→ α<sub>1</sub>`(x₁), thus

λ1−1

ki−→ 1.

Similarly for i −→ ∞,

ui= Aⁱu0

`(Aⁱu0) = α₁x₁+P_n

j=2α_j(^λ_λ^j

1)ⁱx_j

`(α1x1+Pn

j=2αj(_λ^λj

1)ⁱxj)

−→ α1x1

α1`(x1)

Note that:

ki = `(Aⁱu0)

`(Aⁱ⁻¹u0) = λ1

α1`(x1) +Pn

j=2αj(^λ_λ^j

1)ⁱ`(xj) α1`(x1) +Pn

j=2αj(^λ_λ^j

1)ⁱ⁻¹`(xj)

= λ1+ O



| λ2

λ1

|ⁱ⁻¹

 .

That is the convergent rate is   

λ2

λ1

  .

師大

Theorem 4

Let u 6= 0 and for any µ set r_µ= Au − µu. Then kr_µk₂is minimized when

µ = u^∗Au/u^∗u.

In this case r_µ⊥ u.

Proof: W.L.O.G. assume kuk2= 1. Let u U
be unitary and set

 u^∗ U^∗



A u U
≡

 ν h^∗ g B



=

 u^∗Au u^∗AU U^∗Au U^∗AU

 .

Then

 u^∗ U^∗

 rµ =

 u^∗ U^∗



Au − µ

 u^∗ U^∗

 u

=

 u^∗ U^∗



A u U

 u^∗ U^∗

 u − µ

 u^∗ U^∗

 u

=

 ν h^∗

g B

  u^∗ U^∗

 u − µ

 u^∗ U^∗

 u

=

 ν h^∗

g B

  1 0



− µ

 1 0



=

 ν − µ g

 .

It follows that kr_µk²₂= k

 u^∗ U^∗



r_µk²₂ = k

 ν − µ g



k²₂ = |ν − µ|²+ kgk²₂.

師大

Hence

minµ kr_µk₂ = kgk₂ = kr_νk₂. That is µ = ν = u^∗Au. On the other hand, since

u^∗r_µ= u^∗(Au − µu) = u^∗Au − µ = 0, it implies that r_µ⊥ u.

Definition 5 (Rayleigh quotient)

Let u and v be vectors with v^∗u 6= 0.Then v^∗Au/v^∗uis called a Rayleigh quotient.

If u or v is an eigenvector corresponding to an eigenvalue λ of A, then v^∗Au

v^∗u = λv^∗u v^∗u = λ.

Therefore, u^∗_kAu_k/u^∗_ku_kprovide a sequence of approximation to λ in the power method.

Inverse power method

Goal

Find the eigenpair (λ, x) of A where λ is belonged to a given region or closest to a certain scalar σ.

Let λ₁, · · · , λ_nbe the eigenvalues of A. Suppose λ₁ is simple and σ ≈ λ1.Then

µ1 = 1

λ₁− σ, µ2 = 1

λ₂− σ, · · · , µn= 1 λ_n− σ

are eigenvalues of (A − σI)⁻¹ and µ₁→ ∞ as σ → λ₁. Thus we transform λ₁ into a dominant eigenvalue µ₁.

The inverse power method is simply the power method applied to (A − σI)⁻¹.

師大

Algorithm 3 (Inverse power method with a fixed shift) Choose an initial u₀ 6= 0.

For i = 0, 1, 2, . . .

Computev_i+1= (A − σI)⁻¹u_i andk_i+1= `(v_i+1).

Setui+1= vi+1/ki+1

The convergence of Algorithm 3 is |^λ_λ^1−σ

2−σ| whenever λ₁ and λ₂ are the closest and the second closest eigenvalues to σ.

Algorithm 3 is linearly convergent.

Let (λ, x) be an eigenpair of A, i.e.,

Ax = λx ⇒ (A − σI)x = (λ − σ)x ⇒ (A − σI)⁻¹x = 1

λ − σx ≡ µx.

It implies that

λ = σ + µ⁻¹.

Algorithm 4 (Inverse power method with variant shifts) Choose an initial u₀ 6= 0. Given σ₀= σ.

For i = 0, 1, 2, . . .

Computevi+1= (A − σiI)⁻¹ui andki+1= `(vi+1).

Setu_i+1= v_i+1/k_i+1andσ_i+1= σ_i+ 1/k_i+1.

師大

Connection with Newton method

Consider the nonlinear equations:

F

 u λ



≡

 Au − λu

`^Tu − 1



=

 0 0



. (3)

Newton method for (3): for i = 0, 1, 2, . . .

 u_i+1 λi+1



=

 u_i λi



−

 F⁰

 u_i λi

−1

F

 u_i λi



. Since

F⁰

 u λ



=

 A − λI −u

`^T 0

 , the Newton method can be rewritten by component-wise

(A − λ_iI)u_i+1 = (λ_i+1− λ_i)u_i (4)

`^Tui+1 = 1. (5)

Let

v_i+1= u_i+1 λi+1− λ_i. Substituting v_i+1into (4), we get

(A − λ_iI)v_i+1= u_i. By equation (5), we have

ki+1= `(vi+1) = `(ui+1)

λi+1− λ_i = 1 λi+1− λ_i. It follows that

λ_i+1= λ_i+ 1 k_i+1.

師大

Algorithm 5 (Inverse power method with Rayleigh Quotient) Choose an initial u₀ 6= 0 with ku0k₂ = 1.

Compute σ0= u^T₀Au0. For i = 0, 1, 2, . . .

Computevi+1= (A − σiI)⁻¹ui.

Setui+1= vi+1/kvi+1k₂andσi+1= u^T_i+1Aui+1. For symmetric A, Algorithm 5 is cubically convergent.