Special types of matrices

logo

logo

Since Ax = 0, for the fixed k, we have

n

X

j=1

a_kjxj = 0 ⇒ a_kkx_k= −

n

X

j=1,j6=k

a_kjxj

⇒ |a_kk||x_k| ≤

n

X

j=1,j6=k

|a_kj||x_j|,

which implies

|a_kk| ≤

n

X

j=1,j6=k

|a_kj||x_j|

|x_k|≤

n

X

j=1,j6=k

|a_kj|.

But this contradicts the assumption that A is diagonally dominant. Therefore A must be nonsingular.

67 / 89

logo

Theorem 12

Gaussian elimination without pivoting preserve the diagonal dominance of a matrix.

Proof: Let A ∈ R^n×n be a diagonally dominant matrix and A⁽²⁾= [a⁽²⁾_ij ]is the result of applying one step of Gaussian elimination to A⁽¹⁾= Awithout any pivoting strategy.

After one step of Gaussian elimination, a⁽²⁾_i1 = 0for i = 2, . . . , n, and the first row is unchanged. Therefore, the property

|a⁽²⁾₁₁| >

n

X

j=2

|a⁽²⁾_1j |

is preserved, and all we need to show is that

|a⁽²⁾_ii | >

n

X

j=2,j6=i

|a⁽²⁾_ij |, for i = 2, . . . , n.

logo

Using the Gaussian elimination formula (2), we have

|a⁽²⁾_ii | =     

a⁽¹⁾_ii −a⁽¹⁾_i1 a⁽¹⁾₁₁

a⁽¹⁾_1i     

=    

aii−ai1

a₁₁a1i

   

≥ |aii| − |ai1|

|a11||a1i|

= |aii| − |ai1| + |ai1| − |a_i1|

|a11||a1i|

= |aii| − |ai1| + |ai1|

|a11|(|a11| − |a1i|)

>

n

X

j=2,j6=i

|a_ij| + |a_i1|

|a11|

n

X

j=2,j6=i

|a_1j|

=

n

X

j=2,j6=i

|aij| +

n

X

j=2,j6=i

|ai1|

|a11||a1j|

≥

n

X

j=2,j6=i

   

aij− ai1

a11

a1j

   

=

n

X

j=2,j6=i

|a⁽²⁾_ij |.

69 / 89

logo

Thus A⁽²⁾is still diagonally dominant. Since the subsequent steps of Gaussian elimination mimic the first, except for being applied to submatrices of smaller size, it suffices to conclude that Gaussian elimination without pivoting preserves the diagonal dominance of a matrix.

Theorem 13

Let A be strictly diagonally dominant. Then Gaussian elimination can be performed on Ax = b to obtain its unique solution without row or column interchanges.

Definition 14

A matrix A ispositive definiteif it issymmetricandx^TAx > 0

∀ x 6= 0.

logo

Theorem 15

If A is an n × n positive definite matrix, then (a) Ahas an inverse;

(b) a_ii> 0, ∀ i = 1, . . . , n;

(c) max_1≤k,j≤n|a_kj| ≤ max_1≤i≤n|a_ii|;

(d) (a_ij)² < a_iia_jj, ∀ i 6= j.

Proof:

(a) If x satisfies Ax = 0, then x^TAx = 0. Since A is positive definite, this implies x = 0. Consequently, Ax = 0has only the zero solution, and A is nonsingular.

(b) Since A is positive definite, a_ii= e^T_i Ae_i> 0,

where ei is the i-th column of the n × n identify

matrix. 71 / 89

logo

(c) For k 6= j, define x = [x_i]by

xi =







0, if i 6= j and i 6= k, 1, if i = j,

−1, if i = k.

Since x 6= 0,

0 < x^TAx = a_jj+ a_kk− a_jk− a_kj. But A^T = A, so

2akj < ajj+ akk. (5) Now define z = [z_i]by

z_i =

 0, if i 6= j and j 6= k, 1, if i = j or i = k.

logo

Then z^TAz > 0, so

−2a_kj < ajj+ akk. (6) Equations (5) and (6) imply that for each k 6= j,

|a_kj| < a_kk+ ajj

2 ≤ max

1≤i≤n|a_ii|, so

1≤k,j≤nmax |a_kj| ≤ max

1≤i≤n|a_ii|.

(d) For i 6= j, define x = [x_k]by

x_k=







0, if k 6= j and k 6= i, α, if k = i,

1, if k = j,

where α represents an arbitrary real number.

73 / 89

logo

Since x 6= 0,

0 < x^TAx = aiiα²+ 2aijα + ajj ≡ P (α), ∀ α ∈ R.

That is the quadratic polynomial P (α) has no real roots. It implies that

4a²_ij − 4a_iiajj < 0 and a²_ij < aiiajj. Definition 16 (Leading principal minor)

Let A be an n × n matrix. Theupper leftk × ksubmatrix, denoted as

A_k=











a₁₁ a₁₂ · · · a_1k a21 a22 · · · a2k

... ... . .. ... ak1 ak2 · · · akk









 ,

is called theleading k × k principal submatrix, and the determinant of A_k,det(A_k), is called theleading principal

minor. ^{74 / 89}

logo

Theorem 17

A symmetric matrix A is positive definite if and only if each of its leading principal submatrices has a positive determinant.

Theorem 18

The symmetric matrix A is positive definite if and only if Gaussian elimination without row interchanges can be performed on Ax = b with all pivot elements positive.

Corollary 19

The matrix A is positive definite if and only if A can be factored in the form LDL^T, where L is lower triangular with 1’s on its diagonal and D is a diagonal matrix with positive diagonal entries.

75 / 89

logo

Theorem 20

If all leading principal submatrices of A ∈ R^n×nare nonsingular, then A has an LU -factorization.

Proof: Proof by mathematical induction.

1 n = 1, A₁ = [a₁₁]is nonsingular, then a₁₁6= 0. Let L₁ = [1]

and U₁ = [a₁₁]. Then A₁= L₁U₁. The theorem holds.

2 Assume that the leading principal submatrices A₁, . . . , A_k are nonsingular and A_khas an LU -factorization

A_k= L_kU_k, where L_kis unit lower triangular and U_kis upper triangular.

3 Show that there exist an unit lower triangular matrix L_k+1 and an upper triangular matrix U_k+1such that

Ak+1= Lk+1Uk+1.

logo

Write

Ak+1=

 Ak vk

w_k^T a_k+1,k+1

 , where

v_k =









 a_1,k+1 a_2,k+1

... a_k,k+1











and w_k=









 a_k+1,1 a_k+1,2

... a_k+1,k









 .

Since A_kis nonsingular, both L_kand U_k are nonsingular.

Therefore, L_ky_k= v_khas a unique solution y_k∈ R^k, and z^tUk = w^T_k has a unique solution z_k∈ R^k. Let

L_k+1=

 Lk 0 z_k^T 1



and U_k+1 =

 Uk yk

0 a_k+1,k+1− z_k^Ty_k

 .

77 / 89

logo

Then L_k+1 is unit lower triangular, U_k+1 is upper triangular, and Lk+1Uk+1 =

 L_kU_k L_ky_k

z_k^TUk z_k^Tyk+ ak+1,k+1− z^T_kyk



=

 A_k v_k w^T_k a_k+1,k+1



= A_k+1. This proves the theorem.

logo

Theorem 21

If A is nonsingular and the LU factorization exists, then the LU factorization isunique.

Proof: Suppose both

A = L₁U₁ and A = L₂U₂

are LU factorizations. Since A is nonsingular, L₁, U₁, L₂, U₂are all nonsingular, and

A = L₁U₁ = L₂U₂ =⇒ L⁻¹₂ L₁ = U₂U₁⁻¹.

Since L1 and L2are unit lower triangular, it implies that L⁻¹₂ L1

is also unit lower triangular. On the other hand, since U₁ and U₂ are upper triangular, U₂U₁⁻¹ is also upper triangular. Therefore,

L⁻¹₂ L1 = I = U2U₁⁻¹ which implies that L₁ = L₂ and U₁ = U₂.

79 / 89

logo

Lemma 22

If A ∈ R^n×n ispositive definite, then allleading principal submatricesof A arenonsingular.

Proof: For 1 ≤ k ≤ n, let

z_k= [x1, . . . , x_k]^T ∈ R^k and x = [x1, . . . , x_k, 0, . . . , 0]^T ∈ Rⁿ, where x₁, . . . , xk∈ R are not all zero. Since A is positive definite,

z_k^TA_kz_k= x^TAx > 0,

where A_kis the k × k leading principal submatrix of A. This shows that A_k are also positive definite, hence A_kare nonsingular.

logo

Corollary 23

The matrix A is positive definite if and only if

A = GG^T, (7)

where G is lower triangular with positive diagonal entries.

Proof: “⇒” A is positive definite

⇒ all leading principal submatrices of A are nonsingular

⇒ A has the LU factorization A = LU , where L is unit lower triangular and U is upper triangular.

Since A is symmetric,

LU = A = A^T = U^TL^T =⇒ U (L^T)⁻¹ = L⁻¹U^T. U (L^T)⁻¹ is upper triangular and L⁻¹U^T is lower triangular

⇒ U (L^T)⁻¹ to be a diagonal matrix, say, U (L^T)⁻¹= D.

⇒ U = DL^T. Hence

A = LDL^T.

81 / 89

logo

Since A is positive definite,

x^TAx > 0 =⇒ x^TLDL^Tx = (L^Tx)^TD(L^Tx) > 0.

This means D is also positive definite, and hence d_ii> 0. Thus D^1/2is well-defined and we have

A = LDL^T = LD^1/2D^1/2L^T ≡ GG^T,

where G ≡ LD^1/2. Since the LU factorization is unique, G is unique.

“⇐”

Since G is lower triangular with positive diagonal entries, G is nonsingular. It implies that

G^Tx 6= 0, ∀ x 6= 0.

Hence

x^TAx = x^TGG^Tx = kG^Txk²₂> 0, ∀ x 6= 0 which implies that A is positive definite.

logo

The factorization (7) is referred to as theCholesky factorization.

Derive an algorithm for computing the Cholesky factorization:

Let

A ≡ [a_ij] and G =













g₁₁ 0 · · · 0 g₂₁ g₂₂ . .. ... ... ... . .. 0 gn1 gn2 · · · gnn











 .

Assume the first k − 1 columns of G have been determined after k − 1 steps. Bycomponentwise comparisonwith

[aij] =













g11 0 · · · 0 g21 g22 . .. ... ... ... . .. 0 gn1 gn2 · · · gnn























g11 g21 · · · gn1

0 g22 · · · gn2

... . .. ... ... 0 · · · 0 gnn









 ,

one has

akk=

k

X

j=1

g_kj² ,

83 / 89

logo

which gives

g_kk² = akk−

k−1

X

j=1

g_kj² .

Moreover,

aik=

k

X

j=1

gijgkj, i = k + 1, . . . , n,

hence the k-th column of G can be computed by

g_ik=



a_ik−

k−1

X

j=1

g_ijg_kj







g_kk, i = k + 1, . . . , n.

logo

Algorithm 6 (Cholesky Factorization)

Given an n × nsymmetric positive definitematrix A, this algorithm computes the Cholesky factorization A = GG^T.

Initialize G = 0 For k = 1, . . . , n

G(k, k) = q

A(k, k) −Pk−1

j=1G(k, j)G(k, j) For i = k + 1, . . . , n

G(i, k) =



A(i, k) −Pk−1

j=1G(i, j)G(k, j)



G(k, k) End For

End For

In addition to n square root operations, there are approximately

n

X

k=1

[2k − 2 + (2k − 1)(n − k)] = 1 3n³+1

2n²−5 6n floating-point arithmetic required by the algorithm.

85 / 89

logo

In document Direct Methods for Solving Linear Systems (Page 66-86)