logo
logo
Since Ax = 0, for the fixed k, we have
n
X
j=1
akjxj = 0 ⇒ akkxk= −
n
X
j=1,j6=k
akjxj
⇒ |akk||xk| ≤
n
X
j=1,j6=k
|akj||xj|,
which implies
|akk| ≤
n
X
j=1,j6=k
|akj||xj|
|xk|≤
n
X
j=1,j6=k
|akj|.
But this contradicts the assumption that A is diagonally dominant. Therefore A must be nonsingular.
67 / 89
logo
Theorem 12
Gaussian elimination without pivoting preserve the diagonal dominance of a matrix.
Proof: Let A ∈ Rn×n be a diagonally dominant matrix and A(2)= [a(2)ij ]is the result of applying one step of Gaussian elimination to A(1)= Awithout any pivoting strategy.
After one step of Gaussian elimination, a(2)i1 = 0for i = 2, . . . , n, and the first row is unchanged. Therefore, the property
|a(2)11| >
n
X
j=2
|a(2)1j |
is preserved, and all we need to show is that
|a(2)ii | >
n
X
j=2,j6=i
|a(2)ij |, for i = 2, . . . , n.
logo
Using the Gaussian elimination formula (2), we have
|a(2)ii | =
a(1)ii −a(1)i1 a(1)11
a(1)1i
=
aii−ai1
a11a1i
≥ |aii| − |ai1|
|a11||a1i|
= |aii| − |ai1| + |ai1| − |ai1|
|a11||a1i|
= |aii| − |ai1| + |ai1|
|a11|(|a11| − |a1i|)
>
n
X
j=2,j6=i
|aij| + |ai1|
|a11|
n
X
j=2,j6=i
|a1j|
=
n
X
j=2,j6=i
|aij| +
n
X
j=2,j6=i
|ai1|
|a11||a1j|
≥
n
X
j=2,j6=i
aij− ai1
a11
a1j
=
n
X
j=2,j6=i
|a(2)ij |.
69 / 89
logo
Thus A(2)is still diagonally dominant. Since the subsequent steps of Gaussian elimination mimic the first, except for being applied to submatrices of smaller size, it suffices to conclude that Gaussian elimination without pivoting preserves the diagonal dominance of a matrix.
Theorem 13
Let A be strictly diagonally dominant. Then Gaussian elimination can be performed on Ax = b to obtain its unique solution without row or column interchanges.
Definition 14
A matrix A ispositive definiteif it issymmetricandxTAx > 0
∀ x 6= 0.
logo
Theorem 15
If A is an n × n positive definite matrix, then (a) Ahas an inverse;
(b) aii> 0, ∀ i = 1, . . . , n;
(c) max1≤k,j≤n|akj| ≤ max1≤i≤n|aii|;
(d) (aij)2 < aiiajj, ∀ i 6= j.
Proof:
(a) If x satisfies Ax = 0, then xTAx = 0. Since A is positive definite, this implies x = 0. Consequently, Ax = 0has only the zero solution, and A is nonsingular.
(b) Since A is positive definite, aii= eTi Aei> 0,
where ei is the i-th column of the n × n identify
matrix. 71 / 89
logo
(c) For k 6= j, define x = [xi]by
xi =
0, if i 6= j and i 6= k, 1, if i = j,
−1, if i = k.
Since x 6= 0,
0 < xTAx = ajj+ akk− ajk− akj. But AT = A, so
2akj < ajj+ akk. (5) Now define z = [zi]by
zi =
0, if i 6= j and j 6= k, 1, if i = j or i = k.
logo
Then zTAz > 0, so
−2akj < ajj+ akk. (6) Equations (5) and (6) imply that for each k 6= j,
|akj| < akk+ ajj
2 ≤ max
1≤i≤n|aii|, so
1≤k,j≤nmax |akj| ≤ max
1≤i≤n|aii|.
(d) For i 6= j, define x = [xk]by
xk=
0, if k 6= j and k 6= i, α, if k = i,
1, if k = j,
where α represents an arbitrary real number.
73 / 89
logo
Since x 6= 0,
0 < xTAx = aiiα2+ 2aijα + ajj ≡ P (α), ∀ α ∈ R.
That is the quadratic polynomial P (α) has no real roots. It implies that
4a2ij − 4aiiajj < 0 and a2ij < aiiajj. Definition 16 (Leading principal minor)
Let A be an n × n matrix. Theupper leftk × ksubmatrix, denoted as
Ak=
a11 a12 · · · a1k a21 a22 · · · a2k
... ... . .. ... ak1 ak2 · · · akk
,
is called theleading k × k principal submatrix, and the determinant of Ak,det(Ak), is called theleading principal
minor. 74 / 89
logo
Theorem 17
A symmetric matrix A is positive definite if and only if each of its leading principal submatrices has a positive determinant.
Theorem 18
The symmetric matrix A is positive definite if and only if Gaussian elimination without row interchanges can be performed on Ax = b with all pivot elements positive.
Corollary 19
The matrix A is positive definite if and only if A can be factored in the form LDLT, where L is lower triangular with 1’s on its diagonal and D is a diagonal matrix with positive diagonal entries.
75 / 89
logo
Theorem 20
If all leading principal submatrices of A ∈ Rn×nare nonsingular, then A has an LU -factorization.
Proof: Proof by mathematical induction.
1 n = 1, A1 = [a11]is nonsingular, then a116= 0. Let L1 = [1]
and U1 = [a11]. Then A1= L1U1. The theorem holds.
2 Assume that the leading principal submatrices A1, . . . , Ak are nonsingular and Akhas an LU -factorization
Ak= LkUk, where Lkis unit lower triangular and Ukis upper triangular.
3 Show that there exist an unit lower triangular matrix Lk+1 and an upper triangular matrix Uk+1such that
Ak+1= Lk+1Uk+1.
logo
Write
Ak+1=
Ak vk
wkT ak+1,k+1
, where
vk =
a1,k+1 a2,k+1
... ak,k+1
and wk=
ak+1,1 ak+1,2
... ak+1,k
.
Since Akis nonsingular, both Lkand Uk are nonsingular.
Therefore, Lkyk= vkhas a unique solution yk∈ Rk, and ztUk = wTk has a unique solution zk∈ Rk. Let
Lk+1=
Lk 0 zkT 1
and Uk+1 =
Uk yk
0 ak+1,k+1− zkTyk
.
77 / 89
logo
Then Lk+1 is unit lower triangular, Uk+1 is upper triangular, and Lk+1Uk+1 =
LkUk Lkyk
zkTUk zkTyk+ ak+1,k+1− zTkyk
=
Ak vk wTk ak+1,k+1
= Ak+1. This proves the theorem.
logo
Theorem 21
If A is nonsingular and the LU factorization exists, then the LU factorization isunique.
Proof: Suppose both
A = L1U1 and A = L2U2
are LU factorizations. Since A is nonsingular, L1, U1, L2, U2are all nonsingular, and
A = L1U1 = L2U2 =⇒ L−12 L1 = U2U1−1.
Since L1 and L2are unit lower triangular, it implies that L−12 L1
is also unit lower triangular. On the other hand, since U1 and U2 are upper triangular, U2U1−1 is also upper triangular. Therefore,
L−12 L1 = I = U2U1−1 which implies that L1 = L2 and U1 = U2.
79 / 89
logo
Lemma 22
If A ∈ Rn×n ispositive definite, then allleading principal submatricesof A arenonsingular.
Proof: For 1 ≤ k ≤ n, let
zk= [x1, . . . , xk]T ∈ Rk and x = [x1, . . . , xk, 0, . . . , 0]T ∈ Rn, where x1, . . . , xk∈ R are not all zero. Since A is positive definite,
zkTAkzk= xTAx > 0,
where Akis the k × k leading principal submatrix of A. This shows that Ak are also positive definite, hence Akare nonsingular.
logo
Corollary 23
The matrix A is positive definite if and only if
A = GGT, (7)
where G is lower triangular with positive diagonal entries.
Proof: “⇒” A is positive definite
⇒ all leading principal submatrices of A are nonsingular
⇒ A has the LU factorization A = LU , where L is unit lower triangular and U is upper triangular.
Since A is symmetric,
LU = A = AT = UTLT =⇒ U (LT)−1 = L−1UT. U (LT)−1 is upper triangular and L−1UT is lower triangular
⇒ U (LT)−1 to be a diagonal matrix, say, U (LT)−1= D.
⇒ U = DLT. Hence
A = LDLT.
81 / 89
logo
Since A is positive definite,
xTAx > 0 =⇒ xTLDLTx = (LTx)TD(LTx) > 0.
This means D is also positive definite, and hence dii> 0. Thus D1/2is well-defined and we have
A = LDLT = LD1/2D1/2LT ≡ GGT,
where G ≡ LD1/2. Since the LU factorization is unique, G is unique.
“⇐”
Since G is lower triangular with positive diagonal entries, G is nonsingular. It implies that
GTx 6= 0, ∀ x 6= 0.
Hence
xTAx = xTGGTx = kGTxk22> 0, ∀ x 6= 0 which implies that A is positive definite.
logo
The factorization (7) is referred to as theCholesky factorization.
Derive an algorithm for computing the Cholesky factorization:
Let
A ≡ [aij] and G =
g11 0 · · · 0 g21 g22 . .. ... ... ... . .. 0 gn1 gn2 · · · gnn
.
Assume the first k − 1 columns of G have been determined after k − 1 steps. Bycomponentwise comparisonwith
[aij] =
g11 0 · · · 0 g21 g22 . .. ... ... ... . .. 0 gn1 gn2 · · · gnn
g11 g21 · · · gn1
0 g22 · · · gn2
... . .. ... ... 0 · · · 0 gnn
,
one has
akk=
k
X
j=1
gkj2 ,
83 / 89
logo
which gives
gkk2 = akk−
k−1
X
j=1
gkj2 .
Moreover,
aik=
k
X
j=1
gijgkj, i = k + 1, . . . , n,
hence the k-th column of G can be computed by
gik=
aik−
k−1
X
j=1
gijgkj
gkk, i = k + 1, . . . , n.
logo
Algorithm 6 (Cholesky Factorization)
Given an n × nsymmetric positive definitematrix A, this algorithm computes the Cholesky factorization A = GGT.
Initialize G = 0 For k = 1, . . . , n
G(k, k) = q
A(k, k) −Pk−1
j=1G(k, j)G(k, j) For i = k + 1, . . . , n
G(i, k) =
A(i, k) −Pk−1
j=1G(i, j)G(k, j)
G(k, k) End For
End For
In addition to n square root operations, there are approximately
n
X
k=1
[2k − 2 + (2k − 1)(n − k)] = 1 3n3+1
2n2−5 6n floating-point arithmetic required by the algorithm.
85 / 89
logo