By induction, we obtain that for any k ∈ N, xk(t

1. Operator Norms on Space of linear maps

Let A be an n × n real matrix and x0 be a vector in Rⁿ. We would like to use the Picard iteration method to solve for the following system of ordinary linear differential equation

(1.1) x⁰(t) = Ax(t)

x(0) = x0

By integrating the differential equation (1.1), we obtain that x(t) = x₀+

Z t 0

Ax(s)ds.

Let x₀(t) = x₀ for t ∈ [0, b] be the constant function and define x : [0, b] → Rⁿ recursively by

x_n(t) = x₀+ Z _t

0

Ax_n−1(s)ds.

By induction, we obtain that for any k ∈ N, x_k(t) = x₀+(tA)

1! x₀+ · · · +(tA)^k k! x₀. We can rewrite x_k(t) as

x_k(t) =



I_n+(tA)

1! + · · · +(tA)^k k!



x₀, for any k ≥ 1.

Here I_n denotes the n × n identify matrix. It is natural for us to ask: can we define In+

∞

X

k=1

(tA)^k k! ?

To define an infinite series of matrices, we need to define a norm on Mn(R), where Mn(R) is the space of n × n real matrices. Let us define a norm on the space M_mn(R) of m × n matrices where m and n are not necessarily equal.

Let A be an m × n matrix. We define a function L_A : Rⁿ → R^m by L_A(x) = Ax. It is routine to check that LAis a linear map. Conversely, given a linear map T : Rⁿ→ R^m, can we find a unique m × n matrix A such that T = L_A?

Let T : Rⁿ→ R^mbe a linear map. For each x ∈ Rⁿ, we write T (x) = (T₁(x), · · · , T_m(x)).

Since T is linear, Ti : Rⁿ→ R is linear for each 1 ≤ i ≤ m.

Lemma 1.1. Let S : Rⁿ→ R be a linear map. Then there exists a vector a ∈ Rⁿsuch that S(x) = ha, xi for any x ∈ Rⁿ.

Proof. Let β = {e1, · · · , en} be the standard basis for Rⁿ. We write x = Pn

i=1xiei. By linearity of S, S(x) = x₁S(e₁) + · · · + x_nS(e_n). Let a_i = S(e_i) for 1 ≤ i ≤ n. For a = (a1, · · · , an), we have

S(x) = a1x1+ · · · + anxn= ha, xi.



1

Since Ti : Rⁿ→ R is linear for each 1 ≤ i ≤ m, we can choose bi = (ai1, · · · , ain) ∈ Rⁿ such that T_i(x) = hb_i, xi for any x ∈ Rⁿ. Let A be the m × n matrix

A =











a₁₁ a₁₂ · · · a_1n a21 a22 · · · a2n

... ... . .. ... am1 · · · amn









 ,

i.e. A is the matrix whose i-th row vector is b_i. By definition, T = L_A.

Proposition 1.1. Let L(Rⁿ, R^m) be the space of linear maps from Rⁿ to R^m. Then L(Rⁿ, R^m) is a vector subspace of the space of functions F (Rⁿ, R^m) from Rⁿ to R^m. Theorem 1.1. Define φ : Mm×n(R) → L(Rⁿ, R^m) by A 7→ LA. Then φ is a linear isomor- phism.

Proposition 1.2. Let T : Rⁿ→ R^m be a linear map. Then there exists M > 0 such that kT(x)k_R^m ≤ M kxk_Rⁿ for any x ∈ Rⁿ.

Hence T is a Lipschitz function.

Proof. Let us write T = (T1, · · · , Tm). For each 1 ≤ i ≤ m, we choose bi so that Ti(x) = hb_i, xi for all x ∈ Rⁿ. For any x ∈ Rⁿ,

kT (x)k²

R^m =p|T1(x)|²+ · · · + |Tm(x)|²

= q

|hb₁, xi|²+ · · · + |hb_m, xi|² By Cauchy Schwarz inequality, |hbi, xi|² ≤ kb_ik²

Rⁿkxk²

Rⁿ. For any x ∈ Rⁿ, kT (x)k²_Rm ≤ (kb₁k²_Rn+ · · · + kbmk²_Rn)kxk²_Rⁿ. Let M =

q kb₁k²

Rⁿ+ · · · + kb_mk²

Rⁿ. This proves the assertion. 

Let T : Rⁿ→ R^m be a linear map and M > 0 be as above. For kxk_Rⁿ = 1, we find kT (x)k_R^m≤ M.

This allows us to define the operator norm of T by kT k_op= sup

kxk_Rn=1

kT (x)k_R^m.

Theorem 1.2. The space L(Rⁿ, R^m) of linear maps from Rⁿto R^m together with k · k_opis a real Banach space.

Proof. We leave it to the reader to verify that k · kop is a norm on L(Rⁿ, R^m). The com- pleteness of L(Rⁿ, R^m) wth respect to k · k_opwill be proved later. 

It is not difficult for us to prove the following properties.

Proposition 1.3. Let T ∈ L(Rⁿ, R^m) and S ∈ L(R^m, R^p). Then (1) kT (x)k_R^m ≤ kT k_opkxk_Rn, and

(2) kS ◦ T kop≤ kSk_opkT k_op.

Proof. Let us prove (1). When x = 0, the statement is obvious. When x 6= 0, let y = x/kxk_Rⁿ. Then kT (y)k_R^m ≤ kT k_op. By linearity of T,

kT (y)k_R^m = T

 x

kxk_Rⁿ



= 1

kxk_RⁿkT (x)k_Rⁿ ≤ kT k_op. Multiplying the above inequality both side by kxk_Rⁿ, we obtain (1).

For (2), we use (1):

k(S ◦ T )(x)k_Rp ≤ kSk_opkT (x)k_Rm

≤ kSk_opkT k_opkxk_R^m.

Hence kS ◦ T k_op≤ kSk_opkT k_op. 

Corollary 1.1. If T : Rⁿ→ Rⁿ is linear, then kT^kk_op≤ kT k^k_op for any k ∈ N.

Proof. This can be proved by induction. 

For A ∈ Mmn(R), we define the matrix norm of A to be the operator norm of LA, i.e.

kAk = kL_Ak_op.

Theorem 1.3. The space Mmn(R) of m × n real matrices together with the matrix norm k · k is a real Banach space.

Proof. Let us denote by

kAk₂ = v u u t

m

X

i=1 n

X

j=1

|a_ij|²

kAk_∞= max{|a_ij| : 1 ≤ i ≤ m, 1 ≤ j ≤ n}, for each A ∈ M_mn(R). For any A ∈ Mmn(R),

kAk_∞≤ kAk ≤ kAk₂.

(See class note for the detail of the proof of this inequality) Let (Ak) be a Cauchy sequence in Mmn(R). For any  > 0, there exists K ∈ N such that kAk− A_lk < /4mn whenever k, l ≥ K_. Denote A_k by [a_ij(k)]. For any k, l ≥ K_,

|a_ij(k) − a_ij(l)| ≤ kA_k− A_lk <  4mn.

This implies that (a_ij(k)) is a Cauchy sequence in R for 1 ≤ i ≤ m and 1 ≤ j ≤ n. By completeness of R, (aij(k)) is convergent in R. Denote aij = lim_k→∞aij(k). When k ≥ K,

|a_ij(k) − a_ij| = lim

l→∞|a_ij(k) − a_ij(l)| ≤  4mn. For k ≥ K_,

kA_k− Ak ≤ kA_k− Ak₂ =

sX

ij

|a_ij(k) − a_ij|²≤ r²

4 =  2 < .

This shows that limk→∞Ak= A in Mmn(R). Hence Mmn(R) is a real Banach space. 

This theorem implies that L(Rⁿ, R^m) forms a Banach space with respect to the operator norm.

Given an n × n real matrix A, it is make sense for us to ask if the limit (sk) exists in M_n(R), where

(1.2) s_k= In+A

1!+ · · · + A^k

k!, k ≥ 1.

Theorem 1.4. Let (V, k · k) be a real Banach space and (vn) be a sequence of vectors in V. Suppose thatP∞

n=1kv_nk is convergent in R. Then P∞

n=1v_nis convergent in (V, k · k).

Proof. Exercise. 

Theorem 1.5. (Comparison Test) Let (a_n) and (b_n) be a sequence of nonnegative real numbers. Suppose that

(1) 0 ≤ a_n≤ b_n for any n ≥ 1, (2) P∞

n=1bn is convergent in R.

Then P∞

n=1a_n is convergent.

Proof. Exercise. 

Corollary 1.2. (Comparison Test in Banach Space) Let (V, k · k) be a real Banach space and (vn) be a sequence of vectors in V. Suppose that there exists a sequence of nonnegative real numbers (b_n) such that

(1) kvnk ≤ b_nfor any n ≥ 1, (2) P∞

n=1b_n is convergent in R.

Then P∞

n=1vn is convergent in (V, k · k).

Let v1 = In and vk = A^k−1/(k − 1)! for k ≥ 2 and b1 = 1 and bk = kAk^k−1/(k − 1)!

for k ≥ 2. Since kA^kk ≤ kAk^k for any k ≥ 1, we find kv_kk ≤ b_k for any k ≥ 1. Using elementary calculus, we know P∞

k=1b_k is convergent in R (to e^kAk). By comparison test in Banach space, we find P∞

k=1v_k is convergent in (M_n(R), k · k), i.e. (sk) is convergent in (Mn(R), k · k). We define e^A to be the limit of (s_k) in (Mn(R), k · k), i.e.

e^A= lim

k→∞s_k = I_n+

∞

X

k=1

A^k k!.

Here the right hand side is the expression for limk→∞sk. Similarly, we can define cos A and sin A for any n × n real matrix A by

cos A =

∞

X

k=0

(−1)^k

(2k)! A^2k, sin A =

∞

X

k=0

(−1)^k

(2k + 1)!A^2k+1.

We leave it to the reader to verify that cos A and sin A are defined. (The infinite series of n × n matrices are convergent in Mn(R).)

Theorem 1.6. Let x(t) = e^tAx₀ for t ∈ [0, b]. Then x : [0, b] → Rⁿ is the unique differen- tiable function which solves (1.1).

To prove this theorem, we need to introduce the space of vector valued continuous func- tions. This theorem will be proved later. Let us study more about the space Mn(R).

In calculus, we have seen that for each x ∈ R with |x| < 1, 1

1 − x =

∞

X

k=0

x^k.

We may ask ourself that whether the equality holds for any n × n matrix A with kAk < 1.

Since kAk < 1, P∞

k=0kAk^k is convergent in R. Here A⁰ = In the identity matrix. Since M_n(R) is a real Banach space, P∞

k=0A^k is convergent in M_n(R).

Proposition 1.4. Let A ∈ Mn(R) with kAk < 1 and B =P∞

k=0A^k. Then B(In− A) = (I_n− A)B = I_n.

This implies that In− A is invertible with B = (I_n− A)⁻¹.

Proof. Let sk= In+ A + · · · + A^k−1 for each k ≥ 1. For each k ≥ 1, As_k= A + · · · + A^k= s_k+1− I_n= s_kA.

Using the inequality,

kAs_k− ABk = kA(s_k− B)k ≤ kAkks_k− Bk ks_kA − BAk = ks_kA − BAk ≤ ks_k− BkkAk,

we know lim_k→∞As_k = AB and lim_k→∞s_kA = BA. On the other hand, lim_k→∞(s_k+1− In) = B − In. We find that AB = B − In= BA. This implies that

(I_n− A)B = B(I_n− A) = I_n.

This proves our assertion. 

As an application to this proposition, let us prove the following.

Theorem 1.7. Let GL_n(R) be the set of all real n × n invertible matrices. Then GLn(R) forms an open subset of Mn(R).

Proof. Let A ∈ GLn(R). Choose  = 1/kA⁻¹k. Claim that B(A, ) ⊆ GLn(R). This is equivalent to say that if kB − Ak < , i.e. B ∈ B(A, ), then B is invertible. Observe that

B = B − A + A = (B − A)A⁻¹+ I
A.

Let C = (B − A)A⁻¹ Then B = (I + C)A and kCk ≤ kB − AkkA⁻¹k < 1

kA⁻¹k· kA⁻¹k = 1.

By the previous proposition, I + C is invertible. Since A is invertible and product of any two invertible matrices is again invertible, B = (I + C)A is invertible. Hence B ∈ GLn(R).

We proved that B(A, ) ⊆ GLn(R), and hence A is an interior point of GLn(R). Since A is

arbitrarily in GLn(R), GLn(R) is open. 

Theorem 1.8. Let φ : GLn(R) → GLn(R) be the map φ(A) = A⁻¹ for A ∈ GLn(R). Then φ is continuous.

Proof. Let A ∈ GLn(R) and choose d = 1/2kA⁻¹k. For any B ∈ B(A, d), kB − Ak < d < 1

kA⁻¹k.

Hence B is invertible; hence B(A, d) ⊆ GLn(R). Observe that for B ∈ B(A, d), φ(B) − φ(A) = B⁻¹(A − B)A⁻¹

and hence

kφ(B) − φ(A)k ≤ kB⁻¹kkA − BkkA⁻¹k.

Let us estimate kB⁻¹k when B ∈ B(A, d). For each y ∈ Rⁿ, kyk_Rⁿ = kA⁻¹(Ay)k_Rⁿ≤ kA⁻¹kkAyk_Rⁿ.

Hence kA⁻¹k⁻¹kyk_Rⁿ ≤ kAyk_Rⁿ. By triangle inequality and the norm inequality,

kAyk_Rⁿ = k(A − B)y + Byk_Rⁿ ≤ k(A − B)yk_Rⁿ+ kByk_Rⁿ ≤ kA − Bkkyk_Rⁿ+ kByk_Rⁿ. This shows that for any y ∈ Rⁿ,

(kA⁻¹k⁻¹− kA − Bk)kyk_Rⁿ ≤ kByk_Rⁿ.

Since kA − Bk < d = kA⁻¹k⁻¹/2, (kA⁻¹k⁻¹− kA − Bk) > kA⁻¹k⁻¹/2. This shows that for any y ∈ Rⁿ,

kA⁻¹k⁻¹

2 kyk_Rⁿ ≤ kByk_Rⁿ.

Since B is invertible, for any x ∈ Rⁿ, there exists a unique y ∈ Rⁿso that By = x. We find that y = B⁻¹x and hence kB⁻¹xk_Rⁿ ≤ 2kA⁻¹kkxk_Rⁿ. We find that kB⁻¹k ≤ 2kA⁻¹k. Thus for any B ∈ B(A, ),

kφ(B) − φ(A)k ≤ 2kA⁻¹k²kA − Bk.

For any  > 0, we choose

δ_A, = min

 

2kA⁻¹k², d

 . Then for any B ∈ B(A, δA,),

kφ(B) − φ(A)k < 2kA⁻¹k²δ ≤ 2kA⁻¹k²· 

2kA⁻¹k² = .

This proves that φ is continuous. 

2. Spectral Theorem for symmetric matrices with an application to the computation of Operator Norms

Let A : Rⁿ→ R^m be a linear map. The operator norm of A is defined to be kAk = sup

x∈Sⁿ⁻¹

kAxk, where Sⁿ⁻¹ = {x ∈ Rⁿ: kxk = 1}. Then

kAk² = sup

x∈Sⁿ⁻¹

kAxk²

= sup

x∈Sⁿ⁻¹

hAx, Axi

= sup

x∈Sⁿ⁻¹

hA^tAx, xi.

Denote T = A^tA. Then T : Rⁿ→ Rⁿ is a linear map such that (1) T is symmetric, i.e. T^t= T, and

(2) hT x, xi = kAxk² ≥ 0 for any x ∈ Rⁿ.

Definition 2.1. A linear map T : Rⁿ → Rⁿ is nonnegative definite if it satisfies (1) and (2).

For any nonnegative definite linear map Q : Rⁿ→ Rⁿ, we define a function Q_T : Rⁿ→ R by QT(x) = hT x, xi, x ∈ Rⁿ.

If we denote T by [Tij] and x = (x1, · · · , xn), then Q_T(x) =

n

X

i,j=1

T_ijx_ix_j.

We see that QT is a real valued continuous function. Since Sⁿ⁻¹ is closed and bounded, by Bolzano-Weierstrass Theorem, Sⁿ⁻¹ is sequentially compact. By Extreme value Theorem, QT attains its maximum (and also minimum) on Sⁿ⁻¹. Let λ1 be the maximum of QT on Sⁿ⁻¹ and v₁ ∈ Sⁿ⁻¹ so that Q_T(v₁) = λ₁. Since Q_T(x) ≥ 0 for any x ∈ Rⁿ, λ₁ ≥ 0. Since hT v₁, v₁i = λ₁, h(λ₁I − T )v₁, v₁i = 0. Let us prove that, in fact, we have

h(λ₁I − T )v1, wi = 0 for any w ∈ Rⁿ. If the statement is true, then

T v1 = λ1v1,

i.e. λ1 is an eigenvalue of T and v1 is an eigenvector of T corresponding to eigenvalue λ1. Let B = λ₁I − T. Then hBv₁, v₁i = 0. We want to show that hBv₁, wi = 0 for any w ∈ Rⁿ. Since any w ∈ Rⁿ can be written uniquely as w = av1+ y for a ∈ R and y ∈ {v1}^⊥,

hBv₁, wi = ahBv₁, v₁i + hBv₁, yi = hBv₁, yi.

Hence if we can show that for any y ∈ {v₁}^⊥, hBv₁, yi = 0, then hBv₁, wi = 0 for any w ∈ Rⁿ. Thus we only need to prove that hBv1, wi = 0 is true for w ∈ {v1}^⊥. Since Q_T(v) ≤ λ1

for any v ∈ Sⁿ⁻¹, Q_T(x) ≤ λ₁kxk² for any x ∈ Rⁿ. This implies that hBx, xi ≥ 0 for any x ∈ Rⁿ. For any t ∈ R, and any w ∈ Rⁿ,

0 ≤ hB(v₁+ tw), (v₁+ tw)i

= hBv1, v1i + 2hBv₁, wit + hBw, wit²

= 2hBv1, wit + hBw, wit².

This shows that hBv1, wi² ≤ 0. This shows that hBv₁, wi = 0.

Theorem 2.1. Let T : Rⁿ→ Rⁿ be a nonnegative definite linear map. The number λ₁ = max

x∈Sⁿ⁻¹

hT x, xi

is an eigenvalue of T and any v ∈ Sⁿ⁻¹with hT v, vi = λ is an eigenvector of T corresponding to λ.

Lemma 2.1. Let T, λ1 and v1 be as above. Let V1 = span{v1} and W₁ = V₁^⊥. Then Rⁿ= V₁⊕ W₁ and T (W₁) ⊆ W₁.

Proof. If w ∈ W1, then hv1, wi = 0 and hence

hv₁, T wi = hT v₁, wi = λ₁hv₁, wi = 0.

This shows that T w ∈ V₁^⊥= W1. 

For any x ∈ Rⁿ, we can write x = av1+ w for w ∈ W1. Hence T (x) = aλ₁v₁+ T (w).

Let T1 : W1 → W₁ be the map defined by T1(w) = T (w). Since T is linear, T1 is linear.

Furthermore, Since T is symmetric, for any w₁, w₂ ∈ W₁,

hT₁w₁, w₂i = hT w₁, w₂i = hw₁, T w₂i = hw₁, T₁(w₂)i.

This shows that T1 is also symmetric. For any w ∈ W1, hT₁w, wi = hT w, wi ≥ 0.

This shows that T₁ is nonnegative definite. Set λ2 = max

{w∈W1:kwk=1}

hT₁w, wi.

Remark. The set {w ∈ W1 : kwk = 1} is the intersection of W1 and Sⁿ⁻¹. If we denote v1

by (a₁, · · · , a_n), then

W₁= {(x₁, · · · , x_n) ∈ Rⁿ: a₁x₁+ · · · + a_nx_n= 0}.

Hence W1 is a closed subset of Rⁿ. Therefore W1∩ Sⁿ⁻¹ is closed. Since Sⁿ⁻¹ is bounded, W₁ ∩ Sⁿ⁻¹ is bounded. The intersection {w ∈ W₁ : kwk = 1} is closed and bounded. By Bolzano-Weierstrass Theorem, {w ∈ W1 : kwk = 1} is sequentially compact. We apply the extreme value theorem to find λ₂.

Then λ1 ≥ λ₂ ≥ 0. Choose v₂ ∈ W₁ so that kv2k = 1 and λ₂ = hT1v2, v2i. Then v₁ ⊥ v₂ and T₁v₂ = λ₂v₂. This shows that

T v2 = λ2v2.

By induction, we can find a finite nonincreasing nonnegative real numbers λ₁ ≥ λ₂≥ · · · λ_n≥ 0

and an orthonormal basis {vi: 1 ≤ i ≤ n} for Rⁿ such that T vi = λivi for 1 ≤ i ≤ n.

Since {v_i : 1 ≤ i ≤ n} is an orthonormal basis for Rⁿ, for any x ∈ Rⁿ, we can write x = Pn

i=1hx, v_iiv_i. By linearity of T, we obtain T x =Pn

i=1λihx, v_iiv_i. Let Λ = diag(λ1, · · · , λn) and V be the n × n matrix whose i-th column vector is v_i. Then T V = V Λ which implies that T = V ΛV^t by V^tV = I.

Theorem 2.2. (Spectral Theorem for nonnegative definite linear map) Let T : Rⁿ → Rⁿ be a nonnegative definite linear map. Then there exist a finite nonincreasing nonnegative real numbers

λ₁≥ λ₂ ≥ · · · ≥ λ_n≥ 0 and an orthonormal basis {v_i: 1 ≤ i ≤ n} for Rⁿ such that

(1) T v_i= λ_iv_i for 1 ≤ i ≤ n.

(2) T x =Pn

i=1λihx, v_iiv_i.

(3) T = V ΛV^twhere Λ = diag(λ1, · · · , λn) and U is the n × n matrix whose i-th column vector is vi.

Let S : Rⁿ→ Rⁿ be any symmetric linear map. We consider m = min

x∈Sⁿ⁻¹

hSx, xi.

Set T = S − mI. Then T is nonnegative definite. By spectral theorem for T, we choose λ_i ∈ R and vi ∈ Rⁿ such that T v_i = λ_iv_i as above, Then Sv_i = (λ_i+ m)v_i for 1 ≤ i ≤ n.

Let µi = λi+ m. Then

Svi= µivi for 1 ≤ i ≤ n.

We see that vi is also an eigenvector of S with eigenvalue µi. Furthermore, Sx =

n

X

i=1

µiλihx, v_iiv_i. Thus we prove that

Corollary 2.1. Spectral Theorem holds for any symmetric linear map.

Let us go back to the computation of kAk. It follows from the definition that kAk² is the largest eigenvalue of A^tA. Let us denote λ1(A^tA) the largest eigenvalue of A^tA. Then kAk =pλ₁(A^tA).

Definition 2.2. Let λi(A^tA) be the eigenvalue of A^tA such that λ₁(A^tA) ≥ · · · ≥ λ_n(A^tA) ≥ 0.

The i-th singular value of A is defined to be si(A) =p

λi(A^tA).

Corollary 2.2. Let A : Rⁿ→ R^m be a linear map. Then kAk = s₁(A).

Now let us introduce the singular value decomposition for the any linear map. Let us recall a basic theorem in linear algebra.

Proposition 2.1. Let A : Rⁿ→ R^m be a linear map. Then (1) (Im A)^⊥= ker A^t

(2) (Im A^t)^⊥= ker A.

Proof. Let z ∈ (Im A)^⊥. Then hAx, zi = 0 for any x ∈ Rⁿ. hx, A^tzi = hAx, zi = 0 for any x ∈ Rⁿ. Therefore A^tz = 0. We see that z ∈ ker A^t. If z ∈ ker A^t, then A^tz = 0. Hence hx, A^tzi = 0 for any x ∈ Rⁿ which implies that hAx, zi = 0 for any x ∈ Rⁿ. Therefore

z ∈ (Im A)^⊥. 

Choose an orthonormal basis {vi : 1 ≤ i ≤ n} such that A^tAvi = s²_ivi for 1 ≤ i ≤ n. Let us assume that si= 0 for i > k and sk6= 0. Let z_i= Avi for 1 ≤ i ≤ k. We find A^tzi = s²_ivi

for 1 ≤ i ≤ k. Let us compute hz_i, z_ji for any 1 ≤ i, j ≤ k :

hz_i, zji = hAv_i, Avji = hA^tAvi, vji = s²_ihv_i, vji = s²_iδij.

Hence {z_i: 1 ≤ i ≤ k} forms an orthogonal subset of Im A. Since Av_j = 0 for k + 1 ≤ j ≤ n, nullity A = dim ker A = n − k. By rank-nullity lemma,

rank A + nullity A = n

which shows that rank A = k. Since {z_i : 1 ≤ i ≤ k} is orthogonal, it is linearly independent.

We see that {zi : 1 ≤ i ≤ k} forms an orthogonal basis for Im A. Let ui= zi/sifor 1 ≤ i ≤ k.

Then {u_i : 1 ≤ i ≤ k} forms an orthonormal basis for Im A. We write Ax =

k

X

i=1

hAx, u_iiu_i=

k

X

i=1

hx, A^tuiiu_i. Since A^tui= A^t(zi/si) = sivi, we find

Ax =

k

X

i=1

sihx, v_iiu_i.

We can extend {ui : 1 ≤ i ≤ k} to an orthonormal basis {ui : 1 ≤ i ≤ m} for R^m. Let U be the m × m matrix whose i-th column vector is u_i and V be the n × n matrix whose j-th column vector is vi. Then AV = U Σ. Since V^tV = In(V is an orthogonal matrix), then we obtain the matrix form of the singular value decomposition

A = U ΣV^t. Here Σ = diag(s1, · · · , sn).

Theorem 2.3. (Singular Value Decomposition for any linear map) Let A : Rⁿ → R^m be any linear map. Suppose k = rank A. There exists an orthonormal basis {vi} for Rⁿ and orthonormal basis {u_i} for Im A and a finite nonnegative nonincreasing sequence of real numbers s1≥ · · · ≥ s_k such that

(1) A^tAv_i= s²_iv_i for 1 ≤ i ≤ k and A^tAv_j = 0 for k + 1 ≤ j ≤ n and

(2) Avi = siui for 1 ≤ i ≤ k and Avj = 0 for k +1 ≤ j ≤ n and A^tui = sivifor 1 ≤ i ≤ k, and

(3) Ax =Pk

i=1sihx, v_iiu_i for any x ∈ Rⁿ.

(4) A = U ΣV^t, where U is the m × m matrix whose i-th column vector is u_i and V is the n × n matrix whose j-th column vector is vi and Σ = diag(s1, · · · , sn).

Example 2.1. Let A =1 1 2 2

 . (1) Find kAk.

(2) Find the singular value decomposition of A.

The rank of A is one and the image of A is spanned by 1 2



. Take u₁ = ^√1

5

1 2

 and v1 = ^√1

2

1 1



. Then Av1 =√

10u1. Take v2 = ^√1

2

−1 1



and u2 = ^√1

5

−2 1



. Then Av2 = 0.

The larges singular value of A is√

10 and hence kAk =√

10. We see that A

" ₁

√2 −^√1

1 2

√ 2

√1 2

#

=

" ₁

√5 −^√2

2 5

√ 5

√1 5

#√10 0

0 0

 .

Hence we obtain the singular value decomposition of A (in the matrix form) A =

" ₁

√5 −^√2

2 5

√ 5

√1 5

#√10 0

0 0

" ₁

√2

√1 2

−^√1

2

√1 2

# .