It is possible to strengthen theorem 2.4 and some of the interim results used in its derivation using reasonably elementary methods. The key additional idea required is the Wielandt-Hoffman theorem [27], a result from matrix theory that is of independent interest. The theorem is stated and a proof following Wilkinson [28] is presented for completeness.
Lemma 2.4 (Wielandt-Hoffman theorem) Given two Hermitian matrices A and B with eigenvalues αk and βk in nonincreasing order, respectively, then
n−1
nX−1 k=0
|αk− βk|2 ≤ |A − B|2.
Proof: Since A and B are Hermitian, we can write them as A = U diag(αk)U∗, B = W diag(βk)W∗, where U and W are unitary. Since the weak norm is not effected by multiplication by a unitary matrix,
|A − B| = |Udiag(αk)U∗− W diag(βk)W∗|
= |diag(αk)U∗− U∗W diag(βk)W∗|
= |diag(αk)U∗W − U∗W diag(βk)|
= |diag(αk)Q− Qdiag(βk)|,
where Q = U∗W = {qi,j} is also unitary. The (i, j) entry in the matrix diag(αk)Q− Qdiag(βk) is (αi− βj)qi,j and hence
|A − B|2 = n−1
nX−1 i=0
nX−1 j=0
|αi− βj|2|qi,j|2 ∆=
nX−1 i=0
nX−1 j=0
|αi− βj|2pi,j (2.38)
where we have defined pi,j − n−1|qi,j|2 Since Q is unitary, we also have that
nX−1 i=0
|qi,j|2 =
nX−1 j=0
|qi,j|2 = 1 (2.39)
or nX−1
i=0
pi,j =
nX−1 j=0
pi,j = 1
n. (2.40)
2.4. ABSOLUTELY EQUAL EIGENVALUE DISTRIBUTIONS 17 This can be interpreted in probability terms: pi,j = n−1|qi,j|2 is a probability mass function or pmf on{0, 1, . . . , n − 1}2 with uniform marginal probability mass functions. Recall that it is assumed that the eigenvalues are ordered so that α0 ≥ α1 ≥ α2 ≥ · · · and β0 ≥ β1 ≥ β2 ≥ · · ·.
We claim that for all such matrices P satisfying (2.40), the right hand side of (2.38) is minimized by P = n−1I, where I is the identity matrix, so that
nX−1 i=0
nX−1 j=0
|αi− βj|2pi,j ≥nX−1
i=0
|αi− βi|2,
which will prove the result. To see this suppose the contrary. Let ` be the smallest integer in {0, 1, . . . , n − 1} such that P has a nonzero element off the diagonal in either row ` or in column `. If there is a nonzero element in row ` off the diagonal, say p`,a then there must also be a nonzero element in column ` off the diagonal, say pb,` in order for the constraints (2.40) to be satisfied. Since ` is the smallest such value, ` < a and ` < b. Let x be the smaller of pl,a and pb,l. Form a new matrix P0 by adding x to p`,` and pb,a and subtracting x from pb,` and p`,a. The new matrix still satisfies the constraints and it has a zero in either position (b, `) or (`, a). Furthermore the norm of P0 has changed from that of P by an amount
x³(α`− β`)2+ (αb− βa)2− (α`− βa)2− (αb− β`)2´
= −x(α`− αb)(β`− βa)
≤ 0
since ` > b, ` > a, the eigenvalues are nonincreasing, and x is positive.
Continuing in this fashion all nonzero offdiagonal elements can be zeroed out without increasing the norm, proving the result. 2 Applying the Cauchy-Schwartz inequality and the Wielandt-Hoffman the-orem yields the following strengthening of lemma 2.3,
n−1
nX−1 k=0
|αk− βk| ≤ n−1
vu utnX−1
k=0
(αk− βk)2√
n≤ |An− Bn|,
which we formalize as the following lemma.
Lemma 2.5 Given two Hermitian matrices A and B with eigenvalues αn and βn in nonincreasing order, respectively, then
n−1
nX−1 k=0
|αk− βk| ≤ |A − B|.
Note in particular that the absolute values are outside the sum in lemma 2.3 and inside the sum in lemma 2.5. As was done in the weaker case, the result can be used to prove a stronger version of theorem 2.4. This line of reasoning, using the Wielandt-Hoffman theorem, was pointed out by William F. Trench who used special cases in his paper [20]. Similar arguments have become standard for treating eigenvalue distributions for Toeplitz and Hankel matrices. See, for example, [7, 25, 3]. The following theorem provides the derivation. The specific statement result and its proof follow from a private communication from William F. Trench.
Theorem 2.5 Let An and Bnbe asymptotically equivalent sequences of Her-mitian matrices with eigenvalues αn,kand βn,kin nonincreasing order, respec-tively. Since Anand Bn are bounded there exist finite numbers m and M such that
m≤ αn,k, βn,k ≤ M , n = 1, 2, . . . k = 0, 1, . . . , n − 1. (2.41) Let F (x) be an arbitrary function continuous on [m, M ]. Then
nlim→∞n−1
nX−1 k=0
|F (αn,k)− F (βn,k)| = 0. (2.42)
Note that the theorem strengthens the result of theorem 2.4 because of the magnitude inside the sum. Following Trench [21] in this case the eigenvalues are said to be asymptotically absolutely equally distributed.
Proof: From lemma 2.5 n−1X
k=0
|αn,k− βn,k| ≤ |An− Bn|, (2.43)
which implies (2.42) for the case F (r) = r. For any nonnegative integer j
|αjn,k− βn,kj | ≤ j max(|m|, |M|)j−1|αn,k− βn,k|. (2.44)
2.4. ABSOLUTELY EQUAL EIGENVALUE DISTRIBUTIONS 19 By way of explanation consider a, b ∈ [m, M]. Simple long division shows that
aj− bj a− b =
Xj l=1
aj−lbl−1 so that
|aj − bj
a− b | = |aj − bj|
|a − b|
= |
Xj l=1
aj−lbl−1|
≤
Xj l=1
|aj−lbl−1|
=
Xj l=1
|a|j−l|b|l−1
≤ j max(|m|, |M|)j−1,
which proves (2.44). This immediately implies that (2.42) holds for functions of the form F (r) = rj for positive integers j, which in turn means the result holds for any polynomial. If F is an arbitrary continuous function on [m, M ], then from theorem 2.3 given ² > 0 there is a polynomial P such that
|P (u) − F (u)| ≤ ², u ∈ [m, M].
Using the triangle inequality, n−1
nX−1 k=0
|F (αn,k)− F (βn,k)|
= n−1
nX−1 k=0
|F (αn,k)− P (αn,k) + P (αn,k)− P (βn,k) + P (βn,k)− F (βn,k)|
≤ n−1nX−1
k=0
|F (αn,k)− P (αn,k)| + n−1nX−1
k=0
|P (αn,k)− P (βn,k)|
+n−1
nX−1 k=0
|P (βn,k)− F (βn,k)|
≤ 2² + n−1nX−1
k=0
|P (αn,k)− P (βn,k)|
As n → ∞ the remaining sum goes to 0, which proves the theorem since ²
can be made arbitrarily small. 2
Chapter 3
Circulant Matrices
The properties of circulant matrices are well known and easily derived ([15], p. 267,[6]). Since these matrices are used both to approximate and explain the behavior of Toeplitz matrices, it is instructive to present one version of the relevant derivations here.
A circulant matrix C is one having the form
C =
c0 c1 c2 · · · cn−1
cn−1 c0 c1 c2 ... cn−1 c0 c1 . ..
... . .. . .. ... c2 c1 c1 · · · cn−1 c0
, (3.1)
where each row is a cyclic shift of the row above it. The structure can also be characterized by noting that the (k, j) entry of C, Ck,j, is given by
Ck,j = c(j−k) mod n,
which identifies C as a special type of Toeplitz matrix.
The eigenvalues ψk and the eigenvectors y(k) of C are the solutions of
Cy = ψ y (3.2)
or, equivalently, of the n difference equations
mX−1 k=0
cn−m+kyk+
nX−1 k=m
ck−myk= ψ ym; m = 0, 1, . . . , n− 1. (3.3)
21
Changing the summation dummy variable results in
n−1−mX
k=0
ckyk+m+
nX−1 k=n−m
ckyk−(n−m)= ψ ym; m = 0, 1, . . . , n− 1. (3.4) One can solve difference equations as one solves differential equations — by guessing an (hopefully) intuitive solution and then proving that it works.
Since the equation is linear with constant coefficients a reasonable guess is yk = ρk (analogous to y(t) = esτ in linear time invariant differential equa-tions). Substitution into (3.4) and cancellation of ρm yields
n−1−mX
k=0
ckρk+ ρ−n
nX−1 k=n−m
ckρk= ψ.
Thus if we choose ρ−n = 1, i.e., ρ is one of the n distinct complex nth roots of unity, then we have an eigenvalue
ψ =
nX−1 k=0
ckρk (3.5)
with corresponding eigenvector
y = n−1/2³1, ρ, ρ2, . . . , ρn−1´0, (3.6) where the prime denotes transpose where the normalization is chosen to give the eigenvector unit energy. Choosing ρm as the complex nth root of unity, ρm = e−2πim/n, we have eigenvalue
ψm =
nX−1 k=0
cke−2πimk/n (3.7)
and eigenvector
y(m) = n−1/2³1, e−2πim/n,· · · , e−2πi(n−1)/n´
. From (3.7) we can write
C = U ΨU∗, (3.8)
where
U = ny(0)|y(1)| · · · |y(n−1)o
= n−1/2ne−2πimk/n; m, k = 0, 1, . . . , n− 1o
23
Ψ ={ψkδk−j} where δ is the Kronecker delta,
δm =
½1 m = 0 0 otherwise.
To verify (3.8) denote that the (k, j)th element of U ΨU∗ by ak,j and that
ob-serve that ak,jwill be the product of the kth row of U Ψ, which is{n−1/2e−2πimk/nΨk; m = 0, 2, . . . , n− 1}, times the jth row of U, {n−1/2e2πimj/n; m = 0, 2, . . . , n− 1}
so that
ak,j = n−1
nX−1 m=0
e2πim(j−k)/nψm
= n−1
nX−1 m=0
e2πim(j−k)/n nX−1 r=0
cre−2πimr/n
= n−1
nX−1 r=0
cr nX−1 m=0
e2πim(j−k−r)/n. (3.9)
But we have
nX−1 m=0
e2πim(j−k−r)/n =
½n k− j = −r mod n
0 otherwise .
so that ak,j = c−(k−j) mod n. Thus (3.8) and (3.1) are equivalent. Furthermore (3.9) shows that any matrix expressible in the form (3.8) is circulant.
It should also be familiar to those with standard engineering backgrounds that ψm in (3.7) is simply the discrete Fourier transform (DFT) of the sequence ck and (3.8) can be interpreted as a combination of the Fourier inversion formula and the Fourier cyclic shift formula.
Since C is unitarily similar to a diagonal matrix it is normal. Note that all circulant matrices have the same set of eigenvectors. This leads to the following properties.
Theorem 3.1 Let C ={ck−j} and B = {bk−j} be circulant n × n matrices with eigenvalues
ψm =
nX−1 k=0
cke−2πimk/n
βm =
nX−1 k=0
bke−2πimk/n , respectively.
1. C and B commute and
CB = BC = U∗γU ,
where γ ={ψmβmδk,m}, and CB is also a circulant matrix.
2. C + B is a circulant matrix and
C + B = U∗ΩU, where Ω ={(ψm+ βm)δk,m}
3. If ψm 6= 0; m = 0, 1, . . . , n − 1, then C is nonsingular and C−1 = U∗Ψ−1U
so that the inverse of C can be straightforwardly constructed.
Proof. We have C = U∗ΨU and B = U∗ΦU where Ψ and Φ are diagonal matrices with elements ψmδk,m and βmφk,m, respectively.
1. CB = U∗ΨU U∗ΦU
= U∗ΨΦU
= U∗ΦΨU = BC
Since ΨΦ is diagonal, (3.9) implies that CB is circulant.
2. C + B = U∗(Ψ + Φ)U 3. C−1 = (U∗ΨU )−1
= U∗Ψ−1U
if Ψ is nonsingular.
2 Circulant matrices are an especially tractable class of matrices since inverses, products, and sums are also circulant matrices and hence both straightforward to construct and normal. In addition the eigenvalues of such matrices can easily be found exactly.
In the next chapter we shall see that certain circulant matrices asymptoti-cally approximate Toeplitz matrices and hence from Chapter 2 results similar to those in theorem 3.1 will hold asymptotically for Toeplitz matrices.
Chapter 4
Toeplitz Matrices
4.1 Bounded Toeplitz Matrices
In this chapter the asymptotic behavior of inverses, products, eigenvalues, and determinants of finite Toeplitz matrices is derived by constructing an asymptotically equivalent circulant matrix and applying the results of the previous chapters. Consider the infinite sequence{tk; k = 0,±1, ±2, · · ·} and define the finite (n× n) Toeplitz matrix Tn = {tk−j} as in (1.1). Toeplitz matrices can be classified by the restrictions placed on the sequence tk. If there exists a finite m such that tk = 0, |k| > m, then Tn is said to be a finite order Toeplitz matrix. If tk is an infinite sequence, then there are two common constraints. The most general is to assume that the tk are square summable, i.e., that
X∞ k=−∞
|tk|2 <∞ . (4.1)
Unfortunately this case requires mathematical machinery beyond that as-sumed in this paper; i.e., Lebesgue integration and a relatively advanced knowledge of Fourier series. We will make the stronger assumption that the tk are absolutely summable, i.e.,
X∞ k=−∞
|tk| < ∞. (4.2)
This assumption greatly simplifies the mathematics but does not alter the fundamental concepts involved. As the main purpose here is tutorial and we wish chiefly to relay the flavor and an intuitive feel for the results, this paper
25
will be confined to the absolutely summable case. The main advantage of (4.2) over (4.1) is that it ensures the existence and continuity of the Fourier series f (λ) defined by
f (λ) =
Not only does the limit in 4.3) converge if (4.2) holds, it converges uniformly for all λ, that is, we have that
where the righthand side does not depend on λ and it goes to zero as n→ ∞ from (4.2), thus given ² there is a single N , not depending on λ, such that
¯¯¯¯ Note that (4.2) is indeed a stronger constraint than (4.1) since
X∞
Note also that (4.2) implies that f (λ) is bounded since
|f(λ)| ≤ X∞
4.1. BOUNDED TOEPLITZ MATRICES 27 The matrix Tn will be Hermitian if and only if f is real, in which case we denote the least upper bound and greatest lower bound of f (λ) by Mf and mf, respectively. Observe that max(|mf|, |Mf|) ≤ M|f|.
Since f (λ) is the Fourier series of the sequence tk, we could alternatively begin with a bounded and hence Riemann integrable function f (λ) on [0, 2π]
(|f(λ)| ≤ M|f| < ∞ for all λ) and define the sequence of n × n Toeplitz matrices
Tn(f ) =
½
(2π)−1
Z 2π
0
f (λ)e−i(k−j)dλ ; k, j = 0, 1,· · · , n − 1¾ . (4.5) As before, the Toeplitz matrices will be Hermitian iff f is real. The as-sumption that f (λ) is Riemann integrable implies that f (λ) is continuous except possibly at a countable number of points. Which assumption is made depends on whether one begins with a sequence tk or a function f (λ) — either assumption will be equivalent for our purposes since it is the Riemann integrability of f (λ) that simplifies the bookkeeping in either case. Before finding a simple asymptotic equivalent matrix to Tn, we use Corollary 2.1 to find a bound on the eigenvalues of Tn when it is Hermitian and an upper bound to the strong norm in the general case.
Lemma 4.1 Let τn,k be the eigenvalues of a Toeplitz matrix Tn(f ). If Tn(f ) is Hermitian, then
mf ≤ τn,k≤ Mf. (4.6)
Whether or not Tn(f ) is Hermitian,
k Tn(f )k≤ 2M|f| (4.7)
so that the matrix is uniformly bounded over n if f is bounded.
Proof. Property (4.6) follows from Corollary 2.1:
maxk τn,k = max
x (x∗Tnx)/(x∗x) (4.8) mink τn,k = min
x (x∗Tnx)/(x∗x)
so that Combining (4.9)-(4.10) results in
mf ≤
which with (4.8) yields (4.6). Alternatively, observe in (4.11) that if e(k) is the eigenvector associated with τn,k, then the quadratic form with x = e(k) yields x∗Tnx = τn,kPnk=0−1|xk|2. Thus (4.11) implies (4.6) directly. 2 We have already seen in (2.13) that if Tn(f ) is Hermitian, thenk Tn(f )k=
maxk|τn,k|=∆ |τn,M|, which we have just shown satisfies |τn,M| ≤ max(|Mf|, |mf|) which in turn must be less than M|f|, which proves (4.7) for Hermitian ma-trices.. Suppose that Tn(f ) is not Hermitian or, equivalently, that f is not real. Any function f can be written in terms of its real and imaginary parts, f = fr + ifi, where both fr and fi are real. In particular, fr = (f + f∗)/2
4.2. FINITE ORDER TOEPLITZ MATRICES 29