Here are some properties of orthogonal complements (the proof is assigned as Problem 3.31).
Lemma 3.12. If A is a subset of an inner product space H, then the following statements hold.
(a) A⊥ is a closed subspace of H.
(b) H⊥= {0} and {0}⊥ = H.
(c) If A ⊆ B, then B⊥⊆ A⊥. (d) A ⊆ (A⊥)⊥. ♦
Later we will prove that if M is a closed subspace of a Hilbert space, then (M⊥)⊥= M (see Lemma 3.16).
3.6 Orthogonal Projections
The following theorem states that if S is a closed and convex subset of a Hilbert space H, then for each vector x ∈ H there exists is a unique vector y ∈ S that is closest to x. Looking at the proof, we can see why we must assume in this result that H is a Hilbert space and not just an inner product space—we need to know that every Cauchy sequence in H will converge to an element of H.
Theorem 3.13 (Closest Point Theorem). Let H be a Hilbert space, and let S be a nonempty closed, convex subset of H. Given any x ∈ H there exists a unique vector y ∈ S that is closest to x. That is, there is a unique vector y ∈ S that satisfies
kx − yk = dist(x, S) = inf©kx − kk : k ∈ Sª.
Proof. Set d = dist(x, S). Then, by definition of infimum, there exist vectors yn ∈ S such that kx − ynk → d as n → ∞, and for each of these vectors we have kx − ynk ≥ d. Therefore, if we fix ε > 0 then we can find an integer N > 0 such that
d2≤ kh − ynk2≤ d2+ ε2 for all n > N.
Set
p = ym+ yn
2 .
Since p is the midpoint of the line segment joining ymto yn we have p ∈ S, and therefore
kx − p k ≥ dist(x, S) = d.
Using the Parallelogram Law, it follows that if m, n > N, then
kyn− ymk2+ 4d2 ≤ kyn− ymk2+ 4 kx − p k2
= k(x − yn) − (x − ym)k2+ k(x − yn) + (x − ym)k2
= 2¡kx − ynk2+ kx − ymk2¢
≤ 4 (d2+ ε2).
Rearranging, we see that kym−ynk ≤ 2ε for all m, n > N. Therefore {yn}n∈N
is a Cauchy sequence in H. Since H is complete, this sequence must converge, say to y. Since S is closed and yn ∈ S for every n, the vector y must belong to S. Also, since x − yn→ x − y, it follows from the continuity of the norm that
kx − yk = lim
n→∞kx − ynk = d.
Hence y is a point in S that is closest to x.
It only remains to show that y is the unique point in S that is closest to x. If z ∈ S is also a closest point, then kx − yk = d = kx − zk. Further, the midpoint p = (y + z)/2 belongs to S, so kx − p k ≥ d. Applying the Parallelogram Law again, we see that
4d2 = 2¡kx − yk2+ kx − zk2¢
= k(x − y) − (x − z)k2+ k(x − y) + (x − z)k2
= ky − zk2+ 4 kx − p k2
≥ ky − zk2+ 4d2.
Rearranging this yields ky − zk ≤ 0, which implies that y = z. ⊓⊔
In particular, every closed subspace M of H is nonempty, closed, and convex. For this setting we introduce a name for the point in M that is closest to a given vector x. We also use the same name to denote the function that maps x to the point in M that is closest to x.
Definition 3.14 (Orthogonal Projection). Let M be a closed subspace of a Hilbert space H.
(a) Given x ∈ H, the unique vector p ∈ M that is closest to x is called the orthogonal projection of x onto M.
(b) The function P : H → H defined by P x = p, where p is the orthog-onal projection of x onto M, is called the orthogorthog-onal projection of H onto M. ♦
Since the orthogonal projection p is the vector in M that is closest to x, we can think of p as being the best approximation to x by vectors from M. The difference vector e = x − p is the error in this approximation. The following lemma states that the orthogonal projection of x is the unique vector p ∈ H such that the error vector e = x − p is orthogonal to M.
3.6 Orthogonal Projections 45
Lemma 3.15. Let M be a closed subspace of a Hilbert space H. Given vectors x and p in H, the following four statements are equivalent.
(a) p is the orthogonal projection of x onto M, i.e., p is the unique point in M that is closest to h.
(b) p ∈ M and x − p ⊥ M.
(c) x = p + e where p ∈ M and e ∈ M⊥.
(d) e = x − p is the orthogonal projection of x onto M⊥.
Proof. We will prove one implication, and assign the task of proving the remaining (easier) implications as Problem 3.32. We assume that scalars in this problem are complex, but the proof remains valid if we assume that scalars are real.
(a) ⇒ (b). Let p be the (unique) point in M closest to x, and let e = p − x.
Choose any vector y ∈ M. We must show that hy, ei = 0. Since M is a subspace, p + λy ∈ M for any scalar λ ∈ C. Hence,
kx − p k2 ≤ kx − (p + λy)k2 = k(x − p) − λyk2
= kx − p k2− 2 Rehλy, x − pi + |λ|2kyk2
= kx − p k2− 2 Re(λhy, ei) + |λ|2kyk2. Therefore,
∀ λ ∈ C, 2 Re(λhy, ei) ≤ |λ|2kyk2.
If we consider λ = t > 0, then we can divide through by t to get
∀ t > 0, 2 Rehy, ei ≤ t kyk2.
Letting t → 0+, we conclude that Rehy, ei ≤ 0. If we similarly take λ = t < 0 and let t → 0−, we obtain Rehy, ei ≥ 0, so Rehy, ei = 0.
Finally, by taking λ = it with t > 0 and then λ = it with t < 0, and then λ = it with t < 0, it follows that Im(hy, ei) = 0. ⊓⊔
We will use Lemma 3.15 to compute the orthogonal complement of the orthogonal complement of a set.
Lemma 3.16. Let H be a Hilbert space.
(a) If M is a closed subspace of H, then (M⊥)⊥= M.
(b) If A is any subset of H, then
A⊥ = span(A)⊥ = span(A)⊥ and (A⊥)⊥ = span(A).
Proof. (a) We are given a closed subspace M in H. If f ∈ M then hf, gi = 0 for every g ∈ M⊥, so f ∈ (M⊥)⊥. Hence M ⊆ (M⊥)⊥.
Conversely, suppose that f ∈ (M⊥)⊥. Let p be the orthogonal projection of f onto M. Since M is a closed subspace, we have f = p + e where p ∈ M
and e ∈ M⊥. Since f and p belong to M and we have seen that M ⊆ (M⊥)⊥, it follows that e = f − p ∈ (M⊥)⊥. However, we also know that e ∈ M⊥, so e is orthogonal to itself and therefore is zero. Hence f = p + 0 ∈ M. This shows that (M⊥)⊥ ⊆ M.
(b) Now we are given an arbitrary subset A of H. Let M = span(A). We must show that A⊥= M⊥. Since A ⊆ M, we have M⊥ ⊆ A⊥.
Suppose that f ∈ A⊥. Then f ⊥ A, i.e., f is orthogonal to every vector in A. By forming linear combinations, it follows that f ⊥ span(A). By taking limits, it follows from this that f ⊥ span(A) = M. Hence f ∈ M⊥ and therefore A⊥⊆ M⊥.
Thus, we have shown that A⊥= M⊥. Applying part (a), we conclude that (A⊥)⊥= (M⊥)⊥= M. ⊓⊔
By taking A to be a sequence, we obtain the following useful corollary.
Corollary 3.17. Given a sequence {xn}n∈N in a Hilbert space H, the follow-ing two statements are equivalent.
(a) {xn}n∈N is a complete sequence, i.e., span{xn}n∈N is dense in H.
(b) The only vector in H that is orthogonal to every xn is the zero vector, i.e.,
x ∈ H and hx, xni = 0 for every n =⇒ x = 0.
Proof. By definition, {xn} is complete if and only if its finite linear span is dense in H. Lemma 3.16 tells us that span{xn} = {xn}⊥, so we conclude that {xn} is complete if only if {xn}⊥= {0}. ⊓⊔