3.3 Causal Interpretation
Massey [12], [13] shows a way of graphically determining independence of random variables based on causal interpretations. A causal interpretation is an ordered list of random variables. The idea behind a specific choice of order lies in the causality of the system.
Loosely speaking in an engineering way of thinking, we would like to think of some random variables being generated ”first” and some ”later based on” the generation of the others.
Note that a priori every ordered list is a valid causal interpretation, but some choices will be more useful keeping the engineering idea in mind.
As an example consider the vector
V = (M, X1k,Y1k,Hk1,Zk1,Fk1), (3.10) where all components are random variables defined in Chapter 2.
For simplicity assume for the moment that all components take value in discrete al-phabets1. We choose the following causal interpretation:
(M, H1, ...,Hk,Z1, ...,Zk,F1, X1,Y1,F2, X2,Y2, ...,Fk, Xk,Yk). (3.11) If we consider now the entropy of V and write it as a sum using the chain rule
H(V) ='
j
H! V(j))
)V(1), ..., V(j−1)"
, (3.12)
then we see that our choice of a causal interpretation for V simplifies the expression for the entropy significantly:
H(V) = H(M ) + H(H1) + H(H2|H1) + · · · + H(Hk|Hk1) + H(Z1) + · · · + H(Zk) + H(F1)
+ H(X1|F1, M) + H(Y1|X1,H1,Z1) + H(F2|Y1)
+ H(X2|F21, M) + H(Y2|X2,H2,Z2) + · · · + H(Fk|Yk−11 )
+ H(Xk|Fk1, M) + H(Yk|Xk,Hk,Zk) (3.13) Massey calls this a causal-order expansion of H(V). It can easily be depicted graphically in a causality graph, which is a directed graph with an edge from vertex V(j1) to V(j2) if and only if V(j1) is in the conditioning expression for H!
V(j2))
)V(1), ..., V(j2−1)"
. We shall say that a vertex V(j1) is causally prior to vertex V(j2) if there is a directed path from V(j1) to V(j2).
In our case the corresponding graph of (3.11) is shown in Figure 3.2.
Note that once we have established the graph, we do not consider the entropy anymore.
We only used the entropy in order to be able to invoke the chain rule in establishing the
”dependencies” between the different components.
1We will drop this assumption soon again, however, here it simplifies notation considerably because we need not worry about differential entropy. In the end, we are not interested in the entropy at all, but in the ”dependencies” between the components.
Chapter 3 Mathematical Preliminaries
X1 X2 X3 Xk−1 Xk
F1 F2 F3 Fk−1 Fk
Z1 Z2 Z3 Zk−1 Zk
Y1 Y2 Y3 Yk−1 Yk
H1 H2 H3 Hk−1 Hk
M
Figure 3.2: The causality graph of our model.
A causality graph is very useful when determining the statistical dependence between two groups of random variables possibly conditioned on a third group.
To state this property in more clarity let A, B, C ⊂ {1, ..., length(V)} be three index sets. Let V(A) denote a vector containing as components of all components of V whose indices are in A, similarly, define V(B) and V(C).
Any causality graph of V can now be used in order to investigate the independence of V(A) and V(B)when conditioned on V(C). To that goal consider the following procedure:
• from the specify causality graph take the subgraph causally relevant2 to V(A∪B∪C);
• delete all edges leaving any component of V(C);
• drop all directions of the remaining edges;
• if now all components of V(A) are unconnected to the components of V(B), then V(A) is statistically independent of V(B) when conditioned on V(C).
Note that using this procedure we only make statements about the independence, but not about possible dependences, i.e., if the components of V(A)and V(B)are not disconnected, then they might be statistically dependent or independent.
2A subgraph causally relevant to some ˜Vconsists of all those vertices that are either components of ˜V or causally prior to ˜Vin the given causal interpretation, together with the edges connecting these vertices.
Chapter 4
Capacity and Fading Number without Feedback
It has been shown in [8] that the capacity of general regular SIMO fading channels under either an average-power constraint or a peak-power constraint is
C(E) = log(1 + log(1 + E)) + χ({Hk}) + o(1), (4.1) where o(1) denotes terms that tend to zero as E tends to infinity, and
E↑∞lim{C(E) − log log E} < 0. (4.2)
Therefore, we can define χ({Hk})! lim
E↑∞{C(E) − log log E} (4.3)
= hλ
,Hˆ0eiΘ0 ))
)% ˆH"eiΘ!&−1
"=−∞
-− log 2 + nRE#
log %H0%2$
− h! H0)
)H−1−∞"
(4.4) where the second equality is given by [8] and hλ(·) is defined in 3.1.2. Here, {Θk} is IID
∼ U((−π, π]) and independent of {Hk}.
From (4.1) it is obvious that the capacity of the fading channel (2.1) grows extremely slowly at large power. Indeed, log(1 + log(1 + E)) grows so slowly that, for the smallest values of E for which o(1) ≈ 0, the (constant!) fading number χ usually is much larger than log(1 + log(1 + E)). Hence, the threshold between the low-power regime and the capacity-inefficient high-power regime is strongly related to the fading number: the larger the fading number is, the higher the rate can be chosen without operating the system in the inefficient double-logarithmic regime.
Also note that even though the double-logarithmic term on the RHS of (4.1) does not depend on {Hk} or, particularly, on the number of antennas nR, it is still beneficial to have multiple antennas because the fading number χ does depend strongly on the fading process and the number of antennas.
Chapter 4 Capacity and Fading Number without Feedback
From (4.4) one also sees that in the case of a memoryless SIMO fading channel, the fading number is given by
χIID(H) = hλ! ˆHeiΘ"
− log 2 + nRE#
log %H%2$
− h(H), (4.5)
and that therefore the fading number in (4.4) also can be written as χ({Hk}) = χIID(H0) + I!
H0; H−1−∞"
− I,
Hˆ0eiΘ0;% ˆH"eiΘ!&−1
"=−∞
-. (4.6)
In [8], it has also been shown that for an arbitrary value of the power E, the channel capacity can be bounded as follows:
C(E) ≤ CIID(E) + I!
H0; H−1−∞"
, E ≥ 0. (4.7)
From (4.6) we see that this upper bound may not be tight. In particular, asymptotically for E → ∞ it is strictly loose.
Chapter 5
Capacity and Fading Number with Feedback
While it is well-known that feedback has no effect on the capacity of a memoryless channel, in general feedback does increase capacity for channels with memory. The reason for this is that the combination of feedback and memory allows the transmitter to predict the current channel state and thereby adapt to it. Unfortunately, for regular fading channels this increase in capacity due to the feedback turns out to be very limited.
Theorem 5.1 (Capacity Increase by Feedback is Bounded by a Constant) Let a general SIMO fading channel with memory be defined as in (2.1) and consider a noiseless causal feedback link as described in (2.4) (see Figure 2.1). Then the channel capacity under either an average-power constraint (2.5) or a peak-power constraint (2.6) is upper-bounded as follows:
CFB(E) ≤ CIID(E) + I!
H0; H−1−∞"
, E ≥ 0. (5.1)
Proof: When we defined channel capacity, we relied on a result by Dobrusin [14]
which shows that for information stable channels the capacity is given by C= lim
n→∞
1 n sup
Q∈P(Xn)
I(X1n; Y1n), (5.2)
where P(Xn) is the set of all probability measures Q over Xn satisfying the given input constraints.
Here, however, we have feedback and can therefore not rely on the above result, but have to derive a new converse to the coding theorem for the new situation.
Note that since the channel capacity under a peak-power constraint E cannot be larger than the capacity under an average-power constraint E, all upper bounds that are based on an average-power constraint are also valid for the situation with a peak-power constraint.
We will therefore in the following only consider an average-power constraint.
Hence, assume that there is a sequence of code schemes with ,enRFB- codewords of blocklength n—i.e., for each n the rate of the code is not larger that RFB—such that the
Chapter 5 Capacity and Fading Number with Feedback
probability of error
Pe(n)! Pr[ ˆM .= M ] (5.3)
tends to zero as n tends to infinity. Then
H(M ) = log,enRFB- ≥ log(enRFB− 1) = nRFB− )n, (5.4) where )n→ 0 as n → ∞. Therefore,
RFB ≤ 1
nH(M ) + )n
n (5.5)
= 1
nI(M ; Yn1) + 1
nH(M |Yn1) +)n
n (5.6)
= 1
nI(M ; Yn1) + log 2 + Pe(n)log,enRFB
-n + )n
n (5.7)
= 1
nI(M ; Yn1) + log 2
n +Pe(n)nRFB n +)n
n (5.8)
= 1
nI(M ; Yn1) + log 2
n + Pe(n)RFB+)n
n (5.9)
Here (5.6) follows from that definition of mutual information, and the subsequent inequal-ity from Fano’s inequalinequal-ity.
Therefore, for n → ∞ we must have RFB ≤ lim
n→∞
1
nI(M ; Yn1). (5.10)
Hence, any upper bound on the RHS of (5.10) will yield an upper bound on channel capacity in presence of feedback. We will therefore continue with bounding I(M ; Yn1):
1
nI(M ; Yn1) = 1 n
'n k=1
I(M ; Yk|Yk−11 ) (5.11)
= 1 n
'n k=1
,
I(M, Y1k−1; Yk) − I(Y1k−1; Yk)
-(5.12)
≤ 1 n
'n k=1
I(M, Yk−11 ; Yk) (5.13)
≤ 1 n
'n k=1
I(M, Yk−11 ,Hk−11 ; Yk) (5.14)
= 1 n
'n k=1
I(M, Yk−11 ,Hk−11 , Xk; Yk) (5.15)
= 1 n
'n k=1
,
I(Hk−11 , Xk; Yk) + I(M, Yk−11 ; Yk|Hk−11 , Xk)
-(5.16)
= 1 n
'n k=1
I(Hk−11 , Xk; Yk) (5.17)
= 1 n
'n k=1
,
I(Xk; Yk) + I(Hk−11 ,Yk|Xk)
-. (5.18)
Chapter 5
Here the first two equalities follow from the chain rule; the subsequent inequality from the non-negativity of mutual information; the following inequality from adding more terms;
the subsequent equality follows since Xk is a deterministic function of M and Y1k−1 (and hypothetically also Hk−11 ); then we have used the chain rule again; (5.17) follows since3
I(M, Yk−11 ; Yk|Hk−11 , Xk) = 0 (5.19) and finally we have used the chain rule once more.
We have to take into account that Xk depends on past outputs via the feedback in the next step.
I(Hk−11 ; Yk|Xk) ≤ I(Hk−11 ; Yk,Hk|Xk) (5.20)
= I(Hk−11 ; Hk|Xk) + I(Hk−11 ; Yk|Xk,Hk) (5.21)
= I(Hk−11 ; Hk|Xk) (5.22)
= h(Hk|Xk) − h(Hk|Hk−11 , Xk) (5.23)
≤ h(Hk) − h(Hk|Hk−11 , Xk) (5.24)
= h(Hk) − h(Hk|Hk−11 ) (5.25)
= I(Hk; Hk−11 ) (5.26)
Here the first inequality follows from adding one more term; the subsequent equality follows from the chain rule; (5.22) follows since
I(Hk−11 ; Yk|Xk,Hk) = 0 (5.27) which can be seen similarly to (5.19); (5.24) is due to conditioning that reduces entropy;
and the subsequent equality holds since conditional on Hk−11 , Xkand Hkare independent.
Together with (5.18) this yields 1
nI(M ; Y1n) ≤ 1 n
'n k=1
,I(Xk; Yk) + I(Hk; Hk−11 )
-(5.28)
≤ 1 n
'n k=1
,I(Xk; Yk) + I(Hk; Hk−1−∞)
-(5.29)
≤ 1 n
'n k=1
CIID(Ek) + I(H0; H−1−∞) (5.30)
where in the last inequality we have used the stationarity of {Hk} and used CIID(Ek) to denote the capacity without feedback or memory for a given power Ek. Note that the
3To see this keep in mind that Ykis fully determined by Zk, Hk, and Xk. The noise Zkis independent of everything else and can therefore not be estimated from any other random variable; Xk is given; only Hkis not known. However, it can be approximated using the past Hk1−1which again are given. Therefore, conditional on Hk1−1 and Xk, M and Yk1−1 are independent of Yk. This statement can also be proven graphically using a technique based on causal interpretations, see Section 3.3.
Chapter 5 Capacity and Fading Number with Feedback
power allocation depends on the feedback. However, due to (2.5) Ek must satisfy 1
n 'n k=1
Ek ≤ E. (5.31)
Using this together with Jensen’s inequality relying on the concavity of channel capacity in the power, we get
1
nI(M ; Y1n) ≤ CIID .1
n 'n k=1
Ek
/
+ I(H0; H−1−∞) (5.32)
≤ CIID(E) + I(H0; H−1−∞) (5.33) where in the second inequality we used the fact that CIID(·) is nondecreasing.
Therefore,
RFB(E) ≤ lim
n→∞
1
nI(M ; Yn1) ≤ CIID(E) + I(H0; H−1−∞) (5.34) which proves (5.1).
We note that the RHS of (5.1) is identical to the RHS of (4.7). Hence the same (alas potentially loose) bound holds both for the channel capacity with and without feedback.
Moreover, also note that C(E) trivially is a lower bound to CFB(E) since the transmitter can simply ignore the feedback and achieve the same results as without feedback.
An immediate consequence of Theorem 5.1 is that CFB(E) only grows double-logarith-mically in the power at high power and therefore there exists a fading number χFB({Hk}) with a definition as follows:
Corollary 5.2 Because
E↑∞lim{CFB(E) − log log E} < 0, (5.35) we define
χFB({Hk})! lim
E↑∞{CFB(E) − log log E} (5.36) (5.37) Theorem 5.1 can then be applied to χFB({Hk}).
Corollary 5.3 Using the same result as in Theorem 5.1, we learn χFB({Hk}) ≤ χIID({Hk}) + I!
H0; H−1−∞"
. (5.38)
Next, we state a stronger statement.
Chapter 5
Theorem 5.4 (SIMO Fading Number with Feedback) Let a general SIMO fading channel with memory be defined as in (2.1) and consider a noiseless causal feedback link as described in (2.4) (see Figure 2.1). Then the asymptotic channel capacity under either an average-power constraint (2.5) or a peak-power constraint (2.6) is identical to the asymptotic channel capacity for the channel without feedback:
CFB(E) = log(1 + log(1 + E)) + χFB({Hk}) + o(1) (5.39) where the fading number is
χFB({Hk}) = χ({Hk})
= hλ
,Hˆ0eiΘ0 ))
)% ˆH"eiΘ!&−1
"=−∞
-− log 2 + nRE#
log %H0%2$
− h! H0)
)H−1−∞"
. (5.40)
We would like to point out that this result even holds in the (hypothetical) case when the feedback is improved in the sense that in addition to the past channel outputs the transmitter also is informed about the past fading realizations Hk−11 . Note further that since we have assumed the most optimistic form of causal feedback, any type of realistic feedback will yield the same result.
We would like to give a hand-waving explanation of this behavior. Since the fading process is assumed to be regular with a finite differential entropy rate, it is not possible to perfectly predict the future realizations of the process even if one is presented with the exact realizations of the infinite past. Nevertheless, the feedback allows the transmitter to make an estimate of future realizations. Based on these estimates, the transmitter can then perform elaborate schemes of optimal power allocation over time: if the channel state is likely to be poor, it saves power and uses it once the channel state is likely to be good again. Unfortunately, due to the double-logarithmic behavior of capacity, such power allocation has no effect at all as can be seen as follows: for any constant β > 0 (β can be chosen arbitrarily large!),
E↑∞lim{log log βE − log log E} = lim
E↑∞{log(log β + log E) − log log E} (5.41)
= lim
E↑∞{log(log E) − log log E} (5.42)
= 0. (5.43)
So not only the double-logarithmic growth is left untouched, but also the second term, i.e., the fading number, remains unchanged.
Chapter 6
Proof of Theorem 5.4
6.1 Main Line Through the Proof
Since the channel capacity of the system without feedback trivially is a lower bound on the channel capacity with feedback, and since the capacity under a peak-power constraint is a lower bound on the capacity with an average-power constraint, it is sufficient to derive an upper bound on χFB({Hk}) under the assumption of the average-power constraint (2.5) and to show that it is identical to the fading number without feedback and under the assumption of a peak-power constraint.
The proof is very lengthy and we therefore outline the main ideas in the beginning. The basic structure follows the proof of the general fading number of MIMO fading channels with memory given in [9]. However, there are many details that need to be adapted and taken care of. Particularly, we have to consider the following challenges:
• Due to the feedback, the channel input, the fading, and the additive noise become dependent.
• We cannot rely on the important auxiliary result given in [9, Th. 3] that shows that the optimal input is stationary.
• We cannot rely on the important auxiliary result given in [15, Th. 8] that shows that the capacity-achieving input distribution escapes to infinity.
To handle the first challenge, we often rely on the concept of causal interpretations, which is introduced in Chapter 3.3, [12], [13]. This is a tool that allows to graphically proof the independence of random variables when conditioned on certain other random variables.
The missing auxiliary result concerning the capacity-achieving input distribution es-caping to infinity can be proven indirectly inside of the derivation.
The biggest difficulty is caused by the nonstationarity of the channel input that is inherent to the given context because the transmitter continuously learns more about the fading process through the feedback and thereby changes the optimal distribution of the input.
6.1 Main Line Through the Proof Chapter 6
The proof starts with Fano’s inequality (see (5.9)), which states that any given sequence of communication systems with rate RFB and power E must satisfy
RFB(E) ≤ 1
In (6.3), we separate the sum into two parts. The first part, 1 ≤ k ≤ κ, can be considered as transient state. Since κ is a constant, it is bounded anyway and in (6.4) we bound the mutual information term in the sum as in (5.30). Using Jensen’s inequality relying on the concavity of channel capacity in the power, we get
RFB(E) ≤ κ
Here in (6.6), we add the indicator random variable Gk that is defined as
Gk !
41 if %Hk%2 ≥ t,
0 otherwise, (6.9)
for some given t > 0. We will choose t large such that E#
Chapter 6 Proof of Theorem 5.4
and note that by the Markov inequality (Lemma 3.3), γk= Pr# where Hb(·) denoting the binary entropy function, with t large enough and by using (5.30) again with conditioning on Gk= 1, we further bound (6.8) as follows:
I! fading channel, for which we know
˜lim
where the last inequality follows by Jensen’s inequality relying on the concavity of channel capacity in the power again. Putting everything back into (6.5), we get
RFB(E) ≤ κ
6.1 Main Line Through the Proof Chapter 6
The third term in (6.18) is then bounded as follows:
I!
Here in (6.20) the current input Xkand the past fading values Hk−11 are added. In (6.22) we add the indicator random variable Ek that is defined as
Ek!
Note that the three middle terms on the RHS of (6.24) correspond to a memoryless term, a term with memory, and a correction term, respectively. We will show in Section 6.2 that the second term on the RHS of (6.24) can be bounded as follows:
I(Xk; Yk|Ek= 1, Gk= 0)
the third term on the right hand side of (6.24) as I!
Chapter 6 Proof of Theorem 5.4
and the fourth term on the right hand side of (6.24) as I! Plugging (6.27), (6.28), and (6.29) back to (6.24), we get
I!
Note that the four underlining terms in (6.30) cancel each other, and that βknRE#
6.1 Main Line Through the Proof Chapter 6
where (6.32) follows because we add something positive (ν ≥ 0, chosen freely). Therefore, (6.30) becomes
Here, in (6.35) we arithmetically rearrange the terms. We further bound the last term in (6.35) as follows:
Chapter 6 Proof of Theorem 5.4
where (6.36) follows from Jensen’s inequality; (6.37) follows because E#
(6.38) follows because {Hk} is a stationary process; and (6.39) follows because the function x 0→ x log x has its minimum when x = e. Putting (6.40) back into (6.35) and using the stationarity property of {Hk} again, we get
I!
6.1 Main Line Through the Proof Chapter 6
Moreover in Appendix A, we further bound the last part of (6.43) as follows:
I!
and ∆(nR,1) is some finite number. Therefore, we get I!
Chapter 6 Proof of Theorem 5.4 where (6.49) follows because two underlining terms in (6.48) combine to
h!
using Jensen’s inequality for the binary entropy function, adding the sum from (6.18) in front of (6.49), we get
1 the last term in (6.51) can be further bounded as
1
Moreover, in order to get rid of the dependence on the input, {Xk} (note that β depends
6.1 Main Line Through the Proof Chapter 6
-Chapter 6 Proof of Theorem 5.4
Then the bound on the capacity with feedback in (6.18) becomes RFB(E) ≤ hλ! ˆH0eiΘ0)
6.1 Main Line Through the Proof Chapter 6
Note that this bound holds for any system, hence also for a capacity-achieving system.
Therefore we can use (6.61) to upper-bound CFB(E):
χFB({Hk}) = lim
Chapter 6 Proof of Theorem 5.4
Here, in (6.64), we try to find the value of β that achieves the supremum: note that we found that first, Hb(β) and those terms with 1 − β are constant with respect to E; second, the remaining terms do not grow with E except log Γ! ν
log E,E log Eν2 " which means log Γ(·) grows as fast as log log E. So log Γ(·) is the only term inside the sup that grows with E. Therefore, the supremum is achieved if β = 1. Actually, this is related to the property called “escaping to infinity” (see [10, Corollary 2.8]).
Next, note that is implicitly satisfied. As the result, (6.64) becomes
χFB({Hk}) = hλ! ˆH0eiΘ0) definition of )ν in Appendix B. Note further that
ν→0lim
6.2 Detailed Derivations for Three Terms in (6.24) Chapter 6 δ2(κ, ξmin) → 0. Finally, we let κ tend to infinity and the fading number without feedback becomes
6.2 Detailed Derivations for Three Terms in (6.24)
6.2.1 First Term
The second term on the RHS of (6.24) is bounded as follows:
I(Xk; Yk
We continue to bound (6.80) using a duality-based bound, for detail we refer to
Ap-Chapter 6 Proof of Theorem 5.4 Arithmetically rearranging the terms in (6.81), we have the second term on the RHS of (6.24) be bounded as follows:
I(Xk; Yk
The third term on the RHS of (6.24) is bounded as follows:
I!
where the last step follows because conditional on Gk= 0 and all the past values Hk−11 of {Hk}, Hk is independent of Xk and Ek.
6.2.3 Third Term
Recalling the definition of Ek in (6.25), we lower-bound the fourth term on the RHS of (6.24) as follows:
6.2 Detailed Derivations for Three Terms in (6.24) Chapter 6 Here, (6.90) follows because the first term and last term in (6.89) are equal or greater than zero and Hb(Ek|Yk−1k−κ, Gk= 0) ≤ Hb(Ek) = Hb(βk); in (6.92), we add {Θk}, which is IID
∼ U ((−π, π]) and independent of Yk. Because {Θk} is uniformly distributed, it destroys the phase of {Hk} and let {HkeiΘk} becomes circularly symmetric. (6.94) follows because we drop eΘk on both side of mutual information.
By Appendix C.1, we have βkI,
so we further bound (6.96) as follows:
I,
-Chapter 6 Proof of Theorem 5.4
Here, in (6.100), we drop {Zk−1k−κ}, so the mutual information becomes smaller.
By Appendix C.2, we have βkI,%
and we further bound (6.101) as follows:
I,
6.2 Detailed Derivations for Three Terms in (6.24) Chapter 6
Here, (6.105) follows from taking the magnitude from Hk|Xk|eiΘk; (6.106) follows be-cause we drop some terms in mutual information; (6.107) follows from the definition of differential entropy for unit vectors (see Section 3.1.2); (6.110) follows because dropping conditioning increases entropy.
Chapter 7
Discussion and Conclusion
In this thesis, we have shown that the asymptotic capacity of general regular SIMO fading channels with memory remains unchanged even if one allows causal noiseless feedback.
This once again shows the extremely unattractive behavior of regular fading channels at high SNR: besides the double-logarithmic growth [8] and the very poor performance in a multiple-user setup (where the maximum sum-rate only can be achieved if all users apart from one always remain switched off [16]), we now see that any type of feedback does not increase capacity in spite of memory in the channel.
Possible future works for the general regular fading channels with memory and feedback might include the following:
• Considering the case with multiple-input single-output, i.e., having several mobile phones (each having one antenna) communicating with one base station (having only one antenna). The difficulties for this case lies in the fact that now we not only need to optimize the phase and magnitude of the inputs, but also the direction of them.
• Considering the case with multiple-input multiple-output.
• The situation where both transmitter and receiver have access to causal partial side-information Sk about the fading, where by partial we mean that
• The situation where both transmitter and receiver have access to causal partial side-information Sk about the fading, where by partial we mean that