It has been shown that optimal codes for M = 3 or linear optimal codes for M = 4 are weak flip codes with code parameters:
[t∗1, t∗2, t∗3] =
[k + 1, k, k − 1] if n mod 3 = 0, [k + 1, k, k] if n mod 3 = 1, [k + 1, k + 1, k] if n mod 3 = 2,
(11.14)
where we use
k !I n 3 J
. (11.15)
The corresponding pairwise Hamming distance vectors (see Lemma6.7) are
(2k − 1, 2k, 2k + 1) if n mod 3 = 0, (2k, 2k + 1, 2k + 1) if n mod 3 = 1, (2k + 1, 2k + 1, 2k + 2) if n mod 3 = 2.
(11.16)
If we compare this to Theorem11.2:
[t∗1, t∗2, t∗3] =
[k, k, k] if n mod 3 = 0, [k + 1, k, k] if n mod 3 = 1, [k + 1, k + 1, k] if n mod 3 = 2
(11.17)
with corresponding pairwise Hamming distance vectors
(2k, 2k, 2k) if n mod 3 = 0, (2k, 2k + 1, 2k + 1) if n mod 3 = 1, (2k + 1, 2k + 1, 2k + 2) if n mod 3 = 2,
(11.18)
we can conclude the following.
Corollary 11.7 Apart from n mod 3 = 0, the optimal codes for a BSC are identical to the optimal codes for a BEC for M = 3 or linear optimal codes for M = 4 codewords.
It is interesting to note that for n mod 3 = 0 the optimal codes for the BEC are fair and therefore maximize the minimum Hamming distance, while this is not the case for the (very symmetric!) BSC. However, note that the converse is not true: if a code maximizes the minimum Hamming distance, then it is not necessarily an optimal code for the BEC!
So, in particular, it is not clear if binary nonlinear Hadamard codes are optimal.
11.4 Application to Known Bounds on the Error Probabil-ity for a Finite Blocklength
We again provide a comparison between the performance of the optimal code to the known bounds of Chapter7.
Note that the error exponents for M = 3, 4 codewords are E3= E4= −2
3log δ. (11.19)
Moreover, for M = 3, 4, D(BEC)
min
, C(M,n)
*n+13 +,*n3+
-=
−23log δ if n mod 3 = 0
−,n3-+,n+13
-n log δ if n mod 3 = 1
−,n3-+,n+13
-n log δ if n mod 3 = 2.
(11.20)
Figs. 11.15 and 11.16 compare the exact optimal performance for M = 3 and M = 4, respectively, with some bounds: the SGB upper bound based on the weak flip code used by Shannon et al.,10, the SGB lower bound based on the weak flip code (which is suboptimal, but achieves the optimal D(DMC)min and is therefore a generally valid lower bound), the Gallager upper bound, and also the PPV upper and lower bounds.
We can see that the SGB upper bound is tighter to the exact optimal performance than the PPV upper bound. Note, however, the PPV upper bound does not exhibit the correct error exponent. It is shown in [20] that, for n going to infinity, the random coding (PPV) upper bound tends to the Gallager exponent for R = 0 [6], which is of course not necessarily equal to EM for finite M.
Concerning the lower bounds, we see that the PPV lower bound (converse) is much better for finite n than the SGB bound. However, for n large enough, its exponential growth will approach that of the sphere-packing bound [17], which does not equal to EM
either.
Once more we would like to point out that even though the fair weak flip codes achieve the error exponent, they are optimal codes in the BEC, however, they are strictly subop-timal for every n mod 3 = 0 in the BSC.
10The SGB upper bound based on the optimal code performs almost identically (because the BSC is pairwise reversible) and is therefore omitted.
0 5 10 15 20 25 30 35 10−18
10−16 10−14 10−12 10−10 10−8 10−6 10−4 10−2 100
ErrorProbability
Blocklength n
Gallager upper bound
SGB up. b. for t2=$n+13 %, t3=$n3% PPV upper bound
Optimal (exact, t2=t∗2, t3=t∗3)
PPV lower bound
SGB l. b. for t2=$n+13 %, t3=$n3%
Figure 11.15: Exact value of, and bounds on, the performance of an optimal code with M= 3 codewords on the BEC with δ = 0.3 as a function of the blocklength n.
0 5 10 15 20 25 30 35 10−18
10−16 10−14 10−12 10−10 10−8 10−6 10−4 10−2 100
ErrorProbability
Blocklength n
Gallager upper bound
SGB up. b. for t2=$n+13 %, t3=$n3% PPV upper bound
Optimal (exact, t2=t∗2, t3=t∗3)
PPV lower bound
SGB l. b. for t2=$n+13 %, t3=$n3%
Figure 11.16: Exact value of, and bounds on, the performance of an optimal code with M= 4 codewords on the BEC with δ = 0.3 as a function of the blocklength n.
Conclusion
For an arbitrary finite blocklength n, we have studied the optimal code design of ultra-small block-codes for the most general binary input and the binary output discrete memoryless channel, the so-called binary asymmetric channel (BAC), and the binary input and ternary output symmetric discrete memoryless channel: the binary erasure channel (BEC).
We then have put special emphasis on the two most important special cases of binary channels, the Z-channel (ZC) and the binary symmetric channel (BSC). There, again for an arbitrary finite blocklength n, we have derived an optimal code design with four or less messages. In the case of the ZC, we have also conjectured an optimal code design with five messages. Note that since the optimal codes we proposed do not depend on the crossover probability of the channel the optimal codes remain the same even if the channel is nonergodic or nonstationary. Also note that the optimal weak flip codes are by definition coset codes: the M = 3 nonlinear code is always a coset of the M = 4 linear code. However, they are not fixed composition codes.
We have introduced a new way of generating these codes recursively by using a column-wise build-up of the codebook matrix. This column view of the codebook turns out to be far more powerful for analysis than the standard row-wise view (i.e., the analysis based on the codewords). We believe that the recursive construction of codes may be extended to a higher number of codewords and also to more complex channel models. Indeed, we have achieved some first promising results for the binary erasure channel (BEC). Note, however, that in these more complex situations we might need a recursion from n to n + γ with a step-size γ > 1.
We have also investigated the well-known and commonly used code parameter mini-mum Hamming distance. We show that it may not be suitable as a design criterion for optimal codes, even for very symmetric channels like the BSC.
Finally, we would like to point out that the family of weak flip codes defined in Chap-ter 6 (and in particular a subfamily called fair weak flip codes) turn out to have many interesting properties. They can be seen as a subset of a well-known codes: the Hadamard codes. A first closer investigation of some of these properties and these codes’ relation to linear codes have also been presented.
Derivations concerning the BAC
A.1 Proof of Proposition 4.1
Let PX(0) = p, then PX(1) = 1 − p, we have
PY(0) = PX(0)PY|X(0|0) + PX(1)PY|X(0|1) = p(1 − %0) + (1 − p)%1, (A.1) PY(1) = PX(0)PY|X(1|0) + PX(1)PY|X(1|1) = p%0+ (1 − p)(1 − %1). (A.2) Then the mutual information I(X; Y ) of BAC
I(X; Y ) = H(Y ) − H(Y |X) (A.3)
= −PY(0) log PY(0) − PY(1) log PY(1) − $
x=0,1
PX(x)H(Y |X = x) (A.4)
= −Q
p(1 − %0) + (1 − p)%1R logQ
p(1 − %0) + (1 − p)%1R
−Q
p%0+ (1 − p)(1 − %1)R logQ
p%0+ (1 − p)(1 − %1)R
−pHb(%0) log 2 − (1 − p)Hb(%1) log 2 (nats). (A.5) Therefore, take the derivative of the mutual information, we get
−Q
1 − %0− %1R logQ
p(1 − %0) + (1 − p)%1R
− 1 ·"
1 − %0− %1#
−Q
%0− (1 − %1)R logQ
p%0+ (1 − p)(1 − %1)R
− 1 ·"
%0− (1 − %1)#
−Hb(%0) log 2 + Hb(%1) log 2= 0! (A.6)
=⇒ (1 − %0− %1) .
log2 p(1 − %0) + (1 − p)%1
p%0+ (1 − p)(1 − %1) /
= Hb(%1) − Hb(%0) (A.7)
=⇒ p(1 − %0) + (1 − p)%1
p%0+ (1 − p)(1 − %1) = 2Hb(!1)−Hb(!0)
1−!0−!1 (A.8)
=⇒ (1 − %0) + (1−pp )%1
%0+ (1−pp )(1 − %1) = z =⇒ (1 − p
p )(%1− z(1 − %1)) = z%0− (1 − %0) (A.9)
=⇒ 1
p− 1 = 1 − %0− z%0 z(1 − %1) − %1
(A.10)
=⇒ 1
p = z(1 − %1) − %1+ 1 − %0− z%0 z(1 − %1) − %1
(A.11)
=⇒ p = z − %1(1 + z)
(1 + z)(1 − %1− %0). (A.12)
Hecne, the capacity-input achieving distributions are PX∗(0) = z − %1(1 + z)
(1 + z)(1 − %0− %1), PX∗(1) = 1 − %0(1 + z)
(1 + z)(1 − %0− %1) (A.13) Similarly, the capacity-output achieving distributions are
PY∗(0) = (1 + z)(1 − %0− %1) − (1 − %0− %1) (1 + z)(1 − %0− %1) = z
1 + z PY∗(1) = 1
1 + z. (A.14)
Next we substitute this p into I(X; Y ) (A.5), we have CBAC= − z
1 + zlog2( z
1 + z) − 1
1 + zlog2 1
1 + z − PX∗(0)Hb(%0) − PX∗(1)Hb(%1)(A.15)
= − z
1 + zlog22Hb(!1)−Hb(!0)
1−!0−!1 + log2(1 + z)
− z − %1(1 + z)
(1 + z)(1 − %0− %1)Hb(%0) − 1 − %0(1 + z)
(1 + z)(1 − %0− %1)Hb(%1) (A.16)
=
V z
(1 + z)(1 − %0− %1) − z − %1(1 + z) (1 + z)(1 − %0− %1)
W Hb(%0) +
V
− z
(1 + z)(1 − %0− %1) − 1 − %0(1 + z) (1 + z)(1 − %0− %1)
W
Hb(%1) + log2(1 + z)(A.17)
= %1
1 − %0− %1Hb(%0) − 1 − %0
1 − %0− %1Hb(%1) + log2(1 + z). (A.18)