As already mentioned in Section4.5, it is quite common in conventional coding theory to use the minimum Hamming distance or the weight enumerating function (WEF) of a code as a design and quality criterion [12]. This is motivated by the equivalence of Hamming weight and Hamming distance for linear codes, and by the union bound that converts the search for the global error probability into pairwise error probabilities. Since we are interested in the globally optimal code design and the best performance achieved by an ML decoder, we can neither use the union bound, nor can we a priori restrict our search to linear codes. Note that for most values of M, linear codes do not even exist!7
We would like to come back to the example shown in Chapter5and further deepen our analysis of the minimum Hamming distance of our optimal codes on the very symmetric BSC. Although, as (4.16) shows, the error probability performance of a BSC is completely specified by the Hamming distance between codewords and received vectors, we will now demonstrate that a design based on the minimum Hamming distance can fail, even for the very symmetric BSC and even for linear codes. In the case of a more general (and not symmetric) BAC, this will be even more pronounced.
We compare the optimal codes given in Theorem10.2with the following different weak
7Interestingly, a subfamily of the weak flip codes can be shown to have many linear-like properties. For more details see [14].
flip code Csubopt(M,n) with code parameters
This code can be constructed from the optimal code CBSC(M,n−1)∗ by appending a suboptimal column8 and—based on a closer inspection of the proof of Theorem 10.2—can be shown to be strictly suboptimal.
Recalling Lemma6.7, we compute the pairwise Hamming distance vector of the optimal code for M = 3:
For M = 4 we get accordingly:
d"
with the same values for the minimum Hamming distance as for the M = 3.
Comparing this with the suboptimal code (10.11) now yields for M = 3:
d"
8The choice of column depends on n.
i.e., dmin"
Hence, we see that for n mod 3 = 0 the minimum Hamming distance of the optimal code is 2k − 1 and therefore strictly smaller than the corresponding minimum Hamming distance 2k of the suboptimal code.
By adapting the construction of the strictly suboptimal code Csubopt(M,n), a similar state-ment can be made for the case when n mod 3 = 1.
We have shown the following proposition.
Proposition 10.4 On a BSC for M = 3 or M = 4 and for all n with n mod 3 = 0 or n mod 3 = 1, codes that maximize the minimum Hamming distance dmin"
C(M,n)#
can be strictly suboptimal. This is not true in the case of n mod 3 = 2.
As a matter of fact, using a result from [14], one can show that on a BSC for M = 3 or M= 4 and in the case of n mod 3 = 0, all codes that maximize the minimum Hamming distance are strictly suboptimal.
10.4 Application to Known Bounds on the Error Probabil-ity for a Finite Blocklength
We again provide a comparison between the performance of the optimal code to the known bounds of Chapter7.
Note that the error exponents for M = 3, 4 codewords are E3= E4= −2
Figs. 11.15 and 11.16 compare the exact optimal performance for M = 3 and M = 4, respectively, with some bounds: the SGB upper bound based on the weak flip code used by
Shannon et al.,9, the SGB lower bound based on the weak flip code (which is suboptimal, but achieves the optimal D(DMC)min and is therefore a generally valid lower bound), the Gallager upper bound, and also the PPV upper and lower bounds.
We can see that the PPV upper bound is tighter to the exact optimal performance than the SGB upper bound. Note, however, that neither exhibits the correct error exponent.
It is shown in [20] that, for n going to infinity, the random coding (PPV) upper bound tends to the Gallager exponent for R = 0 [6], which is of course not necessarily equal to EM for finite M.
Concerning the lower bounds, we see that the PPV lower bound (meta-converse) is much better for finite n than the SGB bound. However, for n large enough, its exponential growth will approach that of the sphere-packing bound [17], which does not equal to EM
either.
Once more we would like to point out that even though the fair weak flip codes achieve the error exponent, they are strictly suboptimal for every n mod 3 = 0.
9The SGB upper bound based on the optimal code performs almost identically (because the BSC is pairwise reversible) and is therefore omitted.
0 10 20 30 40 50 60 70 80 90 100 11−12
11−10 11−8 11−6 11−4 11−2 11−0 112
ErrorProbability
Blocklength n
Gallager upper bound
SGB up. b. for t2=$n+13 %, t3=$n3% PPV upper bound
Optimal (exact, t2=t∗2, t3=t∗3) PPV lower bound
SGB l. b. for t2=$n+13 %, t3=$n3%
Figure 10.13: Exact value of, and bounds on, the performance of an optimal code with M= 3 codewords on the BSC with % = 0.3 as a function of the blocklength n.
0 10 20 30 40 50 60 70 80 90 100 11−12
11−10 11−8 11−6 11−4 11−2 11−0 112
ErrorProbability
Blocklength n
Gallager upper bound
SGB up. b. for t2=$n+13 %, t3=$n3% PPV upper bound
Optimal (exact, t2=t∗2, t3=t∗3) PPV lower bound
SGB l. b. for t2=$n+13 %, t3=$n3%
Figure 10.14: Exact value of, and bounds on, the performance of an optimal code with M= 4 codewords on the BSC with % = 0.3 as a function of the blocklength n.
Analysis of the BEC
The definition of the flip, the weak flip, and the fair weak flip codes is interesting not only due to their generalization of the concept of linear codes, but also because we can show that they are optimal for the BEC for many values of the blocklength n.
11.1 Optimal Codes with Two Codewords (M = 2)
Theorem 11.1 For a BEC and for any n ≥ 1, an optimal codebook with M = 2 codewords is the flip code of type t for any t ∈0
0, 1, . . . ,2n
2
31.
Proof: Omitted. Similar argument as Theorem10.1.
11.2 Optimal Codes with Three or Four Codewords (M = 3, 4)
Theorem 11.2 For a BEC with arbitrary crossover probability 0 ≤ δ < 1, the optimal code for n = 2 is
C(M,2)∗
BEC =,
c(M)1 c(M)2
-. (11.1)
If we recursively construct a locally optimal codebook for n ≥ 3 by using CBEC(M,n−1)∗ and appending a new column, the total probability increase is maximized by the following choice of appended columns:
c(M)3 if n mod 3 = 0 c(M)1 if n mod 3 = 1 c(M)2 if n mod 3 = 2.
(11.2)
Proof: See Appendix D.1.
While Theorem 11.2 only guarantees optimality under the condition that the opti-mal code are happened to be the code maximizing the total probability increase that constructed recursively.
Theorem 11.3 For a BEC and for any n ≥ 2, an optimal codebook with M = 3 or a linear optimal codebook with M = 4 codewords is the weak flip code of type (t∗2, t∗3), where
t∗2 !G n + 1 3
H
, t∗3!I n 3 J
. (11.3)
Using the shorthands
k !I n 3
J (11.4)
the code parameters of these optimal codes can be written as
[t∗1, t∗2, t∗3] =
[k, k, k] if n mod 3 = 0 [k + 1, k, k] if n mod 3 = 1 [k + 1, k + 1, k] if n mod 3 = 2.
(11.5)
Furthermore, the exact average success probability can be derived recursively in blocklength n, starting with
3Pc(CBEC(3,2)∗) = 3(1 − δ)2+ 4δ(1 − δ) + δ2. (11.6) Then
3Pc(Ct(3,n)∗
2,t∗3 ) = 3Pc(Ct(3,n−1)∗ 2,t∗3 ) +,
δ2k−1+ δ2k−1− δn−1
-;(if n = 3k) (11.7) 3Pc(Ct(3,n)∗
2,t∗3 ) = 3Pc(Ct(3,n−1)∗ 2,t∗3 ) +,
δ2k+ δ2k− δn−1
-; (if n = 3k + 1) (11.8) 3Pc(Ct(3,n)∗
2,t∗3 ) = 3Pc(Ct(3,n−1)∗ 2,t∗3 ) +,
δ2k+ δ2k+1− δn−1
-. (if n = 3k + 2) (11.9) While for M = 4 the exact average success probability is very similar to the case of M = 3.
Proof: Similar to the proof of Theorem 10.3, combing with the proof of Theo-rem 11.2.
Note that the idea of designing an optimal code recursively promises to be a very powerful approach. Unfortunately, for larger values of M, we might need a recursion from n to n + γ with a step-size γ > 1, and this step-size γ might be a function of blocklength n. However, based on our definition of fair weak flip codes and on Conjecture11.5below, we conjecture that the necessary step-size satisfies γ ≤"2#−1
#
#.
We have conjectured that this recursive approach also to the cases of M = 5 and M= 6.
Conjecture 11.4 For a BEC and for any n ≥ 3, if the optimal codebook can be recursively constructed in blocklength n, an optimal codebook with M = 5 codewords can be constructed recursively in the blocklength n. We start with an optimal codebook for n = 3:
C(M,3)∗
BEC =,
c(M)1 , c(M)2 , c(M)5
-(11.10)
and recursively construct the optimal codebook for n ≥ 5 by using CBEC(M,n−γ)∗, γ ∈ {1, 2, 3},
For M = 6 codewords, an optimal codebook can be constructed recursively in the blocklength n by starting with an optimal codebook for n = 4:
C(M,3)∗
BEC =,
c(M)1 , c(M)2 , c(M)6 , c(M)8
-. (11.12)
Then we recursively construct the optimal codebook for n ≥ 6 by using CBEC(M,n−2)∗ and appending
For space reasons we omit the proof and only remark once again that the ideas of the derivation follow the same ideas as shown above in Lemma 9.3and ClaimD.1.
An interesting special case of Conjecture 11.4is as follows.
Conjecture 11.5 For a BEC and for any n being a multiple of 10, an optimal codebook with M = 5 or M = 6 codewords is the corresponding fair weak flip code.
Note that the restriction on n comes from the restriction that fair weak flip codes are only defined for n with n mod"2#−1
#
# = n mod 10 = 0. Even though Conjecture 11.5 actually follows as special case from Conjecture11.4, it can be proven directly and more elegantly using the properties of fair weak flip codes derived in Section6.1.
How about the optimal codes on BEC for higher number of codewords M? We strongly believe that Conjecture 11.5can be generalized to arbitrary M.
Conjecture 11.6 For a BEC and for an arbitrary M, the optimal code for a blocklength n that satisfies n mod"2#−1
#
#= 0 is the corresponding fair weak flip code.