針對非均勻訊源之固定長度整合訊源與通道編碼系統之設計

(1)

國

立

交

通

大

學

電信工程研究所

碩

士

論

文

針對非均勻訊源之固定長度整合訊源與通道編碼系統之設計

Fixed-Length Joint Source-Channel Coding System for Generally Non-uniform

Sources

研究生：林鉑涵

指導教授：陳伯寧教授

(2)

針對非均勻訊源之固定長度整合訊源與通道編碼系統之設計

Fixed-Length Joint Source-Channel Coding System for Generally Non-uniform

Sources

研究生：林鉑涵

Student：Po-Han Lin

指導教授：陳伯寧

Advisor：Po-Ning Chen

國立交通大學

電信工程研究所

碩士論文

A Thesis

Submitted to Institute of Communication Engineering College of Electrical and Computer Engineering

National Chiao Tung University in partial Fulfillment of the Requirements

for the Degree of Master

in

Communication Engineering

July 2013

(3)

tt

針

針對

對

對非

非

非均

均

均勻

勻

勻訊

訊

訊源

源

源之

之

之固

固

定

定長

長

長度

度

度整

整

整合

合

合訊

訊

訊源

源

源與

與

與通

通

通道

道

道編

編

編碼

碼

碼系

系

系統

統

統之

之

之設

設

設計

計

學生: 林鉑涵kkk 指_{導教授: 陳伯寧} 國立交通大學電信工程研究所碩士班摘摘摘要要要固_{定長度整合訊源與通道編碼(FLEC)為一個將訊源編碼與通道編碼結合的系統。在} 這_{篇碩士論文中，我們提出了兩個可用於非均勻訊源(non-uniform source)的FLEC的設計} 方法。第一個方法基於所推導的FLEC的聯集錯誤率上界(union bound)，設計使此上界相_{對較小的FLEC。由於第一個方法僅能適用於碼長較短的FLEC設計，我們因此再提出} 第二個方法。第二個方法基於渦輪碼(turbo code)的架構，重新設計可適用於非均勻訊源的渦輪碼的_{解碼量度。模擬結果顯示我們所提出的第一個方法，系統錯誤率會比傳統結} 合_{赫夫曼(Huffman)來源編碼與BCH碼的分離式設計低。而第二個方法的效能比結合赫夫} 曼(Huffman)來源編碼與渦輪碼的分離式設計好。

(4)

Fixed-Length Joint Source-Channel Coding System for

Generally Non-uniform Sources

Student: Po-Han Lin kkk Advisor: Po-Ning Chen Institute of Communications Engineering

National Chiao Tung University

Abstract

In this thesis, the design of fixed-length joint source-channel error-correcting codes (FLEC) for generally non-uniform source statistics is considered as contrary to the usual variable-length joint source-channel error-correcting coding system. Such a system has the advantage that the receiver can identify easily the codeword margin via a length counter. Two different approaches are attempted. We first derive the union bounds of decoding errors of the FLECs for generally non-uniform sources, and then find the FLECs that have acceptably good union bound values. Since the first approach is only suitable for FLECs of short block length, the second approach assumes the turbo code structure and modifies the turbo decoding metrics to adapt to the tranceiving of non-uniform information. Simulations show that the first pro-posed approach outperforms the traditional tandem scheme that concatenates the Huffman source code with a BCH code, while the second proposed approach beats the concatenation of the Huffman source code with a turbo code of similar rate.

(5)

Acknowledgements

First of all, I would like to express my gratitude to my advisor, Professor Po-Ning Chen, for his patient guidance and support. Secondly, I would like to show my special gratitude to Dr. Ting-Yi Wu for his enthusiastic teaching. This thesis would not have been possible without these helps. Finally, I would like to thank all the members in the NTL lab and all those who have helped me in these two years.

(6)

List of Figures

1.1 Block diagram of a typical digital communication system with separate source and channel coding. . . 2 2.1 The (37,21) recursive systematic encoder . . . 7

4.1 Diagram of a sample turbo decoder. . . 22 5.1 Performance comparison between the joint source-channel block code

con-structed by exhaustive search and that by our proposed algorithm with code-word length ℓ = 8. The source follows Case 1 and hard-decision decoding is employed. . . 36 5.2 Performance comparison between the joint source-channel block code

con-structed by exhaustive search and that by our proposed algorithm with the codeword length ℓ = 8. The source follows Case 2 and hard-decision decoding is employed. . . 37 5.3 Performance comparison between the joint source-channel block code

con-structed by exhaustive search and that by our proposed algorithm with the codeword length ℓ = 8. The source follows Case 3 and hard-decision decoding is employed. . . 38

(9)

5.4 Performance comparison of the proposed joint source-channel block codes for different codeword length. The source follows Case 1 and soft-decision decod-ing is employed. . . 39 5.5 Performance comparison of the proposed joint source-channel block codes for

different codeword length. The source follows Case 2 and soft-decision decod-ing is employed. . . 40 5.6 Performance comparison of the proposed joint source-channel block codes for

different codeword length. The source follows Case 3 and soft-decision decod-ing is employed. . . 41 5.7 Performance comparison between the joint source-channel block code with

codeword length 64 and the tandem scheme (i..e, the Huffman code + (63, 16, 11) BCH code). The source follows Case 1 and hard-decision decoding is em-ployed. . . 42 5.8 Performance comparison between the joint source-channel block code with

codeword length 64 and the tandem scheme (i..e, the Huffman code + (63, 16, 11) BCH code). The source follows Case 2 and hard-decision decoding is em-ployed. . . 43 5.9 Performance comparison between the joint source-channel block code with

codeword length 64 and the tandem scheme (i..e, the Huffman code + (63, 16, 11) BCH code). The source follows Case 3 and hard-decision decoding is em-ployed. . . 44

(10)

5.10 Performance comparison between the joint source-channel turbo code and the tandem scheme (i.e., the Huffman code + a traditional turbo code) under the source with per-letter source entropy 3.9711 bits. The interleaver size is ℓ = 128 × 128. . . 47 5.11 Performance comparison between the joint source-channel turbo code and the

tandem scheme (i.e., the Huffman code + a traditional turbo code) under the source with per-letter source entropy 3.92582 bits. The interleaver size is ℓ = 128 × 128. . . 48 5.12 Performance comparison between the joint source-channel turbo code and the

tandem scheme (i.e., the Huffman code + a traditional turbo code) under the source with per-letter source entropy 3.52516 bits. The interleaver size is ℓ = 128 × 128. . . 52

(11)

5.16 Performance comparison between the joint source-channel turbo code and the tandem scheme (i.e., the Huffman code + a traditional turbo code) under the source with per-letter source entropy 3.24512 bits. The interleaver size is ℓ = 128 × 128. . . 53 5.17 Performance comparison under two different source distributions with the

same per-letter source entropy 3.20 bits. The interleaver size is ℓ = 128 × 128. 54 5.18 Performance comparison between the joint source-channel turbo code and the

tandem scheme (i.e., the Huffman code + a traditional turbo code) under the 26-English-letter text source. The interleaver size is ℓ = 128 × 128. . . 55 5.19 Performance comparison between the joint source-channel turbo code and the

tandem scheme (i.e., the Huffman code + a traditional turbo code) under the normalized 16-English-letter text source. The interleaver size is ℓ = 128 × 128. 56 5.20 Performance comparison between the joint source-channel turbo code and the

tandem scheme (i.e., the Huffman code + a traditional turbo code) under the normalized 16-English-letter text source. The interleaver size is ℓ = 64 × 64. 57 5.21 Performance comparison between the joint source-channel turbo code and the

tandem scheme (i.e., the Huffman code + a traditional turbo code) under the normalized 16-English-letter text source. The interleaver size is ℓ = 256 × 256. 58 5.22 Performance comparison of the modified JSC turbo code by testing different

(12)

Chapter 1 Introduction

The underlying aim of a communication system is to transmit data from the source to the destination over possibly noisy channels. From the aspect of engineering as well as research, the ultimate goal is to find approaches to transmit data more efficiently and meanwhile more reliably.

In leu of transmission efficacy, the procedure of source coding (more often be referred to as data compression) is employed in communication systems to reduce as much redundancy as possible. In a sense, the source coding shortened the information sequence (either dis-tortionlessly or with an acceptable degree of distortion); hence, the efficiency is improved. In reality, there often exist noise and interference when transmitting data over a medium such that the receiver might not ensure the correctness of the received data. So in order to enhance transmission reliability, the channel coding is introduced. As a contrary, channel coding commonly adds redundancy to protect the data from corruption.

In 1948 , Shannon [1] has proved that there exist a source coding scheme and separately a channel coding scheme such that the data can be transmuted at a rate approaching the theoretical transmission limit. Ever since the communication system usually treats the source coding and channel coding separately for their design convenience. A typical digital

(13)

Figure 1.1: Block diagram of a typical digital communication system with separate source and channel coding.

communication system is depicted in Fig. 1.1.

One of the theoretical requirement for achieving the capacity by separate design is the the length of information sequence should approach infinity, which in a sense implies that the communication delay and decoding complexity might go to infinity. However, in reality, we cannot tolerate infinite delay and complexity, and hence under the practical constraints of finite delay and limited complexity, the joint source-channel coding system might outperform the traditional separate one [2].

Along this research direction, the authors in [3] derive the union bound for the so-called joint source-channel variable-length error-correcting (VLEC) codes , and propose a method to construct the VLEC code that possibly minimizes this bound. However, there are some possible drawbacks for a VLEC: e.g., i) its varying decoding delay, ii) error propagation, and iii) an extra mechanism may be needed for the receiver to identify the end of a codeword. For this reason, we will focus on the joint source-channel fixed-length error-correcting code (FLEC) design in this thesis.

(14)

is to derive the union bound for the error rate of FLECs as parallel to [3], and determine the best FLEC that possibly minimizes this bound. The second approach is to presume the turbo coding structure and devises a joint source-channel turbo coding scheme that suits the transceiving of non-uniform sources.

There have been some publications taking the second approach to design FLECs. In [4] and [5], the authors provides encoding and decoding schemes for binary non-uniform sources and binary Markov sources assuming the turbo coding structure. In this thesis, we extend their results to non-binary non-uniform sources. Details will be given in Chapter 4.

The remainder of the thesis is organized as follows. Chapter 2 introduces the notations we use in this thesis and provides the necessary background for the maximum a posteriori (MAP) metric and turbo coding. Chapter 3 derives the union bound of the FLEC, followed by the construction of the FLEC that gives an acceptably low union bound low. Chapter 4 devises the modified turbo coder for n-ary non-uniform sources. Chapter 5 presents and remarks on numerical and simulation results, and Chapter 6 concludes the thesis.

(15)

Chapter 2 Preliminaries

2.1 Codeword and Codebook

In the joint source-channel coding system considered in this thesis, there are n symbols {s1, s2, . . . , sn} in the source alphabet with probabilities of transmission {p1, p2, . . . , pn}, respectively. The ith source symbol si will be mapped to a codeword ci= (ci,1, ci,2, . . . , ci,ℓ) of fixed length ℓ, where ci,j ∈ {0, 1} for every i, j. The code book C can therefore be represented as a matrix below:

C ,      c1 c2 ... cn      =      c1,1 c1,2 . . . cℓ,l c2,1 c2,2 . . . c2,ℓ ... ... ... cn,1 cn,2 . . . cn,ℓ      .

2.2 MAP Decoding Criterion

Denoting by r = (r1, r2, . . . , rℓ) the received vector corresponding to the transmission of codewords through the additive white Gaussian noise (AWGN) channel, we present the respective MAP decoding criterion as follows.

(16)

2.2.1 Hard-Decision Decoding

Define the hard-decision sequence y = (y1, y2, . . . , yℓ) corresponding to received vector r = (r1, r2, . . . , rℓ) by

yi= (

0, if ri > 0 1, otherwise

Based upon y , the maximum a posteriori (MAP) hard-decision decoding is to find the codeword cm such that

Pr (cm | y) ≥ Pr (ci | y) for all 1 ≤ i ≤ n. (2.1) Equivalently, (2.1) can be transformed to:

Pr (y | cm) Pr (cm)

Pr (y ) ≥

Pr (y | ci) Pr (ci) Pr (y )

⇔ Pr (y | cm) Pr (cm) ≥ Pr (y | ci) Pr (ci) . (2.2) Instead of using the above hard-decision decoding aspect, we can directly reduce the AWGN channel with hard-decision decoding to the binary symmetric channel (BSC) with crossover probability b = Qq2EbR N0 , where Q(x) = _√1 2π R∞ x e−u 2_/2

du is the Q-function, Eb is the energy per information bit, N0 is the variance of additive Gaussian noise sample, and R is the code rate. Letting hi = dH(ci, y ), where dH(·, ·) is the Hamming distance, we have

Pr (y | ci) = bhi(1 − b)ℓ−hi. Criterion (2.2) can thus be re-expressed as that for all 1 ≤ i ≤ n,

bhm (1 − b)ℓ−hm Pr (cm) ≥ bhi(1 − b)ℓ−hiPr (ci) ⇔ lnhbhm (1 − b)ℓ−hm Pr (cm) i ≥ lnhbhi_{(1 − b)}ℓ−hi Pr (ci) i ⇔ hmln(b) + (ℓ − hm) ln (1 − b) + ln Pr (cm) ≥ hiln(b) + (ℓ − hi) ln (1 − b) + ln Pr (ci) ⇔ hm[ln(b) − ln (1 − b)] + ln Pr (cm) ≥ hi[ln(b) − ln (1 − b)] + ln Pr (ci) .

(17)

2.2.2 Soft-Decision Decoding

For a soft-decision decoding, the log-likelihood ratio φi in response to the reception of soft received value ri is given by:

φi = ln Pr (ri | 0) Pr (ri | 1) = ln 1 √ πN0e −(ri−1)2/N0 1 √ πN0e −(ri+1)2/N0 = ln e[ −(ri−1)2+(ri+1)2]/N0 ₌ 4 N0 ri. Thus, the soft maximum a posteriori decision must satisfy that for 1 ≤ i ≤ n,

Pr (cm | r) ≥ Pr (ci | r) ⇔ f (r | c_{f (r )}m) Pr (cm) ≥ f (r | c_{f (r )}i) Pr (ci) ⇔ f (r | cm) Pr (cm) ≥ f (r | ci) Pr (ci) ⇔ ℓ Y j=1 f (rj | cm,j) Pr (cm) ≥ ℓ Y j=1 f (rj | ci,j) Pr (ci) ⇔ ℓ X j=1 ln f (rj | cm,j) + ln Pr (cm) ≥ ℓ X j=1 ln f (rj | ci,j) + ln Pr (ci) ⇔ ℓ X j=1 lnf (rj | cm,j) f (rj | ci,j) + ln Pr (cm) ≥ ln Pr (ci) ⇔ ℓ X j=1 lnf (rj | cm,j) f (rj | ci,j) − ln f (rj | ci,j) f (rj | cmj) + 2 ln Pr (cm) ≥ 2 ln Pr (ci) ⇔ ℓ X j=1 [(−1)cm,j φj − (−1)ci,jφj] + 2 ln Pr (cm) ≥ 2 ln Pr (ci) ⇔ ℓ X j=1 (−1)cm,j φj + 2 ln Pr (cm) ≥ ℓ X j=1 (−1)ci,j φj+ 2 ln Pr (ci)

where we use f (·) to denote the probability density function of respective random entity.

2.3 Turbo Encoder

The scheme of a turbo encoder consists of two recursive systematic convolutional encoders (RSC) and an interleaver. The component encoder used in our thesis is the (37,21) RSC as

(18)

Figure 2.1: The (37,21) recursive systematic encoder .

shown in Figure 2.1. In addition, we adopt the the Berrou-Glavieux interleaver, which is described in the following.

Place the input sequence as an s × s matrix. For a bit located at row i and column j, the permuted or interleaved position at row ir and column jr satisfy

ir = 129 (i + j) mod s and

jr= [p (ξ) × (j + 1) − 1] mod s,

where ξ = (i + j) mod 8 and the assignment of function p(·) is given by:                              p(0) = 17 p(1) = 37 p(2) = 19 p(3) = 29 p(4) = 41 p(5) = 23 p(6) = 13 p(7) = 7

(19)

2.4 Turbo Decoding

In this section, we derive the metric that will be used for turbo decoding.

Again, r is the received sequence of length ℓ. Denote by ui is the ith bit in the decision sequence. Then, we derive:

Pr (ui = 0 | r) = X (Swi−1¯ ,Swi)∈T0 Pr S_wi−1_¯ , S_wi _{| r} = X (Swi−1¯ ,Swi)∈T0 Pr Swi−1¯ , Swi, r Pr (r )

where T0 is the set of consecutive two states respectively located at trellis level i − 1 and i, of which the connecting branch is labeled with code bit 0, and subscripts ¯w and w denote the indices of the trellis states. In parallel, we also derive:

Pr (ui = 1 | r) = X (Swi−1¯ ,Swi)∈T1 Pr S_wi−1_¯ , S_wi _{| r} = X (Swi−1¯ ,Swi)∈T1 Pr Swi−1¯ , Swi, r Pr (r )

where T1 is the set of consecutive two states respectively located at trellis level i − 1 and i, of which the connecting branch is labeled with code bit 1.

Define the log-likelihood ratio (LLR) of the ith decision bit as follows: Λ (i) = logPr (ui = 1 | r) Pr (ui = 0 | r) = log P (Swi−1_¯ ,Swi)∈T1Pr S i−1 ¯ w , Swi, r P (Swi−1¯ ,Swi)∈T0Pr S i−1 ¯ w , Swi, r .

Then, the ith bit is declared zero, i.e., ui = 0, if Λ (i) ≤ 0, and is declared one, otherwise. The term inside the summation, i.e., Pr Swi−1¯ , Swi, r can be further decomposed as follows.

Pr Si−1 ¯ w , Swi, r = Pr Si−1 ¯ w , Swi, ri−11 , ri, rℓi+1

= Pr rℓ_i+1_{| S}_wi−1_¯ , S_wi, ri−1₁ , ri Pr Swi−1¯ , Swi, ri−11 , ri (2.3) = Pr rℓ_i+1_{| S}_wi Pr Si−1 ¯ w , Swi, ri−11 , ri (2.4) = Pr ri+1ℓ | Swi Pr Swi, ri | Swi−1¯ , ri−11 Pr Swi−1¯ , ri−11

= Pr S_wi−1_¯ , ri−1₁ Pr Si

w, ri | Swi−1¯ , ri−11 Pr rℓi+1 | Swi

= Pr S_wi−1_¯ , ri−1₁ Pr Si

w, ri | Swi−1¯ Pr rℓi+1| Swi

(20)

Define α S_wi−1_¯ , Pr Si−1 ¯ w , ri−11 β S_wi , Pr rℓ i+1 | Swi γ S_wi−1_¯ , S_wi , Pr Si w, ri | Swi−1¯ Then Pr S_wi−1_¯ , S_wi, r = α Si−1 ¯ w γ Swi−1¯ , Swi β Swi

Therefore, determination of the three functions can decide Λ(i), which in turns decides the ith bit ui.

Functions α and β can be recursively computed as follows. α S_wi = Pr S_wi, ri₁ = 15 X ¯ w=0 Pr S_wi−1_¯ , S_wi, ri₁ = 15 X ¯ w=0 Pr S_wi−1_¯ , S_wi, ri−1₁ , ri = 15 X ¯ w=0 Pr S_wi−1_¯ , ri−1₁ Pr Si w, ri | Swi−1¯ , ri−11 = 15 X ¯ w=0 α S_wi−1_¯ γ Si−1 ¯ w , Swi ,

(21)

β S_wi_¯ = Pr rℓ_i+1 _{| S}_wi_¯ = Pr r ℓ i+1, Swi¯ Pr (Si ¯ w) = 15 X w=0 Pr rℓ i+1, Swi¯, Swi+1 Pr (Si ¯ w) = 15 X w=0

Pr ri+1, rℓi+2, Swi¯, Swi+1 Pr (Si ¯ w) = 15 X w=0 Pr rℓ

i+2 | ri+1, Swi¯, Swi+1 Pr (ri+1, Swi¯, Swi+1) Pr (Si ¯ w) = 15 X w=0 Pr rℓ

i+2 | Swi+1 Pr (ri+1, Swi¯, Swi+1) Pr (Si ¯ w) = 15 X w=0 Pr rℓ

i+2 | Swi+1 Pr (ri+1, Swi+1| Swi¯) Pr (Swi¯) Pr (Si ¯ w) = 15 X w=0

Pr rℓ_i+2 _{| S}_wi+1 Pr ri+1, Swi+1| Swi¯ = 15 X w=0 β S_wi+1 γ Si ¯ w, Swi+1 ,

where for the additive white Gaussian noise channel, function γ is equal to: γ S_wi−1_¯ , S_wi = Pr S_wi, ri | Swi−1¯ = Pr S i−1 ¯ w , Swi, ri Pr Swi−1¯ = Pr ri | S i−1 ¯ w , Swi Pr Swi−1¯ , Swi Pr Swi−1¯ = Pr (ri | xi) Pr S i−1 ¯ w , Swi Pr Swi−1¯ = Pr (ri | xi) Pr S i w | Swi−1¯ Pr Swi−1¯ Pr Swi−1¯ = Pr (ri | xi) Pr Swi | Swi−1¯ = Pr (ci) Pr (ri | xi) = Pr (ci) 1 √ 2πσexp{− || ri− xi ||2 2σ2 }.

(22)

We now state the algorithmic procedures to obtain the three functions. Note that the RSC trellis we adopt has 16 states as shown in Figure 2.1.

Derivation of function α

1. Initialization: Set α (S00) = 1 and α (Sw0) = 0 for all 1 ≤ w ≤ 15. 2. Recursion: For 1 ≤ i ≤ ℓ, compute α (Si

w) = P15 ¯ w=0α Swi−1¯ γ Swi−1¯ , Swi . Derivation of function β 1. Initialization: Set β Sℓ 0 = 1 and β Swℓ = 0 for 1 ≤ w ≤ 15. 2. Recursion: For 1 ≤ i ≤ ℓ, β (Si ¯ w) = P15 w=0β (Swi+1) γ (Swi¯, Swi+1). Derivation of function γ

1. Initialization: Give that ci is the information bit corresponding to the state transition from Swi−1¯ to Swi. Let the corresponding output vector be xi.

2. Computation: γ Swi−1¯ , Swi = Pr (ci)√_2πσ1 exp{−||ri−xi||

2

(23)

Chapter 3 Joint Source-Channel Block Code

In this chapter, the proposed construction approach for the joint source-channel block codes for non-uniform source distributions over additive white Gaussian noise (AWGN) channels will be presented.

3.1 Union bound

In general, it is difficult to determine the close-form formula for the symbol error probability (SEP) corresponding to a codebook and a source distribution. When a criterion for the SEP is needed, researchers will mostly derive the union bound instead. As such, we will use the union bound as a design criterion for the proposed joint source-channel code.

For the derivation of the union bound, a pair-wise error probability should be devised first. Consider two source symbols si and sj respectively with probabilities pi and pj, and assume that they are mapped respectively to codewords ci and cj, between which the Hamming distance is h. In such case, an error occurs when the receiver declares the reception of sj but si is transmitted, or vice versa. Denote the previously mentioned pair-wise error probability

(24)

by Pr(si → sj). Then, when ci is transmitted over the BSC, Pr(si→ sj) =          h P e=(h+1)/2 h eb

e_{(1 − b)}h−e _{for odd h}

1 2 h h/2bh/2(1 − b) h/2 + h P e=h/2+1 h ebe(1 − b)

h−e _{for even h}

Since the above Pr(si → sj) is only a function of h, we will denote it by Kh.

Next, we denote by Pe(h) the probability that a transmitted codeword ci is incorrectly decoded to another codeword at Hamming distance h from it. Then, Pe(h) can be bounded above by: Pe(h) ≤ n X i=1 pi X j:dH(ci,cj)=h Kh,

where dH(ci, cj) is the Hamming distance between codewords ci and cj. The system error probability Pe must then satisfy:

Pe ≤ ℓ X h=1 Pe(h) ≤ ℓ X h=1 n X i=1 pi X j:dH(ci,cj)=h Kh = ℓ X h=1 Kh n X i=1 pi X j:dH(ci,cj)=h

where the last step follows since Kh is nothing to do with i and j. Define Ah , n P i=1 P j:dH(i,j)=h pi to be the average number of codeword pairs with Hamming distance h . Finally, we have

Pe ≤ ℓ X

h=1

KhAh.

We then use the above upper bound as a design criterion for our joint source-channel coding design.

In summary, based on a given code rate R = 1/4 and system signal-to-noise ratio per information bit Eb/N0 = 10 dB, we can compute Kh. Afterwards, we will attempt to find a good joint source-channel block code of length ℓ that hopefully minimizes

ℓ P

h=1

(25)

3.2 Construction of Joint Source-Channel Block Code

For a small codeword length such as ℓ = 8, we can exhaustively search for all the possi-ble code designs and find the one that minimizes Pℓ

h=1

KhAh. However, such an exhaustive search approach may not be feasible for a larger codeword length. We therefore propose a sub-optimal but low-complexity algorithm to construct a joint source-channel code with a prohibitively small union bound value.

For given n source symbol s1, . . . , sn with probabilities of occurrence p1, . . . , pn, and for a specified codeword length ℓ , we propose to construct a joint source-channel block code as follows.

Step 0. Initialize the length index as one, namely, y = 1. Set code matrices

C =      c1,1 c1,2 . . . c1,ℓ c2,1 c2,2 . . . c2,ℓ ... ... ... cn,1 cn,2 . . . cn,ℓ      = 0 and C′ =      c′ 1,1 c′1,2 . . . c′1,ℓ c′ 2,1 c′2,2 . . . c′2,ℓ .. . ... ... c′ n,1 c′n,2 . . . c′n,ℓ      = 0,

where 0 is the all-zero matrix of proper size.

Step 1. For 1 ≤ i, j ≤ n, calculate ai,j,y = dH(ci,1ci,2. . . ci,y−1, cj,1cj,2. . . cj,y−1) .

Step 2. Compute the union bound contribution for each codeword, i.e., for 1 ≤ i ≤ n, gi =

n X

j=1

Kai,j,y × pi.

Step 3. Set the codeword index q = 1. Sort {gi}ni=1 such that gs1 ≥ gs2 ≥ . . . ≥ gsn, where

{si}ni=1 denotes the sequence of the sorted ordering. Step 4. For 1 ≤ u < q, calculate

(

d0,u = (0 ⊕ csu,y) + asq,su,y

(26)

and r0 =    0, q = 1 q−1 P u=1 Kd0,u × psq+ psu , q > 1 , r1 =    0, q = 1 q−1 P u=1 Kd1,u × psq + psu , q > 1

Step 5. Compare r0 and r1 as follows.

If r0 > r1, choose csq,y = 1; else choose csq,y = 0.

Step 6. For 1 ≤ i, j ≤ n, calculate bi,j,y = dH c′i,1c′i,2. . . c′i,y−2, c′j,1c′j,2. . . c′j,y−2.

Step 7. Compute the union bound contribution for each codeword, i.e., for 1 ≤ i ≤ n, gi =

n X

j=1

Kbi,j,y × pi.

Step 8. Set the codeword index q = 1. Sort {gi}ni=1 such that gs1 ≥ gs2 ≥ . . . ≥ gsn, where

{si}ni=1 denotes the sequence of the sorted ordering. Step 9. If y ≥ 2, compute for 1 ≤ u < q,

         d00,u= (00 ⊕ c′su,y−1c′su,y) + bsq,su,y d01,u= (01 ⊕ c′su,y−1c′su,y) + bsq,su,y d10,u= (10 ⊕ c′su,y−1c′su,y) + bsq,su,y d11,u= (11 ⊕ c′su,y−1c′su,y) + bsq,su,y and                                            r00=    0, q = 1 q−1 P u=1 Kd00,u× psq + psu , q > 1 r01=    0, q = 1 q−1 P u=1 Kd01,u× psq + psu , q > 1 r10=    0, q = 1 q−1 P u=1 Kd10,u× psq + psu , q > 1 r11=    0, q = 1 q−1 P u=1 Kd11,u× psq + psu , q > 1 .

(27)

Step 10. Among r00, r01, r10 and r11;

if r00 is the minimum one, choose c′sq,y−1c′sq,y = 00;

else if r01 is the minimum one, choose c′sq,y−1c′sq,y = 01;

else if r10 is the minimum one, choose c′sq,y−1c′sq,y = 10;

else choose c′

sq,y−1c′sq,y = 11.

Step 11. Set q = q + 1 and repeat Steps 4–8 until q = n.

Step 12. Compare the union bound of C and C′_{, and let C}∗ _{be the one with a smaller} union bound.

Step 13. Set y = y + 1, C′ _{= C and C = C}∗_{. Then repeat Steps 1–10 until y = ℓ.} This algorithm will give us a codebook C∗ _{with a satisfiable union bound value.}

3.3 Decoding

For a joint source-channel block code, it has been known that the MAP decoder is optimal in the sense of minimizing the symbol error probability. The MAP decoder in general does not have an efficient implementation. Fortunately, it can be noted that the MAP decoding metric derived in Section 2.2 is strictly decreasing when being calculated in a bit by bit fashion; hence, we can use the priority-first sequential search algorithm as a vehicle to obtain the MAP decision. In this thesis, the priority-first sequential search decoding algorithms have two forms: one for hard-decision decoding and the other for soft-decision decoding.

3.3-A) Priority-first sequential search hard-decision decoding algorithm

Step 0. Construct the binary code tree corresponding to the given codebook, in which a path form the root node to a leaf node is a codeword. Each leaf node (equivalently,

(28)

the end node of a codeword path) is associated with a probability corresponding to the probability of occurrence for the source letter that is encoded to this code-word. The probability associated with a non-leaf node is the largest one among all probabilities associated with those leaf nodes that are offsprings of this non-leaf node.

Step 1. Calculate the hard-decision sequence y. Initialize the metric of the root node as 0, and push the root node into the stack.

Step 2. Extract the node with the maximal metric value from the stack, which we denote it by e. Then, for all child nodes of the extracted node, if there is any, perform the following procedure and then repeat Step 2.

• If the child node of the extracted node is a non-leaf node, set the metric of this child node to

e + (ci,j ⊕ yj) ln b 1 − b ,

where ci,j is the code bit corresponding to the tree branch connecting the extracted node and the child node, and j is the tree level at which the extracted node is located. Push the child node into the stack.

• Else if the child node of the extracted node is a leaf node, set the metric of this child node to

e + (ci,j⊕ yj) ln b 1 − b + ln (pi) ,

where pi is the probability associated with this leaf node. Push the child node into the stack.

If the extracted node is a leaf node (which certainly has has no child nodes), output the codeword corresponding to the leaf node and stop the algorithm.

(29)

3.3-B) Priority-first sequential search soft-decision decoding algorithm Step 0. The same as Step 0 in the algorithm in 3.3-A.

Step 1. Calculate the log-likelihood ratio φj for all 1 ≤ j ≤ ℓ. Initilize the metric of the root node as 0, and push the root node into the stack.

Step 2. Extract the node with the maximal metric value from the stack, which we denote it by e. Then, for all child nodes of the extracted node, if there is any, perform the following procedure, and then repeat Step 2.

• If the child node of the extracted node is a non-leaf node, set the metric of this child node to

e + (−1)ci,j_φ

i,

where ci,j is the code bit corresponds to the tree branch connecting the ex-tracted node and the child node, and and j is the tree level at which the extracted node is located. Push the child node into the stack.

• Else if the child node of the extracted node is a leaf node, set the metric of this child node to

e + (−1)ci,j_φ

i+ 2 ln (pi) ,

where pi is the probability associated with this leaf node. Push the child node into the stack.

If the extracted node is a leaf node (which certainly has has no child nodes), output the codeword corresponding to the leaf node and stop the algorithm.

We remark that when the metric is non-increasing along all paths, the sequential search of the above two algorithms guarantee to find the optimal leaf path with the maximal metric.

(30)

However, the metrics we adopt in the two algorithms are apparently not non-increasing; so the algorithms we propose may end up with a suboptimal codeword path.

Since at low SNR, the complexity of the second sequential search algorithm may be too large to be practical, an amendment refinement to remove some of the tree nodes during the decoding search is additional proposed below.

• When extracting a node at tree level j and associated with probability q, it will be directly discarded if

2 (ℓ − jM) − ln qM ≤ (jM − j) − ln q,

where qM is the largest probability associated with those nodes that have been visited thus far, and jM is the level at which the node that decides qM is located.

(31)

Chapter 4 Modified Turbo Code

4.1 Background

In Chapter 3, we have proposed a novel approach to construct a joint source-channel block code for a non-uniform source. This approach however cannot be feasibly applied when the codeword length of interest is moderately large, and hence the attainable error rates are limited. As such, in this chapter, we turn to the modification of turbo codes that were previously designed by assuming uniform prior probabilities on codewords, and target a joint source-channel turbo coding system that can provide a practically acceptable performance for non-uniform sources.

In the literature, there have been publications working on turbo code design for non-uniform source. In [4], a turbo code has been proposed for binary independent and iden-tically distributed (i.i.d.) source with non-uniform marginal distribution. In [5], the turbo code design has been extended to binary first-order Markov sources. Note that for binary i.i.d. sources, each information bit is statistically independent of all previous bits, which facilitates the derivation of the corresponding decoding metric for turbo decoder. When a first-order Markov source is considered, each bit only depends on the previous bit; nonethe-less, the extension derivation of the turbo decoding metric can make use of this statistical

(32)

structure. In this thesis, we actually consider a source of different statistical nature, where each information bit is dependent on t previous bits with t being a known function of the bit location. Specifically, an English alphabet can be binary-indexed using five bits; so the first bit is surely dependent on the next four bits, and the second bit is statistically affected by the next three bits, etc., while the fifth bit is actually independent of the next bit if the stream of English letters is i.i.d. in nature. Details will be given in later sections.

4.2 Definitions and Notations

Assume there are n symbols {s1, s2, . . . , sn} that are generated according to an independently and identically distributed distribution with marginal probabilities {p1, p2, . . . , pn}. Let g = ⌈log2n⌉. We can then binary-index each source symbol using g bits as s1 = (00 . . . 00), s2 = (00 . . . 01), s3 = (00 . . . 10), etc. As a result, for 1 ≤ a < g, the (mg + a)th bit only depends on the previous a − 1 bits.

For notational convenience, we derive in the following by assuming the ith bit ui is dependent on the previous t bits u_i−1. . . u_i−t. By our setting in the previous paragraph, u_i−1 should be dependent only on bits u_i−2. . . u_i−t. Denote Ui = ui−1. . . ui−t= ui−1Ui−1.

Denote by Si

w the state at level i, and by Tc the set of state pair Swi−1¯ , Swi such that the input bit ui = c will make the trellis transition from state Swi−1¯ to state Swi . Let x be the codeword sequence corresponding to the input information sequence {ui}ℓ1, and let xi be the codeword vector portion corresponding to ui. For convenience, we use xji to denote xixi+1. . . xj. Similarly, denote by ri the channel output due to input xi, and use rji to denote riri+1. . . rj. Also, abbreviate r = rℓ1 = r1r2. . . rℓ. Note that for 1/3-rate turbo coding system, xi = xsi, x 1p i , x 2p i and ri = ris, r 1p i , r 2p

i , where superscripts “1p” and “2p” indicates the first and second parity-check codeword portions, respectively. We then illustrate the general scheme of a turbo decoder in Fig. 4.1, in which Λ(j)_{(i) is the log-likelihood ratio}

(33)

Figure 4.1: Diagram of a sample turbo decoder.

of the ith bit ui computed by the jth component code decoder, and Λ(j)ex (i) is the extrinsic information of the ith bit ui obtained from the jth component code decoder.

4.3 Modified Decoding Metric

The derivation of the decoding metric for the turbo code in the previous chapter is based on the assumption that the source is uniform i.i.d. When the source is not uniformly distributed, the equality of (2.3) and (2.4) is no longer valid. Hence, an alternative decomposition of Pr (ui = c | r) should be done.

Let Vt be the binary bit stream of length t , i.e., Vt = vtvt−1. . . v1, where vi ∈ {0, 1} for every i, and denote Vt= vtVt−1. We then derive:

Pr (ui = c | r) = Pr (ui = c, r ) Pr (r ) = X (Swi−1¯ ,Siw)∈Tc Pr Swi−1¯ , Swi, r Pr (r ) ,

(34)

where X (Swi−1_¯ ,Swi)∈Tc Pr S_wi−1_¯ , S_wi, r = X (Si−1w¯ ,Swi)∈Tc X Vt∈{0,1}t Pr S_wi−1_¯ , S_wi, Ui = Vt, r = X (Si−1w_¯ ,Swi)∈Tc X Vt∈{0,1}t

Pr S_wi−1_¯ , S_wi, Ui = Vt, ri−11 , ri, rℓi+1 = X (Si−1w_¯ ,Swi)∈Tc X Vt∈{0,1}t Pr ri−1₁ Pr Si−1 ¯

w , Ui = Vt| ri−11 Pr Swi, ri, rℓi+1| Swi−1¯ , Ui = Vt, ri−11

(4.1) Given state Swi−1¯ and Ui, the received bit stream ri−11 is independent of Swi, ri, rℓi+1. Hence,

the derivation of (4.1) can be continued as follows. X

(Swi−1¯ ,Swi)∈Tc

X

Vt∈{0,1}t

Pr ri−11 Pr Swi−1¯ , Ui = Vt| ri−11 Pr Swi, ri, rℓi+1| Swi−1¯ , Ui = Vt, ri−11 = X (Si−1w_¯ ,Swi)∈Tc X Vt∈{0,1}t Pr ri−1₁ Pr Si−1 ¯

w , Ui = Vt| ri−11 Pr Swi, ri, rℓi+1| Swi−1¯ , Ui = Vt = X (Si−1w_¯ ,Swi)∈Tc X Vt∈{0,1}t " Pr ri−1₁ Pr Si−1 ¯ w , Ui = Vt| ri−11 Pr S i w, ri, Swi−1¯ , Ui = Vt Pr Swi−1¯ , Ui = Vt ×Pr S i w, ri, Swi−1¯ , Ui = Vt, rℓi+1 Pr Si w, ri, Swi−1¯ , Ui = Vt # = X (Si−1w_¯ ,Swi)∈Tc X Vt∈{0,1}t Pr ri−1

1 Pr Swi−1¯ , Ui = Vt| ri−11 Pr Swi, ri | Swi−1¯ , Ui = Vt

× Pr rℓi+1| ri, Swi−1¯ , Swi, Ui = Vt .

(35)

indepen-dent of ri when given (ui, Ui), we can continued the derivation as: Pr (ui = c, r ) = X (Si−1w¯ ,Swi)∈Tc X Vt∈{0,1}t Pr ri−1

1 Pr Swi−1¯ , Ui = Vt| ri−11 Pr Swi, ri | Swi−1¯ , Ui = Vt × Pr rℓi+1| ri, Swi−1¯ , Swi, Ui = Vt = X (Si−1w_¯ ,Swi)∈Tc X Vt∈{0,1}t Pr ri−1

and observe that when t = 0, bit ui is independent of any previous bits and the above functions are reduced to:

α V0, Swi−1¯ , Pr Si−1 ¯ w | ri−11 β c, V0, Swi , Pr rℓ i+1| ui = c, Swi γ c, V0, Swi−1¯ , Swi , Pr ui = c, Swi, ri | Swi−1¯ = Pr Swi , ri | Swi−1¯

(36)

Hence, Λ(1)(i) = lnPr (ui = 1 | r) Pr (ui = 0 | r) = lnPr (ui = 1, r ) Pr (ui = 0, r ) = ln P (Si−1_w_¯ ,Si_w)_∈T1 P V t∈{0,1}t

Pr(ri−1₁ )Pr(S_wi−1_¯ _{,U i=V t|r}i−1₁ )Pr(Si_{w ,r i |S}i−1_w_¯ _{,U i=V t})Pr(r ℓ_{i+1|ui,U i=V t,S}wi) P

(Si−1_w_¯ ,Si_w)_∈T0 P V t∈{0,1}t

Pr(ri−1₁ )Pr(S_wi−1_¯ _{,U i=V t|r}i−1₁ )Pr(Si_{w ,r i |S}i−1_w_¯ _{,U i=V t})Pr(r ℓ_i+1_{|ui,U i=V t,S}_wi)

= ln P (Swi−1_¯ ,Swi)∈T1 P Vt∈{0,1}t α Vt, Swi−1¯ γ 1, Vt, Swi−1¯ , Swi β (1, Vt, Swi) P (Swi−1¯ ,Swi)∈T0 P Vt∈{0,1}t α Vt, Swi−1¯ γ 0, Vt, Swi−1¯ , Swi β (0, Vt, Swi)

For additive white Gaussian noise channels, function γ is given by γ c, Vt, Swi−1¯ , Swi = Pr ui = c, Swi, ri | Swi−1¯ , Ui = Vt = Pr ui = c, S i w, ri, Swi−1¯ , Ui = Vt Pr Swi−1¯ , Ui = Vt = Pr ri | ui = c, S i w, Swi−1¯ , Ui = Vt Pr ui = c, Swi, Swi−1¯ , Ui = Vt Pr Swi−1¯ , Ui = Vt = Pr ri | ui = c, Swi, Swi−1¯ , Ui = Vt Pr ui = c, Swi | Swi−1¯ , Ui = Vt = Pr (ri | xi) Pr (ui = c | Ui = Vt)

The basic structure of the turbo decoding metric is now done.

Now after the execution of a certain rounds of turbo decoding, we denote by r_iex(2) the extrinsic information from the second component decoder after de-interleaving. This rex(2)_i will be the input to the first component decoder. The Gaussian assumption on r_iex(2) from [6] [7] then gives: Pr (ri | xi) = 1 √ 2πσ2 exp      −(r s i − xsi) 2 +r(1p)_i _{− x}(1p)_i 2 2σ2      Prrex(2)_i ui = c .

(37)

Suppose ¯σ2 _{and ¯}_{M are the estimated variance and mean of r}ex(2)

i , respectively, which are defined as follows: ¯ M = 1 ℓ ℓ X i=1 | rex(2)i | ¯ σ2 = 1 ℓ − 1 ℓ X i=1 | riex(2)| − ¯M 2

The probability Prr_iex(2) ui = c

can then be assigned as:

Prr_iex(2) ui = c =          1, if ¯σ2 _{= 0} 1 √ 2π¯σ2 exp      − h r_iex(2)_{− (2u}i− 1) ¯M i2 2¯σ2      , otherwise

Hence, function γ can be given as: γ c, Vt, Swi−1¯ , Swi = Pr (ui = c | Ui = Vt) 1 √ 2πσ2 exp      −(r s i − xsi) 2 +r_i(1p)_{− x}(1p)_i 2 2σ2      × Prrex(2)_i _{| u}i = c .

Similar to what has been introduced in the previous chapter, function α can be recursively computed through the following steps:

Case 1. i 6= mg + 1, where m ∈ Z+ _{and g = ⌈log}

(38)

on the previous t bits, we get: Pr S_wi−1_¯ , Ui = Vt, ri−11

= Pr S_wi−1_¯ , U_i−1= V_t−1, u_i−1 = v1, ri−21 , ri−1 = X w′_:₍_Si−2 w′ ,S i−1 ¯ w )∈Tv1

Pr S_wi−2′ , S_wi−1_¯ , U_i−1= V_t−1, ri−2₁ , r_i−1

= X w′_:₍_Si−2 w′ ,S i−1 ¯ w )∈Tv1 Pr ri−2

1 Pr Ui−1= Vt−1, Swi−2′ | ri−2₁

× Pr Swi−1¯ , ri−1| Swi−2′ , U_i−1= V_t−1, ri−2₁

= X w′_:₍_Si−2 w′ ,S i−1 ¯ w )∈Tv1 Pr ri−2

1 Pr Ui−1= Vt−1, Swi−2′ | ri−2₁

× Pr Swi−1¯ , ri−1| Swi−2′ , U_i−1= V_t−1

= X w′_:₍_Si−2 w′ ,S i−1 ¯ w )∈Tv1

Pr ri−21 α Vt−1, Swi−2′ γ v1, Vt−1, Swi−2′ , S_wi−1_¯ ,

Pr ri−1₁ = 15 X ¯ w=0 X Vt∈{0,1}t Pr S_wi−1_¯ , Ui = Vt, ri−11 = 15 X ¯ w=0 X Vt∈{0,1}t X w′_:₍_Si−2 w′ ,S i−1 ¯ w )∈Tv1

Pr ri−2₁ α V_t−1, S_wi−2′ γ v1, Vt−1, Swi−2′ , S_wi−1_¯ ,

and α Vt, Swi−1¯ = Pr S_wi−1_¯ , Ui = Vt | ri−11 = Pr S i−1 ¯ w , Ui = Vt, ri−11 Pr ri−1₁ = P w′_:₍_Si−2 w′ ,S i−1 ¯ w )∈Tv1

α V_t−1, S_wi−2′ γ v1, Vt−1, Swi−2′ , Swi−1¯ 15 P ¯ w=0 P Vt∈{0,1}t P w′_:₍_Si−2 w′ ,S i−1 ¯ w )∈Tv1

α V_t−1, S_wi−2′ γ v1, Vt−1, Swi−2′ , Swi−1¯

.

Case 2. i = mg + 1, where m ∈ Z+ _{and g = ⌈log}

(39)

of any previous bits. So we can simply the above derivations to: Pr S_wi−1_¯ , ri−1₁ = 1 X c′₌₀ X w′_:₍_Si−2 w′ ,S i−1 ¯ w )∈Tc′

Pr S_wi−2′ , S_wi−1_¯ , ri−2₁ , r_i−1

= 1 X c′₌₀ X w′_:₍_Si−2 w′ ,S i−1 ¯ w )∈Tc′ Pr ri−2₁ Pr Si−2

w′ | ri−2₁ Pr S_wi−1_¯ , r_i−1| S_wi−2′ , ri−2₁

= 1 X c′₌₀ X w′_:₍_Si−2 w′ ,S i−1 ¯ w )∈Tc′ Pr ri−2₁ Pr Si−2

w′ | ri−2₁ Pr S_wi−1_¯ , r_i−1| S_wi−2′ ,

Pr ri−1₁ = 15 X ¯ w=0 Pr S_wi−1_¯ , ri−1₁ = 15 X ¯ w=0 1 X c′₌₀ X w′_:₍_Si−2 w′ ,S i−1 ¯ w )∈Tc′ Pr ri−2₁ Pr Si−2

w′ | ri−2₁ Pr S_wi−1_¯ , r_i−1| S_wi−2′ ,

and α V0, Swi−1¯ = Pr S_wi−1_¯ _{| r}i−1₁ = Pr S i−1 ¯ w , ri−11 Pr ri−1₁ = 1 P c′₌₀ P w′_:₍_Si−2 w′ ,S i−1 ¯ w )∈Tc′ Pr Si−2

w′ | ri−2₁ Pr Swi−1¯ , ri−1| Swi−2′

15 P ¯ w=0 1 P c′₌₀ P w′_:₍_Si−2 w′ ,S i−1 ¯ w )∈Tc′

Pr S_wi−2′ | ri−2₁ Pr Swi−1¯ , ri−1| Swi−2′

, where Pr S_wi−2′ | ri−2₁ = X Vg∈{0,1}g

Pr S_wi−2′ , U_i−1= Vg | ri−21

= X

Vg∈{0,1}g

α VgS_wi−2′ ,

and

Pr S_wi−1_¯ , r_i−1 _{| S}_wi−2′ = Pr (u_i−1 = c′) Pr (r_i−1 | x_i−1)

= Pr (u_i−1= c′)√ 1 2πσ2 exp{− (rs i − xsi) 2 +r(1p)_i _{− x}(1p)_i 2 2σ2 } Pr rex(2)_i _{| u}i = c .

(40)

We next turn to the recursive computation of function β as follows.

Case 1. i 6= mg. In this case, the next bit ui+1 is dependent on the previous bit ui. By defining Vt+1= Vtc, we derive: Pr rℓ_i+1, ui = c, Ui = Vt, Swi = 1 X c′₌₀ Pr rℓ_i+1, ui = c, Ui = Vt, ui+1= c′, Swi = 1 X c′₌₀

Pr ri+1, rℓi+2, Ui+1= Vt+1, ui+1= c′, Swi, Swi+1¯ = 1 X c′₌₀ Pr rℓ

i+2 | ui+1= c′, Ui+1= Vt+1, Swi+1¯

× Pr ri+1| rℓi+2, ui+1= c′, Ui+1= Vt+1, Swi+1¯ Pr ui+1= c′, Ui+1= Vt+1, Swi+1¯ = 1 X c′₌₀ β c′_{, V}

t+1, Swi+1¯ Pr ri+1| ui+1= c′, Ui+1 = Vt+1, Swi+1¯

× Pr ui+1= c′, Ui+1= Vt+1, Swi+1¯ = 1 X c′₌₀ β c′_{, V}

t+1, Swi+1¯ Pr ri+1, ui+1= c′, Ui+1= Vt+1, Swi+1¯

= 1 X

c′₌₀

β c′, Vt+1, Swi+1¯ Pr ri+1, ui+1= c′, Ui+1= Vt+1, Swi+1¯ , Swi = 1 X c′₌₀ β c′_{, V}

t+1, Swi+1¯ Pr ui+1= c′, Swi+1¯ , ri+1 | Swi, Ui+1= Vt+1 × Pr Swi , Ui+1 = Vt+1 = 1 X c′₌₀

(41)

where β c, Vt, Swi = Pr rℓ_i+1_{| u}i = c, Ui = Vt, Swi = Pr r ℓ i+1, ui = c, Ui = Vt, Swi Pr (ui = c, Ui = Vt, Swi) = 1 P c′₌₀ β c′_{, V}

t+1, Swi+1¯ γ c′, Vt+1, Swi, Swi+1¯ Pr (Swi, Ui+1 = Vt+1) Pr (Ui+1= Vt+1, Swi) = 1 X c′₌₀ β c′, Vt+1, Swi+1¯ γ c′, Vt+1, Swi, Swi+1¯ .

Case 2. i = mg. In this case, the next bit ui+1 is independent of the previous bit ui. This simplifies our derivation to:

Pr rℓ_i+1, ui = c, Ui = Vt, Swi = 1 X c′₌₀

Pr ri+1, rℓi+2, ui = c, Ui = Vt, ui+1= c′, Swi, Swi+1¯ = 1 X c′₌₀ Pr rℓ

i+2 | ui+1= c′, Swi+1¯ Pr ri+1 | rℓi+2, ui+1= c′, ui= c, Ui = Vt, Swi+1¯ × Pr ui+1= c′, ui = c, Ui = Vt, Swi+1¯ = 1 X c′₌₀ β c′_{, V}

0, Swi+1¯ Pr ri+1, ui+1= c′, ui= c, Ui= Vt, Swi+1¯ = 1 X c′₌₀ β c′_{, V}

0, Swi+1¯ Pr ui+1 = c′, Swi+1¯ , ri+1| Swi, ui = c, Ui = Vt × Pr Swi, ui = c, Ui = Vt = 1 X c′₌₀

β c′, V0, Swi+1¯ Pr ui+1= c′, Swi+1¯ , ri+1 | Swi Pr Swi, ui = c, Ui = Vt = 1 X c′₌₀ β c′, V0, Swi+1¯ γ c′, V0, Swi, Swi+1¯ Pr Swi, ui= c, Ui= Vt ,

(42)

where β c, Vt, Swi = Pr r ℓ i+1, ui = c, Ui = Vt, Swi Pr (ui = c, Ui = Vt, Swi) = 1 X c′₌₀ β c′, V0, Swi+1¯ γ c′, V0, Swi, Swi+1¯ .

The initial values of functions α and β are:

α (V0, S00) = 1 , α (V0, Sw0) = 0 for every 0 < w ≤ 16 β 0, Vk, Swℓ = β 1, Vk, Swℓ = 12 for every 0 ≤ w ≤ 16 where k ≡ (ℓ − 1) mod g.

It remains to discuss about the iterative decoding scheme based on our newly derived functions. A well-known iterative decoding equation is to iteratively exchange the so-called extrinsic information as follows.

Λ(1)(i) = Λ(1)_ch (i) + Λ(1)ap (i) + Λ(1)ex (i) ⇐⇒ Λ(1)ex (i) = Λ(1)(i) − Λ (1) ch (i) − Λ (1) ap (i) = Λ(1)_{(i) − ln}Pr (r s i | ui= 1) Pr (rs i | ui= 0) − ln Prr_i(ex)_{| u}i = 1 Prr_i(ex)_{| u}i = 0 = ( Λ(1)_{(i) −} 2 σ2ris if ¯σ2 = 0 Λ(1)_{(i) −} 2 σ2ris− 2 ¯σ¯M2 r (ex) i otherwise ,

where Λch(i) and Λap(i) stand for the quantities due to channel and a priori information, respectively.

After updating the extrinsic information by the first component decoder, it is the turn of the second component decoder to modify the extrinsic information. Let {˜u}ℓ

1 be the input sequence after interleaving. Assume that (˜u)ℓ₁ is i.i.d. in statistics; hence the traditional BCJR algorithm described in Section 2.4 can be applicable. Let Λ(2)_{(i) be the log-likelihood}

(43)

ratio updated in Decoder 2. Then we have: ˜

Λ(2)(i) = ˜Λ(2)_ch (i) + ˜Λ(2)_ap (i) + ˜Λ(2)_ex (i) ⇐⇒ ˜Λ(2)_ex (i) = ˜Λ(2)_{(i) − ˜}Λ(2)_ch _{(i) − ˜}Λ(2)_ap (i)

= ˜Λ(2)_{(i) − ln}Pr ˜r s i | ˜ui = 1 Pr ˜rs i | ˜ui = 0 − ln Pr ( ˜ui = 1) Pr ( ˜ui = 0) = ˜Λ(2)_{(i) −} 2 σ2r˜ s i − ˜Λ(1)ex (i) where {˜r}ℓ

1 is the interleaved sequence for {r}ℓ1, and {˜Λ (1)

ex(i)}ℓ_i=1 is the interleaved sequence for {Λ(1)ex(i)}ℓ_i=1.

Based on [5], we slightly adjust the extrinsic information equation for Decoder 2 as follows. Again, let {Λ(2)ex(i)}ℓ_i=1 be the de-interleaved sequence for {˜Λ(2)ex(i)}ℓ_i=1. Then,

Λ(2)_ex (i) = vΛ(2) ex (i) + (1 − v) ln    P Vt∈{0,1}t Pr (ui = 1 | Ui = Vt) Pr (ûi−t= vt) . . . Pr (ûi−1 = v1) P Vt∈{0,1}t Pr (ui = 0 | Ui = Vt) Pr (ûi−t= vt) . . . Pr (ûi−1 = v1)    where Pr (ûi = 0) = 1 1 + eΛ(2)ex(i) , Pr (ûi = 1) = eΛ(2)ex(i) 1 + eΛ(2)ex(i) , and v = ( 1, _{if (i − 1) mod g = 0} 0.85, otherwise

(44)

Chapter 5 Simulation Results

In this chapter, simulation results are provided to demonstrate the performance of the FLEC that we derived in Chapters 3 and 4. Specifically, we examine what has been proposed in Chapter 3 in Section 5.1, and test the FLEC in Chapter 4 in Section 5.2. Remarks on our simulation results are given in Section 5.3.

5.1 Joint Source-Channel Block Code

Throughout this section, the code rate of FLECs examined is fixed as 1/4. The first-order Markov sources are the chosen source distributions. Three Markov source cases will be simulated.

Case 1. Transition distribution Pr (ui = 0 | ui−1 = 0) = Pr (ui = 1 | ui−1 = 1) = 0.9 with initial probability Pr (u1 = 0) = 1 − Pr (u1 = 1) = 0.9.

(45)

Below we illustrate the details of all figures in Section 5.1.

1. Figure 5.1 compares the SER performances between the JSC block codes constructed by exhaustive search and those constructed by our algorithm. The source under test follows Case 1.

2. The setting of Figure 5.2 is same as that in Figure 5.1 except that the source follows Case 2.

3. The setting of Figure 5.3 is same as that in Figure 5.1 except that the source follows Case 3.

4. Figure 5.4 shows the SER performances of JSC block codes constructed by the algo-rithm introduced in Section 3.2 and decoded by the sequential soft-decision decoding algorithm introduced in Section 3.3 for three different codeword lengths. The source follows Case 1.

5. The setting of Figure 5.5 is same as that of Figure 5.4 except that the source follows Case 2.

6. The setting of Figure 5.6 is same as that of Figure 5.4 except that the source follows Case 3.

7. Figure 5.7 compares the SER performances between the JSC block code we constructed and the tandem scheme. The hard-decision decoding scheme is presumed and the source under test follows Case 1. The codeword length of the JSC block code is 64. The tandem scheme selected for performance comparision is the concatenation of a Huffman code with an (n, k, t) = (63, 16, 11) BCH code. Since the tandem scheme becomes variable length in nature, we use the Levenshtein distance [8] to account for the symbol errors.

(46)

8. The setting of Figure 5.8 is same as that of Figure 5.7 except the source follows Case 2.

9. The setting of Figure 5.9 is same as that of Figure 5.7 except the source follows Case 3.

(47)

5 6 7 8 9 10 11 10-5 10-4 10-3 10-2 SER E_b/N₀(dB) algorithm

exhaustive search from 6dB to 10dB exhaustive search at 5dB exhaustive search at 5.5dB union bound of algorithm

union bound of exhaustive search from 6dB to 10dB union bound of exhaustive search at 5dB union bound of exhaustive search at 5.5dB

Figure 5.1: Performance comparison between the joint source-channel block code constructed by exhaustive search and that by our proposed algorithm with codeword length ℓ = 8. The source follows Case 1 and hard-decision decoding is employed.

(48)

exhaustive search from 5dB to 10dB union bound of algorithm

union bound of exhaustive search from 5dB to 10dB

Figure 5.2: Performance comparison between the joint source-channel block code constructed by exhaustive search and that by our proposed algorithm with the codeword length ℓ = 8. The source follows Case 2 and hard-decision decoding is employed.

(49)

exhaustive search from 5dB to 10dB union bound of algorithm

union bound of exhaustive search from 5dB to 10dB

Figure 5.3: Performance comparison between the joint source-channel block code constructed by exhaustive search and that by our proposed algorithm with the codeword length ℓ = 8. The source follows Case 3 and hard-decision decoding is employed.

(50)

0 1 2 3 4 5 6 10-5 10-4 10-3 10-2 10-1 SER E_b/N₀(dB) length=32 length=48 length=64

Figure 5.4: Performance comparison of the proposed joint source-channel block codes for different codeword length. The source follows Case 1 and soft-decision decoding is employed.

(51)

0 1 2 3 4 5 6 10-5 10-4 10-3 10-2 10-1 SER E b/N0(dB) length=32 length=48 length=64

(52)

0 1 2 3 4 5 6 10-5 10-4 10-3 10-2 10-1 SER E_b/N₀(dB) length=32 length=48 length=64

(53)

1 2 3 4 5 6 10-5 10-4 10-3 10-2 10-1 SER E_b/N₀(dB) Huffman+BCH(n=63,k=16,t=11) JSC block code(length=64)

Figure 5.7: Performance comparison between the joint source-channel block code with code-word length 64 and the tandem scheme (i..e, the Huffman code + (63, 16, 11) BCH code). The source follows Case 1 and hard-decision decoding is employed.

(54)

0 1 2 3 4 5 6 10-5 10-4 10-3 10-2 SER E b/N0(dB) Huffman+BCH(n=63,k=16,t=11) JSC block code(length=64)

(55)

2 3 4 5 6 7 8 10-5 10-4 10-3 10-2 10-1 SER E b/N0(dB) Huffman+BCH(n=63,k=16,t=11) JSC block code(length=64)

(56)

5.2 Modified Turbo Code

For all cases in Section 5.2, the number of iterations for turbo decoder is set as 18. Below we illustrate the details of all figures in Section 5.2.

1. Figure 5.10 compares the SER performance between the modified JSC turbo code and a tandem scheme. The tandem scheme we choose for comparison in this section is the concatenation of the Huffman code with a traditional turbo code of interleaver size ℓ = 128 × 128. The number of source symbols is 16. The source symbol is resulted from a group of four bits, each of which is generated according to binary non-uniform source with p0 = 0.55 and p1 = 0.45. The resulting source entropy per symbol is 3.9711 bits.

2. The setting in Figure 5.11 is the same as that in Figure 5.10 except p0 = 0.58. The resulting source entropy per symbol is 3.92582 bits.

3. The setting in Figure 5.12 is the same as that in Figure 5.10 except p0 = 0.634. The resulting source entropy per symbol is 3.79021 bits.

4. The setting of Figure 5.13 is same as that in Figure 5.10 except p0 = 0.65. The resulting source entropy per symbol is 3.73627 bits.

(57)

8. Figure 5.17 compares the SER performance between two different source statistics of the same source entropy. The two different sources distributions are randomly generated.

9. Figure 5.18 compares the SER performances between the modified JSC turbo code and a tandem scheme. The tandem scheme chosen is the concatenation of Huffman code and a traditional turbo code with interleaver size ℓ = 128 × 128. The source is the 26 English alphabet, and its statistics is tabulated in Table 5.1.

10. The setting of Figure 5.19 is same as that in Figure 5.18 except the source alphabet now consists of only 16 most likely letters in Table 5.1. We normalize the distribution for 16-English-letter alphabet to make their probabilities sum to one (cf. Table 5.2). 11. The settings of Figures 5.20 and 5.21 are the same as that in Figure 5.19 except the

interleaver sizes are changed to ℓ = 64 × 64 and 256 × 256, respectively.

12. Figure 5.22 compares the SER performances of the modified JSC turbo code under three different interleaver sizes. The source alphabet contains the normalized most probable 16 English letters.

(58)

5.4 5.6 5.8 6.0 6.2 6.4 6.6 10-5 10-4 10-3 10-2 10-1 SER E s/N0(dB)

1st order Huffman code+turbo code 2nd order Huffman code+turbo code Modified JSC turbo code

Figure 5.10: Performance comparison between the joint source-channel turbo code and the tandem scheme (i.e., the Huffman code + a traditional turbo code) under the source with per-letter source entropy 3.9711 bits. The interleaver size is ℓ = 128 × 128.

(59)

5.4 5.6 5.8 6.0 6.2 6.4 10-5 10-4 10-3 10-2 10-1 SER E_s/N₀(dB)

(60)

5.2 5.4 5.6 5.8 6.0 6.2 6.4 10-5 10-4 10-3 10-2 10-1 100 SER E_s/N₀ (dB)

(61)

5.2 5.4 5.6 5.8 6.0 6.2 6.4 10-5 10-4 10-3 10-2 10-1 SER E_s/N₀(dB)

(62)

5.0 5.2 5.4 5.6 5.8 6.0 6.2 10-5 10-4 10-3 10-2 10-1 SER E s/N0(dB)

(63)

5.0 5.2 5.4 5.6 5.8 6.0 6.2 10-5 10-4 10-3 10-2 10-1 SER E s/N0(dB)

(64)

4.8 5.0 5.2 5.4 5.6 5.8 10-5 10-4 10-3 10-2 10-1 SER E s/N0(dB)

(65)

4.4 4.6 4.8 5.0 5.2 5.4 5.6 5.8 6.0 6.2 10-5 10-4 10-3 10-2 10-1 SER E_s/N₀(dB)

Figure 5.17: Performance comparison under two different source distributions with the same per-letter source entropy 3.20 bits. The interleaver size is ℓ = 128 × 128.

(66)

5.8 6.0 6.2 6.4 6.6 6.8 10-5 10-4 10-3 10-2 10-1 SER E s/N0

Figure 5.18: Performance comparison between the joint source-channel turbo code and the tandem scheme (i.e., the Huffman code + a traditional turbo code) under the 26-English-letter text source. The interleaver size is ℓ = 128 × 128.

(67)

5.4 5.6 5.8 6.0 6.2 6.4 6.6 10-5 10-4 10-3 10-2 10-1 SER E s/N0(dB)

Figure 5.19: Performance comparison between the joint source-channel turbo code and the tandem scheme (i.e., the Huffman code + a traditional turbo code) under the normalized 16-English-letter text source. The interleaver size is ℓ = 128 × 128.

(68)

5.2 5.4 5.6 5.8 6.0 6.2 6.4 6.6 6.8 10-5 10-4 10-3 10-2 10-1 SER E_s/N₀(dB)

(69)

5.3 5.4 5.5 5.6 5.7 5.8 5.9 6.0 6.1 6.2 6.3 6.4 10-5 10-4 10-3 10-2 10-1 SER E s/N0(dB)

(70)

5.4 5.6 5.8 6.0 6.2 6.4 6.6 10-5 10-4 10-3 10-2 10-1 SER E s/N0(dB)

Modified JSC turbo code with length 64×64 Modified JSC turbo code with length 128×128 Modified JSC turbo code with length 256×256

Figure 5.22: Performance comparison of the modified JSC turbo code by testing different interleaver sizes under the normalized 16-English-letter text source.

(71)

Table 5.1: Source distribution of 26 English alphabet symbols. The source entropy is 4.12091 bits.

English probability English probability

alphabet alphabet E 0.14878610 T 0.09354149 A 0.08833733 O 0.07245769 R 0.06872164 N 0.06498532 H 0.05831331 I 0.05644515 S 0.05537763 D 0.04376834 L 0.04123298 U 0.02762209 P 0.02575393 F 0.02455297 M 0.02361889 C 0.02081665 W 0.01868161 G 0.01521216 Y 0.01521216 B 0.01267680 V 0.01160928 K 0.00867360 X 0.00146784 J 0.00080064 Q 0.00080064 Z 0.00053376

Table 5.2: Normalized source distribution of the 16 most probable English letters. The source entropy is 3.78611 bits.

English probability English probability

alphabet alphabet E 0.1627270 T 0.1023060 A 0.0966141 O 0.0792466 R 0.0751605 N 0.0710741 H 0.0637770 I 0.0617338 S 0.0605662 D 0.0478692 L 0.0450963 U 0.0302101 P 0.0281670 F 0.0268535 M 0.0258319 C 0.0227671

針對非均勻訊源之固定長度整合訊源與通道編碼系統之設計

國

立

交

通

大

學

電信工程研究所

碩

士

論

文

針對非均勻訊源之固定長度整合訊源與通道編碼系統之設計

Fixed-Length Joint Source-Channel Coding System for Generally Non-uniform

Sources

研 究 生：林鉑涵

指導教授：陳伯寧 教授

針對非均勻訊源之固定長度整合訊源與通道編碼系統之設計

Fixed-Length Joint Source-Channel Coding System for Generally Non-uniform

Sources

研 究 生：林鉑涵

Student：Po-Han Lin

指導教授：陳伯寧

Advisor：Po-Ning Chen

國 立 交 通 大 學

電信工程研究所

碩 士 論 文

針

針

針對

對

對非

非

非均

均

均勻

勻

勻訊

訊

訊源

源

源之

之

之固

固

固

定

定

定長

長

長度

度

度整

整

整合

合

合訊

訊

訊源

源

源與

與

與通

通

通道

道

道編

編

編碼

碼

碼系

系

系統

統

統之

之

之設

設

設計

計

研究生：林鉑涵

指導教授：陳伯寧教授

研究生：林鉑涵

國立交通大學

碩士論文