基於改良移動預測之最佳化影像失真保護方法

全文

(1)國立交通大學資訊學院資訊科技（IT）產業研發碩士班碩士論文基於改良移動預測之最佳化影像失真保護方法. A RD Optimized Unequal Error Protection Scheme Based on Modified Motion Estimation. 研究生：吳秉承指導教授：蔡文錦. 教授. 中華民國九十七年九月. i.

(2) 基於改良移動預測之最佳化影像失真保護方法 A RD Optimized Unequal Error Protection Scheme Based on Modified Motion Estimation. 研究生：吳秉承. Student：Ping-Chen Eddie Wu. 指導教授：蔡文錦. Advisor：Wen-Jiin Tsai. 國立交通大學資訊學院資訊科技（IT）產業研發碩士班碩士論文. A Thesis Submitted to College of Computer Science National Chiao Tung University in partial Fulfillment of the Requirements for the Degree of Master in Industrial Technology R & D Master Program on Computer Science and Engineering September 2008 Hsinchu, Taiwan, Republic of China. 中華民國九十七年九月 i.

(3) （基於改良移動預測之最佳化影像失真保護方法）. 學生：吳秉承. 指導教授：蔡文錦教授. 國立交通大學資訊學院產業研發碩士班. 摘. 要. 不對等保護(Unequal Error Protection)是一種透過重要度來改變保護資料比例的方法。當網路在傳輸時封包的掉落會造成影像失真，使用上述方法即可降低錯誤機率的發生。本篇論文中，我們提出了一個透過『不對等保護』來最佳化影片的位元率-失真(Rate-Distortion)，並且使用了 Flexible Macroblock Ordering(FMO)的方式，讓比較重要的影格(Macroblock)可以放在相同的群組 (Slice Group)中，且給予其較高的保護。我們參考了 Rose 所提出的 ROPE 方法 [16]來進行分類，透過影格的失真率，來決定他在影像中的重要度；也參考了 Shih 所提出的方法[15]將影格分類到不同的群組中。接著以最佳化影片的位元率 - 失真演算法，來決定分配每一個群組的前向糾錯碼 (Forward Error Correction)的數量，讓較重要的資料給予更高的保護。我們還更改『全零區塊動態預測』方法用來進行收斂性動態預測，使動態預測能夠提高『不對等保護』的效能。上述的收斂性動態預測方法是 Shih 所提出[15]，是以修改過的動態預測來選取較重要的影格，來確保影格有最好的前向糾錯保護。然而，Shih 的方法有可能會造成資料量增加，而影響到可用的 FEC 數量。因此，我們修改了其方法，實驗結果也顯示出我們提出的『全零區塊動態』預測方法有更優良的效能。. i.

(4) 關鍵字：不對等保護、彈性宏塊次序、全零區塊動態預測、位元率-失真最佳化. ii.

(5) A RD Optimized Unequal Error Protection Scheme Based on Modified Motion Estimation. Advisor: Dr. Wen-Jiin Tsai. Student: Ping-Chen Eddie Wu. Industrial Technology R & D Master Program of Computer Science College National Chiao Tung University. ABSTRACT. Unequal Error Protection (UEP) is a method to efficiently protect the more important portions of the data. It has been widely used for video transmitting over the error-prone channels. In this thesis, we present a Rate-Distortion (R-D) optimized UEP scheme for video transmission over error-prone wireless networks. Our UEP scheme is incorporated with Flexible Macroblock Ordering (FMO), by which the more important macroblocks in a picture are grouped into the same slice groups and protected more than others. We had adopted the ROPE method, proposed by Rose [16], for macroblock distortion estimation in order to determine the importance of each macroblock. Also we adopted a modified dispersed FMO mode proposed by Shih [15] for classifying macroblocks into slice groups. And then, an R-D optimized FEC allocation algorithm is proposed to efficiently protect the more important slice groups than others. In addition, a modified Converged Motion Estimation (CME) based on the concept of all-zero-block (AZB) detection algorithm is proposed to further improve the proposed UEP. The CME is a method proposed in [15]. It makes macroblocks been referenced in a skewed manner so that highly important iii.

(6) macroblocks are converged on only few and the use of FEC is efficient. However, CME may result in an increase in the source bit rate. We modify the CME to cope with this problem and our experiments on many video sequences show promising results.. Keywords: Unequal error protection, Flexible macroblock ordering (FMO), All Zero Block motion estimation (AZB), Rate-Distortion. iv.

(7) Table of Contents Abstract (Chinese) .......................................................................................................... i Abstract (English) ........................................................................................................ iii Table of Contents ........................................................................................................... v List of Figures ............................................................................................................... vi List of Tables ................................................................................................................vii Chapter 1 Introduction ................................................................................................ 1 Chapter 2 Related Works ............................................................................................ 8 2.1. . End-to End Distortion ................................................................................ 8 . 2.2. . Macroblock Assignment ............................................................................ 9 . 2.3. . Forward error correction .......................................................................... 11 . 2.4. . Motion Estimation with AZB................................................................... 12 . 2.5. . Wireless Modulation ................................................................................ 15 . Chapter 3 Proposed UEP Method ............................................................................. 17 3.1. . Macroblock Classification Scheme and Assignment ............................... 18 . 3.2. . Converged Motion Estimation with AZB ................................................ 19 . 3.3. . RD Optimized Channel Rate Allocation .................................................. 22 . Chapter 4 Experimental Results ............................................................................... 27 Chapter 5 Conclusion ............................................................................................... 35 Reference ..................................................................................................................... 37. v.

(8) List of Figures Figure 1.1 A typical video communication system .......................................................1 Figure 1.2 Unequal Error Protections ...........................................................................2 Figure 1.3 Macroblock classifications (a) without FMO, (b) dispersed map, (c) original frame of Foreman, (d) explicit FMO mode .............................4 Figure 1.4 ESI mode (a) combination of the dispersed mode and box-out mode (b) Dispersed mode (c) box-out mode .............................................................4 Figure 2.1 Connected macroblocks would cause the error concealment ineffective ..10 Figure 2.2 Macroblock assignment based on dispersed mode ....................................10 Figure 2.3 AZB threshold ...........................................................................................13 Figure 2.4 Region division in a 4x4 block ..................................................................14 Figure 3.1 Flow Chart .................................................................................................17 Figure 3.2 AZB candidates .........................................................................................21 Figure 3.3 Number of AZB candidates in Foreman ....................................................21 Figure 4.1 The PSNR when packet loss rate reach (a) 10% (b) 20% in Foreman ......29 Figure 4.2 The PSNR when packet loss rate reach (a) 10% (b) 20% in Coastguard................................................................30 Figure 4.3 The PSNR when packet loss rate reach (a) 10% (b) 20% in Stefan ..........31 Figure 4.4 The average PSNR when packet loss rate equal to 5% ~ 25% in video stream (a) Foreman (b) Coastguard (c) Stefan .........................................33 Figure 4.5 The average PSNR according by SNR between 5% ~ 30% in video streams (a) Foreman (b) Coastguard (c) Stefan .......................................34 . . vi.

(9) List of Tables Table 2.1 SAD and Threshold for Moon’s condition ..................................................13 Table 2.2 Sufficient condition for detection zero quantized DCT coefficients in a 4x4 block .........................................................................................................15 Table 2.3 Eight PHY modes of the IEEE 802.11a PHY .............................................16 Table 3.1 The symbol table for rate distortion estimation ..........................................22 Table 3.2 Execute time for Lagrange ..........................................................................25 Table 4.1 Compare Proposed UEP with Shih’s UEP method .....................................28. vii.

(10) Chapteer 1 Introd I ductio on ntrodu uction n In In thhe transmisssion of viddeo over erroor-prone neetworks, paccket loss is a key prob blem. Thee video is veery sensitivve to transm mission errorr after beingg compresssed. Associaating the compresseed bit-streaam with foorward erro or correctioon (FEC) for both error e deteection and error e correcttion can oveercome the problem buut more netw work bandw width is needed.. Figuree 1.1 A typicall video comm munication systtem [1]. A tyypical videoo communication systeem [1] invo olves many steps, as shhown in Figure 1.1. The videoo encoder first comppresses the video souurce into a bit-stream m for reduucing the data rate. Then, the compresseed bit-streaam is assiggned FEC and segm mented into packets. After trannsporting packets p oveer networkss, the Chaannel decooder in the receiver sidde receives the t packets and correctts the error if packet iss lost or damaged. d F Finally, the video deccoder decod des the bit--stream to reconstructt the videeo.. In video v sourcee, some datta are moree important than otherss. For exam mple, the piccture headders are muuch more im mportant thaan the block k data becauuse once it iss lost, the en ntire pictture can nott be construucted well. These T impo ortant data should s be pprotected so that theyy can be dellivered withh a much lower error raate as show wn in Figure 1.2. This iss the 1.

(11) concept of welll-known Unequal U Errror Protectio on (UEP). An effectivve UEP sch heme ponnders the crriteria of im mportance from many y kinds of point of vview and takes t advantage of thhe different sensitivities of the videeo.. Figure 1.2 Unequal U Errorr Protections. hannel disttortion wouuld degradee the In a video traansmission system, souurce and ch quality of the video. Succh quality degradation d can be redduced greattly if we could estim mate the soource and chhannel distoortion correectly. Distorrtion estimaation [2,3,4] is a dom main researcched widelyy and is the basic of UE EP. Here we w classify thhe general UEP U metthods into foour categoriies as follow ws:. . Frame-baased. In a Group of Picture (GO OP), it is obbviously thaat I frames are more im mportant than P fram mes, and P frames are more impoortant than B frames. Iff an I framee lost too many m maccroblocks, not n only the I frame couuld not be concealed c w well, but alsoo would cau use a lot of o error proopagation. With W the saame reason,, the front P frame is m more imporrtant thann the rear P frame in the t same GOP G because the formeer would brring more error e proppagation thaan the latterr. In [5,6], frame-based fr d UEP schem mes are propposed.. . Scalabilitty Layer baased. Thee scalable viideo codingg (SVC) in H.264 H proviides a full sccalability inncluding spaatial, tem mporal and Signal S to Nooise Ratio (SNR) ( scalaability. A SVC stream may consist of manny layers with w one basse layer, annd multiple enhancement layers. The base layer l 1.

(12) which provides basic video quality is much more important than enhancement layers. The higher enhancement layers which depend on the lower enhancement layers are less important. In [1,6], scalability layer based UEP methods are proposed, which allocate different channel rate for different layers according their importance.. . Data Partitioning based. There are three different partition types used in data partition mode of H.264: Partition A for Header information Partition, Partition B for Intra Partition and Partition C for Inter Partition. Among them, Partition A is the most important and Partition C is the least important. Although Partition C is less important than Partition B, the volume of bit stream in Partition C is far more than that in Partition B. Besides, different inter macroblocks in Partition C may have different importance, but there is no further partition for them in current H.264 standard, which means the error protection degree for all the data in Partition C are the same. In [7], an UEP method based on data partitioning for H.264 is proposed. It’s partition method subdivides inter-coded macroblocks in Partition C into several subtypes according to the effect to error propagation.. . Slice-based. Flexible Macroblock Ordering (FMO) is a tool included in the H.264 standard to partition the entire frame into many slice groups. Because the slice groups will be transported separately, we can assign UEP to these slice groups according to their importance, we call slice groups as “SG” in the following of this thesis. In [8,9], data classification schemes with the H.264 explicit FMO mode are proposed as shown in Figure 1.3. They estimate which macroblock would produce more distortion than others, and assign those macroblocks to the same SG with more protection. 1.

(13) Figure 1.3 Macroblock classifications (a) without FMO, (b) dispersed map, (c) original frame of Foreman, (d) explicit FMO mode [9]. Generally, error concealment is well performed in dispersed FMO mode. In [10], a new FMO mode is proposed. They combined dispersed FMO mode and box-out FMO mode, named Explicit Spiral-Interleaved (ESI) FMO mode, as shown in Figure 1.4.. Figure 1.4 ESI mode (a) combination of the dispersed mode and box-out mode (b) Dispersed mode (c) box-out mode [10]. In [11], the importance of each macroblock is first determined called “important factor”. Assume two SGs are used with SG 1 more important than SG 2. The macroblock with highest impact factor will be moved to SG 1. However, since this 1.

(14) action will influence the neighboring macroblocks of the just moved macroblock, the importance of the neighboring macroblocks must be updated accordingly. The recalculation of macroblock importance and macroblock assignment are performed repeatedly till an optimal solution is found. This approach needs a great deal of computation complexity. In [12] they used three SGs and compared the impact factors with two thresholds, Th and Tl, as follows, but the two thresholds are not easy to be determined. z. If impact factor >= Th : high importance SG. z. If impact factor < Th and >= Tl : medium importance SG. z. If impact factor < Tl : low importance SG. For slice-based UEP, the importance of each macroblock will be estimated first. Some researchers determined the importance of a macroblock according to its location [13]. The macroblocks located in the center of the frame are more important that those located on the frame boundaries. Region of Interest (ROI) based UEP [2,10] emphasizes the regions which human favors, therefore, macroblocks located in the foreground regions are more important. In [11], they considered the three parameters: (1) the macroblock’s number of bits, (2) the distortion of the coded macroblock with respect to the original picture, and (3) the distortion of the macroblock if it is lost and concealed only using the surrounding macroblocks. In [12], they considered the reconstructed distortion (that is, quantization error) only.. After what slice for each macroblock is belonged to have been decided, the encoding process can encode bit stream according to the FMO table now and the encoded bit stream will consist of separate slices. The issue then would be how to distribute the available FEC among these slices with different importance. In [14], the expected 1.

(15) length of error propagation (ELEP) is used to describe the effect of packet loss on the decoded video quality in order to achieve optimal FEC assignment. Initially, each slice has the same number of FEC packets, and then processes a while loop with different number of FEC packets until the minimal distortion calculated using ELEP is achieved. Finally, the number of FEC packets is how much protection we will assign.. In [15], Shih proposed Converged Motion Estimation (CME) to achieve a better perForemance of UEP. The CME changes the general concept of motion estimation. Instead of choosing the macroblock with the least Mean Square Error (MSE) as reference for prediction, CME chooses the macroblock which is more important as reference if the increase in MSE is within an acceptable range. To realize the concept above, it modified the definition of MSE by including penalty to the pixels that is referred to. The pixels located inside more protected area will be assigned with less penalty, while inside less protected area will have more penalty. The disadvantage for CME is that it might cause extra bit rate because the macroblocks selected is not the one with minimal MSE.. In the thesis, an R-D Optimized UEP method is proposed. We had adopted the ROPE method, proposed by Rose [16] for macroblock distortion estimation in order to determine the importance of each macroblock and adopted a modified dispersed FMO mode, proposed by Shih [15], for classifying macroblocks into slice groups. And then, an R-D optimized FEC allocation algorithm is proposed to efficiently protect the more important slice groups than others. Besides we modified Converged Motion Estimation (CME) by using All-Zero Block (AZB) algorithm [17] idea to select macroblock located in the important SG for reference. We also take account wireless modulation in to our UEP method. The rest of this thesis is organized as follows. 1.

(16) Chapter 2 gives the introduction to related works of End-to-End distortion, Unequal Error Protection, Forward Error Correction, All-Zero Block and Wireless Modulation. Our proposed scheme is discussed in chapter 3 and the experimental results are shown in chapter 4. The conclusion is given in the last chapter.. 1.

(17) Chapter 2 Related Works Related Works In this Chapter, some precious works relate to this thesis are presented.. 2.1. End-to End Distortion A number of methods have been proposed to estimate end-to-end distortion. The problem was originally considered for optimizing intra-inter decisions to combat temporal error propagation. In [16], the authors suggest a recursive optimal per-pixel estimate (ROPE) for optimal intra/inter mode selection. In ROPE, the expected distortion for any pixel is calculated recursively as follows. Let. denote the. original pixel value at location i in frame n and. the reconstruction of the same. pixel at the decoder. The expected MSE distortion. of the pixel at that location can. then be written as 2. At the encoder, the value. is known and the value. (1). is a random variable. So the. expected distortion at each location can be determined by calculating the first and second moment of the random variable. .. Assume the encoder uses full pixel motion estimation and frame copy error concealment, each lost pixel is reconstructed by copying the pixel at the predicted location k in the previous frame. The predicted location is determined by the median motion vector of the three macroblocks that are nearest to the lost pixel i. Assume a Bernoulli-independent [18] packet loss model where the probability that any packet is 1.

(18) lost is independent of any other packet. Then further assume that the packet lost rate, denoted by p, is available at the encoder. The respective recursion equations of ROPE are as follows and we will reference this concept for determining the importance of macroblocks in this thesis. z. Pixel in an intra-coded macroblock 1. 1. 1 z. (2) (3). 1. Pixel in an inter-coded macroblock 1. ̂ 1. 1. 1. ̂. (4) 1. ̂. 2 ̂ (5). 2.2. Macroblock Assignment The straightest method for macroblock assignment the importance of macroblocks is to sort and then choose the more important ones to be assigned to the SG of high importance. But doing in this way make important macroblocks connected. If one SG was lost, the error concealment would be ineffective because the macroblocks around the lost macroblock are also lost as shown in Figure 2.1.. 1.

(19) Figure 2.1 Connected macroblocks would cause the error concealment ineffective. The macroblock assignment in [15] adopts the dispersed mode with two SGs (say slice 1 and slice 2) first and then adopt K-means clustering algorithm [19] to further split the two SGs into more SGs according to the importance of macroblocks. For example in Figure 2.2, it classify macroblocks into three different levels of importance and six SGs will be generated, three SGs comes from slice 1 (all even macroblocks) and the other three from slice 2 (all odd macroblocks). It is obvious to notice that any macroblock belonging to slice 11, 12, and 13 must have adjacent macroblocks coming from slice 21, 22, as well as 23, and vice versa. Since no contiguous macroblocks will be assigned to the some SG, the error concealment could be effective. In this thesis, we use this method to allocate macroblock into SGs.. Figure 2.2 Macroblock assignment based on dispersed mode 1.

(20) 2.3. Forward error correction Forward error correction (FEC) is a channel coding technique used to recover data from packet losses in the transmission. The type of FEC used depends on the requirements of the application and the nature of the channel. The most commonly studied erasure codes are based on RS codes [13], which have good erasure correcting properties. In this work, we consider RS codes, but the basic framework could easily be applied to other codes.. An RS code is represented as RS(n, k), where k is the number of source symbols and (n-k) is the number of parity symbols. The protection capability of an RS code depends on the block size and the code rate. The block length, n, is usually determined based on the end-to-end system delay constraints, and the code rate of an RS(n, k) is defined as k/n. An RS(n, k) decoder can correct up to (n-k)/2 errors or up to (n-k) erasures, regardless of which symbols are lost. The channel errors in the wired link are typically in the form of packet erasures, so an RS(n, k) code applied across packets can correct up to (n-k) lost packets. Thus, the block unrecoverable probability, , (i.e., the probability that at least one of the original k packets is in error) is. 1. 1. (6).

(21) 2.4. Motion Estimation with AZB In H.264 standard, motion estimation takes the most computation cost in the encoding process. All-Zero Block (AZB) detection is a way to early terminate motion estimation and speed up the encoding. After DCT transform and quantization, it may produce many all zero blocks, where all the coefficients in the blocks are zero. If it can find any AZB before DCT transform and quantization, then motion estimation can be terminated for current macroblock and save time for non-necessary calculation. The detection of AZB is done by checking if the SAD value of a macroblock is inside threshold. If it is inside the threshold, then the coefficients must be all-zero after DCT transform and quantization. Based on the characteristic of DCT transform, Sousa [20] proposed a condition for AZB detection in H.263. Moon [21] analysis the characteristic of DCT transform and quantization in H.264 and modified Sousa’s condition for H.264 as follows. 2 4·. %6,0. (7). The % and f denote the modular operator and a constant ranging from 0 to 2. .. The M(QP%6,0) is the quantization coefficient value. Moon has modified this condition by inserting the threshold with different r values to classify SAD into four modes 2 · where. %6, 2. for. 0,1,2. (8). and T(0) < T(1) < T(2). The four modes defined by Moon are listed Table 2.1, where SADmin is the current minimum SAD value. 1.

(22) Modes. Conditions. The corresponding zero frequency components. M0. SADmin < T(0). r = 0,1,2. M1. T(0) < SADmin < T(1). r = 1,2. M2. T(1) ≤ SADmin ≤ T(2). r=2. M3. T(2) ≤ SADmin. None. Table 2.1 SAD and Threshold for Moon’s condition [21]. Figure 2.3 shows the threshold of T(0), T(1) and T(2) which will detect the AZB in region M0, M1 and M2. From this figure, it can easily tell that the most AZB are located in M0, M1 and M2.. Figure 2.3 AZB threshold [21]. By Table 2.1, Sousa’s algorithm can detect AZB in M0 and the quantized coefficients corresponding to r = 1 and 2 in M1 are already guaranteed to be zero. Moon wants to find more AZB in a special condition for r = 0 in M1. He had proposed another 1.

(23) condition. 0. hs 0,3 2. 1,2. ,. |X |. λ 2. 0. (9). |X |. Wang bring up new condition [17] that will satisfied all r = 0 and r = 1 that will have value zero after quantization in 4x4 macroblock. ,. 1,. 2,. 3. (10). Three threshold Th1, Th2 and Th3 had defined Thi where 1≦ i ≦ 3. 1,1 2. 1. 1 2. 0,1 2. 2. 2 , 1 2. 2 , ,. 3. 2 ,. ,. 2. (12). ,. 0,0. (13) (14). ,. where. (11). ,. According to Figure 2.4, it divided a 4x4 block into four regions: A1, A2, A3 and A4 defined as follow, A1. x, y |x. 0,3, y. 0,3 ,. A2. x, y |x. 0,3, y. 1,2. A3. x, y |x. 1,2, y. 1,2 ,. A4. x, y |x. 1,2, y. 0,3. 0. 1. 2. 3. 0. A1. A2. A2. A1. 1. A4. A3. A3. A4. 2. A4. A3. A3. A4. 3. A1. A2. A2. A1. Figure 2.4 Region division in a 4x4 block [17] 1. (15).

(24) The Table 2.2 describes which condition will be use for different position (u,v). Condition. Position (u,v). ,. /. (1,1). ,. /. (1,3). ,. /. (3,1). ,. /. (3,3). ,. /. (0,1),(2,1). ,. /. (0,3),(2,3). ,. /. (1,0),(1,2). ,. /. (3,0),(3,2). ,. (0,0),(0,2),(2,0),(2,2). Table 2.2 Sufficient condition for detection zero quantized DCT coefficients in a 4x4 block [17]. In this thesis, the idea of AZB algorithm is incorporated into the Converged Motion Estimation (CME). Instead of early terminating motion estimation, we use AZB to avoid the problem of increase in source bit rate for CME.. 2.5. Wireless Modulation The Institute of Electrical and Electronics Engineers, Inc. (IEEE) had defined 802.11a standards [22], for transfer data over wireless network. Inside the standard, it specifies an orthogonal frequency division multiplexing (OFDM) technology for modified data into different modulation. The OFDM physical layer (PHY) has been proposed and was defined in standard. OFDM is characterized by splitting packet from a high data rate data stream into a number of low data rate steams and several modulation modes 1.

(25) can be used. It had defined eight modes in Table 2.3. Depending on each modulation, code rate and data rate it can support data rate from 6 to 54Mbps. In this thesis, we take all the wireless modulation modes into consideration to design the RD-Optimized algorithm for channel rate allocation for wireless network.. Mode Modulation. Code Rate. Data Rate. Bps. 1. BPSK. 1/2. 6 Mbps. 3. 2. BPSK. 3/4. 9 Mbps. 4.5. 3. QPSK. 1/2. 12 Mbps. 6. 4. QPSK. 3/4. 18 Mbps. 9. 5. 16-QAM. 1/2. 24 Mbps. 12. 6. 16-QAM. 3/4. 36 Mbps. 18. 7. 64-QAM. 2/3. 48 Mbps. 24. 8. 64-QAM. 3/4. 54 Mbps. 27. Table 2.3 Eight PHY modes of the IEEE 802.11a PHY [22]. 1.

(26) Chapter 3 Proposed UEP Method Proposed UEP Method In the chapter, we describe our proposed UEP method in detail. In Figure 3.1, the source encoder encodes the bit stream first. After encoding, the macroblock classification module using ROPE algorithm to estimate distortions for each macroblock and treat them as impact factors. After getting the impact factors of macroblocks we adopt Shih’s method in the macroblock assignment module to generate FMO table. Then, the source encoder uses FMO table as input to encode with Converged Motion Estimation (CME) by using modified All-Zero Block motion estimation (CME-AZB) for improve the efficiency of the UEP scheme. Finally, RD Optimized Channel Rate Allocation uses the expect distortion generated by Macroblock Classification module to allocate the channel rates. And then transports the compressed video with FEC.. Figure 3.1 Flow Chart 1.

(27) 3.1. Macroblock Classification Scheme and Assignment This section describes how the macroblock classification module and macroblock assignment module are preformed in our approach. In a frame, different macroblocks have different importance. Our proposed scheme will regard those macroblocks which will cause a great distortion if they were lost as more important macroblocks and give more protection than others. In macroblock classification module, we adopt ROPE end-to-end distortion algorithm [16] (see section 2.1) to calculate the expect distortion. We exploit this result to estimate the importance of macroblocks, called impact factor (IF). Specifically, the IF of a macroblock i in frame n is defined as follows. , where. (16). is defined in equation (1). The inter-coding notation above assume as that pixel i is predicted from pixel j in the previous frame. The above recursions are performed at the encoder in order to calculate the expected distortion at the decoder.. After IF is determined for each macroblock in a frame, we modified Shih’s Dispersed and K-means cluster algorithm [15] (see section 2.2) in our macroblock assignment module for assigning macroblocks into SGs. The advantage for using modified dispersed FMO is to avoid connected macroblocks lost simultaneously. This will make error concealment more efficiency. K-means clustering algorithm is used in Shih’s method for grouping macroblocks with similar importance together. We simply replace the definition of macroblock importance in Shih’s method by our Impact Factor calculate by equation (16).. 1.

(28) 3.2. Converged Motion Estimation with AZB In this section, a Converged Motion Estimation with All-Zero Block detection approach (Called CME-AZB) is proposed to make the UEP more effective. Since the more important macroblocks are typically protected more than less important macroblocks. The use of FEC would be efficient if the more important macroblocks are converged to only few macroblocks. Based on this idea, [15] has proposed a method called Converged Motion Estimation (CME) which tries to modify motion estimation to achieve a skewed macroblock reference pattern. Instead of choosing the macroblock with minimal SAD as reference in motion estimation the macroblock located in highly protected area is chosen even if there is an increase in SAD. However, since the increase in SAD might cause the increase in source bit rate, resulting in reduced bit rate available for error correction code.. To solve this problem, we modify the CME approach by choosing the macroblocks which have quantized coefficients equal to that of the macroblock with minimal SAD. This implies that we only choose macroblock which will not increase the bit rate to be the candidates, among them the one located in the most protected area is selected. However, finding out all the candidate macroblocks imposes a heavy computation overhead in motion estimate because DCT transform and quantization must be performed for every macroblock in the search window of motion estimation (if full search is used). To cope with this problem, we adopt the concept of AZB algorithm. AZB algorithm is originally designed for early termination of motion estimation because it can use the SAD of a macroblock to predict whether the quantized coefficients of this macroblock are all zeros or not. If this macroblock is an All-Zero Block (or AZB block), then motion estimation is terminated for the current. 1.

(29) macroblock. Instead of terminating the motion estimation we modify the AZB algorithm by putting this AZB block into the candidate set and continue motion estimating for searching other AZB block. After all the possible candidates (i.e., All-Zero Block), instead of choosing minimum SAD block, CME-AZB select one with the most pixel located in more protection.. Figure 3.2(a) shows an example of AZB candidates. On frame n, the macroblock in macroblock in red indicates the current macroblock for motion estimation and frame n-1 is the reference frame. Due to too many AZB candidates, we only show four instances that the quantized DCT coefficients are predicted as all zeros, where the macroblock in red, dark-blue, green and light-blue has SAD equal to (38, 43, 58 and 56 respectively). Assume the FMO generated from frame n-1 are shown in Figure 3.2 (b) where the slice group 1 (in purple color) is more important than slice group 2 (in the white color). That is, slice group 1 will get more protection than slice group 2. In this case, the candidate which has the most pixels located in slice group 1 will be selected. That is, that one in light-blue color which has SAD equal to 56 will be selected, although the SAD for red box is 38 which is the smallest one.. 1.

(30) (a). (b) Figure 3.2 AZB candidates. Figure 3.3, shows the number of AZB candidates in different block modes for each frame in Foreman Sequence with QP=28 and search range=32. It is observed that there are 50 to 250 AZB candidates in difference block modes. This implies that there are many choices for the proposed CME-AZB method to select for higher protection without the penalty of increase in bit rate.. Numbers of AZBs. 300. 4X4. 250. 4X8. 200 8X4. 150 100. 8X8. 50. 8X16. 0. 16X8 0. 10. 20. 30. 40 50 60 70 Frame Number. 80. 90 100. Figure 3.3 Number of AZB candidates in Foreman. 1.

(31) 3.3. RD Optimized Channel Rate Allocation Channel rate allocation method is used to determine the amount of FEC assigned to each SG. To utilize FEC efficiently, a rate-distortion optimized algorithm is proposed here and the symbols used are summarized in Table 3.1. protection level for frame n in SG s sth SG group of frame n Total size for frame n in SG s The source size for frame n in SG s SG number in frame Macroblock for frame n in pixel j N. The total size for GOP Number of frames in GOP Number of SGs in frame The maximum bit rate summation diction for entire GOP Distortion for SG group in frame n SG s Distortion for macroblock n in pixel j Summation bit rate for entire GOP. Table 3.1 The symbol table for rate distortion estimation. Since our algorithm organizes SG by grouping together macroblocks that have similar impact factors, we use the same protection level for all the macroblocks belonging to the same SG. Let n, and. ,. denote the protection level assigned to. , the sth SG of frame. denote the parameters of the RS code associated with this protection 1.

(32) level. Since the RS block length is usually determined based on the end-to-end system delay constraints, we assume it is constant (say N) over the entire GOP. So, we have for all the possible n and s in a GOP.. In order to take into account the importance of each SG, more RS packets (i.e., higher protection level) are allocated to SGs carrying important information and less to the / } for. rest. Our channel rate allocation strategy is to optimize the code rates {. each SG in order to minimize the distortion for a given overall transmission rate. Here we present a rate-distortion optimized solution that seeks the best protection level for each SG. Our objective is to seek for the vector of RS coding parameters, ,…,. ,…,. ,…. , that minimizes the expected end-to-end distortion of. the corresponding GOP, where s is the desired number of SGs for each frame and is the number of frames in a GOP. The expected distortion. of a GOP is. given by. (17) ∑. where and the. for all the macroblocks. . However, since. 1. and. is the probability that. ,. has protection level. packet loss rate p in the equations (7) must be substituted with. where. belonging to. , , the. below: (18). is not correctly decoded by the RS decoder,. is the actual channel loss probability. The channel rate allocation problem is. formulated as: min. (19). where. (20). 1.

(33) The above constrained minimization problems (19) and (20) are naturally recast in the standard Lagrange formulation as: (21). By appropriately choosing Lagrange multiplier λ , the above problem (19) can be solved within a convex-hull approximation by solving (21). The search for an appropriate choice of λ can be carried out by the bisection algorithm or a fast convex search technique which is not discussed here. The optimal K then leads to the optimal rate distribution between source and FEC rates.. To solve this equation, we had adopted dynamic program (DP) with the conditions in equation (22) and (23). Equation (22) means that K of slice i must be less than or equal to that of slice j with i < j, assuming that slices are numbered in a descending orders of their important. Since the lost important SG will cause high distortion than the others, we assign more protection (i.e., smaller k) to important SG. Equation (23) means that earlier frames in GOP are more important than the later ones, so we assign more protection on earlier fame in GOP. For the first SG, it calculates J is calculated for all the combination of K. Then for rest of SGs only the J for possible combinations of K that meet the according to condition. , ,. 1~ 1~. (22) (23). To analysis the perForemance of dynamic program, we had use Foreman, Coastguard and Stefan for getting running results. The Table 3.2 shows the execution time for calculating all the combination of K with 105 frames and 6 slices.. 1.

(34) Execution time (s). Foreman. Coastguard. Stefan. 116.466. 116.640. 127.809. Table 3.2 Execute time for Lagrange. When video stream transfer over error-prone wireless network, it will cause packet loss. As discuss earlier, the 802.11a standard had defined eight PHY modes. Assume the sender can receive Signal to Noise Ratio (SNR) from receiver, it can use SNR to determine the status of the network and then select the best modulation for transmitting video data. In [23,24], it will calculate the packet loss rate according by SNR and data rate. We calculated all eight PHY modes and use the packet loss rate for calculate the minimum J. After all combination has been calculated, it can decide which PHY mode has the best K combination. The calculation for bit packet loss rate was shown below where s is SNR value and M is M-ary (M=4, 16, and 64) which depend on Quadrature Amplitude Modulation (QAM): 1. (24). is defiend as follow:. where. 1 where. ·. √. 1. (25). √. is defined as follow:. √. 2· 1. 1 √. ·. 3 1. ·. (26). The Q is defended as 1 √2. (27). The 4-ary QAM and Quadrature Phase Shift Keying (QPSK) modulation are identical. For Binary Phase Shift Keying (BPSK) modulation, the bit error probability is defined 1.

(35) as follow: √2. (28). For calculate minimum J for all the combination of K, we will calculate all the combination of the modulation and use the one with the minimum J: min. 1. 8. (29). These can be calculated from the corresponding packet size and BER. Since equations (24)(28) are calculated probability for bits, we have to summation them together so we can have packet loss rate for bytes where L is the length of bytes: 1. 1. (30). The probability that the packet with L-byte data payload is successfully transmitted within the R retransmission limit under PHY mode m, we replace. as. where i is the number of modulation mode:. ,. ,. 1 1. where. ,. (31). 1 1. ,. is the ACK packet error probability, ,. probability. For calculate Ji, we use replace ,. (32). ,. is the data packet error. ,. as. and the maximum data rate for each modulation: as 1. ,. 1. which is calculate by : (33).

(36) Chapter 4 Experimental Results Experimental Results In the chapter, we compare the proposed method with the “Raster scan” and the “Dispersed” macroblock assignments, both of them are the FMO mode included in the standard. The parameters of our experimental environment are set as follows: z. Test sequence: Foreman, Coastguard, Stefan. z. Group of Picture (GOP): I P P P P ……. z. GOP size: 15 frames. z. Frame rate: 30 fps. z. Frame format: QCIF (176 x 144 pixels). z. SG number: 6 SGs. z. Packet size: 16 bytes. z. Overall bit rate: 340Kbps (Foreman), 360Kbps (Coastguard), 980Kbps (Stefan). The video sequences are encoded and decoded using JM 12.1 [25] where the code of motion estimation is modified to support the proposed CME-AZB. In our experiments, “Raster scan” and “Dispersed” are Equal Error Protection (EEP) with the same overall bit rate. We add the “Shih’s UEP method without CME” and “Shih’s UEP method with CME” for comparison Shih’s UEP method reorders the macroblocks according to their defined impact factors (IF) and then sequentially assign to six SGs of unequal size using dispersed and k-means cluster as described in Section 2.2. Channel rate allocation in Shih’s method is done simply by assign RS depend on number of frames and slices linear proportional to the important of slices. Table 4.1 summarizes the 1.

(37) difference between Shih’s method and the proposed approach.. Macroblock Classification Scheme Macroblock Assignment Method Forward Error Correction. Shih’s UEP method. Proposed UEP method. Shih’s distortion estimation method. Use ROPE formula to calculate expect distortion and treated it as Impact Factor. Shih’s method. Shih’s method. Assign RS by UEP depend on number of frames and slices linear proportional to the. RD Optimized algorithm with Lagrange multiplier. important of slices Motion Estimation. Add penalty on higher slice, may increase bitrates. CME-AZB. Table 4.1 Compare Proposed UEP with Shih’s UEP method. “Proposed UEP method without CME-AZB” and “Proposed UEP method with CME-AZB” are both implements in our experiments for comparison. The measured average PSNR results of Forman, Coastguard, and Stefan with packet loss rate 10% and 20% are shown in Figure 4.1 (a)(b), Figure 4.2(a)(b), and Figure 4.3(a)(b), respectively. The efficiency of using packet loss rate 20% is better than that of using 10% because EEP can not handle high packet loss rate. We can see that Dispersed is better than Raster scan because error concealment is more efficient in Dispersed than Raster scan. Shih’s UEP method performs well than Raster Scan and Dispersed mode in all cases, showing that unequal protection can achieve a better result than EEP. We can see that the proposed UEP method without CME-AZB is better than Shih’s UEP method without CME, it showing that the proposed RD optimized rate distortion channel allocation can improve the quality indeed. The PSNR of Shih’s UEP with. 1.

(38) 37 35 33 31 PSNR. 29 27 25 23 21 19 17 1. 11. 21. 31. 41. 51. 61. 71. 81. 91. 101. 71. 81. 91. 101. Frame Number. (a) 29 27 25. PSNR. 23 21 19 17 15 13 11 1. 11. 21. 31. 41. 51. 61. Frame Number. (b) Figure 4.1 The PSNR when packet loss rate reach (a) 10% (b) 20% in Foreman. CME higher than that if CME is not used showing that CME-AZB will further improve CME method indeed. It is show that our proposed UEP without CME-AZB performs even better than Shih’s UEP with CME. That proves again the superiority of our RD optimized channel rate allocation algorithm. The proposed UEP with. 1.

(39) CME-AZB performs better than proposed UEP without CME-AZB, showing that CME-AZB did take effect in the experiments. Because there are no motion vectors in I frame, CME and CME-AZB can not be performed in I frame Thus CME and CME-AZB can not improve UEP method at I frame in a GOP as shown in each result.. 37 35 33 31 PSNR. 29 27 25 23 21 19 17 1. 11. 21. 31. 41. 51. 61. 71. 81. 91. 101. 71. 81. 91. 101. Frame Number (a). 31. PSNR. 26. 21. 16. 11 1. 11. 21. 31. 41. 51. 61. Frame Number (b) Figure 4.2 The PSNR when packet loss rate reach (a) 10% (b) 20% in Coastguard. 1.

(40) 33. PSNR. 28. 23. 18. 13 1. 11. 21. 31. 41. 51. 61. 71. 81. 91. 101. 71. 81. 91. 101. Frame Number (a) 31 29 27. PSNR. 25 23 21 19 17 15 13 1. 11. 21. 31. 41. 51. 61. Frame Number (b) Figure 4.3 The PSNR when packet loss rate reach (a) 10% (b) 20% in Stefan. Figure 4.4 shows the PSNR result as a function of packet loss rates, ranging from 5% to 25% for Foreman sequence. When the packet loss rate reaches 5%, EEP is better than UEP because the protection rate is high enough to handle most channel errors, but UEP would not be able to in the situation. With the increasing of packet loss rate, the efficiency of UEP is more obvious especially in Coastguard because UEP protect 1.

(41) the important data with more RS code. When packet loss rate reach 25%, the measured PSNR of UEP is almost equal to EEP because the overall bit rate is not high enough to handle the high packet loss rate no matter wheatear UEP or EEP is used. From the above figures, it is clear that our proposed method could maintain the quality.. Given the same SNR, different PHY modes tend to result in different packet loss rate. We use SNR as the input and let encoder choose the best PHY mode according the algorithm present in section 3.3. The experiment results are shown in Figure 4.5 where the SNR is shows that the packet loss rates are around 5% to 25% when SNR between 5 to 30 dBi. We added two tests which are “Normal PHY mode” and “Proposed PHY mode method” as Method 1 and Method 2 with our proposed UEP algorithm with CME-AZB. “Normal PHY mode” uses the fixed PHY mode 4 and “Proposed PHY mode method” will use the proposed method select the best PHY mode according to it SNR. From the above figure, it shows that if it PHY mode is changed dynamically according to SNR, and then it will improve its video quality.. 1.

(42) Raster Scan. 35 Dispersed. PSNR. 30 25 20 15 5. 10. 15 20 Packet Loss Rate. 25. Shih's UEP method without CME Shih's UEP method with CME Proposed UEP method without CME‐AZB Proposed UEP method with CME‐AZB. (a) Raster Scan. 35 Dispersed. PSNR. 30 25 20 15 5. 10. 15 20 Packet Loss Rate. 25. Shih's UEP method without CME Shih's UEP method with CME Proposed UEP method without CME‐AZB Proposed UEP method with CME‐AZB. (b) Raster Scan. 33. Dispersed. PSNR. 28 23 18 13 5. 10. 15 20 Packet Loss Rate. 25. Shih's UEP method without CME Shih's UEP method with CME Proposed UEP method without CME‐AZB Proposed UEP method with CME‐AZB. (c) Figure 4.4 The average PSNR when packet loss rate equal to 5% ~ 25% in video stream (a) Foreman (b) Coastguard (c) Stefan. 1.

(43) 34. PSNR. 29 24 19. Method 1 Method 2. 14 5. 10. 15. 20. 25. 30. SNR (a). 35. PSNR. 30 25 20. Method 1 Method 2. 15 5. 10. 15. 20. 25. 30. SNR (b). 31. PSNR. 26 21 16. Method 1 Method 2. 11 5. 10. 15. 20. 25. 30. SNR (c) Figure 4.5 The average PSNR according by SNR between 5% ~ 30% in video streams (a) Foreman (b) Coastguard (c) Stefan 1.

(44) Chapter 5 Conclusion Conclusion In this thesis, a Rate-Distortion (R-D) optimized UEP scheme for video transmission over error-prone wireless networks scheme has been presented. Our UEP scheme is incorporated with Flexible Macroblock Ordering (FMO) by which the more important macroblocks in a picture are grouped into the same slice groups and protected more than others. We had adopted the ROPE method, proposed by Rose [16], for macroblock distortion estimation in order to determine the importance of each macroblock. Also we adopted a modified dispersed FMO mode proposed by Shih [15] for classifying macroblocks into slice groups. And then, an RD optimized FEC allocation algorithm is proposed to efficiently protect the more important slice groups than others. In addition, a modified Converged Motion Estimation (CME) based on the concept of all-zero-block (AZB) detection algorithm and wireless modulation selection is proposed to further improve the proposed UEP. We can see that the proposed UEP method without CME-AZB is better than Shih’s UEP method without CME, it showing that the proposed RD optimized rate distortion channel allocation can improve the quality indeed. The PSNR of Shih’s UEP with CME higher than that if CME is not used showing that CME-AZB will further improve CME method indeed. It is show that our proposed UEP without CME-AZB performs even better than Shih’s UEP with CME. That proves again the superiority of our RD optimized channel rate allocation algorithm. By given the same SNR, different PHY modes tend to result in different packet loss rate. We use SNR as the input and let encoder choose the best PHY mode according the algorithm. The proposed method select the best PHY mode according to it SNR. The experiment shows that if it PHY mode is 1.

(45) changed dynamically according to SNR, and then it will improve its video quality. The simulation results show that the proposed UEP method improves the quality of the decoded video for H.264 streams, and converged motion estimation with AZB can further improve it.. 1.

(46) Reference [1] Bin-Feng Hung and Chung-Lin Huang, “Content-Based FGS Coding Mode Determination for Video Streaming Over Wireless Networks”, IEEE Journal on Selected Areas in Communications, Volume 21, No. 10, December 2003 [2] Klaus Stuhlmuller, Niko Farber, Michael Link, Bernd Girod, “Analysis of Video Transmission over Lossy Channels”, IEEE Journal on Selected Areas in Communications, Volume 18, No. 6, June 2000 [3] X.K.Yang, Ce Zhu, Z. G. Li, X. Lin, G. N. Feng, S. Wu, and N. Ling, “Unequal loss protection for robust transmission of motion compensated video over the internet”, Signal Processing: Image Communication, Volume 18, Issue. 3, Page 157-167, March 2003 [4] Zhihai He, Hongkai Xiong, “Transmission Distortion Analysis for Real-Time Video Encoding and Streaming Over Wireless Networks”, IEEE Transaction on Circuits and Systems for Video Technology, Volume 16, No. 9, September 2006 [5] Francois Marx, and Joumana Farah, “A novel approach to achieve unequal error protection for video transmission over 3G wireless networks”, Signal Processing: Image Communication, Volume 19, Issue. 4, Page 313-323, April 2004 [6] Tao Fang and Lap-Pui Chau, “GOP-Based Channel Rate Allocation Using Genetic Algorithm for Scalable Video Streaming Over Error-Prone Networks”, IEEE Transactions on Image Processing, Volume 15, No. 6, June 2006 [7] Song Xiao, Chengke Wu, Jianchao Du, Yadong Yang, “Reliable Transmission of H.264 Video over Wireless Network”, IEEE Advanced Information Networking and Applications, 2006. AINA 2006. 20th International Conference, Volume 2, Page 18-20, April 2006 1.

(47) [8] Sio Kei Im, A. J. Pearmain, “An Efficient Data Classification Scheme with the H.264 Flexible Macroblock Ordering”, IEEE Information, Communication and Signal Processing, 2005 Fifth International Conference, Page 1135-1139, Dec. 2005 [9] Thomos, N. Argyropoulos, S. Boulgouris, N.V. Strintzis, M.G. “Error-Resilient Transmission of h.264/AVC Streams Using Flexible Macroblock Ordering”, IEEE Integration of Knowledge, Semantics and Digital Media Technology, 2005. EWIMT 2005. The 2nd European Workshop, Page 183-189, Dec. 2005 [10] Barak Katz, Shlomo Greenberg, Nathalie Yarkoni, Nathan Blaunstien, and Ran Giladi, “New Error-Resilient Scheme Based on FMO and Dynamic Redundant Slices Allocation for Wireless Video Transmission”, IEEE Transactions on Broadcasting, Volume 53, No. 1, March 2007 [11] Yu Wang, Tao Fang, Lap-Pui Chau, Kim-Hui Yap, “Two-dimensional Channel Rate Allocation for SVC over Error-prone Channel”, IEEE Circuits and Systems, 2006. ISCAS 2006. Proceedings. 2006 IEEE International Symposium, Page 21-24, May 2006 [12] Iain E. G. Richardson, “H.264 and MPEG-4 Video Compression”, John Wiley & Sons, UK, 2003 [13] Wicker, S. B. and Bhargava, V. K., ed., “Reed-Solomon Codes and Their Applications”, 1983 [14] Justin Goshi, Alexander E. Mohr, Richard E. Ladner, Eve A. Riskin, and Alan Lippman, “Unequal Loss Protection for H.263 Compressed Video”, IEEE Transactions on Circuits and Systems for Video Technology, Volume 15, No. 3, March 2005. 1.

(48) [15] Jhong-Yu Shih, “An Unequal Error Protection Scheme based on FMO and Converged Motion Estimation”, National Chiao-Tung University, Master Thesis, June 2007 [16] R. Zhang, S. L. Regunathan and K. Rose, “Video coding with optimal inter/intra-mode switch for packet loss resilience”, IEEE Journal on Selected Areas in Communications, Volume 18, Page 966-976, June 2000 [17] Hanli Wang, Sam Kwong and Chi-Wah Kok, “Efficient Prediction Algorithm of Integer DCT Coefficients for H.264/AVC Optimization”, IEEE Transactions on Circuits and Systems for Video Technology, Volume. 16, No. 4, April 2006 [18] Carla Raffaelli and Michele Savi, “Traffic PerForemance of Buffered Multi-Stage Optical Packet Switch”, Photonics in Switching, 2006. PS '06. International Conference on 16-18, Page 1-3, October 2006 [19] S. Z. Selim, M. A. Ismail, “K-MEANS-Type algorithms”, IEEE Transactions PAMI-6, Page 81-87, 1984 [20] L. A. Sousa, “General method for eliminating redundant computations in video coding”, Electron. Lett., Volume 36, Issue 4, Page 306-307, February 2000 [21] Yong Ho Moon, Gyu Yeong Kim, and Jae Ho Kim, “An Improved Early Detection Algorithm for All-Zero Block in H.264 Video Encoding”, IEEE Transactions on Circuits and Systems for Video Technology, Volume 15, No. 8, August 2005 [22] VOCAL Technologies Ltd., “IEEE 802.11a standard White Paper”, 1999 [23] Daji Qiao, Sunghyun Choi and Kang G. Shin, “Goodput Analysis and Link Adaptation for IEEE 802.11a Wireless LANs”, IEEE Transaction on Mobile Computing, Volume 1, No. 4, October 2002. 1.

(49) [24] Yuan Zhang, Wen Gao, Yan Lu, Qingming Huang and Debin Zhao, “Joint Source-Channel Rate-Distortion for H.264 Video Coding Over Error-Prone Network”, IEEE Transactions on Multimedia, Volume 9, No. 3, April 2007 [25] H.264/AVC Reference Software, http://iphome.hhi.de/suehring/tml. 1.

(50)