Intra Mode Switching (IMS)

For the intra-coded w-MBs, we need to change the intra modes to x the wrong reference problem since the intra prediction is performed in the spatial domain. The neighboring samples of the already encoded blocks are used as the prediction reference. Thus, when we replace parts of the background picture with the foreground pixels, the MBs around the borders may have visual artifacts due to the newly inserted samples. Without drift error correction, the distortion propagates spatially all over the whole frame via the intra prediction process in a raster scanning order. A straightforward re nement approach is to apply the R-D optimized (RDO) mode decision to nd the best intra mode from the available pixels and then re-encode new residue.

To reduce complexity we propose an intra mode switching (IMS) technique for the

intra-Sec 3.4. Reduced Frame Memory Transcoding (RFMT)

Figure 3.8: The Wrong Intra Reference Problem within a Macroblock Depending on the Intra Modes

FG

BG 1

3 2

4 5

7 6

Figure 3.9: The Relative Position of Each Case in the Intra Mode Switching Technique

switching approach selects the best mode from the more probable intra prediction modes.

Each block within a MB could be classi ed according to the intra modes as shown in Figure 3.8. Similarly, the mode of the w-block should be re ned while the modes of p-blocks are unchanged. For the w-blocks, the IMS is performed according to the relative position with respect to the foreground pictures as shown in Figure 3.9. To speed up the IMS process, a table lookup method is used to select the new intra mode according to the original intra mode and the relative position. Table 3.2 and Table 3.3 enumerate the IMS selection exhaustively.

Table 3.2: The Cases of the Intra4 Mode Switching

Case Corresponding 4x4 block Original Mode* Switched Mode*

1 Left column of blocks 1, 2, 4, 5, 6, 8 0

* 0: Intra_4x4_Vertical 1: Intra_4x4_Horizontal

2:Intra_4x4_DC 3: Intra_4x4_Diagonal_Down_Left

4: Intra_4x4_Diagonal_Down_Right 5: Intra_4x4_Vertical_Right 6: Intra_4x4_Horizontal_Down 7:Intra_4x4_Vertical_Left 8: Intra_4x4_Horizontal_Up

Table 3.3: The Cases of the Intra16 Mode Switching Case Original Mode* Switched Mode*

1,6 1, 2, 3 1

2 3 2

3,5 0, 2, 3 1

* 0: Intra_16x16_Vertical 1: Intra_16x16_Horizontal 2: Intra_16x16_DC 3: Intra_16x16_Plane

In the following, we will interpret the intra re nement process mathematically. Take Figure 3.10 for example where the block xwis the w-block since it is predicted from the to-be-covered block xc and the other blocks including xp;0, xp;1, xp;2, xp;3, xp;4, xp;5 are the p-blocks. In this dissertation, we use the symbol "bar" above the variables to denote the reconstructed values after decoding, the symbol "prime" to denote the re ned values after transcoding, and the symbol

"hat" to denote the values after transcoding without performing re nement. In the original bitstream, the decoded signal of the block xw can be represented in Eq.(3.1) where rw is the original residue.

xw = rw + IP1(xc) (3.1)

Sec 3.4. Reduced Frame Memory Transcoding (RFMT)

Figure 3.10: An Example of the Intra Prediction Chain

If we do not re ne the block x_w at all, the decoded signal of the block x_w at client side is obtained from the original residue rw and the reference block yc from foreground region as represented in Eq.(3.2).

c

xw = rw+ IP₁(yc) = xw IP₁(xc) + IP₁(yc) = xw+ IP₁(yc xc) (3.2)

, where the symbol IP₁(y_c x_c)indicates the serious mismatch of wrong reference propagated via intra prediction since the blocks in the foreground and background region are quite differ-ent in terms of pixel values. There fore, to improve the video quality, we re ne the residue of w-block after the fully decoding and the intra mode switching. With the re ned intra mode, we compute the new residue and coded block patterns. It should be noted that only the recon-structed values xw is used as the original video is unavailable. The re nement of the block xw

is de ned by

r_w⁰ = x_w IP₂(x_r) = r_w + IP₁(x_c) IP₂(x_r) (3.3)

, where the symbols IP1(x_c) and IP2(x_r) denote intra prediction from the reference pixels x_c and xr by using the original mode and the new mode respectively. The symbol rw is the decoded residue extracted from the source bitstream. Then, the re ned residue is re-quantized and de-quantized as

r_w⁰ = P_d P_e r_w⁰ = P_d P_e [r_w + IP₁(x_c) IP₂(x_r)]

= P_d P_e r_w+ P_d P_e [IP₁(x_c) IP₂(x_r)]

= r_w+ IP₁(x_c) IP₂(x_r) + e_w (3.4)

,where the symbol ewdenotes the quantization error of [IP1(x_c) IP₂(x_r)]. Lastly, the recon-structed data of the block xw is shown in Eq.(3.5)

x⁰_w = rw0+ IP2(xr) = rw+ IP1(xc) + ew = xw+ ew (3.5)

Therefore, the symbol e_w also denotes the re nement error due to the additional quantization process with respect to the original reconstructed value xw. As compared to Eq.(3.2), we can signi cantly reduce the mismatch from IP1(y_c x_c)to ewbecause and [IP1(x_c) IP₂(x_r)]is smaller than IP₁(y_c x_c)and the quantization error of [IP₁(x_c) IP₂(x_r)]is much smaller than [IP1(x_c) IP₂(x_r)]itself.

For the p-blocks, If we do not re ne the residue of the block xp;0, the decoded signal of the block x_p;0 at client side is obtained from the original residue r_p;0 and the reference block x⁰_w

Sec 3.4. Reduced Frame Memory Transcoding (RFMT)

instead of x_was represented in Eq.(3.6)

d

x_p;0 = r_p;0+ IP₁ x⁰_w = x_p;0 IP₁(x_w) + IP₁ x⁰_w

= x_w+ IP₁ x⁰_w x_w = x_w+ IP₁(e_w) (3.6)

Therefore, the re nement of w-blocks may incur drift error that is ampli ed and propagated to the subsequent p-blocks by the intra prediction process. In order to alleviate the error propaga-tion, we re-calculate the coef cients of p-blocks with the re ned samples of w-blocks and the original intra modes as shown in Eq.(3.7), where we assume the block xp;0 is the intra-coded p-block that uses the decoded data of the block xw as prediction.

r_p;0⁰ = x_p;0 IP₁ x⁰_w = r_p;0+ IP₁(x_w) IP₁ x⁰_w

= r_p;0+ IP₁ x_w x⁰_w = r_p;0+ IP₁(e_w) (3.7)

Similarly, the re ned residue should be re-quantized and de-quantized as represented in Eq.(3.8) where the symbol ep;0denotes the drift error in the block xp;0and is identical to the quantization error of intra prediction of re nement error ew in the n-th 4 4 block.

x⁰_p;0 = rp;00+ IP1 x⁰_w = Pd Pe rp;0+ Pd Pe IP1(ew) + IP1 x⁰_w

= r_p;0+ IP₁(e_w) + e_p;0+ IP₁ x⁰_w

= x_p;0 IP₁(x_w) + IP₁ x⁰_w + IP₁(e_w) + e_p;0

= x_p;0+ IP₁ x⁰_w x_w+ e_w + e_p;0 = x_p;0+ e_p;0 (3.8)

Similarly, the next p-block can be derived:

x⁰_p;m+1 = x_p;m+1+ e_p;m+1;

where ep;m+1 = P_d P_e IP (e_m) IP (e_m); m = 0; 1; 2; 3; :::::: (3.9)

The generalized projection theory says that consecutive projections onto two non-convex sets will reach a trap point beyond which future projections do not change the results [33]. After several iterations of error correction, the drift error can not be further compensated. Therefore, we only perform error correction to the p-blocks within intra-coded w-MB rather than all the subsequent p-blocks. We observe that error correction for the p-blocks within intra-coded w-MB improves the averaged R-D performance up to 1.5dB. However, error correction for the intra-coded p-MBs has no signi cant quality improvement (less than 0.2dB).