Four field local motion compensated de-interlacing

FOUR FIELD LOCAL MOTION COMPENSATED DE-INTERLACING

Yu-Lin Chang, Ping-Hao Wu, Shyh-Feng Lin, and Liang-Gee Chen

DSP/IC Design Lab,

Graduate Institute of Electronics Engineering and Department of Electrical Engineering

National Taiwan University, Taipei, Taiwan

ABSTRACT

In this paper, a four field local motion compensated de-interlacing method is proposed to solve the occlusion and true motion vec-tor problems of motion compensated de-interlacing. The proposed de-interlacing method consists of four field motion estimation, 5+5 taps ELA, and intra/MC macroblock mode decision. True motion vectors can be obtained by the four field motion estimation and its matching criterion is extended by the characteristic of interlace-scanned field. The mode decision could detect an occlusive mac-roblock and choose its best mode. Hardware implementation has been considered. The results show that the proposed de-interlacing method can produce images compensated by great majority of the original fields with little intra-field interpolation, and the subjec-tive view results are jag-free and flicker-free high quality images.

1. INTRODUCTION

De-interlacing becomes an important technique today because of the popularity of the progressive TV devices. While traditional SDTV video sequences are all in interlaced scanned format, a con-version must be done if we want to display the SDTV video on the progressive scanned devices(such as PDPs, LCD-TVs). Take DVcam for example, the format recorded in an NTSC DVcam is SDTV format - 720×480i, 30FPS. If the video sequence is di-rectly shown on the progressive devices, defects like line crawling, edge-flicker and jagged effect would appear and make the viewer uncomfortable. So a de-interlacing process is necessary for these applications.

Two low complexity de-interlacing methods, BOB and Weave [1], are commonly adopted in the software approach. BOB is an intra-field interpolation method, which uses a single field to re-construct one progressive frame. However, the vertical resolution is halved and the image is blurred. Weave is a simple inter-field de-interlacing method directly combining two interlaced fields into one progressive frame. However, the line-crawling effect will arise in the motion areas.

Motion adaptive methods are commonly used in consumer electronics. The motion adaptive de-interlacing combines the ad-vantages of both intra-field de-interlacing and inter-field deinter-lacing. Lin proposed a four field horizontal motion adaptive de-interlacing method [2] to reduce the hardware complexity, and it achieves much more accurate motion detection. However, there are still some motion detection error on these motion adaptive methods.

Some motion compensated techniques have been presented to improve image quality. Hilman [3] and Haan [4] proposed motion-compensated frame-rate conversion algorithms with interpolation

to reduce the 3:2 pull-down artifacts. Schutten and Haan [5] pro-posed an object-based true-motion estimation algorithm to elimi-nate the 2-3 pull-down sequences. Chang [6] used global motion estimation to convert the interlaced video with camera panning, zooming, rotating to progressive one. However, these motion com-pensated de-interlacing methods involve too many sub-pixel inter-polation or intra-field interinter-polation and the true image resolution cannot be kept.

In this paper, a four field local motion compensated deinterlac-ing system is proposed. The overview of intra-field and inter-field de-interlacing will be described in section 2 to clarify their advan-tages and disadvanadvan-tages. The proposed method will be discussed in section 3. And the simulation results will be shown in section 4. At last, the conclusion remarks the proposed system.

2. OVERVIEW

Fundamentally, de-interlacing could be characterized into three categories: intra-field de-interlacing, motion adaptive de-interlacing, and motion-compensated de-interlacing.

2.1. Intra-field De-interlacing

The most cost-efficient method is intra-field de-interlacing using the current field. It is widely used in software implementations. The most common method of the intra-field de-interlacing is line doubling, which is used in small LCD panels. However, the jagged effect arises on the oblique line, and flicker will be seen in the fine texture. ELA and DCDi(tm) of the Faroudja Deinterlacer is the most popular algorithm and hardware in this category. ELA is a kind of directional edge interpolation [7]. Three pixels in the previous and the next scan lines are separately referenced to de-termine the obvious edge. This method can eliminate the blur-ring effect of the bilinear interpolation and produce sharp/straight edges. Although intra-field interpolation is very cost-efficient and needs only one line buffer, the resolution of the picture is half of the original. In addition, some defects may occur when an object only exists in the same-parity field.

2.2. Motion Adaptive De-interlacing

Inter-field interpolation means merging two fields into one frame, so it needs one-field buffer. The video quality is better than that of the intra-field interpolation in the static area, but line-crawling effect arises in the motion area. The motion adaptive deinterlac-ing combines the advantages of both intra-field de-interlacdeinterlac-ing and inter-field de-interlacing. It detects the motion areas first, and then

9

Field Buffer 4 Field Motion Estimation Field Buffer Field Buffer Backward Field Current Field Forward Field F.Forward Field 5+5 taps ELA 4 Field Motion Compensation Field In Frame Out Intra/MC MB Mode Decision

Fig. 1. Block Diagram of the Proposed Method

adopts intra-field interlacing in motion areas and inter-field de-interlacing in static areas. This method could preserve video res-olution in static areas and eliminate the line-crawling effect in motion areas. However, many commercial motion adaptive de-interlacing produce blurred images because their motion detection methods detect too many excess motions in a field.

2.3. Motion-Compensated De-interlacing

The motion-compensated method utilizes motion estimation to find the most similar blocks in the neighboring fields and calculates its motion vectors. Then a new field is reconstructed from the neigh-boring fields. Block matching needs extra internal buffer to lo-cate the current macroblock and reference macroblock. Most of the commercial motion compensated de-interlacing often produce blurred images because the using of the sub-pixel motion compen-sation or the mode decision tends to go into the intra-field interpo-lation. Some of them use overlapping block motion compensation (OBMC) to reduce the block effect. However, the OBMC often blurs the block edge to achieve smooth blocks. The main advan-tage of motion compensated de-interlacing should be kept - the resolution. The true image to be compensated with the original field should be produced by integer pixel motion compensation because the sub-pixel motion compensation or excess intra-field interpolation would only lower the image resolution.

3. PROPOSED METHODS

3.1. Proposed Motion Compensated De-interlacing System

The proposed motion compensated de-interlacing is called the four field local motion compensated de-interlacing. As presented in

MV₁ MV₂ -MV₂ F.Forward Field n-2 Forward Field n-1 Current Field n Backward Field n+1 Time SAD₁ SAD₂ Search Window Backward-Forward MacroBlock Current-F.Forward MacroBlock

Fig. 2. Four Field Motion Estimation

Fig.1. It consists of three part: the four field motion estimation /compensation, the 5+5 taps ELA, and the intra/MC Macroblock mode decision.

The four field motion compensated de-interlacing system needs three field buffer to store the data of the forward-forward field, the forward field, and the current field. After the image data storage is done, the four fields - the forward-forward field, the forward field, the current field and the backward field - would be fed into the ”four field motion estimation” module concurrently. The four field motion estimation module will estimate the motion of each non-overlapping macroblock and output their motion information and macroblock information into the ”four field motion compen-sation” module. The four field motion compensation module will compensate the forward field into a new compensated field accord-ing to the motion information provided by the four field motion es-timation module. On the other hand, the image data of the current field is sent to the 5+5 taps ELA module, the module is the same as the enhanced ELA in the [2] which can produce intra-field in-terpolated fields. Finally, the intra-inin-terpolated field, the motion compensated field, and the current field are sent to the ”Intra/MC macroblock mode decision” module. The intra/MC macroblock mode decision module will decide which image data is going to be as the final output. The four field motion estimation and compen-sation, the intra/MC macroblock mode decision are the two main parts of this de-interlacing system. We will describe it below.

3.2. Four Field Motion Estimation/Compensation

The motion estimation methods vary from every motion compen-sated de-interlacing method. The similarity of all these motion estimation methods is to find a true motion vector for each block. It is different from the motion estimation methods for video cod-ing. The motion estimation method for video coding is to find the best-matching block which would produce the minimal residue. However, the motion estimation we need in de-interlacing is to find the best-matching block which would suit with its neighbor-ing block. That is to say, true motion vectors on the true image are needed for the de-interlacing application, not the motion vectors that produce the minimal residues.

The proposed four field motion estimation is shown in Fig.2. The difference between this motion estimation and traditional mo-tion estimamo-tion is that it calculates two same parity field SAD val-ues. TheSAD1 in Fig.2 denotes the block matching SAD value of the current field to the forward-forward field motion estima-tion.SAD1is calculated by accumulating the absolute difference of the current macroblock and the macroblock candidate in the search window of the forward-forward field, which can be written as following equation: SAD1k = X i,j∈MB |f(i + MVkx, j + MVky, n − 2) − f(i, j, n)| (1) wheref(i, j, n) denotes the luminance intensity at the (i, j) location of thenth field, and the MVkxandMVkymeans the x component and the y component of thekth motion vector candi-dateMV_kin the search window of the forward-forward field.

The second SAD value is namedSAD₂. It is calculated by ac-cumulating the absolute difference between the macroblock where the MV2 indicated to in the forward field and the macroblock where−MV₂ indicated to in the backward field. SAD₂ can be shown by SAD2k= X i,j∈MB (|f(i + 1 2MVkx, j + 12MVky, n − 1) −f(i − 1₂MVkx, j − 1₂MVky, n + 1)|) (2)

where the meanings off(i, j, n), MV_kx, andMV_kyare the same asSAD1kin eq.1.

In order to find a motion vector, the matching criterion of the four field motion estimation is minimizing the value ofSAD1+

SAD2, as described in eq.3. And the motion vectorMVkis the

found motion vector. k = Arg( min

k∈SW(SAD1k+ SAD2k)) (3)

If onlySAD₁is considered to be the matching criterion, the pixel values of the forward field and the backward field are not considered. And there will be lots of error if we divide the found motion vector by two and produce the motion compensated field according to the divided motion vector. If onlySAD2is consid-ered to be the matching criterion, the image data of the current field is not considered. While a complex-texture macroblock is on a uniform background, the motion vector would be wrong. So SAD1 andSAD2must be considered together to obtain the cor-rect motion information. Because theSAD1+ SAD2matching criterion involves four field image data, this motion estimation is called ”four field motion estimation”.

As the motion vectors of the macroblocks are gotten, the mo-tion compensamo-tion is done to compensate the current field from the forward field. In the Fig.2, ifMV1is the found motion vector and MV2= 1₂MV1, the macroblock data indicated byMV2would be compensated to the position of the current macroblock. After the compensation of the image data in every macroblock is done, the compensated field would be output to the intra/MC macroblock mode decision module. The four field motion compensation can find true motion vectors due to its ”same parity” property. Com-mon motion compensated de-interlacing may use forward field and

Table 1. Intra/MC Macroblock Mode Decision

Macroblock Macroblock Found Mode SAD Diverse Amount Motion Vector Decision

Small Large Large Intra

Small Large Small MC

Small Small X MC

Large X X Intra

current field for the motion estimation, but the pixel data are some-what different from different parity fields. The proposed four field motion estimation can consider the even field and odd field image data at the same time and get more suitable motion vectors.

3.3. Intra/MC Macroblock Mode Decision

The intra/MC macroblock mode decision module receives the im-age data of the intra-field interpolated field, the current field, the motion compensated field, and the motion vector field and then de-cide which one is going to output. The three criterions for mode decisions are macroblock SAD, macroblock diverse amount, and the found motion vector of each macroblock.

Here the macroblock SAD means the SAD value between the motion compensated macroblock and the current macroblock. This SAD value indicates the accuracy of the four field motion estima-tion. The macroblock diverse amount is a kind of counter which counts how many pixel differences between the motion compen-sated field and the current field in a macroblock are larger than a certain threshold. This macroblock diverse amount may help to identify which macroblock give people uncomfortable feelings. The found motion vector is used with the macroblock SAD and the macroblock diverse amount to help to determine whether a mac-roblock is a special same parity field pattern or not.

The decision rule is described in Table 1. The column ”Mode Decision” is the chosen mode of a macroblock. If the macroblock SAD and the macroblock diverse amount are smaller than certain thresholds, then the mode is the motion compensation mode. Else if the macroblock SAD is larger than a certain threshold, then the intra-field interpolation mode is chosen. There are two spe-cial cases while the macroblock SAD is small and the macroblock diverse amount is large. This means the four field motion estima-tion find its best matching macroblock, however, the found mac-roblocks are quite different from the top field and the bottom field. There are two situations in this condition, one is that the motion estimation found wrong matching macroblock due to the repeated texture. The motion vector would be large in this situation. The other is that the texture is too complex that it exactly divides the macroblock into two parts, and each part only appears on only one parity field. The first situation should be processed with intra-field interpolation, and the second situation should be processed with the motion compensation to preserve the texture resolution.

According to the decision of the mode decision part, the final output is chosen from the current field, the motion compensated results or the intra-field interpolation results and produce the most comfortable image on the progressive devices.

4. SIMULATION RESULTS 4.1. Analysis of the Four Field M.E.

We use 16×8 as the macroblock size, and the search range is from −63 to +64. The motion estimation algorithm is the full search

(a) (b) (c) (d)

Fig. 3. Subjective View Results for Skating (a) Weave (b) Four

Field Motion Adaptive De-interlacing (c) Common Motion Com-pensated De-interlacing (d) Proposed De-interlacing

(a) (b) (c) (d)

Fig. 4. Subjective View Results for Flag (a) Weave (b) Four Field

Motion Adaptive De-interlacing (c) Common Motion Compen-sated De-interlacing (d) Proposed De-interlacing

algorithm which can be implemented by many fast full search al-gorithms to reduce the hardware complexity. The average memory bandwidth of the proposed local motion estimation with CCIR601 720×240, 60 fps format is almost 50MB/sec, and the three field memory need 8Mbits to put all the YUV data in. As the above analysis, it is possible for this method to be implemented on the hardware using the 0.18µm technology today.

4.2. Subjective View

Statistical performance for de-interlacing is hard to compute due to the lack of high resolution progressive video sequence. The common method to test a de-interlacing algorithm is to test the algorithm with several special video sequence and see whether the defects would disappear or not.

There are two interlace-scanned video test sequences which are commonly used in professional TV quality testing. The first one is the ”Skating” sequence, and the second one is the ”Flag” sequence.

The ”Skating” video sequence mainly tests two features of a de-interlacing method. One is the oblique thin line, the other is the repeated texture on the clothes of the skater. The subjective view of Skating is shown in Fig.3. There are many defects like line-crawling, jagged edge in the result of the weave method in Fig.3(a). The four field motion adaptive de-interlacing method and the common motion compensated de-interlacing method in Fig.3(b)(c) could reduce these defects, however, there are many jagged edges on the thin line and the texture, this is caused by the intra-field interpolation. The proposed method in Fig.3(d) com-pensates the thin line with its real part, and the line become straight without blurring or jagging.

The main feature of the ”Flag” video sequence is the edge-preservation. The flag flutters in the breeze, and the motion of the flag is non-rigid body motion. The four field motion adaptive de-interlacing method would detect the whole flag as moving and adapt intra-field interpolation to the flag. As shown in Fig.4(b), if the intra-field interpolation couldn’t support lines with low slope,

jagged edges would still appear. The result of the common motion compensated de-interlacing method is shown in Fig.4(c), jagged edges appear in the high contrast edges. The result of the proposed method is shown in Fig.4(d), the edges are smooth and the stars on the flag are very clear. The result of the four field motion adap-tive de-interlacing in this ”Flag” video sequence is very similar to the industrial Faroudja interlacer. However, the Faroudja De-interlacer is a motion adaptive de-De-interlacer with finite-tap intra-field interpolation, and there would be still some defects while the image is too complex or the line slopes are very low. Here the pro-posed motion compensation can preserve the image resolution and compensate the real image onto the current field.

The proposed de-interlacing system not only achieves high quality on these test sequences but also solves many interlace-scanned problems in many common video sequences. The same parity fieldSAD1+ SAD2matching criterion helps a lot on mo-tion estimamo-tion.

5. CONCLUSION

Traditional motion compensated de-interlacing methods tried to find the true motion vector of the macroblocks in a field. However, the matching criterion should be changed to adapt with the char-acteristic of the interlace-scanned video. The proposed four field local motion compensated de-interlacing system finds the true mo-tion through the proposed four field momo-tion estimamo-tion and uses the motion information to make a intra/MC macroblock mode de-cision. The simulation results of the proposed method show higher quality, jag-free and flicker-free images thanks to the accuracy of the four field motion estimation. The complexity of the algorithm is not too high and is suitable for hardware implementation.

6. REFERENCES

[1] Jed Deame, “Motion compensated de-interlacing: The key to the digital video transition,” SMPTE 141st Technical Confer-ence in NY, Nov. 1999.

[2] Shyh-Feng Lin, Yu-Lin Chang, and Liang-Gee Chen, “Motion adaptive de-interlacing by horizontal motion detection and en-hanced ela processing,” IEEE International Symposium on Circuits and Systems, May 2003.

[3] Kevin Hilman, Hyun Wook Park, and Yongmin Kim, “Using motion-compensated frame-rate conversion for the correction of 3:2 pulldown artifacts in video sequences,” IEEE Trans-actions on Circuits Syst. Video Technol., vol. 10, no. 6, Sept. 2000.

[4] Gerard de Haan and E.B. Bellers, “De-interlacing of video data,” IEEE Transactions on Consumer Electronics, vol. 43, pp. 819–825, Aug. 1997.

[5] R.J. Schutten and G. de Haan, “Real-time 2-3 pull-down elim-ination applying motion estimation/compensation in a pro-grammable device,” IEEE Transactions on Consumer Elec-tronics, vol. 44, pp. 930–938, Aug. 1998.

[6] Yu-Lin Chang, Ching-Yeh Chen, Shyh-Feng Lin, and Liang-Gee Chen, “Motion compensated de-interlacing with adaptive global motion estimation and compensation,” IEEE Confer-ence on Image Processing, Sept. 2003.

[7] T. Doyle, “Interlaced to sequential conversion for edtv appli-cations,” pp. 412–430, Feb. 1988.