Motion Vector Refinement and Vertical Frequency Detection for Motion Compensated De-interlacing

Motion Vector Refinement and Vertical Frequency Detection for

Motion Compensated De-interlacing

Chin-Chuan Liang, Hung-Shih Lin, *Yu-Cheng Fan, *Arvin Chiang, and Hen-Wai Tsao

Department of Electrical Engineering and Graduate Institute of Electronic Engineering

National Taiwan University, Taipei, Taiwan, 106, R.O.C.

*Department of Electronic Engineering and Graduate Institute of Computer and Communication

Engineering, National Taipei University of Technology, Taipei, Taiwan, 106, R.O.C.

E-mail: [email protected]

This paper proposed a novel motion vector refinement and vertical frequency detection method for motion compensated de-interlacing. Motion compensated de-interlacing is the main trend of the next generation of de-interlacing algorithms. However, it suffers visible artifacts caused by incorrect motion vectors. To improve such imperfect pictures, we adopt interleaved sub-sample pseudo frame SAD, further smoothing on motion vectors, undesired vertical high frequency detection and local area expanded weighting coefficient generation techniques to provide more accurate motion estimation and more efficient artifact detection. In particular, the simulated result is better than previous de-interlacing methods.

1: INTRODUCTIONS

Deinterlacing doubles the vertical-temporal sampling density and aims at removing the first repeated spectrum caused by the interlaced sampling of the video. However, the vertical direction sampling process of interlaced TV signals does not follow the Nyquist sampling theorem. The problem has led to several deinterlacing techniques that have been proposed in the literature [1].

In order to solve all circumstances deinterlacing problems, many linear and non-linear methods have been proposed [2]-[7].

Vertical spatial interpolation exploits the correlation between vertically neighboring samples in a field when interpolating intermediate pixels. Their all-pass temporal frequency response guarantees the absence of motion artifacts. However, defects occur with high vertical frequencies since vertical interpolation cannot discriminate between base-band and repeated spectra. The interpolation error causes blurred pictures and line flicker.

Vertical temporal interpolation deinterlacing method

tries to profit from both high spatial correlation and high temporal correlation. As a consequence, motion artifacts are absent for objects moving horizontally without vertical details. Nevertheless, degradation becomes visible at vertical edges.

Motion adaptive deinterlacing detects the motion areas first, and then performs intra-field deinterlacing in motion areas and inter-field one in stationary areas. Therefore, high-resolution and flicker-free deinterlaced pictures can be realized in both stationary and motion areas. Any erroneous detection will cause artifacts in the form of spots on the picture. This will cause a reduction of resolution in detailed regions and flicker in slow motion regions.

In order to solve the above problems, this paper presents a motion vector refinement and vertical frequency detection for motion compensated (MC) de-interlacing using interleaved sub-sample pseudo frame SAD (sum of absolute difference), further smoothing on motion vectors, undesired vertical high frequency detection and local area expanded weighting coefficient generation techniques. This scheme proposes more accurate motion vectors estimation and more efficient vertical high frequency detection. This method produces deinterlaced pictures with better visual quality, imperceptible artifacts and fewer flickers. The proposed motion compensated deinterlacing method is presented in Section 2. The simulation results and comparisons with other conventional methods are described in Section 3. Finally, a conclusion is given in Section 4.

2: MOTION VECTOR REFINEMENT

AND VERTICAL FREQUENCY

DETECTION

This section presents motion vector refinement and vertical frequency detection for motion compensated de-interlacing. Its operation is described below.

2.1:

At first, we proposed an interleaved sub-sample pseudo frame SAD (sum of absolute difference) method. SAD, the most popular choice for error function calculation, is used as the match criterion in the proposed motion estimation. Unlike the predict coding of MPEG, which calculates SAD on consecutive frames, it is improper for the motion estimation of deinterlacing to calculate absolute differences between current and adjacent fields because of differences in parity (even field vs. odd field). On the other hand, it is relevant to calculate SAD using frame or field differences, defined

in Fig. 1, because the differences are obtained from

fields of the same parity (even field vs. even field or odd field vs. odd field). Whatever frame or field difference is used, it is equivalent to calculating SAD on the vertical sub-sample frame. It is well known that the sub-sample method in predictive coding degrades the correctness of motion vectors. To obtain more accurate motion vectors, both frame difference and field difference are used altogether, regardless of the doubling of the processed pixels. Although three fields are processed, it is similar to calculating SAD on a single frame, referred to as pseudo-frame SAD. In order to reduce the pixels used for pseudo-frame SAD, we calculate differences only on the horizontal interleaved sub-sample pixels. This method is named interleaved sub-sample pseudo-frame SAD.

The MSE (Mean Square Error) results of four types

of SAD are shown in Fig.2, where MSE is calculated on

MC interpolation pictures without error protection. The features of the test sequences are listed in Fig. 2. It can be seen that the MSE of pseudo-frame SAD is much less than that of frame difference SAD, even with the use of the interleaved sub-sample scheme [1].

2.2: FURTHER SMOOTHING ON

MOTION VECTORS

The smoothness of motion vector (MV) in MC deinterlacing strongly affects the consistence of interpolated pictures, especially along the temporal axis.

The smoothing method is presented in Fig.3, where the

variance is calculated as:

y x d

d

D var var

var = + (1)

In areas where object motion can be traced well by full search, the MVs of some blocks may occasionally be incorrect because of background noise or non-integer motion. After smoothing, this incorrect MV can be

fixed. From Fig.3, the associated algorithm is: If

occurrences of dominant MV＞＝6, MV5＝dominant MV. In areas where object motion is complex, the MVs of some blocks may be changed to (0,0) after smoothing. From Fig.3, the associated algorithm is: If the variance

of 3 × 3 MVs ＞ ＝ 5, MV5 ＝ (0,0). In such areas, intra-field interpolation is used instead of MC to provide better consistency [1].

2.3: DETECTION OF UNDESIRED

VERTICAL HIGH FREQUENCY

Fig.4 depicts the detected motion area in the

vertical-temporal frequency domain of a frame-difference-based temporal difference detector. Cases A to C are three specific interlaced video sequences. Case A corresponds to a stationary picture with alternating black & white scanning lines (fy=262.5 c/ph and ft=0 Hz for NTSC). In case A, TD(n-1,n+1) is zero, and inter-field interpolation is used to obtain correct results. Case B corresponds to frames with an order of “black-black-white-white” (fy=0c/ph, ft= 15 Hz for NTSC). In case B, TD(n-1,n+1) is large, and intra-field interpolation is used to obtain error-free results. Case C corresponds to frames with an order of “black-white-black-white” (fy=0c/ph, ft= 30 Hz for NTSC). In case C, TD(n-1,n+1) is zero. Inter-field interpolation is used and causes serious artifacts. Case C shows the limitation of frame difference based temporal difference detection. Although Case C would not appear in a general interlaced video sequence, a fast moving object with fine structure may cause artifacts such as those in Case C when the frame difference is zero. Therefore, another detection should be added to solve this problem. As discussed in [5], the vertical spectra of progressive and interlaced pictures are different. In progressive pictures, vertical frequency may be as high as 240 c/ph. However, vertical frequency in interlaced pictures is limited to 170 c/ph, depending on the capture equipment. Therefore, if the interpolated picture has some component with vertical frequency higher than 170c/ph, it may result from erroneous MC interpolation and is thus undesirable. A vertical correlation detector, as shown in Fig.5, is used to detect this kind of artifact. Similar to slope detection, vertical correlation detection is processed on frame_FI(n,n-1) and frame_FI(n,n+1). Using frame_FI(n,n-1) as an example, if diff_1 and diff_2 are smaller, and diff_3 and diff_4 are larger than the threshold, then VC(x,y,n,n-1) is equal to 1, indicating that the vertical correlation of frame_FI(n,n-1) at Pi(x,y,n) is unreasonable. Otherwise VC(x,y,n,n-1) is

equal to 0, indicating that the vertical correlation of frame_FI(n,n-1) at Pi(x,y,n) is reasonable. The function

of the “threshold” block in Fig.5 is [1]: output = 0 , if input≦ threshold

output = 1 , if input＞ threshold (2)

The final VC of Pi(x, y, n) is:

VC(Pi(x,y,n))=VC(n,n–1)|VC(n,n+1) (3)

2.4: LOCAL AREA EXPANDED

WEIGHTING COEFFICIENT

GENERATION

The human eye is more sensitive to line flicker or flicker in an area than to flicker of one pixel. Also, it is less sensitive to artifacts of one pixel. Therefore, while determining the coefficient for soft-switching, we will not only consider the difference detector (TD) and vertical correlation (VC) of the interpolated pixel but also those of the pixels around it. Fig 6(a) demonstrates the local area to be considered while interpolating Pi(x,

y, n). The coefficient α can be determined as [1]:

α = f(TD,VC of the local area) (4) In the design of the proposed method, only seven pixels located on the crossed dash lines in Fig. 6(b) are used. Let TD and VC of the seven pixels be TD(i) and VC(i), where i=0……6; the coefficient α can be calculated by the following equations:

TD3 = 3th_max(TD(i), i =0……6) (5)

∑

where TD3 is the third maximum among the seven TDs.

VCnum isempirically calculated in (6). αtd is evaluated

in (7) using predetermined THL and THH. Finally, α

can be obtained by (8).

3: SIMULATION RESULTS AND

ANALYSES

This section presents the simulation results and the associated analyses of the proposed motion compensated de-interlacing method.

The four test sequences of 352×288 progressive

pictures are shown in Fig.7(a)-(d). Quantitatively, the

PSNRs of our deinterlacing scheme for ten CIF sequences are compared with those of several other

methods, as listed in TABLE I. The proposed method

adopts several standard video sequences (frame 1 to frame 100). Other experimental results cite Ref. [6]. Obviously, our method exhibits better results of PSNR performance than the other methods, even by 3dB in an extreme case. Qualitatively, it can be seen that high quality and artifact-free results are obtained.

4: CONCLUSION

We proposed a motion vector refinement and vertical frequency detection for motion compensated de-interlacing in this paper. Several key techniques are presented. Interleaved sub-sample pseudo-frame SAD method estimates the high accuracy bidirectional motion vector. The MV smoothing method ensures consistence

in the MV fields. Undesired vertical high frequency

detection is used to prevent the artifacts caused by incorrect motion vectors. Finally, the local area expanded weighting generator is adopted to tradeoff flicker and visible artifacts. As a result, the proposed algorithm can provide higher de-interlacing picture quality.

REFERENCES

[1] C. C. Liang, Motion Compensated Deinterlacing for

Video Line Doubler Algorithm Research, Master Thesis,

Taiwan, 2002.

[2] Y. L. Chang, S. F. Lin, and L. G. Chen, “Extended intelligent edge-based line average with its implementation and test method,” 2004 IEEE International Symposium on Circuits and Systems, vol. 2, pp: 341-344, May 2004.

[3] M. K. Park, M. G. Kang, K. Nam, and S. G. Oh, “New edge dependent deinterlacing algorithm based on horizontal edge pattern,” IEEE Transactions on Consumer Electronics, vol. 49, no. 4, pp: 1508-1512, Nov. 2003.

[4] H. S. Oh, Y. Kim, Y. Y. Jung, A. W. Morales, and S. J. Ko, “Spatio-temporal edge-based median filtering for deinterlacing,” 2000 IEEE International Conference on Consumer Electronics, pp: 52-53, June 2000.

[5] K. Sugiyama, and H. Nakamura, “A method of deinterlacing with motion compensated interpolation,” IEEE Trans. on Consumer Electronics, vol. 45, no. 3, pp: 611-616, August 2000.

[6] S. F. Lin, Y. L. Chang, and L. G. Chen, “Motion adaptive interpolation with horizontal motion detection for deinterlacing,” IEEE Trans. Consumer Electron., vol. 49, no. 4, pp. 1256–1265, Nov. 2003.

[7] W. R. Sung, E. K. Kang, and J. S. Choi, “Adaptive motion estimation for motion compensated interframe interpolation,” IEEE trans. Consumer Electronics, vol. 45, no. 3 August 1999.

TABLE I

AVERAGE PSNR(DB)OF DIFFERENT METHODS

FOR VARIOUS CIFVIDEO SEQUENCES

Sequence Merged Bilinear ELA 2-Field 4-Field Proposed

Coastguard 22.59 27.84 26.32 29.32 30.45 33.61 Container 31.01 28.14 25.04 40.27 37.81 34.02 Dancer 23.19 36.15 33.41 31.94 34.06 37.20 Hall Monitor 30.61 30.68 27.75 35.34 39.14 40.78 Mobile 17.83 24.92 23.99 26.20 25.83 28.24 Table 23.89 27.32 25.85 31.55 35.11 36.10 0 TD3 ≦ THL αtd = 1 TD3 ≧ THH (7) L H L TH TH TH TD − − 3 _else αtd－VCnum×0.2 αtd≧VCnum×0.2 0 else (8) α=

Fig. 5. Vertical correlation detector.

Original pixel interpolated pixel Vertical Line y-3 y-2 y-1 y y+1 frame_F1(n,n-1) diff_3 diff_4 diff_2 diff_1 threshold threshold threshold threshold abs abs abs abs diff_1 diff_2 diff_3 diff_4 VC

Fig. 6. Processing window for weighting generation.

Vertical Line y-3 y-2 y-1 y y+1 y+2 y+3 x-2 x-1 x Vertical Line y-3 y-2 y-1 y y+1 y+2 y+3 x-2 x-1 x edge n (a) (b)n (c) (d) (a) (b)

Fig. 7. 352×288 originally progressive sequences: (a) “Coastguard,” (b) “Container,” (c) “Mobile,” (d) “Table Tennis.”

field difference frame difference original sampled pixel pre-deinterlaced pixel interpolated pixel

Bi-directional motion vector D = (dx,dy)

n-1 n n+1 field number y-3 y-2 y-1 y y+1 y+2 y+3 dy dx 0 100 200 300 1 2 3 4 frame difference SAD field difference SAD pseudo frame SAD interleaved sub-sample pseudo frame SAD coastguard 0-100 object moving foreman 100-200 camera pazoning stefan 60-160 fast move table tennis 0-100 zooming sequence frame number feature

Fig. 2 MSE results of four different types of SAD.

Vertical Line

Fig. 1 Bi-directional motion vector.

t y case A t y case B t y case C ft (Hz) fy (c/ph) 60 30 -30 －262.5 262.5 525

Fig. 4. Detected motion area of temporal difference detector

MV1 MV2 MV3 MV4 MV5 MV6 MV7 MV8 MV9