FOUR FIELD VARIABLE BLOCK SIZE MOTION COMPENSATED ADAPTIVE
DE-INTERLACING
Yu-Lin Chang, Ching-Yeh Chen, Shyh-Feng Lin, and Liang-Gee Chen
DSP/IC Design Lab,
Graduate Institute of Electronics Engineering,
National Taiwan University, Taipei, Taiwan
ABSTRACT
A four field variable block size motion compensated adaptive de-interlacing method is proposed to improve the accuracy of the mo-tion vectors and lower the occlusions of momo-tion compensated de-interlacing. The proposed de-interlacing method consists of vari-able block size motion estimation/compensation with four field SAD, interlaced block mode decision, and new block modes. The variable block size motion estimation/compensation improve the accuracy of the motion vectors, especially for spatially-periodic patterns. The new block modes and the interlaced block mode de-cision make block dede-cisions more precisely, and special patterns that motion compensation cannot be compensated are correctly de-interlaced by these two methods. The subjective view shows the improvement of the accuracy of the motion vectors and the cor-rectness of the mode decision.
1. INTRODUCTION
De-interlacing technique is an important part of digital television. It converts interlaced-scanned video sequences into progressive-scanned ones. If an interlaced-progressive-scanned video sequence is directly shown on a progressive-scanned display device, defects like feath-ering, line crawling, edge-flicker and jagged effect would make the viewer uncomfortable, so a good de-interlacing process is neces-sary for these applications.
There are three kinds of interlacing methods: intra-field de-interlacing, motion adaptive de-interlacing (MA), motion compen-sated de-interlacing. The intra-field deinterlacing methods have been broadly used in software and low-level digital televisions. Intra-field de-interlacing uses a single field to reconstruct a pro-gressive frame through intra-field interpolation. The edge-based line average algorithm (ELA) and edge dependent directional in-terpolation (EDDI) are the most common intra-field inin-terpolation algorithms. Zhao [1] proposed some adaptive methods to improve the the edge-directed interpolation algorithms.
Motion adaptive (MA) de-interlacing is the main stream of the de-interlacing method on digital televisions today. The mo-tion adaptive de-interlacing processes the moving areas in a field with intra-field interpolation and the static areas with directly field merging. Lin proposed a four field horizontal motion adaptive de-interlacing method [2] to reduce the hardware complexity, and it achieves much more accurate motion detection. However, there are still some motion detection errors with these motion adaptive methods.
Motion compensated (MC) de-interlacing is the state-of-the-art de-interlacing techinique. Firstly, Nguyen proposed a same
par-ity motion compensated de-interlacing algorithm [3], which finds a best matching block from neighboring fields to compensate with the current block and uses intra-field interpolation for occlusion areas. There are two kinds of methods to improve the accuracy of the motion vector for MC de-interlacing. One is to improve the search method, and the other is to improve the matching criterion. Haan [4] proposed a true motion estimator with 3DRS method to improve the search method of motion estimation in the MC de-interlacing. Chang [5] proposed a new matching criterion - the same parity four field SAD, which utilizes the same parity characteristic of the interlacedscanned video. The four field SAD -can raise the accuracy of the motion vectors and reduce the motion vector errors. However, the accuracy of the motion vectors still has to be improved, and mode decision methods should make correct block mode decision more precisely. Because the more the pixel resolution is kept, the more natural the de-interlaced picture will be.
In this paper, a four field variable block size motion compen-sated adaptive deinterlacing system is proposed. The motivation for this new de-interlacing method will be described in section 2. The problems of MC de-interlacing will be stated. The proposed method will be discussed in section 3. And the simulation results will be shown in section 4. At last, the conclusion remarks the proposed system.
2. MOTIVATION
For MC de-interlacing, there are two kinds of techniques to im-prove the accuracy of the motion estimation - the search method and the matching criterion. There are many search methods for motion estimation, and they have their own characteristics, like fast computational speed, smooth motion vectors, or easy to im-plement. But, the matching criterion of MC de-interlacing is dif-ferent from that of common video coding system. It is not to find a motion vector which makes the residual small after motion compensation, but to find a motion vector which makes the com-pensated block best match with the current block. The four field SAD matching criterion proposed by Chang[5] lacks of the infor-mation from different parity fields. That is to say, if there are some spatially-perioidic patterns moving in a video field sequence, the motion estimation with the four field SAD might not find a correct motion vector. However, this kind of spatially-periodic patterns are very common in the natural video sequences, this problem must be solved for MC de-interlacing. There are two approaches to solve the problem - one is to detect the spatially-periodic pattern, and the other is to change the block size of the motion estimation.
Pe-II - 913
riodic patterns repeat themselves along any direction in a picture. It is hard to find all of them. So the change of the block size should be useful in this problem.
The smaller the block size is, the less the accuracy of the mo-tion vector would be. But, if the block size is enlarged, occlusions appears at object boundaries. The tradeoff between block size and occlusion should be solved to preserve more motion compensated results for MC de-interlacing.
The block modes, and their decision methods are important parts for MC de-interlacing, too. Traditional MC de-interlacing methods adopt two block mode - MC mode or Intra mode. How-ever, MC/Intra mode would fail while de-interlace an object which exists only in single parity field, and this increases the difficulty of the mode decision. And SAD values are often used to do the mode decision, but there are still a lot of miss-detection or false-alarm that degrade the picture quality.
3. PROPOSED METHOD
We proposed a motion compensated adaptive de-interlacing sys-tem to solve the above problems. The variable block size motion estimation with four field SAD is to solve the incorrect of the mo-tion vectors caused by the spatially-periodic patterns and lower the occlusions on the object boundaries. The proposed two new modes - MC/MA mode can improve the picture quality and preserve the correctness of which part of an object should be processed by the intra-field interpolation. The proposed mode decision method is suitable with the two new modes and gets better results.
3.1. The proposed Motion Compensated Adaptive Deinterlac-ing method
The block diagram of the proposed motion compensated adaptive de-interlacing method is shown in Fig. 1. This system needs three field buffers to store the reference field data - the forward-forawrd field, the forward field, and the current field. The four field motion adaptive de-interlacing module uses the MA de-interlacing algo-rithm proposed by Lin[2]. After the four field MC de-interlacing module and the four field MA de-interlacing module processed its data, the MC/MA mode decision part will decide whether a block is going to use the result of MC or MA. The MA mode replaces the intra mode in the traditional MC de-interlacing. After the result is produced, it will be merged with the current field and output to the display device.
3.2. Variable Block Size Motion Estimation with Four Field SAD
As shown in Fig.2, The variable block size motion estimation with four field SAD accmulates three SAD values of a block - SAD1, SAD2, and SAD3 . The variable block size motion estimation
consists of two pass iteration. In the first pass, the motion esti-mation would find a best matching motion vector by SAD1and SAD2under three block sizes. We choose 32×16, 16×8, 8×4 as
the three block sizes. In the second pass, the value of SAD3 is used to determine which block size is going to be the output block size. So there are three kinds of produced block s, as shown in the right side of Fig.2, one 32×16 block, four 16×8 blocks, and three
16×8 blocks with four 8×4 blocks.
In the first pass, SAD1 and SAD2 are defined as following
equations: )LHOG'HOD\ )LHOG'HOD\ )LHOG'HOD\ )LHOG 0RWLRQ $GDSWLYH 'H LQWHUODFLQJ )LHOG 0RWLRQ &RPSHQVDWHG 'HLQWHUODFLQJ 0&0$ 'HFLVLRQ )LHOG,Q %DFNZDUG
)LHOG %DFNZDUG)LHOG &XUUHQW )LHOG )RUZDUG )LHOG ))RUZDUG )LHOG &XUUHQW &RPSHQVDWHG)LHOG&RQWURO 6$',3'&
Fig. 1. Block Diagram of the Proposed Method
09 09 09 ))RUZDUG )LHOG Q )RUZDUG )LHOG Q &XUUHQW )LHOG Q %DFNZDUG )LHOG Q 7LPH 6$' 6$' 6HDUFK:LQGRZ %DFNZDUG)RUZDUG0DFUR%ORFN &XUUHQW))RUZDUG0DFUR%ORFN 6$' %ORFN6L]H[ %ORFN6L]H[ %ORFN6L]H[ Fig. 2. Variable Block Size Motion Estimation with Four Field SAD SAD1k|m= X i,j∈MB |f(i+MVkx, j+MVky, n−2)−f(i, j, n)| (1) SAD2k|m= X i,j∈MB (|f(i + 12M Vkx, j+ 12M Vky, n− 1) −f(i − 12M Vkx, j− 12M Vky, n+ 1)|) (2)
where f(i, j, n) denotes the luminance intensity at the (i, j) location of the nth field, and the MVkxand MVkymeans the x
component and the y component of the kth motion vector can-didate MVk in the search window of the forward-forward field, where m means block size m, m ∈ {32 × 16, 16 × 8, 8 × 4}. SAD1is the SAD value of the current block and the block in the
forward-forward field along MV2with twice MV2length. SAD2
is calculated by accumulating the absolute difference between the macroblock where the MV2indicated to in the forward field and
the macroblock where−MV2indicated to in the backward field.
In order to find a motion vector in the first pass, the match-ing criterion of the four field motion estimation is minimizmatch-ing the value of SAD1+ SAD2, as described in eq.3. And the motion
vector MVkis the found motion vector.
k= Arg( min
k∈SW(SAD1k+ SAD2k)) (3)
The found motion vectors of three block sizes are stored for the second pass block size decision.
In the second pass, the block size will be decided through
SAD3. SAD3 is the SAD value from the current block to the
block in the forward field with the best matching motion vector cal-culated in the first pass motion estimation. Thus, three SAD3with
three block sizes will be produced - SAD3,32x16, SAD3,16x8, and
SAD3,8x4. The purpose of the second pass is to decide the block
size, as the following equation:
b= Arg(min
m (kmSAD3,m)) (4)
where m∈ {32 × 16, 16 × 8, 8 × 4} and kmis a constant for
normalizing the SAD3with different block sizes. The block size
b would be chosen after the second pass calculation. The motion compensation from the forward field will be done according to the block size in the form shown at the right side of Fig.2.
This variable block size motion estimation/compensation can avoid the motion vector errors caused by the spatially-periodic pat-terns and preserve good quality on complex texture areas through smaller block size motion estimation.
3.3. Block Mode Decision
The most suffering defects on video without de-interlacing is the feathering effect. We proposed a method to accumulate the pixels with the feathering effect. If there are many pixels with feather-ing effect in a compensated block, it means the motion compen-sated result may be wrong. So the pixels in that block should be the results of motion adaptive de-interlacing. And the proposed method shown in Fig.3 is called interlaced pixel distortion clas-sification (IPDC). The feathering effect can be detected through IPDC. By IPDC and SAD, the mode decision can determine block mode more accurately.
As shown in Fig.3, there is a matching block in the forward field. And the IPDC calculation in a block is presented as the following pseudo code:
dif f1= abs(fn−1(i + MVx, j+ MVy) − fn(i, j); dif f2= abs(fn(i, j) − fn(i, j + 1);
if ( diff1> th1&& diff2< th2) IPDC++;
M Vx and MVyare the found motion vector. The value of dif f1represents whether a pixel in the compensated block is
sim-ilar to its current block or not, and the value of diff2 shows if
the pixel will be placed in between two similar pixels with cur-rent block. If the pixel in the compensated block is placed in be-tween two different pixels, the possibility for the viewer to suffer the feathering effect is increased, and the IPDC increased. If IPDC is larger than a certain threshold, it means there are too many pixels with feathering effect in the block. So the mode decision module
GLII
!WK
)RUZDUG)LHOG
&XUUHQW)LHOG
GLII
WK
Fig. 3. Interlaced Pixel Distortion Classification should choose the MA mode under large IPDC condition. And an-other condition that the mode decision module would choose the MA mode is the large SAD condition. If SAD1+ SAD2 of the
best matching macroblock is too large, it means the motion estima-tion is incorrect. It is possibly occurred on the object boundaries. Under this condition, the MA mode should be chosen to recover the block with motion adaptive results. And other areas are also processed by MC mode to preserve the pixel resolution as much as possible.
4. SIMULATION RESULTS
The simulation results can be divided into three parts. The first part shows the variable block size motion estimation help to find more accurate motion vectors. The second part demonstrates the picture quality of the mode decision module by IPDC and SAD. The third part brings out the difference between MC/intra mode and MC/MA mode. The test video sequences are flag, ice hockey, and pendulum, which are the common test video sequences used in the de-interlacing method development.
4.1. Variable Block Size Motion Estimation with four field SAD The results in Fig.4 are produced only by motion compensation for the comparison of motion estimation. The search method has been changed to compare their accuracy. The search method 3DRS produces smooth motion vectors, but it cannot correctly determine the motion of non-rigid body, which is shown in Fig.4(b). So the full search method is still needed for complex or non-rigid objects. As shown in Fig.4(c)(d)(e), the full search results vary in different block sizes. For smaller block size(like16 × 8 and 8 × 4, there are more motion vector errors because more blocks would become spatially-periodic patterns, and the motion estimation is prone to be wrong. For larger block size(32 × 16), occlusions and pixels with feathering effect appear on the flag boundaries. However, the variable block size motion estimation with four field SAD adopts proper block size for motion compensation. The result of the vari-able block size motion estimation with four field SAD in Fig.4(f) shows an acceptable result even without intra-field interpolation. 4.2. Block Mode Decision
In Fig.5(a), the traditional block mode decision determined that most of the blocks are MC mode because it only uses the SAD for
(a) (b)
(c) (d)
(e) (f)
Fig. 4. Subjective View Results of flag (a) Directly-Merged (b) Block Size 16x8 3DRS (c) Block Size 8x4 Full Search (d) Block Size 16x8 Full Search (e) Block Size 32x16 Full Search (f) Pro-posed variable block size Full Search
its criterion. The MC mode should not be assigned to the blocks in the middle area of Fig.5(a). As for our mode decision results shown in Fig.5(b), the blocks in the middle area of the picture are chosen as MA mode. The motion adaptive interlacing de-tects these parts as moving parts, then the intra-field interpolation is adopted in these blocks. So the proposed mode decision with SAD and IPDC is better than that with SAD only.
4.3. New Block Modes
The block modes - MC/Intra mode have been replaced by MC/MA mode to ease the burden of the mode decision module. For objects
(a) (b)
Fig. 5. Subjective View Results (a) No IPDC (b) With IPDC
(a) (b)
Fig. 6. Subjective View Results for block modes (a)MC/Intra Mode (b) MC/MA Mode
that only appear in single parity fields, the MA mode is much bet-ter than the Intra mode. As shown in Fig.6(a)(b), the word ”O” and ”K” in the pendulum sequence exist in the different parity fields. The mode decision module is prone to determine the blocks as non-MC mode because of its feathering texture. Traditional meth-ods tried to detect this kind of objects and correct the motion vector and mode, or the result will be the intra-field interpolation result shown in Fig.6(a). If block modes are MC/MA mode, the motion adaptive de-interlacing part will choose the pixels that really need to do intra-field interpolation. And the de-interlacing method with MC/MA mode shows a normal result in Fig.6(b) at these blocks.
5. CONCLUSION
To find accurate motion vectors and to lower the occlusions are two important topics for motion compensated de-interlacing. How-ever, special video patterns, like spatially-periodic patterns, ob-jects only exist in single parity fields, often restrict the perfor-mance of motion compensated de-interlacing. A four field variable block size motion compensated adaptive de-interlacing method is proposed. It consists of the variable block size motion estima-tion/compensation to find more accurate motion vectors, block mode decision with IPDC and SAD to determine the block modes more precisely, and new block mode - MC/MA mode to ease the burden of mode decision and cope with special video patterns. The simulation results of the proposed method show higher quality pic-tures than before. And this method is suitable for advanced digital television to get better de-interlaced pictures.
6. REFERENCES
[1] M. Zhao and G. de Haan, “Intra-field de-interlacing with ad-vanced up-scaling methods,” IEEE International Symposium
on Consumer Electronics, Sept. 2004.
[2] Shyh-Feng Lin, Yu-Lin Chang, and Liang-Gee Chen, “Motion adaptive de-interlacing by horizontal motion detection and en-hanced ela processing,” IEEE International Symposium on
Circuits and Systems, May 2003.
[3] A. Nguyen and E. Dubois, “Spatio-temporal adaptive interlaced-to-progressive conversion,” Signal Processing of
HDTV, IV, E. Dubois and L. Chiariglione, Eds., Elsevier Sci-ence Publishers, pp. 749–756, 1993.
[4] G. de Haan and P.W.A.C. Biezen, “An efficient true-motion estimator using candidate vectors from a parametric motion model,” IEEE tr. on Circ. and Syst. for Video Techn., vol. 8, no. 1, pp. 85–91, Mar. 1998.
[5] Yu-Lin Chang, Ping-Hao Wu, Shyh-Feng Lin, and Liang-Gee Chen, “Four field local motion compensated de-interlacing,”
IEEE International Conference on Acoustics, Speech, and Sig-nal Processing, pp. 253–256, May 2004.
