According to different architectures, we can distinguish four categories of de-interlacing algorithms: the inter-field de-interlacing, the intra-field de-interlacing, the motion adaptive de-interlacing and the motion compensated (MC) de-interlacing.
Moreover, the motion compensated de-interlacing can be divided into two categories:
the motion compensated de-interlacing (non-recursive) and the recursive motion compensated de-interlacing. We illustrate these de-interlacing methods in order.
2.1 Inter-field de-interlacing
Inter-field de-interlacing, also called “Temporal de-interlacing” [1]. It exploits the correlation between neighboring fields in time domain when interpolating pixels.
The simplest form is field insertion, also called “Weave”. This method is usually used to process static objects. Because of the position of static objects are the same in time domain, static objects can be recovered perfectly by field insertion. Fig. 2 shows the diagram of Weave. However, because the position of moving object isn’t the same in time domain, aliasing can be produce when field insertion is in use.
Odd Field Even Field
2.2 Intra-field de-interlacing
Intra-field de-interlacing, also called “Spatial de-interlacing” [1]. It exploits the correlation between neighboring pixels in a field when interpolating pixels. The simplest form is line repetition. Fig. 3 shows the diagram of line repetition. This method produces interpolated pixels from vertically upper pixels or down pixels.
Additionally, other one of the simplest form is line averaging, also called “Bob”. Fig.
4 shows the diagram of Bob. This method produces interpolated pixels from the average between vertically upper pixels with down pixels.
In addition to above methods, the edge-directional de-interlacing is very useful too. Fig. 5 shows the diagram of the edge-directional de-interlacing.It exploits the correlation between three pixels in the previous scan line (line n-1) and the next scan line (line n+1) to determine an obvious edge in the image. The path a is the difference between original pixel 1(O1) and original pixel 6(O6), the path b is the difference between original pixel 2(O2) and original pixel 5(O5), the path c is the difference between original pixel 3(O3) and original pixel 4(O4). Obviously, the path of smallest difference is obvious edge which is used to produce interpolated pixels.
Because the architectures of above methods are very simple, these methods have the advantages of low cost and lower computational complexity [15]. However, interpolated pixels consist of neighboring pixels in a field, so interpolated pixel value approximates to real pixel value, is not equal. Therefore, aliasing, jag will appear in de-interlaced video sequences.
Original Pixel
Interpolated Pixel
O1 O2 O3 O4
O5 O6 O7 O8
I1 I2 I3 I4
O1 O2 O3 O4
O5 O6 O7 O8
Line averaging
I1 = AVG[O1,O5]
I2 = AVG[O2,O6]
I3 = AVG[O3,O7]
I4 = AVG[O4,O8]
O1 O2 O3 O4
O5 O6 O7 O8
O1 O2 O3 O4
O5 O6 O7 O8
O1 O2 O3 O4
O5 O6 O7 O8
Line repetition
Fig. 3 Line repetition Fig. 4 Bob
Fig. 5 Edge-directional de-interlacing
2.3 Motion adaptive de-interlacing
Up to now, the motion adaptive de-interlacing [17],[18] is a very common method since its architecture is not complex. This method uses a motion detector [1] to detect where the motion areas are in the current field and control the importance or “weight”
of these individual pixels at the input of the spatial filter [19]. Therefore, the intra-field de-interlacing is used for motion areas, while the inter-field de-interlacing is used for static areas. Fig. 6 shows the flowchart of the motion adaptive de-interlacing. First, we calculate the difference in corresponding pixels between reference fields and current fields until the last pixel. If the pixel difference is larger than a threshold, the pixel is regarded as the motion pixel. Otherwise, the pixel is regarded as the static pixel.
However, this simple method is easily affected by noise. Thus, the outputs of corresponding pixel difference may include some erroneous information and will cause erroneous motion detection. Thus, noise reduction [20] is necessary to eliminate noises in the difference outputs, and provide robust motion detection. General model of noise reduction consists of low-pass filter and rectifier. S-F Lin proposed the morphological operation [15] for noise reduction as shown in Fig. 7. This method uses a 3x3 min filter (low-pass spatial filter) to eliminate noise, and then a 3x3 max filter (high-pass spatial filter) to rectify the shape of motion objects. However, the min filter seriously destroyed the shape of motion objects even though it is rectified with max filter. Therefore, the shape of motion objects can not be recovered any more. To solve this problem, we propose an adaptive noise filter. It eliminates noises in the difference output according to the shape of motion objects. Thus, the shape of motion objects is not destroyed and accurate motion detection is available for later de-interlacing. This method will be illustrated in Section 3.2 in detail.
Fig. 6 Flowchart of the motion adaptive de-interlacing
F(j-1,k) F(j,k) F(j+1,k)
F(j-1,k-1) F(j,k-1) F(j+1,k-1)
F(j-1,k+1) F(j,k+1) F(j+1,k+1)
G(j,k)=MIN[F(j-1,k-1),F(j,k-1),F(j+1,k-1), F(j-1,k),F(j,k),F(j+1,k), F(j-1,k+1),F(j,k+1),F(j+1,k+1)]
D(j,k)=MAX[G(j-1,k-1),G(j,k-1),G(j+1,k-1), G(j-1,k),G(j,k),G(j+1,k), G(j-1,k+1),G(j,k+1),G(j+1,k+1)]
Fig. 7 the morphological operation
2.4 Motion compensated de-interlacing
Motion compensation is a block-based dynamic compensation technology [16][21]. It uses motion estimation to search the most similar candidate blocks in the neighboring fields, and calculates the offset of corresponding position, as called motion vector (MV) [16]. In the de-interlacing application, they are searched in the neighboring fields because moving objects exist on several continuous video sequences. Fig. 8 shows the diagram of motion compensation for de-interlacing. First, it finds out motion objects in current fields by motion detection, and then motion objects are divided to various current blocks. When searching motion objects, the block-based motion estimation is used to search the most similar candidate blocks in the neighboring fields by SAD (Sum of Absolute Difference) value [16] of candidate block. Finally, the most similar candidate block interpolates the de-interlaced output of current block. Since SAD values of all candidate blocks in the search window [6]
needs to be calculated, the required computational complexity is very large and becomes a problem in practical use.
Besides the huge computational complexity, the motion compensated de-interlacing has the other problem. This problem is that reference field only contains even lines or odd lines, and it results in motion estimation errors or lower probability of finding the most similar candidate block. Motion estimation errors are erroneous motion vectors which are caused by higher SAD threshold value, and higher SAD threshold value causes unsuitable candidate block to interpolate the de-interlaced output of current block. Lower probability of finding the most similar candidate block is caused by lower SAD threshold value, and lower SAD threshold value causes no suitable candidate block to interpolate the de-interlaced output of current block. Therefore, the recursive motion compensated de-interlacing [13] is proposed to solve this problem.
Fig. 8 Motion compensation for de-interlacing
Fig. 9 Recursive motion compensation for de-interlacing
2.5 Recursive Motion compensated de-interlacing
The difference between recursive MC de-interlacing [11][21][22] and MC de-interlacing is that, reference fields of recursive MC de-interlacing are de-interlaced fields instead of original fields when doing motion estimation. Once a perfectly de-interlaced field [1][14] is available, motion estimation is more accurate. A perfectly de-interlaced field is the de-interlaced frame output that the full still image is de-interlaced by field insertion. Because the full still image has no any motion objects in the video image, the de-interlaced frame outputs are the same as original video frames. Fig. 8 shows the diagram of recursive motion compensation for de-interlacing.
Its whole procedure is similar to the motion compensated de-interlacing and thus it also has large computational complexity.
Besides huge computational complexity, the recursive MC de-interlacing has the propagation error due to the recursive operation. The propagation errors contain motion prediction errors and interpolated errors in previously de-interlaced reference field (frame). To reduce the effect of this problem, we propose an adaptive recursive MC de-interlacing with lower propagation errors. It controls the amount of propagation errors according to video signals, and thus congenital defects of recursive MC de-interlacing are eliminated. This method is illustrated in Section 3.4 in detail.