4.1 High-Quality Disparity Estimation Algorithm
4.1.7 Temporal Consistency Enhancement
In previous work and the baseline algorithm, the temporal consistency problems include the flicker artifact and foreground copy artifact due to no enhancement method and over enhancement
103
method, respectively. To address it, we propose the NMR method and the SEP method based on the conventional method in the 3DVC’s DERS algorithm.
1. Conventional Method
First, we introduce the conventional temporal consistency enhancement method in the 3DTV’s DERS algorithm, and point out its drawback. In the conventional method, the main idea is to propagate previous disparity map to current one for no-motion regions by adding the temporal cost
C
temp to cost cube. The conventional method first applies the bilateral filter to smooth the previous frame and the current frame, and then partition the frames into 16×16 macroblocks for calculating the motion absolute difference (MAD) by𝑀𝐴𝐷 = 1
16×16∑(𝑢,𝑣)∈𝑚𝑎𝑐𝑟𝑜𝑏𝑙𝑜𝑐𝑘|𝐼𝐻𝑡(𝑢, 𝑣) − 𝐼𝐻𝑡−1(𝑢, 𝑣)| . (IV-16) If MAD is less than a defined threshold γemp, the block would be regarded as a no-motion block. Thus, the temporal cost can be computed by
𝐶𝑡𝑒𝑚𝑝(𝑥, 𝑦, 𝑑) = {𝜆𝑡𝑒𝑚𝑝|𝑑 − 𝐷𝐻𝑡−1(𝑥, 𝑦)| 𝑖𝑓 𝑀𝐴𝐷 < 𝛾𝑡𝑒𝑚𝑝
0 𝑒𝑙𝑠𝑒 , (IV-17)
where λtemp is a scaling term. In this equation, the no-motion block will suffer from the penalty if its disparity is inconsistent to previous frame.
The conventional method can solve the flicker artifact, but incurs the foreground copy artifact because the background object does not have enough time to update its disparity. On the hand, the previous disparity upsampling step would result in the flicker artifact even if the conventional temporal consistency enhancement is adopted in the HQ-DE algorithm. That is because the object boundary suffers from mixed color of background and foreground, so that the disparity at the boundary would be unstable after the disparity upsampling. To sum up, the HQ-DE algorithm has the foreground copy artifact and the flicker artifact if the convention method is adopted. The following proposed two methods could solve them.
2. No-Motion Registration Method
104
Figure IV-15 illustrates the concept of the proposed no-motion registration (NMR) method by a common case. In which, the pixel is changed from a foreground pixel to a background pixel when a foreground object passes. The conventional method in DERS algorithm propagates the previous disparity to current one when the pixel is no-motion pixel, and takes short frame time to update the disparity from foreground to background while the foreground object is moving out. That would result in foreground copy artifact because of insufficient updating time. To address it, the proposed NMR method extends the motion interval by τNMR frames to provide sufficient updating time. In other word, the no-motion frame count NMC begins to be accumulated while the pixel becomes no-motion one. If
NMC is more than τ
NMR, the temporal cost Ctemp would be computed to propagate previous disparity to current frame.Figure IV-15 Concept of the proposed no-motion registration (NMR) method
Figure IV-16 shows the resultant disparity maps and the synthesized images of the proposed NMR method. Compared to the conventional method in Figure IV-4, the door pivot could be recovered well using the proposed method. In addition, Figure IV-17 shows the change of disparity maps and synthesized images in the seccussive frames while the man is passing away. In Figure IV-17 (c), the door pivot is temporarily distorted as the same as that in Figure IV-4, because the background disparity is still updating. Nevertheless, the distortion could disappear in Figure IV-17 (d).
Pixel
Time t
Previous Disparity Current Disparity
Previous Disparity
Conventional Method
Previous Disparity Current Disparity
Previous Disparity
Proposed NMR Method
no-motion motion no-motion
τ
NMR Foregroundobject
Foreground object
is moving out. Background Object
105
(a) (b) (c)
Figure IV-16 Results of the proposed NMR method in BookArrival (a) the 1st frame, (b) the 25th frame, and (c) the 40th frame
(a) (b) (c) (d)
Figure IV-17 Results of the proposed NMR method in the 32th, 34th, 36th, 38th frames
3. Still-Edge Preservation Method
106
The main idea of the proposed SEP method is to preserve the previous disparity for the still-edge.
In the SEP method, we use the bilateral filter to de-noise image, and apply the Sobel filter with a gradient threshold to detect edges. Combining with the above motion and no-motion information, we could find the still edge, which is no-motion pixel and edge pixel. For the still edges, the current disparity is directly propagated from the previous frame.
Figure IV-18 shows the synthesized result using the disparity maps of SEP method. Compared to the results of baseline algorithm in Figure IV-3, the SEP method could address the flicker artifact on the object boundary.
(a) (b) (c) (d)
Figure IV-18 Results of the proposed SEP method in BookArrival (a) the 9th frame, (b) the 10th frame, (c) the 11th frame, (d) the 12th frame
To sum up, the proposed HQ-DE algorithm could address the temporal consistency and occlusion problems to deliver better disparity maps than the previous work. Taking advantage of the disparity upsampling technique and the fast convergent BP-M approach, the HQ-DE algorithm could also save computation time for high definition disparity estimation. Note that the window sizes in the joint bilateral upsampling and window vote methods are selected from several sampled sizes, and they could be finely tuned to attain the higher quality. The associated objective quality evaluation of HQ-DE algorithm will be presented in Chapter V.