5.1 Experiment Setting
5.2.2 Objective Quality Evaluation
The experiment setting follows the description in previous section. As mentioned in Section 2.3.4, the common-used objective quality evaluation methods are PSNR, SSIM, and T_PSPNR. Their main idea is to evaluate disparity quality by view synthesis results. Thus, they compare the difference between the real captured videos and the synthesized videos, and then analyze the frame difference by different methods. The PSNR and SSIM could be used to evaluate the spatial distortion, and the T_PSPNR could be used to evaluate the temporal distortion. The associated software tools can be obtained from [63], [77]. Note that the view synthesis algorithms are different for the DERS algorithm and our proposed algorithms. The DERS algorithm cooperates with the VSRS algorithm [64], while our proposed algorithms cooperates with the simplified VSRS algorithm [62] that adopts the Gaussian filter for the hole filling and has approximate quality to the original VSRS algorithm.
1. PSNR Evaluation Results
Table V-7 and Table V-8 shows the PSNR evaluation results for luminance channel only, and Figure V-2 plots the corresponding data by column diagram. Note that the “View0” and “View8 mean the left most and the right most views for the 9-view displays. Note that the results of Café, Kendo, and Balloons are not available in the DERS algorithm due to the reason described in previous section.
In addition, the proposed SC-DE algorithm could not generate disparity maps for the sequence Café because of insufficient memory space on PC to support the extremely high resolution and large disparity range. In this table, ∆PSNR is the PSNR difference of our algorithm and the DERS algorithm.
The positive ∆PSNR refers to our algorithm performs better than the DERS algorithm, and vice versa.
Compared to the DERS algorithm, the baseline algorithm could not perform better in most sequences because the baseline algorithm only focuses on the computational reduction, instead of the disparity quality improvement. With the temporal consistency and occlusion improvement methods, the HQ-DE algorithm could has higher PSNR than the DERS algorithm in average.
139
The SC-DE algorithm is accelerated version of HQ-DE algorithm, and suffers from slight PSNR drops. On the other hand, the HE-DE algorithm, the other accelerated version of HQ-DE algorithm, has the slight quality drops in all sequences except the sequence Champagne, and the average PSNR is higher than other algorithms. That implies the proposed cost diffusion method and the new irregular occlusion handling method could deliver better disparity maps than the other proposed algorithms.
Table V-7 Evaluation results of Y-PSNR for View0
DERS Baseline HQ-DE SC-DE HE-DE
PSNR PSNR ∆PSNR PSNR ∆PSNR PSNR ∆PSNR PSNR ∆PSNR BookArrival 34.28 35.54 1.26 35.98 1.70 35.85 1.58 35.80 1.53
LoveBird1 32.45 32.07 -0.38 32.63 0.18 32.58 0.13 31.53 -0.92 Newspaper 29.53 29.27 -0.27 29.90 0.37 29.84 0.31 30.03 0.49
Café N.A. 32.83 - 33.30 - N.A. - 33.22 -
Kendo N.A. 34.66 - 34.84 - 34.82 - 34.88 - Balloons N.A. 34.72 - 35.07 - 34.79 - 34.91 - Champagne 25.32 28.27 2.95 27.63 2.31 24.99 -0.32 31.07 5.75
Pantomime 36.46 37.01 0.55 35.94 -0.52 35.58 -0.88 34.66 -1.80 Average 31.61 33.04 0.82 33.16 0.81 32.64 0.16 33.26 1.01
Unit: dB Table V-8 Evaluation results of Y-PSNR for View8
DERS Baseline HQ-DE SC-DE HE-DE
PSNR PSNR ∆PSNR PSNR ∆PSNR PSNR ∆PSNR PSNR ∆PSNR BookArrival 35.87 35.68 -0.19 35.89 0.02 36.08 0.21 36.02 0.15
LoveBird1 29.31 27.53 -1.78 28.23 -1.08 28.22 -1.09 27.98 -1.33 Newspaper 31.86 31.29 -0.57 31.76 -0.10 31.65 -0.20 31.92 0.06
Café N.A. 32.87 - 33.01 - N.A. - 33.04 -
Kendo N.A. 35.75 - 36.24 - 36.12 - 36.36 - Balloons N.A. 35.24 - 35.63 - 35.40 - 35.58 - Champagne 24.20 28.72 4.52 28.11 3.91 27.46 3.26 29.73 5.53
Pantomime 34.65 35.85 1.20 36.00 1.35 36.13 1.48 35.61 0.96 Average 31.18 32.87 0.64 33.11 0.82 33.01 0.73 33.28 1.08 Unit: dB
140 (a)
(b)
Figure V-2 Evaluation results of Y-PNSR
2. SSIM Evaluation Results
In the SSIM evaluation, we calculate the average of the SSIMs in the three channels, R, G, and B for each sequence. Table V-9 and Table V-10 list the SSIM evaluation results for the View0 and View8, and Figure V-3 shows the corresponding column diagrams. With the SSIM evaluation results, all the proposed algorithms could have the approximate quality to the DERS algorithm but suffer from slight drops less than 0.02.
Table V-9 Evaluation results of SSIM for View0
DERS Baseline HQ-DE SC-DE HE-DE
BookArrival LoveBird1 Newspaper Café Kendo Balloons Champagne Pantomime Y-PSNR for View0 (dB)
BookArrival LoveBird1 Newspaper Café Kendo Balloons Champagne Pantomime Y-PSNR for View8 (dB)
DERS Baseline HQ-DE SC-DE HE-DE
141
Table V-10 Evaluation results of SSIM for View8
DERS Baseline HQ-DE SC-DE HE-DE
Figure V-3 Evaluation results of SSIM
3. PSPNR Evaluation Results
The PSPNR evaluation method [76] consists of the S_PSPNR for spatial distortion and the T_PSPNR for temporal distortion. In this dissertation, we adopt the T_PSPNR to evaluate the temporal consistency of disparity maps. Table V-11 and Table V-12 list the T_PSPNR evaluation results, and Figure V-4 plots the corresponding column diagrams. Compared to the DERS algorithm, the baseline algorithm has serious quality degradation due to no temporal consistency enhancement applied. Taking advantage of the proposed temporal consistency enhancement methods, the HQ-DE
0.85
BookArrival LoveBird1 Newspaper Café Kendo Balloons Champagne Pantomime SSIM for View0
BookArrival LoveBird1 Newspaper Café Kendo Balloons Champagne Pantomime SSIM for View8
142
algorithm could have higher performance than the DERS algorithm. Such the high performance is slightly decreased in the SC-DE and HE-DE algorithms in most sequences because of their acceleration methods. Nevertheless, the two fast algorithms still perform better than the DERS in most of the sequences.
Table V-11 Evaluation results of T_PSPNR (dB) for View0
DERS Baseline HQ-DE SC-DE HE-DE
T_PSPNR T_PSPNR ∆T_PSPNR T_PSPNR ∆T_PSPNR T_PSPNR ∆T_PSPNR T_PSPNR ∆T_PSPNR BookArrival 52.96 49.83 -3.13 53.60 0.64 54.10 1.14 52.94 -0.02
LoveBird1 45.30 43.08 -2.23 46.57 1.26 46.46 1.16 45.70 0.39 Newspaper 43.38 39.44 -3.94 44.09 0.71 44.19 0.82 43.65 0.27
Café N.A. 44.00 - 46.59 - N.A. - 47.83 -
Kendo N.A. 47.57 - 48.08 - 47.90 - 48.15 -
Balloons N.A. 48.25 - 49.99 - 48.25 - 49.93 -
Champagne 34.62 40.34 5.72 41.28 6.66 40.03 5.41 44.56 9.94 Pantomime 51.85 52.10 0.25 52.19 0.35 50.12 -1.72 50.95 -0.90 Average 45.62 45.57 -0.67 47.80 1.92 41.38 1.36 47.96 1.94
Unit dB Table V-12 Evaluation results of T_PSPNR for View8
DERS Baseline HQ-DE SC-DE HE-DE
T_PSPNR T_PSPNR ∆T_PSPNR T_PSPNR ∆T_PSPNR T_PSPNR ∆T_PSPNR T_PSPNR ∆T_PSPNR BookArrival 51.82 50.34 -1.48 53.52 1.70 52.81 0.99 54.62 2.79
LoveBird1 43.33 41.21 -2.11 44.70 1.37 44.75 1.42 43.84 0.51 Newspaper 47.92 43.43 -4.49 47.96 0.04 47.24 -0.67 47.82 -0.09
Café N.A. 43.42 - 46.86 - N.A. - 46.85 -
Kendo N.A. 49.34 - 50.58 - 50.41 - 50.81 -
Balloons N.A. 47.69 - 49.76 - 48.03 - 49.90 -
Champagne 34.16 40.00 5.84 41.18 7.02 41.32 7.16 42.19 8.03 Pantomime 48.45 49.13 0.68 50.12 1.67 49.98 1.53 50.06 1.61 Average 45.14 45.57 -0.31 48.09 2.36 47.79 2.09 48.26 2.57
Unit dB
143 (a)
(b)
Figure V-4 Evaluation results of T_PSPNR
4. Disparity Maps and Synthesized Images
Finally, the disparity maps and view synthesis results are demonstrated in Figure V-5 to Figure V-16. The HQ-DE algorithm could improve the disparity maps and synthesized images better than the baseline algorithm, and has comparable results to the DERS algorithm. Compared to the HQ-DE algorithm, the SC-DE and the HE-DE algorithms has disparity noising at the object boundaries due to their simplified methods.
30 35 40 45 50 55
BookArrival LoveBird1 Newspaper Café Kendo Balloons Champagne Pantomime T_PSPNR for View0 (dB)
DERS Baseline HQ-DE SC-DE HE-DE
30 35 40 45 50 55
BookArrival LoveBird1 Newspaper Café Kendo Balloons Champagne Pantomime T_PSPNR for View8 (dB)
DERS Baseline HQ-DE SC-DE HE-DE
144
145
Figure V-5 Disparity maps and view synthesized images in the 50th frame of BookArrival Results from top to down are the produced by the DERS, baseline, HQ-DE, SC-DE, HE-DE algorithms.
146
147
Figure V-6 Disparity maps and view synthesized images in the 50th frame of LoveBird1
Results from top to down are the produced by the DERS, baseline, HQ-DE, SC-DE, HE-DE algorithms.
148
149
Figure V-7 Disparity maps and view synthesized images in the 100th frame of Newspaper Results from top to down are the produced by the DERS, baseline, HQ-DE, SC-DE, HE-DE algorithms.
Figure V-8 Disparity maps and view synthesized images in the 50th frame of Café Results from top to down are the produced by the baseline, HQ-DE, HE-DE algorithms.
150
Figure V-9 Disparity maps and view synthesized images in the 50th frame of Kendo Results from top to down are the produced by the baseline, HQ-DE, SC-DE, HE-DE algorithms.
151
Figure V-10 Disparity maps and view synthesized images in the 100th frame of Balloons Results from top to down are the produced by the baseline, HQ-DE, SC-DE, HE-DE algorithms.
152
153
Figure V-11 Disparity maps and view synthesized images in the 50th frame of Champagne Results from top to down are the produced by the DERS, baseline, HQ-DE, SC-DE, HE-DE algorithms.
154
155
Figure V-12 Disparity maps and view synthesized images in the 50th frame of Pantomime Results from top to down are the produced by the DERS, baseline, HQ-DE, SC-DE, HE-DE algorithms.
156
Figure V-13 Disparity maps and view synthesized images in the 50th frame of Hall1 Results from top to down are by the DERS, baseline, HQ-DE, SC-DE, HE-DE algorithms.
157
Figure V-14 Disparity maps and view synthesized images in the 50th frame of Hall2 Results from top to down are by the DERS, baseline, HQ-DE, SC-DE, HE-DE algorithms.
158
Figure V-15 Disparity maps and view synthesized images in the 167th frame of CarPark Results from top to down are by the DERS, baseline, HQ-DE, SC-DE, HE-DE algorithms.
159
Figure V-16 Disparity maps and view synthesized images in the 50th frame of CarPark Results from top to down are the produced by the baseline, HQ-DE, SC-DE, HE-DE algorithms.
5.3 Summary
The disparity quality and execution time of the proposed algorithms are examined using the test bench for view synthesis application. Compared to the DERS algorithm, the proposed HQ-DE
160
algorithm has high disparity quality in the temporal PSPNR evaluation, and approximate disparity quality in the spatial PSNR evaluation. For the computational comparison, our proposed HQ-DE algorithm is more efficient than the DERS algorithm because our processing resolution is decreased by the disparity upsampling technique. The computation of HQ-DE algorithm could be significantly reduced by the proposed SC-DE and HE-DE algorithms with slight disparity quality change. Moreover, according to their computational characteristics, the SC-DE algorithm could be further accelerated by processor-based platforms, and the HE-DE algorithm could be accelerated by VLSI design. In the next chapter, the HE-DE algorithm is implemented by VLSI design to achieve the required throughput of high resolution 3DTV applications.
161
VI Design of Disparity Estimation Engine for High Definition 3DTV Applications
The main target of this dissertation is to deliver a disparity estimation engine that can generate three view HD1080p disparity maps in the throughput of 60 frames/s. To achieve this target, we simplify the hardware-efficient disparity estimation (HE-DE) algorithm for lower hardware cost, and propose a corresponding hardware design. The implementation result shows that the proposed disparity estimation engine could achieve the target throughput, and outperform the previous implementation.
This chapter is organized as follows. First, we analyze the data dependency of HE-DE algorithm, and simplify it to reduce more hardware cost. Then, we present the proposed architecture for the simplified HE-DE algorithm. The details of its computational modules and memory access schedule are also described. Finally, the implementation result is demonstrated and compared with previous work.