Discussion

With regard to HDDR system, there are two major issues. One is the limited matching range in near field. When the object distance shrinks, the depth of field becomes shorter as well.

Accordingly, we need more lenslets to extend the depth. However, because the field of view is limited, the range for stereo matching is therefore confined. In other words, we cannot ensure the whole scene is totally captured by every lenslet we add. In Figure 4-26, matching range stands for the region captured by at least three lenslet while blind range means the region(s) that would never be captured. So if we change the number of elemental images for stereo matching, maybe we can extend the matching range. But it is impossible to reduce the blind range unless we increase the field of view. Nevertheless, to increase the field of view will bring about severe lens aberration. Even though we do not increase the field of view, the lens aberration may still influence our elemental images because the objects deviate from the optical axis of some lenslets. The other issue is the reliable stereo matching algorithm. For

79

disparity-based system, stereo matching is always needed to improve. Even though our HDDR system has dealt with the matching error in blur region, the issue of different perspectives still challenges the yield rate because the same feature point would not appear in the two or three elemental images simultaneously. Moreover, occlusion also happens with different perspective as shown in Figure 4-27 [55]. Therefore we cannot make sure that the render depth map is correct even if the elemental images are all in focus. Once the quality of conventional depth maps are enhanced, HDDR depth can be enhanced as well because there will be less misjudgments while thresholding.

Figure 4-26 Matching range and blind range in near field

Figure 4-27 Occlusion geometry

80

Chapter 5 Summary

5.1 Conclusion

3D content plays an important role in 3D technology, so how to capture the depth information is a target that many researchers are digging into. According to the prior arts in chapter one, single camera system based on analysis of disparity is the most practical system, but it requires all-in-focus elemental images to ensure the feature matching. However, extending the depth of field by increasing the f-number is not always suitable. Especially for dimming environment and the image content with movement, capturing with larger f-number confronts the dilemma of exposure time. If the exposure time is insufficient, the image information will be lost. On the other hand, if the exposure time is too long, the ghost image will bring about the blurred images. Two situations cause the imperfections in the rendered depth map because of mismatch in stereo matching algorithm. Hence, we extend the working range by stacking each depth of field instead of increasing the f-number. This concept originates from the high dynamic range (HDR) images which use limited contrast ratio to represent higher luminance difference. Accordingly, our system is named after a similar term of high dynamic depth range (HDDR). The idea of HDDR system can be fulfilled in two manners: temporal or spatial multiplex.

Regarding the algorithm, if DERS can be less vulnerable to noise, the performance will be enhanced. Besides, the limitation of DFEET stems from the texture and shape of objects.

Non-textured regions will contribute no edge information, which will greatly influence the threshold value and cannot filter out the ill-defined objects. As for the shape of object, void with the small aspect ratio region is hard to be reconstructed. In general, current stereo matching algorithm cannot render a depth map with all the details as the color image, so our

81

fusion process is still able to generate the HDDR depth map with acceptable quality.

In chapter four, we’ve demonstrated the experimental results of both of the systems, and from the HDDR depth maps, it’s been proved that the depth range of HDDR system (165 cm) goes beyond that of the result rendered by largest f-number of our camera in the experiment (150 cm) and the capturing time can be minimized by at least 21 times. Furthermore, the feasibility of lens array has been examined by using the moving pitch same as the size of the lenslet. And the distinction of two systems is that temporal HDDR system maintains the resolution of elemental images as the number of depth of field increases, while spatial HDDR system is capable of capturing moving objects with single shot and very short exposure time.

In the end, our HDDR system is compared with the prior arts and the result is illustrated in Table 5-1. The first compared aspect is the size of system which is quite important for commercialization. In addition, image categories should not be restricted and the depth rendering by optical and geometrical measurement is more favorable and reliable. And the last two compared factors are working range and capturing time. The image quality of both color images and depth maps are improved by pursuing higher dynamic range, so in depth maps, the dynamic range, that is working range, is represented as the depth extension. As for capturing time, it goes without saying that the shorter it is, the better the system will be.

Actually, the working range is restricted by the accuracy of stereo matching algorithm and the sensor resolution, no matter which type of the HDDR system is. The disparity increases as the density of resolution increases, so when applying stereo matching algorithm, the risk of mismatch also increases. Therefore, the realization of wonderful depth map rendering requires more robust techniques of feature extraction and matching. Optically, we’ll propose an idea to well utilize the tolerant range of disparity in following part, future work.

82

Table 5-1 Comparison table of our HDDR system with the prior arts

5.2 Future Work

In future work, following the concept of HDDR system, we can apply it in very near field. To reduce the disparity, we have to minimize the size of lenslet. Therefore, liquid crystal lens array is a candidate. Moreover, because high dynamic range image also includes an idea of delicate resolution of luminance, we are eager to render a depth map with not only large depth range but also fine depth resolution. Once this kind of depth map can be generated, we can resize depth range and zoom in the scene.

83

5.2.1 Liquid Crystal Lens Array

Liquid crystal (LC) is a birefringence material whose refractive index varies as the polarization, and it can be controlled by electrical field. Therefore, the character is suitable for phase modulator. LC lens is a kind of GRIN lens and its lens function can be switched in and off as illustrated in Figure 5-1 [56].

Figure 5-1 Mechanism of GRIN lens

By adjusting the applied voltage, the distribution of effective refractive index also can be changed so as to alter the focal length. Besides, LC lens is very small and thin due to its high birefringence. Hence, replacing conventional lens with LC lens benefits our HDDR system in three aspects. First of all, the arrangement of the depth of field can be optimized via the tunable focus capability of LC lens. Secondly, if customers want to shoot conventional 2D images, they don’t have to take off the lens array but shut down the lens function electrically.

Thirdly, smaller pitch of lens array can reduce the disparity of objects. In other words, we are capable of capturing nearer objects in the scene. Furthermore, many small operations are carried out by endoscopes, so if tiny LC lens array with our HDDR system can be embedded with endoscopes, maybe we can use HDDR depth map to build a 3D model of the tissue or the tumor as shown in Figure 5-2 [57] . The 3D model would help surgeons operate the surgery and leave smaller wounds.

84

Figure 5-2 Scheme of the 3D model of a tumor growing

5.2.2 Fine Depth Resolution

Because of the limitation of the searching range in stereo match algorithm as well as the size of pixel, the depth rendering is confined within a specific range of disparity. DERS can tolerate the disparity within 100 pixels, so the larger the pitch of lens array is, the farther positions can be rendered correctly. On the other hand, if the objects is remote from the camera, the disparity smaller than one pixel will not be recorded. As a result, given the size of pixel and the pitch of lens array, the bounded depth range can be calculated as the following equations.

(39)

where p and g are the pitch and the gap between lens array and sensor respectively, and d is depth from lens array. Because the disparity limit is from 1 pixel to 100 pixels, so the depth range becomes

(40) Accordingly, we can design a system to generate a depth map with fine depth resolution by adjusting the pitch as shown in Figure 5-3 and Figure 5-4. We can use three columns of elemental images (say column3 to column5) to render one HDDR depth map and change

85

another set of three columns of elemental images (say column2, column4, and column6) to render another HDDR depth map. For example, if HDDR-2 region in Figure 5-3 is the conventional range of one depth map with 8-bit gray levels, every object in HDDR-3 region would be regarded as no depth different because they don’t have disparity. However, if we can zoom in the scene along the depth, the objects in HDDR-3 region is no more at the same depth. Accordingly, we can use this technique to generate “high definition” depth maps and resize the depth as conventional 2D images.

Figure 5-3 Horizontal scheme of fine depth resolution design

Figure 5-4 Distribution of lens array with different rendering depth ranges

86

Reference

[1] http://en.wikipedia.org/wiki/Digital_camer [2] http://en.wikipedia.org/wiki/Pulfrich_effect

[3] H. Murata, et al., “A Real-Time 2-D to 3-D Image Conversion Technique Using Computed Image Depth”, SID SYM, Vol. 29, pp. 919-922, 1998.

[4] P. Harman, et al., “ Rapid 2D to 3D Conversion”, In Proc. SPIE, Stereoscopic Displays and Virtual Reality Systems IX, Vol. 4660, pp. 78-86, 2002.

[5] S. Battiato, et al., “Depth-Map Generation by Image Classification”, In Proc. SPIE, Three-Dimensional Image Capture and Applications VI, Vol. 5302, pp. 95-104, 2004.

[6] J. Li, et al., “A Novel Image-Based Rendering System with a Longitudinal Aligned Camera array”, EUROGRAPHICS, 2000.

[7] R. Furukawa, et al., “One-shot entire shape acquisition method using multiple projectors and cameras”, In Fourth (PSIVT) of IEEE Computer Society, pp. 107-114, 2010.

[8] R. Furukawa, et al., “Multiview Projectors/Cameras System for 3D Reconstruction of Dynamic Scenes”, In 4DMOD-Workshop on Dynamic Shape Capture and Analysis 2011, hal-00675085, version 1, 2012.

[9] C. Zhou, et al., “Focal Sweep Camera for Space-Time Refocusing”, Columbia University Computer Science Tech. Reports, 2012.

[10] O. Cossairt and S. Nayar, “ Spectral Focal Sweep: Extended Depth of Field from Chromatic Aberrations”, IEEE Conference on Computational Photography, pp. 1-8.

[11] http://en.wikipedia.org/wiki/Plenoptic_illumination_function

[12] E. H. Adelson and J. Y. A. Wang, “Single Lens Stereo with a Plenoptic Camera”, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 14, No. 2, pp. 99-106, 1992.

[13] R. Ng, et al., “Light Field Photography with a Hand-held Plenoptic Camera”, Stanford Tech Report, CTSR 2005-02, 2005.

[14] A. Stern and B. Javidi, “Three-Dimensional Image Sensing, Visualization, and Processing Using Integral Imaging”, In Proc. IEEE, Vol. 94, No. 3, 2006.

[15] J. Arai, et al., “Gradient-index lens-array method based on real-time integral photography for three-dimensional images”, Applied Optics, Vol. 37, No. 11, 1998.

[16] A. Stern and B. Javidi, “Three-dimensional imaging sensing, visualization, and

processing using integral imaging”, In Proc. IEEE on 3-D technologies for imaging and display, 2006.

[17] Y. T. Lim, et al., “Analysis on enhanced depth of field for integral imaging microscope”, Optics Express, Vol. 20, No. 21, 2012.

[18] J. S. Jang and B. Javidi, “Large depth-of-focus time-multiplexed three-dimensional

87

integral imaging by use of lenslets with nonuniform focal lengths and apture sizes”, Optics Letters, Vol. 28, No. 20, 2003.

[19] M. C. Rau´l, et al., “Progress in 3-D Multiperspective Display by Integral Imaging”, In Proc. IEEE, Vol. 97, No. 6, 2009.

[20] http://www.wisegeek.com/what-is-the-difference-between-monocular-and-binocular- vision.htm

[21] N. Holliman, “3D Display System”, 2005.

[22] L. Hill, et al., “Invited paper: 3-D Liquid Crystal Displays and Their Applications”, IEEE, Vol. 94, No. 3, 2006.

[23] J. Mansson, “Stereovision: A model of human stereopsis”, Lund University Cognitive Science, Tech. Report., 1998.

[24] E. Edirisinghe, et al., “Stereo image, an emerging technology”, SSGRR, L’Aquila, Italy, 2000.

[25] http://www.vision3d.com/stereo.html

[26] G. Woodgate, et al., “Flat panel autostereoscopic displays-characterization and

enhancement”, In Proc. SPIE, Stereoscopic Displays and Virtual Reality Systems VII, Vol. 3957, pp. 153-164, 2000.

[27] H. Morishima, et al., “Rear cross lenticular 3D display without eyeglasses”, In Proc.

SPIE, Stereoscopic Displays and Virtual Reality Systems VII, Vol. 3295, pp. 193-202, Apr. 1998.

[28] Y. Schechner and N. Kiryati, “Depth from Defocus vs. Stereo: How Different Really Are They?”, IEEE International Journal of Computer Vision, Vol. 39, No. 2, pp. 141-162, 2000.

[29] C. Zhou, et al., “Coded Aperture Pairs for Depth from Defocus”, IEEE International Conference on Computer Vision, 2009.

[30] J. Hong, et al., “Three-dimensional display technologies of recent interest: principles, status, and issues. [Invited]”, Applied Optics, Vol. 50, No. 34, 2011.

[31] http://ucalgary.ca/pip369/mod4/depthperception/oculomotor

[32] T. C. Shen, “Autostereoscopic 2D-3D Switching Display with Multi-Electrically Driven Cylindrical Liquid Crystal Lens”, Master, Institute of Electro-Optical Engineering, National Chiao Tung University, Hsinchu, Taiwan, ROC, 2009.

[33] http://ucalgary.ca/pip369/mod4/depthperception/monocular/pictorial [34] http://ucalgary.ca/pip369/mod4/depthperception/movement

[35] http://en.wikipedia.org/wiki/Microscopy

[36] W. J. Smith, “Modern Optical Engineering”, Fourth edition, McGraw-Hill, New York, 2008.

[37] C.M. Sparrow, “On Spectroscopic Resolving Power”, Astrophysical Journal, Vol. 44, pp.76-86, 1916.

88

[38] E. Hecht, “Optics”, Fourth edition, Addison Wesley, San Francisco, 2002.

[39] C. A. Bennett, “Principle of PHYSICAL OPTICS”, 吳忠義等譯，滄海書局，台中，民 國九十八年。

[40] http://en.wikipedia.org/wiki/Hyperfocal_distance

[41] http://en.wikipedia.org/wiki/Compound_microscope#Compound_microscope [42] http://en.wikipedia.org/wiki/Dynamic_range

[43] F.. Houlmann and S. Metz, “High Dynamic Range Rendering in OpenGL”, UTBM.

[44] http://en.wikipedia.org/wiki/Dodging_and_burning

[45] J. Kuang, et al., “Evaluating HDR Rendering Algorithm”, Munsell Color Science Laboratory, Rochester Institute of Technology, Rochester, New York, 2006.

[46] K. Devlin, “A review of tone reproduction techniques”, Tech. Report CSTR-02-005, Department of Computer Science, University of Bristol, 2002.

[47] http://en.wikipedia.org/wiki/HDRI

[48] J. W. Goodman, “Fourier Optics”, Third edition, ROBERTS & COMPANY, Colorado, 2005.

[49] C. Fehn, “Depth-image-based rendering (DIBR), compression, and transmission for a new approach on 3D-TV”, In Proc. SPIE, Stereoscopic Displays and Virtual Reality System XI, 2004.

[50] http://mpeg.chiariglione.org/about

[51] O. Stankiewicz, et al., “A soft segmentation matching in Depth Estimation Reference Software (DERS) 5.0”, MPEG, M17049, 2009.

[52] A. Olofsson, “Modern Stereo Correspondence Algorithms: Investigation and evaluation”, Master, Institute of systemteknik, Linköping University, Linköping, Sweden, 2010.

[53] Gonzales and Woods, “Digital image processing, 3/e”, 繆紹綱譯，台灣培生教育出版股 份有限公司，台北，民國九十八年。

[54] S. Lӧsch, “Depth from Blur-Combining Image Deblurring and Depth Estimation”, Master, Department of computer science, Saarland University, Saarbrücken, Germany, 2009.

[55] http://www.cse.yorku.ca/~sizints/ctf_stereo/ctf_stereo_main.html [56] http://www.vdwoxford.org/ourwork/#lenstech

[57] http://biomedicalcomputationreview.org/content/3d-angiogenesis-modeled