The main contribution of this thesis is to propose efficient hardware architecture with three memory reduction methods for real-time integral histogram based JBF. The three proposed memory reduction methods combined reduces the memory cost to 0.003% compare to the original integral histogram based JBF. The efficient hardware architecture can process large amount of parallel histogram bins simultaneously to achieve 1 pixel per cycle high throughput. The ASIC implementation of the architecture can achieve 124Mpixel (60 frames) per second with HD1080p resolution image under 200MHz clock rate. The chip consumes totally 355 K gate counts and 23KBytes internal memory. The off-chip bandwidth requirement is 132.7Mbits per frame, which is 60% of the total bandwidth of 200 MHz clock rate. For higher throughput, the architecture and memory reduction methods can be directly extended to support the processing of multiple cascade pixels.
Future Work
In the thesis, we have proposed efficient architecture for IH based JBF and its design concept is also suitable for any integral image based applications but limited to those use the box spatial kernel. Nevertheless, Mohamed et al. [43] has shown that a more complicated kernel can be approximated by the linear combination of many basic box kernels. This extends the integral image approach to more complex applications. For the complex application, multiple parallel hardware cores of basic box kernel must be put together and thus the overall interface of data transfer and communication, and the analysis of internal memory and bandwidth requirement must be re-estimated elaborately for the best performance.
60
On the other hand, the proposed architecture is suitable for gray-level image process. For extended use for multi-color channels, extra software or hardware has to be further designed for blending color channels to gray level. Nevertheless, these methods usually depend on different applications. For example, for producing human visual consistent gray level images, Faust [44] has to includes human vision knowledge and visual aspects to present an enhance conversion.
61
Reference
[1] C. Tomasi and R. Manduchi, “Bilateral filtering for gray and color images,” in Proc. of IEEE Int’l Conf. on Computer Vision, pp. 839-846, 1998.
[2] J. Kopf, M. F. Cohen, D. Lischinski, and M. Uyttendaele, “Joint bilateral upsampling,” in Proc. of ACM SIGGRAPH, vol. 26, no. 3, p. 96, 2007.
[3] K.-J. Yoon and I. S. Kweon, “Adaptive support-weight approach for correspondence search,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 28, no. 4, pp. 650-656, April 2006.
[4] K.-J. Yoon and I. S. Kweon, “Stereo matching with symmetric cost functions,” in Proc. of IEEE Computer Vision and Pattern Recognition, pp. 2371-2377, 2006.
[5] O. Stankiewicz, K. Wegner, and M. Wildeboer, “A soft-segmentation matching in depth estimation reference software (DERS) 5.0,” ISO/IEC JTC1/SC29/WG11, doc. M17049, Xian, China, October 2009.
[6] S. Paris, P. Kornprobst, J. Tumblin, and F. Durand, “A gentle introducing to bilateral filtering and its applications,” in Proc. of ACM SIGGRAPH tutorial, 2008.
[7] E.P. Bennett, J. L. Mason ,and L. McMillan, “Multispectral bilateral video fusion,”
IEEE Trans. on Image Processing, vol. 16, no. 5, pp. 1185-1194, May 2007.
[8] T.R. Jones, F. Durand ,and M. Desbrun, “Non-iterative, feature-preserving mesh smoothing,” ACM Trans. on Graphics, vol. 22, no. 3, pp. 943-949, July 2003 [9] S. Fleishman, I. Drori ,and D. Cohen-Or, “Bilateral mesh denoising ,” ACM Trans.
62
on Graphics, vol. 22, no. 3, pp. 950-953, July 2003.
[10] E. Eiseman and F. Durand, “Flash photography enhancement via intrinsic relighting,” ACM Trans. on Graphics, vol. 23, no. 3, pp. 673-678, August 2004.
[11] G. Petschnigg, M. Agrawala, H. Hoppe, R. Szeliski, M. Cohen, and K. Toyama,
“Digital photography with flash and no-flash image pairs,” in Proc. of ACM SIGGRAPH, vol. 23, no. 3, pp. 664-672, 2004.
[12] B.M. Oh, M. Chen, J. Dorsey, and F. Durand, “Image-based modeling and photo editing,” in Proc. of ACM SIGGRAPH, pp. 433-442, 2001
[13] F. Durand and J. Dorsey, “Fast bilateral filtering for the display of high-dynamic-range images,” in Proc. of ACM Int’l Conf. on Computer Graphics and Interactive Techniques, pp. 257-266, 2002.
[14] M. Elad, “On the bilateral filter and ways to improve it,” IEEE Trans. on Image Processing, vol. 11, no. 10, pp. 1141-1151, October 2002.
[15] S.B. Bae, E. Paris, and F. Durand, “Two-scale tone management for photographic look,” ACM Trans. on Graphics, vol. 25, no. 3, pp. 637-645, 2006.
[16] H. Yu, Y.-L. Zhao, and H. Wang, “Image denoising using trivariate shrinkage filter in the wavelet domain and joint bilateral filter in the spatial domain,” IEEE Trans.
Image Process., vol. 18, no. 10, pp. 2364-2369, October 2009.
[17] C. Varekamp and B. Barenbrug, “Improved depth propagation for 2D to 3D video conversion using key-frames,” in Proc. of IET European Conf. on Visual Media Production, pp. 167-173, 2007.
[18] C.-C. Cheng, C.-T. Li, P.-S. Huang, T.-K. Lin, Y.-M. Tsai, and L.-G. Chen, “A block-based 2D-to-3D conversion system with bilateral filter,” in Proc. of IEEE
63
Int’l Conf. on Consumer Electronics, pp. 393-394, January 2009.
[19] Q. Yang, R. Yang, J. Davis, and D. Nister, “Spatial-depth super resolution for range images,” in Proc. of IEEE Computer Vision and Pattern Recognition, pp.
1845-1852, 2007.
[20] D. Chan, H. Buisman, C. Theobalt, and S. Thrun, “A noise-aware filter for real-time depth upsampling,” in Proc. of Workshop on Multi-camera and Multi-modal Sensor Fusion Algorithms and Application – M2SFA2, 2008.
[21] A. K. Riemens, O. P. Gangwal, B. Barenburg, and R-P. M. Berretty, “Multi-step joint bilateral depth upsampling,” in Proc. of SPIE Visual Communications and Image Processing, p. 72570M, 2009.
[22] M.-C. Chuang, Y.-N. Liu, T.-H. Chen, and S.-Y. Chien, “Color filter array demonsaicking using joint bilateral filter,” in Proc. of IEEE Int’l Conf. on Multimedia and Expo, pp. 125-128, 2009.
[23] C. Xiao, Y. Nie, W. Hua, and G. Feng, “Fast multi-scale joint bilateral image and video texture upsampling,” The Visual Computers, Springer Berlin/Heidelburg, pp.154-157, December 2009.
[24] L. Wang, M. Liao, M. Gong, R. Yang, and D. Nister, “High quality real-time stereo using adaptive cost aggregation and dynamic programming,” in Proc. of Int’l Symposium on 3D Data Processing, Visualization and Transmission (3DPVT), pp. 798-805, 2006.
[25] Z. Gu, X, Su, Y. Liu, and Q. Zhang, “Local stereo matching with adaptive support-weight, rank transform, and disparity calibration,” Pattern Recognition Letter, vol. 29, issue 9, pp. 1230-1235, July 2008.
[26] Q. Yang, L. Wang, R. Yang, H. Stewenius, and D. Nister, “Stereo matching with color-weighted correlation, hierarchical belief propagation, and occlusion
64
handling,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 31, no. 3, pp.2347-2354, March 2009.
[27] J. Lu, S. Rogmans, G. Lafruit, and F. Catthoor, “Stream-centric stereo matching and view synthesis: a high-speed approach on GPUs,” IEEE Trans. on Circuits and Systems for Video Technology, vol. 19, no. 11, pp. 1598-1611, November 2009.
[28] N. Y.-C. Chang, T.-H. Tsai, P.-H. Hsu, Y.-C. Chen, and T.-S. Chang, “Algorithm and architecture of disparity estimation with mini-census adaptive support weight,”
IEEE Trans. Circuits Sys. Video Technol, vol. 20, no. 6, pp. 792-805, June, 2010.
[29] Q. Yang, K.-H. Tan, and N. Ahuja, “Real-time O(1) bilateral filtering,” in Proc. of IEEE Computer Vision and Pattern Recognition, pp. 557-564, 2009.
[30] S. Paris and F. Durand, “A fast approximation of the bilateral filter using a signal processing approach,” International Journal of Computer Vision, vol. 81, no. 1, pp.24-52, 2006.
[31] T. Q. Pham and L. J. van Vliet, “Separable bilateral filtering for fast video processing,” in Proc. of IEEE Int’l Conf. on Multimedia & Expo, pp. 454-457, 2005.
[32] T.-S. Huang, “Two-dimensional digital signal processing II: transforms and median filters,” Spring-Verlag, New York, pp. 209-211, 1981.
[33] B. Weiss, “Fast median and bilateral filtering,” ACM Trans. on Graphics, vol. 25, no. 3, pp. 519-526, July 2006.
[34] F. Porikli, “Constant time O(1) bilateral filtering,” in Proc. of IEEE Computer Vision and Pattern Recognition, pp.3895-3902, 2008.
[35] R. Deriche, “Recursively implementing the Gaussian and its derivatives”, in Proc. of International Conference on Image Processing, pp. 263-267, 1992.
65
66
[36] J. Chen, S. Paris, and F. Durand, “Real-time edge-aware image processing with the bilateral grid,” ACM Trans. on Graph, vol. 26, no. 3, article 103, pp. 1-9, July 2007.
[37] A. Adams, N. Gelfand, J. Dolson, and M. Levoy, “Gaussian KD-trees for fast high dimensional filtering,” ACM Trans on Graph, vol. 28, no. 3, p. 21, 2009.
[38] M.-H. Ju, and H.-B. Kang, “Constant time stereo matching,” in Proc. of Int’l Machine Vision and Image Processing Conf., pp. 13-17, 2009.
[39] Micron Technology, “Synchronous DRAM MT48LC2M32B2-1 Meg x 32 x 4 banks,”
Available:http://download.micron.com/pdf/datasheets/dram/sdram/128MbSDRA Mx32.pdf
[40] A. Wong, “NVIDIA GeForce 8800 GTX/GTS Tech Report,” Available:
http://www.techarp.com/showarticle.aspx?artno=358&pgno=0
[41] A. L. Shimpi and D. Wilson, “Nvidia’s 1.4 billion transistor GPU: GT200 arrives as the GeForce GTX 280 & 260,” Available:
http://www.anandtech.com/show/2549
[42] “CPU World,” Available: http://www.cpu-world.com/index.html
[43] M. Hussein, F. Porikli, and L. Davis, “Kernel integral images: a framework for fast non-uniform filtering,” in Proc. of IEEE Computer Vision and Pattern Recognition, pp. 1-8, 2008.
[44] L. Neumann, M. Cadik, and A. Memcsics, “An Efficient Perception-based Adaptive Color to Gray Transformation”, in int’l Conf. of on Wireless Communications and Signal Processing, pp. 1-4, 2009.
Biographical Notes
姓名︰許博雄
學歷︰
國立交通大學電子所系統組 碩士 (民國 97 年 09 月 ~ 民國 99 年 08 月) 國立交通大學電資學院學士班 學士 (民國 93 年 09 月 ~ 民國 97 年 06 月) 國立台南第一高級中學 (民國 90 年 09 月 ~ 民國 93 年 06 月)
著作︰
[1] Po-Hsiung Hsu, Yu-Chen Tseng and Tian-Sheuan Chang, “Low Memory Cost Bilateral Filtering Using Stripe-based Sliding Integral Histogram,” in proceeding of IEEE International Symposium on Circuit and System, pp. 3120-3123, 2010.
得獎事蹟︰
9 98 學年度大專院校積體電路設計競賽標準元件數位電路設計組「佳作」
9 97 學年度大專院校積體電路設計競賽標準元件數位電路設計組「設計完成 獎」