單張影像之特質影像萃取

全文

(1)Characteristic Image Decomposition from a Single Image A dissertation presented by Yun-Chung Chung to Department of Computer Science and Information Engineering in fulfillment of the requirements for the degree of Doctor of Philosophy in the subject of Computer Science and Information Engineering. National Taiwan Normal University Taipei, Taiwan July 2009.

(2) 國立台灣師範大學資訊工程研究所博士論文. 單張影像之特質影像萃取 Characteristic Image Decomposition from a Single Image. 研究生：鍾允中指導教授：陳世旺博士. 中華民國九十八年七月.

(3) Acknowledgement. First, I would like to gratefully and sincerely thank Prof. Sei-Wang Chen for his guidance, understanding, patience, and most importantly, his friendship during my studies. His insightful guidance is indispensable to the accomplishment of this dissertation. For everything you’ve done for me, Prof. Chen, I thank you. I would like to thank all the members of my doctoral committee for their input, valuable discussions and accessibility. The committee include research fellow Mark Hong-Yuan Liao, Institute of Information Science, Academia Sinica; professor Yi-Ping Hung, Department of Computer Science and Information Engineering, National Taiwan University; professor Kuo-Liang Chung, Department of Computer Science and Information Engineering, National Taiwan University of Science and Technology; professor Din-Chang Tseng, Department of Computer Science and Information Engineering, National Central University; professor Greg Lee and Chiung-Yao Fang, Department of Computer Science and Information Engineering, National Taiwan Normal University. Thank you for all of your wise efforts to make this dissertation more perfect then ever. I would also like to thank to all of the dearest members of the Intelligent Transportation System Lab, National Taiwan Normal University, for sharing innumerous days and nights in the lab. Thanks for all of your encouragements and supports when I was in need. In particular, I would like to thank Jung-Ming Wang and Dr. Shyang-Lih Chang, who have spent all the years inspiring me in the long research ages.. Characteristic image decomposition from a single image. Ack-1.

(4) Finally, and most importantly, I would like to thank my family. All of your support, encouragement, quiet patience and unwavering love are undeniably the bedrock upon which the past years of my life have been built. You provide me with unending encouragement and support to let me finish the incredible mission. My daughter and son, Rosalyn and Josh, give me the most joyful experience that I have never had in my life. I thank you all in my life who has been with me, sincerely. Thank you.. Anthony Yun-Chung Chung CSIE, NTNU July 2009. Characteristic image decomposition from a single image. Ack-2.

(5) Abstract. Many computer vision applications have had successful results in limited environmental conditions. However, they often fail when the constraints are loosened as in real world scenes. One of the most common restrictions imposed on vision algorithms is the illumination condition. Techniques that are able to tolerate illumination variations will be useful for general and realistic scenes. In this study, a solution is proposed to get around the undesired effects of illumination such as shadows, highlights and interference reflections. They are called characteristic images and decomposed from the input image. In view of that edges are one of the keys to understand an image; a computational framework for characteristic image decomposition from a single image based on the edges of the image is developed. The major idea is to classify the edge pixels of the image to target characteristic subsets. The proposed computational framework for characteristic decomposition consists of four major steps: boundary detection, evidence extraction, boundary classification, and characteristic image reconstruction. Given an image, the boundaries of the image are first detected. Evidence is extracted to classify the edge pixels to characteristic subsets. Based on the classification result of edge pixels, an integration process is applied to the classified edges to reconstruct the characteristic images. Three applications of this computational framework, i.e., interference reflections, highlight reflections, and intrinsic images, are developed in this dissertation. For interference reflections, a technique for separating reflection and object components of a single interference image in an automated manner is presented. The key idea of the proposed method is to classify edges of the interference image into either reflection or Characteristic image decomposition from a single image. Abs-1.

(6) object, and to use integration to reconstruct reflection and object images. The method utilizes TV model, blur measure, and region segmentation results as evidence with fuzzy integral technique to classify the edge pixels. Based on the classification results of edge pixels, an integration method is applied to reconstruct the reflection and object components of the input image. For separating specular and diffuse components, Shafer’s dichromatic reflection model is utilized, which assumes that light reflected at a surface point is linearly composed of diffuse and specular reflections. The major idea is to classify the boundary pixels of an image as specular or diffuse. A fuzzy integral process is proposed to classify boundary pixels based on their local evidences, including specular and diffuse estimation information. Based on the classification result of boundary pixels, an integration method is applied to reconstruct the specular and diffuse components of the input image. Unlike previous research, the proposed method has no color segmentation or iterative operations. For intrinsic images, the proposed approach first convolves an input image with a prescribed set of derivative filters. The pixels of the derivative images are next classified as reflectance or illumination according to three measures: chromatic, intensity contrast and edge sharpness, which are calculated in advance for each pixel from the input image. Finally, an integration process is applied to the classified derivative images to obtain the intrinsic images of the original image. The experimental results have demonstrated that the proposed methods can perform characteristic image decomposition from a single image effectively with small misadjustments and rapid convergence.. Abs-2. Characteristic image decomposition from a single image.

(7) 摘要. 對於許多電腦視覺方面的應用而言，從輸入單一張的影像中粹取特徵影像（characteristic images）是非常重要的一個課題，例如陰影的分析、輔助光影的研究、反光消除、反射影像的移除等。舉例來說，對於許多視覺化的智慧型交通運輸應用系統（Intelligent Transportation System）而言，例如交通監控、交通的違規執法、駕駛安全輔助、自動車輛導引等，這些視覺系統若不是架設於戶外，就是裝設於車輛中，它們都遭遇到一共同的困擾，就是光影（包括陰影、反光等現象）常常會干擾甚至降低相關系統的可靠度，使得後續的處理工作增加不少的困難。然而，想要直接從輸入單一張的影像中粹取特徵影像並非一件容易的事。本論文提出一個可靠的架構從輸入單一張的影像中粹取特徵影像，本架構包括四個主要步驟：邊緣偵測（boundary generation）、分類資訊擷取（information extraction）、邊緣分類（boundary classification）以及特徵影像組成（image composition）。在本論文中共提出應用本架構所解決的三像主要問題，包括反射干擾、反光消除、陰影移除，其特徵影像分別定義如下：反射干擾影像（Interference images）之特徵影像包括：觀測物體影像（Object image）以及反射影像（Reflection image）；反光影像（Dichromatic reflection images）之特徵影像包括：強光影像（Specular image）以及物體影像（Diffuse image）；本質影像（Intrinsic images）之特徵影像包括：實體影像（Reflectance image）以及光影影像（Illumination image）。本論文所提出的架構，可從輸入單一張的影像中粹取特徵影像的技術，除了可以提供上述應用系統一個問題的解決方法之外，並可以應用於任何戶外或是與光影有關的系統。. Characteristic image decomposition from a single image. CAbs-1.

(8) Characteristic Image Decomposition from a Single Image. Contents. Abstract Figure List Table List Chapter 1 Introduction 1.1 Motivation. 1-2. 1.2 Literature Review. 1-6. 1.2.1 Interference Image Decomposition. 1-6. 1.2.2 Dichromatic Reflection Image Decomposition. 1-9. 1.2.3 Intrinsic Image Decomposition. 1-12. 1.3 The Proposed Techniques. 1-15. 1.4 Contributions. 1-19. 1.5 Organization of this Dissertation. 1-20. Chapter 2 Computational Framework for Characteristic Image Decomposition 2.1 Characteristic Image Decomposition Design. 2-2. 2.2 System Architecture. 2-5. 2.2.1 Boundary Generation. 2-5. 2.2.2 Information Extraction. 2-6. 2.2.3 Boundary Classification. 2-7. 2.2.4 Characteristic Image Composition. 2-7. 2.3 Proposed Applications Characteristic image decomposition from a single image. 2-9 C-1.

(9) Characteristic image decomposition from a single image. Chapter 3 Interference Reflection Separation 3.1 Introduction. 3-2. 3.2 Interference Image Decomposition. 3-5. 3.2.1 Boundary Detection. 3-9. 3.2.2 Evidence Extraction. 3-9. 3.2.3 Boundary Classification. 3-14. 3.2.4 Component Image Reconstruction. 3-16. 3.3 Details of Techniques. 3-16. 3.3.1 Total Variation Decomposition. 3-16. 3.3.2 Blur Measure. 3-19. 3.3.3 Region Segmentation. 3-22. 3.3.4 Fuzzy Integral. 3-26. 3.3.5 Classification refinement. 3-30. 3.3.6 Component Image Reconstruction. 3-32. 3.4 Experimental Results. 3-33. 3.5 Concluding Remarks. 3-37. Chapter 4 Dichromatic Reflection Decomposition 4.1 Introduction. 4-3. 4.2 System Architecture. 4-5. 4.3 Information Extraction. 4-8. 4.3.1 Specular Estimation Map. 4-8. 4.3.2 Diffuse Estimation Map. 4-10. 4.4 Boundary Classification. C-2. 4-14. 4.4.1 Fuzzy Integral. 4-15. 4.4.2 Fuzzy Classification. 4-15. 4.4.3 Defect Compensation. 4-17. 4.5 Experimental Results and Discussions. 4-19. 4.6 Concluding Remarks. 4-23.

(10) Contents. Chapter 5 Intrinsic Image Extraction 5.1 Introduction. 5-3. 5.2 System Architecture. 5-4. 5.2.1 Logarithmic Edge Generation. 5-5. 5.2.2 Characteristic Measure Calculation. 5-5. 5.2.3 Hierarchical Edge Classification. 5-8. 5.2.4 Intrinsic Image Formation. 5-12. 5.3 Chromatic Characteristics. 5-13. 5.3.1. Image Formation Model. 5-13. 5.3.2. Photometric Reflectance Model. 5-15. 5.3.3. Chromatic Invariants. 5-16. 5.3.4. Measurements. 5-18. 5.4 Experimental Results and Discussions. 5-19. 5.4.1 Chromatic Characteristic Selection. 5-20. 5.4.2 Intrinsic Image Extraction. 5-27. 5.5 Concluding Remarks. 5-36. Chapter 6 Concluding Remarks and Future Work 6.1 Concluding Remarks. 6-2. 6.1.1. Contributions. 6-4. 6.2 Future Work. 6-5. Appendix A The PURDUE RVL SPEC-DB Color Image Database. Bibliography. A-2 B-1. Publication List Refereed Papers. P-2. Submitted Papers. P-2. International Conference Papers. P-3 C-3.

(11) Characteristic image decomposition from a single image. C-4. Domestic Conference Papers. P-4. Theses. P-6.

(12) Characteristic Image Decomposition from a Single Image. Figure List Chapter 1 Introduction Figure 1.1. Illumination effect examples, (a) shadow, (b) highlight reflection, (c) interference reflection.. 1-3. Figure 1.2. An interference image (a) original image; (b) an image taken by a camera with a polarizing filter manually adjusted to reduce as much reflection as possible; (c) the image obtained by rotating the polarizing filter 90° in (b). 1-6 Figure 1.3. Mona Lisa, (a) original picture, (b) gray scale image, (c) pentagon distortion image, (d) partial image, (e) brush stroke image, (f) line drawing by hand [ENC 07], (g) topographical contour image, (h) edge image.. 1-16. Chapter 2 Computational Framework for Characteristic Image Decomposition Figure 2.1. The outline of characteristic image decomposition process.. 2-2. Figure 2.2. The flowchart of the computational framework of characteristic image decomposition from a single image.. 2-4. Chapter 3 Interference Reflection Separation Figure 3.1. (a) An interference image ; (b) an image taken by a camera with a polarizing filter manually adjusted to reduce as much reflection as possible; (c) the image obtained by rotating the polarizing filter 90° in (b).. Characteristic image decomposition from a single image. 3-3. F-1.

(13) Characteristic image decomposition from a single image. Figure 3.2. Bock diagram for the proposed approach to decomposing an interference image into its reflection and object images. 3-5 Figure 3.3. Flowchart for separating reflection and object images from a single image.. 3-7. Figure 3.4. The intermediate results of an interference image separation (a) Input image, I; (b)(c)(d) red, green and blue component images of (a), Ir, Ig, Ib, and I = (Ir, Ig, Ib); (e)(f)(g) horizontal edge magnitude images of the red, green and blue component image, Erh, Egh, Ebh; (h) horizontal edge magnitude image Eh = (Erh, Egh, Ebh); (i)(j)(k) vertical edge magnitude images of the red, green and blue component image, Erv, Egv, Ebv; (l) vertical edge magnitude image Ev= (Erv, Egv, Ebv). Figure 3.5. Formation of the interference image.. 3-8 3-10. Figure 3.6. The intermediate results of an interference image separation (a) texture map X; (b) cartoon map C; (c) horizontal edge magnitude image Xh; (d) vertical edge magnitude image Xv; (e) horizontal edge magnitude image Ch; (f) vertical edge magnitude image Cv; (g) horizontal strong edge magnitude image Fh from (c) and (e); (h) vertical strong edge magnitude image Fv from (d) and (f); (i) horizontal weak edge magnitude image Uh from (c); (j) vertical weak edge magnitude image Uv from (d); (k) blur map Bv of vertical edge direction; (l) blur map Bh of horizontal edge direction. Figure 3.7. The intermediate results of an interference image separation (a) vertical object edge map oh; (b) horizontal object edge map ov; (c) vertical reflection edge map rh; (d) horizontal reflection edge map rv; (e) object image before F-2. 3-12.

(14) Figure List. refinement; (f) reflection image before refinement; (g) region map S; (h) region edge map Se; (i) refined vertical object edge map Oh; (j) refined horizontal object edge map Ov; (k) refined vertical reflection edge map Rh; (l) refined horizontal reflection edge map Rv; (m) object image O; (n) reflection image R. Figure 3.8. Flowchart of the proposed retinex based blur measure. 3-14 3-19. Figure 3.9. Gradient magnitude at an edge point p(x,y) in the direction. ζ of the gradient. (a) sharp edge and (b) blurred edge. 3-20. Figure 3.10. Illustration of the blur measure for an edge point p’(x, y), (a) the edge point location, (b) fitting em(ζ) by normal distribution φ(ζ).. 3-21. Figure 3.11. (a) The input image, (b) the split result, (c) the merged result, (d) the boundary map.. 3-26. Figure 3.12. Classification refinement, (a) the nearby area close to p(x, y) in S, (b) the edge map Se, (c) the classified edge map before refinement.. 3-32. Figure 3.13. Results of interference image decomposition, where each interference image and its corresponding separation results are shown in a row. The first column images are the input interference images I, the second column images are the separated reflection images R, and the third column images are the separated object images O.. 3-34. Figure 3.14. More results of interference images decomposition, where each interference image and its corresponding separation results are shown in a row. The first column images are the input interference images I, the second column images are the separated reflection images R, and the third column images are the separated object images O.. 3-36. F-3.

(15) Characteristic image decomposition from a single image. Chapter 4 Dichromatic Reflection Decomposition Figure 4.1. The interaction of a light hits a surface of opaque objects. 4-2 Figure 4.2. Flowchart for extracting dichromatic reflection images from a single image.. 4-4. Figure 4.3. Specular estimation map, (a) a sample image, (b) eI map, (c) eS map. Figure 4.4. Flowchart for diffuse estimation map, Pi, generation.. 4-9 4-10. Figure 4.5. The processes to calculate the diffuse estimation image of Fig. 4.3(a), (a) chromaticity map, Ci, (b) maximum chromaticity map, M, (c) maximum intensity map, IM, and (d) diffuse estimation map, Pi with Da = 0.5.. 4-11. Figure 4.6. A sample distribution of pixels of a single color surface (including highlight area) on the maximum chromaticityintensity space.. 4-13. Figure 4.7. Dichromatic reflection separation result (synthetic image), (a) input image, (b) fuzzy integral result, (c) defect compensation result, (d) specular image, and (e) diffuse image.. 4-18. Figure 4.8. Dichromatic reflection separation results (synthetic images), (a), (d) input images, (b), (e) specular images, and (c), (f) diffuse images.. 4-20. Figure 4.9. Dichromatic reflection separation results (real images), (a), (d) input images, (b), (e) specular images, and (c), (f) diffuse images. Figure 4.10. Dichromatic reflection separation results, (a) input image, (b) the proposed specular image, and (c) Tan and Ikeuchi’s specular image [TAN 05], (d) the proposed diffuse image, and (e) diffuse image of [TAN 05], (f). F-4. 4-21.

(16) Figure List. enlargement of (a), (g) enlargement of (d), (h) enlargement of (e).. 4-22. Chapter 5 Intrinsic Image Extraction Figure 5.1. Proposed flowchart for extracting intrinsic images from a single image.. 5-4. Figure 5.2. Hierarchical edge classification process.. 5-7. Figure 5.3. Example illustrating edge classification.. 5-9. Figure 5.4. Scenes under (a) diffuse, (b) ambient, and (c) direct lighting conditions.. 5-22. Figure 5.5. Degrees of invariance of chromatic characteristics of the same objects under different lighting conditions.. 5-24. Figure 5.6. Degrees of invariance of chromatic characteristics of distinct objects under (a) diffuse lighting, (b) ambient lighting, and (c) direct lighting.. 5-25. Figure 5.7. Degrees of invariance of chromatic characteristics of distinct objects under different lighting conditions.. 5-27. Figure 5.8. A synthetic image example: (a) input image, (b) reflectance image, (c) illumination image, and (d) illumination image by I / R.. 5-28. Figure 5.9. Examples using synthetic images: (a) input images, (b) reflectance images, and (c) illumination images.. 5-30. Figure 5.10. Examples using real scenes: (a) input images, (b) reflectance images, and (c) illumination images.. 5-31. Figure 5.11. Examples using real scenes: (a) input images, (b) reflectance images, and (c) illumination images.. 5-32. Figure 5.12. The importance of edge classification: (a) input image, (b) edge classification map, (c) illumination image, (d) improved edge classification map, and (e) improved illumination image.. 5-33 F-5.

(17) Characteristic image decomposition from a single image. Figure 5.13. Modify the edge classification result by noise removal: (a) input image, (b) edge classification map, (c) reflectance image,(d) illumination image, (e) edge classification map after noise removal, (f) resulting reflectance image, and (g) illumination image.. 5-35. Chapter 6 Concluding Remarks and Future Work Appendix Figure A.1. Images used from the PURDUE RVL SPEC-DB Color Image Database.. F-6. A-22.

(18) Characteristic Image Decomposition from a Single Image. Table List Chapter 1 Introduction Table 1.1. The characteristic components to be decomposed from a single image. 1-5. Chapter 2 Computational Framework for Characteristic Image Decomposition Chapter 3 Interference Reflection Separation Chapter 4 Dichromatic Reflection Decomposition Chapter 5 Intrinsic Image Extraction Chapter 6 Concluding Remarks and Future Work Appendix. Characteristic image decomposition from a single image. T-1.

(19) Chapter 1 Introduction. Characteristic image decomposition from a single image.

(20) Characteristic image decomposition from a single image. 1.1 Motivation Images provide much greater information than words. It is effortless for humans to comprehend a complex image. Nevertheless, it is an extremely difficult task for computers to interpret and understand images. Typically, computer vision applications may consist of sequentially arranged processing processes, including: (1) low-level processing, such as: edge detection, thinning, image segmentation, feature extraction, and (2) high-level processing, such as: object detection and tracking, and recognition. Much of computer vision research, e.g., object detection and tracking [HAN 04, WAN 04, WAN 06, HAV 07, PAN 07, MOR 07], and recognition [BIN 03, ROY 03, CHA 04, CHU 04], have had successful results in limited environmental conditions. However, they often failed if the constraints are loosened in real world scenes. One of the major reasons is that high-level processing (object detection and tracking, and recognition) relies on the performance of the low-level processing results. It is impossible to expect good results of object tracking and recognition with low quality of feature extractions. The quality of low-level processing is often degraded by the illumination variations, and for real scenes, the illumination is uncontrollable, and so shadows and highlight reflections are unavoidable. Thus, in order to avoid the issues caused by shadows and highlight reflections, many studies assume perfect or controlled indoor lighting conditions. Thus, one of the most common restrictions imposed on vision algorithms is the illumination condition. Consequently, techniques which are able to remove illumination restrictions will be valuable to make the computer vision algorithms mentioned above applicable to more general real scenes. In this dissertation, a solution is proposed to eliminate the illumination restrictions, including interference reflections, highlights and shadows.. 1-2.

(21) Chapter 1 Introduction. (a). (b). (c). Figure 1.1. Illumination effect examples, (a) interference reflection, (b) highlight reflection, (c) shadows.. First, oftentimes when viewing objects through a glass cover at particular angles, the glass will reflect a scene that is located in front of the glass. This reflection will interfere with the object behind the glass. This situation is called an interference reflection, and the image formed under this condition is called the interference image. When taking pictures of objects behind a transparent glass cover, the interference reflection is hard to avoid, and photographers usually use a polarized filter to eliminate the interference reflection. For example, Fig. 1.1(a) demonstrates the picture of a shop window with interference reflection, where the displayed watches are combined with the reflection of the street image. Without a polarized filter, the proposed technique can still remove the undesired reflection image by separating the interference image into its component images. The interference image is made up of two types of images, the reflection image, which contains the reflected image of objects in front of the glass cover (e.g. the reflected street scene in Fig. 1.1(a)), and the exhibition object image, which contains the exhibition objects behind the glass cover in the image (e.g. the. 1-3.

(22) Characteristic image decomposition from a single image. exhibition watches in Fig. 1.1(a).) In this dissertation, they are simply called the reflection and object images. Next, in the case of smooth surface objects, highlight reflections often exist and depend on the viewing direction of the camera angle of surfaces and illumination source. Highlight reflections may cause vision algorithms to failure and decrease the quality of their results. The dichromatic reflection images decomposition technique proposed in this dissertation is one solution to this problem. Dichromatic reflection images consist of two types of images, the specular image, which contains highlight effects, and the diffuse image, which contains the remaining reflections from the objects in the scene. Without worrying about the highlight effects, the vision algorithms can be applied to more general cases. Fig. 1.1(b) illustrates an example of highlight reflections, which may lead vision algorithms to fail to perform vehicle tracking and counting at an intersection. In this example, the highlight reflection distorts the shape of vehicles, which may then be miss-tracked or misclassified. Third, the intensity of an image reveals the brightness of a scene, which in turn is determined by two major factors, the amount of light incident on the scene and the reflectance of the scene. As a consequence, an image is often modeled as a product of two components: illumination and reflectance [Far 99, SZE 00, YUI 99]. The images formed from these two components are referred to as the illumination image and the reflectance image, and together are called the intrinsic images [BAR 78, FIN 04, MAR 02, SIN 93] of the original image.Using the algorithm proposed in this dissertation, the intrinsic images can be extracted from a single image. Vision algorithms can then be applied directly to the reflectance image to avoid illumination effects and shadows, greatly improving their applicability. Fig. 1.1(c) illustrates an example of a vehicle. 1-4.

(23) Chapter 1 Introduction. tracking and counting application at a road intersection, where vision algorithms may be confused by shadows. In this example, the shadow of the bus connects the bus with a taxi and a motorcycle. The vehicle counter algorithm requires additional computations to remove the shadow to get the correct count. In this dissertation, the components of interference images, dichromatic reflection images, and intrinsic images are called characteristic components of images, and are summarized in Table 1.1. The proposed technique will extract the characteristic components from a single image and is referred to as the characteristic images decomposition process. The proposed algorithm for characteristic image decomposition has four major steps: boundary detection, evidence extraction, boundary classification, and characteristic image reconstruction, and is described in Chapter 2. In addition, applications of this algorithm, i.e., interference images separation, dichromatic reflection images decomposition, and intrinsic images extraction, will be introduced in Chapters 3, 4, and 5, respectively.. Table 1.1. The characteristic components to be decomposed from a single image Interference images. Dichromatic. Intrinsic images. reflection images Characteristic. Reflection image. Specular image. Illumination image. components. Object image. Diffuse image. Reflectance image. 1-5.

(24) Characteristic image decomposition from a single image. 1.2 Literature Review In this section, recent research results and related methodologies of interference images, dichromatic reflection images, and intrinsic images decompositions are discussed. The surveys are discussed in Section 1.2.1, 1.2.2, and 1.2.3, respectively.. (a). (b). (c). Figure 1.2. An interference image (a) original image; (b) an image taken by a camera with a polarizing filter manually adjusted to reduce as much reflection as possible; (c) the image obtained by rotating the polarizing filter 90° in (b).. 1.2.1 Interference Image Decomposition Many methods [AGR 05, BRO 03, FAR 99, IRA 92, LEV 04, LEV 04-1, LEV 07, SAR 04, SAR 05, SHE 99, SZE 00, TSI 03] have been proposed for extracting component images from interference images. One of the common methods is the employment of polarizing filters mounted in front of cameras. The polarizer angle that minimizes the reflections from a transparent cover has been called the Brewster’s angle. This angle depends on the viewing direction of the camera, the orientation of the cover, and the positions of lighting sources. In practice, the Brewster’s angle is difficult to. 1-6.

(25) Chapter 1 Introduction. reach by hand. Figure 1.2(b) shows an image taken by a camera with a polarizing filter manually adjusted to reduce as many reflections as possible from the glass shield. The image obtained by further rotating the polarizing filter 90° is displayed in Figure 1.2(c). This image is hardly visible in the original image (see Figure 1.2(a)). The proposed technique, which in a sense acts as a pseudo-polarizer, decomposes an interference image into its object and reflection images. The interference decomposition problem is an ill-posed problem because two unknowns Io and Ir are to be determined from a single input Ii. Additional information should be included. In [BRO 03, FAR 99, SHE 99], multiple images acquired by rotating at different angles a polarizing filter accompanying a camera were employed. The techniques of blind source separation [BRO 03], independent components analysis [FAR 99], and polarization-based decorrelation [SHE 99] have been applied to polarization images to decompose them into object and reflection images. In [IRA 92, SAR 04, SAR 05, TSI 03, ZIB 01], video sequences were used. Unlike polarization images, video images are temporally correlated. The techniques of temporal integration [IRA 92], layer information exchange [SAR 04], global-to-local space-time alignment [SAR 05], constrained least squares and compositing operations [TSI 03], as well as stereo matching [ZIB 01] have successfully extracted component images from video sequences. The above methods using multiple images have suffered from the indefinite number of images to be employed to accomplish the purpose. The number of images to be used depends on working conditions, such as the distribution of ambient lighting sources, the orientation of the transparent cover, and the viewing direction of the camera. Furthermore, multiple images increase both the processing time and storage. In order to. 1-7.

(26) Characteristic image decomposition from a single image. reduce the number of images being used, stereo images [TSI 03] and flash/no-flash image pairs [AGR 05] have been shown to provide enough information for solving the interference decomposition problem. However, extra devices and equipments were incorporated. This somewhat restricts the practical applications of the techniques. Recently, researchers [LEV 04, LEV 04-1, LEV 07] have turned their attentions to extract component images from a single interference image. Levin, Zomet and Weiss [LEV 04] assumed that interference decomposition should result in component images having fewer edges and corners than the original image. Regardless of the realism of this assumption, there are still a large number of ways to decompose an interference image. In order to reduce the search space of possible decompositions, Levin et al. divided the input image into small overlapping patches. For each patch, a number of patch pairs were selected from a database of natural images. Each selected patch pair approximately summed to the input patch. Suppose there are n overlapping patches extracted from the input image. Let mi denote the number of patch pairs selected from the database for input pair i. There are in total. ∏ i=1 mi n. possible decompositions for the. input image. A max-product belief propagation process determined as the result the decomposition with the lowest predefined cost in the set of possible decompositions. The belief propagation process, which is actually a probabilistic relaxation labeling process, is iterative in nature. It could take a long time to examine all the possible decompositions. In fact, a fixed number of iterations were performed. As a consequence, the returned result is often not optimal. Moreover, the belief propagation process may not converge when complex images are considered. In [LEV 04-1, LEV 07], Levin and Weiss considered a simpler problem, in which the user provides labels of component images for a number of critical gradients in the. 1-8.

(27) Chapter 1 Introduction. interference image. However, the problem is still ill-conditioned. A sparsity prior was introduced, which states that the output of any derivative filter tends to be sparse. More explicitly, the histogram of the output of a derivative filter is peaked at zero and fall off rapidly out to the two extreme ends of the histogram. A probability function characterized by the sparsity prior was constructed, which served as the criterion of interference decomposition. An interference image was decomposed by maximizing the probability function using the iterative reweighted least squares technique. In view of the iterative character of this technique, the decomposition of an interference image could take up to 12 minutes. However, the major embarrassment of the Levin and Weiss’ method may be the user involvement. In this study, we propose a technique to automatically label boundary pixels. To this end, a number of evidences are derived from the input interference image, based on which boundary pixels are classified as being object- or reflection-related. Since no iteration is involved in the reconstruction of component images, our method greatly reduced the processing time and achieved reasonable results.. 1.2.2 Dichromatic Reflection Images Decomposition Specular reflection is often an obstacle to computer vision applications because it generates saturated highlight spots and also changes color properties which annoy object recognition processes. Most computer vision applications assume perfectly diffuse surfaces and ignore the specular reflections. However, specular reflection is inevitable in natural scenes especially outdoors when the sun is shinning, and a method to separate specular and diffuse reflections is desired. In addition, the separated specular and diffuse reflection images can provide extra information about the types and. 1-9.

(28) Characteristic image decomposition from a single image. smoothness of surface materials. Many techniques have been developed to separate specular and diffuse reflections from images. They can be broadly categorized into three classes: multiple images, segmented color surfaces in a single image, and single image using pixel-based analysis. First, in early research, multiple images were used. Wolff and Boult [WOL 91] proposed a polarization-based method to separate reflection components. A set of polarization filters with different polarization angles are utilized to generate several gray scale images with different polarizing effects, from which highlights are located. Later, Nayar et al. [NAY 97] extended their work by integrating color and polarization information. Criminisi et al. [CRI 02] uses a collection of images taken under different locations of light sources to create a spatial-temporal EPI (Epipolar Plane Images) plane. EPI-strips are extracted and individual EPI-strips are analyzed and decomposed into specular and diffuse components. Lin and Shum [LIN 01] proposed a neutral interface reflection model for separating the reflection components of color images. From two photometric images, the RGB intensities of the two reflection components are computed for each pixel using a linear model of surface reflectance. Many other researchers used multiple images to separate specular and diffuse components [LIN 02, LEE 92, SAT 94]. All of them have successful results on multiple images; however, using several input images is not appropriate for many situations. Later, researchers tried to solve the separation problem using segmented color surfaces in a single image, and analyzed one segmented color surface at a time. To analyze colors, a dichromatic reflection model was proposed by Shafer [SHA 85] and used by Klinker et al. [KLI 90] and Bajscy et al. [BAJ 96] to develop specular and diffuse separation methods. They analyzed the color distribution on the surface of a. 1-10.

(29) Chapter 1 Introduction. single color object, and transformed it to their specified color space to separate specular and diffuse pixels based on the color information. One common limitation of these methods is that they all require prior color segmentation to mark a single color area before performing color analysis. Color segmentation techniques have been applied to help categorize color regions; however, textured surfaces and complex scenes are very common in the real world, and the applications of their methods are thus limited. Recently, researchers have tried to use pixel-based approach to avoid the above limitations of single color surfaces. Tan et al. [TAN 03] successfully used an image inpainting technique to remove highlights based on the local information of pixels. Image inpainting technique is to modify an image in a non-detectable form, and its applications include the restoration of damaged paintings and photographs, and the removal or replacement of selected objects. Unlike traditional inpainting, occluded image regions are directly filled by neighboring pixels. Here, highlight color analysis with uniform illumination color assumption are utilized to estimate the underlying diffuse color. The experimental results demonstrate their method works well. However, it requires manual interaction to mark highlights, and image inpainting methods are iterative in nature. Mallick et al. [MAL 06] transformed image pixels to a SUV color space where specular pixels have particular grouping areas in S channel. They proposed a partial differential equation (PDE) to describe the property, and the PDE iteratively erodes the specular component at each pixel to remove the specularity. For different image types, i.e., textureless and highly textured scenes, a family of PDEs with multi-scale morphological templates are designed. The experimental results show that their method works well.. 1-11.

(30) Characteristic image decomposition from a single image. Tan and Ikeuchi [TAN 05] proposed a specular-free image which estimates diffuse surface. Note that the diffuse colors are not kept in the specular-free image. The specular-free image is utilized to perform iterative operations (include specularity reduction and diffuse verification) to decrease the specularity of pixels and get diffuse values. Their method produced satisfactory separation results. However, pixel-based approaches require iterative operations, which consume a lot of computation time to the pixel-wise local information to separate specular and diffuse components. In this dissertation, a feature-based approach to separate specular and diffuse components from a single image is proposed. The proposed method utilizes specular and diffuse estimation, and a fuzzy integral classification algorithm to classify the edge pixels. The proposed method requires neither color segmentation nor iterative pixel-based operations. The key idea of the proposed method is to classify edges of the input image into either specular-related or diffuse-related, and to use integration to reconstruct specular and diffuse images. The proposed method produces satisfactory separation results in a short time. In addition, comparisons with the recent work of Tan and Ikeuch [TAN 05] are provided.. 1.2.3 Intrinsic Image Decomposition In many computer vision applications, it is desirable that the reflectance and illumination components be decomposed from the input image. Both components have their own advantages. Since the reflectance component is related to the scene characteristic, the reflectance image in principle remains constant under different illumination conditions. For applications such as object recognition [CHA 04], pattern classification [LEU 99], scene interpretation [FAN 03], and visual surveillance [KAM. 1-12.

(31) Chapter 1 Introduction. 00, MAT 03, TER 99], it is preferable to use reflectance images. The illumination component varying with different lighting conditions can be used for tasks such as illumination assessment [DRE 03, FIN 06], shading analysis [BEL 01, FUN 92, PAR 03], color constancy [FIN 01], and geometric modeling [SHA 03]. Decomposing an image into its reflectance and illumination components is an ill-posed problem [BAR 78]. There are two unknowns (illumination and reflectance components) that are to be derived from one given data (the input image). Additional information is needed to separate the components. Weiss [WEI 01] used multiple images. Let I i (i = 1,..., n) be a set of images taken of a scene under different illumination conditions. Since the reflectance component is assumed to be constant, say R, a set of n equations, I i = R × Li (i = 1, ... , n), can be constructed, where Li is the illumination component of image I i . However, this set of n equations is still not enough to solve for the n + 1 unknowns (R and Li ). Weiss further introduced a sparseness assumption [SIM 97], which states that the filtered images obtained by applying gradient operators to the input images are sparse (i.e., they contain mostly zeros) so that the histograms of the filtered images can be fitted with a Laplacian function. With this assumption, the decomposition problem becomes solvable. Weiss then estimated R and Li using a maximum likelihood technique. Yuille et al. [YUI 99] also used as the input data a set of images taken of an object under different and unknown lighting conditions. A singular value decomposition technique was applied to the images to separate the images into components depending on surface characteristics (geometry and albedo) and illumination conditions. Based on the extracted surface characteristics, a generative model of the object [GEO 00], which approximates the object’s appearance under a restricted range of illumination conditions, was determined. 1-13.

(32) Characteristic image decomposition from a single image. Since multiple images were used by Weiss and Yuille et al., the applicability of their techniques is somewhat limited. Tappen et al. [TAP 02] proposed a method for recovering intrinsic components from a single image. A set of derivative filters are first applied to the input image giving rise to a set of derivative images. The pixels of the derivative images are classified as being reflectance-related or illumination-related based on their color and intensity. However, unsatisfactory results were observed. Tappen introduced a process, called the generalized belief propagation process, to improve the results. Thereupon, a de-convolution process was applied to the classified derivative images to obtain the intrinsic images of the input image. The Tappen method took about six minutes to categorize pixels and another six minutes to perform the generalized belief propagation process. To use with real-time applications, the time complexity of the Tappen method should be reduced. Recently, Matsushita et al. [MAT 03] introduced an illumination eigenspace into Tappen’s computational framework. The eigenspace, which is built in advance, provides information for categorizing the pixels of derivative images. Since no information is computed during pixel classification, the Matsushita approach can operate in real time. However, over a period of 120 days Matsushita collected a set of 2048 images from a scene for generating its illumination eigenspace. Apparently, Matsushita’s method can not be applied to time-varying scenes (i.e., dynamic scenes). To be applicable to dynamic scenes, the information for classifying pixels of derivative images must be computed directly from the input image, and for real-time applications, the computation should be efficient.. 1-14.

(33) Chapter 1 Introduction. 1.3 The Proposed Techniques An image usually consists of millions of pixels of different intensities (R, G, and B color values). The meaning of an image is not simply the intensity of pixels, but rather in recognizable groups of pixels which humans see as one entity. This is supported by the theory of the human recognition of objects, recognition-by-components [HOF 84, BIE 87]. The perceptual recognition of objects is conceived of as a process where an input image is segmented into an arrangement of simple geometric components. The recognition-by-components theory is based on how people recognize objects in an image by decomposing their shapes into basic geometrical parts. It also answers why people can recognize objects even when the intensity changes considerable from its original value. Several modifications to the image in Fig. 1.3(a) are shown in Fig. 1.3(b) - 1.3(h). In all cases people can still easily recognize Mona Lisa. Note that the topographical contour reproduces the look of a topographical map by covering the image with contour lines. All of them have different intensities of pixels, and even the sizes and scales of images are not the same. However, humans have no problem recognizing them as Mona Lisa. This can be explained by the recognition-by-components theory, and if the major components of Mona Lisa (i.e., the face, and eyes) are kept in the modified images, no matter how the modifications are applied on the image, the Mona Lisa can be still recognized. From all the modifications above, people can see that edges represent most of the meaning of the picture as in Fig. 1.3(f) and (g).. 1-15.

(34) Characteristic image decomposition from a single image. (a). (b). (c). (d). (e). (f). (g). (h). Figure 1.3. Mona Lisa, (a) original picture, (b) gray scale image, (c) pentagon distortion image, (d) partial image, (e) brush stroke image, (f) line drawing by hand [ENC 07], (g) topographical contour image, (h) edge image.. An edge is defined as an imaginary line that separates two regions with different characteristics, e.g. intensity, color, illumination, reflectance, specular, and diffuse, etc. Accordingly, edge detection techniques are an essential technique for computer vision, and edge detection methods can be categorized [SAL 07] as spatial domain, frequency domain, and multi-resolution analysis. Classical spatial domain techniques try to find the maximum gradient values to detect edges (Sobel, Prewitt, Canny [CAN 86], Roberts). 1-16.

(35) Chapter 1 Introduction. or the zero crossings of the second derivative (Laplacian of Gaussian). Edge information exists in the high frequency area of the Fourier transform of an image. Fourier analysis reveals global information about edges, and it is not suitable to detect specific local edges. Multi-resolution analysis uses hierarchical structures to detect edges in different resolutions. Saleem et. al. [SAL 07] implemented a multi-resolution edge detector using Wavelet transform, which are less sensitive to noise since noise edges do not appear at low resolutions. Edges in a blurry image [WU 07] can also be better detected using multi-resolution approach. However, weak or isolated edges may be treated as noise and ignored. In this dissertation, all the edges in the image are required to be classified to compose the characteristic components, and for this a traditional edge gradient mask is the most suitable. In this dissertation, based on the idea that edges are the key to recognizing the meaning of images, a computational framework is developed for characteristic image decomposition from a single image using environmental supporting information to classify the edges of the image. The classified edges are separated into desired components, and each component is integrated to form the target characteristic image. Three applications of this computational framework (interference reflections, highlight reflections, and intrinsic images) are developed in this dissertation. For interference reflections, a technique for separating reflection and object components of a single interference image in an automated manner is presented. The key idea of the proposed method is to classify edges of the interference image into either reflection or object, and to use integration to reconstruct reflection and object images. The method utilizes TV model, blur measure, and region segmentation results as evidence with fuzzy integral technique to classify the edge pixels. Based on the. 1-17.

(36) Characteristic image decomposition from a single image. classification results of edge pixels, an integration method is applied to reconstruct the reflection and object components of the input image. The experimental results have demonstrated that the proposed method can perform separation of a single interference image effectively with small misadjustments and rapid convergence. For highlight reflections, a feature based technique for separating specular and diffuse components of a single image is presented. In the proposed approach, Shafer’s dichromatic reflection model is utilized, which assumed light reflected from a surface point is additively composed of diffuse and specular reflections. The idea behind the proposed method is to classify the boundary pixels of the input image as specular or diffuse. A fuzzy integral process is proposed to classify boundary pixels based on their local evidences, including specular and diffuse estimation information. Based on the classification result of boundary pixels, an integration method is used to reconstruct the specular and diffuse components of the input image. The experimental results demonstrate that the proposed method can perform dichromatic reflectance separation effectively with small misadjustments and rapid convergence. For intrinsic images, the proposed approach first convolves an input image with a prescribed set of derivative filters. The pixels of the derivative images are classified as reflectance or illumination according to three measures: chromatic, intensity contrast and edge sharpness, which are calculated in advance for each pixel from the input image. Finally, an integration process is applied to the classified derivative images to obtain the intrinsic images of the original image. Both synthetic and real images have been utilized in the experiments. The results reveal the feasibility of the proposed technique in rapidly and effectively decomposing intrinsic images from a single image.. 1-18.

(37) Chapter 1 Introduction. 1.4 Contributions There are four major contributions from this dissertation: (i) The computational framework for characteristic image decomposition from a single image The proposed computational framework which performs characteristic image decomposition from a single image has four major steps: boundary generation, information extraction, boundary classification, and image composition. This framework can be applied to many valuable applications, some of which are addressed in this dissertation, such as interference reflection separation, dichromatic reflection decomposition, and intrinsic image extraction from a single image. The decomposed components are useful for making computer vision algorithms applicable to more general and realistic scenes. (ii) The solution to the reflection image separation by interference images separation Interference images extraction can separate objects reflected by a glass cover and objects behind the glass cover in a single image. The proposed separation technique is valuable to vision applications since it can eliminate annoying reflections in an image. Unlike previous researches, the proposed method is fast, fully automatic, and requires no manual operations or iterative operations. (iii) The solution to highlight reflection removal by dichromatic reflection image extraction The dichromatic reflection image extraction algorithm can eliminate highlight reflections from the input image and help avoid errors caused by highlights in computer vision applications. Unlike previous research, the proposed method has no color segmentation or iterative operations.. 1-19.

(38) Characteristic image decomposition from a single image. (iv) The solution to shadow removal by intrinsic image extraction Intrinsic image extraction can extract the illumination and reflection images from a single input image. In many computer vision applications, it is desirable that the reflectance and illumination components be separated from the input image. The proposed algorithm is useful to in computer vision applications which are plagued by the illumination conditions. However, this task is not at all simple because it is an ill-conditioned problem. Previous researchers have relied on multiple images to solve this problem. But, the use of multiple images restricts the application domain. To overcome this limitation, the technique proposed in this dissertation uses a single image.. 1.5 Organization of this Dissertation In this dissertation, a technique for intrinsic decomposition from a single image and its applications is proposed. This dissertation is organized as follows. The computational framework for intrinsic decomposition is addressed in Chapter 2. The framework has four steps: boundary generation, information extraction, boundary classification, and image composition. Next, applications of the computational framework, to interference images separation, dichromatic specular reflection images extraction, and intrinsic images extraction, are presented in Chapters 3, 4, and 5, respectively. In these chapters, a brief introduction and literature review is given, and then the design and system architecture is presented. Next, the information extraction and classification methods will be discussed in each chapter. Experimental results and discussions are given at the end of. 1-20.

(39) Chapter 1 Introduction. each chapter. Finally, concluding remarks and future work are contained in Chapter 6.. 1-21.

(40) Chapter 2 Computational Framework for Characteristic Image Decomposition. Characteristic image decomposition from a single image.

(41) Characteristic image decomposition from a single image. Figure 2.1. The outline of characteristic image decomposition process.. 2.1 Characteristic Image Decomposition Design In view of that edges are one of the keys to understand an image; a computational framework for characteristic image decomposition from a single image based on the edges of the image is developed. The classified edges are separated into desired characteristic components, and each component is integrated to form the target image. Three applications of this computational framework (interference reflections, highlight reflections, and intrinsic images) are developed in this dissertation. The outline of the proposed characteristic image decomposition process is depicted in Fig. 2.1. Four major steps are designed to decompose the characteristic images from a single image: boundary detection, evidence extraction, boundary classification, and characteristic image reconstruction. In the first step, the boundary maps are generated from the input image. The boundary maps are composed of horizontal and vertical components, which are obtained by using spatial edge detection masks. Sometimes, particular characteristic boundary maps are required, which can be obtained with characteristic maps from the evidence 2-2.

(42) Chapter 2 Computational framework for characteristic image decomposition. extraction step. Second, the characteristic maps are extracted from the image at the evidence extraction step. The characteristic maps are designed to reveal the information about the desired characteristics hidden in the input image. For example, TV model, blur measure, and region segmentation results are the characteristic maps for interference boundary classification. Specular estimation and diffuse estimation are the characteristic maps for specular boundary classification, and chromatic, intensity contrast and edge sharpness are the characteristic maps for intrinsic boundary classification. In the third step, boundary classification classifies the edges in the boundary maps to related characteristic categories according to the characteristic maps obtained from the previous step. A variety of classification methods can be employed in this step, and a hierarchy-based architecture and a fuzzy integral classification method are employed in the experiments. Additionally, a defect compensation process may be added to increase the quality of the classification results. In the final step, an integration process [WEI 01] is applied to the classified derivative components, from which the characteristic images are reconstructed. The design of the four steps is detailed in the next section.. 2-3.

(43) Characteristic image decomposition from a single image. Figure 2.2. The flowchart of the computational framework of characteristic image decomposition from a single image.. 2-4.

(44) Chapter 2 Computational framework for characteristic image decomposition. 2.2 System Architecture From the outline of the proposed characteristic image decomposition depicted in Fig. 2.1, the computational framework is designed with the four major steps to decompose the characteristic images from a single image as shown in Fig. 2.2. The four steps are boundary detection, evidence extraction, boundary classification, and characteristic image reconstruction, which are described in the following sections.. 2.2.1 Boundary Detection Let I = (Ir, Ig, Ib) denote the input color image, where Ir, Ig and Ib are the red, green and blue components of the input image. It is assumed that the color components can be decomposed by their characteristic components. For example, from Shafer’s dichromatic reflection model, each color component Ii (i = r, g, b) is modeled as Ii = Si + Di, where Si and Di are the specular and diffuse components of Ii, respectively. For intrinsic image decomposition, each color component is modeled as Ii = Ri × Li, where Ri and Li are the reflectance and illumination components of Ii, respectively. Taking the logarithm of Ii, I i′ = log I i = log( R i × Li ) = log R i + log Li = R i′ + Li′ , results in an. additive composition of reflectance and illumination. The components are called the characteristic components of Ii. The decomposition process begins with boundary detection stage in which Ii is convolved with a horizontal derivative filter fh and a vertical derivative filter fv, resulting in derivative component images Iih and Iiv. In this dissertation, the Prewitt derivative filters are used. The boundary map Bi contains horizontal derivative values Iih and vertical derivative values Iiv, i.e., Bi = (Iih, Iiv).. 2-5.

(45) Characteristic image decomposition from a single image. 2.2.2 Evidence Extraction. In the second stage, evidence extraction, the characteristic maps are extracted from the input image as the information to be used in the next stage for classifying the pixels of derivative component images into characteristic categories of edge pixels. This step should be custom designed for each particular application. In this dissertation, different characteristic maps are designed for the proposed applications as described below. For interference images separation, the proposed method utilizes TV model, blur measure, and region segmentation results as evidence with fuzzy integral technique to classify the edge pixels. The TV model has the advantages of multiscale, but intensity-independent, decomposition, and easy parameter selection. Blur measure is a measure of the sharpness or blurriness of edges in an image can be useful for a number of applications in image processing. The region segmentation technique uses invariant chromatic characteristics and steerable filters to perform segmentation process. To perform highlight reflection separation, the specular estimation map and the diffuse estimation map are computed. The specular estimation map reveals the degrees to which pixels belong to highlight areas by calculating the pixel’s intensity and saturation information. The diffuse estimation map is calculated by removing specular effects from the input image by nonlinearly shifting each pixel’s intensity and maximum chromaticity. These two maps are used in the next stage for classifying the pixels of derivative components as specular or diffuse. For intrinsic image extraction, three measure maps (chromaticity, intensity contrast and edge sharpness) are calculated for each image pixel. The calculated measures will be used in the next stage for classifying the pixels of derivative component images into illumination or reflectance edge pixels.. 2-6.

(46) Chapter 2 Computational framework for characteristic image decomposition. 2.2.3 Boundary Classification. In the boundary classification step, the information obtained form the previous step is used to classify pixels of the boundary map to get the boundary classification map. The classified derivative component images can be used to generate the results in step 4. A variety of classification methods can be employed in this step. A hierarchy-based architecture is designed to classify the edge pixels as illumination or reflectance categories. A fuzzy integral classification is utilized to classify the edge pixels as specular or diffuse from the highlight image, and also the reflectance and object images from the interference image.. An additional process may be added to compensate for erroneous classifications results in areas low edge magnitudes, where the extracted information has a high noise content.. 2.2.4 Characteristic Image Reconstruction. The final step of the proposed computational framework is characteristic image reconstruction. In this step, an integration process [WEI 01] is applied to the classified derivative components, from which the characteristic images are composed. Denote the classified components as Si, Di and classified derivative components with filter j as Sij, Dij, respectively. From each convolution equation j, fj * Si = Sij, a reverse filter fjr is designed and convolved it with each equation as fjr * fj * Si = fjr * Sij. Summing all the equations over j as. 2-7.

(47) Characteristic image decomposition from a single image. ∑f. r j. j. * f j * S i = ∑ f jr * S ij , j. and. ∑f. r j. j. * f j * D i = ∑ f jr * D ij .. (2.1). j. To solve the specular Si and diffuse Di components in the convolution Eq. (2.1), let. η be the normalization function, such that. η ∗ (∑ f jr ∗ f j ) = δ , j. and δ is the Kronecker delta function. The specular and diffuse components can be calculated as. S i = η *( ∑ f jr * S ij ) , j = h ,v. D i = η *( ∑ f jr * D ij ) .. (2.2). j = h ,v. Note that fjr is a user-defined reversed function of fj, and η can be obtained after fjr is defined. Here, the reversed function is defined as fjr (p) = fj (-p). Practically, Eq. (2.1) can be solved directly by applying Fourier transform ℑ,. ⎛⎛ ⎞ ⎞ ⎛ ⎞ ℑ ⎜ ⎜ ∑ f jr * f j ⎟ * S i ⎟ = ℑ ⎜ ∑ f jr * S ij ⎟ . ⎜ j ⎟ ⎠ ⎝ j ⎠ ⎝⎝ ⎠. (2.3). By the convolution theorem,. ⎛ ⎞ ⎛ ⎞ ℑ ⎜ ∑ f jr * f j ⎟ ℑ S i = ℑ ⎜ ∑ f jr * S ij ⎟ ⎝ j ⎠ ⎝ j ⎠,. ( ). therefore,. ⎛ ⎛ ⎞ ⎛ ⎞⎞ S i = ℑ−1 ⎜ ℑ ⎜ ∑ f jr * S ij ⎟ / ℑ ⎜ ∑ f jr * f j ⎟ ⎟ , ⎜ ⎟ ⎠ ⎝ j ⎠⎠ ⎝ ⎝ j. 2-8. (2.4).

(48) Chapter 2 Computational framework for characteristic image decomposition. where ℑ-1 is the inverse Fourier transform. Likewise,. ⎛ ⎛ ⎞ ⎛ ⎞⎞ D i = ℑ−1 ⎜ ℑ ⎜ ∑ f jr * D ij ⎟ / ℑ ⎜ ∑ f jr * f j ⎟ ⎟ . ⎜ ⎟ ⎠ ⎝ j ⎠⎠ ⎝ ⎝ j. (2.5). After obtaining color components, Si and Di, the characteristic images, S and D, of the input image I are composed as S = (Sr, Sg, Sb). (2.6). and D = (Dr, Dg, Db).. (2.7). 2.3 Proposed Applications In this dissertation, a computational framework is developed for characteristic image decomposition from a single image using environmental supporting information to classify the edges in the image. The classified edges are separated into desired characteristic components, and each component is integrated to form the target image. Three applications of this computational framework (interference reflections, dichromatic reflections, and intrinsic images) are developed in this dissertation. The methodologies of interference reflection images, highlight reflection images, and intrinsic images extraction are described in Chapters 3, 4 and 5, respectively.. 2-9.

(49) Chapter 3 Interference Reflection Separation. Characteristic image decomposition from a single image.

(50) Characteristic image decomposition from a single image. The interference image is defined as the superpositioning of a reflection image and an object image. A technique for separating reflection and object components of a single interference image is presented. The proposed method does not try to separate reflection and object components directly from the interference image, but to classify edges of the interference image into either reflection or object related. Our method utilizes total variation (TV) method, blur measure, and region segmentation as evidence with a fuzzy integral technique to classify the edge pixels. Based on the results of edge pixel classification, the reflection and object components of the input image are reconstructed. Compared to previous published research, the proposed method is fast and requires no manual operations. The experimental results have demonstrated that the proposed method can perform separation of a single interference image effectively with small misadjustments and rapid convergence.. 3.1 Introduction Transparent covers made of glass or plastic are commonly used to protect demonstration objects in galleries, museums, jewel shops, and department stores. However, transparent covers easily reflect the scenes located in front of the covers while viewing exhibition objects behind the covers. Figure 3.1(a) shows an image taken for a kitchen in a restaurant through a glass shield. The reflection from the glass interferes with the view of the kitchen behind the shield. We call such an image an interference image. Figure 3.1(b) shows an image taken by a camera with a polarizing filter manually adjusted to reduce as many reflections as possible from the glass shield. The. 3-2.

(51) Chapter 3 Interference reflection separation. image obtained by further rotating the polarizing filter 90° is displayed in Figure 3.1(c). This image is hardly visible in the original image (see Figure 3.1(a)). The proposed technique, which in a sense acts as a pseudo-polarizer, decomposes an interference image into its object and reflection images. In this paper, a technique for eliminating the effect of reflections from an interference image is presented. Let Ii denote the input interference image and Io denote the image after removing the effect of reflections from Ii. We refer to Io as the object image, which contains the exhibition objects of our interest. Let Ir = Ii - Io, which is referred to as the reflection image. In this paper, Io and Ir are collectively called the component images of Ii. Although we are interested in object image Io, reflection image Ir preserves the information about the reflected scene, which can be of use in some applications.. (a). (b). (c). Figure 3.1. (a) An interference image ; (b) an image taken by a camera with a polarizing filter manually adjusted to reduce as much reflection as possible; (c) the image obtained by rotating the polarizing filter 90° in (b).. 3-3.

(52) Characteristic image decomposition from a single image. Many methods [AGR 05, BRO 03, FAR 99, IRA 92, LEV 04, LEV 04-1, LEV 07, SAR 04, SAR 05, SHE 99, SZE 00, TSI 03] have been proposed for extracting component images from interference images. They can be broadly categorized into three classes: using multiple images (acquired by rotating at different angles a polarizing filter accompanying a camera,) video sequences, stereo images and flash/no-flash image pairs. Extracting component images from a single interference image assuming that interference decomposition should result in component images having fewer edges and corners than the original image. A simpler problem to extracting component images from a single interference image, in which the user provides labels of component images for a number of critical gradients in the interference image. The literal review of interference images extractions have been discussed in Sec. 1.2.1. In this study, we propose a technique to automatically label boundary pixels. To this end, a number of evidences are derived from the input interference image, based on which boundary pixels are classified as being object- or reflection-related. Since no iteration is involved in the reconstruction of component images, our method greatly reduced the processing time and achieved reasonable results. The rest of this paper is organized as follows. The fundamental idea and the flowchart of the proposed interference image decomposition method are addressed in Sec. 3.2. The details of techniques, including total variation decomposition, blur measure, region segmentation, fuzzy integral, classification refinement, and component image reconstruction are addressed in Sec. 3.3. The experimental results of the proposed method are demonstrated in Sec. 3.4. Concluding remarks and future work are finally given in Sec. 3.5.. 3-4.

(53) Chapter 3 Interference reflection separation. 3.2 Interference Image Decomposition Figure 3.2 shows a block diagram for the proposed approach, which consists of four major steps: (1) boundary detection, (2) evidence extraction, (3) boundary classification, and (4) component image reconstruction. Let I = (Ir, Ig, Ib) represent the interference input image, where Ir, Ig and Ib are the red, green and blue components of the input image. The first three steps of the proposed approach are applied separately to each of the three color component images. The results are then integrated in the last step to form the object and reflection images of the input image.. Figure 3.2. Bock diagram for the proposed approach to decomposing an interference image into its reflection and object images.. 3-5.

(54) Characteristic image decomposition from a single image. Consider any color component image, say Ii, i = r, g, b. In the first step of boundary detection, Ii is convolved with a horizontal derivative filter fh and a vertical derivative filter fv. Let Eih and Eiv denote the resultant derivative magnitude images, respectively. In this study, Prewitt filters are used as the derivative filters. Thereafter, four evidence maps, including a cartoon map C, a texture map X, a blur map B, and a region map S are derived from image I. The first two maps are derived using a total variation method to be addressed in Sec. 3.3.1. The derivations of the blur map and the region map are discussed in Sec. 3.3.2 and Sec. 3.3.3, respectively. In the third step of boundary classification, edges are classified as being object- or reflection-related based on the first third evidence maps using a fuzzy integral approach. Moreover, the classification results is further refined based on the information of the region map. The reflection and object images are finally reconstructed from the associated edges in the last step. The flowchart of detail processes to decompose reflection and object images from the interference image is illustrated in Fig. 3.3. In the following, we address in depth the individual steps.. 3-6.

(55) Chapter 3 Interference reflection separation. Input single image I = (Ir, Ig, Ib) Ii = Oi + Ri, i=r, g, b. Region segmentation. Evidence extraction. Total variation decomposition. Region map S. Cartoon map C. Texture map X. Region edge map Se. Magnitude images Cih , Civ. Magnitude images Xih , Xiv. Derivative filters fh , fv. Magnitude images Eih , Eiv Boundary detection. Boundary classification. Strong edge magnitude image Fih , Fiv. Blur map B (Bh , Bv). Weak edge magnitude image Uih , Uiv. Object edge maps oih, oiv. Fuzzy Classification. Reflection edge maps rih, riv. Refined object edge maps Oih, Oiv. Classification refinement. Refined reflection edge maps Rih, Riv. Image reconstruction Component image reconstruction. Blur measure. Component images Oi, Ri. Interference images O, R. Figure 3.3. Flowchart for separating reflection and object images from a single image.. 3-7.

(56) Characteristic image decomposition from a single image. (a). (b). (c). (d). (e). (f). (g). (h). (i). (j). (k). (l). Figure 3.4. The intermediate results of an interference image separation (a) Input image, I; (b)(c)(d) red, green and blue component images of (a), Ir, Ig, Ib, and I = (Ir, Ig, Ib); (e)(f)(g) horizontal edge magnitude images of the red, green and blue component image, Erh, Egh, Ebh; (h) horizontal edge magnitude image Eh = (Erh, Egh, Ebh); (i)(j)(k) vertical edge magnitude images of the red, green and blue component image, Erv, Egv, Ebv; (l) vertical edge magnitude image Ev= (Erv, Egv, Ebv).. 3-8.

(57) Chapter 3 Interference reflection separation. 3.2.1 Boundary Detection In the first step of boundary detection, Ii is convolved with a horizontal derivative filter fh and a vertical derivative filter fv. Refer to the example shown in Fig. 3.4. The input image is displayed in Fig. 3.4(a), in which some books behind the glass of a bookcase and the reflected photographer wearing a plaid shirt from the glass are present. Figures 3.4(b), (c) and (d) show the red, green and blue component images of the input image denoted by Ir, Ig, and Ib, respectively. Fig. 3.4(e), (f) and (g) show the horizontal edge magnitude images of the red, green and blue component image denoted by Erh, Egh, and Ebh, respectively. For convenience, Fig. 3.4(h) shows their composition, Eh = (Erh, Egh, Ebh). Fig. 3.4(i), (j) and (k) show the vertical edge magnitude images of the red, green and blue component image denoted by Erv, Egv, Ebv, respectively. Fig. 3.4(l) shows their composition, Ev= (Erv, Egv, Ebv).. 3.2.2 Evidence Extraction Refer to Fig. 3.3, in the evidence extraction step, four kinds of characteristics are computed from the input image. The cartoon map C, the texture map X, and the blur map B serve as criteria for classifying the edges extracted in the previous step into reflection or object edges. The region map S is used as the criterion to refine the classified result. Refer to Fig. 3.5, the camera is focused on the exhibition object. Since exhibition objects are more close to the protection glass than scene objects, i.e., ls >> le, the reflection of the scene object is in general out of focus for the camera. The exhibition object image and the reflected scene object image are then hybrid as the interference image. As a consequence, the exhibition object typically has a relatively clearer appearance than that of the scene object while presenting in the image taken by. 3-9.

(58) Characteristic image decomposition from a single image. the camera. Both the intensity and the boundary magnitude of the exhibition object are larger than those of the scene object.. Figure 3.5. Formation of the interference image.. The total variation (TV) model [AN 08, CHA 04-1, CHE 06] to be addressed in Sec. 3.3.1 provides an elegant tool for decomposing an image I into two component images: a large-scale component image u and a small-scale component image v. Mathematically, f = u + v. Images u and v have commonly been referred to as the texture and cartoon images, respectively. The large-scale image has simpler object boundaries than the small-scale image. The results of TV model are texture map X and cartoon map C. Refer to the example image shown in Fig. 3.4(a). Figure 3.6(a) and (b) are the results of TV model, texture map X and cartoon map C, respectively. Basically, the reflections of scene objects are shown in the texture map X, and the exhibition objects are shown in 3-10.