使用透視法於肺部 PET/CT DICOM 影像之三維立體顯示

全文

(1)國立高雄大學資訊工程學系研究所碩士論文. 使用透視法於肺部 PET/CT DICOM 影像之三維立體顯示 Stereoscopic Display of Lung PET/CT DICOM Scans using Perspective. 研究生：陳玥汝撰指導教授：殷堂凱博士中華民國一百一年七月.

(2) 致謝首先誠摯的感謝指導教授殷堂凱博士，老師悉心的教導使我得以一窺醫學影像處理領域的深奧，不時的討論並指點我正確的方向，且總能在我迷惘時為我解惑，使我在這些年中獲益匪淺，老師對學問的嚴謹更是我輩學習的典範。本論文的完成另外亦得感謝西洋語文學系的黃柏瀧大力協助，因為有你的幫忙，使得本論文能夠更完整而嚴謹。也感謝大學時代同學們及惠普實習期間同事的經驗分享與支持，使我得以順利完成學業。兩年裡的日子，實驗室裡共同的生活點滴，學術上的交流、討論進度前一起緊張、趕作業的革命情感，和飯後的八卦閒聊都是寶貴的回憶和經驗，感謝眾位學長、同學、學弟妹的共同砥礪，你/妳們的陪伴讓兩年的研究生活變得絢麗多彩，尤其常常一起修課和幫了我很多忙的陳弘晉、黃聖文和黃群雄同學，恭喜我們順利走過這兩年。家人在背後的默默支持更是我前進的動力，沒有你們的體諒、包容，相信這兩年的生活將是很不一樣的光景。最後，謹以此文獻給我摯愛的雙親。. 陳玥汝謹誌於高雄大學 2012 年 7 月 17 日. i.

(3) 使用透視法於肺部 PET/CT DICOM 影像之三維立體顯示. 指導教授：殷堂凱博士國立高雄大學資訊工程所. 學生：陳玥汝國立高雄大學資訊工程所. 摘要. 隨著立體顯示技術進步，越來越多的影像利用此技術來提供二維平面影像所缺乏的深度資訊。本文將多張序列的二維的電腦斷層掃描影像截取胸腔肺部的部份建立為三維的立體模型並將正子攝影所計算出的標準攝取值(SUV)與電腦斷層掃描影像相疊，透過透視投影法取得比最大強度投影法(MIP)較好的立體效果並於主動式立體顯示呈現。並將其與傳統二維平面影像 MIP 和其它研究做出比較，除了增加視覺效果與擴展深度知覺外，立體顯示深度資訊還可以幫助診斷、應用在教學或手術路徑規劃。. 關鍵字：立體顯示, 透視, 視差, 正子攝影, 電腦斷層掃描, 標準攝取值. ii.

(4) Stereoscopic Display of Lung PET/CT DICOM Scans using Perspective Advisor: Dr. Tang-kai Yin Institute of Computer Science and Information Engineering National University of Kaohsiung Student: Yueh-ju Chen Institute of Computer Science and Information Engineering National University of Kaohsiung. ABSTRACT. In recent years, as stereoscopic display technology is applied more and more widely, more images tend to use it to provide the depth information that 2D images cannot bring. In this thesis, we present stereoscopic images using perspective projection of lung and display 3D effect on active shutter glasses system. Stereoscopic images are created by a calculation of standardized uptake value (SUV) on positron emission tomography (PET) and overlapped with computer tomography (CT). Than shown the difference with traditional 2D MIP image. Stereoscopic image using perspective and the others works. Applied stereoscopic technology to medical image processing to expend our vision from 2D to 3D and helpful for diagnosis such as improve diagnosis efficiency or surgical path planning. Keywords: stereoscopic, 3D display, parallax, perspective, Standardized Uptake Value (SUV), Positron Emission Tomography combined Computed Tomography (PET/CT). iii.

(5) Content 摘要 ........................................................................................................................................ ii ABSTRACT .......................................................................................................................... iii Content .................................................................................................................................. iv List of Figures ........................................................................................................................ vi Chapter 1. Introduction ...................................................................................................... 1. 1.1.. Related Works.......................................................................................................... 2. 1.2.. Motivation ................................................................................................................ 3. 1.3.. Thesis Contribution.................................................................................................. 5. 1.4.. Thesis Layout ........................................................................................................... 6. Chapter 2 2.1.. Medical Scans .......................................................................................................... 7. 2.1.1. 2.2.. Fundamentals of Medical Image Processing and Stereoscopy......................... 7 Image Specification .......................................................................................... 7. Morphological Image Processing ............................................................................ 8. 2.2.1.. Erosion .............................................................................................................. 8. 2.2.2.. Dilation ............................................................................................................. 8. 2.2.3.. Opening ............................................................................................................ 9. 2.2.4.. Closing .............................................................................................................. 9. 2.3.. Maximum Intensity Projection (MIP).................................................................... 10. 2.4.. Standardized Uptake Value (SUV) ........................................................................ 10. 2.5.. Stereoscopic ........................................................................................................... 11. 2.5.1.. Anaglyph ........................................................................................................ 13. 2.5.2.. Polarization Systems (Passive) ....................................................................... 14. 2.5.3.. Shutter Systems (Active) ................................................................................ 15. 2.5.4.. Holographic .................................................................................................... 16. 2.6.. NVIDIA 3D Vision ................................................................................................ 16. Chapter 3. DICOM Analysis ............................................................................................ 18. 3.1.. From 2D Slice to 3D Volume Data ....................................................................... 19. 3.2.. Multithreaded Computing ...................................................................................... 20. 3.3.. Calculation of Standardized Uptake Value ............................................................ 20. 3.3.1.. Step 1: Convert Pixel Value to Activity Concentration ................................. 21. 3.3.2.. Step 2: Decay Calibration Factor.................................................................... 22. 3.3.3.. Step 3: Calculate SUV .................................................................................... 22. iv.

(6) 3.4.. Non-body Pixel of CT............................................................................................ 24. Chapter 4. Stereoscopy..................................................................................................... 29. 4.1.. Real 3D and Fake 3D ............................................................................................. 29. 4.2.. Stereoscopic Display Principle .............................................................................. 29. 4.3.. Perceived Depth Deviation .................................................................................... 31. 4.4.. Optical Angle ......................................................................................................... 31. 4.5.. Eye Separation ....................................................................................................... 33. 4.6.. Defining Reference Plane and Screen Depth ......................................................... 33. 4.7.. Perspective ............................................................................................................. 35. 4.8.. Parallax .................................................................................................................. 39. 4.8.1.. Positive Parallax (In Screen) .......................................................................... 40. 4.8.2.. Zero Parallax (2D Object) .............................................................................. 41. 4.8.3.. Negative Parallax (Out of Screen) .................................................................. 41. 4.8.4.. Parallax Adjustment ....................................................................................... 41. Chapter 5 5.1.. Stereoscopic Display ...................................................................................... 46. NVIDIA 3D Vision ................................................................................................ 46. 5.1.1.. Shutter Glasses ............................................................................................... 46. 5.1.2.. Stereo Driver................................................................................................... 46. 5.2.. The Principle of the 3D Vision .............................................................................. 47. 5.2.1.. OpenGL QuadBuffer ...................................................................................... 47. 5.2.2.. NVAPI ............................................................................................................ 47. 5.2.3.. 3D Video ........................................................................................................ 48. 5.3.. Experiment Environment ....................................................................................... 49. 5.4.. Stereo Display with NVIDIA 3D Vision ............................................................... 50. 5.4.1.. MIP ................................................................................................................. 50. 5.4.2.. Perspective Projection with Mean Volume Rendering Stereo ....................... 53. 5.4.3.. Perspective Projection with Bilinear Interpolation Stereo ............................. 54. 5.5.. Result ..................................................................................................................... 56. Chapter 6. Conclusion ...................................................................................................... 57. 6.1.. Conclusion ............................................................................................................. 57. 6.2.. Future Work ........................................................................................................... 58. Reference .............................................................................................................................. 60. v.

(7) List of Figures Figure 2-1 Paper anaglyph filter glasses. Retrieved from Wikipedia, the free encyclopedia, http://en.wikipedia.org/wiki/File:Anaglyph_glasses.png .......................... 14 Figure 2-2 RealD circular polarized glasses. Retrieved from Wikipedia, the free encyclopedia, http://en.wikipedia.org/wiki/File:REALD.JPG ......................................... 15 Figure 2-3 A pair of LCD shutter glasses. Retrieved from Wikipedia, the free encyclopedia, http://en.wikipedia.org/wiki/File:Xpand_LCD_shutter_glasses.jpg ......... 16 Figure 2-4 NVIDIA 3D Vision toolkit. Retrieved from NVIDIA 3D Vision, http://www.nvidia.com.tw/object/3d-vision-main-tw.html .............................................. 17 Figure 3-1 Algorithm. ....................................................................................................... 19 Figure 3-2 Original PET slice number 195, including air and non-body pixel. ............... 23 Figure 3-3 After SUV calculation, lots of air and non-body pixels have been removed. 23 Figure 3-4 SUV value threshold by 2.5, keeping the pixel only larger than 2.5. ............. 24 Figure 3-5 Removing non-body pixel of CT. ................................................................... 25 Figure 3-6 Original CT DICOM image. ........................................................................... 26 Figure 3-7 Transforming the original image to binary image. ......................................... 26 Figure 3-8 Eroding the image with disk value 9. ............................................................. 27 Figure 3-9 Dilating the image with disk value 8. ............................................................. 27 Figure 3-10 Closing the image with disk value 20 to reduce the black hole of Fig. 3-9. . 28 Figure 3-11 Producting the image with the mask we created. We get the body CT DICOM image. ................................................................................................................. 28 Figure 4-1 Principle of stereoscopic display. ................................................................... 30 Figure 4-2 Optical angle of two eye. ................................................................................ 31 Figure 4-3 Left eye and right eye view............................................................................. 33 Figure 4-4 Rotation of eye points and center point. In the larger circle, the red points represent the rotation of right eye point while the black points represent that of the left eye point. .......................................................................................................................... 34 Figure 4-5 In the middle, blue points are the center point of right eyes and the red point is center point of left eye. ........................................................................................ 35. vi.

(8) Figure 4-6 Relation between the eye points, center point, and the projected 2D planes on the x-y axes. The projected 2D planes are perpendicular to the CP-EP line and parallel to the z axis. ......................................................................................................... 36 Figure 4-7 New coordinates ( x' , y ' , z ' ) were obtained by translating the origin (0, 0, 0) to EP, rotating the x-y axes by around the new. θ1. around the z axis, and then rotating the y-z axes. x' axis such that the new y ' axis is from EP to CP. ............................ 38. Figure 4-8 All the points of a line of projection will be projected to a single point on the ( u , v ) plane. ............................................................................................................... 39 Figure 4-9 Positive parallax, zero parallax and negative parallax. ................................... 40 Figure 4-10 Parallax adjustment algorithm. ..................................................................... 42 Figure 4-11 Original left CT perspective projection. ....................................................... 43 Figure 4-12 Positive parallax of left CT projection.......................................................... 43 Figure 4-13 Negative parallax of left CT projection. Pixels need to switch to the right CT. .................................................................................................................................... 43 Figure 4-14 Right CT projection’s negative parallax pixels. ........................................... 44 Figure 4-15 After switching negative parallax pixels. ..................................................... 44 Figure 4-16 Result with overlapping SUV projection. ..................................................... 44 Figure 4-17 Right image with overlapping SUV projection. ........................................... 45 Figure 4-18 Side-by-side image. ...................................................................................... 45 Figure 5-1 3D video principle. ......................................................................................... 48 Figure 5-2 ASUS G53Jw. ................................................................................................. 49 Figure 5-3 The initial eye point projection of patient of MIP, three points look as in the same plane. ....................................................................................................................... 51 Figure 5-4 By rotation we can know the upper red points is in front of the others in fig. 5-3, but left hand and right hand look like in the same plane, and we cannot distinguish which one is the left and which one is the right of MIP. ............................... 51 Figure 5-5 The patient’s back of MIP. ............................................................................. 52 Figure 5-6 Difficultly to distinguish the 3D relationship between ribs and hands. .......... 52 Figure 5-7 Showing the initial eye point perspective projection with mean volume rendering of target objects ................................................................................................ 53. vii.

(9) Figure 5-8 Mean volume rendering method had better bone resolution but poor 3D effect because of the see-through effect on the bone........................................................ 53 Figure 5-9 Back scene of chest. ........................................................................................ 54 Figure 5-10 Perspective is still better than traditional MIP for that Perspective enhanced the 3D experience. ............................................................................................ 54 Figure 5-11 Perspective with bilinear interpolation side-by-side stereo image. .............. 55 Figure 5-12 With active shutter glass and 3D display, left hand has the out of screen 3D effect. .......................................................................................................................... 55 Figure 5-13 Right hand shows in the screen with depth vision. ....................................... 55 Figure 5-14 Bilinear interpolation keeps the bone surface smoother. .............................. 56. viii.

(10) Chapter 1. Introduction. In recent years, as stereoscopic (also called stereoscopy or 3D imaging) display technology is applied more and more widely, more media tend to use it to provide the depth information that 2D images cannot bring. 3D stereoscopic display hardware has recently developed rapidly and made it possible to achieve high quality interactive volumetric rendering of diagnostic data on low-cost platforms [1].. 3D visualization for medical. analysis is not yet widely accepted and applied, although most modern radiology workstations now start including 3D modules that can generate impressive virtual representations of the imaged structures. The possibility caused is medical data sets that contain many overlapping structures, leading volumetric techniques to generate cluttered images, which may become difficult to understand when projected onto a 2D screen. Stereoscopy is a technique for creating or enhancing the illusion of depth in an image by means of stereopsis for binocular vision. are used together [2].. Binocular vision is vision in which both eyes. It can give stereopsis in which parallax provided by the two eyes'. different positions on the head give precise depth perception [3].. Binocular vision with. volume perception of surrounding objects allows human beings to orient freely in the environment [4].. In volume space, projection plane represents to the retina of the human’s. eye. The projection plane is able to perceive an object image in two different projections. Each point in the space corresponds to specific physiological parallax in projection plane. In our work, we present a stereoscopic image created by a standardized uptake value (SUV) calculation in positron emission tomography (PET) scans of lung and overlapped it with computer tomography (CT), showing it by 3D effects via shutter glasses. We apply stereoscopic technology to medical image processing to expend our vision from 2D to 3D and show that the biggest difference between them is the depth information.. The. advantage of depth in stereoscopic images built by computed tomography is helpful for diagnosis such as avoiding misdiagnosis or path planning.. 1.

(11) 1.1. Related Works De Silva et al. pointed out the most distinguishing feature of 3D display systems, compared to the traditional 2D counterparts, is their ability to provide an additional perception of depth to its viewers [5].. Thus, the mechanisms behind human depth. perception play a significant role in 3D video systems. While there have been significant amounts of research carried out to understand human depth perception, in the areas of physiology and psychology, its applicability to 3D display systems is seldom spoken. Understanding the mechanisms of depth perception is of utmost importance to the development of 3D video technologies that are heavily based on exploitation of human perception. It is explained with the aid of existing physiological and psychological models how humans perceive depth in 3D video displays.. Based on these explanations a. mathematical model is derived to explain the just noticeable difference in depth (JNDD) as perceived by a viewer, watching 3D video. The derived model is experimentally validated on an auto-stereoscopic display. This model is expected to be useful in both 3D content productions as well as in 3D content processing and compression. An autostereoscopic display with image quality comparable to ordinary 2D displays has recently been developed [6]. ambiguity is a problem [6].. Abildgaard et al. found with MIP models depth. When binocular stereoscopy with special glasses is used, the. two images are not changed if the observer moves his or her head, apart from the obvious change in the viewing angle relative to the screen. In contrast to binocular stereoscopy, the prototype used in the present study produces a range of different views in different horizontal angles. The number of different views is configurable.. Wang et al. used a. fully crossed ROC paradigm in which eight radiologists interpreted chest CT examinations using each of three display modes, which included slice-by-slice, orthogonal MIP, and stereoscopic display [6, 7, 8]. There are a variety of compelling reasons to believe that stereographic display methods will provide considerable benefits for the display of 3D datasets, though the efficacy of these methods needs to be tested empirically. Santhanam et al. present a medical display and visualization framework for radiation therapy that couples a computer-based simulation of real-time lung tumor motion and its dose accumulation during treatment with an Augmented Reality Center (ARC) based display. 2.

(12) system [9].. Agus’s system allowed multiple untracked naked-eye users in a sufficiently. large interaction area to coherently perceive rendered volumes as real objects, with stereo and motion parallax cues [1].. Nelson et al. proved stereoscopic viewing improved. visualization of small structures when there were multiple overlapping structures. Identification of high-contrast structures such as the fetal skull or spine showed improvements with stereoscopic viewing compared with conventional viewing [10].. They. also noted that stereoscopic viewing enhanced visualization of structures near the observer compared with those that were far from the observer, some of which could be partially obscured by the nearer structure. Beurden et al. reviewed empirical studies concerning the effectiveness of stereoscopic displays in medicine [11].. The review domains covered diagnosis, pre-operative planning,. minimally invasive surgery (MIS) and training or teaching.. Kickuth et al. compared the. accuracy of stereoscopic and standard 3D CT in the classification of acetabular fractures [12].. Hernandez et al. presented a technique for stereoscopic visualization applied to. three-dimensional (3D) ultrasonic breast data.. Two conical transparent projections of the. volume were computed from two slightly different viewpoints [13]. projections made up the stereoscopic pair.. These two. The pair was displayed on a stereoscopic. monitor for the visualization of the 3D data with the depth dimension. Yamagishi found it is important task for medical school lecturers to teach thorough knowledge of the 3D extent of the vascular structure [14]. Recently, isotropic voxel data have been easily obtained by multi-detector CT (MD-CT).. They made use of MD-CT data of clinical cases and created. stereoscopic images for the purpose of practical education. Stereoscopic viewing yielded an exact 3D observation of the abdominal vascular structure. Real time rendering of the stereoscopic images could be performed by VOXEL-MAN.. Using the data of clinical. cases gave an impact on medical students.. 1.2. Motivation Lung cancer affects more than 100,000 Americans each year in all ages but particularly in the smoking population more than 50 years of age [8]. cancer can be effectively treated if they are detected early.. 3. Most types of lung. Radiographic imaging plays an.

(13) important role in the early detection and diagnosis of lung cancer. The primary radiographic imaging tools for lung nodule screening and early detection have shifted from film-based projection radiography to computed tomography (CT). Maximum intensity projection (MIP) is the most common volumetric display method used for the task of lung nodule detection and allows thicker slabs comprised of multiple slices to be viewed.. MIP projects the slab by taking the highest voxel value along a. projection path as a final pixel value for display. In addition, MIP does not always correctly portray geometric structure and therefore may lead radiologists to misinterpret rendered images, especially images with small structures.. In our preliminary study survey, we. found that some of the radiologists spent most of their time viewing slabs comprised of a single slice in the stereographic display mode. The most likely explanation for this is that they were relatively more familiar with slice-by-slice display than with stereo display [6]. Wang et al. indicated that most of the radiologists preferred reading 3D images with a slab thickness equivalent to about five slices, and there was no difference in the time distribution patterns between the stereo and MIP modes [8]. An individual X-ray image alone fails to provide information about size, position, and shape of the object of interest [4]. complete information.. A stereoroentgenographic image provides more. 3D techniques for stereoscopic display of radiological images have. been introduced for many years, but none of these have found wide uses [15, 16, 17].. A. limitation with the majority of the stereoscopic techniques is the requirement of special viewing glasses.. Fortunately, recent advances in computer hardware and software. performance have made such stereoscopy affordable on desktop computers, facilitating viewing of volume data in offices as well as ultrasound equipment and laboratories. More powerful laptop computers make it possible to interactively display volume data in educational and scientific meetings, enhancing comprehension of anatomic structures [10]. Recently, several groups have begun to investigate the application of stereoscopic displays for radiological imaging [6, 18, 19]. Unlike other volumetric display methods, stereo display exploits visual stereopsis to avoid ambiguities associated with structure superimposition. Chan et al. had shown that stereo mammographic display improved detection and characterization performance compared to projection mammograms [18, 19].. 4.

(14) If we can implement stereoscopic technology with medical diagnosis, it may improve the efficiency of diagnosis with the depth information that 2D image cannot provide.. Or it. can be implemented in surgical simulation or education training. Agus et al. present a prototype medical data visualization system exploiting a light field display and custom direct volume rendering techniques to enhance understanding of massive volumetric data, such as CT, MRI, and PET scans [1].. Both the 3D complexity and the enormous size of. medical data thus create a growing demand for interactive high quality 3D visualization. Several researches have emphasized the potential benefits of stereoscopic display [20]. However, the usefulness of stereoscopic 3D CT must lie in the ability to classify them exactly.. There is a clear need for more empirical evidence that quantifies the added value. of stereoscopic displays in medical domains, and more and more researches are needed to assess the potential benefits of stereoscopic displays in those applications [11].. For those. reasons, we compared stereoscopic and standard 2D image in the classification of in 2D MIP, perspective projection with mean volume rendering and perspective projection with bilinear interpolation stereoscopic images to show the improvement of our work.. 1.3. Thesis Contribution Many researches had been done the stereoscopic technique on medical domain before. Abildgaard et al. used an autostereoscopic display in visualization of radiological anatomy from MR angiography [21].. Santhanam showed the display system to monitor the. radiation dose delivery on the tumor [9].. Hernandez generated two conical transparent. projections of high quality from an ultrasonic volume [13].. Those projections computed. fewer than two slightly different viewing angles that represented a stereo pair. active shutter glasses that have better resolution with less expensive device.. We use. The display. systems not only focus on lung but with entire chest and ribs will have more 3D effect of position depth relation.. ARC display system needs one to wear a head mount, yet it. makes one get more uncomfortable than shutter glasses.. The stereoscopic display using. CT and PET DICOM as a 3D display source data can have a better whole body structure rather than ultrasonic.. In this thesis, we applied stereoscopic technology to medical image. 5.

(15) processing. We have SUV to help diagnose and ribs to judge the precise position of body and stereoscopic display’s depth information.. 1.4. Thesis Layout In section 2, we introduce fundamentals of medical image processing of this thesis. Section 3 explains the methodology of calculation of SUV and other medical image analyzing and processing. In section 4, we describe the stereoscopic display principle and create 3D image of DICOM scans using perspective. In section 5, we demonstrate the results of the proposed perspective methods of stereo images and discuss with the 2D MIP image.. 6.

(16) Chapter 2. Fundamentals of Medical Image. Processing and Stereoscopy 2.1. Medical Scans Digital Imaging and Communications in Medicine (DICOM) is the international standard for medical images and related information [22]. It defines the formats for medical images that can be exchanged with the data and quality necessary for clinical use. DICOM is implemented in almost every radiology, cardiology imaging, and radiotherapy device. With tens of thousands of imaging devices in use, DICOM is one of the most widely deployed healthcare messaging standards in the world.. DICOM has enabled. advanced medical imaging applications that have “changed the face of clinical medicine.” In this research, we use MATLAB as tool of calculating and analyzing standardized uptake values. DICOM can be divided into two parts: the image and header (also called metadata) files, which can record more details information of patients.. 2.1.1. Image Specification In this thesis, we use 3D volumetric CT scan sets [23]. sequence of 2D slices. set.. Each of CT data is a. Individual slices are stacked into 3D space that forms a volumetric. Our CT is low-dose whole-body CT sets, which is provided by the Department of. Nuclear Medicine at National Cheng Kung University Hospital in Tainan, Taiwan. data is obtained by Siemens’s PET/CT scanner.. This. The average amount of slices in each. examination is about 350 with slice thickness equal to 4.0 mini-meter.. Each slice has. 512 × 512 pixels of 1.36 × 1.36 mm spacing in width and length. The slice are stored. in 12 bits monochromatic images, but stored as 16 bits resolution. It is acquired with the following parameters: . 110 ~ 130kVp and 50 ~ 100mAs;. . Spiral mode, 3mm collimation and 1.5 pitch; 7.

(17) . 4 ~ 5 mm thickness;. . 512 × 512 matrix. 2.2. Morphological Image Processing Mathematical morphology (MM) is a theory and technique for the analysis and processing of geometrical structures, basing on set theory, lattice theory, topology, and random functions.. MM is most commonly applied to digital images, but it can be. employed as well on graphs, surface meshes, solids, and many other spatial structures.. 2.2.1. Erosion Erosion is one of two fundamental operations in Morphological image processing from which all other morphological operations are based.. It was originally defined for. binary images, later being extended to grayscale images, and subsequently to complete lattices.. Denoted 𝐴𝛩𝐵, is defined as the set operation formula 2-1.. 𝐴𝛩𝐵 = { z | 𝐵𝑧 ⊆ A }. ( 2-1 ). 2.2.2. Dilation Dilation is one of the basic operations in mathematical morphology.. Originally. developed for binary images, it has been expanded first to grayscale images, and then to complete lattices.. The dilation operation usually uses a structuring element for probing. and expanding the shapes contained in the input image. Denoted 𝐴 ⊕ 𝐵, is defined as the set operation formula 2-2.. �𝑧 ∩ A ≠ ∅ � 𝐴⊕𝐵 = �z|𝐵. 8. ( 2-1 ).

(18) 2.2.3. Opening In mathematical morphology, opening is the dilation of the erosion of a set A by a structuring element B.. Denoted as A◦B = (AΘB)⊕B. ( 2-2 ). ,where Θ and ⊕ denote erosion and dilation, respectively.. Together with closing, the. opening serves in computer vision and image processing as a basic workhorse of morphological noise removal.. Opening removes small objects from the foreground of an. image, placing them in the background, while closing removes small holes in the foreground, changing small islands of background into foreground. also be used to find specific shapes in an image.. These techniques can. Opening can be used to find things into. which a specific structuring element can fit.. 2.2.4. Closing In mathematical morphology, the closing of a set binary image A by a structuring element B is the erosion of the dilation of that set.. Denoted as. A ● B = (A⊕B)ΘB ,where ⊕ and Θ denote the dilation and erosion, respectively.. ( 2-3 ) In image processing,. closing is, together with opening, the basic workhorse of morphological noise removal. Opening removes small objects, while closing removes small holes.. 9.

(19) 2.3. Maximum Intensity Projection (MIP) In scientific visualization, a maximum intensity projection (MIP) is a volume rendering method for 3D data which projects in the visualization plane the voxels with maximum intensity that falls in the way of parallel rays traced from the viewpoint to the plane of projection. This implies that two MIP renderings from opposite viewpoints are symmetrical images if they are rendered using orthographic projection. This technique is computationally fast, but the 2D results do not provide a good sense of depth of the original data. To improve the sense of 3D, animations are usually rendered of several MIP frames in which the viewpoint is slightly changed from one to the other; they thus create the illusion of rotation. This helps the viewer's perception to find the relative 3D positions of the object components.. However, since the projection is. orthographic the viewer cannot distinguish between left or right, front or back and even if the object is rotating clockwise or anti-clockwise. MIP is used for the detection of lung nodules in lung cancer screening programs which utilize computed tomography scans. MIP enhances the 3D nature of these nodules, making them stand out from pulmonary bronchi and vasculature. MIP imaging was invented for use in Nuclear Medicine by Jerold Wallis, MD, in 1988, and subsequently published in IEEE Transactions in Medical Imaging [24].. In the setting of Nuclear. Medicine, it was originally called MAP (Maximum Activity Projection).. Additional. information can be found in other articles by the same author [25, 26]. Use of depth weighting during production of rotating cines of MIP images can avoid the problem of difficulty of distinguishing right from left and clockwise versus anticlockwise rotation. MIP imaging is used routinely by physicians in interpreting Positron Emission Tomography (PET) or Magnetic Resonance Angiography studies.. 2.4. Standardized Uptake Value (SUV) The main source to analyze and calculate SUV is DICOM files. Actually SUV is also referred to as the dose uptake ratio. It also can be called the Differential Uptake Value, Dose Uptake Ratio, or Dose Absorption Ratio. The Uptake Value is represented by Pixel intensity value in the image commonly used in PET imaging for semi-quantitative 10.

(20) analysis of PET. The cut-off value for malignant lesion is 2.5, which is used to distinguish between malignant lesions and benign tumor. In medical radiology, positron emission tomography (PET) plays an important role in the diagnosis of lesions. The Standardized Uptake Value (SUV) is widely used in PET imaging for PET analysis. It is helpful for a simple analysis of fluorodeoxyglucose (FDG) images. Masa-Ah et al. used a novel scheme for SUV calculation in PET scans. The results were compared with the SUV taken from the well-known application software from GE healthcare [27]. GE healthcare's and Masa-Ah's systems correlated significantly with 95% confidence. It showed that the novel scheme for SUV calculation provides the correct SUV value.. Then Masa-Ah et al. used the novel SUV calculation scheme for PET. DICOM files using MATLAB [28].. The strength of this scheme is that the. interchangeability of the DICOM files can be done conveniently without the special application software from any vendors.. 2.5. Stereoscopic Stereoscopic (also called stereoscopy or 3D imaging) display refers to a technique for creating the illusion of depth in an image by presenting two offset images separately to the left and right eye of the viewer. Both of these 2D offset images are then combined in the brain to give the perception of 3D depth. Stereoscopic 3D display includes head mount system, anaglyph system, polarized filter system, field sequential system and so on [29]. Namely, three strategies have been used to accomplish the following: 1) have the viewer wear eyeglasses to combine separate images from two offset sources, 2) have the viewer wear eyeglasses to filter offset images from a single source separated to each eye, and have the light source split the images directionally into the viewer's eyes (no glasses required). In our work, we use the have viewer wear glassed to combine separate images from two offset sources. Kim and Sohn found that in stereoscopic images is visual fatigue, but they used a depth-based view synthesis algorithm to handle the whole regions in rendering process. And then they performed depth-based view synthesis to the contents which were predicted. 11.

(21) to cause visual fatigue [30]. A subjective evaluation showed that the proposed depth adjustment method generated comfortable stereoscopic images. Ramos-Diaz et al. allow the minimization of artifacts like ghosting and overlapping areas, and they reduce dynamic range of depth map value [31]. Okuyama et al. mention that informed consent for patients becomes important in clinics where the doctor explains diseases and treatments [32]. But patients have difficulty understanding Medical Images, i.e. X-rays, CT or MRI because the patient is not an expert in reading these medical images. They tried to reconstruct the stereoscopic image of CT and MRI and display these using a simple and low cost auto-stereoscopic viewer. Su et al. apply augmented reality overlay of reconstructed 3D-computed tomography images onto real-time stereo video footage which is able to use iterative closest point and image-based surface tracking technology that does not use external navigation tracking systems or preplaced surface markers [33]. Kersten et al. present empirical studies that consider the effects of stereopsis and simulate aerial perspective on depth perception in translucent volumes [34].. They consider a purely. absorptive lighting model, in which light is not scattered or reflected, but is simply absorbed as it passes through the volume. A purely absorptive lighting model is used, for example, when rendering digitally reconstructed radiographs (DRRs), which are synthetic X–ray images reconstructed from CT volumes. Surgeons make use of DRRs in planning and performing operations, so an improvement of depth perception in DRRs may help diagnosis and surgical planning. Bouguila and Yoshida have undertaken experiments to prove that the observer’s perceived depth is not the same with the theoretical depth completely [35, 36]. Bouguila proposed integrating haptic sensation with stereopsis cues to improve the task of localizing objects [35]. Yoshida presented the possible factors that affect perceived depth, such as measuring errors of the eye separation, the distance between the viewpoint and screen [36]. Then an effective method was performed by modification of both the viewing position and the screen. Lin et al. supposed there is deviation between perceived depth and theoretical depth of virtual object in stereoscopic virtual environment [37]. They analyzed its possible causes with optical geometry.. In addition, a correcting method performed by setting. synthetic viewpoint dynamically is proposed to show that this method can reduce the depth deviation to less than 10 millimeter, and meets the requirements for locating and direct. 12.

(22) manipulation in virtual environment.. Yamagishi et al. discuss stereoscopic viewing. yielded an exact 3-D observation of the abdominal vascular structure. Real time rendering of the stereoscopic images could be performed by VOXEL-MAN [14]. Using the data of clinical cases gave an impact on medical students. Liao et al. proposed a high-resolution stereoscopic surgical display using the integral videography (IV) and multiple projectors [38]. IV projects a computer generated graphical object by multiple rays through “fly’s eye lens,” which can display geometrically accurate auto-stereoscopic images and reproduce motion parallax in 3-D space.. The. technique requires neither special glasses nor sensing device to track viewer’s eyes. Thus it is suitable for pre-operative diagnosis and intra-operative use and for reporting the use of multiple reduction projection display system and a parallel-calculation to achieve highresolution IV image. They evaluate the feasibility of this display by developing a 3-D CT stereoscopic image and applying it to surgical planning and intra-operative guidance. The main contribution of it is application and modulation of medical stereoscopic technique originally developed in high-resolution multi-projector stereoscopic display system.. Liao. et al. also developed the auto-stereoscopic display and analysis system to aid the development of 3-D cardiac imaging for use during minimally invasive cardiac surgery [39]. Also they integrated the high-quality auto-stereoscopic display system into image guided cardiac surgery. The human heart was scanned in a sequence of volumetric images which is based on dynamic information of heart from CT scanning. It represented the motion of the human heart throughout the cardiac cycle. Time sequence 3-D image analysis can be particularly useful in cardiac applications. This approach will allow them to acquire the optimum method to produce 3-D cardiac image for planning and guidance of minimally invasive cardiac surgery.. 2.5.1. Anaglyph Beyond those methods we mentioned, the most accessible is the anaglyph stereo [40], which is based on the principle of spectral separation and uses the properties of filters. It filters the rays of one color and delays the rays of other colors. Anaglyph images are much easier to view than either parallel, diverging or crossed-view pairs stereograms.. 13.

(23) Each pixel of the computer image is composed of three color components: red, green and blue. On the screen simultaneously displaying two images forms the stereo pair. At the same time in each pixel of the total image red component corresponds to the red in the left image, and the green components or blue components in the right. However, these sideby-side types offer bright and accurate color rendering, which anaglyphs can hardly achieve. Figure 2-1 is Paper anaglyph filters producing an acceptable image at low cost and suitable for inclusion in magazines.. Figure 2-1 Paper anaglyph filter glasses. Retrieved from Wikipedia, the free encyclopedia, http://en.wikipedia.org/wiki/File:Anaglyph_glasses.png. 2.5.2. Polarization Systems (Passive) To present stereoscopic pictures, two images are projected superimposed onto the same screen through polarizing filters or presented on a display with polarized filters. For projection, a silver screen is used so that polarization is preserved. The viewer wears lowcost eyeglasses which also contain a pair of opposite polarizing filters. As each filter only passes light which is similarly polarized and blocks the opposite polarized light, each eye only sees one of the images, and the effect is achieved.. 14.

(24) Figure 2-2 RealD circular polarized glasses. Retrieved from Wikipedia, the free encyclopedia, http://en.wikipedia.org/wiki/File:REALD.JPG. 2.5.3. Shutter Systems (Active) With the shutter method using the concept of alternate-frame sequencing, a shutter blocks light from each appropriate eye when the converse eye's image is projected on the screen or displayed on the computer display,. This was the basis of the Teleview system which was used briefly in 1922. There have been many examples of shutter glasses over the past few decades, such as SegaScope 3-D glasses for the Sega Master System and the Atari/Tektronix Stereotek 3D system, but the NVIDIA 3D Vision gaming kit introduced this technology to mainstream consumers and PC gamers.. Also see Time-division. multiplexing. Fig. 2-3 is a pair of LCD shutter glasses used to view XpanD 3D films. The thick frames conceal the electronics and batteries.. 15.

(25) Figure 2-3 A pair of LCD shutter glasses. Retrieved from Wikipedia, the free encyclopedia, http://en.wikipedia.org/wiki/File:Xpand_LCD_shutter_glasses.jpg. 2.5.4. Holographic Similar to stereoscopic display, a holographic display is a kind of 3D display. The viewer can move around the image in any direction because it incorporates a true parallax element and the image maintains its integrity. Holographic displays are of great interest because users can view 3D images without wearing glasses. Creating a virtual image in space can be viewed from any direction as if it were a real physical object. However, unless the stored images can be updated, holographic technology will remain unsuitable for a wide variety of 3D display applications [41].. 2.6. NVIDIA 3D Vision 3D Vision (previously GeForce 3D Vision) is a stereoscopic gaming kit from NVIDIA which consists of LC shutter glasses and driver software which enables stereoscopic vision for any Direct3D game, with various degrees of compatibility. There have been many examples of shutter glasses over the past decade, but the NVIDIA 3D Vision gaming kit introduced this technology to mainstream consumers and PC gamers. The kit is specially designed for 120 Hz LCD monitors. Also it is compatible with CRT monitors (some of which may work at 1024×768×120 Hz and even higher refresh rates), DLP-projectors, and other. It requires a graphics card from NVIDIA.. 16.

(26) Figure 2-4 NVIDIA 3D Vision toolkit. Retrieved from NVIDIA 3D Vision, http://www.nvidia.com.tw/object/3d-vision-main-tw.html. 17.

(27) Chapter 3. DICOM Analysis. In this section, we introduce the detail of DICOM analysis, which is imported from DICOMDIR file, and store all CT/PET series as well as each slice name from all 18 patients. Then we calculate SUV and remove the CT’s platform. It is because the 3D calculation needs a lot of calculation that we only focus on the lung and chest to reduce the total time of whole process. In chapter 4, we go into the stereo process of the producing of stereoscopic image. We define which part of the view plane and showing plan should be set as negative parallax and which of that should not. Also, we define the proper eye separation to create comfort 3d vision feeling and to avoid the bad experience.. The. projection project from the center of objective to the eye point includes left eye and right eye, so that makes a true 3D effect. Then we adjust the parallax and use the true 3D from the projection of two viewpoints. It only needs to adjust the out of screen pixel by change the left and right pixel. After the stereoscopic image has been created, in this research we use positive shutter glasses via NVIDIA’s 3D vision toolkit. By the middleware of 3D vision, we finally can see the 3D vision of medical image.. 18.

(28) Figure 3-1 Algorithm.. 3.1. From 2D Slice to 3D Volume Data In this study, we only use CT and PET DICOM image to analyze one slice that usually does not take a long time. In 3D volume data, every change needs a couple of time to calculate it. To reduce the time of whole process, we save the entire series with every slice's file name and file path with every patient in a structure of matrix. When we need to analyze DICOM, we can only import the demand series or slices and then construct to. 19.

(29) volume matrices stored in memory. Thus, we can calculate SUV and remove the nonbody pixel of lung by calling slice image of the chest part only.. 3.2. Multithreaded Computing For that we use MATLAB as an image processing tool, lots of matrix calculations really take a long time. MATLAB has the advantage of faster coding time with a lots library to implement, but it has a speed limitation of loop time. To reduce the time we need to spend on computing three-dimensional matrices of 18 patients, we use multithreaded computing and GPU acceleration of MATLAB to get a better performance. It decreased from 23 hours to 15 hours.. 3.3. Calculation of Standardized Uptake Value SUV can be seen as Differential Uptake Value, Dose Uptake Ratio or Dose Absorption Ratio. And the SUV value is all calculated form the DICOM. DICOM files are the main source to analyze and calculate SUV. The content of the DICOM file can be divided into 2 main parts, which are the image and metadata respectively. The image is composed by image data and the metadata. The metadata record the details of the data such as dose of injection, patient’s weight and height, etc. The Uptake Value is represented by pixel value in the DICOM file that stores the data in 16 bits ranging from 0 to 32767. PET DICOM file has been through the attenuation correction.. SUV calculation is the conversion of the pixel value to the activity. concentration. The related attributed tags for the conversion are Rescale Slope tag, and Rescale Intercept tag. Those tags are different for each image slice of the patients. So the tissue activity is calculated using formula (3-1): 𝑈 = 𝑚．𝑆𝑉 + 𝑏. ( 3-1 ). ,where m is Rescale Slope (which is different in each image slice), SV is the stored value (Pixel Intensity Value), b is rescale intercept (for PET scan is always Zero), and U is units of value after conversion (for our study is Bq/ml, Shown in Unit tag). 20.

(30) The conversion formula above is the fundamental defined by National Electrical Manufacturers Association (NEMA), however the formula maybe slightly different for each vendor. In this study we used the DICOM file as an input source. The method of calculation is based on the formula (3-2) as follows:. 𝑆𝑈𝑉 =. 𝐴𝑐𝑡𝑖𝑣𝑖𝑡𝑦 𝐶𝑜𝑛𝑐𝑒𝑡𝑟𝑎𝑡𝑖𝑜𝑛 𝐼𝑛𝑗𝑒𝑐𝑡𝑒𝑑 𝐷𝑜𝑠𝑒 � � 𝐵𝑜𝑑𝑦 𝑊𝑒𝑖𝑔ℎ𝑡. ( 3-2 ). ,where Pixel Value is original pixel value form DIOCM, 𝐼𝑚𝑎𝑔𝑒 𝑆𝑐𝑎𝑙𝑒 𝐹𝑎𝑐𝑡𝑜𝑟 = 𝑅𝑒𝑠𝑐𝑎𝑙𝑒 𝑆𝑙𝑜𝑝𝑒 × 0.027027027 , Actual Activity is injected activity at the time of scan,. Body Weight (g) is patient’s body weight and can be get from Patient Weight (0010, 1010), and Dose Calibration factor must be multiplied by 2.7 × 10−8 for Bq to mCi conversion.. The steps of the SUV calculation scheme are summarized as follows:. 𝑆𝑈𝑉 =. 𝑃𝑖𝑥𝑒𝑙 𝑉𝑎𝑙𝑢𝑒 𝐼𝑚𝑎𝑔𝑒 𝑆𝑐𝑎𝑙𝑒 𝐹𝑎𝑐𝑡𝑜𝑟 𝐴𝑐𝑡𝑢𝑎𝑙 𝐴𝑐𝑡𝑖𝑣𝑖𝑡𝑦 � �×(𝐷𝑜𝑠𝑒 𝐶𝑎𝑙𝑖𝑏𝑟𝑎𝑡𝑖𝑜𝑛 𝐹𝑎𝑐𝑡𝑜𝑟 ) 𝐵𝑜𝑑𝑦 𝑊𝑒𝑖𝑔ℎ𝑡. ( 3-3 ). 3.3.1. Step 1: Convert Pixel Value to Activity Concentration After extracting pixel value, conversion of Pixel Value (Bq/ml) to Activity Concentration (nCi/ml) is the next step. The data in this conversion are based on some of the detail part (metadata).. 𝐴𝑐𝑡𝑖𝑣𝑖𝑡𝑦 𝐶𝑜𝑛𝑐𝑒𝑟𝑡𝑟𝑎𝑡𝑖𝑜𝑛 = 𝑃𝑖𝑥𝑒𝑙 𝑉𝑎𝑙𝑢𝑒. 𝐼𝑚𝑎𝑔𝑒 𝑆𝑐𝑎𝑙𝑒 𝐹𝑎𝑐𝑡𝑜𝑟. 𝑃𝑖𝑒𝑥𝑒𝑙 𝑉𝑎𝑙𝑢𝑒. 𝐼𝑚𝑎𝑔𝑒 𝑆𝑐𝑎𝑙𝑒 𝐹𝑎𝑐𝑡𝑜𝑟. ( 3-4 ). = 𝑃𝑖𝑥𝑒𝑙 𝑉𝑎𝑙𝑢𝑒 × 𝑅𝑒𝑠𝑐𝑎𝑙𝑒 𝑆𝑙𝑜𝑝𝑒 × 0.027027027 ( 3-5 ). 21.

(31) The related tags in this part are Rescale Slope, Rescale Intercept and Units. In PET DICOM, only Rescale slope is different in each slice.. 3.3.2. Step 2: Decay Calibration Factor We calculate decay calibration factor with the related tags, which, in this part, are Total Dose (0018, 1074), Series Date (0008, 0021), Series Time (0008, 0031), Radiopharmaceutical Start Time (0018, 1072), and Radiopharmaceutical Half Life (0018, 1075).. In Code Meaning (0008, 0104) metadata value is “F^18^[^18^Fluorine]”, it shows. that we use Fludeoxyglucose (F-18). Fludeoxyglucose’s isotope half-life is 109.74 minute.. 𝐴𝑐𝑡𝑢𝑎𝑙 𝐴𝑐𝑡𝑖𝑣𝑖𝑡𝑦 × 𝐷𝑜𝑠𝑒 𝐶𝑎𝑙𝑖𝑏𝑟𝑎𝑡𝑖𝑜𝑛 𝐹𝑎𝑐𝑡𝑜𝑟 =. 𝐼𝑛𝑗𝑒𝑐𝑡𝑒𝑑 𝐷𝑜𝑠𝑒×(2.7×108 ) 𝑇𝑖𝑚𝑒 𝐷𝑖𝑓𝑓𝑒𝑟𝑒𝑛𝑐𝑒. ( 3-6 ). 2𝐼𝑠𝑜𝑡𝑜𝑝𝑒 𝐻𝑎𝑙𝑓−𝑙𝑖𝑓𝑒. 3.3.3. Step 3: Calculate SUV We calculate each slice’s SUV by formula (3-7) with previous step.. 𝑆𝑈𝑉 = 𝑆𝑈𝑉 =. 𝑃𝑖𝑥𝑒𝑙 𝑉𝑎𝑙𝑢𝑒 𝐼𝑚𝑎𝑔𝑒 𝑆𝑐𝑎𝑙𝑒 𝐹𝑎𝑐𝑡𝑜𝑟 𝑇𝑖𝑚𝑒 𝐷𝑖𝑓𝑓𝑒𝑟𝑒𝑛𝑐𝑒 2𝐼𝑠𝑜𝑡𝑜𝑝𝑒 𝐻𝑎𝑙𝑓−𝑙𝑖𝑓𝑒 𝐵𝑜𝑑𝑦 𝑊𝑒𝑖𝑔ℎ𝑡. ( 3-7 ). ×𝐼𝑛𝑗𝑒𝑐𝑡𝑒𝑑 𝐷𝑜𝑠𝑒. (𝑃𝑖𝑥𝑒𝑙 𝑉𝑎𝑙𝑢𝑒×𝑅𝑒𝑠𝑐𝑎𝑙𝑒 𝑆𝑙𝑜𝑝𝑒)×𝐵𝑜𝑑𝑦 𝑊𝑒𝑖𝑔ℎ𝑡 𝑇𝑖𝑚𝑒 𝐷𝑖𝑓𝑓𝑒𝑟𝑒𝑛𝑐𝑒 𝐼𝑠𝑜𝑡𝑜𝑝𝑒 𝐻𝑎𝑙𝑓−𝑙𝑖𝑓𝑒 ×𝐼𝑛𝑗𝑒𝑐𝑡𝑒𝑑 𝐷𝑜𝑠𝑒 2. ( 3-8 ). The cut-off value of distinguish malignant lesions and benign tumor is 2.5, so we calculated SUV and then we deleted each pixel that value is under 2.5. In other word, we set a threshold value 2.5. Finally, we outputted each SUV matrix to a DICOM image for label map or other image processing purpose.. 22.

(32) Figure 3-2 Original PET slice number 195, including air and non-body pixel.. Figure 3-3 After SUV calculation, lots of air and non-body pixels have been removed.. 23.

(33) Figure 3-4 SUV value threshold by 2.5, keeping the pixel only larger than 2.5.. 3.4. Non-body Pixel of CT Platform is where the patients laid down with when they do the radiology scanning. Focusing on the back part of CT DICOM, we transform DICOM image to binary image that contains 0’s and 1’s to make a mask for CT to remove the platform. Then we need image processing method (erosion and dilation) to remove the platform and reserve the detail we need for medical analyzing. We tried several setting to get the best result of erode and dilate values. Finally, we get a method by erosion with a disk value 9 and dilation with disk value 8. Through the platform we cannot entirely remove non-body pixel clearly but we can keep the detail pixel of body. Erosion is for clearing the non-body pixel, and dilation is for getting the body pixel back. Doing closing is for recovering the hole and reserving the detail of chest. After the mask is created, we can filter platform by simply producting two matrices, the original CT DICOM and mask. At first, we do erosion and dilation twice. However, after several try, we found that we could do only once on each step and get the same result. We. 24.

(34) can see the rough of mask is created when doing the first dilation in Fig. 3-9. We do closing because we only need to remove the non-body pixel but we do not want to remove the other details that might affect diagnose. Finally, we can get a patient DICOM image without non-body pixels. But why do we do closing after dilation rather than doing dilation with larger disk or erosion with smaller disk? It is because if we erode with smaller disk, non-body pixel will not be removed by erosion and if we dilate with larger disk, the dilation will also recover the non-pixel part. Closing will only reduce the black hole of body mask and keep more details than disk value 8.. Figure 3-5 Removing non-body pixel of CT.. 25.

(35) Figure 3-6 Original CT DICOM image.. Figure 3-7 Transforming the original image to binary image.. 26.

(36) Figure 3-8 Eroding the image with disk value 9.. Figure 3-9 Dilating the image with disk value 8.. 27.

(37) Figure 3-10 Closing the image with disk value 20 to reduce the black hole of Fig. 3-9.. Figure 3-11 Producting the image with the mask we created. We get the body CT DICOM image.. 28.

(38) Chapter 4. Stereoscopy. 4.1. Real 3D and Fake 3D Recently a large amount of films claimed that they are 3D films, but actually many of those “fake 3D” were converted into 3D post-production from 2D films. Instead of created by two projections from different viewpoints, the “fake 3D” films are actually produced with a depth map which can display the other eye view to the audience.. However,. audiences do not know what is the different between “real 3D” and “fake 3D”. Therefore, this leads them to have worse impression to the 3D experience when watching the so-called 3D films.. 4.2. Stereoscopic Display Principle In virtual reality system, in order to create stereoscopic imagery, we generate a unique perspective view of the scene for each eye. Human adults’ eye separation is about 5.5 to 7.0 cm, so there is a slightly difference between the left and right views. Stereoscopic or the perception of depth is created based on this discrepancy.. 29.

(39) Figure 4-1 Principle of stereoscopic display. Fig. 4-1 shows the stereoscopic display principle’s relation. Distance D is the distance between the viewpoint and the screen. Disparity distance R is the distance of stereo pair of images on the screen. And Screen depth S identifies the distance between the virtual object and the screen. The following formula (4-1) and (4-2) are obtained from Fig. 4-1. 𝑆𝑐𝑟𝑒𝑒𝑛 𝐷𝑒𝑝𝑡ℎ. 𝐷𝑖𝑠𝑡𝑎𝑛𝑐𝑒−𝑆𝑐𝑟𝑒𝑒𝑛 𝐷𝑒𝑝𝑡ℎ. 𝑆𝑐𝑟𝑒𝑒𝑛 𝐷𝑒𝑝𝑡ℎ＝. =. 𝐷𝑖𝑠𝑝𝑎𝑟𝑖𝑡𝑦 𝐷𝑖𝑠𝑡𝑎𝑛𝑐𝑒 𝐸𝑦𝑒 𝑆𝑝𝑎𝑟𝑎𝑡𝑖𝑜𝑛. 𝐷𝑖𝑠𝑡𝑎𝑛𝑐𝑒×𝐷𝑖𝑠𝑝𝑎𝑟𝑖𝑡𝑦 𝐷𝑖𝑠𝑡𝑎𝑛𝑐𝑒. 𝐸𝑦𝑒 𝑆𝑒𝑝𝑎𝑟𝑎𝑡𝑖𝑜𝑛＋𝐷𝑖𝑠𝑝𝑎𝑟𝑖𝑡𝑦 𝐷𝑖𝑠𝑡𝑎𝑛𝑐𝑒. ( 4-1 ). ( 4-2 ). Therefore, the theoretical screen depth is decided by eye separation, distance between the viewpoint and screen and the disparity distance. If they are constant values, the depth will be constant too.. 30.

(40) 4.3. Perceived Depth Deviation Both the viewpoint and object were located in the central location of the virtual environment to make observers face the screen squarely. But, in fact, in the situation that observer stands at the central axis of the screen and that the object’s stereo pair of images is projected on the edge of the screen, observers will tilt their head naturally, instead of facing the screen vertically. As a result, depth deviation is produced.. 4.4. Optical Angle In volume space we do not know the real distance between two eyes, so we need the optical angle to calculate the viewpoint as two eyes. The best viewing range for our two eyes and brain to experience the 3D stereo effect is about 70 to 500 cm between the viewers’ eyes and the screen.. Figure 4-2 Optical angle of two eye. If the distance between the two eyes is 6.5 to 7.0 cm, then the target distance 70 cm is about 10 times of eye distance. In this distance the optical angle is about 5 to 6 degree. Parallax with this optical angle the stereo effect has the best 3D experience. However, if the object is closer than 70 cm to the viewpoint, the viewer probably cannot tolerate the 31.

(41) parallax with the angle larger than 6 degree and will feel uncomfortable. On the other hand, the viewers will start to derecognize the proximity of the object with distance farther than 500 cm. For that the location of the 500 cm distance to our eyes, the optical angle is about 0.7 to 0.8 degree. The optical angle formula is as follows:. 𝜃 2. 𝑡𝑎𝑛 =. 𝐸𝑦𝑒 𝑆𝑒𝑝𝑎𝑟𝑎𝑡𝑖𝑜𝑛 2. ( 4-3 ). 𝐷𝑖𝑠𝑡𝑎𝑛𝑐𝑒 𝑓𝑟𝑜𝑚 𝑉𝑖𝑒𝑤𝑒𝑟 𝑡𝑜 𝑇𝑎𝑟𝑔𝑒𝑡 𝑂𝑏𝑗𝑒𝑐𝑡. 𝑂𝑝𝑡𝑖𝑐𝑎𝑙 𝐴𝑛𝑔𝑙𝑒 𝜃 = 2 × 𝑡𝑎𝑛−1 (. 𝐸𝑦𝑒 𝑆𝑒𝑝𝑎𝑟𝑎𝑡𝑖𝑜𝑛 2. 𝐷𝑖𝑠𝑡𝑎𝑛𝑐𝑒 𝑓𝑟𝑜𝑚 𝑉𝑖𝑒𝑤𝑒𝑟 𝑡𝑜 𝑇𝑎𝑟𝑔𝑒𝑡 𝑂𝑏𝑗𝑒𝑐𝑡. ). ( 4-4 ). Eye separation, distance between two cameras, has a deep relationship with optical angle. Optical angle depends on the distance to the target object. Different viewing range will create different optical angle, which also causes the different parallax and stereo effect. We mention that the ratio with distance and eye separation is about 10 times so we can get the relation formula.. When eye separation is from 6.5 to 7 cm and optimal viewing range. is from 70 to 500 cm, the value of. 1. 𝐸𝑦𝑒 𝑆𝑒𝑝𝑎𝑟𝑎𝑡𝑖𝑜𝑛 𝑂𝑝𝑡𝑖𝑚𝑎𝑙 𝑉𝑖𝑒𝑤𝑖𝑛𝑔 𝑅𝑎𝑛𝑔𝑒. 𝐷𝑖𝑠𝑡𝑎𝑛𝑐𝑒 𝑏𝑒𝑡𝑤𝑒𝑒𝑛 𝑣𝑖𝑒𝑤𝑝𝑜𝑖𝑛𝑡𝑠 =. is about from 10 to 70 times.. 𝐷𝑖𝑠𝑡𝑎𝑛𝑐𝑒 𝑓𝑟𝑜𝑚 𝑉𝑖𝑒𝑤𝑒𝑟 𝑡𝑜 𝑇𝑎𝑟𝑔𝑒𝑡 𝑂𝑏𝑗𝑒𝑐𝑡 𝐸𝑦𝑒 𝑆𝑒𝑝𝑎𝑟𝑎𝑡𝑖𝑜𝑛 𝑂𝑝𝑡𝑖𝑚𝑎𝑙 𝑉𝑖𝑒𝑤𝑖𝑛𝑔 𝑅𝑎𝑛𝑔𝑒. ( 4-5 ). The optimal viewing range will differ from various viewing scene. To maximum the parallax with 70 cm distance and optical angle is about 6 degree, it is good for a close-up take. However, normally stereoscopic image uses 1.4 to 2.1 m, and the optical angle is from 2.86 to 1.91 degree; this is most suitable for the 3D effect. In this value, people will feel most comfortable and has a better 3D experience.. 32.

(42) 4.5. Eye Separation In previous section, we mentioned that it is hardly to define the pixel and CT/PET scale, so we choose an eye angle between 2.86 and 1.91 degree for stereoscopic from half of the range of optimal optical angle. We define the first center point and an eye point and define 512 pixels between center point and eye points.. Then we rotate optical angle to get. a temporary right eye viewpoint for finding the distance of left eye point and right eye point and seeing what the eye separation is. The distance between the eyes is called intraocular. It is 4.5 to 7 cm on average and equivalent to the visible parallax on screen for objects at infinity. We define a normalized eye separation depending on varies screen width. A reference maximum value for the separation is used in the stereo projection for a comfortable stereo experience, and it is different from each screen model. Because the viewers’ eyes do not view the same image on each eye, it may cause the uncomfortable experience to them.. In order to let user viewer feel comfortable, the. maximum parallax at infinity is eye separation. Eye separation should be shorter than the maximum separation value to get the better stereo effect. When the screen is close to the users, most of the users cannot handle more than 50% of the eye separation.. Figure 4-3 Left eye and right eye view.. 4.6. Defining Reference Plane and Screen Depth After all the parameters have been calculated, we can start to define the eye point and the center point of 360 degree rotation. We set the projection plan in the center point of. 33.

(43) objective to make the upper of body out of screen.. For the limitation of massive. calculation and time eclipse, we need to produce the 360 degree’s image for result in advance. We choose 10 degree for each view, but each view needs a left eye and a right eye viewpoint as well as the center point, which have to parallel with the eye point. Thus, every rotation needs a new center point. Fig. 4-4 and 4-5 show the rotation of eye point and center point.. In the larger circle. of fig. 4-4 and 4-5, the red points represent the rotation of right eye point while the black points represent that of the left eye point.. In the center of the figures, there are many blue. points which represent the center point of right eye.. The only red point in the center is the. center point of left eye as well as center point of body.. Figure 4-4 Rotation of eye points and center point. In the larger circle, the red points represent the rotation of right eye point while the black points represent that of the left eye point.. 34.

(44) Figure 4-5 In the middle, blue points are the center point of right eyes and the red point is center point of left eye.. 4.7. Perspective From the DICOM file in this research, both CT and PET datasets are 3D ones that the sizes of CT and PET are 512×512× n z and 168×168× n z , respectively, with n z being the number of transverse slices differing from patient to patient. To visualize the 3D datasets, volume rendering was used, simulating X-rays emanating from the source of the X-rays, penetrating the object, and projecting on a 2D image plane [42]. Specifically, perspective rendering was employed here, all rays emanating from an eye point (EP) as shown in Fig. 4-6 we can view the object from different eye points around the object at the same z-axis height, and so that the center point (CP) of the projected 2D plane was put at the center of lung.. 35.

(45) Figure 4-6 Relation between the eye points, center point, and the projected 2D planes on the x-y axes. The projected 2D planes are perpendicular to the CP-EP line and parallel to the z axis. Using the same notations as in [43], a new coordinate system ( x' , y ' , z ' ) was set by translating the origin (0, 0, 0) to the eye point ( xe , ye , ze ), rotating the x-y axes around the z axis by. θ1 , and finally rotating the y-z axes around the new x'. axis by. θ2. (Fig. 4-. 7). Denote the coordinate of CP as ( xc , yc , zc ). Here,. s1 = sin θ1 = c1 = cos θ1 = Then the vector (. xc − xe ( xc − xe ) 2 + ( y c − y e ) 2. and. yc − ye ( xc − x e ) 2 + ( y c − y e ) 2. .. xc − xe , yc − ye , zc − ze ) is rotated to be (0,. s1 ( xc − xe ) + c1 ( yc − ye ) , zc − ze ) = (0,. ( xc − xe ) 2 + ( yc − ye ) 2 ,. 36. zc − ze ) after θ . 1.

(46) Then. s2 = sin θ 2 =. c2 = cos θ 2 =. zc − ze. and. ( xc − xe ) 2 + ( yc − ye ) 2 + ( zc − ze ) 2. ( xc − xe ) 2 + ( yc − ye ) 2 ( xc − xe ) 2 + ( yc − ye ) 2 + ( zc − ze ) 2. As shown in Fig. 2, a point ( x ,. .. y , z ) was mapped to ( x' , y ' , z ' ) as. 0 0  c1 − s1 0  x − xe   x '  1  y − y   s  y ' = 0 c s c 0 1 1 2 2 e        z '  0 − s 2 c2   0 0 1  z − z e  c1 ( x − xe ) − s1 ( y − y e )   =  s1c2 ( x − xe ) + c1c2 ( y − y e ) + s 2 ( z − z e )  − s1 s 2 ( x − xe ) − c1 s 2 ( y − y e ) + c2 ( z − z e ) A coordinate ( u , v ) on the projected 2D plane was related to ( x ' , y ' , z ' ) and. d = ( xc − xe ) 2 + ( yc − ye ) 2 + ( zc − ze ) 2 as. u x' v z' = = and d y' d y'. ( 4-6 ). , where the point ( x ' , y ' , z ' ) would be projected to ( u , v ) on the projected 2D plane.. 37.

(47) Figure 4-7 New coordinates ( x' , y ' , z ' ) were obtained by translating the origin (0, 0, 0) to EP, rotating the x-y axes by new. θ1. around the z axis, and then rotating the y-z axes around the. x' axis such that the new y ' axis is from EP to CP. When a coordinate ( u , v ) was given, formula (4-6) would be a set of equations:. u=. c1 x − s1 y c1 x − s1 y and v = , s1c2 x + c1c2 y + s2 z s1c2 x + c1c2 y + s2 z. ,where ( x , y , z ) = ( x − xe , y − ye , z − z e ). The solution to the above equations was a line as shown in Fig. 3. This line could be parameterized as (a) ( x ,. ∆z = − (b) (. ∆y ∆. x,. ss s cs c ∆z x ) with ∆ = −c1 + 1 2 v − 1 c2 , ∆ y = s1 + 1 2 v − 1 c2 , and u u u u ∆. c2 s v − 2 , or u u. ∆x ∆ cs c ss s y , y , z y ) with ∆ = − s1 − 1 2 v + 1 c2 , ∆ x = c1 − 1 2 v + 1 c2 , and ∆ u u u u ∆. ∆z =. c2 s v+ 2 . u u. 38.

(48) If. u = 0 , we assume that y ≠ 0 and c1 ≠ 0 . This particular case does not. happen if the EP is not included in the projected line as in usual applications. Then the parameterized line is (. s 2 + c2 v s1 y ). y, y , c1c2 − c1s2 v c1. The points of the line inside the object are recorded. The projected voxel value can be taken as (a) the voxel value of the nearest voxel to the EP, (b) the maximum voxel value of these recorded voxels, or (c) the average voxel value of these recorded voxels.. Figure 4-8 All the points of a line of projection will be projected to a single point on the ( u ,. v ) plane.. 4.8. Parallax Parallax is a difference or change in the apparent position or direction of an object as seen from two different points [44, 45]. Some people call it Disparity or Binocular Retinal Disparity as well. In fact, parallax and disparity refers to the same thing, but parallax is generally used in the 3D stereo images performance or display and disparity for the retina. Though the greater the optical angle the larger the parallax, but the parallax should not be too much or it will cause eye discomfort. In other words, 3D effect will become bigger when the target is closer to viewer but it will make our eyes feel uncomfortable. 39.

(49) Therefore, we must define specific viewing distance. Distinguishing the depth information viewing range from 70 to 500 cm is the best viewing distance for human. With this range 3D image has the best stereo effect, and its target optical angle is about 0.8 degrees to 6 degrees. If target is closer than 70 cm (optical angle is greater than 6 degrees), our eyes will begin to feel uncomfortable. On the other hand, if the target is farther than 500 cm (optical angle is less than 0.8 degrees), the 3D effect will be poor. Basically, when the eyes have different positions and the perspective will naturally have different parallax. According to the sight changes, the parallax from different scene can be divided into positive parallax, zero parallax and negative parallax. Fig. 4-9 shows the difference of three parallax types, where the L and R point is eye sight cross with the screen or reference plane.. Screen can be an overlapped image with left and a right sight. image.. Figure 4-9 Positive parallax, zero parallax and negative parallax.. 4.8.1. Positive Parallax (In Screen) Positive Parallax is the cross point of each eye sight, and its 3D image will appear behind the screen. The left and right eye images were placed in the location of the plane with the positive distance from point L and point R (see Fig 4-9). 40.

(50) 4.8.2. Zero Parallax (2D Object) Zero Parallax is left and right eye images overlap on the screen at point L and point R position. The intersection of the two focus attention would fall on the screen, and the image will show on the screen without 3D effect.. 4.8.3. Negative Parallax (Out of Screen) Negative parallax is left and right eye images cross screen plane. At the point L and point R position of the two eyes, the focus of attention will be in front of the screen, and its 3D images will appear as if they are in front of the screen plane. To produce depth of field effect, we can use negative parallax. Negative parallax objects look closer than the screen, and it makes people feel the objects flying out of the screen. The closer the screen is the smaller depth of field the image generates. When watching the original 3D image or film, the viewer may be easier to adapt to a different 3D scene than a “fake 3D” one.. 4.8.4. Parallax Adjustment Stereoscopic separation of parallax shift is applied to the geometry vertices. All the 3D objects should be rendered using a unique perspective projection in a given frame. Parallax controls the depth sense of brain vision of viewer. When the parallax is larger the viewer feel the object is much closer. Yet when the parallax is negative, it will show as if it is out of the screen plane, and the viewer will feel object is in the air. It has more vision effect to them. Due to that we project twice of the CT and SUV, when we overlap two images, we need to define which is much closer to the viewer. So we use the depth relation from the perspective projection to keep the closer pixel and then to delete the deeper pixel. In the negative parallax situation, we have to arrange the left pixel and right pixel, because we have four projection images, left CT and SUV, right CT and SUV. 41.

(51) projection after parallax adjustments. While switching the pixels to the other sides, we have to check the right image’s CT and SUV pixel’s overlapping situation simultaneously.. Figure 4-10 Parallax adjustment algorithm.. 42.

(52) Figure 4-11 Original left CT perspective projection.. Figure 4-12 Positive parallax of left CT projection.. Figure 4-13 Negative parallax of left CT projection. Pixels need to switch to the right CT.. 43.

(53) Figure 4-14 Right CT projection’s negative parallax pixels.. Figure 4-15 After switching negative parallax pixels.. Figure 4-16 Result with overlapping SUV projection.. 44.

(54) Figure 4-17 Right image with overlapping SUV projection.. Figure 4-18 Side-by-side image.. 45.