HIGH DYNAMIC RANGE IMAGE PROCESSING TO SCREEN DIABETIC RETINOPATHY
Ko-Yang Chen (陳科仰), Chiou-Shann Fuh (傅楸善), Kung-Bin Sung (宋孔彬), Ching-Yao Tsai(蔡景耀)
Graduate Institute of Biomedical Electronics and Bioinformatics, National Taiwan University E-mail: bebintu@kchen.org
ABSTRACT
In this paper, we process the ophthalmologic images of patients with diabetes clinically. By utilizing the tone-mapping techniques in high dynamic range imaging, we help facilitate the screening rate of diabetic retinopathy, and thus, decreasing the blindness rate of diabetic patients.
1. INTRODUCTION
In modern photography imaging, high dynamic range images are often generated in scenes with high contrast, such as a picture with both outdoor and indoor objects during daylight. If the outdoor objects are clear, which means the exposure level is correct for the outdoor objects, the indoor objects are often plain dark.
Similarly, when indoor objects are clear, outdoor objects are usually over-exposed. By applying tone mapping techniques to these images, we may produce an image that shows both outdoor and indoor objects clearly.
In patients with diabetes, we tend to request the patients coming back to the out-patient department for eye fundus examinations in order to screen diabetic retinopathy. Among these ophthalmologic images, the contrast between blood vessels and fundus is very limited, and it is often the case that clinical physicians are not able to spot the problems under ophthalmoscope.
This paper utilizes the tone mapping techniques in high dynamic range images and processes ophthalmological images so that patients with diabetes will benefit from successful early screening. A treatment plan can be drafted early in the stage, which will decrease the rate of blindness, improve the life quality of diabetic patients, and elongate the life span of such patients.
Diabetic retinopathy is one of the most important factors of visual loss. It is also a major cause
of morbidity in patients with diabetes [4]. When the duration of diabetes is longer, the prevalence of the diabetic retinopathy is also greater. Several preventive and therapeutic methods were reviewed to minimize the morbidity rate of diabetic retinopathy. Treatment of diabetic retinopathy should be done both at prevention by glycemic control and at treatment by early detection.
Most patients with diabetic retinopathy have no symptoms until late stages, which is usually too late for treatment.
Retinopathy is a major cause of blindness or morbidity in patients with Types 1 and 2 diabetes.
Incidence of blindness is 25 times higher [3]. Figure 1.1 is an example of normal vision, and Figure 1.2 is the same view with patients with diabetic retinopathy [11].
Figure 1.1 Normal vision.
Figure 1.2 The same view with diabetic retinopathy.
Diabetic Retinopathy (DR) is the most common cause of blindness in middle-aged subject [3].
Common phenomena of diabetic patients with retinopathy include cotton wool spots, microaneurysms, edema, exudates, and neovacularization of the retina.
These characteristics help clinical physicians to determine if a patient with diabetes is affected by retinopathy. Figure 1.3 is an example of the characteristic image of diabetic retinopathy [1].
Figure 1.3 Characteristics of retinal changes for diabetic patients.
Macular edema (ME) can occur at any stage of diabetic retinopathy. It is defined as retinal thickening and edema involving the macula, and it may be visualized by specialized fundus examination with stereoscopic viewing as shown in Figure 1.4 [12]., fluorescein angiography, and most directly, by Optical Coherence Tomography (OCT; a non-invasive low energy laser imaging technology) [5].
Clinically Significant Macular Edema (CSME) is defined as retinal thickening within 500 microns of the fovea, hard exudates within 500 microns of the fovea if associated with adjacent retinal thickening, or one or more areas of retinal thickening at least 1500 microns in diameter that is within one disc diameter (1500 microns) of the fovea [5].
Figure 1.4 Macular edema of diabetic patients
Retinal detachments can be rhegmatogenous (caused by a break in the retina; rhegma is Greek for tear) or nonrhegmatogenous (caused by leakage or exudation from beneath the retina or vitreous traction pulling on the retina). On occasion the retina appears to be detached but is actually not; this is termed pseudo retinal detachment.
A full thickness retinal break may exist as a round retinal hole or a linear or horseshoe-shaped retinal tear.
In both cases, there is a discontinuity in the retina that allows vitreous fluid to pass through the retinal break into the subretinal space, resulting in retinal detachment [2].
Diabetic retinopathy, shown in Figure 1.5 [9], falls into the category of traction retinal detachment, which is a certain kind of nonrhegmatogenous retinal detachment.
Traction retinal detachments occur when the vitreous has an abnormally strong attachment to the retina and contracts, thus pulling the retina off the back of the eye.
This is typically seen in patients who have proliferative diabetic retinopathy, retinopathy of prematurity, or sickle cell retinopathy, and who subsequently develop retinal neovascularization.
The areas of neovascularization and fibrosis between the retina and the vitreous create a strong adhesion between these two tissues. With vitreous contraction the retina is pulled away from the back of the eye and a traction retinal detachment is created [2].
Figure 1.5 Retinal detachment of diabetic patients.
Retinal haemorrhage is a disorder of the eye in which bleeding occurs into the retina. Figure 1.6 [15]
demonstrates the view under ophthalmoscope of patients with retinal hemorrhage.
The retina is a thin disc-shaped layer of light-sensitive tissue on the back wall of the eye. Its job is to translate what we see into neural impulses and send them to the brain via the optic nerve. A retinal haemorrhage can be caused by hypertension, retinal vein occlusion (a blockage of a retinal vein), or diabetes mellitus (which causes small fragile blood vessels to form, which are easily damaged). Retinal haemorrhages can also occur due to shaking, particularly in young infants (shaken baby syndrome) or from severe blows to the head [21].
Retinal haemorrhages that take place outside of the macula can go undetected for many years, and may sometimes only be picked up when the eye is examined in detail with an ophthalmoscope. However, some
retinal haemorrhages can cause severe impairment of vision.
Figure 1.6 Retinal hemorrhage of diabetic patients.
With complete vessel occlusion, parts of the retina become starved for nourishment. The ischemic retina responds by releasing angiogenic molecules like VEGF to promote new vessel growth. These new vessels serve to bypass the clogged arteries in order to resupply the starved retina [13]. A picture of neovascularization is shown in Figure 1.7 [13].
Figure 1.7 Retinal neovascularization of diabetic patients.
The newly formed vessels are abnormal in both appearance and function. The new vessels are friable and prone to leaking. They also grow in the wrong place, spreading and growing along the surface of the retina.
They can even grow off the retina, sprouting up into the vitreous fluid [13], as shown in Figure 1.8 [13].
Neovascularization is not just limited to the retina, but can also occur on the iris itself. NVI (Neo- Vascularization of the Iris) is an ominous sign, as the new vessels can cover the trabecular meshwork and create a sudden neovascular glaucoma [13].
Pathogenesis of DR is multifactorial but is primarily caused by the metabolic effects of chronic hyperglycemia. Possible causes include, results in vascular changes and retinal injury and ischemia, chronic hyperglycemia: primary cause of diabetic retinopathy [4], autoregulation of retinal blood flow,
growth factors, carbonic anhydrase, genetic factors, medications, relation to nephropathy.
2. SCREENING METHODS OF DIABETIC RETINOPATHY
Screening for Diabetic Retinopathy (DR) is important because the majority of patients who develop DR have no symptoms until Macular Edema (ME) and/or Proliferative Diabetic Retinopathy (PDR) are already present. In addition, the efficacy of laser photocoagulation in preventing visual loss from PDR and ME is well established in randomized trials[7].
Most patients who develop diabetic retinopathy have no symptoms until the very late stages, which is often too late for effective treatment. Efficacy of laser photocoaguation in preventing visual loss from Proliferative Diabetic Retinopathy (PDR) and Macular Edema (ME) is well established [7].
Laser therapy is more beneficial in preventing visual loss than reversing diminished visual acuity. Early detection through screening is important to preserve vision in patients with diabetes [7]. Figure 2.1 illustrates how laser therapy is performed on patients with retinopathy [17].
Figure 2.1 Illustration of how laser therapy is performed on patients with retinopathy.
Ophthalmoscopy is a test that allows a health professional to see inside the back of the eye (called the fundus) and other structures using a magnifying instrument (ophthalmoscope) and a light source. It is done as part of an eye examination and may be done as part of a routine physical examination. [18]. Figure 2.2 illustrates ophthalmoscopy [14].
Figure 2.2 Illustration of ophthalmoscopy.
Accuracy is lower when performed by primary care physicians than specialized ophthalmological technicians [10].
Seven-field stereoscopic fundus photography is a method for screening and keeping record of patients with diabetic retinopathy. It requires a trained photographer and a trained reader to interpret the results.
Figure 2.3 shows an image from seven-field stereoscopic fundus photography [16].
Figure 2.3 Seven-field stereoscopic fundus photography.
Similar to ophthalmoscopy except that the examinee can lean his forehead and keep in a fixed position.
Figure 2.4 shows a piece of typical fundal photography equipment [8].
Figure 2.4 Typical fundal photography equipment.
Digital stereoscopic retinal imaging allows remote interpretation by an ophthalmologist. The examination takes 15 to 20 minutes and does not require dilation of the eyes. Compared with dilated fundoscopic examination for retinopathy screening, digital imaging was concordant in 86 percent of cases [7]. Figure 2.5 demostrates how fundal photography is performed [6].
Figure 2.5 Demostration of how fundal photography is performed.
3. OUR PROPOSED METHOD
Several tone mapping operators were developed in the recent years [22]. They can all be divided in two main types:
1. Global (or spatially uniform) operators: they are non-linear functions based on the luminance and other global variables of the image. Once the optimal function has been estimated according to the particular image, every pixel in the image is mapped in the same way, independent of the value of surrounding pixels in the image. Those techniques are simple and fast (since they can be implemented using look-up-tables), but they can cause a loss of contrast [22].
2. Local (or spatially varying) operators: the parameters of the non-linear function change in each pixel, according to features extracted from the surrounding parameters. In other words, the effect of the algorithm changes in each pixel according to the local features of the image. Those algorithms are more complicated than the global ones, they can show artifacts (e.g. halo effect and ringing), the output can look un-realistic, but they can provide the best performance, since the human vision is mainly sensitive to local contrast [22].
During the experiment processes, both operators were tried out on ophthalmologic images. Local operators were used in the first place. However, different local operators more or less produce a ring-like bright area surrounding the edges of objects with high contrast. This phenomenon is called “halo-effect”, and
is not acceptable in medical images. The edges of blood vessels, optic discs, central fovea, and so on, are not in original state, and thus, they could result in incorrect diagnoses because the edges are an important factor determining whether the tissue or structure is normal.
Global operators were used next, and the results are satisfatory.
The images from ophthalmoscope are not necessarily exposed correctly, which means some images can be underexposed while others overexposed. Therefore, the first step of our method starts with histogram equalization in order to make each image exposure to be adjusted to its fullest potential.
After the intensities of the images are standardized, gamma compression is introduced to extract the brightness of the pixels that we want to observe. The luminances are compressed into Low Dynamic Range (LDR) display ranges via
where γ is a predetermined value of 0.48 after several tests so that the fundus and vessels are both more prominent in the images, while at the same time, preserved the vessels near optic discs, which is normally the brightest area in the image.
In order to get the maximum displayable dynamic range of a monitor out of an ophthalmologic image, we have to discard pixels that are of little clinical value. Pixels with intensities lower than 60 in 8-bit images are discarded before gamma compression is performed, since these pixels provide very limited information on the fundus and vessels [19].
The flow chart of our proposed method is shown as follows:
Function Process
Convert to Y, Cr and Cb Color Space
Histogram Equalization for
Y Channel in Unmasked
Pixels Mask Out Pixel
Intensity < 20
Convert Back to RGB Color
Space Gamma Correction, γ
= 0.48 Discard Pixel Intensity < 60
Save Image
4. EXPERIMENTS AND RESULTS
4.1 Experimental Environment
CPU: Intel (R) Core (TM)2 Duo CPU E4500
@2.20GHz Memory: 4 GB OS: Windows XP
Programming Language: Dev C++ 5.9.9.2
4.2 Image Results
new
vessels emboli
Dr.
Tsai Dr.
Li Dr.
Huan g
Dr.
Chen Dr. Tsai Dr.
Li Dr. HuangDr.
Chen
253-5330 3 14 11 11 0 0 0 1
253-5331 0 10 8 7 0 0 0 2
253-5332 5 11 14 8 0 0 0 2
253-5333 0 12 14 12 2 0 0 0
253-5334 3 8 6 9 2 0 0 0
255-5587 2 1 2 1 0 0 0 0
255-5588 0 1 1 1 2 0 3 1
255-5589 0 1 1 1 2 0 4 2
48 . 0 , =
= inγ
γ
out V
V
255-5590 1 0 5 3 8 10 10 7
255-5590_1 2 0 2 3 6 9 13 7
255-5591 0 0 2 3 0 8 0 7
258-5824 3 3 6 9 1 3 0 2
258-5826 4 6 10 3 0 1 0 3
258-5827 0 4 3 6 0 1 0 0
258-5828 0 5 1 3 0 0 0 1
258-5829 4 9 7 6 0 3 0 6
258-5830 3 10 6 8 0 0 0 0
263-6381 3 2 1 1 0 0 0 0
263-6384 1 2 1 1 0 0 0 0
263-6385 5 7 7 8 0 0 0 0
263-6395 4 2 4 2 0 0 0 0
270-7060 1 5 3 3 3 3 8 14
270-7063 0 3 2 2 0 0 4 1
270-7064 2 1 1 1 3 3 3 4
270-7065 0 3 2 2 0 5 7 5
270-7066 2 3 2 3 6 7 7 10
270-7067 3 0 4 4 4 7 8 10
270-7068 2 1 0 4 4 6 5 6
270-7069 0 2 3 3 0 6 4 10
270-7070 0 0 1 1 5 3 5 6
270-7071 0 3 1 1 7 3 5 6
270-7072 0 0 1 0 5 4 6 5
270-7073 2 1 2 2 0 2 4 7
270-7074 1 2 8 3 4 3 6 6
284-8443 2 6 5 6 0 0 0 0
284-8444 3 4 4 3 0 0 0 0
284-8445 3 2 5 3 0 0 0 0
284-8446 3 2 1 2 0 0 0 0
284-8447 1 4 2 3 0 0 0 0
284-8448 2 3 5 4 0 0 0 0
284-8449 2 4 4 3 0 0 0 0
284-8452 2 5 6 5 0 0 0 0
284-8453 5 2 5 4 0 0 0 0
284-8454 1 3 4 4 0 0 0 0
284-8455 6 5 4 5 0 0 0 0
284-8456 6 9 8 9 0 0 0 0
292-9242 5 3 3 2 0 1 1 1
292-9244 0 0 1 0 3 2 0 2
292-9245 2 3 2 2 2 1 1 2
292-9246 2 1 2 3 0 0 1 1
292-9247 4 3 1 2 0 0 2 1
292-9251 4 3 4 3 4 1 3 4
292-9252 3 2 4 3 0 0 1 0
292-9255 3 4 3 2 5 3 3 4
292-9257 3 3 5 4 1 2 0 2
Sum 117 203 220 207 79 97 114 148
Table 4.1 Results from ophthalmologists.
The results could be categorized as,
253-5330 to 251-5334: many vessels. This could result from cataract and opaque lenses since the original pictures are murky. A lot of details are retrieved from the image and intensified.
255-5587 to 255-5589: some vessels, some emboli. Most of the images in this group show few emboli in the originals. However, after image processing, emboli could be easily seen, and thus help clinical doctors to diagnose more quickly and easily.
255-5590 to 255-5591: some vessels, prominent emboli increase in numbers.
258-5824 to 263-6395: several vessels. Clarity of the originals are acceptable, thus, image processing is not so effective on these images.
270-7060 to 270-7074: some vessels, several emboli. Pre-existing emboli is obvious in the originals.
Therefore, image processing only boosts the image quality a little bit.
284-8443 to 284-8456: some vessels. The original images are clear.
292-9224 to 292-9257: some vessels, some emboli. The original images are clear and provide sufficient information for the doctors. But our method enhances the contrast, and thus, facilitates clinical diagnosis and treatment.
The resulting images show that our method improves the contrast of the fundus and blood vessels, making clinical doctors identify the emboli and clotting arteries more easily.
4.3 Discussion
Some details of the images are lost during the processing. On the images comparison, the membrane is not seen after processing. Bruch’s membrane is the innermost layer of the choroid. It is also called the vitreous lamina, because of its glassy microscopic appearance. Because of histogram equalization, the five layers of membrane cannot be seen because the fundus is brighter after processing. However, among diabetic retinopathy patients, the membrane is not one of the key elements in diagnosing.
Most images which suffer from detail loss are the one showed in Table 4.4. However, the vessels in this area are usually very thick, and if there is any kind of obstruction, it will show in peripheral vessels.
Therefore, it is safe to disregard the vessels disappearing in the final images.
These detail lost only happened in images 270-7069, 251-5331, 284-8447, among a total of 110 images processed. This only consists of less than 0.5% of the total images. Membrane disappearance only occurred in 270-7069. Other images showed missing vessels near optic discs because the brightness of these areas are usually the brightest part of the image, and some details were sacrificed during histogram equalization.
5. CONCLUSION
Our proposed method produces clearer blood vessels and emboli in most of the images, particularly the ones with cataract and opaque lenses since patients with these phenomena usually result in murky images under ophthalmoscope.
The images generally appeared to be dark before processing. After histogram equalization, the bright pixels and dark ones are more evenly distributed.
Further gamma compression cuts off unnecessary dark pixels which only include background fundus. As we are focusing on the vessels of the ophthalmologic images, fundus information can be omitted. However, bright pixels are preserved because there are major arteries surrounding optic disc, which is usually very bright. By sacrificing some dark pixels, we can get more available dynamic range to display vessel and blood clot edges more clearly.
Since laser therapy is very effective in early stages of patients with diabetic retinopathy, raising the screening rate simply by image processing is very helpful both to the doctors diagnosing and to the patients receiving proper treatment. Our proposed method enables clinical ophthalmologists to see blood clots (emboli) and distribution of blood vessels more easily, which decreases the blindness rate of diabetic patients effectively.
Current processing techniques emphasize on vessels and blood clots, especially on patients with cataract and opaque lenses. The resulting images are very helpful to clinical doctors when diagnosing diabetic retinopathy.
Optic discs and vicinities are usually overexposed after the processing. This phenomenon is acceptable with diabetic patients since the clotting vessels are usually located outside this area.
We could adjust the parameters to preserve the details in the optic disc area for ophthalmologists who
specialize in such area; we could also optimize for the fundus so the gradient colors can be emphasized for other specialties. These functions can be integrated in the program, so the clinical doctors can pick the area he or she wants to focus on in order to diagnose or make treatment plans.
REFERENCES
[1] Arleo Eye Associates, “What Is Diabetic Retinopathy,”
http://www.arleoeye.com/Pages/retin.html, 2009.
[2] J. G Arroyo, "Retinal Tear and Detachment,"
http://www.utdol.com/online/content/topic.do?topicKey=p riophth/8273&selectedTitle=1~79&source=search_result, 2009.
[3] Centers for Disease Control and Prevention, “National Estimates on Diabetes,” National Diabetes FactSheet, http://www.cdc.gov/diabetes/pubs/estimates.htm#complica tions, 2006.
[4] R. N. Frank, “Diabetic Retinopathy,” New England Journal of Medicine, Vol. 350, No. 1, pp. 48-58, 2004.
[5] C. E. Fraser and D. J. D'Amico, "Classification and Clinical Features of Diabetic Retinopathy,"
http://www.utdol.com/online/content/topic.do?topicKey=d iabetes/21756&selectedTitle=1~57&source=search_result
#5, 2009.
[6] Liverpool Ocular Oncology Centre, "First Visit,"
http://www.eyetumour.com/visit_first.php, 2009.
[7] D. K. McCulloch, "Screening for Diabetic Retinopathy,"
http://www.utdol.com/online/content/topic.do?topicKey=d iabetes/11804&selectedTitle=3~90&source=search_result, 2009.
[8] Image-Acquire.com, "Canon CF-1 Digital Retinal Camera," http://www.image-acquire.com/canon-cf-1- digital-retinal-camera, 2009.
[9] T. L. Jackson, “Clinical Notes,”
http://www.timothyjackson.net/GP%20clinical%20notes.h tml, 2009.
[10] J. P. O'Hare, A. Hopper, C. Madhaven, M. Charny T.
S. Purewell, B. Harney, and J. Griffiths, "Adding Retinal Photography to Screening for Diabetic Retinopathy: A Prospective Study in Primary Care," British Medical Journal, Vol. 16;312(7032), pp. 679-82, 1996.
[11] National Institute of Health, “NIHSeniorHealth:
Diabetic Retinopathy - Causes and Risk Factors,"
http://nihseniorhealth.gov/diabeticretinopathy/causesandris kfactors/02.html, 2005.
[12] National Eye Institute, National Institutes of Health,
“Diabetic Macular Edema,”
http://www.nei.nih.gov/photo/eyedis/index.asp, 2006.
[13] Ophthobook.com, "Beginner's Guide to the Retina,"
http://www.ophthobook.com/chapters/retina, 2008.
[14] H. D. Riley, "Using the Direct Ophthalmoscope,"
http://www.opt.indiana.edu/riley/HomePage/Direct_Oscop e/Text_Direct_Oscopt.html, 2009.
[15] S. M. Sastry, "Eye Disease Library,"
http://www.bethesdaretina.com/library.htm, 2009.
[16] Visual Pathways, "Automated Retinal Imaging System," http://www.vispath.com/aris.html, 2009.
[17] Vitreoretinal Consultants, "Diseases and Surgery of
the Retina and Vitreous,"
http://www.houstonretina.com/laserTreatment.html, 2006
[18] WebMD, "Eye Health Center,"
http://www.webmd.com/eye-health/ophthalmoscopy, 2007
[19] Wikipedia, "Gamma Correction,"
http://en.wikipedia.org/wiki/Gamma_compression, 2009.
[20] Wikipedia, "Histogram Equalization,"
http://en.wikipedia.org/wiki/Histogram_equalization, 2009.
[21] Wikipedia, "Retinal Hemorrhage,"
http://en.wikipedia.org/wiki/Retinal_haemorrhage, 2009.
[22] Wikipedia, "Tone Mapping,"
http://en.wikipedia.org/wiki/Tone_mapping, 2009.
