server, they proposed a tone mapping operator for grey-level HDRIs. Ward [18] adopts a simple linear operator in his global tone mapping method that can preserve apparent contrast and visi-bility.
Ferwerda et al.[4] first applied psychophysical studies to their tone mapping operator, which can capture the image properties of color appearance, visual acuity and light/dark adaptation.
Ward et al. [19] used the histogram adjustment technique to define their tone mapping function.
As human eyes have the best view in the fovea, they filtered an input HDRI to obtain a foveal sample image. Histogram adjustment is then performed according to the foveal image. Ward et al. assume that all pixels would participate in the adaptation; however, in human visual system, eye movements are critical for acquiring and processing visual information [20]. Therefore, it should be more reasonable to compute the foveal sample image based on a viewers fixation positions.
Pattanaik et al. [10] used an adaptation model, which is a simplification of Hunt’s model [21], to transform the luminance of a scene into retinal-response-like vectors, which are con-verted into appearance vectors. The dispaly intensity of the rendered scence is then computed from the appearance vectors using the inverse appearance and adaptation model. Mantiuk et al.
[22] formulate tone mapping as an optimization problem, which minimizes the visible contrast distortions between a human visual system and the display. Van Hateren [23] used the model of human cones to perform tone mapping in which all components are represented as temporal kernels to transform HDR Images/videos into LDR images/videos.
2.2 Local tone mapping methods
Global tone mapping is usually used to compress an HDRI [5] for its computational efficiency;
however, it may cause some loss of details since all pixels are transformed using the same mapping function
2.2 Local tone mapping methods 6
It is more desirable to have the tone mapping function at different locations be adaptively adjusted to preserve the details. Chiu et al. [24] defined a scaling function and used it as guide-line to scale the pixel values. Tumblin and Greg [25] noticed that the artists usually compress the contrast of large features and add the details in their drawings. They applied anisotropic diffusion [26], which treats the intensity as heat, to find the boundary of an object in an image and proposed a detail-preserving contrast reduction method to mimic an artist’s drawing pro-cess. Fattal et al. [27] compressed large magnitude of gradient and solved a Poisson equation to obtain LDR images. Their method avoids some noisy appearance of LCIS methods.
Durand and Dorsey [28] showed that the bilateral filtering [29] is a robust statistical esti-mator. It is an edge-preserving smoothing operator and has the similar property of anisotropic diffusion. The authors used two-scale decomposition: base image and detail image. The base image was created by bilateral filtering the input HDRI such that the high luminance is pre-served. The detail image was the division of the intensity by the base image. They performed contrast reduction on the base image and multiplied the result with the detail image to produce LDRI.
Inspired by the photographic technique, Reinhard et al. [30] adopted the concepts of the zone system [31][32][33] and the dodging-and-burning technique to perform the dynamic range compression. Chen et al. [34] defined different tone mapping functions for different objects in an image, in which objects are detected using the Earth Mover’s Distance (EMD) [35].
Recently, several interactive tools are developed to perform tone mapping locally. Lischinski et al. [36] used brush and stroke to set constraints, which are propagated to form a completely adjusted tone map under an image-guided energy minimization framework. Liang et al. [37]
used a simple touch screen to set constraints and modified the stroke-based algorithm [36] so that the local tone mapping can be executed efficiently on mobile devices
C H A P T E R 3
Background
In this chapter, we introduce attention and adaptation in the human visual system and their com-putational models that are used in our tone mapping algorithm. There are two kinds of attention activities in human visual system [38]. One is transient or exogenous attention and the other is sustained or endogenous attention. Transient attention is an unconscious and bottom-up pro-cess, which is affected by salient stimulus in the scene. Sustained attention is a task-relevant and top-down process. It is voluntary and the reason why we can pay attention to our interest-ing information. Attention influences the performance of many visual tasks. In particular, it affects the visual adaptation mechanism as well. We will introduce the psychophysical studies about the interaction between attention and adaptation in the fields of neuroscience and vision in section 3.1.
In section 3.2, we briefly describe the visual attention model used in our tone mapping approach. We adopt the HDR saliency map, which is developed based on the saliency map proposed by Itti et al. [3], to simulate the activity of transient attention because saliency map is constructed from the low-level image features, which is a bottom-up process. Saliency map
7
3.1 Adaptation and attention in Human Visual System 8
has also been widely used in video compression, image processing, global illumination, and computer vision. We do not explicitly model the effect of sustained attention in tone mapping function for two reasons. First, sustained attention is task-dependent and related to cognition;
however, the relation between cognition and sustained attention is still study. Therefore, it is difficult to build the computational model of sustained attention. Second, the interaction be-tween sustained attention and contrast sensitivity can be roughly considered as a mixed effect of transient attention and adaptation according to the psychophysical study on sustained atten-tion [2]. Therefore, the effect of sustained attenatten-tion is implicitly modeled in the adaptaatten-tion mechanism in our approach.
3.1 Adaptation and attention in Human Visual System
In the vision field, there are many studies about the interaction between attention and adaptation [1][12][13][14]. Figure 3.1 describes how attention and adaptation affect the contrast response in visual neurons. The black curve represents the situation when there are no effects of attention and adaptation. When our eyes adapt to the luminance of a scene, our visual neurons would be less sensitive to contrast and the black curve would shift toward the green curve. In other words, the contrast of the scene needs to be increased to trigger the same level of neural response when the adaptation effect occurs. Adaptation also affects our ability to detect the just-noticeable contrast difference [39]. Our eyes can discriminate finer contrast difference more efficiently before adaptation.
In contrast to adaptation, attention induces an opposite effect to the contrast response of visual neurons. Attention effect would move the black curve toward the red curve in Figure 3.1, i.e., neurons need less contrast to attain the same response. Contrast appears more intense when humans’ eyes pay attention to as attention stimulates the visual neurons causing stronger response. Psychophysical studies [40] suggest that transient attention changes the response gain modulation, which leads to greater attention modulation for high contrast level and affects the
3.1 Adaptation and attention in Human Visual System 9
asymptotes of the curve. Sustained attention influences the contrast gain modulation, which shifts the curve but does not affect the asymptotes.
Figure 3.1: This figure shows a typical contrast response function of a hypothetical visual neuron summarized by Pestilli et al.[1]. The black curve is the response of the neuron under neutral condition. The red curve shows the effect of attention on the response, which increases the sensitivity of the neuron. The green curve describes the effect of adaptation which causes the neuron to need more contrast in order to achieve the same magnitude of response.
3.1 Adaptation and attention in Human Visual System 10
The interaction between transient attention and adaptation was explored in the psychophys-ical study by Pestilli et al. [1]. Figure 3.2 shows their experiments. At the beginning of each trial, subjects must adapt to one of the two adaptation conditions: adapt-0 and adapt-100. At the adapt-0 condition, subjects adapt to the background of the stimulus for 20 s. At the adapt-100 condition, subjects were stimulated with two counterphase flickering Gabor patches with 100%
contrast for 70 s. After adapting, a white rectangle was shown as a guide cue in order to ma-nipulate transient attention during the test trial. This guide cue presented at the fixation point (neutral) or above one of the Gabor patches (peripheral) for 50 ms. Transient attention would be directed to the location of white rectangle instantaneously. After 50 ms, there was an inter stimulus interval (ISI) for 50 ms. After the duration of ISI, the two tilted test Gabor patches with contrast less than 100% were presented simultaneously for 30 ms. After presenting stimulus, subjects need to answer the orientation of stimulus, which is indicated by response cue.
There are three attentional conditions in each test trials as shown in Figure 3.3. In the neutral-cue condition, which is the control group, the guide cue was in neutral condition. Sub-jects would be asked to discriminate the orientation of the Gabor patch indicated by the response cue. In the valid-cue or attentive condition, the guide cue and response cue were at the same direction and subjects needed to report the tilt of the Gabor patch presented by response cue. In the invalid-cue or nonattentative condition, subjects would need to discriminate the tilt of Gabor filter not preceded by peripheral cue.
3.1 Adaptation and attention in Human Visual System 11
Figure 3.2: stimuli used in the experiments of Pestilli et al. [1]. Their experiment is a two-alternative forced-choice (2FAC) orientation-discrimination task on Gabor patches.
Figure 3.3: This figure shows three attentional conditions in Pestilli et al.[1].
3.1 Adaptation and attention in Human Visual System 12
Figure 3.4 shows the results of Pestilli et al. [1]. In their findings, in order to obtain the same accuracy in the attentive condition, subjects need to receive higher contrast if they do not attend to the Gabor patch indicated by the response cue in advance. Attention would increase the contrast sensitivity in the valid-cue situation. The blue horizontal lines show the estimated threshold values at a 70% accuracy. Full adaptation (adapt-100) at the beginning would increase the contrast threshold. They also noted that adaptation only affects the threshold, while attention affects both the threshold and the asymptotes.
Figure 3.4: This figure shows the findings in Pestilli et al. [1]. Each column reports data from an observer. The abscissa is contrast. The ordinate is the percentage of questions that a subject answered correctly. The black, red, and green curve represents the neutral, valid, and invalid condition, respectively The blue horizontal line means the estimated threshold value at 70%
accuracy