Face Configuration Processing in the
Human Brain: The Role of Symmetry
Chien-Chung Chen1,2, Kai-Ling C. Kao1and Christopher W. Tyler2
1
Department of Psychology, National Taiwan University, Taipei 106, Taiwan and2Smith-Kettlewell Eye Research Institute, San Francisco, CA 94115, USA
Symmetry is an important cue in face perception. We manipulated symmetry and other configurational variables to study their role in face processing in the human brain. We employed 2 types of symmetry: image symmetry (where one part of the image is defined as the mirrored transform of the other part about an axis) and object symmetry (where the spatial relationships among the image components are interpreted as parts of a symmetric 3-dimensional object). We compared blood oxygenation level dependent re-sponses in healthy human observers for upright front-view faces with responses to different symmetry-controlled images. The cortical areas activated by the face images, relative to Fourier-matched scrambled images, were the fusiform (FFA) and occipital (OFA) face areas, the middle occipital gyri (MOG), and areas around the superior temporal and intraoccipital sulci (IOS). Contrasting faces and their image-symmetric scrambled versions showed a similar activation pattern except in the right OFA, suggesting an involvement in facial symmetry processing. The upright versus inverted faces (with the same image symmetry but unfamiliar object identity) showed robust differential activation in the FFA, OFA, MOG, IOS, and precuneus. The response to frontal-view versus 3/4-view faces (having the same object symmetry but disrupted image symmetry) showed little differential activation in the FFA or the OFA but strong responses in the MOG and IOS, suggesting that face processing in the FFA and the OFA is holistic and viewpoint invariant.
Keywords: fMRI, fusiform, intraoccipital sulcus, inverted face, object perception, occipital face area
Introduction
Face recognition and discrimination may be one of the most developed perceptual skills of visual object processing. An adult can discriminate and recognize hundreds of faces (Bharick and others 1975). Much evidence has shown that face perception cannot be achieved by analyzing facial features alone and that facial configuration, or the spatial relationships among facial features, is crucial to face perception (Fantz 1961; Valentine 1988; O’Toole and others 1994; Leder and Bruce 2000; Webster and others 2004).
There are, however, many possible spatial relations among facial features. It is not clear what types of spatial relations are essential for a human observer to analyze the facial configura-tion. Much attention on facial configuration has focused on the relative distance between features (e. g., Fellous 1997), such as the distance from the center of the face to several landmarks on a face (e.g., Anderson and Wilson 2005), from the eyes to the eyebrows (e.g., Brown and Perrett 1993), from the eyes to the nose and the mouth (Hosie and others 1988), and so on. On the other hand, there are reports that configuration cues such as
mirror symmetry are important in face perception. There is a well-established relationship between facial symmetry and perceived attractiveness (or beauty) of potential mates in humans (Grammer and Thornhill 1994; Mealey and others 1999; Rhodes and others 2001, 2002; Jacobsen and Hofel 2002). There is also a preference for facial asymmetry that is related to emotion expressiveness (Swaddle and Cuthill 1995; Zaidel and others 1995; Kowner 1996). Recently, Rhodes and others (2005) showed that humans can detect symmetry better in upright faces than in either inverted faces or contrast reverted faces and argued that there should be a mechanism in the human visual system specific for the symmetry of faces.
Brain areas supporting face processing have been reported both in monkey (Perrett and others 1982) and in humans (Kanwisher and others 1997; McCarthy and others 1997; Halgren and others 1999; Haxby and others 2000). In particular, part of the fusiform gyrus and the inferior occipital gyrus have been reported as specific for face perception and are designated the fusiform face area (FFA, Kanwisher and others 1997) and occipital face area (OFA, Halgren and others 1999; Rossion and others 2003), respectively (although there is disagreement on the exact role of the FFA, see Gauthier and others 1999; Haxby and others 2001). It is, however, not clear whether these brain areas are involved in the analysis of local facial features, face-specific configurations, or other gestalt properties of faces. In behavioral studies, the configuration-specific system is generally isolated by contrasting performance for upright faces and inverted faces (Yin 1969; Carey and Diamond 1977; Thompson 1980). A manipulation that causes performance degradation in inverted relative to upright faces is considered to reflect processing by mechanisms specific to the absolute facial con-figuration because the relative concon-figuration of the features is undisturbed by rotation.
There have been several neuroimaging studies contrasting brain activation to upright and inverted faces, but significant differential activations for upright face vs. inverted face have not been reported in either FFA or OFA (Kanwisher and others 1998; Aguirre and others 1999; Haxby and others 1999) until recently (Yovel and Kanwisher 2004). Because faces are much less recognizable when upside down, the implication of the predominantly negative result is that these areas are responding to the basic aspects of facial stimuli that are equally present when the face is inverted, such as the local features and the configuration aspects such as symmetry, rather than the higher-level perceptual processes of facial recognition, which are substantially degraded by inversion.
Our purpose here is to identify brain areas specific for the configuration properties of facial symmetry. Given recent reports on brain areas responding to image symmetry per se Cerebral Cortex June 2007;17:1423--1432
doi:10.1093/cercor/bhl054
(Sasaki and others 2005; Tyler and others 2005a) and the behavioral evidence for facial symmetry mechanisms (Rhodes and others 2005), we are interested in the role of symmetry in brain areas responding to faces. We may conceptualize 2 types of face symmetry. The first type, ‘‘image symmetry,’’ occurs when one image component is a mirrored transform of another image component about some axis of transformation. This type of symmetry specifies the direct spatial relations between the parts of the 2-dimensional (2D) projections of the faces and is not concerned with the interpretation of the image as an object in space. The second type of symmetry is ‘‘object symmetry,’’ which specifies the spatial relationships among the components of the image interpreted as the representation of a 3D object. Hence, those components or facial features will be processed through a face gestalt or 3D template of the human head. This kind of object symmetry may not be consistent with image symmetry due to an asymmetric viewpoint of the observer or the rotation of the object’s axis of symmetry away from the line of sight. Thus, a 3/4 view of a symmetric face forms an asymmetric image, even though it is the image of a symmetric object and can be understood to have accurate object symme-try. Our experiments were designed to test the role of these 2 types of symmetry in the facial representations of the FFA and the OFA. In detail, our goals were as follows:
1. To reveal all relevant areas supporting face processing (except those for early visual processing), we first contrasted brain activations between faces and asymmetric scrambled versions of the same images with identical Fourier energy. These images should generate the same net activation for arrays of neurons acting as local linear filters.
2. To identify the cortical areas responding to symmetry per se, we compared the activation for the asymmetric random images and symmetricized versions of the same images (as in Tyler and others 2005a).
3. To identify cortical regions involved in the processes of face recognition, we then compared brain activation of the front-view faces and their inverted versions. Inverted faces are known to disrupt the facial configuration mechanisms and degrade the function of the face recognition mechanisms. Thus, because inverted faces maintain the same local features, the same image symmetry, and the same object symmetry as the upright faces, the null hypothesis is that they should produce no activation to regions containing either local mechanisms of symmetry processing for facial recognition. Hence, any significant differences in activation for the upright versus inverted faces should reveal the operation of holistic facial configuration mechanisms.
4. To examine whether the differential activation patterns from (3) are due to face-specific symmetry or to early image symmetry processing, we also contrasted front-view upright faces with their phase-scrambled versions that were verti-cally symmetric. A contrast between faces and vertiverti-cally symmetric scrambled images should reveal all relevant areas supporting face processing that are insensitive to symmetry. A comparison of the last 2 experiments should thus uncover whether areas that are attributed to face processing are actually responding to the symmetry of the faces rather than more informative cues.
5. Finally, to study the role of object versus image symmetry (under the assumption that faces are symmetric objects), we compared the responses to upright frontal-view faces and 3/4-view faces of the same persons. The 3/4-view faces retain the same semantic and object-centered properties as the frontal-view faces but have radically altered image symmetry. Cortical areas encoding image symmetry should be differentially activated by this comparison, but those encoding object symmetry or semantic content should show little or no differential activation with the change in viewpoint of the same faces.
Methods Stimuli
Figure 1 shows examples of the stimuli used in our experiment. We had 5 types of images: upright frontal-view faces, inverted faces, 3/4-view faces, symmetric scrambled images, and asymmetric scrambled images. The face images were taken from The Facial Recognition Technology (FERET) Database (Phillips and others 2000) provided by the National Institute of Standards and Technology, USA. We computed the symme-try index (Tyler and others, 2003) of the frontal-view faces of the 1010 subjects in the FERET database and selected the 36 face images with greatest symmetry index values. The symmetry index is computed based on the power spectrum of the Fourier transform of the face images. Here we were only interested in the horizontal symmetry. Hence, we computed the difference of the power at the points (kx, ky)
and (–kx, ky), where kx and ky are horizontal and vertical spatial
frequencies of the images (in the upper half-plane, excluding the horizontal axis). The symmetry index is computed as a function of the root mean square difference of the power between corresponding frequencies summed over the spectrum.
The inverted faces were simply the upside-down versions of the same 36 images. The 3/4-view images were pictures of the same persons available from the database with their head rotated 67.5° either to their left (17 images) or to their right (19 images). The 2 types of scrambled images were created by manipulating the phase spectrum of the frontal-view face images. We first normalized the face images by subtracting their mean luminance values to remove the DC component and then computed their Fourier transforms. The phase spectra were scrambled
randomly for the asymmetric scrambled images and set to zero phase for the symmetric scrambled images. We then generated the control images by taking inverse Fourier transforms of the original amplitude spectra combined with the new phase spectra of the 2 types. Note that this novel zero-phase manipulation introduces noticeable edge structures throughout the images similar to that of the intact faces, so it tends to equate for nonlinear edge processing in the face images in addition to the presence of bilateral symmetry. The luminance and contrast of the inverse Fourier-transformed images were normalized to match the mean luminance and contrast energy of the face images. The phase zeroed scrambled images had point symmetry, so they were converted to bilateral symmetry by inverting their left half relative to the right half.
A fourth-power Gaussian aperture (1/e scale parameter=3.2°) was applied to all images to remove image features outside face areas (cf., Tyler and Chen forthcoming). A fixation point of 6.75936.759 size was inserted at the center of the face image.
Procedure
We used the blood oxygenation level dependent (BOLD) signal from a block-design functional Magnetic Resonance Imaging (fMRI) paradigm to measure the cortical response to the stimulus contrasts. Each of the 4 experimental runs had 18-s epochs of upright frontal-view faces alternating with 18-s epochs of one of the other image types in 6 cycles (36-s period) of the 2-epoch block alternation. Each 18-s epoch consisted of 18 presentations of the images for 850 ms followed by a 150-ms blank period. To keep the observers’ attention on the presented images regardless of block content, the observers were instructed to press a response key when the currently presented image physically matched the previous one (one-back task).
Stimuli were delivered with MRVision 2000 goggles from Resonance Technology, Inc. (Northridge, CA). At 800 (H) by 600 (V) resolution, the pixel size of the display was specified as 2.259.
Six observers participated in this study, including 2 of the authors and 4 paid observers naı¨ve to the purpose of the study. All observers had normal or corrected-to-normal (by the focusing lenses included with the goggles) visual acuity. All observers gave informed consent for their participation.
Data Acquisition and Analysis
The images were collected with a Bruker 3T scanner located at National Taiwan University. Within each scanning session, both functional (T2*
weighted, BOLD) responses and anatomical (T1weighted) images were
acquired in identical planes. The images were collected in 20 transverse planes parallel to the anterior commissure--posterior commissure line. An echo-planar imaging (EPI) sequence (Stehling and others 1991) was used to acquire the functional data (time repetition=3000 ms, time echo=
40 ms, flip angle=90°, voxel resolution=2.3432.3433 mm). A high-resolution anatomical (T1weighted) MRI volume scan of the entire head
was run once on each observer (voxel size=13131 mm). The main experiment lasted 225 s (75 images). The first 9 s (3 images) were excluded from further analyses to avoid the start-up transient. Thus, the data analyzed for each scan spanned 216 s (72 images). To correct head-motion artifacts, we used SPM (Friston and others 1995) to realign the EPI images acquired. The realigned images, as well as the anatomic images, were then normalized to a standard template with SPM. The normalized images were fed to the mrVista software (Wandell and others 2000) for coregistration, data analysis, and 3D visualization. Statistical analysis of the BOLD activation was based on the spectral correlation between the BOLD activation time series and the experi-mental sequence (Engel and others 1997). For each run, the Fourier transform was calculated for each BOLD time series. The spectral correlation, or coherence, was then computed as the power at the frequency of block alternation divided by the square root of the energy of the spectrum for each voxel. The signal amplitude is given by the square root of the signal power.
There were situations where we needed to test differential contrasts across 2 or more block design runs. For this purpose, we followed the procedure proposed by Joseph and others (2002). That is, we first identified a region of interest (ROI) defined by a specific parametric contrast from coherence maps for individual runs. The signal amplitude of the averaged waveform over the voxels in this ROI from each
observer was then fed to a repeated measure analysis of variance (ANOVA) that assessed the effect of conditions for the specified contrast. In addition, to get detailed information of the differential contrasts in specific ROIs defined by the localizers, we followed up significant main effects in the ANOVA with pairwise comparison t-tests of differences in activation between conditions in the specified ROIs. Localizers
In addition to the main experiments, all observers participated in localizer experiments in a separate session, in order to identify relevant brain areas reported in the literature. The lateral occipital complex (LOC) was identified by the differential BOLD activation to pictures of common objects (objects that can be seen everyday life) versus block-scrambled versions of themselves, as developed by Kourtzi and
Kanwisher (2001). The human motion sensitive complex (hMT+) was
identified by the differential BOLD activation to low-contrast moving dots and their stationary versions. The moving dots (4.59 by 4.59 square) oscillated between expansion and contraction once per second with a spatial density of 2% and a speed of 4.5°/s. The retinotopic areas were identified with checkerboard rotating wedges that spanned 180° and rotated 30° every 3 s, taking 36 s to go around the display, with 6 periods in one retinotopic run. The symmetry areas were identified by the differential BOLD activation to the symmetric versus the asymmetric scrambled images.
Result and Discussion Localizers
The symmetry localizer consisted of two types of nonsense scrambled images computed from the Fourier spectrum of the face stimuli: one vertically symmetric and the other nonsym-metric. Figure 2(a) shows the activation map averaged across observers on 3 views of the inflated brain. The gray shading denotes the gyral (light) and sulcal (dark) layout. The pseudo-color denotes the average coherence between the response waveform and the experimental sequence. The predominant symmetry response occurred in the inferior occipital gyrus, the middle occipital gyrus (MOG) and the areas around intra-occipital sulcus (IOS). The MOG activation is posterior to hMT+(red contour) and partially overlaps with LOC (magenta contour). The dorsal predominance of the symmetry response is consistent with that observed by Tyler and others (2005a) and Sasaki and others (2005).
Figure 2(b) shows areas localized by localizers on a flatmap. The flatmaps here and in the remainder of this paper had their center near the occipital poles and extended 80 mm in radius around this point. The areas delineated by yellow borders are the first-tier retinotopic areas (V1--V3), identified with a rotating wedge. The criterion for delineating visual areas is discussed elsewhere (Tyler and others 2005b). The magenta borders denote the LOC as identified by contrasting BOLD activation to pictures of objects and their scrambled versions. The red borders denote hMT+ as identified by the motion localizer. The blue contours delimit the extent of the symmetry response. The outline of these areas will be shown for reference in the subsequent flat maps.
Brain Areas for Face Processing
Figure 3 (row 1) shows the averaged activation map across observers for a face localizer contrasting the upright frontal-view faces and the asymmetric scrambled images. We show selected slices of all conditions contrasting upright front-view faces with other images in this figure for ready compar-ison. The pseudocolored voxels in the figures denote voxels
with coherence above 0.475 or Bonferroni corrected P-value (considering all cortical voxels) less than 0.001.
Compared with the scrambled images, the face images bilaterally activated the fusiform, the inferior occipital, and the middle occipital gyri areas around the superior temporal sulcus (STS) and areas around the IOS. The Talairach coordi-nates (Talairach and Tournoux, 1988) of those activated areas are given in Table 1. Notice that the first-tier retinotopic areas did not show significant differential activation between the two types of images. On average, the BOLD signal modulations in V1 of the left and the right hemispheres were 0.24% and 0.20% (with confidence intervals of 0.29% and 0.24%), respectively. These values are not statistically significant and are much (significantly) smaller than typical V1 responses to contrast energy change, which is about 1--2% modulation, with a confi-dence interval of about 10% of this value (Engel and others 1997; Tootell and others 1998; Boynton and others 1999). This null result demonstrates that our phase spectrum manipulation equated all the relevant activations in early vision within experimental error. This is an important result because many previous approaches to equating images with their scrambled versions have resulted in significant activation in the early retinotopic areas (Murray and others 2002, 2004; Altmann and others 2003, 2004). It is therefore a worthwhile advance to have
developed a form of scrambling that reduces the residual activation in early areas below the typical level of fMRI detectability.
The IOS activation extended to a more dorsal region along the sulcus than did the symmetry localizer. The green symbols denote the mean locations of the FFA reported in the literature (triangles: averaged from individuals in Kanwisher and others 1997; diamonds: average reported in Rossion and others 2003; circle: reported value in Halgren and others 1999). The cyan symbols denote the OFA reported in Rossion and others (2003). The fusiform form gyrus activation is consistent with the FFA Figure 3. Average group activations shown in standardized brain slices for (a) face localizer, (b) symmetry localizer, (c) face versus inverted face, and (d) frontal versus rotated faces. Colored symbols indicated mean locations of activations from other studies (see key). Activation colors as in Figure 2.
Figure 2. (a) Activation maps for the symmetry localizer (vs. Fourier-equated random images) averaged across observers on 3 views of the inflated brain. Gyral (light) and sulcal (dark) denoted by gray shading. Orange color denotes the average activation pattern. Anterior (A) and posterior (P) directions indicated on underside view. (b) The symmetry-specific activation (yellow-orange coloration) depicted in flatmaps centered on the occipital poles. Yellow borders: V1--V3; magenta borders: LOC; red borders: hMT+. Yellow-orange coloration codes amplitude of significant activation at corrected P<0.001 (color bar). The dark blue border denotes the extent of the activated areas.
Table 1
Brain areas responding to face information
Activated areas Talairach coordinates
Fusiform Left (ÿ32, ÿ46, ÿ17)
Right (28,ÿ57, ÿ14)
Inferior occipital gyrus Left (ÿ28, ÿ70, ÿ15)
Right (29,ÿ77, ÿ15)
IOS Left (ÿ31, ÿ75, 16)
Right (24,ÿ86, 13)
MOG Left (ÿ30, ÿ84, 1)
and the inferior occipital gyrus activation is consistent with the OFA reported in the literature. Regions around the STS have been reported as relating to face processing both in humans (Puce and others 1998) and in monkeys (Perrett and others 1982). The IOS activation partially overlaps with the area identified by the symmetry localizer.
For a better understanding of the spatial relationships between the activated areas, Figure 4 shows an occipital flatmap and inflated version of the same activation. As in Figure 2, the flatmap is centered at the occipital pole and displays the spatial location of the activation across the cortical surface. The first-tier retinotopic areas and the human motion area, identified in separate scans, are retained for reference. It may be seen that the face-specific activation runs in a strip lying between these two reference regions. The concentrations corresponding to the FFA and OFA from other studies are identified by triangles and diamonds, respectively. In addition, there is a pronounced dorsal activation in the IOS centered at Talairach (–37, 75, 16) for the left and (24, –86, 13) for the right hemisphere. This region has rarely been mentioned in previous studies of face-specific activation, but appears to be the strongest site of activation in the present dataset. The subsequent conditions are designed to disentangle their particular roles in the agglomer-ation of cues present in the face localizer.
Image Symmetry Effect
The contrast between the upright frontal-view faces and the symmetric scrambled images (Fig. 3, row 2) was designed to remove the factor of symmetry from the cortical face response in the previous condition. This contrast showed significant activation in the fusiform, the inferior occipital and the middle occipital gyri, and also the area around the IOS. Figure 5 compares the differential activation (pseudocolor patches) in relation to the symmetry (blue contour) and the face areas (FFA: green contours, OFA: cyan contours, and IOS: black contours) identified above. Qualitatively speaking, the major difference between this and the previous condition contrasting with asymmetric scrambled was in the right OFA. The number of activated voxels dropped from 82 to 10 in the right OFA when the symmetry information was available in the scrambled images. The differential contrast (Joseph and others 2002) between the face versus symmetric scrambled condition and the face versus asymmetric scrambled was also significant (F5,10= 9.98, P=0.0251 <0.05) in the right OFA. The average response amplitude drops from 0.33% to 0.19% (t (5)=3.87, P=0.0059 <
0.05). On the other hand, the left OFA and the bilateral FFA did not show significant activation response change when symme-try was present in the scrambled images. The activation changes were only from 0.49% to 0.46% (t (5)=0.58, P=0.29
> 0.05, not significant [NS]) for the left OFA, from 0.49% to 0.41% (t (5)=1.56, P=0.09 >0.05, NS) for the right fusiform, and 0.44% to 0.35% (t (5)=1.21, P=0.14>0.05, NS) for the left.
Figure 5. Activation maps for the face responses versus symmetry-equated scrambled averaged across observers (a) on 3 views of the inflated brain and (b) occipital-pole flatmaps. FFA: green borders; OFA: cyan borders; IOS: black borders; blue borders: symmetry activation. Other details as in Figure 2.
Figure 4. Activation maps for the face localizer (vs. Fourier-equated random images) averaged across observers (a) on 3 views of the inflated brain and (b) occipital-pole flatmaps. Yellow borders: V1--V3; magenta borders: LOC; red borders: hMT+; green borders: FFA; cyan borders: OFA; black borders: IOS; blue borders: symmetry activation—Colored symbols indicated locations of activations in individual subjects from other studies (see key). Activation colors as in Figure 2.
Thus, the data for the latter 3 ROIs are consistent with the null hypothesis that the face activation in these areas was insensitive to the symmetry information in the faces, whereas the right OFA showed a significant symmetry component in its response.
Because the difference between the asymmetric and symmet-ric scrambled images lies in their image symmetry, the activation difference between the 2 conditions should reveal the areas processing 2D image symmetry information. Therefore, the OFA activation is apparently influenced by image symmetry. How-ever, the OFA is not one of the areas identified by our symmetry localizer but lies just next to it. The plausible explanation is that the OFA processes face-specific symmetry information (Rhodes and others 2005). The adjacent relation between the OFA and the symmetry areas hints that the OFA may receive input from nearby symmetry area for facial processing.
The IOS activations, on the other hand, are significantly stronger for the symmetry-equated face/scrambled contrast than for the symmetric/asymmetric noise contrast (F5,10 = 17.85, P = 0.0083 < 0.05, for the right hemisphere, F5,10 = 10.39, P=0.032 <0.05 for the left; activation change increased from 0.89% to 1.18%, t (5)=2.86, P=0.017 <0.05 for the right and from 0.46% to 0.62%, t (5)=3.73, P=0.007 <0.05 for the left). Thus, although it is the major locus of activation by symmetry, the IOS is even more strongly activated by the face configuration with symmetry equated. The implication is that this is a cortical region specialized for processing configuration effects, of which symmetry and face arrangements are 2 separate forms.
Face Configuration/Inverted Face Effect
We next compared mean brain activation for the upright versus inverted faces. This is an interesting manipulation because it distinguishes between absolute and relative configuration effects. An absolute configuration effect is one that is specific to the particular orientation and scale of the configuration. For instance, the mouth is located 3° below the eyes with a horizontal position midway between the eyes. This type of configuration is also called the first-order configuration in some face perception literature (Carey and Diamond 1977). A relative configuration effect is one that maintains all the geometric relations among a set of features, regardless of their absolute orientation or scale. For instance, the mouth and the 2 eyes form a triangle regardless whether the faces are upright, tilted, or inverted. This type of configuration should not be confused with the second-order configuration specified in some face perception litera-ture (Carey and Diamond 1977), which refers to the relative distances between facial features. The relative configuration here emphasizes the orientation of the configuration, which is not generally discussed in the second-order configuration literature. In terms of the particular property of symmetry, inverted faces are equated to the upright faces with respect to both the relative and absolute configurations of ‘‘image’’ symmetry. However, the absolute face configuration (or face template) response would be disrupted by the face inversion. Hence, if a brain area such as the OFA processes symmetry in faces in the same way as generic image symmetry, this area should not show differential activation for upright versus inverted face stimuli. On the other hand, if the symmetry is processed as a part of an orientation-specific face gestalt (as would be implied by our constant environmental exposure to upright faces), this area should still show differential activation for these 2 types of faces.
As shown in Figure 3 (row 3), upright versus inverted faces showed robust differential activation in the areas in the fusiform, the inferior occipital, and the middle occipital gyri, and also the areas around IOS and the precuneus. Figure 6 shows the flatmap version of the same activation. The differen-tial activation in the FFA and the OFA was no less than that between faces and asymmetric scrambled images (F5,10 = 0.0277, P = 0.88 > 0.05). These results confirm that the symmetry processing in the OFA found in the previous condition is indeed face specific, or at least orientation specific with respect to face stimuli.
BOLD activation for face inversion has been in reported in many areas throughout the brain (Servos and others 1999; Leube and others 2003; Yovel and Kanwisher 2004, 2005). Reports of the face inversion effect in the literature for the FFA, however, are mixed. Early fMRI studies showed either no (Aguirre and others 1999; Haxby and others 1999) or a weak (Kanwisher and others 1998) face inversion effect in the FFA. More recent fMRI studies have shown a robust face inversion effect in the FFA (Yovel and Kanwisher 2004, 2005). The Yovel and Kanwisher (2005) face inversion effect extended beyond the FFA to an extensive area on the ventral and the lateral occipital surfaces, which is quite similar to ours. Although the cause of this discrepancy in the literature is not resolved, it is possible that the face stimuli used in those studies played a role. The papers reporting no face inversion effect in FFA (Aguirre and others 1999; Haxby and others 1999) used stimuli that had the external contours of the face removed and the one that
Figure 6. Activation maps for upright faces versus inverted faces averaged across observers (a) on 3 views of the inflated brain and (b) occipital-pole flatmaps. FFA: green borders; OFA: cyan borders; IOS: black borders; blue borders: symmetry activation. Other details as in Figure 2.
reported weak face inversion effect (Kanwisher and others 1998) had part of the face covered. On the other hand, studies reporting robust face inversion effects (Yovel and Kanwisher 2004, 2005; and the current study) showed pictures of whole faces as stimuli, including the hair and the neck. It has been suggested that the external contour of a face is also an important part of the face template (Sinha and Poggio 1996; Maurer and others 2002). An incomplete face stimulus may not optimally activate the neurons responsible for the full-face template and may in turn produce insufficient differential ac-tivation between upright face and inverted face.
The precuneus activation has a temporal phase opposite to that of the other significant activations. It has been reported that the precuneus is involved in mental rotation (Bonda and others 1995; Cohen and others 1996; Jordan and others 2001). Hence, one likely explanation for the greater activation in the precu-neus is that the observers were rotating the inverted face information to enhance recognition. To clarify the exact role of the precuneus in inverted face processing would, however, require further study.
Object Structure
The upright frontal-view faces and the 3/4-view faces share the same 3D object structure but not the same image symmetry. Hence, a brain area for image symmetry would show differential activation for these 2 types of images, whereas an area processing symmetry based on the 3D object structure inferred from the image information should respond equally to these 2 types of faces. Perceptually, we understand that the face is still symmetric although it is 3D rotated, so there must be some brain circuitry that carries this perceptual equivalence. This analysis is supported by the finding that sparsely sampled information about the positions of object features relies on an exclusively 3D coding of object properties (Likova and Tyler 2003). The fact that the face retains the same object structure when 3D rotated relative to the plane of projection can be appreciated only by a neural circuit that has encoded the 3D structure of the projected images.
The upright frontal-view faces and the 3/4-view faces showed less differential activation (Fig. 3, row 4, also see Fig. 7 for inflated and flat versions) than the face template areas identified by faces versus inverted faces. The pseudocolor patches in Figure 7(c) identify the regions showing significant differential activation (F5,10 =8.95, P =0.03 < 0.05) between faces and inverted faces but not between faces and 3/4-view faces. This reduction is quite pronounced in the FFA, as more than 61% of voxels showing significant differential activation for faces versus inverted faces (52% for the right and 87% for the left) did not show significant activation in this condition. The activation modulation reduces from 0.49% for faces versus inverted faces to 0.20% for faces versus 3/4-view faces in the left FFA (t (5)=2.43, P=0.029 <0.05). A similar reduction was observed in the right FFA as well (from 0.43% to 0.25%) though the difference is not statistically significant (t (5)=0.79, P=0.23
>0.05, NS).
The activated area in the OFA was also substantially reduced (see Figs 6 vs. 7), although there were some spots still showing activation. The activation modulation was reduced from 0.49% to 0.20% (t (5)=2.43, P=0.0296 <0.05) for the left OFA and 0.47% to 0.26% for the right OFA (t (5)=2.28, P=0.036<0.05) The implication of these results is that much of the coding in
the FFA and OFA is based on the 3D representation of the faces, which ‘‘look the same’’ to these neural circuits despite the radical change in image symmetry. The areas near the IOS and the MOG, however, still showed robust activation to the 3D rotation of the faces as 65% (right) to 90% (left) of IOS voxels showed significant activation in this condition. This strong response implies that they are coding image symmetry rather than object symmetry.
To understand this activation, we may consider the factors that change versus those that are invariant in the image of a 3D-rotated face (Fig. 1a vs. 1c). The invariant factors are low-level ones such as the local edge structure and feature properties, midlevel ones such as the 3D representation of the face structure and high-level ones such as the emotional expression, social position, and personal identity of the individual depicted. Figure 7. Activation maps for frontal-view faces versus 3/4-view faces averaged across observers (a) on 3 views of the inflated brain and (b) occipital-pole flatmaps. (c) Flatmaps showing the regions of differential activation between faces and inverted faces but not between faces and 3/4-view faces (yellow-orange coloration). FFA: green borders; OFA: cyan borders; IOS: black borders; blue borders: symmetry activation. Other details as in Figure 2.
The factors that do change with 3D rotation are the local relations between local feature properties (such as angles between the edges) and 2D configurational relations among the features as a whole, including the symmetry relations that activated these regions in the first experiment. We therefore conclude that the MOG and IOS regions are more involved in 2D configural processing, including symmetry relations, than the FFA and OFA, which are mostly concerned with properties that are invariant under rotation.
Viewpoint Invariance
Our result on viewpoint invariance is consistent with those of Grill-Spector and others (1999), which showed that the fusiform activation in both hemispheres had greater viewpoint invariance than did the dorsal occiput. Vuilleumier and others (2002) used the repetition priming technique to show that left (but not right) fusiform activation had viewpoint-invariant behavior for viewing objects, although size invariance was exhibited bi-laterally. It is interesting to note that both our stimulus sets and those of Grill-Spector and others (1999) contained faces, whereas those of Vuilleumier and others (2002) contained only nonbiological objects. Hence, the discrepancy with the latter results may be due to the sampling of stimuli.
Behavioral studies of face recognition showed a viewpoint-dependent effect. When asked whether an image of a face was from the same person as a previously shown face image, the observers‘ recognition performance depends on the viewpoint of the learned face (Troje and Bulthoff 1996; Hill and others 1997) but not the test face (Troje and Bulthoff 1996). This evidence seems contradictory to our fMRI results, which show little differential activation between 3/4-view and front-view faces in the FFA and OFA. Such viewpoint invariance is com-patible with models suggesting that viewpoint invariance can be achieved from the concatenation of view-based processes (for reviews, see Riesenhuber and Poggio 2000; Rolls 2000). However, it should be noted that an invariance property in a brain area as a whole does not necessary mean that cells in this area all have the same invariance property. Given the spatial resolution of the fMRI, it is likely that each voxel covers a network of view-based units. Hence, brain areas with such units would appear as viewpoint invariant in their fMRI responses, even though neural recognition responses have access to the individual components of such networks. Ventral and Dorsal Face Processing
By this point, the roles of the ventral areas in face configuration processing are clear. The right OFA, located just next to common image symmetry areas, does process face-related image symmetry. Hence, it shows differential activation be-tween faces and asymmetric scrambled images but not bebe-tween faces and symmetric scrambled images.
The symmetry information processed in the right OFA is either orientation specific or a part of face gestalt. After all, the OFA does show differential activation between upright and inverted faces, along with the reduction of differential activation between front view and 3/4 view. The FFA (at least its anterior part) on the other hand, has little interest in image symmetry. Alternation between symmetric- and asymmetric scrambled images produces no detectable effect in this area. In terms of face-specific stimuli, the FFA shows strong differential activa-tion between upright and inverted faces but not between
front-view and 3/4-front-view faces. Hence, the FFA is likely to incorporate an orientation-specific face gestalt (that is, however, viewpoint invariant with respect to lateral rotation).
The roles of the dorsal areas may be understood in terms of relative versus absolute configuration analysis. The symmetry localizer activated part of the areas around the IOS. Those dorsal symmetry areas, at least in the left hemisphere, did show differential activation for the symmetry/asymmetry manipula-tion and for front-view versus 3/4-view faces, and also between upright and inverted faces, which are equally symmetric. Hence, the IOS area may be specialized for processing spatial config-urations, either purely of the symmetry type or of the face-inversion type (with symmetry equated). Because this area responds well to configuration changes whether or not faces are present, whether or not symmetry was manipulated, and whether or not the relative configuration was varied, the implication is that the IOS region is involved in a wide variety of configuration analyses.
The only manipulation that reduced the response in the symmetry activation area was 3D face rotation, which elimi-nated significant response in much of the MOG region. This region has been particularly associated with the processing of 3D structure (Tyler and others 2006), so the lack of response may be understood as a similar response to the 2 forms of 3D structure carried by both the frontal and rotated faces. The strong response to faces versus scrambled textures makes sense in this context as a response to the depth structure in the faces, but the response to symmetric versus asymmetric textures and to face inversion is harder to understand. It may be that there is a stronger perception of depth structure in upright than in inverted faces due to the match to the 3D gestalt of the upright face, and a stronger sense of 3D structure from the luminance cues of the symmetric versus asymmetric noise (especially in the case of noise with the 1/f-type spectrum of the faces). These are suggestions that would require further study to pin down.
Conclusions
We draw 5 main conclusions from the pattern of results to the symmetry manipulations: 1) The complex of cortical areas activated by the face images, relative to Fourier-matched scram-bled images, were the fusiform, inferior, and middle occipital gyri, and areas around the STS and IOS. This is a larger range of dorsal cortical areas than has been previously reported for face-specific activation. 2) The areas responding to symmetry per se were predominantly in the MOG and the IOS. Conversely, the OFA and the FFA did not show significant correlation changes for symmetry in the scrambled images. 3) To identify cortical regions involved in the processes of face recognition, we compared brain activation of the front-view faces and their inverted versions. The upright versus inverted faces showed robust differential activation in the areas in the fusiform, inferior occipital and middle occipital gyri, around the IOS, and in the precuneus. 4) Contrasting the responses to faces and to their vertically symmetric phase-scrambled versions showed a response pattern similar to that produced by contrasting the responses to faces and asymmetric scrambled images (except in the right OFA). This result suggests that the OFA may be involved in processing symmetry specific to face images (in addition to other aspects of the facial configura-tion). 5) Finally, to study the role of object versus image symmetry, we compare the responses to frontal and 3/4-view faces of the same individuals. This contrast showed little differential activation
in the FFA or the OFA but strong responses in areas near the MOG and the IOS, indicating that viewpoint dependence is measurable in the BOLD responses of some cortical areas. Taken together, these results suggest that face processing in the FFA and the OFA is holistic and relatively invariant to the viewpoint from which the face is seen.
Notes
Supported by National Science council (Taiwan) 94-2752-H-002-007-PAE to CCC and National Institutes of Health/National Eye Institute (USA) EY 13025 to CWT. Conflict of Interest: None declared.
Address correspondence to Chien-Chung Chen, Department of Psychology, National Taiwan University, Taipei 106, Taiwan. Email: c3chen@ntu.edu.tw.
References
Aguirre GK, Singh R, D’Esposito M. 1999. Stimulus inversion and the responses of face and object-sensitive cortical areas. Neuroreport 10:189--194.
Altmann CF, Bulthoff HH, Kourtzi Z. 2003. Perceptual organization of local elements into global shapes in the human visual cortex. Curr Biol 13:342--349.
Altmann CF, Deubelius A, Kourtzi Z. 2004. Shape saliency modulates contextual processing in the human lateral occipital complex. J Cogn Neurosci 16:794--804.
Anderson ND, Wilson HR. 2005. The nature of synthetic face adaptation. Vis Res 45:1815--1828.
Bharick HP, Bharick PO, Wittlinger RP. 1975. Fifty years of memory for names and faces: a cross-section approach. J Exp Psychol Gen 104:54--75.
Bonda E, Petrides M, Frey S, Evans A. 1995. Neural correlates of mental transformations of the body-in-space. Proc Natl Acad Sci USA 92:11180--11184.
Boynton GM, Demb JB, Glover GH, Heeger DJ. 1999. Neuronal basis of contrast discrimination. Vis Res 39:257--269.
Brown E, Perrett DI. 1993. What gives a face its gender? Perception 22:829--840.
Carey S, Diamond R. 1977. From piecemeal to configural representation of faces. Science 195:312--314.
Cohen MS, Kosslyn SM, Breiter HC, DiGirolamo GJ, Thompson WL, Anderson AK, Brookheimer SY, Rosen BR, Belliveau JW. 1996. Changes in cortical activity during mental rotation: a mapping study using functional MRI. Brain 119:89--100.
Engel SA, Glover GH, Wandell BA. 1997. Retinotopic organization in human visual cortex and the spatial precision of functional MRI. Cereb Cortex 7:181--192.
Fantz RL. 1961. The origin of form perception. Sci Am 204:66--72. Fellous J. 1997. Gender discrimination and prediction on the basis of
facial metric information. Vis Res 37:1961--1973.
Friston KJ, Holmes AP, Worsley KJ, Poline JP, Frith CD, Frackowiak RSJ. 1995. Statistical parametric maps in functional imaging: a general linear approach. Hum Brain Mapp 2:189--210.
Gauthier I, Tarr MJ, Anderson AW, Skudlarski P, Gore JC. 1999. Activation of the middle fusiform ‘‘face area’’ increases with expertise in recognizing novel objects. Nat Neurosci 2:568--573. Grammer K, Thornhill R. 1994. Human (Homo sapiens) facial
attrac-tiveness and sexual selection the role of symmetry and averageness. J Comp Psychol 108:233--242.
Grill-Spector K, Kushnir T, Edelman S, Avidan G, Itzchak Y, Malach R. 1999. Differential processing of objects under various viewing condi-tions in the human lateral occipital complex. Neuron 24:187--203. Halgren E, Dale AM, Sereno MI, Tootell RBH, Marinkovic K, Rosen B.
1999. Location of human face-selective cortex with respect to retinotopic areas. Hum Brain Mapp 7:29--37.
Haxby JV, Gobbini MI, Furey ML, Ishai A, Schouten JL, Pietrini P. 2001. Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science 293:2425--2430.
Haxby JV, Hoffman, EA, Gobbini MI. 2000. The distributed human neural system for face perception. Trends Cogn Sci 4:223--233.
Haxby JV, Ungerleider LG, Clark VP, Schouten JL, Hoffman EA, Martin A. 1999. The effect of face inversion on activity in human neural systems for face and object perception. Neuron 22:189--199. Hill H, Schyns PG, Akamatsu S. 1997. Information and viewpoint
dependence in face recognition. Cognition 62:201--222.
Hosie JA, Ellis HD, Haig ND. 1988. The effect of feature displacement on the perception of well-known faces. Perception 17:461--474. Jacobsen T, Hofel L. 2002. Aesthetic judgments of novel graphic patterns
analyses of individual judgments. Percept Mot Skills 95:755--766. Jordan K, Heinze HJ, Lutz K, Kanowski M, Jancke L. 2001. Cortical
activations during the mental rotation of different visual objects. Neuroimage 13:143--152.
Joseph JE, Partin DJ, Jones M. 2002. Hypothesis testing for selective, differential, and conjoined brain activation. J Neurosci Methods 118:129--140.
Kanwisher N, McDermott J, Chun M. 1997. The fusiform face area: a module in human extrastriate cortex specialized for face percep-tion. J Neurosci 17:4302--4311.
Kanwisher N, Tong F, Nakayama K. 1998. The effect of face inversion on the human fusiform face area. Cognition 68:B1--B11.
Kourtzi Z, Kanwisher N. 2001. Representation of perceived object shape by the human lateral occipital complex. Science 293:1506--1509. Kowner R. 1996. Facial asymmetry and attractiveness judgment in
developmental perspective. J Exp Psychol Hum Percept Perform 22:662--675.
Leder H, Bruce V. 2000. When inverted faces are recognized: the role of configuration information in face recognition. Q J Exp Psychol 53A:513--536.
Leube DT, Yoon HW, Rapp A, Erb M, Grodd W, Bartels M, Kircher TT. 2003. Brain regions sensitive to the face inversion effect: a functional magnetic resonance imaging study in humans. Neurosci Lett 342:143--146.
Likova LT, Tyler CW. 2003. Peak localization of sparsely sampled luminance patterns is based on interpolated 3D surface representa-tion. Vis Res 43:2649--2957.
Maurer S, Le Grand R, Mondloch, CJ. 2002. The many faces of configural processing. Trends Cogn Sci 6:255--260.
McCarthy G, Puce A, Gore JC, Allison T. 1997. Face-specific processing in the human fusiform gyrus. J Cogn Neurosci 9:605--610.
Mealey L, Bridgstock R, Townsend GC. 1999. Symmetry and perceived facial attractiveness a monozygotic co-twin comparison. J Pers Soc Psychol 76:151--158.
Murray SO, Kersten D, Olshausen BA, Schrater P, Woods DL. 2002. Shape perception reduces activity in human primary visual cortex. Proc Natl Acad Sci USA 99:15164--15169.
Murray SO, Schrater P, Kersten D. 2004. Perceptual grouping and the interactions between visual cortical areas. Neural Netw 17:695--705. O’Toole AJ, Deffenbacher KA, Valentin D, Abdi H. 1994. Structural aspects of face recognition and the other-race effect. Mem Cogn 22:208--224.
Perrett DI, Rolls ET, Cann W. 1982. Visual neurons responsive to faces in the monkey temporal cortex. Exp Brain Res 47:329--342.
Phillips P J, Moon H, Rauss PJ, Rizvi S. 2000. The FERET evaluation methodology for face recognition algorithms. IEEE Trans Pattern Anal Mach Intell 22:1090--1104.
Puce A, Allison T, Bentin S, Gore JC, McCarthy G. 1998. Temporal cortex activation of human viewing eye and mouth movement. J Neurosci 18:2188--2199.
Rhodes G, Geddes K, Jeffery L, Dziurawiec S, Clark A. 2002. Are average and symmetric faces attractive to infants? Discrimination and looking preferences. Perception 31:315--321.
Rhodes G, Peters M, Lee K, Morrone MC, Burr D. 2005. Higher-level mechanisms detect facial symmetry. Proc Biol Sci 272:1379--1384. Rhodes G, Yoshikawa S, Clark A, Lee K, McKay R, Akamatsu S. 2001.
Attractiveness of facial averageness and symmetry in non-western cultures in search of biologically based standards of beauty. Perception 30:611--625.
Riesenhuber M, Poggio T. 2000. Models of object recognition. Nat Neurosci Suppl 3:1199--1204.
Rolls ET. 2000. Functions of the primate temporal lobe cortical visual areas in invariant visual object and face recognition. Neuron 27:205--218.
Rossion B, Caldara R, Seghier M, Schuller AM, Lazeyras F, Mayer E. 2003. A network of occipito-temporal face-sensitive areas besides the right middle fusiform gyrus is necessary for normal face processing. Brain 126:2381--2395.
Sasaki Y, Vanduffel W, Knutsen T, Tyler CW, Tootell RH. 2005. Symmetry activates extrastriate visual cortex in human and nonhuman pri-mates. Proc Natl Acad Sci USA 102:3159--3163.
Servos P, Engel SA, Gati J, Menon R. 1999. fMRI evidence for an inverted face representation in human somatosensory cortex. Neuroreport 10:1393--1395.
Sinha P, Poggio T. 1996. I think I know that face. Nature 384:404. Stehling MK, Turner R, Mansfield P. 1991. Echo-planar imaging:
mag-netic resonance imaging in a fraction of a second. Science 254: 43--50.
Swaddle JP, Cuthill IC. 1995. Asymmetry and human facial attractiveness symmetry may not always be beautiful. Proc R Soc Lond B Biol Sci 261:111--116.
Talairach J, Tournoux P. 1988. Co-planar stereotaxic atlas of the human brain. New York: Thieme Medical Publishers.
Thompson, P. 1980. Margaret Thatcher: a new illusion. Perception 9:483--484. Tootell RB, Hadjikhani NK, Vanduffel W, Liu AK, Mendola JD, Sereno MI, Dale AM. 1998. Functional analysis of primary visual cortex (V1) in humans. Proc Natl Acad Sci USA 95:811--817.
Troje NF, Bulthoff HH. 1996. Face recognition under varying poses: the role of texture and shape. Vis Res 36:1761--1771.
Tyler CW, Baseler HA, Kontsevich LL. 2001. A global symmetry model predicts both fMRI time-series and psychophysical responses to symmetry patterns. Soc Neurosci Abs 11.11.
Tyler CW, Baseler HA, Kontsevich LL, Likova LT, Wade AR, Wandell BA. 2005a. Predominantly extra-retinotopic cortical response to pattern symmetry. Neuroimage 24:306--314.
Tyler CW, Chen CC. Spatial summation of face information. J Vis. Forthcoming.
Tyler CW, Likova LT, Chen CC, Kontsevich LL, Schira MM, Wade AR. 2005b. Extended concepts of occipital retinotopy. Curr Med Imaging Rev 1:319--329.
Tyler CW, Likova LT, Kontsevich LL, Wade AR. 2006. The specificity of cortical area KO to depth structure. Neuroimage 30:228--238. Valentine T. 1988. Upside-down faces: a review of the effect of inversion
upon face recognition. Br J Psychol 79:471--491.
Vuilleumier P, Henson RN, Driver J, Dolan RJ. 2002. Multiple levels of visual object constancy revealed by event-related fMRI of repetition priming. Nat Neurosci 5:491--499.
Wandell BA, Chial S, Backus BT. 2000. Visualization and measurement of the cortical surface. J Cogn Neurosci 12:739--752.
Webster MA, Kaping D, Mizokami Y, Duhamel P. 2004. Adaptation to natural facial categories. Nature 428:557--561.
Yin RK. 1969. Looking at upside-down faces. J Exp Psychol 81:141--145. Yovel G, Kanwisher N. 2004. Face perception: domain specific, not
process specific. Neuron 44:889--898.
Yovel G, Kanwisher N. 2005. The neural basis of the behavioral face-inversion effect. Curr Biol 15:2256--2262.
Zaidel DW, Chen AC, German C. 1995. She is not a beauty even when she smiles: possible evolutionary basis for a relationship between facial attractiveness and hemispheric specialization. Neuropsychologia 33:649--655.
