Learning Gender with Support Faces

Learning Gender with Support Faces

Baback Moghaddam, Member, IEEE, and Ming-Hsuan Yang, Member, IEEE

AbstractÐNonlinear Support Vector Machines (SVMs) are investigated for appearance-based gender classification with low-resolution ªthumbnailº faces processed from 1,755 images from the FERET face database. The performance of SVMs (3.4 percent error) is shown to be superior to traditional pattern classifiers (linear, quadratic, Fisher linear discriminant, nearest-neighbor) as well as more modern techniques such as Radial Basis Function (RBF) classifiers and large ensemble-RBF networks. Furthermore, the difference in classification performance with low-resolution ªthumbnailsº (21-by-12 pixels) and the corresponding higher resolution images (84-by-48 pixels) was found to be only 1 percent, thus demonstrating robustness and stability with respect to scale and degree offacial detail.

Index TermsÐSupport vector machines, gender classification, linear, quadratic, Fisher linear discriminant, RBF classifiers, face recognition.

æ 1 I

AS Human-Computer Interaction technology (HCI) evolves, computer vision systems for people monitoring will play an increasingly important role in our lives. Examples include human (face) detection, face/body tracking, action (gesture) recognition, person identification (face recognition) and estimation of age, ethnicity and perhaps most fundamentally gender. This informa- tion will not only enhance existing HCI systems but can also serve as a basis for passive surveillance andcontrol in ºsmart buildingsº (e.g., restricting access to certain areas basedon gender) and collecting valuable demographics (e.g., the number of women entering a retail store on a given day). We have developed an appearance-basedgender classifier for low-resolution images (extractedby an automatic face detection system) which uses Support Vector Machine (SVM) learning. This system exhibits performance far superior to existing classifiers.

In recent years, SVMs have been successfully appliedto various tasks in computational face-processing, including face detection [22], face pose discrimination [19], and face recognition [24]. The goodempirical results can be explainedby the fact that SVM is an optimal discriminant based on large margin learning theory. For the cases where it is difficult to estimate the density model in high- dimensional space, e.g., images, the discriminant approach is preferable to the generative approach. Furthermore, SVMs provide an efficient discriminant method, not only to handle the patterns that are not linearly separable, but to also achieve lower general- ization error for unseen test examples. In this paper, we develop an appearance-basedmethodto classify gender from facial images using nonlinear SVMs andcompare their performance with traditional classifiers (e.g., linear, quadratic, Fisher linear discri- minant, andnearest-neighbor) as well as more modern techniques such as RBF networks andlarge ensemble-RBF classifiers. We have focusedour study on very low-resolution ªthumbnailº images in which only the main frontal facial regions (inside the ªovalº of the

face) are visible andalmost completely excludedhair information (outside the ªovalº). The motivation for using these particular images is two-fold. First, hair styles can change in appearance easily andfrequently. Therefore, in a robust face recognition system face images are usually croppedto keep only the main facial regions. Second, we wished to investigate the minimal amount of face information (resolution) requiredto learn male and female faces by various classifiers. Previous studies on gender classification have usedhigh-resolution images with hair informa- tion andrelatively small data sets for their experiments. In our study, we demonstrate that SVM classifiers are able to learn and classify gender from a large set of hairless low-resolution images with very high accuracy. Furthermore, SVM classifiers showed negligible difference between their error rates with low and high- resolution facial images. In our experimental study, little or no hair information was usedas input to the classifiers. This is in contrast to previous results reportedin the literature where almost all methods include some hair information.

2 B

Although gender classification has attracted much attention in the psychological literature [3], [6], [23], relatively few learning-based machine vision methods have been proposed. In this section we briefly review andsummarize the prior art in visual gender classification. The studies referred to are also summarized in Fig. 1 where the final entry [21] reports some of the preliminary results reportedin this paper.

Gollomb et al. [16] traineda fully connectedtwo-layer neural network, SEXNET, to identify gender from 30-by-30 face images.

Their experiments on a set of 90 photos (45 males and45 females) gave an average error rate of 8.1 percent comparedto an average error rate of 11.6 percent from a study of five human subjects.

Cottrell [9] also appliedneural networks for emotion andgender classification. The dimensionality of a set of 160 64-by-64 face images (10 males and10 females) was reducedfrom 4,096 to 40 with an auto-encoder. These vectors were then presented as inputs to another one-layer network for training. They reportedperfect classification (albeit for only 20 individuals). Brunelli and Poggio [4] developed HyperBF networks for gender classification in which two competing RBF networks, one for male andthe other for female, were trainedusing 16 geometric features as inputs (e.g., pupil to eyebrow separation, eyebrow thickness, andnose width).

The results on a data set of 168 images (21 males and 21 females) show an average error rate of 21 percent. Using similar techniques as Golomb et al. [16] andCottrell [9], Tamura et al. [29] used multilayer neural networks to classify gender from face images at multiple resolutions (from 32-by-32 to 8-by-8 pixels). Their experiments on 30 test images show that their network was able to determine gender from 8-by-8 images with an average error rate of 7 percent. Insteadof using a vector of gray levels to represent faces, Wiskott et al. [31] usedlabeledgraphs of two-dimensional views to describe faces. The nodes were represented by wavelet- based local ªjetsº and the edges were labeled with distance vectors.

They useda small set of controlledmodel graphs of males and females to encode ªgeneral face knowledge,º in order to generate graphs of new faces by elastic graph matching. For each new face, a composite reconstruction was generatedusing the nodes in the model graphs. The gender of the majority of nodes used in the composite graph was usedfor classification. The error rate of their experiments on a gallery of 112 face images was 9.8 percent.

Recently, Gutta et al. [17] proposeda hybridclassifier basedon neural networks (RBFs) andinductive decision trees with Quinlan's C4.5 algorithm with 3,000 FERET faces of size 64-by- 72 pixels. The best average error rate was foundto be 4 percent.

. B. Moghaddam is with Mitsubishi Electric Research Laboratories, 201 Broadway, Cambridge, MA 02139. E-mail: [email protected].

. M.-H. Yang is with Honda Fundamental Research Laboratories, 800 California St., Mountain View, CA 94041. E-mail: [email protected].

Manuscript received 27 Sept. 2000; revised 18 June 2001; accepted 21 Oct.

2001.

Recommended for acceptance by R. Sharma.

For information on obtaining reprints of this article, please send e-mail to:

[email protected], and reference IEEECS Log Number 112913.

0162-8828/02/$17.00 ß 2002 IEEE

3 G

C

A generic appearance-basedgender classifier is shown in Fig. 2.

An input facial image x generates a scalar output f…x† whose polarityÐsign of f…x†Ðdetermines class membership. The magni- tude kf…x†k can usually be interpretedas a measure of belief or certainty in the decision made. Nearly all binary classifiers can be viewedin these terms; for density-basedclassifiers (linear, quadratic and Fisher) the output function f…x† is a log-likelihood ratio, whereas for kernel-basedclassifiers (nearest-neighbor, RBFs, and SVMs) the output is a ªpotential fieldº related to the distance from the separating boundary.

3.1 Support Vector Machines

A Support Vector Machine is a learning algorithm for pattern classification, regression anddensity estimation [30], [8], [11]. The basic training principle behindSVMs is finding the optimal linear hyperplane such that the expectedclassification error for unseen test samples is minimizedÐi.e., good generalization performance.

According to the structural risk minimization inductive principle [30], a function that classifies the training data accurately and which belongs to a set of functions with the lowest VC dimension [8] will generalize best, regardless of the dimensionality of the input space. Basedon this principle, a linear SVM uses a systematic approach to finda linear function with the lowest capacity. For linearly nonseparable data, SVMs can (nonlinearly) map the input to a high-dimensional feature space where a linear hyperplane can be found. Although there is no guarantee that a linear solution will always exist in the high-dimensional space, in practice it is feasible to finda working solution.

Given a labeledset of M training samples …xi; yi†, where xi2 R^N and yi is the associatedlabel …yi2 f 1; 1g†, a SVM classifier finds the optimal hyperplane that correctly separates (classifies) the largest fraction of data points while maximizing the distance of either class from the hyperplane (the margin). Vapnik [30] shows that maximizing the margin is consistant with to minimizing the VC dimension in constructing the optimal hyperplane. Computing the optimal hyperplane is posedas a constrainedoptimization problem andsolvedusing quadratic programming techniques. The discriminant hyperplane is defined by:

f…x† ˆX^M

iˆ1yii
k…x; xi† ‡ b;

where k…
;
† is a kernel function, b is a bias term andthe sign of f…x† determines the class membership of x. Constructing an optimal hyperplane is equivalent to finding all the nonzero iand is formulatedas a quadratic programing (QP) problem with linear constraints [5]. Any vector xithat corresponds to a nonzero iis a support vector (SV) of the optimal hyperplane. A desirable feature of SVMs is that the number of training points which are retainedas

support vectors is usually quite small, thus providing a compact classifier. Solving the constraint optimization problem for a SVM with a large data set is a nontrivial task, many methods have been proposedto tackle such problems. In our study, we useda public- domain SVM package which uses conjugate gradients for the QP optimization [26]. For more recent advances in fast optimization methods for SVMs, see [27].

For a linear SVM, the kernel function is just a simple dot product in the input space while the kernel function in a nonlinear SVM effectively projects the samples to a feature space of higher (possibly infinite) dimension via a nonlinear mapping function:

 : R^N! F^M; M  N

andthen constructs a hyperplane in F. The motivation behindthis mapping is that it is more likely to finda linear hyperplane in the high-dimensional feature space. Using Mercer's theorem [10], the expensive calculations requiredin projecting samples into the high-dimensional feature space can be replaced by a simpler kernel function satisfying the condition

k…x; xi† ˆ …x†
…xi†;

where  is the nonlinear projection function. Several kernel functions, such as, polynomials andradial basis functions, have been shown to satisfy Mercer's theorem andhave been used successfully in nonlinear SVMs:

k…x; xi† ˆ ……x
xi† ‡ 1†^d k…x; xi† ˆ exp… jx xij²†;

where d is the degree of freedom in a polynomial kernel and is the spreadof a Gaussian cluster. In fact, by using different kernel functions, SVMs can implement a variety of learning machines, some of which coincide with classical classifiers, e.g., Bayesian classifier, radial basis function networks, maximum entropy approaches. Nevertheless, automatic selection of the ªrightº kernel function andits associatedparameters remains problematic andin practice one must resort to trial anderror with validation set for model selection. However, see [7] for a recently proposed method for multiple parameter selection for SVMs.

3.2 Radial Basis Function Networks

A radial basis function (RBF) network is also a kernel-based technique for improvedgeneralization, but it is basedinsteadon regularization theory [25], [18]. A typical RBF network with K Gaussian basis functions is given by

f…x† ˆX^K

i wiG…x; ci; ²_i† ‡ b;

where the G is the ith Gaussian basis function with center ciand variance ²_iand b is a bias term. The weight coefficients wicombine the basis functions into a single scalar output value, with b as a bias term. Training a Gaussian RBF network for a given learning task Fig. 1. Comparison ofrepresentative gender classification studies (see text).

Fig. 2. Gender classifier.

involves determining the total number of Gaussian basis functions, locating their centers, computing their corresponding variances, andsolving for the weight coefficients andbias using the regularization theory [25]. Judicious choice of K, ci, and ²_i, can yieldRBF networks which are quite powerful in classification and regression tasks. The number of radial bases in a conventional RBF network is predetermined before training, whereas the number for a large ensemble-RBF network is iteratively increaseduntil the error falls below a set threshold. The RBF centers in both cases are usually determined by k-means clustering. In contrast, a SVM with the same RBF kernel will automatically determine the number and location of the centers, as well as the weights andthresholdthat minimize an upper boundon the expectedrisk. Recently, Evgeniou et al. [14] have shown that both SVMs andRBF networks can be formulatedunder a unifiedframework in the context of Vapnik's theory of statistical learning [30]. As such, SVMs provide a more systematic approach to classification than classical RBF and various other neural networks.

3.3 Classical Discriminant Methods

Fisher linear discriminant (FLD) is an example of a class specific subspace methodthat finds the optimal linear projection for classification [15], [12], [2]. Rather than finding a projection that maximizes the projectedvariance as in principal component analysis, FLD determines a projection that maximizes the ratio between the between-class scatter andthe within-class scatter.

Consequently, classification is simplifiedin the projectedspace. In our experiments, we useda single Gaussian to model the distribution of male or female class in the resulting one dimensional space. The class membership of a sample was then determined using the maximum a posteriori probability, or equivalently by a likelihoodratio test. See [1], [13], [28] for face recognition methods using FLD.

The decision boundary of a quadratic classifier is defined by a quadratic form in x, derived through Bayesian error minimization [15], [2], [12]. Assuming that the distribution of each class is

Gaussian, the classifier output is given by finding the minimum Mahalanobis distance to a cluster center. A linear classifier is a special case of the quadratic form, based on the assumption all the clusters have the same covariance matrix. For both classifiers, the sign of the discriminant function determines class membership andis also equivalent to a likelihoodratio test.

4 E

In our study, 256-by-384 pixel FERET ªmug-shotsº were pre- processedusing an automatic face-processing system which compensates for translation, scale as well as slight rotations.

Shown in Fig. 3a, this system is described in detail in [20] and uses maximum-likelihoodestimation for face detection, affine warping for geometric shape alignment andcontrast normal- ization for ambient lighting variations. The resulting output ªface- printsº in Fig. 3a were standardized to 80-by-40 (full) resolution.

These ªface-printsº were further subsampledto 21-by-12 pixel ªthumbnailsº for our low-resolution experiments. Fig. 3b shows a few examples of processedface-prints (note that these faces contain little or no hair information). A total of 1,755 thumbnails (1,044 males and711 females) were usedin our experiments. For each classifier, the average error rate was estimatedwith five-fold cross validation (CV)Ði.e., a five-way data set split, with 4/5th usedfor training and1/5th usedfor testing, with four subsequent nonoverlapping rotations. The average size of the training set was 1,496 (793 males and713 females) andthe average size of the test set was 259 (133 males and126 females).

The SVM classifier was first testedwith various kernels in order to explore the space of possibilities andperformance. In all the experiments, we set the soft margin C value to infinity so that no training error is allowed[30]. Meanwhile, each training andtesting vector was scaledto be between -1 and1, andeach optimization problem was solvedby the conjugate gradient methodwith a decomposition method similar to [22]. Fig. 4 shows the empirical results with various kernels andparameters (basedon one training Fig. 3. (a) Face alignment system. (b) Some aligned faces.

Fig. 4. Empirical results with thumbnails using various kernels. (a) Polynomial kernel. (b) RBF kernel.

set). The mediocre results achieved by the first order polynomial kernel indicated that the linear decision surface is not able to effectively classify all the data points, which also indicated the ªhardnessº of this data set (also, see the results of linear and Fisher linear discriminant classifiers in Table 1). On the other hand, nonlinear decision surfaces constructed by second, third and fourth order polynomial kernel achievedgoodresults. Meanwhile, the variance of the overall error rate among these nonlinear SVMs was not significant. For RBF kernels, we foundthat the variance of the overall error rate was not significant when we chose reasonable values.

A Gaussian RBF kernel was foundto perform the best (in terms of error rate), followedby a cubic polynomial kernel as second best. In the large ensemble-RBF experiment, the number of radial bases was incrementeduntil the error fell below a set threshold.

The average number of radial bases in the large ensemble-RBF was foundto be 1,289 which corresponds to 86 percent of the training set. The number of radial bases for classical RBF networks was heuristically set to 20 prior to actual training andtesting. The quadratic, linear, and Fisher classifiers were all implemented using single Gaussian distributions. In each case, a likelihood ratio test was usedfor classification. The average error rates of all the classifiers testedwith 21-by-12 pixel thumbnails are reportedin Table 1 andsummarizedin Fig. 5.

The SVMs out-performedall other classifiers, although the performance of large ensemble-RBF networks was close to SVMs. However, nearly 90 percent of the training set was retainedas radial bases by the large ensemble-RBF. In contrast, the number of support vectors foundby both SVMs was only about 20 percent of the training set. We also appliedSVMs to classification basedon high-resolution images (84-by-48 pixels).

The Gaussian andcubic kernel SVMs performedequally well at

both low- andhigh-resolutions with only a slight 1 percent error rate difference. Fig. 6 shows three pairs of opposite (male andfemale) support faces from an actual SVM classifier. This figure is, of course, a crude low-dimensional depiction of the optimal separating hyperplane (hypersurface) andits associated margins (shown as dashed lines). However, the support faces shown are positionedin accordance with their basic geometry.

Each pair of support faces across the boundary was the closest pair of images in the projectedhigh-dimensional space. It is interesting to note not only the visual similarity of a given pair but also their androgynous appearance. Naturally, this is to be expectedfrom a face locatednear the boundary of the male andfemale domains. We also note that as seen in Table 1, all the classifiers hadhigher error rates in classifying females. This phenomenon is most likely due to the general lack of prominent anddistinct facial features in female faces.

5 D

We have presenteda comprehensive evaluation of various classification methods for determination of gender from facial images. The nontriviality of this task (made even harder by our ªhairlessº low-resolution faces) is demonstrated by the fact that a linear classifier hadan error rate of 60 percent (i.e., worse than a random coin flip). Furthermore, an acceptable error rate (< 5 percent) for the large ensemble-RBF network required storage of 86 percent of the training set (SVMs requiredabout 20 percent). Storage of the entire data set in the form of the nearest-neighbor classifier yielded too high an error rate (30 percent). Clearly, SVMs succeeded in the difficult task of finding a near-optimal gender partition in face space with the added economy of a small number of support faces.

Given the relative success of previous studies with low- resolution faces it is re-assuring that 21-by-12 faces can in fact be usedfor reliable gender classification. Unfortunately, most of the previous studies used data sets of relatively few faces. The most directly comparable study to ours is that of Gutta et al. [17], which TABLE 1

Experimental Results with Thumbnails

Fig. 5. Error rates ofvarious classifiers.

Fig. 6. Support faces at the boundary.

also usedFERET faces. With a data set of 3,000 faces at a resolution of 64-by-72, their hybridRBF/Decision-Tree classifier achieveda 4 percent error rate. In our study, with 1,800 faces at a resolution of 21-by-12, a Gaussian kernel SVM was able to achieve a 3.4 percent error rate. Both studies use extensive cross validation to estimate the error rates. Given our results with SVMs, it is clear that better performance at even lower resolutions is made possible with this learning technique.

R

[1] V. Belhumeur, J. Hespanha, andD. Kriegman, ªEigenfaces vs. Fisherfaces:

Recognition Using Class Specific Linear Projection,º IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 19, no. 7, pp. 711-720, July 1997.

[2] C.M. Bishop, Neural Networks for Pattern Recognition. OxfordUniv. Press, 1995.

[3] V. Bruce, A.M. Burton, N. Dench, E. Hanna, P. Healey, O. Mason, A.

Coombes, R. Fright, andA. Linney, ªSex Discrimination: How do We Tell the Difference between Male andFemale Faces?º Perception, vol. 22, pp. 131- 152, 1993.

[4] R. Brunelli andT. Poggio, ªHyperbf Networks for Gender Classification,º Proc. DARPA Image Understanding Workshop, pp. 311-314, 1992.

[5] C.J.C. Burges, ªA Tutorial on Support Vector Machines for Pattern Recognition,º Data Mining and Knowledge Discovery, vol. 2, no. 2, 1998.

[6] A.M. Burton, V. Bruce, andN. Dench, ªWhat's the Difference Between Men andWomen? Evidence from Facial Measurement,º Perception, vol. 22, pp. 153-176, 1993.

[7] O. Chapelle, V. Vapnik, O. Bousquet, andS. Mukherjee, ªChoosing Multiple Parameters for Support Vector Machines,º Machine Learning, vol.

46, nos. 1-3, pp. 131-159, 2002.

[8] C. Cortes andV. Vapnik, ªSupport Vector Networks,º Machine Learning, vol. 20, 1995.

[9] G.W. Cottrell, ªEmpath: Face, Emotion, andGender Recognition Using Holons,º Advances in Neural Information Processing Systems 3, R.P. Lippmann, J.E. Moody, and D.S. Touretzky, eds. pp. 564-571, 1991.

[10] R. Courant andD. Hilbert, Methods of Mathematical Physics. vol. 1, New- York: Interscience, 1953.

[11] N. Cristianini andJ. Shawe-Taylor, An Introduction to Support Vector Machines. Cambridge Univ. Press, 2000.

[12] R.O. Duda, P.E. Hart, and D.G. Stork, Pattern Classification. Wiley Interscience, 2001.

[13] K. EtemadandR. Chellappa, ªDiscriminant Analysis for Recognition of Human Face Images,º J. Optical Soc. of Am., vol. 14, no. 8, pp. 1724-1733, 1997.

[14] T. Evgeniou, M. Pontil, andT. Poggio, ªRegularization Networks and Support Vector Machines,º Advances in Computational Math., vol. 13, no. 1, pp. 1-50, 2000.

[15] K. Fukunaga, Introduction to Statistical Pattern Recognition. Academic Press, 1990.

[16] B.A. Golomb, D.T. Lawrence, andT.J. Sejnowski, ªSexnet: A Neural Network Identifies Sex from Human Faces,º Advances in Neural Information Processing Systems 3, R.P. Lippmann, J.E. Moody, and D.S. Touretzky, eds.

pp. 572-577, 1991.

[17] S. Gutta, H. Wechsler, andP.J. Phillips, ªGender andEthnic Classification,º Proc. Third IEEE Int'l Automatic Face and Gesture Recognition, pp. 194-199, 1998.

[18] S. Haykin, Neural Networks : A Comprehensive Foundation. Prentice Hall, 1998.

[19] J. Huang, X. Shao, andH. Wechsler, ªFace Pose Discrimination Using Support Vector Machines,º Proc. 14th Int'l Conf. Pattern Recognition, pp. 154- 156, 1998.

[20] B. Moghaddam and A. Pentland, ªProbabilistic Visual Learning for Object Representation,º IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 19, no. 7, pp. 696-710, July 1997.

[21] B. Moghaddam and M.-H. Yang, ªGender Classification with Support Vector Machines,º Proc. Fourth Int'l Conf. Automatic Face and Gesture Recognition, pp. 306-311, 2000.

[22] E. Osuna, R. Freund, and F. Girosi, ªTraining Support Vector Machines: An Application to Face Detection,º Proc. IEEE Computer Soc. Conf. Computer Vision and Pattern Recognition, pp. 130-136, 1997.

[23] A.J. O'Toole, T. Vetter, N.F. Troje, andH.H. Bulthoff, ªSex Classification is Better with Three-Dimensional Structure than with Image Intensity Information,º Perception, vol. 26, pp. 75-84, 1997.

[24] P.J. Phillips, ªSupport Vector Machines Appliedto Face Recognition,º Advances in Neural Information Processing Systems 11, M.S. Kearns, S. Solla, andD. Cohen, eds., MIT Press, vol. 11, pp. 803-809, 1998.

[25] T. Poggio andF. Girosi, ªNetworks for Approximation andLearning,º Proc.

IEEE, vol. 78, no. 9, vol. 1481-1497, 1990.

[26] C. Saunders, M.O. Stitson, J. Weston, L. Bottou, B. Schoelkopf, and A.

Smola, ªSupport Vector Machine Refernce Manual,º Technical Report CSD- TR-98-03, Dept. Computer Science, Royal Holloway, Univ. of London, 1998.

[27] Advances in Kernel Methods. B. Schlkopf, C.J.C. Burges, andA.J. Smola, eds.

MIT Press, 1998.

[28] D. Swets andJ. Weng, ªUsing Discriminant Eigenfeatures for Image Retrieval,º IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 8, pp. 831-836, Aug. 1996.

[29] S. Tamura, H. Kawai, andH. Mitsumoto, ªMale/Female Identification from 8  6 Very Low Resolution Face Images by Neural Network,º Pattern Recognition, vol. 29, no. 2, pp. 331-335, 1996.

[30] V. Vapnik, The Nature of Statistical Learning Theory. Springer, 1995.

[31] L. Wiskott, J.-M. Fellous, N. KruÈger, andC. von der Malsburg, ªFace Recognition andGender Determination,º Proc. First Int'l Workshop Face and Gesture Recognition, pp. 92-97, 1995.

. For more information on this or any other computing topic, please visit our Digital Library at http://computer.org/publiciations/dilb.