Accuracy enhanced thermal face recognition

Accuracy enhanced thermal face recognition

Chun-Fu Lin

, Sheng-Fuu Lin

a_{Institute of Electrical Control Engineering, National Chiao Tung University, Taiwan, ROC} b

Instrument Technology Research Center, National Applied Research Laboratories, Taiwan, ROC

h i g h l i g h t s

The recognizer employs both critical thermal features and geometric features.

The critical geometric features would not be influenced by hair style.

The topography of blood vessels is unique for every human.

The recognizer uses direct information of the topography of blood vessels.

Performance of the recognizer is invariable when the hair of frontal bone varies.

a r t i c l e

i n f o

Article history:

Received 28 January 2013 Available online 7 September 2013

Keywords: Face recognition Thermal face recognizer Recognition performance

a b s t r a c t

Human face recognition has been generally researched for the last three decades. Face recognition with thermal image has begun to attract significant attention gradually since illumination of environment would not affect the recognition performance. However, the recognition performance of traditional ther-mal face recognizer is still insufficient in practical application. This study presents a novel therther-mal face recognizer employing not only thermal features but also critical facial geometric features which would not be influenced by hair style to improve the recognition performance. A three-layer back-propagation feed-forward neural network is applied as the classifier. Traditional thermal face recognizers only use the indirect information of the topography of blood vessels like thermogram as features. To overcome this limitation, the proposed thermal face recognizer can use not only the indirect information but also the direct information of the topography of blood vessels which is unique for every human. Moreover, the recognition performance of the proposed thermal features would not decrease even if the hair of frontal bone varies, the eye blinks or the nose breathes. Experimental results show that the proposed features are significantly more effective than traditional thermal features and the recognition performance of thermal face recognizer is improved.

Ó 2013 Elsevier B.V. All rights reserved.

1. Introduction

Personal recognition with computer is important since the need to recognize persons automatically in various areas, such as sur-veillance, attendance identification, human computer interface, airport security checks, and immigration checks[1,2]. The tradi-tional recognition methods, such as secret code and ID card are no longer able to satisfy the need. Biometric recognition has be-come an important recognition method [3,4]. Biometric systems work in a similar procedure[5–8]. In the beginning, every user of

the system is registered into a database with a specified method. A certain characteristic of every user is capture. When a person needs to be identified, the biometric system will compare his/her characteristic with all characteristics stored in the database to find out the possible match. Biometrics use physical characteristic or personal trait to identify the person. Physical characteristics are generally obtained from living human body. The most common physical features used are facial features, eye features (iris and retina), hand geometry, fingerprints and so on [9,10]. Personal traits are more appropriate for applications which need interac-tion. It is more convenient but less secure. The commonly used personal traits are signature and voices [11,12]. Among these biometric systems, face recognition has become a significant research topic for no physical interaction is required in its operation[13,14].

Most of the face recognition systems used visible image due to the availability of low cost visible band optical cameras. But these

1350-4495/$ - see front matter Ó 2013 Elsevier B.V. All rights reserved.

http://dx.doi.org/10.1016/j.infrared.2013.08.011

⇑Corresponding author. Address: 1001, University Road, Hsinchu 30010, Taiwan, ROC. Tel.: +886 3 5712121x54365; fax: +886 3 5715998.

E-mail addresses: [email protected] (C.-F. Lin), sfl[email protected]

(S.-F. Lin).

1 _{Address: 20, R&D Rd.VI, Hsinchu Science Park, Hsinchu 30076, Taiwan, ROC.}

Tel.: +886 3 5779911x554; fax: +886 3 5773947.

Contents lists available atScienceDirect

Infrared Physics & Technology

high and low frequency signals in thermal face image as features, for instance, LBP (Local Binary Pattern) and eigenface which may be preprocessed by DCT (Discrete Cosine Transform) or wavelet or polar[22,23]. Following feature extraction, the extracted fea-tures are sent to a classifier for recognition. Neural network is an efficient face recognition classifier and the back-propagation feed-forward learning algorithm is one of the most useful and pop-ular method of neural networks[24].

However, the practical recognition performance is still insuffi-cient. There are three major reasons which are described below.

1. Traditional thermal face recognizers cannot employ facial geo-metric features which have been used generally and success-fully in face recognition of visible face images since thermal face images have lower resolution and higher noise than visible face images.

2. The recognition performance of traditional thermal face recog-nizers decrease easily when the hair of frontal bone vary, the eye blink or the nose breathe.

3. Traditional thermal face recognizers only use the indirect infor-mation of the topography of blood vessels like thermogram as features rather than direct information of the topography of blood vessels.

In this paper, a novel thermal face recognizer which can overcome the three major problems described above is proposed to significantly improve the recognition accuracy. The image pre-processing techniques and extracted features of the recognition scheme are described. A three-layer back-propagation feed-for-ward neural network is applied as the classifier[24].

The rest of this paper is organized as follows. The recognition algorithm is introduced in Section2. Experimental results and dis-cussion are presented in Section3. Section4draws the conclusion. 2. Recognition algorithm

In this section, procedure of the proposed recognition system is presented and divided into three parts. The three parts are the im-age preprocessing that segments the face area out of the back-ground, the features extraction that extracts features from a segmented image, and the classification that uses features as inputs to neural network classifier. The image preprocessing part employs Otsu’s method to find the optimal binarization threshold for seg-menting the background, and then precisely indicates the location of nose tip, eye, and mouth of thermal face image[25]. The features extraction part extracts six kinds of features: (1) the angle of the bridge of the nose, (2) the area of the bridge of the nose, (3) the perimeter of cheek square of face, (4) the temperature distribution statistics of cheek square of face, (5) the location of high tempera-ture area of cheek square of face, and (6) the distance between high temperature area of cheek square of face. The third part, face clas-sifier, uses a back-propagation feed-forward neural network as face classifier. The details of the techniques which are used in these three parts are described in subsequent subsections.

skin, lateral of thermal face is used for recognition in this study. Grayscale image (as seen inFig. 1a) is converted into binary image (as seen in Fig. 1b) in order to segment the background, and then precisely indicates the location of nose tip, eye, and mouth of thermal face image. Otsu’s method is adopted to find the optimal binarization threshold k⁄_{[25]. Threshold k}⁄_{used to}

transfer grayscale image f(x, y) to binary image g(x, y) is given by gðx; yÞ ¼ 0 if f ðx; yÞ 6 k



1 if f ðx; yÞ > k 

Although the outline of the top of the head is not clear, the outline of the nose, eye and mouth is in clear shape inFig. 1(b). Thus, the exact location of the nose tip, eye and mouth can be labeled. Fur-ther, a precise and complete cheek square of thermal lateral face image can be determined, as described in Section2.4. When image is scanned with column from left to right and pixels are scanned from top to bottom in a column, the nose tip is the first pixel which has gray value. As soon as the nose tip is found, eye is the first trough in vertical direction from nose tip to top of the image. Sim-ilarly, mouth is second trough in vertical direction from nose tip to bottom of the image as shown inFig. 1(b).

2.2. The angle of the bridge of the nose

The angle of the bridge of the nose varies from person to person especially for different races. After the location of nose tip, eye, and mouth of thermal face image are precisely indicated in Section2.1, the angle of the bridge of the nose can be found. As can be seen in Fig. 2which is part area ofFig. 1(b), the included angle h is included by y1which connects eye to nose tip and y2which connects eye to

mouth. Consequently, the included angle h can represent the angle of the bridge of the nose and does not be influenced by hair style. Further, h hold with the same person even though there is some tilt of chin or there is different distance between capturing instrument and face.

2.3. The area of the bridge of the nose

The area of the bridge of the nose varies much from person to person even for same races. As soon as the straight line y1which

connects eye to nose tip are determined in Section2.2, the area of the bridge of the nose can be found. As can be seen inFig. 2, the area A can represent the area of the bridge of the nose and does not be influenced by hair style. This feature A can be obtained with few calculations which calculate the number of pixels locating above y1. This feature depends on distance between capturing

instrument and face. The thermal images were normalized to same size in order to avoid this effect.

2.4. The perimeter of cheek square of face

After the method that indicates the precise ambit of cheek at thermal lateral face is define and described as follows, the perim-eter of cheek square of face can be calculated.

Human faces are naturally symmetric and even the profiles fol-low geometric rule. Since the location of eye, nose tip and mouth have been identified by the method that is described in Section2.1, cheek square at thermal lateral face image can be define clearly. As can be seen in theFig. 3, cheek square at thermal lateral face image is circled. The left boundary of cheek is twice distant as the dis-tance from nose tip to mouth. The vertical disdis-tance from mouth to the bottom boundary of cheek is half of the distance from nose

tip to mouth. The top boundary of cheek is middle of the distance from nose tip to eye. Every side of cheek square has the same length.

The location and dimension of cheek squares on different faces are different because the geometric length of different faces is different. Therefore, the perimeter of cheek square of face is an effi-cient feature to recognize face. This feature can be obtained with few calculations by calculating the number of pixels of the cheek square.

2.5. The temperature distribution statistics of cheek square of face Individual cheek has individual representation of temperature which is according to the topography of blood vessels of cheek. Cheek square at thermal lateral face image which can represent temperature has been circled inFig. 3. The gray value pixels in cheek square are converted to corresponding temperature value according to the formula provided by the far-infrared capturing instrument.

Before gather temperature distribution statistics of cheek square, the temperature range is set from 30.5 °C to 33.5 °C. The range is divided into nine sections with the interval 0.375 °C;

Fig. 1. Thermal lateral face and corresponding binary image. (a) Grayscale image of thermal lateral face. (b) Corresponding binary image.

Fig. 2. The angle of the bridge of the nose h and the area of the bridge of the nose A.

The distribution of high temperature area of cheek is according to the blood vessels which transport warm blood throughout the cheek. The distribution of high temperature area of cheek is unique since the topography of blood vessels of cheek is different even at identical twins[28]. Therefore, the locations of two highest tem-perature areas which are the direct information of the topography of blood vessels are identified in this section.

Histogram is calculated from cheek of thermal image since the gray value in thermal image is positive relative to temperature. Histogram records the number of pixels of each gray value in a thermal image. As shown inFig. 5, brightest 10% pixels of histo-gram are retained in thermal image. Thermal image is converted into binary image by setting brightest 10% pixels with 1 (white) and setting else pixels with 0 (black).

After thermal image is converted into binary image, the two highest temperature areas can be extracted using ‘‘8-connected component labeling’’ algorithm as shown in Fig. 6 [29]. In ‘‘8-connected component labeling’’ algorithm, a pixel is considered as connected when it has neighbors on the same row, column and diagonal.

The cheek square is divided into nine sub-blocks which are numbered.

high temperature areas. The temperature is positive relative to gray value in thermal image. Therefore, the distance between two largest components with high gray value are identified in this section. As can be seen inFig. 7, the two largest components with high gray value of cheek square are chosen by the method which has been described in Section2.6.

In order to measure the distance between two largest compo-nents in the binary image, centroide of each component (X, Y) is calculated by X ¼ P x P ygðx; yÞx P x P ygðx; yÞ ð2Þ Y ¼ P y P xgðx; yÞy P y P xgðx; yÞ

where x, y are the coordinate of the binary image and g(x, y) is the intensity value,that is, g(x, y) = 0 or 1[30]. As soon as the centroides of two largest components are measured, the distance D between two largest components is calculated by

D ¼

ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ðX1 X2Þ2þ ðY1 Y2Þ2

q

ð3Þ

where (X1, Y1) and (X2, Y2) are the coordinate of centroides of two

largest components.

3. Experimental results and discussion

Sixty persons are involved in Infrared face image for our data-base. Each of the sixty persons is taken six pictures in different

day. Therefore, there are 360 images of thermal lateral face, and each image has size 720  480. One hundred eighty images of them are used to form training set, and the other one hundred eighty images of them are used to form testing set. Some example images of the dataset are shown inFig. 8.

Three kinds of Experiment which use different features are per-formed as shown below. In each experiment, back-propagation feed-forward neural network is applied as the classifier. The neural network is three layers which are input layer, hidden layer and output layer. The number of neurons of input layer is decided by the number of used features. Since 12, 13 and 15 features are used respectively for three different experiments, the number of neu-rons of input layer are 12, 13 and 15 respectively for three different experiments. The number of neurons of output layer is decided by the number of participated persons which is 60. The number of neurons of hidden layer is decided by

Nhidden¼

Ninputþ Noutput

2 ð4Þ

where Nhidden, Ninputand Noutputis the number of neurons of hidden

layer, the number of neurons of input layer and the number of neu-rons of output layer respectively.

The learning speed

g

neu-ral network employs momentum

a

equal to the sum of a fraction of the last weight change and the new change. Momentum allows the network to react to local gra-dient in the error surface, and it ranges between 0 and 1. The momentum

a

Fig. 5. Choose pixels from brightest 10% pixels of histogram of cheek square. (a) Thermal image of cheek square. (b) Binary image with brightest 10% pixels of histogram of cheek square.

Fig. 6. Define component using ‘‘8-connected component labeling’’ algorithm and then extract the two largest component. (a) The brightest 10% pixels can be divided with ‘‘8-connected component labeling’’ algorithm. (b) The two largest components of brightest 10% pixels of histogram of cheek square.

Fig. 7. As soon as the centroides of two largest components are measured, the distance D is calculated.

3.1. Experiment I

In this experiment, all thermal features of cheek which are de-scribed in Sections2.5–2.7are used to be input features. That is the nine temperature distribution statistics, the location of high tem-perature area and the distance between high temtem-perature areas. Thus, there are 12 input features and 12 input neurons of the neu-ral network. The hidden layer and output layer have 36 and 60 neurons, respectively. The recognition result for the testing sam-ples is shown inTable 1. The control group uses traditional thermal features which are extracted from whole face and uses same archi-tecture of neural network classifier as this study[31]. In the Ref. [31], traditional thermal features which are extracted from whole face are presented. The thermal image of whole face is processed with Haar wavelet transform and the LL band and the average of LH/HL/HH bands subimages are created for face image. Then a total confidence matrix is formed for the thermal image of whole face by taking a weighted sum of the corresponding pixel values of the LL band and average band. The total confidence matrix is used as a feature vector and is fed into neural network classifier. The recog-nition performance of the proposed thermal features of cheek and the traditional thermal features which are presented in Ref.[31]is evaluated by the precision and recall which are defined as precision ¼jfrelevant documentsg \ fretrieved documentsgj

jfretrieved documentsgj

ð5Þ recall ¼jfrelevant documentsg \ fretrieved documentsgj

jfretrieved documentsgj ð6Þ

Precision represents the ability of a measurement to be consistently reproduced. Recall represents the ability to remember experiences. Therefore, precision and recall can be calculated by

precision ¼ TP

TP þ FP ð7Þ

recall ¼ TP

TP þ FN ð8Þ

where TP, FP and FN represent True Positive, False Positive and False Negative of recognition results respectively.

As can be seen fromTable 1, the proposed thermal features of cheek have significantly higher recognition performance than tra-ditional thermal features.

3.2. Experiment II

In this experiment, cheek correlative features which are de-scribed in Sections 2.4–2.7are used to be input features. That is the perimeter of cheek square, the nine temperature distribution statistics, the location of high temperature area and the distance between high temperature areas. Thus, there are 13 input features and 13 input neurons of the neural network. The hidden layer and output layer have 37 and 60 neurons, respectively. The recognition result for the testing samples is shown in Table 2. The control group is the same control group in experiment I. The recognition performance of experimental group in experiment II is slightly higher than it in experiment I, since the perimeter of cheek square has been acceded as a feature for classifier in experiment II. 3.3. Experiment III

In this experiment, the all thermal features and facial geometric features which are described in Sections2.2–2.7are used to be in-put features for the proposed recognizer. That is the angle of the bridge of the nose, the area of the bridge of the nose, the perimeter of cheek square, the nine temperature distribution statistics, the location of high temperature area and the distance between high temperature areas. Thus, there are 15 input features and 15 input neurons of the neural network. The hidden layer and output layer have 38 and 60 neurons, respectively. The recognition result for the testing samples is shown inTable 3. The control group is the same control group in experiment I. The precision and recall of the pro-posed recognizer reach 97% and 92% respectively.

Fig. 9shows the summary of the experiments. Since the pro-posed thermal features which can directly represent the topogra-phy of blood vessels of face are used in experimental group I, the precision and recall are significantly higher than control group. The rising of precision is faster than the recall in the three experi-mental groups. The precision rises fastest and reaches highest when the critical facial geometric features which would not be

Table 1

The recognition result for experiment I.

Precision (%) Recall (%) Traditional thermal features which are extracted

from whole face

85 87

Thermal features of cheek proposed by this study 90 89

Table 2

The recognition result for experiment II.

Precision (%) Recall (%) Traditional thermal features which are extracted

from whole face

85 87

Cheek correlative features proposed by this study 92 90

Table 3

The recognition result for experiment III.

Precision (%) Recall (%) Traditional thermal features which are extracted

from whole face

85 87

influenced by hair style are involved in experimental group III. In addition, the precision is higher than the recall in the three exper-imental groups, whereas the precision is lower than the recall in the control group. Therefore, the proposed features are more con-scientious and careful than traditional thermal features which are extracted from whole face because precision represents the ability of a measurement to be consistently reproduced and recall repre-sents the ability to remember experiences. This property is quite suitable for personal recognition of access control system. 4. Conclusions

In this paper, we proposed a novel hybrid neural network face classifier with the facial geometric features and the thermal fea-tures. First of all, grayscale thermal face image of far-infrared is captured with lateral, and the image preprocessing technology seg-ments the face area out of the background. Then, several features are extracted from the processed face image. The facial geometric features which are extracted from nose and cheek include the an-gle of the bridge of the nose, the area of the bridge of the nose, and the perimeter of cheek square of face. The thermal features which are extracted from the cheek square of face include the tempera-ture distribution statistics, the location of high temperatempera-ture area, and the distance between high temperature areas. Finally, back-propagation neural network is used as classifier.

As documented in the experimental results, the proposed recog-nizer significantly improves the recognition performance. Possible explanations are:

1. Traditional thermal face recognizers can only use thermal fea-tures but facial geometric feafea-tures, since thermal face image have lower resolution and higher noise than visible face image so that the facial geometric features are difficult to identify. However, the proposed method can indicate the location of nose tip, eye, and mouth from thermal face image precisely so that it extracts not only the thermal features but also two crit-ical facial geometric features which would not be influenced by hair style to improve the recognition performance.

2. Traditional thermal face recognizers usually extract thermal features from whole face. Hence, the recognition performance decreases easily when the hair of frontal bone varies, the eye blinks or the nose breathes. However, the proposed method identifies the cheek area accurately and extracts thermal fea-tures from cheek absorbedly. Therefore, the recognition ability of the thermal features which are extracted in this study would not decrease when the hair of frontal bone varies, the eye blinks or the nose breathes. Furthermore, the cheek area which is identified in this study is bigger and more complete than the traditional thermal face recognizers. Therefore, the temperature response of the topography of blood vessels of cheek is more

obvious than tradition so that the proposed thermal features are more representative and can be extracted easily to improve the recognition performance.

3. The topography of blood vessels of face is unique even for iden-tical twins. Traditional thermal face recognizers only use the indirect information of the topography of blood vessels, like thermogram as features. However, the proposed method uses not only the indirect information, like thermogram, but also the direct information of the topography of blood vessels, such as the location of high temperature area and the distance between high temperature areas. Therefore, the recognition performance is significantly improved.

However, the proposed method is only applicable to lateral views without glasses.

Acknowledgment

The financial support of instrument technology research center is gratefully acknowledged.

References

[1]A. Kumar, D. Zhang, Personal recognition using hand shape and texture, IEEE Trans. Image Process. 15 (2006) 2454–2461.

[2]K. Nakajima, Y. Mizukami, K. Tanaka, T. Tamura, Footprint-based personal recognition, IEEE Trans. Biomed. Eng. 47 (2000) 1534–1537.

[3]W. Zhao, R. Chellappa, P.J. Phillips, A. Rosenfeld, Face recognition: a literature survey, ACM Comput. Surveys 35 (2003) 399–458.

[4]P.J. Phillips, A. Martin, C.L. Wilson, M. Przybocki, Introduction to evaluating biometric systems, Computer 33 (2000) 56–63.

[5] A. Kumar, D. Zhang, Biometric recognition using feature selection and combination, in: Proc. AVBPA, New York, 2005, pp. 813–822.

[6]A.K. Jain, A. Ross, S. Prabhakar, An introduction to biometric recognition, IEEE Trans. Circ. Syst. Video Technol. 14 (2004) 4–20.

[7]K.A. Toh, W.Y. Yau, Combination of hyperbolic functions for multimodal biometrics data fusion, IEEE Trans. Syst. Man, Cybern. B, Cybern. 34 (2004) 1196–1209.

[8]A. Ross, A.K. Jain, Information fusion in biometrics, Patt. Recogn. Lett. 24 (2003) 2115–2125.

[9] A. Kumar, D.C.M. Wong, H. Shen, A.K. Jain, Personal verification using palmprint and hand geometry biometric, in: Proc. AVBPA, Guildford, UK, 2003, pp. 668–675.

[10] Y. Wang, T. Tan, A.K. Jain, Combining face and iris for identity verification, in: Proc. AVBPA, Guildford, UK, 2003, pp. 805–813.

[11]D.S. Guru, H.N. Prakash, Symbolic representation of on-line signatures, Proc. Intl. Conf. Comput. Intell. Multimedia Appl. 2 (2007) 312–317.

[12]Q. Ibrahim, N. Abdulghani, Security enhancement of voice over Internet protocol using speaker recognition technique, IET (2012) 604–612. [13]B. Klare, A.K. Jain, Face recognition across time lapse: on learning feature

subspaces, Proc. Int. Joint Conf. Biomet. (2011) 1–8.

[14]J. Haddadnia, K. Faez, M. Ahmadi, An efficient human face recognition systemusing Pseudo ZernikeMoment Invariant and radial basis function neural network, Int. J. Patt. Recogn. Artific. Intell. 17 (2003) 41–62. [15]P. Phillips, J. Beveridge, B. Draper, G. Givens, A. O’Toole, D. Bolme, J. Dunlop,

Y.M. Lui, H. Sahibzada, S. Weimer, An introduction to the good, the bad, and the ugly face recognition challenge problem, Proc. Autom. Face Gest. Recogn. (2011) 346–353.

[16]A.K. Jain, B. Klare, U. Park, Face matching and retrieval in forensics applications, IEEE Multimedia 19 (2012) 20–25.

[17] D.A. Socolinsky, A. Selinger, A comparative analysis of face recognition performance with visible and thermal infrared imagery, in: Proceedings of the International Conference on Pattern Recognition, Quebec, Canada, 2002, pp. 217–222.

[18]D.A. Socolinsky, A. Selinger, J.D. Neuheisel, Face recognition with visible and thermal infrared imagery, Comput. Vision Image Understand. 91 (2003) 72– 114.

[19]M.U. Akram, A. Khanum, Retinal images: blood vessel segmentation by threshold probing, IEEE Symp. Ind. Electron. Appl. (2010) 475–479. [20]Shiqian Wu, Weisi Lin, Shoulie Xie, Skin heat transfer model of facial

thermograms and its application in face recognition, Patt. Recogn. 41 (2008) 2718–2729.

[21] D.A. Socolinsky, A. Selinger, Thermal face recognition over time, in: Proceedings of the International Conference on Pattern Recognition, 2004, pp. 187–190.

[22] Bai-Ling Zhang, Haihong Zhang, Shuzhi Ge, Face Recognition by Applying Wavelet Subband Representation and Kernel Associative Memory, in: IEEE Transaction on, Neural Networks, vol. 15, 2004.