Chapter 5 The Experimental Results
5.3 On-Line Experiment using a Robot Platform
5.3.3 Results of On-Line Testing
Online experiments were carried out using the embedded vision system, IPC and an pet robot as shown in Figure 5-9. We collected facial data of five persons to train the emotion classifiers. The online recognition result of five trained persons is shown in Table 5-11. The average recognition rate is 80.6%.
Subsequently, we invited four new persons to interact with Momobear, the pet
62
Figure 5-7 HRI procedure of the proposed emotion recognition system
Figure 5-8 The designed actions of Momobear
63
robot. Every person expressed five time of each emotion category under three lighting conditions in front of the CMOS image sensor. Table 5-12 shows that the recognition rate of the first new person is lower in some facial expressions. But that was increased greatly by the proposed learning algorithm. Table 5-13 gives the new recognition result after online learning. The results reveal that the average recognition rate of the first new person can be raised from 57.3% to 82.7%.
Figure 5-9 The interaction scenario
Table 5-11 The recognition result of five trained persons
anger happiness neutral sadness surprise
recognition rate 80% 77% 81% 85% 80%
Table 5-12 Recognition rate of the first new person before online learning
anger happiness neural sadness surprise
right light on 40% 0% 80% 80% 20%
all lights on 100% 0% 80% 80% 80%
left light on 40% 0% 80% 100% 80%
64
Table 5-13 Recognition rate of the first new person after online learning
anger happiness neural sadness surprise
right light on 80% 100% 80% 80% 60%
all lights on 100% 80% 80% 80% 80%
left light on 80% 80% 80% 100% 80%
The experimental procedures of the next three persons are the same as the first one.
The original recognition results of three persons are shown in Table 5-14, Table 5-16 and Table 5-18. It is observed that original recognition rates of new subjects are very low, but they can be increased dramatically by the proposed learning system. Table 5-15, Table 5-17 and Table 5-19 show the results after online learning. Each recognition result is evaluated on a new SVM classifier that is learned from the original trained one. The average rates of the four new subjects are shown in Tables 5-20~5-21. In summary, the average recognition rate of four new persons after each on-line learning can be increased from 58% to 81.3 %, which is as high a recognition rate as that of training persons (80.6%).
Table 5-14 Recognition rate of the second new person before online learning
anger happiness neural sadness surprise
right light on 20% 0% 80% 80% 20%
all lights on 20% 0% 80% 80% 20%
left light on 80% 20% 80% 80% 80%
Table 5-15 Recognition rate of the second new person after online learning
anger happiness neural sadness surprise
right light on 80% 80% 80% 80% 80%
all lights on 60% 80% 80% 80% 80%
left light on 80% 80% 80% 80% 80%
65
Table 5-16 Recognition rate of the third new person before online learning
anger happiness neural sadness surprise
right light on 20% 0% 100% 80% 100%
all lights on 80% 40% 80% 80% 0%
left light on 80% 20% 80% 100% 80%
Table 5-17 Recognition rate of the third new person after online learning
anger happiness neural sadness surprise
right light on 80% 80% 100% 80% 100%
all lights on 80% 60% 80% 80% 80%
left light on 80% 80% 80% 100% 80%
Table 5-18 Recognition rate of the fourth new person before online learning
anger happiness neural sadness surprise
right light on 80% 80% 80% 100% 40%
all lights on 100% 80% 0% 80% 0%
left light on 80% 60% 80% 40% 40%
Table 5-19 Recognition rate of the fourth new person after online learning
anger happiness neural sadness surprise
right light on 80% 80% 80% 100% 100%
all lights on 100% 80% 80% 80% 80%
left light on 80% 80% 80% 80% 80%
Table 5-20 Average recognition results of four new persons before online learning anger happiness neural sadness surprise
right light on 40% 20% 85% 85% 45%
all lights on 75% 30% 60% 80% 25%
left light on 70% 25% 80% 90% 60%
66
Table 5-21 Average recognition results of four new persons after online learning
anger happiness neural sadness surprise
right light on 80% 80% 85% 85% 85%
all lights on 85% 75% 80% 80% 80%
left light on 80% 75% 80% 90% 80%
Table5-22 shows that the recognition rates of previously data also can be maintained (78.67% on the average) even when the SVM classifiers are adjusted for the new individual. With the proposed learning, the pet robot not only accommodates itself to new facial data but also keeps the satisfactory performance of old data.
Table 5-23 presents the experimental results of comparing the proposed method with the previous method developed in [41]. We see that the performances of the previous method are much lower than that of the proposed method. This is because their method cannot accommodate the FER system to facial expressions of new subjects and feature extraction of the method may fail under illumination variation.
Therefore, we conclude that the developed FER system with a learning function outperforms those without learning or accommodating functions while categorizing facial expressions of new subjects.
Figure 5-10 shows an example of emotional interaction with Momobear. Figure 5-10 (a) is a successful recognition with Momobear. Figure 5-10 (b) shows that Momobear misclassifies surprise expression. Figure 5-10 (c) illustrates that the user informs Momobear to learn surprise expression through learning buttons. Figure 5-10 (d) is a successful recognition of surprise expression after online learning.
67
Table 5-22 The recognition results of five trained person after online learning
anger happiness neutral sadness surprise Average recognition rate
after SVM classifier learning
82% 77% 82% 82% 72% 78.67%
Table 5-23 Recognition results of the method proposed in [41]
anger happiness neural sadness surprise
right light on 45% 20% 65% 60% 80%
all lights on 70% 50% 90% 25% 75%
left light on 75% 60% 15% 70% 70%
(a) (b)
(c) (d) Figure 5-10 An example of interaction with Momobear.
68
Chapter 6
Conclusions and Future Work
6.1 Conclusions
This thesis presents a fast facial expression learning algorithm for a pet robot. An emotion recognition system can accommodate itself to new facial data. The proposed learning method adjusts parameters of SVM hyperplane. After adjusting hyperplane parameters, the new classifier not only recognizes new facial data but also keeps acceptable recognition rates of classifying previous old data. Because only new erroneous samples combined with historical critical sets (CSs) are used to restrain a new SVM classifier, the proposed algorithm can speed up the learning procedure.
Further, to obtain facial features correctly, Gabor wavelet based feature extraction is employed in the FER system.
The proposed FER algorithm has been evaluated on the AR Face Database and the database built in the lab. These offline experimental results show that recognition rate is 81.7% on The AR Face Database and 81.5% on the lab database. Moreover, the learning algorithm also has been verified using self-built database and a robot platform. The average recognition rate of new persons after online learning can be raised from 58% to 81.3%. In the meantime, new SVM classifier also keeps satisfactory performance of recognizing old data (78.7%).
6.2 Future work
Gabor wavelet based feature extraction method has robust properties against the changes of lighting conditions, but the computation cost of extracting facial points is very high. Moreover, extracting feature points around the mouth is not stable enough.
69
In the future, some popular facial feature models such as active appearance model (AAM) can be combined with Gabor based feature extraction method to improve the performance of extracting facial points. One alternative solution is to use Gabor based feature matching to detect fiducial points in the first frame of image sequences. Then, in the subsequent image frames, AAM or other facial feature models are applied to reduce the computation cost and raise the recognition rates of detecting facial points.
On the other hand, we will also work on new methods for improving the error rate of the original trained data with SVM learning in order to apply the proposed learning algorithm to practical robotic applications.
70
References
[1] M. Fujita, “On Activating Human Communications with Pet-type Robot AIBO,”
Proc. of the IEEE, vol. 92, no. 11, pp. 1804-1813, 2004.
[2] P. Ekman and W. V. Frisen, Emotion in the Human Face. Englewood Cliffs, NJ：
Prentice-Hall, 1975.
[3] P. Ekman and W. V. Friesen：The Facial Action Coding System (Consulting Psychologists Press, San Francisco 1978).
[4] http://face-and-emotion.com/dataface/facs/manual/TitlePage.html
[5] M. Pantic and L.J.M. Rothkrantz, “Automatic Analysis of Facial Expressions：
The State of the Art,” IEEE Trans. on Pattern Analysis and Machine Intelligence, vol. 22, no. 12., pp. 1424 - 1445, 2000.
[6] Gengtao Zhou, Yongzhao Zhan and Jianming Zhang, “Facial Expression Recognition Based on Selective Feature Extraction,” in Proc. of IEEE 6th Int. Conf.
on Intelligent Systems Design and Applications, Shandong, China, 2006, pp. 412 – 417.
[7] F. Bourel, C.C. Chibelushi, and A.A. Low, “Robust Facial Expression Recognition Using a State-Based Model of Spatially-Localised Facial Dynamics,” in Proc. of IEEE 5th Int. Conf. on Automatic Face and Gesture Recognition, Washington, D.C, USA, 2002, pp. 106 – 111.
[8] WeiFeng Liu and ZengFu Wang, “Facial Expression Recognition Based on Fusion of Multiple Gabor Features,” in Proc. of IEEE 18th Int. Conf. on Pattern Recognition, Hong Kong, China, 2006, pp. 536 – 539.
[9] M. Valstar and M. Pantic, “Fully Automatic Facial Action Unit Detection and Temporal Analysis,” in Proc, IEEE Int. Conf. on Computer Vision and Pattern Recognition Workshop, New York, USA, 2006, pp. 149-156.
[10] G. Guo and C.R. Dyer, “Learning from Examples in the Small Sample Case:
Face Expression Recognition,” IEEE Trans. on Systems, Man and Cybernetics, Part B, vol. 35, no. 3, pp. 477 – 488, 2005.
[11] V. Vanik, The Nature of Statistic Learning Theory, New York ： Springer-Verlag,1995.
71
[12] N. Cristianini and J. Shawe-Taylor, An Introduction to Support Vector Machines and Other Kernel-based Learning Methods, NY Cambridge University Press 2000.
[13] M.Yeasin, B. Bullot, and R. Harma, “Recognition of Facial Expressions and Measurement of Levels of Interest From Video,” IEEE Trans. on Multimedia,vol.8, no.3, pp.500-508, 2006.
[14] Zhengyou Zhang, M. Lyons, M. Schuster, and S. Akamatsu, “Comparison Between Geometry-Based and Gabor-Wavelets-Based Facial Expression Recognition Using Multi-Layer Perception,” in Proc. of 3rd IEEE Conf. on Automatic Face and Gesture Recognition, Nara, Japan, 1998, pp. 454 – 459.
[15] Y. Z. Zhan, J. F. Ye, D. J. Niu, and P. Cao, “Facial Expression Recognition Based on Gabor Wavelet Transformation and Elastic Templates Matching,” in Proc. of 3rd Int. Conf. on Image and Graphics, Hong Kong, China, 2004, pp. 254 – 257.
[16] Y. Saatci and C. Town, “Cascaded Classification of Gender and Facial Expression Using Active Appearance Models,” in Proc. of 7th Int. Conf. on Automatic face and Gesture Recognition, Southampton, UK, 2006, pp. 393 – 398.
[17] G. J. Edwards ,T. F. Cootes and C. J. Taylor, “Face Recognition Using Active Appearance Models, ” in Proc. of European Conf. Computer Vision, Freiburg, Germany, 1998, pp.581-695.
[18] I. Kotsia and I. Pitas, “Facial Expression Recognition in Image Sequences Using Geometric Deformation Features and Support Vector Machines,” IEEE Trans. on Image Processing, vol.16, no.1 , pp. 172-187, 2007
[19] T. Kobayashi, Y. Ogawa, K. Kato and K. Yamamoto,“ Learning Systems of Human Facial Expression for a Family Robot,” in Proc. of the 6th IEEE Int. Conf.
on Automatic Face and Gesture Recognition, Seoul, Korea, 2004, pp. 481-486.
[20] S. Ozawa, S.L. Toh, S. Abe, Pang Shaoning and N. Kasabov,“ Incremental Learning for Online Face Recognition,” In Proc. of Int. Joint Conf. on Neural Networks, Montreal, Canada, 2005, pp. 3174 – 3179.
[21] W. M. Huang, B. H. Lee, L. Y. Li and K. Leman,“ Face Recognition by Incremental Learning,” In Proc. of IEEE Int. Conf. on Systems, Man and Cybernetics, Washington, D.C., USA, 2003, pp.4728-4723.
[22] R. Xiao, J. Wang and F. Zhang, “An Approach to Incremental SVM Learning Algorithm,” in Proc. of the 12th International Conference on Tools with Artificial Intelligence, Vancouver, Canada, 2000, pp.268– 273.
72
[23] J. L. An, Z. O. Wang and Z. P. Ma, “An Incremental Learning Algorithm for Support Vector Machine,” in Proc. of Int. Conf. on Machine Learning and Cybernetics, Xian,, China, 2003, pp. 1153 – 1156.
[24] L. Xuchun, Z. Yan and E. Sung, “Sequential Bootstrapped Support Vector Machines - a SVM accelerator,” in Proc. of IEEE Int. Joint Conf. Neural Networks, Montréal, Canada, 2005, pp. 1437 – 1442.
[25] W. H. Zeng and J. Ma, “A Novel Incremental SVM Learning Algorithm,” in Proc. of Int. Conf. on CSCW in Design, Xiamen, China, 2004, pp. 658 – 662.
[26] P. Mitra, C. A. Murthy and S. K. Pal, “Data Condensation in Large Databases by Incremental Learning With Support Vector Machines,” In Proc. of Intl. Conf. on Pattern Recognition, Catalonia, Spain, 2000, pp. 708 – 711.
[27] Y. Liu, Q. He and Q. Chen, “Incremental Batch Learning with Support Vector Machines,” in Proc. of 5th world congress on Intelligent Control and Automation, Hangzhou, China, 2004, pp. 1857– 1861.
[28] Y. Liu, Q. He and Y. Tang, “Support Vector Pursuit Learning,” in Proc. of IEEE Int. Conf. on System, Man and Cybernetics , The Hague, The Netherlands, 2004, pp.5841– 5846.
[29] C.M. Chou: Real-Time Face Tracking Under Illumination Variation, Master. Eng.
Thesis, National Chiao Tung University, Hsinchu, Taiwan, R.O.C., 2005.
[30] A. McAndrew, Introduction to Digital Image Processing with Matlab, Thomson Course Technology, 2004
[31] J.H. Lai, P.C Yuen, W.S. Chen, S. Lao and M. Kawade, “Robust Facial Feature Point Detection Under Nonlinear Illuminations,” in Proc. of IEEE ICCV Workshop on Recognition, Analysis and Tracking of Faces and Gestures in Real-time Systems, Canada, 2001, pp.168-174.
[32] J. K. Kamarainen and H. Kalviainen, “Invariance Properties of Gabor Filter-Based Features---Overview and Application,” IEEE Trans. on Image Processing, vol. 15, no 5, pp.1088-1099, 2006.
[33] L. Wiskott, J. M. Fellous, N. Kruger and von der Malsburg, “Face Recognition by Elastic Bunch Graph Matching,” IEEE Trans. on Pattern analysis and Machine Intelligence, vol. 19, no. 7, pp. 775-779, 1997.
[34] E. J. Hokdeb and R. Owens, “Automatic Facial Point Detection,” in Proc. of 5th Asian Conference on Computer Vision, Melbourne, Australia, 2002, pp. 731-736.
[35] D. Vukadinovic and M. Pantic, “Fully Automatic Facial Feature Point Detection Using Gabor Feature Based Boosted Classifiers,” in Proc. of IEEE Int. Conf. on Systems, Man and Cybernetics, Hawaii, U.S.A., 2005, pp. 1692 – 1698.
73
[36] Y. Zhang and Q. Ji, “Active and Dynamic Information Fusion for Facial Expression Understanding from Image Sequences,” IEEE Trans. on Pattern Analysis and Machine Intelligent, vol. 27, no 5, pp. 699-714, 2005
[37] http://cobweb.ecn.purdue.edu/~aleix/aleix_face_DB.html [38] IC-MEDIA Corp.: ICM205B Datasheet, Oct 2002.
[39] Henrry Andrian and K. T. Song, “Embedded CMOS Imaging System for Real-Time Robotic Vision,” in Proc. of IEEE/RSJ International Conference on Intelligent Robots and Systems, Edmonton, Alberta, Canada, 2005, pp. 3694-3699.
[40] http://www.pololu.com/products/pololu/0727/
[41] J.H. Hsu: Bimodal Emotion Recognition System Using Image and Speech Information, Master. Eng. Thesis, National Chiao Tung University, Hsinchu, Taiwan, R.O.C., 2006.