本論文提出一個利用二階段臉部表情紋理特徵的擷取技術，結合使用 TLBP 及 CS-LBP 兩種方法。在第一階段我們使用 TLBP 獲得紋理特徵影像，因為 TLBP 對於取得全域紋理特徵影像較 LBP 優秀，可去除雜訊及盡量避免因影像平滑所 產生的誤差。第二階段區塊式擷取影像特徵直方圖的部分，我們則使用 CS-LBP。
由於 CS-LBP 對於區域特徵的表現極佳，加上其特徵維度僅是 LBP 的十六分之 一；因此，在使用區塊式方法建構特徵直方圖時，可有效降低影像特徵直方圖維 度，減少後續分類器訓練時間。本論文之二階段式方法乃結合上述兩種方法，比 起單獨使用 TLBP 或 CS-LBP 方法，更能提高臉部表情的辨識效能。由於 TLBP 及 CS-LBP 運算簡單且速度快，非常適合用於即時臉部表情辨識系統中。
針對本論文提出的方法，主要的研究結論有兩部分：
特徵擷取部分：能否有效取得影像特徵資訊，將是影響臉部表情辨識系統的 成敗關鍵。本論文所提方法，實驗結果證實具有一定的辨識能力，就算是低解析 度影像的臉部表情辨識，仍有一定的穩定性。本論文所提方法比 LBP、LDP、
Gabor 小波及幾何特徵的方法辨識準確率高，而且結果相對穩定。此外，我們更 改善其他文獻方法特徵維度較大，後續所需分類訓練時間較長的問題。
分類辨識部分：影像樣本數的多寡及各類影像的特徵資訊，直接影響分類器 的分類能力，造成分類器的分類能力隨著樣本數的增減而產生大幅度的變化。本 論文所提的方法能夠有效擷取出每類表情之間差異性較大的特徵，實驗結果證實，
即使每種表情類別採用的樣本數大幅減少，也不會對系統的辨識能力產生巨大的 影響。但是若樣本本身的影像特徵資訊不明顯，則後續分類效果仍會受其影響；
如此，可考慮結合其他特徵資訊(例如：生理訊號、語音等)進行表情的辨識。
對於未來主要的研究重點有：
1. 增加影像資料庫：目前特徵資訊擷取僅限於現成資料庫，未來可增加更多的 臉部表情影像以測試系統，提供更客觀的分析數據。
2. 強化特徵資訊：本論文所提方法主要針對影像進行特徵資訊擷取。若遇到如 JAFFE 資料庫中部分表情特徵不明顯，造成辨識率下降的問題，未來從臉部 特徵區域的取得，或是特徵資訊擷取方法的改進，做為強化系統效能的目 標。
3. 實際應用層面：除了原有的兩組表情資料庫之外，可增加其他層面的表情資 料庫(例如：自閉症兒童、嬰幼兒或老人等)。這些不同層面的資料庫應用可 作為未來本系統延伸與改進的空間。此外，目前系統主要以單張影像為辨識 對象，未來可朝向辨識視訊中人臉的各式表情，應用上將會更為廣泛。
4. 朝多目標發展：雖然本系統的發展主要用於表情分類，但本論文所提之方法 架構可應用於其他人機介面的實例(例如：性別分類或年齡估測等)，推廣至 更多的分類應用。
參考文獻
[1] P. Ekman, Emotions Revealed: Recognizing Faces and Feelings to Improve Communication and Emotional Life, Times Books, 2003.
[2] P. Ekman and W. V. Friesen, “Constants across cultures in the face and emotion,”
Journal of Personality and Social Psychology, vol. 17, pp.124-129, 1971.
[3] P. Ekman and W. V. Friesen, Unmasking the Face, Malor Books, Los Altos, 2003.
[4] W. Liu, Y. Wang, and S. Li, “LBP feature extraction for facial expression recognition,” Journal of Information & Computational Science, vol. 8, no. 3, pp.
412-421, 2011.
[5] C. Shan, S. Gong, and P. W. McOwan, “Conditional mutual information based boosting for facial expression recognition,” in Proc. British Machine Vision Conference, Sep. 2005, vol. 1, pp. 399-408.
[6] C. Shan and T. Gritti, “Learning discriminative LBP-histogram bins for facial expression recognition,” in Proc. British Machine Vision Conference, 2008, pp.
2-12.
[7] C. Shan, S. Gong, and P. W. McOwan, “Facial expression recognition based on local binary patterns: A comprehensive study,” Image and Vision Computing, vol.
27, no. 6, pp. 803-816, 2009.
[8] Y. Tian, L. Brown, A. Hampapur, S. Pankanti, A. Senior, and R. Bolle, “Real world real-time automatic recognition of facial expression,” in Proc. IEEE Workshop on Performance Evaluation of Tracking and Surveillance, 2003, pp.
2-5.
[9] Y. Tian, “Evaluation of face resolution for expression analysis,” in Proc.
Computer Vision and Pattern Recognition Workshop on Face Processing in Video, 2004, pp. 82-88.
[10] S. Liao, W. Fan, C. S. Chung, and D.-Y. Yeung, “Facial expression recognition using advanced local binary patterns, tsallis entropies and global appearance features,” in Proc. IEEE International Conference on Image Processing, 2006, pp. 665-668.
[11] T. Jabid, M. H. Kabir, and O. Chae, “Robust facial expression recognition based on local directional pattern,” ETRI Journal, vol. 32, no. 5, pp. 784-794, 2010.
[12] X. Tan and B. Triggs, “Enhanced local texture feature sets for face recognition under difficult lighting conditions,” in Proc. IEEE International Workshop on Analysis and Modeling of Faces and Gestures, 2007, vol. 4778, pp. 168-182.
[13] B. Hossain, A. Faisal, and H. Emam, “Person-independent facial expression
recognition based on compound local binary pattern (CLBP),” The International Arab Journal of Information Technology First Online Publication, 2012.
[14] M. Kabir, T. Jabid, and O. Chae, “Local Directional Pattern Variance (LDPv) : A robust feature descriptor for facial expression recognition,” The International Arab Journal of Information Technology, vol. 9, no. 4, pp. 382-391, 2012.
[15] T. Gritti, C. Shan, V. Jeanne, and R. Braspenning, “Local Features based facial expression recognition with face registration errors,” in Proc. IEEE International Conference on Automatic Face and Gesture Recognition, 2008, pp. 1-8.
[16] J. Meng, Y. Gao, X. Wang, T. Lin, and J. Zhang, “Face recognition based on local binary patterns with threshold,” in Proc. IEEE International Conference on Granular Computing, 2010, pp. 352-356.
[17] M. Heikkila, M. Pietikäinen, and C. Schmid, “Description of interest regions with center-symmetric local binary patterns,” in Proc. 5th Indian Conference on Computer Vision, Graphics and Image Processing, 2006, pp. 58-69.
[18] T. Kanade, J. F. Cohn, and Y. Tian, “Comprehensive database for facial expression analysis,” in Proc. IEEE International Conference on Automatic Face and Gesture Recognition, Grenoble, France, 2000, pp. 46-53.
[19] M. J. Lyons, S. Akamatsu, M. Kamachi, and J. Gyoba, “Coding facial expressions with Gabor wavelets,” in Proc. Automatic Face and Gesture Recognition, Apr. 1998, pp. 200-205.
[20] M. Pantic and L. Rothkrantz, “Expert system for automatic analysis of facial expression,” Image and Vision Computing, vol. 18, no. 11, pp. 881-905, 2000.
[21] Y. Tian, T. Kanade, and J. Cohn, “Recognizing action units for facial expression analysis,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.
23, no. 2, pp. 97-115, Feb. 2001.
[22] R. E. Kaliouby and P. Robinson, “Real-time inference of complex mental states from facial expressions and head gestures,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition, 2005, pp. 181-200.
[23] M. Pantic and L. J. M. Rothkrantz, “Facial action recognition for facial expression analysis from static face images,” IEEE Transactions on Systems, Man, and Cybernetics -- Part B: Cybernetics, vol. 34, no. 3, pp. 1449-1461, 2004.
[24] M. Pantic and I. Patras, “Dynamics of facial expression: recognition of facial actions and their temporal segments from face profile image sequences,” IEEE Transactions on Systems, Man, and Cybernetics -- Part B: Cybernetics, vol. 36, no. 2, pp. 433-449, 2006.
[25] I. Kotsia and I. Pitas, “Using geometric deformation features and support vector machines,” IEEE Transactions on Image Processing, vol. 16, no. 1, pp. 172-187,
2007.
[26] C. Padgett and G. Cottrell, “Identifying emotion in static face images,” in Proc.
Second Joint Symp. Neural Computation, 1995, vol. 5, pp. 91-101.
[27] P. Ekman and W. V. Friesen, “Facial action coding system,” Consulting Psychologists Press, College Avenue, Palo Alto, CA, 1978.
[28] Z. Zhang, M. J. Lyons, M. Schuster and S. Akamatsu, “Comparison between geometry-based and Gabor-wavelets-based facial expression recognition using multi-layer perceptron,” in Proc. IEEE International Conference on Automatic Face and Gesture Recognition, Apr. 1998, pp. 454-459.
[29] G. Donato, M. Bartlett, J. Hager, P. Ekman, and T. Sejnowski, “Classifying facial actions,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 21, no. 10, pp. 974-989, 1999.
[30] M. S. Bartlett, G. Littlewort, M. Frank, C. Lainscsek, I. Fasel, and J. Movellan,
“Recognizing facial expression: machine learning and application to spontaneous behavior,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition, 2005, vol. 2, pp. 568-573.
[31] M. S. Bartlett, J. R. Movellan, and T. J. Sejnowski, “Face recognition by independent component analysis,” IEEE Transactions on Neural Networks, vol.
13, no. 6, pp. 1450-1464, 2002.
[32] S. Bashyal and G. K. Venayagamoorthy, “Recognition of facial expressions using Gabor wavelets and learning vector quantization,” Engineering Applications of Artificial Intelligence, vol. 21, no. 7, pp. 1056-1064, 2008.
[33] C. Shan, S. Gong, and P. W. McOwan, “Robust facial expression recognition using local binary patterns,” in Proc. IEEE International Conference on Image Processing, 2005, vol. 2, pp. 370-373.
[34] X. Feng, M. Pietikäinen, and A. Hadid, “Facial expression recognition with local binary patterns and linear programming,” Pattern Recognition and Image Analysis, vol. 15, no. 2, pp. 546-548, 2005.
[35] X. Feng, A. Hadid, and M. Pietikäinen, “A coarse-to-fine classification scheme for facial expression recognition,” in Proc. International Conference on Image Analysis and Recognition, 2004, vol. 3212, pp. 668-675.
[36] S. M. Lajevardi and Z. M. Hussain, “Facial expression recognition using Log-Gabor filters and local binary pattern operators,” in Proc. International Conference on Communication, Computer and Power, Feb. 2009, pp. 15-18.
[37] T. Ahonen, A. Hadid, and M. Pietikäinen, “Face recognition with local binary patterns,” in Proc. European Conference on Computer Vision, 2004, vol. 3021, pp. 469-481.
[38] O. G. Cula and K. J. Dana, “Compact representation of bidirectional texture
functions,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition, 2001, vol. 1, pp. 1041-1047.
[39] K. I. Laws, “Texture energy measures,” in Proc. Image Understanding Workshop, 1980, pp. 47-51.
[40] T. Leung and J. Malik, “Representing and recognizing the visual appearance of materials using three-dimensional textons,” International Journal of Computer Vision, vol. 43, no. 1, pp. 29-44, 2001.
[41] C. Zhu, C. E. Bichot, and L. Chen, “Multi-scale color local binary patterns for visual object classes recognition,” in Proc. International Conference on Pattern Recognition, 2010, pp. 3065-3068.
[42] G. W. Yuan, Y. Gao, D. Xu, and M. R. Jiang, “A new background subtraction method using texture and color information,” in Proc. International Conference on Intelligent Computing, 2011, pp. 541-548.
[43] P. Viola and M. J. Jones, “Robust real-time face detection,” International Journal of Computer Vision, vol. 57, no. 2, pp. 137-154, 2004.
[44] J. A. R. Castillo, A. R. Rivera, and C. Oksam, “Facial expression recognition based on local sign directional pattern,” in Proc. IEEE International Conference on Image Processing, 2012, pp. 2613-2616.
[45] C. Cortes and V. Vapnik, “Support-vector networks,” International Journal of Machine Learning, vol. 20, Issue 3, pp. 273-297, 1995.
[46] OpenCV, http://opencv.org.
[47] T. Ojala, M. Pietikäinen, and T. Mäenpää, “Multiresolution gray-scale and rotation invariant texture classification with local binary patterns,” IEEE Trans.
Pattern Anal Mach. Intel., vol. 24, no. 7, pp. 971-987, 2002.
[48] T. Ojala, M. Pietikäinen, and D. Harwood, “A comparative study of texture measures with classification based on featured distribution,” Pattern Recognition, vol. 29, no. 1, pp. 51-59, 1996.
[49] C. W. Hsu and C. J. Lin, “A comparison of methods for multiclass Support Vector Machines,” IEEE Trans. on Neural Networks, vol. 13, no. 2, pp. 415-425, 2002.
[50] C. C. Chang and C. J. Lin, LIBSVM: a library for support vector machines.
Software available at http://www.csie.ntu.edu.tw/~cjlin/libsvm, 2001.
[51] 莊順旭，基於權重式區域方向特徵和區域二元化形態之表情辨識方法。中華 大學資訊工程學系碩士論文，99年8月。
[52] 劉懿萱，植基於全域與局部紋理特徵之臉部表情辨識。國立高雄大學電機工 程學系碩士論文，101年6月。