遠距點選輸入之手勢互動控制系統開發

行政院國家科學委員會專題研究計畫 成果報告

遠距點選輸入之手勢互動控制系統開發(第 2 年) 研究成果報告(完整版)

計 畫 類 別 ： 個別型

計 畫 編 號 ： NSC 96-2628-E-011-007-MY2

執 行 期 間 ： 97 年 08 月 01 日至 98 年 07 月 31 日 執 行 單 位 ： 國立臺灣科技大學工業管理系

計 畫 主 持 人 ： 李永輝

計畫參與人員： 碩士班研究生-兼任助理人員：陳建州 碩士班研究生-兼任助理人員：劉燕萍 碩士班研究生-兼任助理人員：黃美甄 博士班研究生-兼任助理人員：林蕙如

報 告 附 件 ： 出席國際會議研究心得報告及發表論文

處 理 方 式 ： 本計畫可公開查詢

中 華 民 國 98 年 09 月 23 日

行政院國家科學委員會補助專題研究計畫

報 告

遠距點選輸入之手勢互動控制系統開發

(二年計畫之第二年)

A remote input device base on controls from finger gesture recognition system

計畫類別：□ 個別型計畫

計畫編號：NSC 96－2628- E －011 － 007 －MY2

執行期間： 96 年 8 月 1 日至 98 年 7 月 31 日

計畫主持人：李永輝 國立台灣科技大學 工業管理系 計畫參與人員：柯志涵、葉陳鴻、吳淑楷 工業管理系

成果報告類型(依經費核定清單規定繳交)：□精簡報告

本成果報告包括以下應繳交之附件：

□赴國外出差或研習心得報告一份

□赴大陸地區出差或研習心得報告一份

█出席國際學術會議心得報告及發表之論文各一份

□國際合作研究計畫國外研究報告書一份

處理方式：除產學合作研究計畫、提升產業技術及人才培育研究 計畫、列管計畫及下列情形者外，得立即公開查詢

執行單位： 國立台灣科技大學 工業管理系

中 華 民 國 98 年 9 月 20 日

可供推廣之研發成果資料表

□ 可申請專利 □ 可技術移轉

日期：98 年 9 月 20 日

國科會補助計畫

計畫名稱：遠距點選輸入之手勢互動控制系統開發 計畫主持人： 李永輝

計畫編號：NSC 96－2628- E －011 － 007 －MY2 學門領域：工業工程與管理

技術/創作名稱 遠距點選輸入之手勢互動控制系統 發明人/創作人 李永輝

中文：

利用兩部攝影機發展出的三度空間，以自然手部動作為輸入控制設備，萃取 手部與頭部的膚色，並以投手連線投射到螢幕，進行遠距的點選控制。

技術說明

A GOAL-DIRECTED POINTING TRACKING SYSTEM: we

combine stereoscopic range information and skin-color

classification to achieve a robust tracking system of the free hand movement in 3D. The setup consists of two fixed-baseline cameras connected to PC.

可利用之產業 及 可開發之產品

智慧環境配合的遠距輸入裝置

技術特點 三度空間自然姿勢的輸入方法

推廣及運用的價值 與智慧環境結合，最適合於遠距與大型銀幕的互動需求

※ 1.每項研發成果請填寫一式二份，一份隨成果報告送繳本會，一份送 貴單位

成果推廣單位（如技術移轉中心）。

A remote, hand-free, 3-dimensional finger trajectory tracking system and its application

Lee, Yung-Hui and Ko, Chin-Han Department of Industrial Management,

National Taiwan University of Science and Technology, No. 43 Kee-lung Road, Section 4, Taipei, Taiwan, ROC

yhlee@im.ntust.edu.tw

Keyword：Finger and Head Orientation, 3-dimensional tracking, remote interaction

In this paper, we describe a simple and inexpensive solution to providing real time 3D input of hand and head orientation called “goal-directed pointing tracking system (GPTS).” We provided 1) a presentation of GPTS that tracked the hand and head orientations and 2) an experimental verification of the reliability of GPTS, and (3) an interaction with a music player. GPTS was demonstrated to recognize a range of gestures which was performed in our 3D video based recognition environment.

The recognition rate of this scenario is above 90% when user’s head and hands were tracked.

INTRODUCTION

As an alternative to traditional input devices, researchers are exploring the possibility of adopting goal-directed pointing movements as an input function to various appliances of computer games, interactive computer graphics, and remote control for home appliances in a smart environment. Pointing is a movement of the hand/arm towards a specific object, location, and/or a direction. Among the set of gestures performed by humans when communicated with each other and/or with machines, pointing movement has the most spatial compatibility. In recognition pointing movements, the detections of the occurrence of the finger-hand-arm movements and the pointing directions have to be addressed (Kai, et al., 2003).

Most of pointing movement system recognition forearm orientation only. Our previous study of remote pointing accuracy in a distance of 3 meters showed that pointing accuracy to a target size of 7.5 cm in radius with no visual cue, the hitting rate was only 66.0% and the spread ranges was 7.02±5.30cm (Lee, et al., 2008). When pointing an object, the eye, the finger, and the object should be collinear. It was hypothesized in this study that pointing accuracy can be improved by the inclusion of the hand and head orientation into the determination of pointing location, (Lee, et al., 2001).

Figure 1 showed the environment of this remote interaction, where hand and head orientation was recognize, and the direction vector was determined according to the extended line of the hand and head vector. What unknown is the accuracy of this goal-directed pointing movement.

Figure 1: Example of remote interaction with electronic systems. In this paper, we describe a simple and inexpensive solution to providing real time 3D input of hand and head orientation called “goal-directed pointing tracking system (GPTS).” We provided 1) a presentation of GPTS that tracking the hand and head orientations and 2) an experimental verification of the reliability of GPTS by calibrating the pointing accuracies in 3D, and (3) an interaction with a music player, which contains functions of play, pause, volume controls, move to a previous and/or a next control.

GOAL-DIRECTED POINTING TRACKING SYSTEM In the study, we combine stereoscopic range information and skin-color classification to achieve a robust tracking performance. The setup consists of two fixed-baseline cameras connected to PC.

3-D Calibration

Based on data from a 2 camera system, direct linear transformation (DLT) was used to obtain 3D coordinates for the system. Measurements of the focus, positions, rotation angles, and distance parameters of the two cameras were used as inputs for the calibration.

2-D Hand/Head Area Detection

GPTS uses skin detection and background subtraction techniques to isolate the image of hand and head on each frame, which is then used for direction calculation. Del/The skin detection algorithm is a modified version of that of McKenna and Morrison (2004). Areas of hand and head were detected by identifying ellipse cylinder skin areas in the ycbcr color space of the images from two cameras. Only pixels of skin-color-liked are collected to a coordinate sheet.

The most memory-saving format is to use a binary dataset.

Erosion algorithm was used to filter out the noise. Finally, a K-means clustering mechanism (K=3) was used to cluster the coordinates sheet into 3 skin-area sets (2-D positions of the head and 2 hands). These operations are applying onto the both 2 images in the initial state.

2-D Hand/Head Position Tracking

It is not efficient to scan the whole stereo-image, after obtaining the initial 2-D hand and head coordinates, method of kernel based object tracking (Comaniciu et al., 2003) was adopted. When color histogram that describe the object population, a comparisons of the current and the previous target populations were conducted, update by gradient information until the correlation coefficient is large enough and then the scan stopped. Since the moving trajectories of hand and head are nearby-differentiable, the method works efficiently in this application.

Figure 2: hand/head position tracking and the pointing movement

3-D Goal-directed pointing movement

The trajectories of pointing movement were smoothed by Kalman filter (Keskin, et al., 2003). Combine the paired 2-D hand and head positions from each image by DLT parameters, the 3-D coordinates of the goal directing pointing were calculated. Sequences of hand and head information are then used for goal-directed pointing controls.

The system is capable of recognizing gestures at a speed of 20 Hz. It was then the velocity and the accelerations of the

pointing movements were calculated.

SYSTEM VALIDATIONS

In order to evaluate the performance of our system, we prepared a calibration test and an aiming stability tests. There were 16 different markers on the calibration frame. The geometric relationships and the 3D Cartesian coordinates of the 16 markers were known. The frame was of a size of one cubic meter. The frame was moved around and into the filed of view of the two cameras in a distance of 3.6 m. The 3D coordinates of the markers were calculated and the standard deviations of the differences of the mean coordinates in three axes were presented in Table 1.

Table 1: Reliability of 3-D calibration (N=16) Standard Deviation(cm) in Experiment x(cm) z(cm)

Moving ˆσRange Moving ˆσ Range 1 2.26 9.3 1.66 4.60 2 2.18 8.6 1.83 6.32 3 2.94 12.1 1.56 5.44 4 2.54 10.9 1.48 4.89 5 2.04 8.9 1.42 5.72 6 2.66 11.2 1.87 6.08 7 2.29 10.2 1.45 5.42 8 2.61 11.3 2.03 6.48 9 1.97 8.5 1.49 5.68 10 2.74 10.2 1.63 5.52 2.44 10.1 1.66 5.62

Table 2 presents the aiming stability and the range of 10 goal-directed pointing movements. It showed that the aiming accuracy, in a distance of 3.2 m which is about 5 times of the length of head/hand link, can be achieved in a range of 10 cm in X-axis and 6 cm in Z-axis.

Table 2 Point Stability in 3.2 m No. X-axis Y-axis Z-axis

1 0.012 0.124 0.032 2 0.011 0.326 0.012 3 0.006 0.475 0.021 4 0.011 0.102 0.015 5 0.007 0.079 0.014 6 0.003 0.207 0.033 7 0.008 0.292 0.007 8 0.016 0.166 0.017 9 0.004 0.143 0.036 10 0.012 0.078 0.009 total σ 0.031 0.739 0.070

Reliability test of the 3D coordinates showed that the most variations (<0.8 cm) appeared in Y-axis (forward and backward movements)

Figure 3: Aiming stability in 3.2 m

SYSTEM APPLICATON

To verify the advantage of GPTS, we built a remote controlled music player with free-hand comments. The comments were achieved by continuous monitoring the changes of the 3-D coordinates and the rotational angles of the pointing movements, either clockwise or counter-clockwise.

Figure 4: Tracking of the changes of the rotational angle both clockwise and counter-clockwise

In this application, comments were defined as following:

1. Comment initiation: continuously monitoring 20 frames of

formation of a circle with continuously changed rotational angles. This starts when there are forward movements (push) in the Y-axis and the velocity is greater than a threshold.

2. Play: when there are forward movements (push) in the Y-axis and the velocity is greater than a threshold (Same as Comment initiation).

3. Stop: when there are backward movements (pull) in the Y-axis and the velocity is greater than a threshold. This has to be done after comment initiation.

4. Volume up and down: pointing movement move toward the right/left and the velocity is greater than a threshold.

This has to be done after comment initiation.

5. Move to the next song: circling movements, clockwise, longer than 1 sec, and then push.

6. Back to the previous song: circling movements, counter clockwise, longer than 1 sec, and then push.

Experiment result showed that the average time-spending for gesture command is less than 4 seconds, and the recognition rate of all comments is more than 90%.

Figure 5: Application of GPTS for a control of music player

CONCLUSION

In this paper, GPTS was demonstrated to recognize a range of gestures in our 3D video based recognition environment in a distance of 3 meters. The system was designed to interact with a music player. The recognition rate of this scenario is greater tan 90% when user’s head and hands are well tracked. Other intended future work includes (a) a strong recognition engine using recurrent neural network model or hidden Markov model, (b) a more complete elements of gesture will be defined that movements in 3D spaces, moving speeds, and the strength of movement are more meaningful, and (c) interaction with a larger display at a longer distance.

Acknowledgements

This study is supported by a grant from the National Science Council, R.O.C. (Project No. NSC NSC 96-2628-E-011 -007 -MY2). The authors wish to acknowledge this financial support.

References

[1]Comaniciu, D., Ramesh, V., and Meer, P., 2003, Kernel-based object tracking, IEEE Transactions on Pattern Analysis and Machine Intelligence, 25, 564-575.

[2]Kai, N., Seemann, E., and Stiefelhagen, R., 2004, 3D-tracking of head and hands for pointing gesture recognition in a human-robot interaction Scenario, Automatic Face and Gesture Recognition, Proceedings. Sixth IEEE International Conference, 565-570.

[3]Kai, N. and Stiefelhagen, R., 2003, Pointing gesture recognition based on 3D-tracking of face, hands and head orientation, ICMI’03, November, 5-7, 2003, Vancouver, British Columbia, Canada.

[4]Lee, Y.H., Yeh, C.H., and Wu, S.K., 2008, Accuracy measurement of distant goal- directed hand pointing movements, Conference Proceeding of the 1^st East Asia Ergonomics Symposium, Nov. 12 – 14, Kitakyushu, Japan.

[5]McKenna, S. J. and Morrison, K. 2004, A comparison of skin history and trajectory-based representation schemes for the recognition of user-specified gestures, Pattern Recognition 37, 5, 999-1009

[6]Lee, M. S., Weinshall, D., Cohen-Solal, E., Comenarez, A., and Lyons, D., 2001, A Computer Vision System for On-Screen Item Selection by Finger Pointing, in Proc IEEE Conference on Computer Vision and Pattern Recognition, Hawaii, Dec 2001.

[7]Keskin, C., Erkan, A., Akarun , L., 2003, Real time hand tracking and 3D gesture recognition for interactive interfaces using HMM, ICANN 2003

日本人間工學會第五十回紀念大會

開催日：2009 年 6 月 10 日（水）、11 日（木）、12 日（金）

会 場：産業技術総合研究所 つくば中央 李永輝

國立台灣科技大學 工業管理系 (一) 前言

日本人間工學會(Japan Ergonomic Society, JES)成立於 45 年前，由於往年曾經一年 兩次的舉辦學術研討會，因此今年舉行第五十回紀念大會，是一重要里程碑。學會 會長 Susumu Saito, Ph.D. President, Japan Ergonomics Society, Institute for Science of Labour，任職日本勞工部，曾受邀以貴賓身份，來台參加中華民國人因 工程學會(Ergonomic Society of Taiwan, EST)今年(2009)的年會，因此促成本人以 特使之身分，代表 EST 前往參加第五十回紀念大會。除了參加學術研討會外，於 6/10(三)下午，與 JES 的國際事務成員討論 EST 與 JES 的合作模式，另受邀參加 JES 的慶祝晚宴，將 EST 所提供的紀念品在所有 JES 的成員面前送給 Saito 先生，鏈結 兩學會的合作。圖一、二、及三為本人參加晚宴及贈送紀念品的合照。

圖一：大會宣傳海報

圖二：李老師參加日本人間工學會紀念大會並受邀致詞

圖三：由左到右為大會主席赤松幹之先生，李老師，JES 理事長 Saito 先生，與日本

大學教授擔任 JES 國際事務長的 Dr. Yoshinori Horie

(二) 會議內容

JES現有會員約2000人，成員遍及日本各地，相較於台灣的人因學會，JES有更多的 業界、實務界、尤其是職安衛醫師，物理及職能治療師的參與，JES是僅次於美國人 因工程學會次大的組織。大會主題為” 人間工學的過去、現在、與未來”，研討會的 規模相當驚人，論文發表與對談內容亦較實務導向並具本土需求的鏈結性。主題如 下：

1. Aviation Human Factors 2. Ergonomic Design

3. Kansei Information Processing 4. Information and Social Ergonomics 5. Medical Safety

6. Nursing Ergonomics 7. Dental Ergonomics 8. Auditory Ergonomics 9. Clothes

會議的特色之一，JES推出”人間工学の歴史（写真）”展示。該展覽是由學會成 員石松健男(がとらえた)，透過累積四十多年的事件照片，說明日本企業及社會，

自学会設立至今與人因工程相關的歴史證據。照片由黑白到彩色，內容由傳統到現 代，技術由機器到資訊科技，彌足珍貴。

會議的特色之二是”人間工学活用事例展示”。包括感性(人間生活技術戦略)，

人間生活工学研究の活動紹介，ISO/TC159 の活動および人間工学JIS規格，特定 非営利活動法人キッズデザイン協議会，国際ユニヴァーサルデザイン協議会 (IAUD)活動ご紹介，人間工学への二関節筋力学体系の導入，伝統みらい研究セン ターにおける暗黙値の抽出。顯示JES在企業互動、國際鏈結、以及與政府法人的互 動成效。

(三) 與會心得

„ JES的發展路線圖: 會議的最大收穫在於瞭解JES的發展計畫。JES為深化該 組織對於日本社會的貢獻，並進一步為其組織的長遠發展準備，特別依其社 會、企業、及學術研究的需求，提出JES的road map規劃。JES發展路線的規劃，

是源發於企業的road map以及政府的road map。 規劃的基本內容包括：

Introduction, underlying considerations, requirement for ergonomics,

important ergonomics issues, work environments & systems friendly to

people, diversified needs in human lives, safe and comfortable mobility, safe

and healthy environments; and good social communications. 對於落實這些 規劃，在執行面上JES成立了九個工作小組，並以Intentional approaches based on the social & ergonomics needs, Effective organization and cooperative system, Promotion of practical application of ergonomics做為推動未來的工 作(21世紀の人間工学戦略課題)的主軸。

„ JES人因設計能量的展現： JES以產品設計的人因工程運用為主題，在會場 展示人因設計對於產品設計的加值。展示的商品包括兒童傢具、辦公人因工程 座椅、吹風機、刮鬍刀、電鍋….以及各式各樣，以人性因素為出發並具使用性 創意的商品，充分展現人因工程專業對於日本企業的實質貢獻，此一手段亦是 值得EST仿效。

圖四：產品的人因工程設計

„ JES人因工程現場改善個案資料庫： 在作業現場推動現場人因工程改善，

一直是人因工程的核心價值，透過個案的報導、說明、與標竿學習，是放大與 複製現場改善價值的最佳手法。Ergonomic good practices 資料庫在JES不只已 是網站上的重要資產，也翻譯成英文供各界引用，是一非常值得EST學習與合 作的專業資源。

圖四：現場人因工程改善

„ JES 的指引訂定與參與ISO法規制訂： 為協助企業並展現JES的專業能 力，JES推出了改善VDT（visual display terminals）作業疲勞的指引、護理工 作人員改善下背痛指引、兒童讀書與工作場所人因工程指引、不良工作姿勢評 估指引等。此外、JES成員亦積極參與ISO法規制訂，JES成員參與SC1 General Ergonomic Principles、SC3 Anthropometry and Biomechanics、SC4

Ergonomics of Human System Interaction、SC5 Ergonomics of the Physical Environment、WG2 Ergonomics for People with Special Requirements、

Advisory Group for Accessible Design等，其國際化與國際影響力亦值得EST 學習。

(四)結論

„ JES 與EST合作與鏈結：經過與JES重要成員重視與非正式的溝通與討論，提出 以下建議：(1)仿效學會與法國的模式，推動兩年一次互訪的”中日人因工程研 討會”，促動日本與台灣的互動與實質合作的產生，(2)兩學會可就現有的資源，

包括產品設計的人因工程、現場人因工程改善案例的發表、翻譯、推廣，啟動 立即的合作事項，(3)八月在北京IEA2009的會議中，JES與EST應就實質合作項 目達成共識，並承諾實質的行動。

„ 與日本大學與研究單位的合作: 利用此次會議，本人亦有機會與日本相關領域 的學者專家互動，包括日本勞動科學研究所齊藤進博士、產業技術總和研究所 人間福祉醫學研究部長橫井孝志博士、日本大學名譽教授 OHKUBO Takao、

日本大學 Yoshinori Horie 教授、關西大學 Kotani Kentaro 教授，北海道大學 環境人間工學教授橫山真太郎。相信這些專業領域的連結，亦是引領本校與他 校或是機構合作的開端。

(五) 攜回資料名稱

1. JES 日本人間工學會紀念大會手冊及議程表 2. 日本人間工學會紀念大會論文集一份 3. JES road map .ppt file

4. JES人間工学の歴史（写真）.pdf files