具全自主操作3D物件能力之雙臂服務型機器人

行政院國家科學委員會專題研究計畫 成果報告

具全自主操作 3D 物件能力之雙臂服務型機器人(第 3 年) 研究成果報告(完整版)

計 畫 類 別 ： 個別型

計 畫 編 號 ： NSC 98-2221-E-011-081-MY3

執 行 期 間 ： 100 年 08 月 01 日至 101 年 07 月 31 日 執 行 單 位 ： 國立臺灣科技大學機械工程系

計 畫 主 持 人 ： 林其禹

計畫參與人員： 博士班研究生-兼任助理人員：鄭立傑 博士班研究生-兼任助理人員：黎德興 博士班研究生-兼任助理人員：張友倫

報 告 附 件 ： 出席國際會議研究心得報告及發表論文

公 開 資 訊 ： 本計畫可公開查詢

中 華 民 國 101 年 10 月 31 日

中 文 摘 要 ： 本計劃之目的在研發製作一多功能全自主式及可遠端控制之 新型雙臂工作機器人。該輪型機器人具備雙機器手臂及立體 視覺及感測系統，可以在視角內自動辨識 3D 物件。全自主 式機器人可使用身上配備之一對機器手臂自發式地對工作物 件進行一些預先設定之操作，當脫離預定工作環境時，以可 使用遠端操控技術由遠端機器人操控人員根據機器人身上收 集到之視覺及其他感測訊號，指揮機器人運動或操控機器手 臂動作執行物件操作。本計劃研發重點在(1)發展高效率和高 可靠度電腦視覺演算法，使得機器人得以快速搜尋到隨意擺 置的 3D 工作物件，並進行操作和(2)發展配合多攝影機和可 變化光線軸向等條件下之多機器手臂之遠端遙控技術。計畫 執行後研發出之立體視覺系統可以提供初期物件之座標位置 供機器手臂作立即規劃和行動，接著視覺控制技術將連續導 引機器手臂執行較精準動作。即時視覺引導機器手臂軌跡規 劃和運動控制技術也已成功實現來協助機器人完成全自主式 操作。

中文關鍵詞： 服務型機器人，雙機器手臂，立體視覺

英 文 摘 要 ： This search project aims to build a multi-purpose mobile robot that can either operate autonomously or be manually operated in a remote distance. This wheeled robot will have stereo vision capabilities and ultrasonic sensors with which the robot can identify objects of interests within its viewing angles for operations. The robot will be equipped with two multi-link manipulators so that the robot can perform a number of tasks with its two arms.

Research efforts will be placed on: (1) developing efficient and reliable computer vision algorithms so that the robot can search and find the randomly

located objects (work pieces), and (2) developing new methods of synthesis of telecontrol systems by

multilink manipulators, when there are several

television cameras with changed spatial orientations of their optical axes. Stereo vision algorithm has been successfully developed to provide estimated coordinates of objects at first glance for initial manipulator trajectory planning, and visual servoing techniques can continually guide manipulators to conduct precise operations. Online vision-guided manipulator trajectory planning and instant motion

control techniques have be developed in this research to successfully assist the robot to accomplish the autonomous operations. Remote operated system has been also implemented on this robot so that the robot can provide a reliable performance in a harsh

environment or on untrained operations where autonomy is not possible.

英文關鍵詞： Service Robot, Dual Robot Arms, Stereo Vision

行政院國家科學委員會補助專題研究計畫 行政院國家科學委員會補助專題研究計畫 行政院國家科學委員會補助專題研究計畫

行政院國家科學委員會補助專題研究計畫 ■■■■成果報告成果報告成果報告 成果報告

□

□

□

□期中進度報告期中進度報告期中進度報告期中進度報告

具全自主操作 3D 物件能力之雙臂服務型機器人

Two-armed service robot with autonomous 3D objects handling capabilities

計畫類別：■個別型計畫 □整合型計畫

計畫編號：NSC 98－2221－E－011－081－MY3 執行期間：98 年 8 月 1 日至 101 年 7 月 31 日 執行機構及系所：台灣科技大學機械工程系

計畫主持人：林其禹 共同主持人：

計畫參與人員：計畫參與人員：周柏嘉，鄭立傑，張友倫，黎德興 成果報告類型(依經費核定清單規定繳交)：□精簡報告 ■完整報告

本計畫除繳交成果報告外，另須繳交以下出國心得報告：

□赴國外出差或研習心得報告

□赴大陸地區出差或研習心得報告

■出席國際學術會議心得報告

□國際合作研究計畫國外研究報告

處理方式：除列管計畫及下列情形者外，得立即公開查詢

□涉及專利或其他智慧財產權，□一年□二年後可公開查詢

中 華 民 國 101 年 10 月 25 日

I

目錄 目錄 目錄 目錄

1. 中文摘要 ... 1

2. English Abstract: ... 1

3. 前言 ... 2

4. 研究目的 ... 2

5. 文獻探討 ... 2

6. 研究方法 ... 4

I. Progress of Hardware Design ... 4

II. Robot vision for object recognition ... 13

1. Simple object ... 13

2. Complex object ... 13

3. Stereo Vision Cameras for 3D Imaging ... 14

III. Tele-control ... 15

IV. Object operation parameters definition and parameter-based robot arm autonomous operations . 17 7. 結果與討論 ... 22

8. 參考文獻 ... 23

1

1. 中文摘要

本計劃之目的在研發製作一多功能全自主式及可遠端控制之新型雙臂工作機器人。該輪型機器人具 備雙機器手臂及立體視覺及感測系統，可以在視角內自動辨識 3D 物件。全自主式機器人可使用身上 配備之一對機器手臂自發式地對工作物件進行一些預先設定之操作，當脫離預定工作環境時，以可使 用遠端操控技術由遠端機器人操控人員根據機器人身上收集到之視覺及其他感測訊號，指揮機器人運 動或操控機器手臂動作執行物件操作。本計劃研發重點在(1)發展高效率和高可靠度電腦視覺演算 法，使得機器人得以快速搜尋到隨意擺置的 3D 工作物件，並進行操作和(2)發展配合多攝影機和可變 化光線軸向等條件下之多機器手臂之遠端遙控技術。計畫執行後研發出之立體視覺系統可以提供初期 物件之座標位置供機器手臂作立即規劃和行動，接著視覺控制技術將連續導引機器手臂執行較精準動 作。即時視覺引導機器手臂軌跡規劃和運動控制技術也已成功實現來協助機器人完成全自主式操作。

關鍵字 關鍵字 關鍵字

關鍵字: : : : 服務型機器人，雙機器手臂，立體視覺

2. English Abstract:

This search project aims to build a multi-purpose mobile robot that can either operate autonomously or be manually operated in a remote distance. This wheeled robot will have stereo vision capabilities and ultrasonic sensors with which the robot can identify objects of interests within its viewing angles for operations. The robot will be equipped with two multi-link manipulators so that the robot can perform a number of tasks with its two arms. Research efforts will be placed on: (1) developing efficient and reliable computer vision algorithms so that the robot can search and find the randomly located objects (work pieces), and (2) developing new methods of synthesis of telecontrol systems by multilink manipulators, when there are several television cameras with changed spatial orientations of their optical axes. Stereo vision algorithm has been successfully developed to provide estimated coordinates of objects at first glance for initial

manipulator trajectory planning, and visual servoing techniques can continually guide manipulators to conduct precise operations. Online vision-guided manipulator trajectory planning and instant motion control techniques have be developed in this research to successfully assist the robot to accomplish the autonomous operations. Remote operated system has been also implemented on this robot so that the robot can provide a reliable performance in a harsh environment or on untrained operations where autonomy is not possible.

Keywords: Service Robot, Dual Robot Arms, Stereo Vision

2

3. 前言

本研究計畫目的在使用三年時間研製出ㄧ部全自主式(autonomous)運作的具雙機器手臂及立體視 覺的新式服務型機器人，實現以立體視覺及其他感應訊號快速即時引導機器人移動定位，並配合立體 視覺使用雙手臂進行多種動作的技術與系統整合平台。這套技術與系統平台包含能即時操控機器手臂 作動之快速位置與力量控制系統合成方法，可即時以高動力準確度方式移動手臂終端挾持器(grippers;

end effectors)內工作物件，並對該物件施以一定力量之操作。具多機器手臂系統及立體視覺系統之全自 主式機器人，能夠連續接收環境資訊並考慮結構特徵及動力限制下，進行即時軌跡規劃及最佳運動模 式；本計劃也將發展多攝影機連續於不同位置及角度提供的物件及工作空間資訊下，可對多機器手臂 系統進行遠端半自動遙控之系統合成方法，及使用一般立體攝影機和微處理器條件下，能進行自動辨 識出各工作物件之空間座標及週遭條件，並提供全自主式機器人和多手臂系統運動及軌跡規劃所需參 數之有效自動辨識系統；本計劃也探討同時可對雙機器手臂進行力量和位置控制並執行協同複雜技術 操作之命令系統。

有效整合上述多項創新技術後，可創造可以在許多不同控制環境下全自主式地使用雙機器手臂執 行複雜工作的智慧型機器人。此種新型服務型機器人，具備高度發展潛力，可以進入工廠和家庭，取 代現在只能操作擺放於固定位置的機械手臂，具有相當大的產業價值。

4. 研究目的

本計劃預計研發的機器人必須能在複雜的工作環境中移動定位，並且使用立體視覺和機器手臂，

精確且快速的使用適當力量條件下進行物件操作。這個移動式機器人將有立體視覺能力，讓機器人在 操作過程的視角下可以辨識 3D 目標物件。這個機器人也將有兩個多軸手臂控制器讓這個機器人可以透 過兩支機器手臂從事許多工作。立體視覺公式將會在雙眼擷取影像後提供物件座標作為初始操控規劃 使用，同時連續的視覺伺服機制及技術將會繼續引導操控機器手臂對 3D 物件做精準的定位和操作。以 力回饋為基礎的線上視覺引導操控設計和即時動態控制技術將會在這個研究中被開發來協助機器人完 成自動化工作。在計畫的第三年，也將發展出機器人的遠端控制系統，可讓自動化機器人得以在惡劣 的環境中、或針對事先未定義物件，提供緊急工作指示和操作能力，或是臨時執行一些事先未經訓練 的緊急任務。

5. 文獻探討

目前具雙手臂的輪型移動的研究相當多， ARMAR-III 的底盤有 3 個全向輪採 120 度擺放，其手臂 具有 7 個自由度，5 指手掌則是具有 8 個自由度，在硬體設計上較特別為單一眼睛具有 2 個攝影機，

左右共 4 個，此機器人具有 3D 物件辨識與定位能力，其建立 3D 物件的方式與總計劃所提之概念類似，

透過立體視覺的資訊來判斷實體物件的形狀與位置[1]；2007 年 Grunwald、Schreiber、Albu-Schaffer 與 Hirzinger 學者研製出 DLR 雙機械手臂系統，其設計目的在於減輕機械手臂重量，達到整體機構輕量化，

3

故將位置感測器、扭力感測器、電源供應器等裝置結合於關節桿件中，利用碳纖維取代一般金屬材料 減少整體機構重量，並將其應用於機器人與人類互動之研究[2]；TWENDY-ONE 為早稻田大學所開發 之服務型機器人，底盤採用 4 個全向輪，主要任務為老人照顧與護理，其手臂有 7 個自由度，手臂表 面佈有大量的壓力感測器，手掌為 4 指 13 個自由度，手指表面具有軟性材料，手掌佈滿力量與壓力感 測元件，因此手臂與手掌之受力狀況可以被得知[3]；Riba 機器人為 RI-MAN 機器人的改良，其底盤使 用 4 個全向輪驅動方式，手臂的表面亦具有壓力感測器，最特別的部分為此機器人可以抱起 60 公斤重 之病人，對於老人或是病患造顧相當實用[4] [5]；Wakamura 亦為輪型移動雙臂機器人，手臂僅有 4 個 自由度，其手臂的每一個自由度皆有防夾的設計[6]；Monica 為台灣大學羅仁權教授研究團隊所研發之 機器人，手臂具有 4 個自由度，手掌為 10 個自由度，其手指從雞蛋到原子筆都可輕鬆夾取[7]；TOYOTA Partner 為雙輪自主平衡機器人，其機器人可以自己維持平衡而不傾倒，手臂具有 6 個自由度，手掌為 5 個，其手掌之主要功能為吹奏樂器[8]；EMIEW 2 亦為雙輪自主平衡機器人，不僅可以站立，且還可 以採用跪姿，並且以此種姿態移動，此腿部機構相當特別，也可根據不同的需求而變換[9]。上述八種 機器人多採用全向輪為移動平台，採 3 輪或是 4 輪的擺設，而在手臂設計上也有採用力量或是壓力感 測元件，當手臂受到不正常的負載時，機器人可以根據感測器資訊產生不同的反應，這也是下一代服 務型機器人的必要條件。

人類手掌可以執行的工作相當多，手掌的自由度也不少，若設計出與人類手掌相同自由度之機器 手掌是相當複雜的，且製作成本也過高，因次，適當的簡化自由度是必要的考量，而設計與製作出適 當的機器手掌是需要相當不容易的。Ichiro Kawabuchi 所開發之機器手掌，此手掌相當接近人類手掌，

具有 5 之手指，15 個自由度，手掌可執行的工作相當多樣，亦可使用於手語表達[10]；Gifu Hand III 機器手掌與前者相當類似，具有 5 隻手指，16 個自由度，此機器手掌最特殊之處為具有觸覺感測器覆 於表面，因此當手掌再拿持物體時，可以根據觸覺訊號，來調整手掌的拿取姿勢[11]；Shadow 之致動 器為氣壓缸或直流馬達，此手掌同樣具有 5 隻手指，更驚人的是具有 21 個自由度，可以做出相當多樣 的動作[12]；NTU-Hand IV 為 5 指設計，共有 11 個自由度，且重量相當巧僅有 650 克，其體積與真人 手掌相當[13]。上述這四種機器手掌皆為模仿真人手指數量設計，且自由度也較多，優點為可以做的動 作相當多，且拿持物品較容易，但過多的自由度容易使得手掌的尺寸過大，主要原因為馬達與減速機 體積導致，此外控制電路也相對複雜，因此這樣的設計對移動型機器人來說，並不是一個理想的設計，

移動式的機器人因為體積與耗電量有著一定的限制，因此在手掌自由度上都會朝向簡化的設計，以縮 小手掌的體積與降低耗電量。

如德國 Karlsruhe 大學開發的 ARMAR 雙臂服務型機器人[14]、DESIRE 計畫的整合型機器人[15][16]

為了成為人類生活上的幫手，操作許多生活用品，必須對操作物品有足夠的認識，物件資訊當中，包 含外表的幾何形狀、物件重量、操作種類。不同的操作種類需要不同的操作參數。譬如說機器人要做 倒水的工作，同樣的動作可以對不同形狀的瓶子如進行操作，但是對機器人手臂進行參數化手臂全自 主操作技術就必須要改變一些參數，例如抓取的位置、瓶口的方向等，為了使機器人可操作許多物件，

則必須具備有快速學習新物件的能力。

4

6. 研究方法

I. Progress of Hardware Design

Due to the limited financial support of this project, the content of the robot hardware design is not completely new, but a restructured version of an existing mobile robot. We modified the mechanism to get rid of the existing drawbacks, to ensure the new mobile robot can fit for these requirements of the current project.

錯誤 錯誤 錯誤

錯誤! 找不到參照來源找不到參照來源找不到參照來源找不到參照來源。。。。 is the overview of the mobile robot. There are a total of four driving wheels, two on the each side of the robot. This enables this robot to move forward, backward and turning around.

Furthermore there are two degrees of freedom at the bottom of the chassis. These components can be used to maintain balance when walking on an uneven terrain. This robot has two arms and hands. Left and right arms are the original designs from an existing mobile robot. Both left and right hands are the new version of the design. A stereo vision system is installed on the top of the robot. The head can rotate up and down, also left and right. In addition, a computer and motor drivers are installed inside the Robot’s chassis. All of the motor drives are connected by series communications with RS-232.

Fig. 1 Overview of the armed mobile robot

The detail specification is shown in Table 1. The height of robot is 140 ~ 150 cm. There are totally 34 degrees of freedom. DC servo motors are used to actuate the robot, while harmonic drivers and pulleys are used to reduce the speed. The camera used as the eye is a Logitech webcam 5000. Maximum loading is 650 grams by the hand. There are seven degrees of freedom in each arm. 錯誤錯誤錯誤錯誤! 找不到參照來源找不到參照來源找不到參照來源找不到參照來源。。。 shows the 。 rotating range of each degree of freedom.

Because the gripping ability of the older hand is very limited, it needs to re-design a new one. The degree of freedom of the new robotic hand takes reference to the real human hand. There are 5 degrees of freedom on the new hand as shown in 錯誤錯誤錯誤錯誤! 找不到參照來源找不到參照來源找不到參照來源。找不到參照來源。。。. It’s a simplified model from the humans. DOF-1~2 for the finger’s joint, these joints can change the direction and angle of fingers. DOF-4~5 are for opening and closing three fingers.

5 Table 1 Specification of the robot

Height 140~160 cm

DOF

Head:

Body:

Arm:

Hand:

Chassis:

Total:

2 2 7 ×2

5×2 6 32 Actuators DC Servo Motor Reduction

Mechanism

Harmonic Drive Gear Timing-Belt/Pulley Camera Webcam ×2 Maximum Load 650g (Hand)

Table 2 Moveable range of two arms Movable Range of Arm θ₁ -180° ~ 180°

θ₂ -17° ~ 196°

θ₃ -180° ~ 180°

θ₄ -10° ~ 130°

θ₅ -180° ~ 180°

θ₆ -47° ~ 47°

θ₇ -115° ~ 47°

Fig. 2 Arrangement of 5 DOF of hand

Most important part is the ability of grapping. By clever arrangement of the 5 degrees of freedom of hand, this robotic hand can hold on objects like cube, bottle, cylinder, square column, slice and wand. The maximum load of this robotic hand is 650g.

Cube Bottle-1 Bottle-2 Cylinder-1 Cylinder-2

Cylinder-3 Square column Bar Screw Key

Fig. 3 Simulation results of grasping different objects

6

After the three finger hand design, we designed a new type of five fingers hand. Because of the objects for the robot to grasp are all things in human’s daily, we want the robot hand to best resemble the real human’s hand. Furthermore, we wish that the robot hand size is closer to the real human so that the robot hand can grasp the target items with higher confidence. The simulation results of grasping using the three finger hand can be seen in Fig. 3.

The new design of the five fingers hand has 8 degrees of freedom and 18 joints in total as shown in 錯誤錯誤錯誤! 錯誤 找不到參照來源

找不到參照來源 找不到參照來源

找不到參照來源。。。 and 錯誤。 錯誤錯誤錯誤! 找不到參照來源找不到參照來源找不到參照來源找不到參照來源。。。。. The whole mechanism can be divided into three kinds of mechanisms: fingers bent/straighten mechanism, thumb twisting/bending mechanism and fingers outreach/contract mechanism.

Table 3 Degrees of freedom and the joint definitions DOF Joint DOF Joint

1 J1,1 5 J2,2+J2,3+J2,4

2 J1,2 6 J3,2+J3,3+J3,4

3 J1,3 7 J4,2+J4,3+J4,4

4 J2,1+J4,1+J5,1 8 J5,2+J5,3+J5,4

Fig. 4 Free body diagram

Fingers bent and straighten mechanism

The index finger, middle finger and ring finger share the same mechanism design. 錯誤錯誤錯誤錯誤! 找不到參照來找不到參照來找不到參照來找不到參照來 源

源 源

源。。。。 shows the mechanism design of the index finger. Motor are embedded in a first knuckle and the helical gears are used to turn 90 degrees from the output axis. A coordinated motion of the two sets of four-bar

7

linkage mechanism shown in 錯誤錯誤錯誤! 找不到參照來源錯誤 找不到參照來源找不到參照來源找不到參照來源。。。。, achieves a degree of freedom bending motion of three joint chain.

Fig. 5 Finger mechanism design

Fig. 6 Linkage mechanism diagram

With the same movement principle of the little finger and index finger, the motor is embedded within the first knuckle and uses two sets of four-bar linkage mechanism to drive the chain bending joints. The length and the strength of human little finger are usually smaller than those of the other four fingers, therefore you can use a smaller torque, smaller length and smaller outer diameter, and a smaller DC motor to drive the mechanism. 錯誤錯誤錯誤! 找不到參照來源錯誤 找不到參照來源找不到參照來源。找不到參照來源。。。 shows such little finger mechanism design.

8

Fig. 7 Little finger mechanism design Thumb twisting and bending mechanism

Accounting for the many important activities of the hands, we set each of three joints as an independent degree of freedom, and it needs a larger torque than the other fingers.

And also we use worm gears reducer mechanism to increase the reduction ratio. Since this mechanism has a self-locking function, the worm gears will not reversely rotate when the thumb bents resistant objects.

The motor embedded in the last knuckle uses a helical gear to drive as shown in 錯誤錯誤錯誤錯誤! 找不到參照來源找不到參照來源找不到參照來源找不到參照來源。。。. 。

Fig. 8 Thumb bending mechanism design

The twisting joint design is co-axial to the motor 1, and is fitted with a spur gear. Motor 1 and the bending mechanism rotate on the axis, and avoid interference linkage as shown in 錯誤錯誤錯誤錯誤! 找不到參照來源找不到參照來源找不到參照來源。找不到參照來源。。。.

9

Fig. 9 Thumb twisting mechanism design

Fingers outreach and adduction mechanism

Outreach and contraction mechanisms of the index finger, ring finger and little finger use the seven-bar mechanism to drive. The middle finger due to a small swing angle is fixed in the palm base. The motor with worm gear drives the ring finger linkages with the maximum swing angle14° and the index finger and the little finger the maximum angle 13° and 21°, respectively. The fingers outreach and adduction mechanism is shown in 錯誤錯誤錯誤錯誤! 找不到參照來源找不到參照來源找不到參照來源找不到參照來源。。。. 。

Fig. 10 Fingers outreach and adduction mechanism design

The mechanism design and the real manufactured palm is shown in 錯誤錯誤錯誤錯誤! 找不到參照來源找不到參照來源找不到參照來源找不到參照來源。。。。 and 錯誤錯誤錯誤! 錯誤 找不到參照來源

找不到參照來源 找不到參照來源

找不到參照來源。。。。, respectively. The major structural dimensions comparison is shown in 錯誤錯誤錯誤錯誤! 找不到參找不到參找不到參找不到參 照來源照來源

照來源照來源。。。. 。

10

Fig. 11 The whole robot hand mechanism design

Fig. 12 Real robot hand mechanism assembly

11

Fig. 13 Robot hand mechanism dimensions

After we assemble and install robot hand, we performed some robot hand mechanism actuation test as show in Table 4. And also did the robot hand object handling tests as show in Table 5.

Table 4 Robot hand mechanism actuation test

Motion 1: stone Motion 2: OK motion

Motion 3: number 3 Motion 4: number 5

12

Table 5 Robot hand mechanism crawl and hold the test

Catch the ball Grasping disc Grasping notebook

Grasping a box Grip PET bottle Grip the cup

Holding the phone Grip plier Clip business card

The experiments shows that the mechanical hand can grasp a tennis ball (sphere), a disc (disk), a rectangular box (box), a plastic bottle, a cup (cylinder), a telephone, and a pair of pliers. Also shown in 錯誤錯誤錯誤錯誤!

找不到參照來源 找不到參照來源 找不到參照來源

找不到參照來源。。。。, the robot hand can also pinch the business card between the fingers and thumb clip.

Because we have designed the new robot hand, we make a comparison between the 3 fingers and the 5 fingers hand designs as shown in 錯誤錯誤錯誤! 找不到參照來源錯誤 找不到參照來源找不到參照來源。找不到參照來源。。。.

13

Table 6 Design compare list about robot hand

Design type/item 3 fingers 5 fingers

Finger number Three Five

Degree of freedom 5 DOF 8 DOF

Degree type 3 close and open 2 rotation

6 close and open 1 rotation(Thumb)

1 Stretch Drive way DC & sever motor DC & sever motor Driving mechanism Built-in actuator

Slider and linkage

Built-in actuator Slider and linkage Finger motion Under-actuated Under-actuated

Hand size(cm) 12(L) x 10(W) 18(L) x 16(W)

Maximum load 650g 535g

In order to increase the work space of the hands and the practicality of the robot, we set two degrees of freedom on the body. The first degree of freedom can make the robot stoop, with the two arms degrees of freedom, significantly increasing the working space. The second degree of freedom can change the robot’s height, which is useful and convenient when the robot picks up the object on different height. The chassis was originally designed by Professor Nakajima, the Department of Advanced Robotics, Chiba Institute of Technology [21]. We modified their original design so as to integrate with the upper body mechanisms. There are 2 degrees of freedom at the front and rear chassis’ mechanism. It helps the robot move on uneven terrain as shown in Fig 14.

Fig. 14 (left) Body’s degrees of freedom. (right) The chassis’ degrees of freedom

14

II. Robot vision for object recognition

1. Simple object

For the simple appearance object we defined, by using the color and edge information to detect object localization and orientation. There are three major steps for the recognition. First step is the pre-processing of the input images. In this step, we extract the target color region by using the HSV color space as shown in Fig.

15.

Fig. 15 Input image pre-processing

The second step is matching the edge map with the object’s database. It is shown in Fig. 16. In the step two, we get the correct orientation of the object. For the robot to operate this object, it needs to know the localization in the 3D space. In the third step, by using the stereo vision, calculate the object center in right and left camera.

Fig. 16 Match with the database 2. Complex object

For the complex object recognition, we use the SIFT (Scale Invariant Feature Transform)[22] to extract features from the object’s appearance. SIFT is calculate based on the following four steps.

a. Find scale-space extrema

b. Keypoint localization & filtering. (improve keypoints and throw out bad ones) c. Orientation assignment (remove effects of rotation and scale)

d. Create descriptor (using histograms of orientations) e. Image with features is shown in the Fig. 15

For recognize the specific object, we can build up the database by different viewing angles 2D images. In

15

Fig. 17 (left), consider one rotation axis of images. Once get the input image, calculate the SIFT features, and compare with the database, find the correct orientation in Fig. 17 (right).

Fig. 17 (Left) Different viewing angle of one axis for a cup. (Right) SIFT feature matching.

3. Stereo Vision Cameras for 3D Imaging

Stereo vision works in a similar way to 3D sensing in human vision. It begins with identifying image pixels that correspond to the same point in a physical scene observed by multiple cameras. The 3D position of a point can then be established by triangulation using a ray from each camera. The more corresponding pixels are identified, the more 3D points that can be determined with a single set of images. Correlation stereo methods attempt to obtain correspondences for every pixel in the stereo image, resulting in tens of thousands of 3D values generated with every stereo image [23]-[25] as shown in Fig. 18.

The underlying equation that performs the conversion is:

where Z = distance along the camera Z axis, f = focal length (in pixels), B = baseline (in meters), d = disparity (in pixels). The disparity d measures the displacement of a point between the two images. After Z is

determined, X and Y can be calculated using the usual projective camera equations: , , where u and v are the pixel location in the 2D image, X, Y, Z is the real 3D position.

(Original image) (Points in 3D with color info) (Disparity image) Fig. 18 Color information combined with the depth map

16

III. Tele-control

Tele-control description

Tele-control operation allows the operator to control a mechanical or robot from the safe distance through the wireless communication [17]. In the project, the tele-operation is designed to control a service robot with 2 arms. The robot will be attached 2 webcam to transmit the image to the operator and also received the command from the operator by the wireless. The operator will command directly for the robot by pointing on the screen or type the command for the robot. The system is shown in Fig. 19.

Fig. 19 The model used in the project

In order to create the tele-operation function, the following issues must be addressed

The robot must be programmed to receive the command from the operator through the wireless and the robot will take the role as the client. The operator will take the role as the server and transmit the command for the robot through wireless by pointing on the screen displayed the image transmit from the client. The tele-operation is used in wireless LAN network. The speed through the wireless must be fast enough and stable.

The interface of the server and client are firstly designed based on the targets:

● Both of the server and client must be firstly confirm as logged in before operate the task, by provide the username of each other.

● The server and client communicate with each other by the receive the data region.

● The server order the command for the client by typing on the command region.

In the client the interface is almost the same except the command region.

Some basic operations have been completed in the server and the client The function in server:

● Receive the image and current coordinate from the client.

● Display the image received from client.

● Order command for the client by pointing on the displayed image receive from the client or type in the command region.

The function in Client:

● Receive the image from 2 webcams.

● Transmit the images and current coordinate (x,y) of the detecting object to the server.

The operation of the server and the client has been tested as shown in Fig. 20.

17

Fig. 20 The interface in the server (left) and client (right).

The server have 2 displays region (camera 1 and camera 2) this is the image received from the webcam of the client. After server and client sign in, the client will transmit the string “hi” and transmit the image from the webcam and then waiting the command from the server. Interface is shown in Fig. 21.

The operation step

● Step 1:

Point on the camera 1, the coordinate of the point will be transmit to the client Point on the camera 2, the coordinate of the point will be transmit to the client

● Step 2:

The client after received the coordinate from step 1 will use that factor to track the object pointed Then transmit the current coordinated of the object to the server

Because the transmit in the wireless environment have some delay so we have to concentrate more in the speed of the operation, example the speed to receive the picture and the speed of transmit the data between the server and client. More over the system must be obtained more stable and more reliable when process the task.

Fig. 21 The server interface.

Combine the tele-control with stereo vision I) Approaching the target by stereo-vision

As previous discussion the stereo vision is setup on the mobile platform as shown in Fig. 22. The stereo

18

system comprises 2 cameras. It is setup at the client side. When a person sitting at the server wants to grasp an object, they will point at that object on the screen. By doing the image processing to recognize and localize the object in the Robot vision for object recognition part, the stereo vision will calculate and give the relative coordinate of the object, which is pointed, with the current robot’s position. The red ball is the object pointed on the screen and it is marked by the red ellipse. After the object is marked, the relative position (X, Y, and Z) of the ball to the robot’s position is calculated as the previous stereo algorithm.

Fig. 22 Position of the ball in the camera image

After getting the position of the ball by the stereo vision, Robot will control 2 motors at the platform to approach the ball by reducing the vector error between the position of robot and the position of the ball:

With

X Y Z X Y Z

are the coordinate of the ball and robot. The approaching task will finish when robot enter the working zone of the hand described in the hardware part.

II) Inverse Kinematic for the 7-DOF arm

For the grasping task the inverse kinematic for the 7 -DOF manipulator is derived. By assigning the coordinate as shown in Fig. 23, where the DH table is built, the angle of each joint can be calculated for the grasping task when the position of the object is determined by the stereo vision.

IV. Object operation parameters definition and parameter-based robot arm autonomous operations

Integration of a specific task in one program function is called parameter based function. Parameter based for industrial robot is the program that receives the output data of the image processing part and the data input from an interactive user and commands the task for industrial robot as shown in Fig. 23. By using the parameter based programming, the robot can reproduce or duplicate the same task with another complicated object without re-programming. The data of the image processing are the location (O), the grasping point (A) of the objects, the location of the target task (B) or the position of the circle obstacle (C). The data input from an interactive user are illustrated by the direction of the engagement or other commands.

19

Fig. 23 The data input for the parameter based program.

After received the data input, by solving the inverse kinematic, the robot will grasp the object at A point and according to the direction command of the user, the robot will move the object to B point. On the way to approach the B point. The robot will automatically avoid the obstacle. The trajectory to move the robot to the B point and the method to avoid the obstacle are discussed in the next section.

Image based visual servoing

As discussed above, to overcome the difficulties associated with stereo vision and to improve the precision of robot positioning, many researchers have focused on using eye-in-hand visual servoing to

perform grasping tasks. There are many camera-robot configurations used in visual servoing control as shown in Fig. 24 [26].

Since the image processing noise generated while the calculations for stereo vision are performed, the stereo vision generally involves error. Using the VM1 structure, the location of the object can be precisely adjusted and calculated.The VM1 structure was used for image-based visual servoing in this research.

After the end-effector roughly approaches the object, the orientation of the rectangular object is viewed by the 3rd camera attached to the end-effector, as shown in Fig. 25 (in 2D space).

Fig. 24 (from left to right): VM1 monocular eye-in-hand, VM2 monocular stand-alone, VM3 binocular eye-in-hand, VM4 binocular stand-alone and VM5 redundant camera system [26].

20

The center coordinate in the image space of the rectangle cubic is calculated based on the color filter and the volume filter. The maximum of the

x

y

As describe in [27] and [28], the aim of the image based visual control is to minimize the error by using the interaction matrix L_s which is defined as the following:

2

2

1 0 (1 )

0 1 1

f

f f f f

s

f

f f f f

x x y x y

Z Z

L y

y x y x

Z Z

− 

− +

 

 

= − 

+ − −

 

 

(1)

where ^xf and ^yf are the 2D point coordinates that are projected by a 3D point (X, Y, Z) into the camera frame. In this matrix, Z is the depth of the 3D point. To use this control loop, the depth Z must be estimated at each sampling time. Thus, ^Ls in Eq. 8 cannot be directly used, and an estimation or approximation method must be used.

Moreover, from Fig. 25 it can be realized that, the stereo vision cannot always assure the view from the 3rd camera can always watch the full object or have at least three points for the purpose of controllability as mentioned in [28]. When the system is setup for a real application, some environmental parameters that can affect the system performance need a further consideration. These include the error of the setup device, the error associated with image processing when environments change and especially the error associated with the calibration of the camera and the system. As a result, it is necessary to retune or redesign the controller parameters and to recalibrate the system to effectively cope with the changed environment.

To alleviate the burden of the controller tuning, reduce the time spent in recalibrating the system, and improve the learning process in the presence of uncertain parameters, this study applied artificial intelligence methods to reduce tracking errors.

Fig. 25 Orientation of a rectangle cubic viewed in 3rd camera.

21

The fuzzy controller block consists of three parts: fuzzification, fuzzy inference and defuzzification.

Triangle membership functions were used in this work as shown in Fig. 26. The position error is partitioned into five fuzzy sets and the change rate of position error is partitioned into three fuzzy sets. Finally, the output signal has four fuzzy sets. The range ofepi,ediand u are all [-1, 1] where and are the 2D point coordinates that are projected by a 3D point (X, Y, Z) into the camera frame. In this matrix, Z is the depth of the 3D point.

To use this control loop, the depth Z must be estimated at each sampling time. Thus, in Eq. 8 cannot be directly used, and an estimation or approximation method must be used.

Moreover, it is noted that, the stereo vision cannot always assure the view from the 3rd camera can always watch the full object or have at least three points for the purpose of controllability as mentioned in [11]. When the system is setup for a real application, some environmental parameters that can affect the system performance need a further consideration. These include the error of the setup device, the error associated with image processing when environments change and especially the error associated with the calibration of the camera and the system. As a result, it is necessary to retune or redesign the controller parameters and to recalibrate the system to effectively cope with the changed environment.

To alleviate the burden of the controller tuning, reduce the time spent in recalibrating the system, and improve the learning process in the presence of uncertain parameters, this study applied artificial intelligence methods to reduce tracking errors. An advantage of the fuzzy logic is that the solution to the original problem can be transformed such that a human can understand and design the controller based on the past learning experiences

In this research, the fuzzy controller block consists of three parts: fuzzification, fuzzy inference and defuzzification. Triangle membership functions were used in this work as shown in Fig. 26. The position error is partitioned into five fuzzy sets and the change rate of position error is partitioned into three fuzzy sets.

Finally, the output signal has four fuzzy sets. The range of epi, edi and u are all [-1, 1].

epi

(epi) µ

edi

(e_di)

µ µ( )u

u

(a) (b) (c)

Fig. 26 (a) The membership function of the position error. (b) The membership function of the change rate of the position error. (c) The membership function of the output signal of the fuzzy

One important step in the fuzzy logic controller design is to make a rule table. In this work, 5 fuzzy subsets of and 3 subsets , there exist 15 rules, and therefore, the rule base is represented in a 3x5 matrix, as shown in Table 7.

22

Table 7 Rule table of the fuzzy controller

In the experiment, the algorithm that was applied to command the robot to approach and grasp the object is described in Fig. 27. This method uses a combination of stereo vision and image-based visual servoing with a fuzzy controller. The experimental data demonstrate that the accuracy of the system can be effectively improved using the IBVSFC. The error of the system is small, approximately 1 mm. Using the eye-in-hand structure and IBVSFC will help alleviate the burdens of the high computational cost associated to the system calibration while providing a high success rate and high accuracy when working under weak calibrations.

Moreover, the structure of the system is not complicated; it is easy to set up in complex environments, and it has a lower cost compared with other systems on the market.

Fig. 27 Multitask working flow chart

23

7. 結果與討論

本研究為全自主式機器人/遠端操控機器人，包含機器人運動底盤和雙機器手臂硬體設計和整合、

手臂運動控制技術，遠端操控技術/全自主式運動技術，和視覺及感測訊號為基礎之物件辨識及操作之 多領域整合，透過四種不同領域的技術，來建立一個使用立體視覺即時導引全自主機器人和可由遠端 機器人操控員利用影像資訊即時遙控機器人運動，並使用雙手臂進行物件操作之前瞻技術和系統。

本計畫完成了機器人的設計，機器人為一輪型移動平台，包括雙機器手臂與立體視覺，定義了機 器人的自由度與尺寸。在機器人的硬體設計方面，機器人的高度為 140 公分，全身共有 32 個自由度，

受限於核准經費不足的限制，目前僅完成右手臂的製作，其機器人右手的尺寸不僅與真人相近，且能 拿持多種不同形狀的物體，而左手則使用本實驗室過去已製作的機器手臂。在視覺的技術發展上分成 兩個階段來進行，包含簡單物件與複雜物件兩者。簡單物件為利用顏色的產生的輪廓來進行物體辨識，

透過邊緣的特徵來判斷為何種物體，再進而比對其物體的擺放角度，以利將來機器人夾持；若為表面 紋路複雜物件時，則採用 SIFT 技術做為程式的判斷方式，透過物體表面的圖形特徵來做為判斷的依 據，進而在辨識出物體在空間的角度與位置。在遠端操控技術上，目前已成功完成在區域網路的環境 下，讓使用者可以順利的登入另一台電腦，使用者除了可以傳輸文字指令外，還可以同時接受到兩台 攝影機的影像資訊，由立體視覺提供的物件辨識可以供給遠端機器人的移動與操作。在整合的部分上，

我們使用立體視覺結合模糊控制理論達成物件抓取，本實驗驗證於六個自由度的機械手臂上並且證明 此擺放於任意位置的物件可以被抓取並操作。本計畫以有限的經費，成功完成以多軸機器手臂對 3D 物 件執行全自主操作技術，具高度前瞻實用性。未來將可持續以本基礎開發各式可應用在產業界的 3D 物 件全自主操作技術，以提升整體生產能力和效率。

24

8. 參考文獻

[1] T. Asfour, K. Regenstein, P. Azad, J. Schr¨oder, A. Bierbaum, N. Vahrenkamp and R. Dillmann,

“ARMAR-III: An Integrated Humanoid Platform for Sensory-Motor Control,” IEEE-RAS International Conference on Humanoid Robots, pp. 169-175, 2006

[2] Albu-Schaffer A., Haddadin S., Ott Ch., Stemmer A., Wimbock T. and Hirzinger G., “The DLR lightweight robot：design and control concept for robots in human environments,” Industrial Robot：An International Journal, pp. 376-385, 2007.

[3] H. Iwata and S. Sugano, “Design of human symbiotic robot TWENDY-ONE,” IEEE International Conference on Robotics and Automation, pp. 580-586, 2009.

[4] RIBA機器人網站，http://rtc.nagoya.riken.jp/RIBA/

[5] T. Mukai, M. Onishi, T. Odashima, S. Hirano and Luo Zhiwei, “Development of the Tactile Sensor System of a Human-Interactive Robot “RI-MAN”,” IEEE Transactions on Robotics, vol. 24, pp. 505-512, 2008.

[6] Wakamura 機器人網站，http://www.mhi.co.jp/kobe/wakamaru/english/

[7] 台灣大學羅仁權教授網站，http://www.ee.ntu.edu.tw/profile?id=758

[8] TOYOTA Partner 機器人網站，http://www.toyota.co.jp/en/special/robot/index.html [9] EMIEW 2 機器人網站，http://www.hitachi.co.jp/rd/research/robotics/emiew2_01.html

[10] I. Kawabuchi, “A Designing of Humanoid Robot Hands in Endo skeleton and Exoskeleton Styles,”

Humanoid Robots, New Developments, Book edited by Armando Carlos de Pina Filho, pp. 401-426, 2007.

[11] T. Mouri, H. Kawasaki, K. Yoshikawa, J. Takai, and S. Ito, “Anthropomorphic Robot Hand: Gifu Hand III,” Proc. of Int. Conf. ICCAS2002, pp.1288-1293, 2002.

[12] Shadow 機器人公司網站，http://www.shadowrobot.com/

[13] 台灣大學 黃漢邦教授 機器人實驗室網站，http://robot0.me.ntu.edu.tw/tc/lab_research_BE.htm [14] Azad, P., Asfour, T., and Dillmann, R., “Stereo-based 6D object localization for grasping with humanoid

robot systems”, Intelligent Robots and Systems, 2007. IROS 2007. IEEE/RSJ International Conference on Oct. 29 2007-Nov. 2 2007 Page(s):919 – 924

[15] Grundmann, T., Eidenberger, R., Zoellner, R.D., Zhixing Xue, Ruehl, S., Zoellner, J.M., Dillmann, R., Kuehnle, J., Verl, A., “Integration of 6D Object Localization and Obstacle Detection for Collision Free Robotic Manipulation”. 2008 IEEE/SICE International Symposium on System Integration: SI International 2008 - The 1st Symposium on Systems Integration, art. no. 4770428, pp. 66-71.

[16] Zhixing X., Kasper, A., Zoellner, J.M., and Dillmann, R., “An automatic grasp planning system for service robots”, Advanced Robotics, 2009. ICAR 2009. International Conference on 22-26 June 2009, 1 – 6

[17]http://codeproject.com

[18] Nakajima, S., “Concept of a Novel Four-wheel-type Mobile Robot for Rough Terrain, RT-Mover,”

Intelligent Robots and Systems, pp. 23257–3264.

[19] Nakajima, S., “Development of Four-wheel-type Mobile Robot for Rough Terrain and Verification of Its

25

Fundamental Capability of Moving on Rough Terrain,” International conference on Robotics and Biomimetics, pp. 1968-1973.

[20] Nakajima, S., “Proposal for Step-up Gait of RT-Mover, A Four-Wheel-Type Mobile Robot for Rough Terrain with Simple Leg Mechanism,” International Conference on Robotics and Biomimetics, pp.

351-356.

[21] Nakajima, S., “Kinematics for a rough terrain mobile robot to climb up a step,” Emerging Trends in Mobile Robotics, pp. 825-833

[22] Lowe, David G., “Distinctive image features from scale-invariant key points,” International Journal of Computer Vision 60(2), 2004, 91-110

[23] Richard, “Multiple View Geometry in computer vision”

[24] Nassir Navab , “3D Computer Vision II”

[25] Point Grey Research inc, stereo catalog

[26] D. Kragic and H. I. Christensen, “Survey on visual servoing for manipulation,” Comput. Visi. Act.

Percept. Lab. (CVAP), ISRN KTH/NA/P—02/01—SE CVAP259, Tech. Rep., 2002

[27] I. Siradjiddin, L. Behera, T. M. McGinnity and S. Coleman, “Image based visual servoing of a 7 DOF robot manipulator using a distributed fuzzy proportional controller,” IEEE Int. Conf. Fuzzy Systems, pp.

1-8, 2010.

[28] F. Chaumette and S. HutChinson, “Visual servo control Part 1: Basic approaches,” IEEE Robot. Automat.

Mag., Vol. 13, pp. 82-90, 2006.

國科會補助 國科會補助 國科會補助

國科會補助專題研究計畫項下出席國際學術會議心得報告專題研究計畫項下出席國際學術會議心得報告專題研究計畫項下出席國際學術會議心得報告專題研究計畫項下出席國際學術會議心得報告

日期： 101 年 10 月 29 日

計畫編號 NSC98－2221－E－011－083－MY3

計畫名稱 具全自主操作 3D 物件能力之雙臂服務型機器人 出國人員

姓名 林其禹

服務機 構及職

稱

台灣科技大學機械系教授

會議時間

100 年 9 月 13 日 至

100 年 9 月 15 日

會議地 點

Livingstone, Zambia

會議名稱

(中文) 第十屆 IEEE 第八區 Africon 國際研討會

(英文) The 10

IEEE Region 8 Africon 2011 Conference

發表論文 題目

(中文)以 DSP 為基具立體視覺的的捕手機器人系統

(英文) The DSP Based Catcher Robot System with Stereo Vision

一、參加會議經過

Africon 是一個每兩年 IEEE 在非洲舉辦一次的尖端控制研討會。本屆研討會係由 IEEE

Zambia Section, the IEEE Industrial Electronics Society (IES), 和 the Engineering Institution of Zambia (EIZ)共同主辦。

IEEE AFRICON 2011 主要技術領域包含電機、電子、機械、土木和資訊科技。研討會

論文包含下列更細分的技術領域:

Electromagnetics & Antennas

Communication & Signal Processing

Computational Semiotics

Power Electronics & Devices

Electron Devices & Circuits

Energy & Power Systems

Engineering Applications & Health

Lasers & Electro-Optic Systems

Engineering Management

Education & Technology

Computational Methods & Applications

Control & Automation

Software Engineering

Modeling & Simulation

Information Systems

Engineering Education

Micro-Electro-Mechanical Systems (MEMS)

Renewable Energy Technologies

Asset Management & Terrotechnology

在研討會上午發表了一篇論文，題目是”The DSP Based Catcher Robot System with Stereo Vision”，內容探討使用簡單的設備，包含DSP和一套立體視覺系統，讓一個雙自由度機 器手臂系統得以接住從4m外投來的球。

二、與會心得

本研討會為難得在非洲主辦，以控制為主軸的國際研討會，聚集包含機械、電機、電 子、土木等各領域人員。本人很難得可以在以非洲學者為主的研討會內跟許多人溝 通和交換意見，也爭取到一些參訪新國家的機會，也順便宣傳招收優秀非洲學生到 台灣就讀，功效還算不錯。

三、建議

非洲的大學數目很少，許多優秀學生缺乏進入研究所就讀機會。台灣正在積極招攬外 籍學生到台灣就讀研究所，因此非洲正是一個非常適當的區域。值得台灣各大學前 去開發。

四、攜回資料名稱及內容

大會論文集

Dear Chyi-Yeu,

Thank you for registering for the 2011 IEEE AFRICON, held in Zambia from September 13 - 15, 2011. Below is a summary of your registration. Please review it for accuracy.

Event: 2011 IEEE AFRICON Attending: Chyi-Yeu Lin

Registration Information:

Registration Items

Chyi-Yeu Lin R01 – IEEE Member Additional Information

Chyi-Yeu Lin Paper ID 1:

ND-005061

Confirmation number: KWNMND7V96P

To view or modify your registration, please click the link below. You will be asked to enter the confirmation number shown above.

Click here to Modify your Registration

Click here to view your Registration

NOTE: If you are paying by Wire Transfer, please send an email to [email protected] with "Wire Transfer" in the subject line to get instructions and information about the transfer process.

Sincerely,

2011 IEEE AFRICON

Non discrimination: "IEEE is committed to the principle that all persons shall have equal access to programs, facilities, services, and employment without regard to personal characteristics not related to ability,