Architecture and implementation of a multisensor system for robotic assembly environment

Architecture and Implementation of a Mulltisensor

System

for Robotic Assembly Environment

Yi-Tin

Ho, Li-Chen Fu and Han-Shen Huang

Department of Computer Science and Information Engineering

National Taiwan University, Taipei, Taiwan,

R.O.C.

Abstract

To upgrade the robot intelligence adopted in a

variant task environment, we have t o integrate var- ious sensors f o r environmental information and therefore we have t o develop sensory data process- ing techniques OT t o find a negotiated solution f r o m disparate senso y data. W e propose

a

multisensor

s s t e m architecture f o r the prototype of general con- d u r a t i o n of sensor system desi ning, especially f o r robotic assembly environment. #he utmost objective is t o provide user (assembly system designer) with a n environment without extra consideration of sen- sory module configuration or information p o w .

1

Introduction

ANY specific robotic systems with multiple sensors integrated have been published [9-111. Multisensor systems are surveyed in different areas of application; industrial tasks like material han- dlin part fabrication, military command and con- trolyor battle management, space, target tracking inertial navigation, and remote sensing of coastai waters [7,8]. With the aid of multisensor, robot would be endowed more intelligent t o sense envi- ronment.

Robotic assembly systems are usually controlled by a hierarchical structural scheme with different levels, namely, task planning, subtask decomposi- tion, trajectory planning tra'ectory execution and manipulation, as depicted in big. 1.

lanning level, it performs the task what sequence would be the most efficient (with cost and time consideration) way t o sequentially complete the assembly task. A task command should always be decomposed into some subtask commands for ease of im lementation. Trajector is responsible for the task of planning trajectories for a robot arm. Trajectory execution takes charge of fulfilling trajector plans and monitoring execu- tion processes. The yast level, manipulation, is

$-

ways implemented in a closed-loop with sensor in- formation about the environment and the robot ac- tions.

Modern automatic assembly systems utilize many types of sensors [1,3-51. Various sensors are inte- grated t o a n assembly system for meeting the in- creasing need of automation. The robots employ

At the task

such as assem

f:

ly sequence planning t o determie

planning, following the su

B

task decomposition levee

the sensor as a measuring device; that is, it com- putes the sensor information and acts upon that in- formation about working space [2]. With the hierar- chical structure of the assembly system mentioned above, the functional structure of the inte ration becomes obvious. The sensor s stem shoul

8

meet the needs of different levels in t i e assembly hierar- chy. One must decide how automatic (intelligent) he or she wants the system t o be when selecting an integration mechanism. The hig,her level informa- tion the sensor system can provide, the more intel- ligent the system is.

2

Problem Formulation

This paper aims t o a multisensor system architec- ture which can be adopted in moist of the assembl tasks, and to find general rules for organizing d i f ferent sensors with necessary assembly information. 2.1 Different Phases in a Robotic As-

sembly Process

Robotic assembl process can be divided hierar- chically into three Lvels, namely, task level, trajec- tory level, and mani ulation level. In each level,

a robot performs d i g r e n t kinds of work with the corresponding way of interaction with environment. They may use sensory data t o accomplish more complicated tasks or t o achieve higher-level ma- chine intelligence. Before develo ing the sensor s s process and the necessary environmental informa- tion.

In temporal analysis, the task execution in a

robotic assembly system can be divided into two phases, namely, location/recognition (L.R.) phase

and assembly ( A . ) phase, which are discussed b e

low.

tem, we have t o investigate the f? ow of the assem&;

1. (a) Location/recognition phise

In L.R. phase, the main tasks of are t o ac-

quire as accurate information about the assem- bly work cell as possible. Furthermore, addi- tional information on the task environment is also needed in this phase, such as object iden- tification as well as the number and shapes of the existing obstacles.

1.)

Task

level ;

the so-called pre-processing planning by mak- ing the necessary decisions, including determi- nation of the suitable assembly types and of the robot adions t o be taken for each particu- lar part assembly.

2.) Trajectory level :

In the tra’ectory level, the main task is t o de- termine the robotic motion trajectories so as

t o s eed up, if possible, the robot transference Whig ensuring collision avoidance when in a workspace cluttered with obstacles.

3.) Manipulation level :

In L.R. phase, manipulation means the skills

which should be chosen t o perform the task determined at the task level.

2. (b) Assembly phase

The second phase of robotic assembly task is similar t o the L.R. hase, which is divided into

three levels with diaerent types of jobs. 1.) Task level :

Similar t o the L. R. phase, the task level here is

t o process the highest level decisions, such as determination of the assembly (part mating) strategy

,

and of the assembly sequence, etc.

2.) Trajectory level :

In the assembly hase, trajectory represents phase, especially when it is a rigid arm. Some methods also have t o be employed t o avoid any damage such as collision.

3.) Manipulation level :

The matin o eration is taken in this level. centrate on the insertion operation for peg-in- hole t e assembly, which is the majority in more delicate ro

E

otic motion than in L.R.

There may

t P

e ots of mating types, but we con- assem

isP

y processes.

2.2 Sensory Functions Required in

Each Processing Phases

To rovide helpful information for the operAtions descrited in the previous section, several observa- tions should be made for an assembly process in each processing level.

The extremely critical information of the task level is the object identification In the trajectory level, we have t o carry the part ready for assem- bly t o the mating point (configuration), from which the matin process roceeds. Such mating point is usually setermine8 by soft or compliant contact detected by force monitoring. In the manipulation level of the L.R. phase, both the buffering or grasp-

ing operations need the pose information (location and orientation) of the object. In the A . phase, a

match of the contact pattern between some contact information and the modeled force attern should be accurately reached in the manipuyation level.

After the above discussion, we are now ready t o propose a n architecture of a multisensor system.

3

Multisensor System Architecture

Our multisensor system is hierarchically divided into three levels, the organization level, the coordi-

nation level, and feature extraction level. The or a nization level mainly deals with the lanning tas& and command execution, and the feature extractor in the feature extraction level can be viewed as a complete sensor equipped with various functions.

A simple diagram that shqws .the hierarchical structure of the sensor system is given in Fig. 2.

The organizer, which is the hi hest level in the

instructed by the lanner into a number of eyemen- tar subtasks, ea% is with some prior knowledge, an$ will then pass them as a sequence of input commands t o its successor, namely, the coordana- tor. After execution of the iven commands, the

other information t o the organizer for modifying or

updating the knowledge base.

The coordinator is an intermediate level of mech-

anism serving as a supervisor of the feature extrac- tors between the organizer and the HFE structure.

Essentially, the way the coordinator executes the or-

ganizational commands is through passing message t o the HFE structure and incidentally supervising the progress.

HEF(Hierarchica1 Feature Extraction) Structure is the lowest level in the hierarchical multisensor system, and is the practical interface with the work- ing environment, which is originated from the log- ical sensor structure proposed by Henderson [6]. Here, we keep the main characteristics of the logical sensor structure, namely, the abstraction of sensor devices and equipment with algorithm module, and refine the conceptual development so as t o lead to a feature extractor.

Each feature module in the HFE structure is an independent controller to its succeeding sub-feature extractors. There are command line and message channel connecting the feature module and its suc- cessors hierarchically. The lowest level module (fea- ture extractor) in the HFE structure is the physical sensor device driver, which works with the sim lest t o get an image. The feature extraction modules which are connected hierarchically throu h com- mand lines and message channels will then {e called the HFE (hierarchical feature extractor) structure as a whole.

The benefit of using such structure is that when the coordinator requires a piece of specific informa-

tion it communicates with the ap ro riate feature the lowest level of processing actions. The commu- nication can be at any point in the structure, so

that the whole HFE structure behaves ‘ust like a dedicated module. The HFE structure wkich incor- porates the coordinator is depicted in Fig. 3.

the coordination level supervises a1

P

the data flow

sensor system, is responsible for

B

ecomposin tasks

coordinator may or may not

B

eedback the result or

hardware control, i.e t o switch on, t o switch o

E

,

or

module without concerns about t

R P

e ower level or

3.1 Flexibility in HFE Structure

Here, we take the concept of the logical sensors (Henderson, 1988) t o construct our feature extrac- tion unit (of feature extractor). Each unit contains four parts, the input command, the communication, the processin program, and the selector

part.

They will be descrited below and is depicted in Fig. 4.

1.) Input command : which is used t o actuate the unit. This enabling command ma be a very simple one without any parameter a t t a d e d , or with some instructive parameters like the buffer number t o be selected t o store the processed image data.

2.) Communication : which behaves as a spe- cific 110 port of the feature extraction unit. The way of the communication can be a procedure in- voking with call by reference parameters, or with a mailbox for sending message t o and/or receiving messages from the other unit (remote procedures) in a distributed environment. These parameters t o be passed are determined when the unit is created.

3.) Selector : which is t o select the proper set of inputs. Each feature extraction unit contains two kinds of input, one is the kind of complementary sub-feature input and the other is the kind of com- petitive inputs. General1 the selector is respon- sible for selecting a suitabe input based on some specific criterion (e.g., within certain time bound) from a set of competitive inputs.

4.) Program : which constitutes the main part of the unit. The unit calculates the target feature from the set of inputs selected b the selector follow- in the instructions given by t i e input commands wkch come either from the coordinator or from a

parent feature extraction unit.

3.2 Implementation and Performance We use object-oriented programming t o singular- ize the flexibility and the extensibility of our sensor

s stem. The "overload" characteristic of C++ en- atles the transparency of the invoking of sensing functions. We can use the same command name t o invoke each feature extraction unit. The communi- cation portion of a unit is easily performed b object pointers. A piece of pro ramming codes oJtypica1 ogical sensor followed t i e feature extraction unit (see Fig. 4) is shown below, and the performance is discussed at the end of the section.

class logical-unit {

private data :

character string : sensorname[l5]; number of competitive subfeature : npara; number of complementary subfeature : nsub

/*

to record how many subfeaure input is

connected

*/

subfeature pointers :

logical-unit *p[maxsub-unit];

/*

the linkage to the subfeatures

*/

/*

same actuating name in each feature

public data and function :

virtual actuating function : sensing;

extractor

*/

1;

It's hard t o evaluate the performance of a com- plicated sensor system, but we can investigate this roblem more detailedly by analysis of the influence Factors like response time and accuracy.

A lo ical sensor in the hierarchical structure can be modieled as

xi *i(xil, xi29 ..

.

,

Z i k ) ,

where KPi is the uncertainty model of ith logical sen- sor, and xij is the information provided by the j t h

sub-logical sensor (successor) fcr constituting the result of the ith logical sensor, and

xij =

where the atomic sensor means the lowest level

nodes in the HFE structure ,i.e. the hrqrdware d e vice driver, it can be analyzed more delicately by

if sensor i is a pure algorithm module

if sensor i is an atomic sensor

{

where t is the transducer portion of the mth ph si-

cal sensor, and represents the sensing portion. %'or like

8Cd

chip as the sensing portion, and with

d:

170 as the transducer portion.

The cost of response time can lbe formularized by the cost function

exam le a

CC

6

camera have an intensitive arra

where ti is the time period that si takes t o res ond with a command from the coordinator. The Rnc- tion f i is monotonically increasiing with respect t o

the time period

ti,

and tf is the hard time constraint that bounds the response time. And the average cost can be defined by

where pi represents the probability that the re-

s onse time is equal t o t i . Ancl the total cost of t i e responding time can be formulated as

N

CostT =

C

w ~ E [ c o s ~ ~ ] , i=l

where N is the total number of lo ical sensors, wi is the weighting factor indicating t i e relative impor- tance (or the rate of usage) of the corresponding logical sensor (feature extractor).

Similarly, the sensor uncertainty (inaccuracy) can be modeled in the same way

where

x

;

represents a vector of natural state, z<

represents the estimated vector, Lzj represents the hard error constraint. The avera e and the total cost of sensor uncertainty can be

B

ormulated as

cost?

= E[cost;]

N

CostU = alicostf

k = 1

Feature '

Sveed

The hard constraints in both models represent the bound of respond time and error which would cause failure, for example, a part on a running con- veyor belt would crash down without being grasped in time, or the opened jaws of a manipulator would run into a damage hit with the part if the pose error is too large.

We can use the performance evaluation analysis as a criterion t o evaluate the efficiency of a sensor system, but it's difficult t o tell what kind of archi- tecture for a sensor system is better or worse. It depends on individual application task. But in the general environment of automatic robotic assembl task, the roposed architecture would works we8 and give te! assembly designer confidence to follow with the rules. In our laboratory, several sensing units are implemented by Borland C++ with pro- posed structure. The sensing time is listed below

(unit in seconds).

Average sens- Condition

ing time 4.97 sveed of obiect Orient

I

0.84 = 30.97 "/sec

1 -

Location I I 0.29 Obj. Pose

I

1.15

It can be compared with the time of sensing with non-structured programming of sensor system, whose results are listed at the end of next cha ter. It reveals that the sensing speed is slowed {own by constructing the structure, but the performance is acceptable compared t o the non-structured pro- gramming.

4

Experiment

In this section we present an automatic robotic assembly cell which is set up in our Intelligent Robotic Laboratory.

4.1 System Configuration

The emplo ed equi ment in our assembl cell is de icted in

d g .

5. T i e main piece is the &+axis AfeptOne robot arm with all revolute joints. This robot is mainly res onsible for the assembly p r e cess whose reachabye workspace covers almost the whoie working range of the cell. Referrin to the configuration shown in Fig. 5, the conveyor%elt sit- ting in the middle of the work cell transports the assembly parts into the cell. It is the main trans- ferring mechanism in our system.

Two pairs of optical switches are placed across the conveyor belt as our proximity sensors, which are responsible for monitoring the presence of a part at some location of the conveyor belt, and calculates the speed of the movement of the conveyor belt.

In the works ace there are three CCD cameras connected t o D4-2881 image processor board. They rovide ray-level images with 512*512 ixel reso- Pution o?the scene. One of them is calleaoverhead camera, which is located right above the pairs of

o tical switches in order t o detect the presence of tge incoming part. Another is called side camera, which is located right above the end of the conveyor belt in order t o monitor all events which take place on the conveyor belt. The third one is the eye-in- hand camera which is equipped on the end of the third link of tke AdeptOne arm. The block diagram of the visual operation is depicted in Fig. 6.

A force/torque sensor, Lord FT-75/250, is e ui ped onto the wrist of the AdeptOne arm wlic! is shown in Fig. 7. It detects the force and torque information as a contact sensor, and can an- alyze force and torque in Cartesian coordinate.

The image data and the si nals of the optical switches are connected t o a P6-486 through some hard-wires communication time seems t o dominate the overall computation task in the assembl pro- cess. The AdeptOne robot is under control &om a dedicated robot controller.

4.2 System Control Scheme

The functional architecture of the whole assembly system is similar t o the sensor system, that is de- scribed in the last section. It contains an assembly planner or state monitor, and a task coordinator which instructs the other two execution modules, the sensor system and the robot motion system. The block diagram is shown in Fig. 8.

Similar to the coordinator in a sensor system (see

Fig. 2), the assembly task coordinator mainly deals with how t o interpret the input commands and how to supervise various modules. In the last section, the organizer of the sensor system receives com- mands from the lanner and issues the translated instructions t o tEe lower modules, but there the planner in fact consists of a real assembly planner and a task coordinator, as shown in Fig. 8, in a practical assembly system. The combination of the assembly planner and the task coordinator instructs the sensor system t o fetch the necessary features. After that the task coordinator would evaluate or

judge the !etched features, and then command the robot motion system to complete a process. In Fig. the sensor system may fetch the informa- tion either from the environment or directly from the investigation of the result of robot motion. By the cooperation between the sensor system and the robot controller, the assembl process is completed The starting point of the assembly work is in the task coordinator, wherefrom the data, after infer- ence, flow into the planner, sensor system, and the robot motion system. The task coordinator plays the role of the central supervisor t o negotiate among various modules. We proceed t o describe the de- tails of the process flow in the L.R. phase and the

A . phase introduced in section 2 according t o the processing order.

by this closed-loop control sc

K

eme.

4.2.1 Location/Recognition Phase

In the L.R. phase, we com lete all the work except

the mating part. The rogot motion module will perform dynamic gras ing of the incoming part on the running conveyor gelt, carr the part t o a des- ignated location in order to perform a more precise

grasping, and then finally carr the recisely held

part into the assembly site. Tiese jogs are shown in Fig. 9.

After

determining

the

speed

of

the moving ob-

rt,

the robot arm can track it along the conveyor rasp the object which is with slight friction wit{ t%e belt. The dynamic grasping procedure is depicted in Fig. 10.

Before the tracking o eration, we have t o find the precise 3D location ofthe object center and the orientation of the object. In our experiment, we simply use the 2D area information,. which comes from the overhead camera t o recognize the incom- ing art and the height of this part which can be easi{ got from the knowledge base.

elt and smooth1

SDeed 4.2.2 Assembly Phase

4.943 The parts used in the assembly task alon with the

fixture board are pictured in Fig. 11, whica includes a base a mainbody, a slide bar, and a round peg, which have t o be mated together in a specific order. The flow chart of the assembly process is depicted in Fig. 13, where the thin lines with arrows rep- resent the processing flow and the bold lines with arrows relate the working process with the system supporting modules. For example, the

‘

Take to the mating p o d process is supported by both the sen- sor system module and the robot motion module, and these two modules are coordinated b the task Foordinator. And the flow chart also slows that Take t o the matin point’ is the second step after ‘Carry into assemb

7

y celF.

area, orientation and location Refining scan

Take to the mating point

0.856 4.848

In a static environment, the mating points can be well defined prior to the processing. However, in

a dynamic environment, it needs to employ some

locating techniques with visual or contact sensors. 4.3

Results

The assembly task performed b the 5-axis Adep- t o n e robot described in section z 2 is implemented with a special robot language, System V, on Adep- t o n e supervisor and Borland C++ on PC-486.

The assembly time is related t o robot motion speed under the consideration of robot accuracy and safety. The average and the worst assembly time of three parts is listed below (unit in seconds).

mainbody slide bar

round peg

The total time counts from the instant of part’s appearance on the conve or belt to the instant of completion of the assemb6 task. The numbers pre- fixed by asterisk (*) denote the time obtained for the worst case, but those without the prefixes de- note the average time. In the last row with round peg, (n) means that square searching strategy is applied up to n counts. Clearly, wte can see that it needs about 4.42 seconds for one more trial count.

The sensin time is listed below, and we can com- pare the res& with the one listed in the last sec- tion, which is implemented with structured sensor system architecture. It reveals that the structural- ization and the systemization seem t o cost more (in time) than a simple intelligent intlegrated piece of programming.

5

Conclusion

Sensors incor orated in robotic assembly cells usually suffered gom the problems of nonstructured management which results in lack of reusability and extensibilit

.

We analyze the robotic assembly task into severaf processing levels and propose a multi- sensor system architecture with associated feature templates for related usage in corresponding levels. A structured sensor system with benefit of reusabil- it flexibility or extensibility is introduced and im- p2rnented.

References

[l] John W. Cook, “Applying Sensors to Automatic A 5 sembly Systems”, IEEE tmnsactions on Industry Ap- plications. Vol. 27, No. 2, March/April 1991

[2] Rex Maus and RandallAllsup, “Robotics : A Manager’s Guide”, John Wiley and Sons, Inc. 1986

[3] M. Myrup Andreasen, S. Kahler, T.Lund “Design for Assembly”, IFS Publications, UK 1988

[4] IEEE “Proceedings offnternational Conference on Com- puter Integrated Manufacturing”, Rcnsselaer Polytech- nic Institute, Troy, New York, May 1988

[5] IEEE “Proceedings of International Conference on Computer Integrated Manufacturing”, Rensselaer Poly- technic Institute, Troy, New York, May 1992

[6] Tom Henderson and Esther Shilcrai:, “Logical Sensor Systems”, Joumal of Robotic System, Vol. 1, No. 2,

pp.169-193, 1984

[7] Norbert Roth and Peter Mengel, “Sensor Integration

-

Getting The Whole Picture”, Sensor Review, Vol. 12, No. 1, pp.28-33, 1992

[8] Mohan Manubhai Trivedi, Mongi A. Abidi, Richard 0.

33.471 12.057 Eason, Ralph C. Gonzales, “Develo~ping Robotic Sys-

*35.214 *13.086 tems with Multiple Sensors”, IEEE ’Ifnnsactions on

Systems, Man, and Cykmetics, Vol. 20, No. 6 , pp. 1285-1300, November/December 1990

46.929 24.986

[SI M. A. Abidi, R. C. Gonzales, “The TJse of Multisensor

*47.729 *25.786

14.907 6.425 _{Data for Robotic Applications”, IEE.F 2”sactions on} *32.286(3) *22.600(3) Robotics and Automation, Vol. 6, No. 2, pp.159-177,

*36.229(4) *26.557(4) April 1990

m

Figure 1: Robotic Assembly Control Flow

... ..I?.-..!-

--

.. :

Figure 3: The Coordinator and the HFE structure

t

I

Figure 4. Feature Extraction Unit in HFE Struc-

ture

Figure 5 Configuratlon of the Assembly CeU

Figure 9: Working Situation of a Robot in the

L.R.phase

I

/ I

Figure 11: T h e Assembly Parts and the Fixture

Board