VR-Based Teleoperation for Robot Compliance
Control
CHENG-PENG KUAN and KUU-YOUNG YOUNG
Department of Electrical and Control Engineering, National Chiao-Tung University, Hsinchu, Taiwan
(Received 4 April 2000; in final form 21 July 2000)
Abstract. Robots governed by remote human operators are excellent candidates for work in haz-ardous or uncertain environments such as nuclear plants or outer space. For successful teleoperation, it is important to let the operator feel physically present at the remote site. When the telerobotic system is used to execute compliance tasks in which simultaneous control of both position and force may be demanded and inevitable contact with environments is encountered, information about the interactions between the robot manipulator and the environment are especially crucial for the operator to make proper decisions. This paper proposes a VR-based telerobotic system for such compliance tasks. The proposed system provides both visual and haptic information. A local intelligence con-troller, capable of surface tracking and force regulation, is equipped on the robot manipulator to tackle the time-delay problem usually present in teleoperation and to share control load with the operator. The proposed telerobotic system is developed in a virtual environment due to recent gains in the capabilities and popularity of virtual reality to generate realistic telepresence. Experiments based on the surface-tracking and peg-in-hole compliance tasks demonstrate the effectiveness of the proposed system.
Key words: teleoperation, robot compliance control, virtual reality.
1. Introduction
Teleoperation technologies have been applied to hazardous or uncertain environ-ments such as nuclear plants, outer space, or deep oceans, and also to highly automated systems that are not necessarily hazardous but which demand human intervention for detecting and monitoring abnormalities, such as aviation [26, 29]. These technologies let operators enter remote environments with scales or physical laws much different from those in the normal world. Successful implementation of teleoperation systems demands creation of environments that make the operator feel actually present at the remote site. This generation of telepresence becomes much more challenging, however, when the teleoperation system is used to exe-cute compliance tasks, because simultaneous control of both position and force is demanded in compliance task execution, and because contact with environments is inevitable [15, 19, 24]. Furthermore, due to the transmission time delay and the incompatibility between the manipulative devices used by the operator and
the manipulator in the remote site, the operator usually experiences unnatural and ineffective manipulation in teleoperating a compliance task [1, 16, 17].
One key issue in teleoperation is the coordination and cooperation between the human operator and the manipulator. To let the manipulator be more autonomous in dealing with various tasks and uncertain environments, many systems have been proposed to incorporate more intelligence into the manipulator in the remote site, for instance, the concepts of shared autonomy control, tele-sensor-programming, virtual mechanism, and predictive telemanipulation control [13, 14, 18, 29]. As only limited intelligence has been successfully implemented on the manipulator, human intervention is still crucial at the current stage. Following this idea, we propose to equip on the telerobot a local intelligence controller, which possesses a certain degree of intelligence to tackle the interacting robot manipulator and environment in the remote site autonomously. This local intelligence controller can also join force with the human operator for compliance task execution. As it can respond and adapt to the real environment in a real-time manner, its presence also alleviates the time-delay problem.
Because the proposed telerobotic system is designed to tackle the compliance tasks, proper compliance control strategies should be included for force manage-ment. Among various compliance control strategies, the hybrid control exerts po-sition and force control along different degrees of freedom [28]. The impedance control deals with control of dynamic interactions between manipulators and en-vironments as wholes instead of controlling position and force individually [15]. Extensions from hybrid control and impedance control include hybrid impedance control for dealing with different types of environments [2], the parallel approach to force/position control for tackling conflicting situations when both position and force control are exerted in the same direction [7], generalized impedance control for providing robustness to unknown environmental stiffnesses [20], among oth-ers. The compliance control strategy used in the proposed telerobotic system is basically the type of hybrid control. To deal with the varying shape and unknown stiffness of the environmental surface, we develop the proposed compliance control strategy to be capable of learning.
Based on the discussions above, this paper proposes a VR-based telerobotic system for compliance tasks. As information about the situation at the remote site is crucial for the operator to make proper decisions, the proposed system provides both visual and haptic information [16, 27, 30]. A local intelligence controller is equipped on the robot manipulator to tackle the time-delay problem, deal with the interaction between the robot manipulator and environment, and share control load from the operator. We developed this telerobotic system in a virtual environment for the capabilities and popularity of virtual reality to generate realistic telepres-ence [3, 4, 9, 10]. With the inclusion of visual and haptic interfaces such as head mounted displays (HMD), data gloves, and force-reflection joysticks, virtual real-ity nowadays goes beyond pure graphics and images to emulating realreal-ity, letting operators visualize, manipulate, and interact with objects in the virtual world more
naturally [5, 11, 31, 33]. We also implemented object deformation in the virtual environment to achieve realistic descriptions of the object behaviors in response to the forces induced during the interaction between the robot manipulator and remote objects [12, 25]. The rest of this paper is organized as follows. The proposed VR-based telerobotic system and its implementation are described in Sections 2 and 3, respectively. In Sections 4 and 5, simulations and experiments report the effectiveness of the proposed system. Conclusions are given in Section 6.
2. Proposed VR-Based Telerobotic System
The organization of the proposed VR-based telerobotic system is shown in Fig-ure 1. The main modules in this system are the human operator, the force-reflection joystick, and the VR simulator on the operator site; on the remote site, one has the local intelligence controller, the robot manipulator, and the environment. In Figure 1, from the operator site, the human operator inputs motion command Mh
via the force-reflection joystick, which in turn generates motion command Mj and
sends it to the local intelligence controller on the remote site. In addition to acting as an input device for receiving motion commands from the operator, the force-reflection joystick also provides the haptic feeling Hf to the operator, which is
generated by processing the position and force feedbacks, PR and Fc, induced
when the robot manipulator is interacting with the environment. Except for the haptic feeling from the force-reflection joystick, the operator also receives the vi-sual information Vs from the VR simulator, which generates virtual environments
that emulate the remote real environments, through utilizing PR and Fc. On the
remote site, the local intelligence controller, consisting of the command-mapping module and the dynamic interaction controller, modulates motion command Mj
sent from the operator site into command CR, which in turn is sent to the robot
drive unit for generating torques to move the robot manipulator. To tackle the time delay problem, the local intelligence controller, discussed in Section 2.1, is designed to be able to deal with the interaction between the robot manipulator and the environment autonomously without the intervention of the operator.
In Figure 1, the human operator receives the haptic feeling Hf from the
force-reflection joystick and the visual information Vs from the VR simulator. Through
Hf and Vs, the operator senses the happenings in the remote environment and
makes decisions. As the operator (together with the local intelligence controller) is an important part in the bilateral telemanipulation control loop, how Hf and Vs
are presented to the operator is crucial for a successful teleoperation. To furnish the operator with better haptic feeling, we let the force-reflection joystick generate virtual reflected forces using a VR force-reflection strategy, rather than receiving contact forces measured by the force sensor mounted on the remote robot ma-nipulator directly [1, 6, 17]. The basic idea for this VR force-reflection strategy is to generate the VR reflected force using the estimated stiffness of the remote
Figure 1. System organization of the proposed VR-based telerobotic system.
object derived from the measured position and force data, PRand Fc, described in
Equation (1):
Fr = Ko· (PR− Pcs), (1)
where Fr stands for the generated reflected force, Kothe estimated object stiffness,
and Pcsthe location of the contact surface. In addition to the consideration of time
delay, the use of the estimated stiffness for VR force generation can also avoid the sensitivity problem usually encountered in using the sensed force directly. The least-square linear regression method was used for estimating real-time stiffness through processing a series of continuously measured position and force data. With the real-time stiffness estimation, the method can deal with varying stiffnesses, and consequently unknown environments. Note that because the measured contact force may be larger than the maximal force the human operator can tolerate or larger than the maximal output force the force-reflection joystick can provide, Fr
may have to be scaled down to match the comfortable range of the human operator or the output-force range of the force-reflection joystick.
As for the visual information Vs, we used a simple spring model, similar to that
described in Equation (1), to generate VR object deformation due to the forces induced when the robot manipulator interacts with remote objects. Thus, the oper-ator can also visualize the effect of the interactive force as well as feel it. To speed up the graphical display, we did not implement the entire shape varying during object deformation, but approximated the curved surfaces of the deforming object by linearly varying planes. Note that for the proposed VR force-reflection strategy and the object deformation generation, a more general mass-spring-damper model can also be used at the expense of processing times, described below:
Fr = Mo· ¨PR+ Bo· ˙PR+ Ko· (PR− Pcs), (2)
where Mo, Bo, and Kostand for the mass, damping, and stiffness, respectively.
The issue on how the haptic feeling and visual information provided by the proposed telerobotic system can help the operator in teleoperation has been inves-tigated through a series of experiments about maze passing and contour following,
previously performed in our laboratory [21]. We evaluated the experimental results based on the elapsed time and the contact force during task execution. According to the experimental results, we considered that the visual information did help the operator in task execution by providing VR object deformation in addition to better views and various viewing angles to look at the remote environment. The force reflection also helped the operator in manipulation, and the VR force reflection was more helpful than the direct force feedback, which might lead to large contact forces and even instability in some cases. And visual information was a more important factor than force feedback was, as skillful operators could use visual information alone to achieve good performance. Therefore, we concluded that the haptic feeling and visual information provided are helpful for sensing the remote environment, and thus place the operator in a better position to cooperate with the local intelligence controller for compliance task execution.
2.1. LOCAL INTELLIGENCE CONTROLLER
The local intelligence controller, consisting of the command-mapping module and the dynamic interaction controller, is designed to tackle the interacting robot ma-nipulator and environment autonomously and cooperate with the human operator for compliance task execution. When the local intelligence controller can first let the robot manipulator track the surface of the environment and maintain a desired contact force, it will be very helpful for the operator to perform the subsequent ma-nipulation. This function is implemented in the dynamic interaction controller. In addition, the local intelligence controller should also provide the mapping between the command sent from the operator site and that sent to the robot manipulator. This function is provided by the command-mapping module, with the command mapping function, MP (·), described as
Dj = MP (Mj), (3)
where Mj is the motion command sent from the force-reflection joystick and Dj
is that sent to the dynamic interaction controller, which, after processing, will then be forwarded to the robot drive unit. MP (·) can be as simple as a scaling constant for adjusting the movement resolution or matching the working range of the force-reflection joystick and that of the robot manipulator, or as complicated as a nonlinear function or algorithm specifically designed for certain task [14, 22]. In this paper, we did not focus on the development of MP (·) and let it be just as a scaling constant.
The dynamic interaction controller is designed to execute surface tracking with force regulation. As shown in Figure 1, it receives Dj from the command-mapping
module, which represents the hand movement of the operator and provides the direction for the robot manipulator to follow. With Dj and the position and force
information sent from the sensors equipped on the robot manipulator, the dynamic interaction controller derives CR, which is then sent to the robot manipulator for
Figure 2. Command derivation of the proposed dynamic interaction controller for surface
tracking with force regulation.
execution. For the free space case, CRis simply given as Djplus the sensed current
position PR, because only the movement of the operator needs to be considered.
When in contact, however, CR is derived using the compliance control strategy,
because both surface tracking and force regulation have to be dealt with. The proposed compliance control strategy is basically the type of hybrid control. Yet, to determine a proper CR when the stiffness of the environment and the shape
of the environmental surface are not known in advance, a learning algorithm is developed for force regulation. Compared with previous approaches for on-line estimation and adaptation of uncertain environments [8, 32], the proposed learning algorithm is simple and with fewer parameters for adjustment, as demonstrated in the derivation of CRfor the contact case below.
Referring to Figure 2, CRconsists of Ds for surface tracking, Df for achieving
a desired contact force, and PR, as described in Equation (4):
CR(k+ 1) = Ds(k)+ Df(k)+ PR(k), (4) where CR = [XCR, YCR, ZCR] t, D s = [XDs, YDs, ZDs] t, D f = [XDf, YDf, ZDf] t, PR= [XPR, YPR, ZPR]
t, and k is the time step. D
sand Df are derived from Dj and
the measured contact force Fc =
Fcx, Fcy, Fcz
t
, as follows. In Figure 2, when the robot manipulator moves to contact with the environment from the free space, Dj
is usually not along the direction of the environmental surface due to the imprecise manipulation of the operator. Dj can generally be divided into two portions: one
is along and the other perpendicular to the environmental surface. As shown in Figure 2, Ds is taken just as the projection of Dj on the environmental surface.
However, since the shape of the environmental surface is not known exactly, Dj
cannot be projected onto the surface directly. Instead, the measured contact force
Fc, providing the directional information for the environmental surface, is used for
deriving this projection, as described in Equation (5):
Ds(k)= 1 Fc(k) 2 · F 2 cy+ Fcz2 −Fcx· Fcy −Fcx· Fcz −Fcx· Fcy Fcx2 + Fcz2 −Fcy· Fcz −Fcx· Fcz −Fcy· Fcz Fcx2 + F 2 cy · Dj(k). (5)
On the other hand, Df, perpendicular to Ds, is not the projection of Dj normal
to the environmental surface. The use of Df is to let the measured contact force Fc
approach a desired force Fd, and it is derived using a learning process, as described
in Equation (6):
Df(k)= η · ef(k)·
Fc(k)
Fc(k)
, (6)
where η is a learning rate that determines the convergence speed, and ef(k) (=
Fc(k) − Fd(k) ) stands for the difference between the measured and desired
contact forces. Using Equation (6), Fc will gradually approach Fd via learning.
Thus, with Ds and Df, the dynamic interaction controller achieves surface tracking
with force regulation without knowing the shape of the environmental surface and the stiffness of the environment in advance, as demonstrated in Theorem 1. THEOREM 1. Given command CR derived from Equations (4)–(6), the robot
manipulator will track the environmental surface with the desired contact force according to the human operator’s motion command. Here, the precise shape and stiffness of the environmental surface may not be known in advance.
Proof. Command CR, described in Equation (4), uses Ds for surface tracking
and Df to achieve the desired contact force. We first show that Ds, described in
Equation (5), can lead the robot manipulator to track the environmental surface ac-cording to the operator’s command. Assuming that Dj, representing the operator’s
command, is not parallel to Fc, it is quite straightforward to find
Dst(k)· Dj(k) >0. (7)
Equation (7) means that the angle between Ds and Dj is less than 90◦,
indicat-ing that Ds is following the direction of the operator’s command to move along
the environmental surface. We then show that the measured contact force Fc will
approach the desired contact force Fd using Df, described in Equation (6). By
modeling the environment as a linear spring with stiffness Ke and approximating
the environmental surface by planes defined as aX+bY +cZ+d = 0, the measured contact force Fc(k)at time step k can be expressed as
Fc(k) = Ke(aXPR(k)+ bYPR(k)+ cZPR(k)+ d)
= Ke(PRt(k)· En+ d), (8)
where (a, b, c, d) are parameters varying along with the surface curvature with
En = [a, b, c]t, the normal unit vector to the surface, and stiffness Ke may vary
at different locations of the surface. Here, we let Enpoint to the negative direction
of Fc. Many industrial robot manipulators (including the Mitsubishi RV-M2 type
five-axis robot manipulator used in our proposed telerobotic system) use the point-to-point control, and this kind of control can let the robot manipulator closely approach the position PR specified by command CR. Therefore, letting PR(k+ 1)
equal to CR(k+ 1) and using Equations (4) and (8), the measured contact force
Fc(k+ 1) at time step k + 1 can be expressed as
Fc(k+ 1) = Ke(PRt(k+ 1) · En+ d)
= Ke(CRt(k+ 1) · En+ d)
= Ke(Dst(k)· En+ Dft(k)· En+ PRt(k)· En+ d). (9)
Because Ds is along the environmental surface, Dts(k)· En= 0. Then, with
Equa-tion (6), Fc(k+ 1) in Equation (9) now becomes
Fc(k+ 1) = 0 + Ke· η · ef(k)·
Fct(k)
Fc(k)
· En+ Fc(k) . (10)
In Equation (10), with the learning rate η larger than zero, when ef(k)(= Fc(k) −
Fd(k) ) < 0, Fc(k+ 1) > Fc(k) ; when ef(k) >0, Fc(k+ 1) < Fc(k) .
That means Fc will gradually approach Fd in each learning step with η properly
chosen. The fastest convergence rate occurs when η= 1/Ke. However, as Kemay
not be known in advance, we chose η to be proportional to 1/ Fc(k) instead.
Based on the proof above, we conclude that command CRcan let the robot
manip-ulator track the environmental surface with the desired contact force according to
the operator’s command. ✷
In the proof above, the environment is modeled as a linear spring. However, the real environment may exhibit certain degrees of nonlinearity and the friction also exist. Yet, as aforementioned, the point-to-point control makes the robot manipula-tor closely follow the commands sent from the local intelligence controller. Thus, when the environmental surface was not too rough, the presence of the nonlinearity and friction would not deviate the robot manipulator from the desired trajectory too much. Therefore, Df derived through learning can still keep the robot
manipula-tor in contact with the environment and let Fc approach Fd. The experiments in
Section 5 demonstrate that the local intelligence controller performed quite well in the real environments. In addition, this controller was also successful in tackling environments with curved or discontinuous surfaces.
3. System Implementation
According to the organization shown in Figure 1, we developed the proposed VR-based telerobotic system. Its system implementation is shown in Figure 3(a) and the system view in Figure 3(b). At the operator site, the impulse engine, developed by the Immersion Corporation (U.S.A.), was chosen as the force-reflection joystick. This joystick had five degrees of freedom in motion, with three of them equipped with force reflection, and its maximal output force was about 8.9 N. At the remote site, a Mitsubishi RV-M2 type five-axis robot manipulator equipped with a JR3 force-moment sensor was adopted. The standard specifications of the Mitsubishi
Figure 3. The proposed VR-based telerobotic system: (a) system implementation and (b)
system view.
RV-M2 type robot manipulator are listed in Table I. Two personal computers (PC1 and PC2) with Pentium CPUs were located at the operator and remote site, respec-tively. PC1 was mainly used for developing the VR simulator and communicating with the force-reflection joystick, with PC2 for developing the local intelligence controller and communicating with the RV-M2 drive unit and JR3 force-moment sensor. For implementing the VR simulator, the WorldToolKit (WTK) developed by the Sense8 Corporation (U.S.A.) was used to develop the simulation manager and TrueSpace2 (Caligari Corporation, U.S.A.) was the 3D modeling software.
Table I. Standard specifications of the Mitsubishi RV-M2 type robot manipulator
Item Specification Remaks
Mechanical structure 5 degree-of-freedom robot, vertical articulated robot
Waist rotation 300◦(max. 140◦/sec) J1 axis
Shoulder rotation 130◦(max. 79◦/sec) J2 axis
Operation Elbow rotation 120◦(max. 140◦/sec) J3 axis
range Wrist pitch ±110◦(max. 163◦/sec) J4 axis
Wrist roll ±180◦(max. 223◦/sec) J5 axis
Offset 120 mm
Arm length Upper arm 250 mm
Fore arm 200 mm
Weight capacity Max. 2 kgf (including the hand weight) Maximum path velocity 1500 mm/sec
Position repeatability ±0.1 mm
Drive system Electrical servo drive using DC servo motors
Robot weight Approx. 28 kgf
Motor capacity J1, J2 axes: 60 W; J3 axis: 40W; J4, J5 axes: 23W
As Figure 3(a) shows, the human operator sends in motion command Mh to
move the VR robot manipulator in the virtual environment using the force-reflection joystick; this modifies Mhinto Mj. Via PC1 and then PC2, motion command Mj
is sent to the RV-M2 drive unit which in turn generates torques Tato move the real
robot manipulator. Actual robot positions PR are fed back to the VR simulator
residing at PC1 via the drive unit and PC2 to synchronize the motions of the VR and real robot manipulators. Contact forces Fc induced when the robot
ma-nipulator interacts with remote objects are measured using the JR3 force-moment sensor mounted on the robot manipulator. The measured contact forces Fcare first
processed by the JR3 interface and then sent to the local intelligence controller re-siding at PC2. By using PRand Fc, the local intelligence controller derives Ds and
Df that constitute CR, described in Section 2.1, to perform surface tracking with
force regulation. Fcis also sent to the VR simulator to generate realistic VR object
deformation and to the force-reflection joystick to generate haptic feeling Hf.
4. Simulation
Simulations based on the surface-tracking compliance task, shown in Figure 4(a), were executed to evaluate the performance of the proposed VR-based telerobotic system. In Figure 4(a), the operator manipulated a virtual robot in the virtual en-vironment and let the peg equipped on the end-effector move along the surface of
Figure 4. (a) The surface-tracking compliance task. (b) The peg-in-hole compliance task.
the block while maintaining a desired contact force. The virtual contact force Fc
was generated by the VR simulator according to a preset stiffness and shape of the block. In turn, Fc was used to generate the haptic feeling Hf via the
force-reflection joystick. With Hf and the visual information Vs from the VR simulator,
the operator generated motion commands to maneuver the virtual robot. The effect of friction was ignored in the simulations.
In the first set of simulations, the block was placed vertically such that the nor-mal direction of the contact surface was parallel to the X axis of the coordinate frame of the robot manipulator, i.e., the peg would contact the block in the X
Figure 5. Simulation results for surface tracking using the proposed system (I), a vertical
direction. Results are shown in Figure 5. In Figure 5(a), contact force Fcwas close
to the desired value 10 N with a steady-state error of 0.2 N. This steady-state error was caused by the robot movement resolution (about 0.1 mm). Figure 5(b) shows the position trajectory of the peg which was maneuvered by the operator to move from the free space, make contact with the block, and then slide along the surface of the block in the Y direction. From the results, the human operator successfully cooperated with the local intelligence controller to accomplish this surface-tracking task.
In the second set of simulations, the block was placed aslant such that the nor-mal direction of the contact surface was 45◦relative to the X axis of the coordinate frame of the robot manipulator. Results are shown in Figure 6. In Figure 6(a), the contact force was still very close to 10 N, and also with a small steady-state error. Figure 6(b) shows the position trajectory of the peg, which was maneuvered along an aslant surface. From the results, the local intelligence controller successfully tracked an aslant surface while maintaining the desired contact force. We also performed simulations using different kinds of virtual objects with various kinds of shapes, orientations, and stiffnesses. Similar results as those shown in Figures 5 and 6 were obtained.
5. Experiment
We performed two sets of experiments to demonstrate the effectiveness of the pro-posed VR-based telerobotic system. The first set of experiments, corresponding to the simulations, was for the surface-tracking compliance task. In the experiments, we attached a rectangular sponge of an unknown stiffness on a desk. The operator manipulated the robot manipulator and let a wooden peg move along the surface of the sponge while maintaining a desired contact force. The second set of experi-ments was for the peg-in-hole compliance task, shown in Figure 4(b). For this task, the operator manipulated the robot manipulator to insert a wooden peg into a hole located on a carton. The peg-in-hole compliance task was chosen because it entails many typical situations that may be encountered in compliance task execution, including tracking desired trajectories, maintaining desired contact forces, reduc-ing impact forces in transitionreduc-ing from free to constrained motion, and ensurreduc-ing stability in constrained motion. Since a more delicate manipulation was demanded when the robot manipulator made contact with the environment, the command-mapping module in the local intelligence controller described in Equation (3) was set to be
Dj = 0.5 · Mj. (11)
In contrast, when the peg was in free space, Dj was just set to be equal to Mj.
In the first set of experiments for surface tracking, the desk was first placed vertically such that the normal direction of the contact surface was parallel to the
Figure 6. Simulation results for surface tracking using the proposed system (II), an aslant
Figure 7. Experimental results for surface tracking using the proposed system (I), a vertical
the sponge in the X direction. Results are shown in Figure 7. In Figure 7(a), contact force Fc was close to the desired value 10 N with small varying errors, mainly
resulting from surface ruggedness, the robot movement resolution, and the sensor noise. Figure 7(b) shows the position trajectory of the peg which was maneuvered by the operator to move from the free space, make contact with the sponge, and then slide along the surface of the sponge. This peg position trajectory was not exactly along the Y direction and was with small oscillations because the sponge presented friction, nonlinear stiffnesses, and a not particularly smooth surface. We then rotated the desk by an unknown angle to construct an aslant surface for the next experiment. Results are shown in Figure 8. In Figure 8(a), contact force Fcwas
still very close to 10 N with small varying errors. Figure 8(b) shows the position trajectory of the peg along an aslant surface. These results demonstrate that the proposed VR-based telerobotic system can perform surface tracking with force regulation for environments of unknown stiffnesses and orientations, even in the presence of environmental nonlinearity and friction.
To further investigate the effect of the dynamic interaction controller in the local intelligence controller, we performed a third experiment similar to the first one except that the dynamic interaction controller was not included. Results are shown in Figure 9. In Figure 9(a), the desired contact force Fc could not be maintained;
in Figure 9(b), surface tracking along the sponge could not be achieved. This was because the operator alone needed to take care of both surface tracking and force regulation in the presence of time delay and imprecise human hand movements. During some experiments violent oscillations even occurred; this might lead to system damage when not properly dealt with. These results demonstrate that the proposed dynamic interaction controller can effectively tackle the time-delay prob-lem, perform surface tracking and force regulation, and share control load with the operator.
In the second set of experiments, we applied the proposed system to execute the peg-in-hole compliance task, shown in Figure 4(b). During task execution we first moved the peg toward the carton until it contacted the surface, moved the peg along the surface with a desired contact force of 10 N until the hole was reached, and then inserted the peg into the hole until the peg pressed on the bottom of the hole with a contact force of 10 N. To keep the demonstration simple, the hole was arranged to be located in the Y direction of the initial peg location, the normal to the surface of the carton to be along the Z direction, and the contact between the peg and the hole in the X direction to be avoided when the peg was inserted into the hole. Thus, only force responses in the Y and Z directions and the position trajectory of the peg in the Y –Z plane were recorded. Figures 10(a) and (b) show the force responses in the
Yand Z directions, respectively, and Figure 10(c) the position response in the Y –Z plane. During the first 15 seconds of task execution, the peg made contact with the surface and then slid on the surface along the negative Y direction, and a contact force of 10 N (with small variations) in the Z direction was measured. At this time interval, the force response in the Y direction was close to 0 N. From 15 to 20
Figure 8. Experimental results for surface tracking using the proposed system (II), a slanting
Figure 9. Experimental results for surface tracking using the proposed system without the
dynamic interaction controller in the local intelligence controller: (a) force response and (b) position response in the X–Y plane.
Figure 10. Experimental results for the peg-in-hole compliance task using the proposed
sys-tem: (a) force response in the Y-direction, (b) force response in the Z-direction, and (c) position response in the Y –Z plane.
seconds, the peg was near the hole and started to enter it. During this transition, the contact force in the Z direction decreased while that in the Y direction increased. From 20 to 30 seconds, the peg had been inserting into the hole along the negative
Z direction and made contact with the hole in the Y direction. The contact force in the Y direction increased to around the desired value of 10 N, and that in the Z direction decreased to about 0 N. Finally, from 30 to 40 seconds, the peg reached the bottom of the hole. The contact force in the Z direction gradually increased to around 10 N. Meanwhile, the contact force in the Y direction decreased to close to zero as the operator moved the peg toward the other side of the hole along the negative Y direction to avoid contact. The successful execution of the peg-in-hole compliance task demonstrates that the proposed system is able to tackle those typical situations usually encountered during compliance task execution.
6. Conclusion
In this paper we have proposed a VR-based telerobotic system for compliance tasks. In the proposed system, both visual and haptic feedbacks are provided for the operator to make better manipulation. In addition, a local intelligence con-troller, equipped on the remote site for time-delay consideration, has been devel-oped to tackle the interaction between the robot manipulator and the environment autonomously. Simulations in the virtual environment and experiments based on the surface-tracking and peg-in-hole compliance tasks demonstrate the effective-ness of the proposed VR-based telerobotic system. According to our study, the human operator is better at planning and navigation and the local intelligence con-troller is suitable for command execution and force management. To further apply the proposed system to various kinds of compliance tasks, it is important to find out how the human operator may cooperate with the local intelligence controller according to their specialties and the specific characteristics of the given compli-ance task, and how the intelligence in control be distributed between them. As one of our future works, we will continue working on this issue. Besides, we are also interested in how the visual and haptic information influence system performance from both qualitative and quantitative standpoints, and how they can be properly integrated [21, 23]. In addition, we also plan to investigate the effect of time delay on long-distance teleoperation.
Acknowledgement
This work was supported in part by the National Science Council, Taiwan, under grant NSC 88-2213-E-009-115.
References
1. Anderson, R. J. and Spong, M. W.: Bilateral control of teleoperations with time delay, IEEE
Trans. Automatic Control 34(5) (1989), 494–501.
2. Anderson, R. J. and Spong, M. W.: Hybrid impedance control of robotic manipulators, IEEE
J. Robotics and Automation 4(5) (1988), 549–556.
3. Bomon, D.: International survey: Virtual-environment research, Computer 28(6) (1995), 57– 65.
4. Burdea, G. and Coiffet, P.: Virtual Reality Technology, Wiley, 1994. 5. Burdea, G.: Force and Touch Feedback for Virtual Reality, Wiley, 1996.
6. Chan, T. F., Dubey, R. V., and Everett, S. E.: Variable damping impedance control of a bilateral telerobotic system, IEEE Control Systems Magazine 17(1) (1997), 37–45.
7. Chiaverini, S. and Sciavicco, L.: The parallel approach to force/position control of robotic manipulators, IEEE Trans. Robotics Automat. 9(4) (1993), 361–373.
8. Colbaugh, R., Seraji, H., and Glass, K.: New results in adaptive impedance control of manipulators, in: IEEE Conf. on Decision and Control, 1993, pp. 3410–3415.
9. Ferretti, G., Filippi, S., Maffezzoni, C., Magnani, G., and Rocco, P.: Modular dynamic virtual-reality modeling of robotic systems, IEEE Robotics and Automation Magazine 6(4) (1999), 13–23.
10. Göbel, M.: Projects in VR: Industrial applications of VEs, IEEE Computer Graphics and
Applications 16(1) (1996), 10–13.
11. Goto, A., Inoue, R., Tezuka, T., and Yoshikawa, H.: A Research on tele-operation using virtual reality, in: IEEE Int. Workshop on Robot and Human Communication, 1995, pp. 147–152. 12. Hirota, K. and Hirose, M.: Providing force feedback in virtual environments, IEEE Computer
Graphics and Applications 15(5) (1995), 22–39.
13. Hirzinger, G., Brunner, B., Dietrich, J., and Heindl, J.: Sensor-based space robotics – ROTEX and its telerobotic features, IEEE Trans. Robotics Automat. 9(5) (1993), 649–663.
14. Joly, L. and Andriot, C.: Imposing motion constraints to a force reflecting telerobot through real-time simulation of virtual mechanisms, in: IEEE Int. Conf. on Robotics and Automation, 1995, pp. 357–362.
15. Hogan, N.: Impedance Control: An approach to manipulation: Parts I, II, and III, ASME J.
Dynamic Systems, Measurement, and Control 107 (1985), 1–24.
16. Kazerooni, H. and Her, M. G.: The dynamics and control of a haptic interface device, IEEE
Trans. Robotics Automat. 10(4) (1994), 453–464.
17. Kim, W. S., Hannaford, B., and Bejczy, A. K.: Force-reflection and shared compliant control in operating telemanipulators with time delay, IEEE Trans. Robotics Automat. 8(2) (1992), 176–185.
18. Kotoku, T.: A predictive display with force feedback and its application to remote manipula-tion system with transmission time delay, in: IEEE/RSJ Int. Conf. on Intelligent Robots and
Systems, 1992, pp. 239–246.
19. Kuan, C. P. and Young, K. Y.: Reinforcement learning and robust control for robot compliance tasks, J. Intelligent and Robotic Systems 23 (1998), 165–182.
20. Lee, S. and Lee, H. S.: Intelligent control of robot manipulators interacting with an uncertain environment based on generalized impedance, In: IEEE Int. Symposium on Intelligent Control, 1991, pp. 61–66.
21. Liao, Y. Y., Chou, L. R., Horng, T. J., Luo, Y. Y., Young, K. Y. and Su, S. F., Force reflection and manipulation for a VR-based telerobotic system, accepted by Proceedings of the NSC –
Part A: Physical Science and Engineering, Taiwan.
22. Lloyd, J. E., Beis, J. S., Pai, D. K., and Lowe, D. G.: Programming contact tasks using a reality-based virtual environment integrated with vision, IEEE Trans. Robotics Automat. 15(3) (1999), 423–434.
23. Lumelsky, V.: On human performance in telerobotics, IEEE Trans. Systems, Man, Cybernet. 21(5) (1991), 971–982.
24. Mason, M. T.: Compliance and force control for computer controlled manipulator, IEEE Trans.
Systems, Man, Cybernet. 11(6) (1981), 418–432.
25. McNeely, W. A.: Robotic graphics: A new approach to force feedback for virtual reality, in:
IEEE Virtual Reality Annual Int. Symposium, 1993, pp. 336–341.
26. Mitsuishi, M., Hori, T., and Nagao, T.: Predictive information display for tele-handling/machining system, in: IEEE/RSJ/GI Int. Conf. on Intelligent Robots and Systems, 1994, pp. 260–267.
27. Piantanida, T., Boman, D., and Gille, J.: Human perceptual issues and virtuality, Virtual Reality
Systems 1(1) (1993), 43–52.
28. Raibert, M. H. and Craig, J. J.: Hybrid position/force control of manipulators, ASME J.
Dynamic Systems, Measurement, and Control 102 (1981), 126–133.
29. Sheridan, T. B.: Telerobotics, Automation, and Human Supervisory Control, MIT Press, 1992. 30. Shimoga, K.: A survey of perceptual feedback issues in dextrous telemanipulation, in: IEEE
Virtual Reality Annual Int. Symposium, 1993, pp. 263–279.
31. Yokokohji, Y., Hollis, R. L., and Kanade, T.: What you can see is what you can feel – Develop-ment of a visual/haptic interface to virtual environDevelop-ment, in: IEEE Virtual Reality Annual Int.
Symposium, 1996, pp. 46–53.
32. Yoshikawa, T. and Sudou, A.: Dynamic hybrid position/force control of robot manipulators – on-line estimation of unknown constraints, IEEE Trans. Robotics Automat. 9(2) (1993), 220–226.
33. Yoshikawa, T., Yokokohji, Y., Matsumoto, T., and Zheng, X. Z.: Display of feel for the ma-nipulation of dynamic virtual objects, ASME J. Dynamic Systems, Measurement, and Control 117(4) (1995), 554–558.
