4. Simulations
4.2 VR-based predictive display
In this section, the VR-based predictive display together with virtual force reflection is used to provide real-time prediction of slave position and virtual force reflection at the master site. With the assumptions that the models of the slave manipulator and remote environment in the VR simulator well approximated the actual ones, we intend to show that realistic feel of the remote environment can be achieved. Fig. 4.5 shows the simulation results of including VR-based predictive display in the proposed scheme when the same compliance task that moved the slave to contact with a wall was executed. Figs. 4.5(a)-(c) show the master position, predicted position, and predicted force, respectively, and Figs. 4.5(d)-(f) the slave position, contact force, and feedback force response, respectively. In Figs. 4.5(b)-(c), we found that the predicted position was not with lag and the predicted force reflection provided from the VR simulator did not vibrate. Figs. 4.5(d)-(f) show the position of the slave with small lag and the feedback force vibrated a little bit due to varying time delay. By comparing Fig. 4.5(c) with Fig. 4.5(f), we found that the shape of force responses were almost the same under varying time delay, implying that we successfully incorporate the predicted force reflection into the control loop for providing real-time force reflection and did not destabilize the system. Thus, with the VR-based predictive display, the synchronicity in position and force were achieved between the master and virtual environment. In addition, the human operator could observe the movement of the virtual robot and feel the virtual contact force in real-time with no influence from varying time delay. However, we also considered that this method did not really achieve system synchronicity since the slave position and feedback force were still with lag phenomena. The human operator generated the commands depending on the beedback information from the VR simulator. Thus, if
the predicted position and force were not correctly provided by the VR simulator, improper commands would be generated by the human operator to move the remote slave manipulator. Thus, in this system architecture, we must make sure that the models of the slave and environment constructed in the VR simulator approximates the real ones as close as possible, which is certainly a challenge to overcome.
0 1 2 3 4 5
Fig. 4.5 Position and force responses of the master and slave using the VR-based predictive display: (a) position response for the master, (b) position response for the predictive display, (c) force response for the predictive display, (d) position response for the slave, (e) force response for the slave, and (f) feedback force response.
4.3 Transparency
In this section, we analyze system transparency from frequency aspect. We utilize the impedance transfer function described in Eq.(3.31) to analyze transparency.
We consider only the stiffness of the remote environment. The same compliance task described Sec. 4.1-4.2 was performed. The environmental impedance encountered by the slave is formulated as
( ) Ke
where Ke denotes the stiffness of the remote environment (a wall).
Fig. 4.6 shows the simulation results of letting the slave manipulator contact with a hard wall with the stiffness set to be ke =1000N m/ . Figs. 4.6(a)-(b) show the magnitude and phase responses of the impedance transfer function, respectively.
Fig. 4.7 shows the simulation results of letting the slave manipulator contact with a soft wall with the stiffness set to be ke =100N m/ . Figs. 4.7(a)-(b) show the magnitude and phase responses of the impedance transfer function, respectively. The forward and backward delays for the simulations in Figs. 4.6 and 4.7 were both set to be 25 msec.
By examining the magnitude and phase responses shown in Figs. 4.6 and 4.7, we found that complete transparency was not achieved. Nevertheless, the proposed scheme preserved a scaled environmental impedance to the operator during compliance tasks in the presence of time delay. Figs. 4.6 and 4.7 show that the transmitted impedance, Z , and the environment impedance, t Z , have almost linear e and scaled relationship between each other over the bandwidth of human capability.
The abrupt variation of magnitude and phase response of the impedance transfer
function in high frequency was mainly due to the delay effect. In Figs. 4.6 and 4.7, we found that the delay effect did not degrade the system transparency obviously in the bandwidth of human capability. We concluded that the scaled “feel” of the remote environment was preserved.
Fig. 4.6 Magnitude and phase responses of the impedance transfer function when contact with hard wall: (a) magnitude response and (b) phase response.
10-1 100 101 102 103 -20
0 20 40 60
MagnudeB(d)it
10-1 100 101 102 103
-200 -100 0 100 200
frequency(rad/sec) pha
se(d egr ee) (a)
(b)
Fig. 4.7 Magnitude and phase responses of the impedance transfer function when contact with soft wall: (a) magnitude response and (b) phase response.
4.4 Event-based teleoperation system
In this section, simulations were performed to evaluate the performance of the event-based system we have developed in Sec. 3.6. Fig. 4.8 shows the simulation results of the of the event-based teleoperation system in the presence of varying time delay in event domain. Figs. 4.8(a)-(c) show the position and force response and Fig.
4.8(d) the plot of time versus , the event. In this set of simulations, the system position scaling factor was set to be
s
p 1
k = . The control parameters described in Eq.(3.35) were set to be kx =200N m/ and kv =2Ns m/ , respectively. The parameters ε and described in Eq.(3.36) were set to be q ε =0.01 and , respectively. The forward delay was set to vary from 150 to 250 msec and the backward delay vary from 200 to 300 msec.
0.25 q=
In Fig. 4.8, we found that indeed the event-based teleoperation system achieved event-synchronicity in event domain. Besides, the slave followed the master position commands and maintained a proper contact force with the event-based controller. Fig.
4.9 shows the simulation results in time domain (from 20 to 45 secs), which is a zoom-in version of Fig. 4.8. Fig. 4.9(a) shows the master position commands of each event generated from the human operator, and the slave position response in each event in time domain. Fig. 4.9(b) shows the measured contact force. Fig. 4.9(c) shows the feedback force from the slave. In Fig. 4.9, we found that in the event-based system the time-synchronicity was not achieved. The slave position response and the feeback force were still with lag under network transmission. It only achieved event synchronization.
In the event-based teleoperation system, it only concerns about whether the slave
states followed the master commands at each event. The slave states during the intervals between two events were not the main consideration. From Sec. 3.6, it is clear that the frequency of event is a function of time delay (if the round-trip delay is 0.5sec, the frequency of events is 2 events/sec). Since we have an event per round-trip delay, then the frequency by which the force measured and fed back is a function of time delay. This implies that the force is sampled once for each round-trip delay and since this delay is variable then the sampling rate of force is variable. If the time delay is large, the sampling rate of the actual contact force decreases. Thus, the measured force signal reconstructed at the master as force reflection for the human operator would much differ from the actual contact force, yielding unnatural feeling to the human operator and degrading the realism. The event-based control method is extensively used, though. It does not require complicated mathematical knowledge because the chosen of action reference as command cycle is intuitive, simple, and suitable for network transmission environment.
(a)
(b)
(c)
(d)
Fig. 4.8 Position and force responses of the event-based teleoperation system in event domain: (a) position response (black: master, red: slave, yellow: wall), (b) contact force, (c) feedback force, and (d) time versus event.
(a)
(b)
(c)
Fig. 4.9 Position and force responses of the event-based teleoperation system in time domain: (a) position response (black +: master, blue∗ : slave, yellow: wall), (b) Measured force, and (c) feedback force.
Chapter 5
Experiments
The following experiments were performed under real network connection to evaluate the performance of the proposed scheme. The Winsock, one of the network API (Application Programming Interface), was used as the networking communication software. Under the Visual C++ 6 programming environment, we initialized Windows sockets and created the connection between the master and slave site’s computers with Winsock functions. A socket is a communication endpoint.
When both computers start to communicate, each one can use the socket as the interface to connect each other. All of the data flows in the network can be received or sent through these sockets. We first investigated the teleoperation between two
computers connected via the Internet in National Chiao Tung University (NCTU).
The sampling period of master site PC was 0.001 msec. Thus, there were 5000 position commands generated at the local site and send to remote site for task execution within 5 sec. Fig. 5.1 shows the profile of round-trip delay that each master command encountered during task executed between two computers in NCTU.
Fig. 5.2 shows the Internet experimental results. Figs. 5.2(a)-(c) show the master position, predicted position, and predicted force, respectively, and Figs. 5.2(d)-(f) the slave position, contact force, and feedback force, respectively. In Figs. 5.2(b)-(c), we observed that VR has provided real-time predicted position and force reflection information to the operator, which were almost similar to the actual slave positions and contact forces. Fig. 5.2(d) shows that the slave did follow the master trajectory quite well in free motion. In Fig. 5.2(e), the slave maintained a stable contact force in contact motion. From the results, we concluded that under the small varying time delay between the two computers in NCTU, the system performance was satisfactory.
Fig. 5.1 The profile of round-trip delay that each master command encountered between two computers in NCTU.
(a) (d)
(b) (e)
(c) (f)
Fig. 5.2 Internet experimental results in performing the compliance task between two computers in NCTU: (a) position of the master, (b) predicted position in the master site, (c) predicted virtual force in the master site, (d) position of the slave, (e) contact force in the slave site, and (f) feedback force from the slave site.
We then executed the task between the computer in our laboratory and that in National Cheng Kung University (NCKU), which is located at southern part of Taiwan. Fig. 5.3 shows the profile of round-trip delay that each master command encountered during task execution between the two computers in NCTU and NCKU.
Fig. 5.4 shows the Internet simulation results. We found that the system stability was still maintained and performance was not affected by the larger time delay between NCTU and NCKU.
Fig. 5.3 The profile of round-trip delay that each master command encountered between two computers in NCTU and NCKU.
0 1 2 3 4 5
Fig. 5.4 Internet experimental results in performing the compliance task between two computers in NCTU and NCKU: (a) position of the master, (b) predicted position in the master site, (c) predicted virtual force in the master site, (d) position of the slave, (e) contact force in the slave site, and (f) feedback force from the slave site.
In the third set of Internet experiments, we executed the task between the computers in our laboratory and that in Nanotechnology Lab. of UC Irvine, Ca., USA.
Fig. 5.5 shows the profile of round-trip delay that each master command encountered during the task executed between the two computers in Taiwan and USA. Fig. 5.6 shows the Internet experimental results. Figs. 5.6(a)-(c) show the master position, predicted position, and predicted force, respectively, and Figs. 5.6(d)-(f) the slave position, contact force, and feedback force, respectively. In Figs. 5.6(b)-(c), we observed that the real-time predicted position and force reflection at the master site still approximated the actual slave position and contact force. Fig. 5.6(d) shows that the slave followed the master trajectory robustly even under much larger varying time delay than previous ones. In Fig. 5.6(e), the stability of contact force in contact motion did not degrade severely. System stability was still maintained and performance almost not affected by the much larger time delay between Taiwan and USA.
Fig. 5.5 The profile of round-trip delay that each master command encountered between two computers in Taiwan and USA.
0 1 2 3 4 5
Fig. 5.6 Internet experimental results in performing the compliance task between two computers in Taiwan and USA:(a) position of the master, (b) predicted position in the master site, (c) predicted virtual force in the master site, (d) position of the slave, (e) contact force in the slave site, and (f) feedback force from the slave site.
In this set of experiments, the laparoscopic impulse engine produced by Immersion Corporation, shown in Fig. 5.7, was used for the user to generate position commands and receive feedback reflection forces. The impulse engine is a 5 DOF joystick designed specifically for force reflection in the VR MIS (Minimally Invasive Surgery) simulation. Fig. 5.8 shows the experimental setup. The 3-D VR scene including a VR mobile manipulator (This manipulator was constructed by the advanced home robot research group in NCTU), was used for the experiment. When the human operator manipulated the laparoscopic impulse engine, the operation commands would be sent into the local site PC to generate position commands. Then, the local site PC displayed the predicted robot position in the 3-D VR scene on the monitor to provide real-time visual information and sent the position commands to the remote site via the Internet. The real-time predicted force reflection also generated by local site PC at the same time. Master site impedance controller was used to prevent too large force reflection from damaging the impulse engine. Another 3-D VR scene was located at the slave site to represent the actual slave robot and environment.
When the remote PC receives delayed commands form the master site, it would display the controlled robot position on the slave site monitor and calculate the contact force. The Sliding-mode-impedance controller in the slave site was used to perform path tracking and force stability maintenance. We performed the same compliance task as that in the simulations and Internet experiments that moved remote virtual mobile manipulator to contact the virtual wall in 3-D VR scene between two computers in NCTU. The experimental results are shown in Fig. 5.9. Fig. 5.9(a) shows the master position commands generated from the local site PC according to the operation commands we applied on the impulse engine. Figs. 5.9(b)-(c) show the calculated predicted position and contact force of the remote slave system, respectively. The predicted force vibrated a little bit as shown in Fig. 5.9(c), mainly
due to the path resolution of the position commands. When we increased the commanded path resolution, the robot would move very slowly because the frame rate for the display of VR scene was around 60 to 70 Hz. Figs. 5.9(d)-(e) show the controlled slave position and force response in the VR scene according to the delay commands from the master. We found that the slave did follow the master, and the force response of the slave was very similar to the predicted force, as shown in Fig.
5.9(c). Fig. 5.9(f) shows the round-trip time delay measured during the experiments.
From the experimental results, the system stability was achieved and the performance was satisfactory, in the presence of small experimental error, demonstrating the feasibility and effectiveness of the proposed bilateral control scheme.
Fig. 5.7 The laparoscopic impulse engine.
Fig. 5.8 The experimental setup.
(a) (d)
(b) (e)
(c) (f)
Fig. 5.9 Experimental results under the experimental setup as described in Fig. 5.8: (a) position of the master, (b) predicted position in the master site, (c) predicted virtual force in the master site, (d) position of the slave, (e) contact force in the slave site, and (f) the round-trip delay.
Chapter 6
Conclusion and Future work
In this thesis, we have developed a bilateral control scheme based on the sliding-mode-impedance controller for the networked VR-based telerobotic system. It is much simplified when compared with our previous work, thus making system stability and transparency analyses more meaningful. We then analyzed system transparency after system stability has been guaranteed. The system has preserved scaled environmental impedance within the bandwidth of human capability. We have also developed an event-synchronized teleoperation system for performance comparison. Simulation and experimental results have shown that the proposed scheme can achieve system stability under varying time delay. And, the system has exhibited much smoother motion and more natural force response when compared with the event-based system.
6.1 Future Work
To enhance this bilateral telerobotic system, some future works are suggested below:
1. The use of Internet to support the communications between the operator and the robot is quite attractive due to its worldwide availability for a telerobotic system, although time delay present in the network stands as the obstacle to achieve system synchronicity. Event-based control is a very popular method to eliminate the asynchronous phenomenon invoked by time delay. However, the event-based system exhibits unsmooth behavior and unnatural force response, which is not preferred in a telerobotic system. In our works, we have used the VR-based predictive display technique to provide predicted position and virtual contact force of the slave for the human operator in a real-time manner. However, the remote situation remains untackled. In others words, system synchronicity is still not achieved actually. As a future work, we will continue to develop a stable, transparent, and synchronous teleoperation system, which is certainly very challenging.
2. In this thesis, the experiments are basically performed in the simulated environment. In future, the physical system should be implemented to make the experimental results more convincing.
3. For real-time consideration, we should also develop more efficient network transmission method that is robust and fast in data transmission.
Bibliography
[1] N. Ando, J.-H. Lee, and H. Hashimto, “A Study on Influence of Time Delay in Teleoperation,” IEEE/ASME International Conference on Advanced Intelligent Mechatronics, pp. 317-322, 1999.
[2] R. J. Anderson and M. W. Spong, “Bilateral Control of Teleoperators with Time Delay,” IEEE Trans. on Automation Control, Vol. 34, No. 5, pp. 494-501, 1989.
[3] R. J. Anderson and M. W. Spong, “Bilateral Control of Teleoperators with Time Delay,” IEEE Trans. on Robotics and Automation, Vol. 13, No. 5, pp. 494-501, 1989.
[4] D. Barney and T. Ken, “Distributed Robotics over the Internet,” IEEE Robotics &
Automation Magazine, Vol. 7, No. 5, pp. 22-27, 2000.
[5] H. C. Cho, J. H. Park, K. Kim, and J. O. Park, “Sliding Mode-Based Impedance Controller for Bilateral Teleoperation under Varying Time-Delay,” IEEE Int. Conf.
on Robotics and Automation, pp. 1025-1030, 2001.
[6] I. Elhajj, N. Xi, and Y. H. Liu, “Real-Time Control of Internet Based Teleoperation
with Force Reflection,” IEEE Int. Conf. on Robotics and Automation, pp. 3284-3289, 2000.
[7] I. Elhajj, N. Xi, W. K. Fung, and Y. H. Liu, “Haptic Information in Internet-Based Teleoperation,” IEEE/ASME Trans. on Mechatronics, Vol. 6, No. 3, pp. 295-303, 2001.
[8] I. Elhajj, N. Xi, B. Song, M. M. Yu, W. T. Lo, and Y. H. Liu, “Transparency and Synchronization in Supermedia Enhanced Internet-Based Teleoperation,” IEEE Int. Conf. on Robotics and Automation, pp. 2713-2718, 2002.
[9] I. Elhajj, N. Xi, W. K. Fung, Y. H. Liu, Y. Hasegawa, and T. Fukuda,
“Supermedia-Enhanced Internet-Based Telerobotics,” IEEE Trans. on Robotics and Automation, Vol. 91, No. 3, pp. 396-421, 2003.
[10] A. Goto, R. Inoue, T. Tezuka, and H. Yoshikawa, “A Research on Tele-operation Using Virtual Reality,” IEEE Int. Workshop on Robot and Human Communication, pp. 147-152, 1995.
[11] W. Gao, Y. Wang, and A. Homaifa, “Discrete-Time Variable Structure Control
Systems”, IEEE Transactions on Industrial, vol. 42, No. 2, pp. 117-122, 1995.
[12] K. Hirota and M. Hirose, “Providing Force Feedback in Virtual Environment,”
IEEE Computer Graphics and Applications, Vol. 15, No. 5, pp. 22-30, 1995.
[13] K. Kosuge, H. Murayama, and K. Takeo, “Bilateral Feedback Control of Telemanipulators via Computer Network,” IEEE Int. Conf. on Robotics and Automation, pp. 1380-1385, 1996.
[14] K. Kosuge and H. Murayma, “Bilateral Feedback Control of Telemanipulator via Computer Network in Discrete Time Domain,” IEEE Int. Conf. on Robotics and Automation, pp. 2219-2224, 1997.
[15] D. A. Lawence, “Stability and Transparency in Bilateral Teleoperation,” IEEE Trans. on Robotics and Automation, Vol. 9, No. 5, pp. 624-637, 1993.
[16] W. Leland, M. Taqqu,W. Willinger, and D. Wilson, “On the Self-Similar Nature of Ethernet Traffic,” IEEE/ACM Trans. on Networking, Vol. 2, No. 1, pp. 1-15, 1994.
[17] Y. Liu, C. Chen, and M. Meng, “A Study on the Teleoperation of Robot Systems via WWW,” IEEE Canadian Conference on Electrical and Computer Engineering, pp. 836-840, 2000.
[18] G. Niemeyer, J. Jacques, and E. Slotine, “Towards Force-Reflecting Teleoperation over the Internet,” IEEE Int. Conf. on Robotics and Automation,
[18] G. Niemeyer, J. Jacques, and E. Slotine, “Towards Force-Reflecting Teleoperation over the Internet,” IEEE Int. Conf. on Robotics and Automation,