5.1 Conclusion
We have developed an image processing algorithm to track surgical instruments by using natural features. The spiking neural network demonstrates satisfactory detection performance in instrument recognition. The Kalman filter enhances the tracking performance for multiple instruments in the scene. This algorithm detects the instruments by their geometric features and texture. So it will not be affected by the tissue reflection on the metal surfaces and the illumination problem.
Experimental results show that the target kernels can track the instruments in the actual laparoscopic surgery despite of lighting variation and pose change in the surgical images. Furthermore, the kernels can localize and distinguish the surgical instruments in the endoscope images. Through the great amount of data training, the system can achieve robust tracking even the robot move in a wide range. The utility of the buffer zone design has also been verified. It helps to stable the image when surgeons operate the instruments in a specific range of area. They can also use the instruments to guide the endoscope move according to their thinking. The instrument in their hands is not only the tools for treatment but also a mouse to control the pose of endoscope. Since the safety problem is a significant issue in surgery, the control role should be reliable in the practical application. In our design, if the surgeons need to change the type of the instrument during surgery, the instrument will not be detected. In this condition, the robot will keep stationary.
We have implemented an endoscope tracking system that can help the surgeons to concentrate on the operation of the instruments. The recognition rate of the system is sufficient to guide the robot to a proper location stably.
53
5.2 Future Work
Consider that in most case of MIS, surgeons need two instruments at a time for treatment. Currently, the computational speed to detect two instruments is about 8 fps in our experiment. However, some of the surgery would be more complex and have the necessary to use more than two instruments in the same time. To speed up the computation capability is therefore become the significant task in the next work. An improved method is to use parallel processing hardware as the external server for powerful computation, and transmit the recognition result to the robot through the wireless network.
54
References
[1] F. Tendick and M. C. Cavusoglu, “Human-Machine Interfaces for Minimally Invasive Surgery,” in Proc. of IEEE International Conference on Engineering in Medicine and Biology Society, Chicago, America, Oct 30- Nov 5, 1997, pp.2771-2776.
[2] C. A. Grimbergen and J. E.N. Jaspers, “Robotics in Minimally Invasive Surgery,”
in Proc. of IEEE International Conference on Systems, Man and Cybernetics, The Hague, The Netherlands, Oct 10-13, 2004, pp.2486-2491.
[3] K. Kong, J. H. Li, J. M. Li, K. Liang and Y. Gao, “Development of a Robotic Arm for Minimally Invasive Surgery,” in Proc. of IEEE International Conference on Complex Medical Engineering, Harbin, China, May 22-25, 2011, pp.388-393.
[4] P. Lago, C. Lombardi and I. Vallone, “From Laparoscopic Surgery to 3-D Double Console Robot-Assisted Surgery,” in Proc. of IEEE International Conference on Information Technology and Applications in Biomedicine, Corfu, Greece, Nov 3-5, 2010, pp.1-4.
[5] “Wikipedia,” available: http://en.wikipedia.org/wiki/ZEUS_robotic_surgical_sy stem.
[6] “Medsys,” available: http://www.medsys.be/surgical-robots/lapman/technology.
asp.
[7] “Freehand Ltd,” available: http://www.freehandsurgeon.com/
[8] O. Weede, F. Dittrich, H. Worn, B. Jensen, A. Knoll, D. Wilhelm, M. Kranzfelder, A. Schneider and H. Feussner, “Workflow Analysis and Surgical Phase Recognition in Minimally Invasive Surgery,” in Proc. of IEEE International Conference on Robotics and Biomimetics, Guangzhou, China, Dec 11-14, 2012, pp.1068-1074.
[9] F. Nageotte, P. Zanne, C. Doignon and M. de Mathelin “Visual Servoing-Based Endoscopic Path Following for Robot-Assisted Laparoscopic Surgery,” in Proc.
of IEEE/RSJ International Conference on Intelligent Robots and Systems, Beijing, China, Oct. 9-15, 2006, pp. 2364-2369.
[10] L. Bouarfa, O. Akman, A. Schneider, Pieter P. Jonker and J. Dankelman,
“In-vivo Real-time Tracking of Surgical Instruments in Endoscopic Video,”
Minimally Invasive Therapy & Allied Technologies, vol.21, no.3, pp.129-134, 2012.
[11] X . Sun and S . Payandeh, “Toward Development of 3D Surgical Mouse Paradigm,” in Proc. of IEEE/ RSJ International Conference on Intelligent Robots and Systems, San Francisco, USA, Sep. 25-30, 2011, pp. 2096-2101.
[12] S. Speidel, J. Benzko, S. Krappe, G. Sudra, P. Azad, B. Peter, M. Stich, C. Gutt and R.Dillmann, “Automatic Classification of Minimally Invasive Instruments based on Endoscopic Image Sequences,” in Proc. of SPIE on Medical Imaging,
55
vol. 7261, pp. 1-8, 2008.
[13] Y. M. Baek, S.Tanaka, K. Harada, N. Sugita, A. Morita, S. Sora, R. Mochizuki and M. Mitsuishi, “Full State Visual Forceps Tracking under a Microscope using Projective Contour Models,” in Proc. of IEEE International Conference on Robotics and Automation, Minnesota, USA, 2012, pp. 2919-2925.
[14] V. Sa-Ing, S. S. Thongvigitmanee, C. Wilasrusmee and J.Suthakorn, “Object Tracking for Laparoscopic Surgery Using the Adaptive Mean-Shift Kalman Algorithm,” International Journal of Machine Learning and Computing, Vol. 1, No. 5, pp.441-447, 2011.
[15] J. Ryu, J. Choi and H. C. Kim, “Endoscopic Vision Based Tracking of Multiple Surgical Instruments in Robot-assisted Surgery,” International Conference on Control, Automation and Systems, Jeju Island, Korea, 2012, pp. 2195-2198.
[16] S. Voros, E. Orvain, P. Cinquin and J.-A. Long, “Automatic Detection of Instruments in Laparoscopic Images: a First Step Towards High Level Command of Robotized Endoscopic Holders,” Proc. of IEEE/RAS-EMBS International Conference on Biomedical Robotics and Biomechatronics, Pisa, Italy, 2006, pp.
1107– 1112.
[17] S. Voros, G.-P. Haber, J.-F. Menudet, J.-A. Long and P. Cinquin, “ViKY Robotic Scope Holder: Initial Clinical Experience and Preliminary Results Using Instrument Tracking,” IEEE/ASME Transactions on Mechatronics, vol. 15, no. 6, pp. 879-886, 2010.
[18] R. Sznitman, K. Ali, R. Richa, R.H. Taylor, G.D. Hager and P. Fua,
“Data-Driven Visual Tracking in Retinal Microsurgery,” i n P r o c . o f International Conference on Medical image computing and computer assisted intervention, Nice, France, 2012, pp. 568-575.
[19] K. Ali, F. Fleuret, D. Hasler and P. Fua, “A Real-time Deformable Detector,” in Proc. of IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.
34, no. 2, pp. 225-239, 2011. monkey striate cortex," Journal of Physiology, vol. 195, pp. 215-243, 1968.
[24] M. Mishkin and L. G. Ungerleider, "Contribution of striate inputs to the visuospatial functions of parieto-preoccipital cortex in monkeys," Behavioural Brain Research, vol. 6, pp. 57-77, 1982.
[25] A. Delorme and S. Thorpe, “Face Identification Using one Spike per Neuron:
Resistance to Image Degradation,” Neural Networks, Vol. 14, pp. 795-803, 2001.
56
[26] Chou P. Hung, Gabriel Kreiman, Tomaso Poggio, James J. DiCarlo, "Fast Readout of Object Identity from Macaque Inferior Temporal Cortex," Science, Vol. 310, no. 5749, pp. 863-866 Nov., 2005.
[27] “Intuitive Surgical, Inc.” available: http://www.intuitivesurgical.com/products/
instruments/.
[28] G. Welch and G. Bishop. “An Introduction to the Kalman Filter,” Course 8. In Proc. of the ACM SIGGRAPH Conference, Los Angeles, August. 2001.
[29] J.S. Chen, A. Huertas and G. Medioni, “Fast Convolution with Laplacian-of-Gaussian Masks,” in Proc. of IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. PAMI-9, No. 4, pp. 584-590, 1987.
[30] Simona E. Grigorescu, Nicolai Petkov, and Peter Kruizinga, “Comparison of Texture Features Based on Gabor Filters,” in Proc. of IEEE Transactions on Image Processing, Vol. 11, No. 10, pp. 1160-1167, 2002.
[31] A. Delorme J. Gautrais, R. Van Rullen and S. Thorpe, “SpikeNet: A Simulator for Modeling Large Networks of Integrate and Fire Neurons,” Neurocomputing, vol. 26-27, 1999, pp. 989-996.
[32] L. Perrinet, A. Delorme and S. Thorpe, “Network of Integrate-And-Fire Neurons Using Rank Order Coding A: How to Implement Spike Timing Dependant Plasticity,” Neurocomputing, Vol. 38-40, pp. 817-822, 2001.
[33] Kai-Tai Song and Chun-Ju Chen , “Autonomous and Stable Tracking of Endoscope Instrument Tools with Monocular Camera,” in Proc. of IEEE AIM 2012, Kaohsiung, Taiwan, 2012, pp.39-44.
[34] “WeBSurg® IRCAD®,” available: http://www.websurg.com/.
[35] Hsiao-Wei Tang, “Huatuo robot,” available: http://www.huatuorobot.com/.
[36] Hsiao-Wei Tang, “Vesalius Robot,” available: http://www.vesaliusrobot.com/t echnology.