Virtual reality orthopedic surgery simulator
Ming-Dar Tsaia, Ming-Shium Hsiehb;∗, Shyan-Bin JouaaInstitute of Information and Computer Engineering, Chung Yuan Christian University, Chung-Li, Taiwan bDepartment of Orthopaedics and Traumatology, Taipei Medical University Hospital, Taipei Medical University,
252 Wu Hsing Street, Taipei, 11031 Taiwan
Received 18 September 2000; received in revised form 12 April 2001; accepted 12 April 2001 Abstract
This paper describes a highly interactive virtual reality orthopedic surgery simulator. The simulator allows surgeons to use various surgical instruments to operate on virtual rigid anatomic structures, such bones, pros-theses and bone grafts, to simulate every procedure on the rigid structures for complex orthopedic surgeries, including arthroplasty, corrective or open osteotomy, open reduction of fractures and amputation. A com-parative study of the simulator with paper simulation was performed and showed that interns and residents found the simulator to be a useful learning tool, and that visiting doctors could use it e0ectively for planning veri1cation and rehearsal of operations. c 2001 Elsevier Science Ltd. All rights reserved.
Keywords: Volume visualization; Volume manipulation; Orthopedic surgical simulation; Preoperative rehearsal and training; Arthroplasty
1. Introduction
Orthopedic surgeries usually involve complex geometry and require awareness of topology changes in skeletal morphology. Current training methods for interns and residents in teaching hospitals do not adequately raise spatial perception about geometry and topology changes of skeletal morphology. Two reasons for the inability are trainees can only observe an operation before he participates in a surgery and preoperative rehearsal usually involves 2D (two dimensional) paper surgical simulations based on X-ray images. Moreover, orthopedic visiting doctors may also fail in real operations even after evaluating topology and geometry changes on rigid bones, prostheses and bone grafts of all procedures by 2D paper simulations. The failure rates were reported as 10–20% for high tibia osteotomy [1–3] and 5–15% for anterior fusion of the spine [4–9]. These failures, including poor anatomic section lines, poor contact surfaces, inappropriate size and shape of bone graft and improper
∗Corresponding author. Tel.: +886-2-2737-2181(3118); fax: +886-2-2737-5618.
E-mail addresses: tsai@ice.cycu.edu.tw (M.-D. Tsai), shiemin@mail.tmc.edu.tw (M.-S. Hsieh), jou@earth.ice.cycu.edu.tw (S.-B. Jou).
0010-4825/01/$ - see front matter c 2001 Elsevier Science Ltd. All rights reserved.
reduction position, are caused by insuGcient spatial data from the 2D paper simulations. (Soft tissues on healing and open-up are usually not evaluated before a surgery because surgeons can easily open and close up soft tissues and modify soft tissues to obtain satisfactory shapes during the surgery [2,8].)
The application of virtual reality (VR) to surgical training gives a more realistic human ma-chine interaction than traditional 2D simulations and has already become a useful surgical planning and training tool. Several VR surgical simulators have been developed which provide detailed information regarding simulated tissues, tools and actions of surgeons [10]. Simulation systems usually provide a virtual environment by rendering a surface model that may be recon-structed from video data (for simulating endoscopic or laparoscopic surgery, e.g. Satava [11]), X-ray projections (for bone surgery, e.g. Caponetti [12]), transversal tomographic slices (for arthroscopic or oral implant surgery [13,14]), or synthetic surfaces (for ophthalmic surgery [15]). However, the surface model is diGcult to employ for computing topology changes on anatomic structures because no interior information is available. In contrast to surface models, a volume (stack of parallel 2D grayscale images) model represents a body via regularly positioned cuboids (voxels) and is suitable to convey relations between adjacent tissues or structures with a high accuracy (with resolution limits but without projection errors [16]), and to simulate surgeries with topol-ogy changes [17]. Volume manipulation algorithms can easily simulate topoltopol-ogy changes of struc-tures such as structure removal, section and fusion (local manipulations) [17,18], however, are still insuGcient in simulating global manipulations such as a structure reposition and distant fusion, and in identifying a new-sectioned structure. These manipulations are also necessary in orthopedic surgeries.
This paper describes an (VR) orthopedic surgery simulation system based on volume data. The system can compute geometric and topologic changes on bones, prostheses and bone grafts of every orthopedic procedure and provide stereographic images (a pair of 3D images for two eyes) of the simulation results. Through the 3D visual input and output environment, the stereographic images allow surgeons to obtain a spatial perception of every procedure. We have been using this system in preoperative meetings to verify the surgical modality and to simultaneously train and educate residents and interns in our orthopedic department. This paper is structured into several parts. Section 2 intro-duces current volume-based orthopedic surgical simulation methods and describes their limitations. Section 3 introduces our methods and the structure of our prototype system. Section 4 introduces the kinds of orthopedic surgeries that our system can simulate and shows some simulation examples. Section 5 describes a comparative study between the computer simulations and paper simulations to show the performance of the VR orthopedic simulator. Conclusions and future study are introduced in Section 6.
2. Volume-based orthopedic surgery simulators
Many excellent algorithms have been developed for visualizing a volume. For example, tissue sur-faces can be well approximated by hundreds of thousands of triangulated isosursur-faces at the sub-voxel level (e.g. [19]). These isosurfaces can be quickly rendered even by general PC platforms with smooth shading models such as the Gouraud shading model. Instead of manipulating triangulated isosurfaces, surgical simulation algorithms usually manipulate voxels directly to simulate surgeries
especially the ones with topology changes on tissues. For example, most commercial imaging sys-tems usually use a simple method of manipulating voxels, a cut-away operation to remove the voxels of one side of a cutting plane for removing obscurations.
Two approaches to manipulating voxel-represented structures have been discussed. One approach extends the contents of each voxel; for example, Gibson employed 6 links to represent relations between a voxel and its six face neighbors [17]. Adding and deleting the links easily implement local manipulations, such as sectioning a structure or fusing two structures. However, link additions and deletions are computationally demanding. Therefore, global manipulations such as repositioning a structure or fusing distant structures are diGcult because these manipulations need many link deletions and additions to represent swapping the structure. Another approach uses a 2D pointer array to manipulate a structure [18]. The array element is a list of triplets, each of which records the depth of a boundary voxel of the structure, a unit normal vector and 6 facing codes indicating which voxel faces of the voxel are on the boundary. Using the surface normal and face codes, a 3D image can be quickly rendered. By adding a translation to the depth of the boundary voxels in the lists, the structure in the pointer array can be translated. However, translating the structure not along the depth direction is still diGcult by this data representation scheme because it needs many time-consuming list additions and deletions.
The above data representations are inconvenient in repositioning a structure, and moreover insuf-1cient in representing a sectioned surface in a structure and identifying (recognizing) voxels inside the closed boundary of a new sectioned structure. The recognition is especially important in VR orthopedic surgery simulations, because when a surgeon cut out a new structure and then points to the structure to manipulate (reposition or remove), or to two structures requiring manipulation (fusing) the system must 1nd out all voxels belonging to the involved structures.
For achieving this purpose, we assign a structure code to every voxel of a volume in order to distinguish voxels in a sub-tissue level. That means two voxels can be of the same tissue but belong to di0erent anatomic structures. By searching the voxels with the same structure code, all voxels of a structure can be traversed in a manipulation (removal, repositioning or assignment to fuse with another structure). We then assign face-codes to every voxel in order to represent sectioned surfaces in structures. We also developed several algorithms that eGciently recognize a new sectioned structure and manipulate the structural voxels to simulate orthopedic procedures including section, fusion, removal, recognition and repositioning of structures of bones, prostheses and bone grafts and testing collisions between a moving structure with other structures, vessels and nerves. By the functions of our prototype system, a surgeon can simulate an operation on a real patient and ensure the accuracy of anatomic morphology in interactive responses.
3. Software
The software system including the simulator software and the driver software was developed in Chung Yuan Christian University, Chung-Li, Taiwan. The system was 1rst reported in 1996 [20], and has since been modi1ed, improved and used to train interns and residents, and to rehearse the surgical plans for visiting doctors in the Orthopedic Department of Taipei Medical University Hospital. The software is implemented in C + + (Visual C + + ver. 5.0) using the OpenGL libraries to render isosurfaces without special graphics hardware.
Fig. 1. System architecture.
Fig. 1 shows the system architecture. A surgeon wears a shuttle eyeglass to observe stereographic images and uses a surgical instrument attached to a six-dimensional degree tracker to simulate surgical procedures. The system includes an interface module, volume conversion module, isosurface reconstruction module, rendering module, and simulation module.
3.1. Interface module
The interface module provides virtual instruments and selectors including menus and data slide-bar. Using the menus, the surgeon can choose a volume to simulate, determine a simulation function to operate, and input bone grafts and prostheses that have been designed by an AutoCAD system and change parameters of the shading model about light and material properties. Through the slide-bars, the surgeon can easily change (slide) perspective conditions including viewing positions and angles, disparities of stereographic images to choose suitable ones.
The tracker is attached to one end of a surgical instrument to simulate a virtual instrument. Based on the position and attitude of the tracker and the shape data obtained from the instrument, the system can compute spatial data for the virtual instrument. Using the spatial data, the system can render the instrument to obtain its 3D image and compute the intersections between the instrument and the volume for simulating surgeries. The system currently provides the following virtual instru-ments: bone saws and osteotomes for sectioning bone, virtual plates and staples for 1xation, virtual dissectors and currectors for removing tumors, and a virtual hand for moving bones, bone grafts and prostheses.
The tracker is also used as a positioning instrument that partitions a volume into several subvol-umes for the convenience in rendering tissue surfaces.
3.2. Data conversion module
For manipulating voxel-represented structures, every voxel is assigned three 6-bit distance levels to simulate tissue surface changes, six 1-bit face codes indicating whether the voxel faces are on the boundary and one byte indicating a tissue type and structure number. A total of 4 bytes of memory are used for each voxel. Bone grafts and prostheses are designed by the AutoCAD system 1rst, and then converted to voxel-represented structures.
Currently, our system can convert gray-scale volumes from CT (computer tomographic) slices and bordered volumes from MRI (magnetic resonance imaging) slices.
3.3. Isosurface reconstruction module and rendering module
In contrast to thresholding techniques that determine a sample point on a tissue surface (isosur-face) by one over-threshold voxel and one under-threshold voxel, one distance-measured voxel can determine a sample point [21]. Therefore, the three distance-levels are interpreted as three sample points on the three main axes respectively. Our system then uses the marching cube algorithm that employs the sample points on the main axes to reconstruct triangulated isosurfaces [19].
The resulting isosurfaces help to convey visual information about anatomical features, and are compatible with the standard pipeline of 3D rendition thus enabling the system to take uses ad-vantage of standard 3D graphics software and hardware. The isosurfaces need not be reconstructed when changing the perspective, and can be rendered quickly even on PC-based platforms. However, reconstruction of the isosurfaces may take several seconds. If a volume is divided into several sub-volumes, the system can only reconstruct the isosurfaces of operated subvolumes in order to save isosurface reconstruction time. Currently, the volume partitioning speci1cation is set by the user, who can divide a volume into several subvolumes using the positioning instrument.
Manipulation of the three distance-levels can simulate changes of tissue surfaces. These manipu-lations are employed in the “section”, “fusing” and “healing” simumanipu-lations.
3.4. Simulation module
The “section” function 1rst interprets the positions and attitudes of a tracker as swept sectioning surfaces, computes distance-levels for sectioned boundary voxels, and assigns a structure code to the voxels. The “recognition” function uses an eGcient 3D seed and Oood algorithm to assign the voxels the same structure code inside a closed boundary composed of voxels with the structure code (sectioned boundaries) and voxels with di0erent tissue codes (natural boundaries). Unlike straightfor-ward seed and Oood algorithms that put six neighbors into a stack for recursion, voxels along some axis are directly computed and not stacked in the algorithm [22]. Therefore, voxels for recursion are considerably reduced.
The “removal” function assigns all voxels of a structure to be air voxels that can also be im-plemented by the 3D seed and Oood algorithm. The “fusion” function re-recognizes one anatomic structure (separate bone, prosthesis or bone graft) into another to joins them together. The structures may contact each other and no new structure voxels are generated. New structural voxels are gener-ated in the fusing process if the structures are distant. In this situation, the system generates a closed
boundary connecting user-speci1ed curves on the two fusing structures and then re-recognizes the voxels inside the new boundary with the old structures as one structure.
The “collision test” function detects collision among anatomic structures of bones, prostheses and bone grafts, and vessels and nerves. He proposed an eGcient collision detection method that maps all objects into a map of regular cells, and then detects collisions if objects occupy other object’s spaces [23]. This grid intersection method was not adopted to detect collisions in our system because other functions are implemented during the collision test. One such function determines the distance be-tween structures when a structure is placed onto another structure in a “fusing” simulation. The other function assigns a structure code to traversed soft-tissue voxels to obtain associated soft tissues in a “healing” simulation. We adopted an eGcient ray traversal algorithm to detect collisions (whether bone or nerve voxels exist on the path of a moving anatomic structure or surgical instrument). This algorithm is the most eGcient because it has the fewest additions and comparisons [24]. The “reposi-tion” function translates all voxels of a structure to another position by 1rst implementing a “collision test” to detect collisions, then pushing the structure into a series of stacks and clearing the structure of the old position by the seed and Oood algorithm before popping the structure to a new position. FEM (1nite element methods) and mass-spring techniques are often employed to simulate object deformation in computer graphics [25,26]. These techniques are not anatomic but still can predicate soft tissue prognosis in some degree of precision. However, we do not adopt these methods because they are time demanding, and so diGcult to achieve the interactive (fast response) requirement. We let the surgeon determine the soft tissue morphology. The “healing” function determines an associated soft tissue of a moving bone, and moves it with the bone and fuses the associated soft tissue with original soft tissues similar to the bone fusion. The surgeon should modify (by sections) interactively the inconsistencies between the fused associated soft tissues and the original soft tissues. Although this healing method is not procedure-like, surgeons can predict a satisfactory shape of soft tissues after the healing.
4. Complicated orthopedic surgery simulations
Surgeons can use this system to simulate procedures on rigid bones, prostheses and bone grafts in orthopedic surgeries such as arthroplasty, corrective or open osteotomy, fusion, open reduction for complicated fractures and amputation via the simulation functions outlined above. In the following, we explain how to use the simulation functions to simulate the surgeries. The exercises of the simulation functions implemented on one knee arthroplasty, one open osteotomy and one fusion are demonstrated in the following.
The following simulation CPU times were obtained under implementing on a PC with a Pentium-III 800 MHz CPU and 256 Mbytes of main memory. For setting up such a VR simulator platform, the PC must be equipped with a shutter glass (CrystalEyes PC by StereoGraphics) and a tracker (InsideTRAK by Polhemus). It costs approximately U.S. $2,000 to set up the platform.
4.1. Arthroplasty and corrective osteotomy
In the simulation of arthroplasty operations, the surgeon sections (using the “section” function) bones until one or more anatomic structures are separated (and thus recognized by the “recognition”
function) from the skeleton. The surgeon may remove (using the “removal” function) the structures to correct the skeletal morphology, to accommodate the prosthesis or in the case where the structures are abnormal bones. A prosthesis structure is used to replace a removed joint in arthroplasty. The surgeon may reposition the structures (using the “reposition” function) to correct the skeletal on morphology, and then 1x the structures and fuse (using the “fusing” function) them into the skeleton. 4.2. Example of knee arthroplasty
Fig. 2 shows the image-rendering results of an example knee arthroplasty operation that was per-formed to replace a destroyed joint and correct a mal-position of the tibia. The volume was constructed with 24 CT slices at a 256 × 256 resolution. However, we enlarged the volume as 35 × 256 × 256 resolution for manipulating a user-input prosthesis. The computation time was 2:2 s to reconstruct the bone isosurfaces for the whole volume and 0:29 s to obtain a 3D image with this system. After the prosthesis was input into the volume, the isosurface reconstruction time be-comes 3:2 s for the bone and prosthesis isosurfaces and the rendition time bebe-comes 0:42 s.
For saving the isosurface reconstruction time, the volume was divided into 1ve subvolumes: joint, near-joint tibia, near-joint femur, far-joint tibia and far-joint femur. The isosurface reconstruction time for a subvolume was near 1=6 ∼ 1=4 of the one for the whole volume. The system reconstructed the isosurfaces of operated parts after a procedure. Because most of the procedures operated only one subvolume and some procedures operated two subvolumes, the volume partitioning is eGcient in saving simulation time.
Fig. 2(a) shows the proximal tibia being sectioned by the saw. The interface slidebars for deter-mining the parameters of the various perspectives and menus for deterdeter-mining a simulation function are also shown in the left top and bottom, respectively. Fig. 2(b) shows the results after two Oat sections on the femur and tibia, respectively, followed by recognition and removal of a near Oat bone fragment of the femur and a wedge-shaped fragment of the tibia. A hand (a virtual instrument) began to reposition the tibia. Fig. 2(c) shows that the tibia was repositioned to correct the mal position. Fig. 2(d) shows that a vertical bone fragment was sectioned away so that the femur can accommo-date the posterior of the prosthetic femur. The virtual hand was removing the bone fragment. Fig. 2(e) shows that a vertical bone fragment and an oblique one were sectioned away such that the femur can accommodate the anterior of the U-shaped prosthetic femur. An oblique section on the posterior of the femur for accommodating the U-shaped prosthetic femur and an oblique section on the patella for accommodating the prosthetic patella were then sectioned away.
Fig. 2(f) shows that the prosthesis has been recognized (by recognition functions) as three separate structures: a curved femur part, disk-like tibia part and dome-like patella part. The tibia part has been repositioned for insertion on the tibia by the virtual hand. This 1gure also shows that the dome of the prosthetic patella can well slide inside the groove of the prosthetic femur and the prosthetic tibia also well matched to the tibia plateau. Fig. 2(g) shows that the U-shaped prosthetic femur has been repositioned for insertion on the femur. The prosthetic patella has already been inserted on the patella. However, we cannot observe it well because it is almost hidden by the patella. The prosthetic femur can match the residual femur well, and therefore the previous sections on the femur were appropriate. The U-shaped curve of the prosthetic femur can also slide well inside the grooves of the prosthetic tibia. Therefore, the prosthetic femur and tibia are considered well positioned.
Table 1
Total simulation time for knee arthoplasty simulationsa
1 2 3 4 5 6 7 8 9 10 11 12
Time 0.95 1.24 1.98 0.96 1.95 0.93 1.88 1.97 1.75 2.05 3.20 0.93 spent (s)
Voxels 8879 11845 37365 5252 4990 4880 3985 3754 4750 19605 39630 1250 involved
a1, sectioning away (sectioning, recognizing and removing) the high-tibia; 2, sectioning away the low-femur; 3, recognizing
and repositioning the tibia; 4, a vertical section on the anterior part of the femur; 5, recognizing and removing the vertical section; 6, an oblique section on the anterior part of the femur; 7, recognizing and removing the oblique part; 8, A vertical section on the posterior part of the femur then moved away; 9, an oblique section on the posterior part, then moved away; 10, repositioning the prosthetic tibia; 11, repositioning the prosthetic femur; 12, repositioning the prosthetic patella.
The simulation example provides an anatomical demonstration that the knee arthroplasty can cor-rect the mal-position of the tibia, accommodate the tibia and femur to 1t the prosthesis and insert the prosthesis into the correct position. The sizes of three prosthetic parts are good choices for working the knee functions. The complex changes in skeletal morphology involved in this surgery were well simulated by our system. The results of every procedure on the bones and the prosthesis can be thoroughly demonstrated with high-quality 3D images. Table 1 shows the computer response times for the simulations involved in the knee arthroplasty. A complete simulation is de1ned as including completion of the speci1ed function, reconstruction of the isosurfaces and rendering of the corre-sponding image. Because most the simulations responded in 2 s, we considered that our system can achieve the requirement of interactive responses.
4.3. Open osteotomy
Open osteotomy is used to open a bone in order to remove tumors inside the bone. Upon simu-lation of using our system, the surgeon sections a bone until a window structure separates. He then repositions the structure away to indicate opening the bone by using the recognition function and then the reposition function. Then, he dissects the tumor (using the section function) and removes it. The surgeon may simulate implantation of a bone graft by inputting a bone graft and repositioning it to the tumor position. He then 1nally repositions the separate window structure to the original position and fuses the window structure with the original bone together to simulate closure of the bone.
←−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−
Fig. 2. Arthoplasty for replacing the knee joint and correcting mal-position of the knee: Gray areas: surfaces of bones or a prosthesis in (f) and (g). Red areas: virtual instruments including (1) a virtual hand in (b), (c), (d), (e), (f) and (g) for repositioning a bone or a prosthetic part; (2) a virtual saw in (a) for sectioning a bone. (a) The saw is sectioning proximal tibia; (b) Knee joint has been sectioned away, and tibia is being repositioned; (c) Tibia has been repositioned to correct the mal-position; (d) Posterior femur part has been sectioned away to accommodate the U-shaped femur part of the prosthesis; (e) Anterior femur part has been sectioned away to accommodate the U-shaped femur part of the prosthesis; (f) Tibial part of the prosthesis has been inserted into to the knee joint; (g) Femur part of the prosthesis has been inserted into the knee joint.
4.4. Example of knee open osteotomy
Fig. 3 shows the rendering results of a knee open osteotomy for removing a tumor inside the proximal tibia. The volume was constructed with 28 CT slices at a resolution of 256×256. Fig. 3(a) shows a 3D image of a knee where the proximal tibia was being sectioned by a virtual saw. Fig. 3(b) shows a window-shaped bone fragment that has been sectioned, recognized and repositioned away using the virtual hand. The area of the tumor is marked with an orange color. A dissector (indicated by a red color) is available to dissect the tumor. Fig. 3(c) shows the tumor being dissected by the dissector. Fig. 3(d) shows that the tumor has been removed and a graft bone (lower left corner) has been prepared (already recognized) to 1ll the space of the resected tumor. Fig. 3(e) shows that the graft bone has been implanted and that the window-shaped bone fragment was being repositioned again to its original location. Fig. 3(f) shows the results after fusing the bone fragment with the knee. The results suggest that the position and size of the window fragment is a reasonable choice for opening the knee and allowing the tumor to be completely removed. The graft bone is suitable to 1ll up the tumor space.
4.5. Fusion
Fusion is used to implant bone grafts to treat patients with anatomic instability between bones. To simulate an operation involving fusion, the surgeon can use the section and the recognition functions to section away unstable bones, input a recognized bone graft, and then use the fusion function to fuse the bone grafts and the original bones together.
4.6. Example of spine fusion
Fig. 4 shows an example of a fusion operation between vertebral bodies in a patient with instability with L-HIVD (lumbar herniated inter-vertebral disc) at L4-5 (fourth lumbar spine-1fth lumbar spine) and L5-S1 (1rst sacrum spine) [27]. The volume is constructed with 6 MRI bordered slices with a 256 × 256 resolution. Fig. 4(a) shows a frontal view of the bone surfaces (gray area) of the lumbar spine for this simulation, including disk spaces (green area) and spinal cord and roots (red area), while Fig. 4(b) depicts this area after bone disarticulation. Fig. 4(c) demonstrates the result after the simulated procedures of anterior decompression, disectomy, and partial corpectomy by sectioning away parts of L4 and L5 and part of the disk between L4 and L5. After sectioning the disk, we can observe the spinal cord, root and posterior column of the bone tissue. Fig. 4(d) demonstrates the result after the operation of decompressing the nerve root and cord, and fusing L4-5 anteriorly by inserting a bone graft into the sectioned L4-L5 disk space. The results of the procedures of the preoperative simulation suggests that the anterior fusion of L4-5 with osteotomy of L4 and L5 plus autogenous bone graft with an iliac bone block graft would achieve the desired result in this case. 4.7. Open reduction and amputation
Open reduction is used to reposition the bone fragments caused by a complicated fracture to their normal anatomical positions and to then fuse them into the skeleton. A graft bone can be used to compensate for defects if some of the bone fragments are compressed. In the simulation of open
Fig. 3. Open osteotomy for removing a tumor in the tibia. Gray areas: surfaces of bones or a bone graft in (d). Orange area: a tumor. Red areas: virtual instruments including (1) a virtual hand in (b), (d) and (e) for repositioning a bone or a bone graft; (2) a virtual saw in (a) for sectioning a bone; (3) a virtual dissector in (b) and (c) for sectioning the tumor. (a) The saw is sectioning proximal tibia; (b) Window-shaped bone fragment is sectioned and repositioned away; (c) The dissector is dissecting tumor; (d) Bone graft is prepared to 1ll-up the space of excised tumor; (e) Bone graft has been implanted, and the window shaped piece of bone is being repositioned to the original position; (f) Window-shaped bone is repositioned and fused with the tibia.
Fig. 4. Anterior fusion for solving instability between L4 and L5 veteral bones: (a) Frontal view of bone surfaces (gray area) of the lumbar spine, including disk spaces, spinal cord and roots; (b) After bone disarticulation; (c) Sectioning away parts of the disk between L4 and L5 for anterior decompression, disectomy and partial corpectomy; (d) Inserting bone graft into the sectioned L4-L5 space for decompression of the nerve root and cord, and anterior fusion of L4-5.
reduction surgery, surgeons can use the recognition function to identify separate bone fragments by the location of the fracture, the reposition function to position the bones into their normal positions, and the fusion and healing functions to evaluate the prognosis of bones and soft tissues.
Amputation is used to cut the abnormal parts of upper or lower extremities o0 in order to preserve the normal parts. Surgeons use the section function to section an extremity, and the recognition and removal functions to recognize and then remove the abnormal extremity.
5. Comparative study between VR simulations and paper simulations
From July 1997–June 1998, a number of surgeries are simulated simultaneously using the VR computer system and the traditional paper surgery methods by a group of surgeons. We compared the results of the paper and computer simulations to know which method give more real, precise and preferable results and is easy to use.
5.1. Study participants and VR exercises
All participants including 4 visiting doctors, 8 residents and 4 interns were male, right handed with no visual defects and no previous experience with VR systems. The average age was 45 for the visiting doctors, 32 for the residents and 26 for the interns. The mean number of career operations
performed by the visiting doctors was 1800. Before participating the program, all the residents and interns were already passed the 20-h regular course about the paper surgeries.
Each participant performed a VR training exercise to gain familiarity with the system before beginning the project. Operators who are inexperienced with VR may not be familiar with the disparity of stereo images, and therefore, may not section the correct position. These operators may also incorrectly reposition a structure because of this disparity. The recognition function may fail if the structure codes speci1ed on divided structures during the sectioning were not consistent. In addition, the fusing simulation may fail if the speci1ed fusing surfaces on fusing structures were not consistent. A system developer stood by in order to instruct the participants to solve the problems, choose a good parity, viewing position and angles, and specify correct fusing surfaces and structure codes. The criteria for a participant to pass the exercise were if he could operate all of the system functions and to know how to redo the simulation when an error encountered during a simulation. These criteria were the same with the ones of passing the paper course. Because re-dosing was allowed in both paper and computer simulations, therefore familiarities about using the simulation functions did not a0ect the results.
The average exercise time was about 5 h for the visiting doctors, 10 h for the residents and 15 h for the interns. The reason of the visiting doctors had a shorter time to become familiar with the VR sim-ulator is, they were well experienced then had more spatial perception about the orthopedic surgeries. The spatial perception was helpful to them in becoming familiar with the use of the VR simulator. 5.2. Materials
During the test period, over 1000 orthopedic surgeries were performed at the Orthopedic Depart-ment of Taipei Medical University Hospital. We selected 50 of these orthopedic surgeries including 20 arthroplasties, 5 corrective osteotomies, 5 open osteotomies, 12 fusions, 5 open reductions and 3 amputations as the samples for the study. The ratios among the selected surgeries correspond to the ones of the total surgical categories during the period.
Standard imaging procedures for the all selected surgeries included an anterior–posterior X-ray and a lateral X-ray, and a set of traversal (either CT or MRI) slices and the fees were paid by the Central Health Insurance Bureau under the compulsory insurance system. As the result, this study did not incur any additional expenses for the procedures. The traversal slices were scanned into the VR simulator as a volume data, and then used for the VR simulations performed by all the participants, and the pair of X-rays was used for the paper simulations. Each participant implemented the simulations individually not simultaneously; therefore only one VR system had been used.
After the simulations, the surgery was performed by one of the visiting doctor, two of the residents and one of the interns. After a certain period depending on the type of the performed surgery, an anterior–posterior X-ray and a lateral X-ray were taken to obtain data on bone fusion and soft tissue healing. A questionnaire was then given to every participant for comparing the computer simulation with the paper simulation based on the following criteria: realism, ease of use, precision, helpfulness to surgeries and preference. The values assigned to the criteria could only be an integer from 1 to 5 (poor, not good, fair, good and excellent).
The numbers of the replies to the criteria of realism, ease of use and precision were 400 (1 reply × 50 surgeries × 8 persons) from the residents and 200 (1 reply × 50 surgeries × 4 persons) from the interns or the visiting doctors, respectively. For the criteria of helpfulness during the actual
surgery, the replies were 100 (1 reply × 50 surgeries × 2 persons) from the residents and 50 from the interns or the visiting doctors (1 reply × 50 surgeries × 1 person). For the criteria of preferences, the replies were 8 from the residents (1 reply × 8 persons) and 4 from the interns or visiting doctors (1 reply × 4 persons).
5.3. Results and analysis
Fig. 5 shows the horizontal full-range boxplots that were used to describe the individual dis-tributions of the evaluated criteria (except for preferences) in the paper simulation and computer simulation, respectively. A boxplot indicates the highest and lowest values of the replies for a crite-rion. A rather thick line inside each boxplot shows the mean value of the replies to the corresponding criterion. Because the replies to the preferences are too few, only the mean value but no boxplots are illustrated.
All three groups rated the computer simulation as good realism, while the paper simulation was rated as not good. The paper simulation provides only a two-dimensional view. In contrast, the VR simulation system provides a visual VR environment that can better simulate the actual visual experience of the surgery. The participants also commented that the main drawbacks of the VR system are response speed and insuGciency of tactile feeling. They hoped for real-time responses and haptic capabilities. They considered that these drawbacks degraded the sense of realism of the system. Although the system provides collision tests that can prevent penetration among structures and instruments, the collision tests cannot give any tactile feeling.
The paper simulation was considered as having the greatest ease of use. Almost no equipment was needed to implement the paper simulations except for paper and pencils. In contrast to this, the operators in the VR system had to be equipped with VR devices (shuttle glass and tracker) in order to perform the computer simulations. The user also had to wait for the computer response when simulating a surgical procedure. However, some participants commented that this inconvenience is acceptable since they could obtain good simulation results.
The relative precision of the paper and computer simulations was compared with the X-rays of the surgical results after bone fusion and soft tissue healing. The simulation predictions about dimensions of bones and soft tissues, angles and distances between anatomic structures were compared with the actual data. The visiting doctors achieved excellent results in the computer simulations and good results in the paper simulations. The surgical experiences of the visiting doctors can help them to achieve good paper simulations even under a 2D environment. The residents achieved excellent results in the computer simulations but fair results in the paper simulations. The interns achieved good results in the computer simulations but not good results in the paper simulations. The residents and especially the interns had no or limited surgical experience; and therefore did not achieve good results with the paper simulations.
The computer simulations were found to be very helpful in the training and planning for a surgery by all of the participants mainly for the following reasons: (1) the system could simulate surgical procedures on bones, prostheses and bone grafts, and provided a realistic stereographic image auto-matically not manually as the paper simulations, and (2) the stereographic images could be observed from an arbitrary perspective which enabled the operators to increase their understanding of the involved anatomy during the surgery. This enabled the operators to gain a greater understanding of the changes in skeletal morphology that will result from these surgical procedures. The visiting
Fig. 5. Comparisons between paper simulations and surgery simulations based on the criteria of realism, ease, helpfulness to surgery and preference with three groups of surgeons (interns, residents and visiting doctors): (◦) Horizontal full-range boxplot: individual distributions of the evaluated criterion; () Thick line: mean value of the replies to a criterion.
doctors also considered the paper simulations to be a useful tool in the preparation and planning for surgery. However, because the visiting doctors were already well trained, they could evaluate the procedures well using manual simulations on 2D papers and mentally reconstruct the 3D images. However, the residents and especially the interns had no such abilities.
The residents and interns expressed their preference to use the VR simulation system as a learning tool as having the following reasons: (1) the simulation system automatically provided a realistic simulation of the surgical process, and (2) preference for using a computer-based system since they already used computers which in many other aspects of their work such as documenting and communication. Therefore, they preferred the style of our preoperative simulation tool. In contrast, the visiting doctors were already accustomed to paper simulation methods. They only expressed a small preference for using the computer simulations.
6. Discussion and future works
Simulators must achieve a useful degree of realism for a corresponding surgery. Our simulation methods can manipulate voxel-represented structures of bones, prostheses and bone grafts to model interactions between the structures such as: cutting, fusing, repositioning, recognition, and collision testing of a moving structure. By these functions, our system can simulate complex geometry and topology changes of skeletal morphology for every orthopedic procedure. These capabilities are necessary to provide helpful spatial information for most orthopedic surgeries.
In our study of the e0ectiveness of this system, residents and interns were able to obtain much better prognostic information using the computer simulations than using the paper simulations. They also preferred the VR simulator to paper simulation. These 1ndings indicate that the VR simulator is a better training tool than traditional paper surgery methods. It is especially useful in the preparation for many kinds of diGcult surgical procedures that are often performed in the orthopedics department. The visiting doctors also considered the VR simulator as an excellent tool for planning surgery and even to experiment with new surgical techniques without putting patients at risk.
In the future our work will focus on improving some drawbacks of the prototype system. Im-provements in the user interface can ease surgeons to operate the system. For example, we hope to assign distinct colors to di0erent structures for highlighting some structure. As the arthroplasty example shows, the bones and the prosthesis were processed as the same tissue. The more important prosthesis should be better to be highlighted for easy observation. The improvement in reducing response time of the system would help improve realism and ease to use. Because the system uses a volume to simulate every surgical procedure, rendering isosurfaces reconstructed from a volume to obtain a 3D image is usually computationally demanding. Use of decimation techniques to re-duce triangles of isosurfaces or other volume visualization techniques such as accelerated volume rendering techniques can be tried to save the rendition time.
Another drawback of the system is in the simulation of post-surgical procedures that rely on the speci1cations of the operator, that is, fusing surfaces in the fusion simulation and morphology modi1cation in the simulation of soft tissue healing. Unreasonable speci1cations may be assigned by some user and so an inaccurate prognosis results. The system should be optimized to automatically determine the corresponding fusing surfaces on structures and to recover soft tissues under a given speci1cation (including age, sex, race and so forth).
Further investigation will be needed in order to allow the introduction of physical constraints and force-feedbacks into the system. In this initial study, we avoided the diGculties involved in simulating tissue deformation, tissue resistance against surgical instruments and precise computation of soft tissue recovery. Studies of the physical constraints that are present in orthopedic surgeries could be used to augment a VR tactile environment to our system.
7. Summary
Computed tomography (CT) or magnetic resonance imaging (MRI) scanning has become a stan-dard procedure to reveal interior anatomies. Visualizing a volume constituted by transversal slices can ease observation of anatomies to improve diagnoses. Beyond the volume visualization, manip-ulating volume data to simulate deformation or topology changes of tissues during a surgery can verify surgical plans, rehearse procedures and predict prognoses. In the presented work, an additional link is deleted or added between any two voxels (discrete volume element) to represent structure section and fusion. However, repositioning a structure and recognizing whether tool swept surfaces section out a new structure cannot be simulated because insuGcient data of every voxel.
So we extended each voxel with a structure code to distinguish voxels of the same tissue but distinct structures. Then, we used an eGcient 3D seed and Oood algorithm to recognize continuous voxels of a structure remove and reposition all the voxels the structure. By manipulating the structural volume, we can also test collisions between a moving structure, such as bone, prosthesis and bone graft, with vessels, nerves and other structures, and simulate topology and geometry changes on structures including sectioning a structure, recognizing whether the sectioned structure is separate and fusing separate structures into one. We have succeeded in using the volume manipulation algorithms to simulate complex orthopedic surgeries such as arthroplasty, corrective and open osteotomy, fusion, open reduction for complicated fractures and amputation.
A highly interactive prototype system equipped with VR devices has been developed and modi1ed by the computer specialist in Institute of Information and Computer Engineering of Chung-Yuan Christian University. The surgeons in Orthopedics Department of Taipei Medical University Hospital has tested, evaluated and criticized the prototype system. This VR simulator has o0ered an e0ective alternative to conventional paper surgery. Our comparative study has indicated that, the simulator allows surgeons to obtain more real and precise simulations about surgical procedures compared to paper-based simulations, therefore it is helpful in verifying orthopedic modalities, and training interns and residents in preoperative orthopedic meetings.
Acknowledgements
The authors would like to thank the National Science Council for 1nancial support of this research under Contract No. NSC-86-2213-E-033-036, NSC-87-2213-E-033-005, NSC-89-2320-B-038-019-M08 and NSC-89-2213-E-033-070. We would also like to thank all of our colleagues at the computer research lab, the visiting doctors, residents and interns at our orthopedic department for participation in the study. We also like to thank the Radiology Department of Taipei Medical University Hospital
for providing the CT and MRI data and the United Orthopedic Corporation of Taiwan for providing the prosthesis data.
References
[1] J.A. Finkelstein, A.E. Gross, A. Davis, Varus osteotomy of the distal part of the femur: A survivorship analysis, J. Bone J. Surg. Am. 78A (1996) 1348–1352.
[2] A. Nagel, J.N. Insall, G.R. Scuderi, Proximal tibial osteotomy. A subjective outcome study, J. Bone J. Surg. Am. 78A (1996) 1353–1358.
[3] H. James, Beaty, Orthopaedic Knowledge Update. In: San. Howard, T.L. Randall (Eds.), Spine, Rosemont, American Academy of Orthopaedic Surgeons, 1999, pp. 561–754.
[4] S.R. Gar1n, A.R. Vaccaro, Orthopaedic knowledge update, in: S.R. Gar1n, A.R. Vaccaro (Eds.), Spine, Rosemont, American Academy of Orthopaedic Surgeons, 1997, pp. 55–61.
[5] J.E. Hall, Dwyer instrumentation in anterior fusion of the spine, J. Bone J. Surg. Am. 63A (1981) 1188–1190. [6] A.R. Hodgson, F.E. Stock, Anterior spinal fusion: a preliminary communication on the radical treatment of Pott’s
disease and Pott’s paraplegia, Br. J. Surg. 44 (1956) 266–275.
[7] J.J. Regan, H. Yuan, P.C. Mc Afee, Laparoscopic fusion of the lumbar spine: minimally invasive spine surgery, a prospective multicenter study evaluating open and laparoscopic lumbar fusion, Spine 24 (1999) 402–411.
[8] M. Muschik, H. Zippel, C. Perka, Surgical management of severe spondylolisthesis in children and adolescents. Anterior fusion in situ versus anterior spondylodesis with posterior transpedicular instrumentation and reduction, Spine 22 (17) (1997) 2036–2042.
[9] J.C. Flynn, M.A. Hoque, Anterior fusion of the lumbar spine. End-result study with long-term follow-up, J. Bone J. Surg. Am. 61 (1979) 1143–1150.
[10] R.M. Satava, Virtual reality, the current status of the future, in: S. Weghorst, H. Siegurg, K. Morgan (Eds.), Health care in the Information Age, IOS Press, Amsterdam, 1996, pp. 542–545.
[11] R.M. Satava, Virtual reality surgical simulator.: The 1rst steps, Surgical Endoscopy 7 (3) (1993) 203–205. [12] L. Caponetti, A.M. Fanell, Computer-aided simulation for bone surgery, IEEE Comput. Graphics Appl. 13 (6) (1993)
87–91.
[13] R. Ziegler, G. Fischer, W. Muller, M. Gobel, Virtual reality arthroscopy training simulator, Comput. Biol. Med. 25 (2) (1995) 193–203.
[14] S. Seipel, I.V. Wagner, S. Koch, W. Schneide, Oral implant treatment planning in a virtual reality environment, Comput. Methods Programs Biomed. 57 (2) (1999) 95–103.
[15] I.W. Hunter, L.A. Jones, M.A. Sagar, S.R. Lafontaine, P.J. Hunter, Ophthalmic microsurgical robot and associated virtual environment, Comput. Biol. Med. 25 (2) (1995) 173–182.
[16] S. Baumrind, F. MoGt, S. Curry, The geometry of three-dimensional measurement from paired coplanar X-ray images, Am. J. Orthod. 84 (1983) 313–326.
[17] J.K. Udupa, D. Odhner, Fast visualization, manipulation and analysis of binary volumetric objects, IEEE Comput. Graphics Appl. 11 (5) (1991) 53–63.
[18] S.F.F. Gibson, Using linked volumes to model object collision, deformation, cutting, carving, and joining, IEEE Trans. Visualization Comput. Graphics 5 (4) (1999) 333–348.
[19] W.E. Lorensen, H.E. Cline, Marching cubes: a high resolution 3D surface construction algorithm, Proceedings of ACM SIG Graph Computer Graphics, Addison Wesley Press, Reading, MA, 1987, pp. 163–169.
[20] M.D. Tsai, M.S. Hsieh, W.C. Chang, S.K. Wang, Volume manipulation algorithms for simulating musculoskeletal surgery, in: S.S. Yang, D. Thalmann (Eds.), Proceedings of the Fourth Paci1c Conference on Graphics and Applications, National Chiao Tung University Press, Hsinchu, Taiwan, 1996, pp. 220–234.
[21] G. Sealy, K. Novins, E0ective volume sampling of solid models using distance measures, in: B. Werner (Ed.), Proceedings of IEEE Computer Graphics International, IEEE Computer Society Press, Canmore, Alberta, Canada, 1999, pp. 26–32.
[22] L. Feng, S.H. Soon, An e0ective 3D seed 1ll algorithm, Comput. Graphics 22 (5) (1998) 641–644.
[23] T. He, A. Kaufman, Collision detection for volumetric objects, Proceedings of IEEE Visualization, IEEE Computer Society Press, Phoenix, Arizona, USA, 1997, pp. 159–166.
[24] D. Cohen, A. Kaufman, 3D line voxelization and connectivity control, IEEE Comput. Graphics Appl. 17 (6) (1997) 80–87.
[25] S. Cotin, N. Delingette, N. Ayche, Real-time elastic deformations of soft tissues for surgery simulation, IEEE Trans. Visualization Comput. Graphics 5 (1) (1999) 62–73.
[26] R.M. Koch, M.H. Gross, F.R. Carls, D.F.V. Buren, G. Fankhauser, Y.I.H. Parish, Simulating facial surgery using 1nite element models, in: J. Fujii, H. Rushmeier (Eds.), Proceedings of ACM SIG Graph Computer Graphics, Addison Wesley Press, New Orleans, Louisiana, USA, 1996, pp. 421–428.
[27] M.S. Hsieh, M.D. Tsai, Diagnosis of herniated intervertebral disc assisted by 3-dimensional, multiaxial, magnetic resonance imaging, J. Formosan Med. Assoc. 98 (5) (1999) 347–355.
Ming-Dar Tsai received the BS degree in Mechanical Engineering from National Taiwan University, Taipei, Taiwan, in 1983, and MS and Ph.D. degrees in Machinery Precision Engineering from the University of Tokyo, Tokyo, Japan, in 1988 and 1991, respectively. Since 1991, he has been on the faculty of the Department of Information and Computer Engineering, Chung Yuan Christian University, Chungli, Taiwan, where he is currently an Associate Professor. His research interests include computer graphics, virtual reality, scienti1c visualization and computer in medical applications.
Ming-Shium Hsieh received the MD degree in Medicine from Taipei Medical College, Taiwan, in 1974, and the Ph.D. degree in Orthopedic Surgery from Essen University, Essen, Germany, in 1982. Since 1986, he has been on the faculty of the Department of Orthopedics at Taipei Medical College, Taipei, Taiwan, where he is currently the Chairman and an Associate Professor. His research interests inclue image studies and orthopedic 1eld including spine surgery, arthroplasty and traumatology.
Shyan-Bin, Jou is a Ph.D. student at Chung Yuan Christian University. He received the BS degree in Civil Engineering from Chung Yuan Christian University and MS degree in Computer and Information Science from New Jersey Institute of Technology in 1984, 1990, respectively. Since 1991, he is a lecturer at Nanya Institute of Technology, Chungli, Taiwan. His research interests include computer graphics and image processing.
