The effect of haptic degrees of freedom on task performance in virtual surgical environments.

Force and touch feedback, or haptics, can play a significant role in the realism of virtual reality surgical simulation. While it is accepted that simulators providing haptic feedback often outperform those that do not, little is known about the degree of haptic fidelity required to achieve simulation objectives. This article evaluates the effect that employing haptic rendering with different degrees of freedom (DOF) has on task performance in a virtual environment. Results show that 6-DOF haptic rendering significantly improves task performance over 3-DOF haptic rendering, even if computed torques are not displayed to the user. No significant difference could be observed between under-actuated (force only) and fully-actuated 6-DOF feedback in two surgically-motivated tasks.

Telerobotic system concept for real-time soft-tissue imaging during radiotherapy beam delivery.

PURPOSE: The curative potential of external beam radiation therapy is critically dependent on having the ability to accurately aim radiation beams at intended targets while avoiding surrounding healthy tissues. However, existing technologies are incapable of real-time, volumetric, soft-tissue imaging during radiation beam delivery, when accurate target tracking is most critical. The authors address this challenge in the development and evaluation of a novel, minimally interfering, telerobotic ultrasound (U.S.) imaging system that can be integrated with existing medical linear accelerators (LINACs) for therapy guidance. METHODS: A customized human-safe robotic manipulator was designed and built to control the pressure and pitch of an abdominal U.S. transducer while avoiding LINAC gantry collisions. A haptic device was integrated to remotely control the robotic manipulator motion and U.S. image acquisition outside the LINAC room. The ability of the system to continuously maintain high quality prostate images was evaluated in volunteers over extended time periods. Treatment feasibility was assessed by comparing a clinically deployed prostate treatment plan to an alternative plan in which beam directions were restricted to sectors that did not interfere with the transabdominal U.S. transducer. To demonstrate imaging capability concurrent with delivery, robot performance and U.S. target tracking in a phantom were tested with a 15 MV radiation beam active. RESULTS: Remote image acquisition and maintenance of image quality with the haptic interface was successfully demonstrated over 10 min periods in representative treatment setups of volunteers. Furthermore, the robot's ability to maintain a constant probe force and desired pitch angle was unaffected by the LINAC beam. For a representative prostate patient, the dose-volume histogram (DVH) for a plan with restricted sectors remained virtually identical to the DVH of a clinically deployed plan. With reduced margins, as would be enabled by real-time imaging, gross tumor volume coverage was identical while notable reductions of bladder and rectal volumes exposed to large doses were possible. The quality of U.S. images obtained during beam operation was not appreciably degraded by radiofrequency interference and 2D tracking of a phantom object in U.S. images obtained with the beam on/off yielded no significant differences. CONCLUSIONS: Remotely controlled robotic U.S. imaging is feasible in the radiotherapy environment and for the first time may offer real-time volumetric soft-tissue guidance concurrent with radiotherapy delivery.

Validation of a rhinologic virtual surgical simulator for performing a Draf 3 endoscopic frontal sinusotomy.

BACKGROUND: We recently introduced a patient-specific rhinologic virtual surgical environment (VSE) that has shown potential for surgical rehearsal of various skull base lesions. Our aim in this study was to validate the usefulness of the rhinology VSE in performing the Draf 3 procedure. METHODS: An outside-in Draf 3 procedure was performed on 4 cadaver heads. Computed tomography (CT) scans were obtained before and after cadaver dissection (CD). Pre-dissection CT scans were used to construct a cadaver-specific VSE. A virtual Draf 3 dissection (VD) was performed using the same technique. Validation was conducted by comparing the final common frontal outflow tract. A subjective comparison of the post-dissection endoscopic findings (CD vs VD) and an objective measurement using the post-dissection CT scan for the CD and the reconstructed CT scan obtained from the data after the VD was performed. RESULTS: Subjective overall resemblance of the 2 dissections (CD vs VD) assessed by the 4-point Likert scale (0-3) was 2.5 (median interquartile range [IQR], 0.25) for the 4 cadavers. The median difference for the anteroposterior dimension of the frontal neo-ostium (CD vs VD) assessed in the midsagittal view was 0.11 mm, whereas the median difference for the lateral dimension assessed in the coronal view was 2.71 mm. Thus, no statistical difference was observed. CONCLUSION: VD showed nearly matching results with the actual cadaver dissection. With further validation, our rhinologic VSE may be used for presurgical planning and rehearsal before the actual Draf 3 procedure is performed in the operating room.

Automatic preparation, calibration, and simulation of deformable objects.

Many simulation environments - particularly those intended for medical simulation - require solid objects to deform at interactive rates, with deformation properties that correspond to real materials. Furthermore, new objects may be created frequently (for example, each time a new patient's data is processed), prohibiting manual intervention in the model preparation process. This paper provides a pipeline for rapid preparation of deformable objects with no manual intervention, specifically focusing on mesh generation (preparing solid meshes from surface models), automated calibration of models to finite element reference analyses (including a novel approach to reducing the complexity of calibrating nonhomogeneous objects), and automated skinning of meshes for interactive simulation.

Providing metrics and performance feedback in a surgical simulator.

One of the most important advantages of computer simulators for surgical training is the opportunity they afford for independent learning. However, if the simulator does not provide useful instructional feedback to the user, this advantage is significantly blunted by the need for an instructor to supervise and tutor the trainee while using the simulator. Thus, the incorporation of relevant, intuitive metrics is essential to the development of efficient simulators. Equally as important is the presentation of such metrics to the user in such a way so as to provide constructive feedback that facilitates independent learning and improvement. This paper presents a number of novel metrics for the automated evaluation of surgical technique. The general approach was to take criteria that are intuitive to surgeons and develop ways to quantify them in a simulator. Although many of the concepts behind these metrics have wide application throughout surgery, they have been implemented specifically in the context of a simulation of mastoidectomy. First, the visuohaptic simulator itself is described, followed by the details of a wide variety of metrics designed to assess the user's performance. We present mechanisms for presenting visualizations and other feedback based on these metrics during a virtual procedure. We further describe a novel performance evaluation console that displays metric-based information during an automated debriefing session. Finally, the results of several user studies are reported, providing some preliminary validation of the simulator, the metrics, and the feedback mechanisms. Several machine learning algorithms, including Hidden Markov Models and a Naïve Bayes Classifier, are applied to our simulator data to automatically differentiate users' expertise levels.

Representing fluid with smoothed particle hydrodynamics in a cranial base simulator.

We describe the implementation of irrigation and blood simulation using Smoothed Particle Hydrodynamics (SPH) in a cranial base surgical simulator. Graphical accuracy of virtual surgery is a significant goal for improving the realism and immersive experience of computerized training environments. For temporal bone micro-surgery fluids contribute not only to the visual integrity of the surgical field but provide relevant anatomic cues as well. The skill of 3-D sensory and navigation has become increasingly viable in surgery with the rising popularity of laparoscopic, catheter angiography and other minimally invasive approaches. The introduction of realistic simulated blood and irrigation enables the practice and coordination of two-handed microdissection techniques and the timing needed for safe bone removal and cautery.

Early experience with a patient-specific virtual surgical simulation for rehearsal of endoscopic skull-base surgery.

BACKGROUND: With the help of contemporary computer technology it is possible to create a virtual surgical environment (VSE) for training. This article describes a patient-specific virtual rhinologic surgical simulation platform that supports rehearsal of endoscopic skull-base surgery. We also share our early experience with select cases. METHODS: A rhinologic VSE was developed, featuring a highly efficient direct 3-dimensional (3D) volume renderer with simultaneous stereoscopic feedback during surgical manipulation of the virtual anatomy, as well as high-fidelity haptic feedback. We conducted a retrospective analysis on 10 patients who underwent various forms of sinus and ventral skull-base surgery to assess the ability of the rhinologic VSE to replicate actual intraoperative findings. RESULTS: In all 10 cases, the simulation experience was realistic enough to perform dissections in a similar manner as in the actual surgery. Excellent correlation was found in terms of surgical exposure, anatomical features, and the locations of pathology. CONCLUSION: The current rhinologic VSE shows sufficient realism to allow patient-specific surgical rehearsal of the sinus and ventral skull base. Further validation studies are needed to assess the benefits of performing patient-specific rehearsal.

Evaluating drilling and suctioning technique in a mastoidectomy simulator.

This paper presents several new metrics related to bone removal and suctioning technique in the context of a mastoidectomy simulator. The expertise with which decisions as to which regions of bone to remove and which to leave intact is evaluated by building a Naïve Bayes classifier using training data from known experts and novices. Since the bone voxel mesh is very large, and many voxels are always either removed or not removed regardless of expertise, the mutual information was calculated for each voxel and only the most informative voxels used for the classifier. Leave-out-one cross validation showed a high correlation of calculated expert probabilities with scores assigned by instructors. Additional metrics described in this paper include those for assessing smoothness of drill strokes, proper drill burr selection, sufficiency of suctioning, two-handed tool coordination, and application of appropriate force and velocity magnitudes as functions of distance from critical structures.

Validating metrics for a mastoidectomy simulator.

One of the primary barriers to the acceptance of surgical simulators is that most simulators still require a significant amount of an instructing surgeon's time to evaluate and provide feedback to the students using them. Thus, an important area of research in this field is the development of metrics that can enable a simulator to be an essentially self-contained teaching tool, capable of identifying and explaining the user's weaknesses. However, it is essential that these metrics be validated in able to ensure that the evaluations provided by the "virtual instructor" match those that the real instructor would provide were he/she present. We have previously proposed a number of algorithms for providing automated feedback in the context of a mastoidectomy simulator. In this paper, we present the results of a user study in which we attempted to establish construct validity (with inter-rater reliability) for our simulator itself and to validate our metrics. Fifteen subjects (8 experts, 7 novices) were asked to perform two virtual mastoidectomies. Each virtual procedure was recorded, and two experienced instructing surgeons assigned global scores that were correlated with subjects' experience levels. We then validated our metrics by correlating the scores generated by our algorithms with the instructors' global ratings, as well as with metric-specific sub-scores assigned by one of the instructors.

Anatomy-Specific Virtual Reality Simulation in Temporal Bone Dissection: Perceived Utility and Impact on Surgeon Confidence.

Objective To evaluate the effect of anatomy-specific virtual reality (VR) surgical rehearsal on surgeon confidence and temporal bone dissection performance. Study Design Prospective pre- and poststudy of a novel virtual surgical rehearsal platform. Setting Academic otolaryngology-head and neck surgery residency training programs. Subjects and Methods Sixteen otolaryngology-head and neck surgery residents from 2 North American training institutions were recruited. Surveys were administered to assess subjects' baseline confidence in performing 12 subtasks of cortical mastoidectomy with facial recess. A cadaver temporal bone was randomly assigned to each subject. Cadaver specimens were scanned with a clinical computed tomography protocol, allowing the creation of anatomy-specific models for use in a VR surgical rehearsal platform. Subjects then rehearsed a virtual mastoidectomy on data sets derived from their specimens. Surgical confidence surveys were administered again. Subjects then dissected assigned cadaver specimens, which were blindly graded with a modified Welling scale. A final survey assessed the perceived utility of rehearsal on dissection performance. Results Of 16 subjects, 14 (87.5%) reported a significant increase in overall confidence after conducting an anatomy-specific VR rehearsal. A significant correlation existed between perceived utility of rehearsal and confidence improvement. The effect of rehearsal on confidence was dependent on trainee experience and the inherent difficulty of the surgical subtask. Postrehearsal confidence correlated strongly with graded dissection performance. Subjects rated anatomy-specific rehearsal as having a moderate to high contribution to their dissection performance. Conclusion Anatomy-specific virtual rehearsal improves surgeon confidence in performing mastoid dissection, dependent on surgeon experience and task difficulty. The subjective confidence gained through rehearsal correlates positively with subsequent objective dissection performance.

Virtual 3D planning and guidance of mandibular distraction osteogenesis.

We present a system for 3D planning and pre-operative rehearsal of mandibular distraction osteogenesis procedures. Two primary architectural components are described: a planning system that allows geometric bone manipulation to rapidly explore various modifications and configurations, and a visuohaptic simulator that allows both general-purpose training and preoperative, patient-specific procedure rehearsal. We provide relevant clinical background, then describe the underlying simulation algorithms and their application to craniofacial procedures.

Achieving proper exposure in surgical simulation.

One important technique common throughout surgery is achieving proper exposure of critical anatomic structures so that their shapes, which may vary somewhat among patients, can be confidently established and avoided. In this paper, we present an algorithm for determining which regions of selected structures are properly exposed in the context of a mastoidectomy simulation. Furthermore, our algorithm then finds and displays all other points along the surface of the structure that lie along a sufficiently short and straight path from an exposed portion such that their locations can be safely inferred. Finally, we present an algorithm for providing realistic visual cues about underlying structures with view-dependent shading of the bone.

High-fidelity haptic and visual rendering for patient-specific simulation of temporal bone surgery.

Medical imaging techniques provide a wealth of information for surgical preparation, but it is still often the case that surgeons are examining three-dimensional pre-operative image data as a series of two-dimensional images. With recent advances in visual computing and interactive technologies, there is much opportunity to provide surgeons an ability to actively manipulate and interpret digital image data in a surgically meaningful way. This article describes the design and initial evaluation of a virtual surgical environment that supports patient-specific simulation of temporal bone surgery using pre-operative medical image data. Computational methods are presented that enable six degree-of-freedom haptic feedback during manipulation, and that simulate virtual dissection according to the mechanical principles of orthogonal cutting and abrasive wear. A highly efficient direct volume renderer simultaneously provides high-fidelity visual feedback during surgical manipulation of the virtual anatomy. The resulting virtual surgical environment was assessed by evaluating its ability to replicate findings in the operating room, using pre-operative imaging of the same patient. Correspondences between surgical exposure, anatomical features, and the locations of pathology were readily observed when comparing intra-operative video with the simulation, indicating the predictive ability of the virtual surgical environment.

Haptic feedback improves surgeons' user experience and fracture reduction in facial trauma simulation.

Computer-assisted surgical (CAS) planning tools are available for craniofacial surgery, but are usually based on computer-aided design (CAD) tools that lack the ability to detect the collision of virtual objects (i.e., fractured bone segments). We developed a CAS system featuring a sense of touch (haptic) that enables surgeons to physically interact with individual, patient-specific anatomy and immerse in a three-dimensional virtual environment. In this study, we evaluated initial user experience with our novel system compared to an existing CAD system. Ten surgery resident trainees received a brief verbal introduction to both the haptic and CAD systems. Users simulated mandibular fracture reduction in three clinical cases within a 15 min time limit for each system and completed a questionnaire to assess their subjective experience. We compared standard landmarks and linear and angular measurements between the simulated results and the actual surgical outcome and found that haptic simulation results were not significantly different from actual postoperative outcomes. In contrast, CAD results significantly differed from both the haptic simulation and actual postoperative results. In addition to enabling a more accurate fracture repair, the haptic system provided a better user experience than the CAD system in terms of intuitiveness and self-reported quality of repair.

An interactive simulation environment for craniofacial surgical procedures.

Recent advances in medical imaging and surgical techniques have made possible the correction of severe facial deformities and fractures. Surgical correction techniques often involve the direct manipulation - both relocation and surgical fracture - of the underlying facial bone. The work presented here introduces an environment for interactive, visuohaptic simulation of craniofacial surgical procedures, with an emphasis on both mandibular distraction procedures and traditional orthognathic surgeries. The simulator is intended both for instruction and for procedure-specific rehearsal, and can thus load canonical training cases or patient-specific image data into the interactive environment. A network module allows remote demonstration of procedure technique, a form of 'haptic tutoring'. This paper discusses the simulation, haptic feedback, and graphic rendering techniques used to drive the environment. Particular emphasis is placed on techniques for fracture and subsequent rigid manipulation of bone structures, a key component of the relevant procedures.

Quantifying risky behavior in surgical simulation.

Evaluating a trainee's performance on a simulated procedure involves determining whether a specified objective was met while avoiding certain "injurious" actions that damage vulnerable structures. However, it is also important to teach the stylistic behaviors that minimize overall risk to the patient, even though these criteria may be more difficult to explicitly specify and detect. In this paper, we address the development of metrics that evaluate the risk in a trainee's behavior while performing a simulated mastoidectomy. Specifically, we measure the trainee's ability to maintain an appropriate field of view so as to avoid drilling bone that is hidden from view, as well as to consistently apply appropriate forces and velocities. Models of the maximum safe force and velocity magnitudes as functions of distances from key vulnerable structures are learned from model procedures performed by an expert surgeon on the simulator. In addition to quantitatively scoring the trainee's performance, these metrics allow for interactive 3D visualization of the performance by distinctive coloring of regions in which excessive forces or velocities were applied or insufficient visibility was maintained, enabling the trainee to pinpoint his/her mistakes and how to correct them. Although these risky behaviors relate to a mastoidectomy simulator, the objectives of maintaining visibility and applying safe forces and velocities are common in surgery, so it may be possible to extend much of this methodology to other procedures.

Virtual reality simulation in neurosurgery: technologies and evolution.

Neurosurgeons are faced with the challenge of learning, planning, and performing increasingly complex surgical procedures in which there is little room for error. With improvements in computational power and advances in visual and haptic display technologies, virtual surgical environments can now offer potential benefits for surgical training, planning, and rehearsal in a safe, simulated setting. This article introduces the various classes of surgical simulators and their respective purposes through a brief survey of representative simulation systems in the context of neurosurgery. Many technical challenges currently limit the application of virtual surgical environments. Although we cannot yet expect a digital patient to be indistinguishable from reality, new developments in computational methods and related technology bring us closer every day. We recognize that the design and implementation of an immersive virtual reality surgical simulator require expert knowledge from many disciplines. This article highlights a selection of recent developments in research areas related to virtual reality simulation, including anatomic modeling, computer graphics and visualization, haptics, and physics simulation, and discusses their implication for the simulation of neurosurgery.

Online image-based monitoring of soft-tissue displacements for radiation therapy of the prostate.

PURPOSE: Emerging prolonged, hypofractionated radiotherapy regimens rely on high-dose conformality to minimize toxicity and thus can benefit from image guidance systems that continuously monitor target position during beam delivery. To address this need we previously developed, as a potential add-on device for existing linear accelerators, a novel telerobotic ultrasound system capable of real-time, soft-tissue imaging. Expanding on this capability, the aim of this work was to develop and characterize an image-based technique for real-time detection of prostate displacements. METHODS AND MATERIALS: Image processing techniques were implemented on spatially localized ultrasound images to generate two parameters representing prostate displacements in real time. In a phantom and five volunteers, soft-tissue targets were continuously imaged with a customized robotic manipulator while recording the two tissue displacement parameters (TDPs). Variations of the TDPs in the absence of tissue displacements were evaluated, as was the sensitivity of the TDPs to prostate translations and rotations. Robustness of the approach to probe force was also investigated. RESULTS: With 95% confidence, the proposed method detected in vivo prostate displacements before they exceeded 2.3, 2.5, and 2.8 mm in anteroposterior, superoinferior, and mediolateral directions. Prostate pitch was detected before exceeding 4.7° at 95% confidence. Total system time lag averaged 173 ms, mostly limited by ultrasound acquisition rate. False positives (FPs) (FP) in the absence of displacements did not exceed 1.5 FP events per 10 min of continuous in vivo imaging time. CONCLUSIONS: The feasibility of using telerobotic ultrasound for real-time, soft-tissue-based monitoring of target displacements was confirmed in vivo. Such monitoring has the potential to detect small clinically relevant intrafractional variations of the prostate position during beam delivery.

Reconstruction and exploration of virtual middle-ear models derived from micro-CT datasets.

BACKGROUND: Middle-ear anatomy is integrally linked to both its normal function and its response to disease processes. Micro-CT imaging provides an opportunity to capture high-resolution anatomical data in a relatively quick and non-destructive manner. However, to optimally extract functionally relevant details, an intuitive means of reconstructing and interacting with these data is needed. MATERIALS AND METHODS: A micro-CT scanner was used to obtain high-resolution scans of freshly explanted human temporal bones. An advanced volume renderer was adapted to enable real-time reconstruction, display, and manipulation of these volumetric datasets. A custom-designed user interface provided for semi-automated threshold segmentation. A 6-degrees-of-freedom navigation device was designed and fabricated to enable exploration of the 3D space in a manner intuitive to those comfortable with the use of a surgical microscope. Standard haptic devices were also incorporated to assist in navigation and exploration. RESULTS: Our visualization workstation could be adapted to allow for the effective exploration of middle-ear micro-CT datasets. Functionally significant anatomical details could be recognized and objective data could be extracted. CONCLUSIONS: We have developed an intuitive, rapid, and effective means of exploring otological micro-CT datasets. This system may provide a foundation for additional work based on middle-ear anatomical data.

Integration of patient-specific paranasal sinus computed tomographic data into a virtual surgical environment.

BACKGROUND: The advent of both high-resolution computed tomographic (CT) imaging and minimally invasive endoscopic techniques has led to revolutionary advances in sinus surgery. However, the rhinologist is left to make the conceptual jump between static cross-sectional images and the anatomy encountered intraoperatively. A three-dimensional (3D) visuo-haptic representation of the patient's anatomy may allow for enhanced preoperative planning and rehearsal, with the goal of improving outcomes, decreasing complications, and enhancing technical skills. METHODS: We developed a novel method of automatically constructing 3D visuo-haptic models of patients' anatomy from preoperative CT scans for placement in a virtual surgical environment (VSE). State-of-the-art techniques were used to create a high-fidelity representation of salient bone and soft tissue anatomy and to enable manipulation of the virtual patient in a surgically meaningful manner. A modified haptic interface device drives a virtual endoscope that mimics the surgical configuration. RESULTS: The creation and manipulation of sinus anatomy from CT data appeared to provide a relevant means of exploring patient-specific anatomy. Unlike more traditional methods of interacting with multiplanar imaging data, our VSE provides the potential for a more intuitive experience that can replicate the views and access expected at surgery. The inclusion of tactile (haptic) feedback provides an additional dimension of realism. CONCLUSION: The incorporation of patient-specific clinical CT data into a virtual surgical environment holds the potential to offer the surgeon a novel means to prepare for rhinologic procedures and offer training to residents. An automated pathway for segmentation, reconstruction, and an intuitive interface for manipulation may enable rehearsal of planned procedures.