Computer-assisted simulated workplace-based assessment in surgery: application of the universal framework of intraoperative performance within a mixed-reality simulation.

Objectives: Workplace-based assessment (WBA) is a key requirement of competency-based medical education in postgraduate surgical education. Although simulated workplace-based assessment (SWBA) has been proposed to complement WBA, it is insufficiently adopted in surgical education. In particular, approaches to criterion-referenced and automated assessment of intraoperative surgical competency in contextualized SWBA settings are missing.Main objectives were (1) application of the universal framework of intraoperative performance and exemplary adaptation to spine surgery (vertebroplasty); (2) development of computer-assisted assessment based on criterion-referenced metrics; and (3) implementation in contextualized, team-based operating room (OR) simulation, and evaluation of validity. Design: Multistage development and assessment study: (1) expert-based definition of performance indicators based on framework's performance domains; (2) development of respective assessment metrics based on preoperative planning and intraoperative performance data; (3) implementation in mixed-reality OR simulation and assessment of surgeons operating in a confederate team. Statistical analyses included internal consistency and interdomain associations, correlations with experience, and technical and non-technical performances. Setting: Surgical simulation center. Full surgical team set-up within mixed-reality OR simulation. Participants: Eleven surgeons were recruited from two teaching hospitals. Eligibility criteria included surgical specialists in orthopedic, trauma, or neurosurgery with prior VP or kyphoplasty experience. Main outcome measures: Computer-assisted assessment of surgeons' intraoperative performance. Results: Performance scores were associated with surgeons' experience, observational assessment (Objective Structured Assessment of Technical Skill) scores and overall pass/fail ratings. Results provide strong evidence for validity of our computer-assisted SWBA approach. Diverse indicators of surgeons' technical and non-technical performances could be quantified and captured. Conclusions: This study is the first to investigate computer-assisted assessment based on a competency framework in authentic, contextualized team-based OR simulation. Our approach discriminates surgical competency across the domains of intraoperative performance. It advances previous automated assessment based on the use of current surgical simulators in decontextualized settings. Our findings inform future use of computer-assisted multidomain competency assessments of surgeons using SWBA approaches.

Three-dimensional-Printed Computed Tomography-Based Bone Models for Spine Surgery Simulation.

INTRODUCTION: We present a novel 3-dimensional (3D) printing method for low-cost and widely available reproduction of computed tomography (CT)-based synthetic bone models for spine surgery simulation, optimized to reproduce realistic haptic properties. The method allows reproduction of either normal or abnormal patient anatomy. The models are fluoroscopy compatible and contain deformities and fractures present in the underlying CT data. METHODS: Spine models created from CT data were printed on a 3D printer using 2 different materials for cortical and cancellous bone. Printing parameters were iteratively optimized with surgical experts and 3 candidate spine models were evaluated in a study regarding haptic properties. X-ray images of a spine section printed with final printing parameters were evaluated by surgical experts regarding fluoroscopic properties. RESULTS: Eleven surgical experts performed a trocar insertion, a typical workflow step in spine surgery procedures, on the models. We observed agreement that cortical structures and strong agreement that cancellous structures of the final model are haptically comparable with human vertebral bone. Ten surgical experts evaluated x-ray images of the model. They expressed strong agreement on the similarity with x-ray images of the human spine and confirmed the presence of a fracture. Material cost of a typical spine model is around US $11. CONCLUSIONS: Models created using the novel methodology realistically reproduce the haptic properties during a trocar placement into the vertebral body. The models are compatible with conventional x-ray imaging. Because the models correspond to real patient CT data, those can alternatively be used in simulation environments that simulate fluoroscopy or CT image guidance to produce highly realistic, radiation-free imaging output.

Technical and Nontechnical Skills in Surgery: A Simulated Operating Room Environment Study.

STUDY DESIGN: Observational simulation study. OBJECTIVE: The goal of this study was to investigate the relationship between technical and nontechnical skills (NTS) in a simulated surgical procedure. SUMMARY OF BACKGROUND DATA: Although surgeons' technical and NTS during surgery are crucial determinants for clinical outcomes, little literature is available in spine surgery. Moreover, evidence regarding how surgeons' technical and NTS are related is limited. METHODS: A mixed-reality and full-scale simulated operating room environment was employed for the surgical team. Eleven surgeons performed the vertebroplasty procedure (VP). Technical skills (TS) were assessed using Objective Structured Assessment of Technical Skill scores and senior expert-evaluated VP outcome assessment. NTS were assessed with the Observational Teamwork Assessment for Surgery. Kendall-Tau-b tests were performed for correlations. We further controlled the influence of surgeons' experience (based on professional tenure and number of previous VPs performed). RESULT: Surgeons' NTS correlated significantly with their technical performance (τ = 0.63; P = 0.006) and surgical outcome scores (τ = 0.60; P = 0.007). This association was attenuated when controlling for surgeons' experience. CONCLUSION: Our results suggest that spine surgeons with higher levels of TS also apply better communication, leadership, and coordination behaviors during the procedure. Yet, the role of surgeons' experience needs further investigation for improving surgeons' intraoperative performance during spine surgery. LEVEL OF EVIDENCE: 3.

Say, What Is on Your Mind? Surgeons' Evaluations of Realism and Usability of a Virtual Reality Vertebroplasty Simulator.

BACKGROUND: Virtual reality (VR)-based simulations offer rich opportunities for surgical skill training and assessment of surgical novices and experts. A structured evaluation and validation process of such training and assessment tools is necessary for effective surgical learning environments. OBJECTIVE: To develop and apply a classification system of surgeon-reported experience during operation of a VR vertebroplasty simulator. METHODS: A group of orthopedic, trauma surgeons and neurosurgeons (n = 13) with various levels of expertise performed on a VR vertebroplasty simulator. We established a mixed-methods design using think-aloud protocols, senior surgical expert evaluations, performance metrics, and a post-simulation questionnaire. Verbal content was systematically analyzed using structured qualitative content analysis. We established a category system for classification of surgeons' verbal evaluations during the simulation. Furthermore, we evaluated intraoperative performance metrics and explored potential associations with surgeons' characteristics and simulator evaluation. RESULTS: Overall, 244 comments on realism and usability of the vertebroplasty simulator were collected. This included positive and negative remarks, questions, and specific suggestions for improvement. Further findings included surgeons' approval of the realism and usability of the simulator and the observation that the haptic feedback of the VR patient's anatomy requires further improvement. Surgeon-reported evaluations were not associated with performance decrements. DISCUSSION: This study is the first to apply think-aloud protocols for evaluation of a surgical VR-based simulator. A novel classification approach is introduced that can be used to classify surgeons' verbalized experiences during simulator use. Our lessons learned may be valuable for future research with similar methodological approach.

Stepwise development of a simulation environment for operating room teams: the example of vertebroplasty.

BACKGROUND: Despite the growing importance of medical simulation in education, there is limited guidance available on how to develop medical simulation environments, particularly with regard to technical and non-technical skills as well as to multidisciplinary operating room (OR) team training. We introduce a cognitive task analysis (CTA) approach consisting of interviews, structured observations, and expert consensus to systematically elicit information for medical simulator development. Specifically, our objective was to introduce a guideline for development and application of a modified CTA to obtain task demands of surgical procedures for all three OR professions with comprehensive definitions of OR teams' technical and non-technical skills. METHODS: To demonstrate our methodological approach, we applied it in vertebroplasty, a minimally invasive spine procedure. We used a CTA consisting of document reviews, in situ OR observations, expert interviews, and an expert consensus panel. Interviews included five surgeons, four OR nurses, and four anesthetists. Ten procedures were observed. Data collection was carried out in five OR theaters in Germany. RESULTS: After compiling data from interviews and observations, we identified 6 procedural steps with 21 sub-steps for surgeons, 20 sub-steps for nurses, and 22 sub-steps for anesthetists. Additionally, we obtained information on 16 predefined categories of intra-operative skills and requirements for all three OR professions. Finally, simulation requirements for intra-operative demands were derived and specified in the expert panel. CONCLUSIONS: Our CTA approach is a feasible and effective way to elicit information on intra-operative demands and to define requirements of medical team simulation. Our approach contributes as a guideline to future endeavors developing simulation training of technical and non-technical skills for multidisciplinary OR teams.

Associations of Intraoperative Flow Disruptions and Operating Room Teamwork During Robotic-assisted Radical Prostatectomy.

OBJECTIVE: To identify type and severity of surgical flow disruptions and to determine their impact on the perception of intraoperative teamwork. METHODS: Forty radical prostatectomy cases were studied in an academic department for urology. A standardized observational tool for identification of type and severity of flow disruptions was applied during real-time prostatectomy procedures. Additionally, all operating room team members evaluated intraoperative teamwork immediately after the procedure. Procedures were divided into 4 phases: prerobot, docking, console time, and postrobot. RESULTS: A total of 2012 flow disruptions were observed, with an average rate of 16.27 events per hour. The highest rate was during the robot docking phase. Although the frequency of disruption types varied across phases, the most severe disruptions were related to communication and coordination during the prerobot and docking phase. Equipment- and communication-related disruptions were mostly severe during the time the surgeons were on the console. Among the surgeons, we identified a significant relationship between disruptions and intraoperative teamwork such that during procedures with frequent severe disruptions, surgeons experienced inferior teamwork (ß = -0.40, P = .01). This was not the case for nurses and anesthetists. CONCLUSION: Emphasis on improving operating room team communication and coordination would help to establish efficient and smooth surgical workflow.

Virtual reality-based simulators for spine surgery: a systematic review.

BACKGROUND CONTEXT: Virtual reality (VR)-based simulators offer numerous benefits and are very useful in assessing and training surgical skills. Virtual reality-based simulators are standard in some surgical subspecialties, but their actual use in spinal surgery remains unclear. Currently, only technical reviews of VR-based simulators are available for spinal surgery. PURPOSE: Thus, we performed a systematic review that examined the existing research on VR-based simulators in spinal procedures. We also assessed the quality of current studies evaluating VR-based training in spinal surgery. Moreover, we wanted to provide a guide for future studies evaluating VR-based simulators in this field. STUDY DESIGN AND SETTING: This is a systematic review of the current scientific literature regarding VR-based simulation in spinal surgery. METHODS: Five data sources were systematically searched to identify relevant peer-reviewed articles regarding virtual, mixed, or augmented reality-based simulators in spinal surgery. A qualitative data synthesis was performed with particular attention to evaluation approaches and outcomes. Additionally, all included studies were appraised for their quality using the Medical Education Research Study Quality Instrument (MERSQI) tool. RESULTS: The initial review identified 476 abstracts and 63 full texts were then assessed by two reviewers. Finally, 19 studies that examined simulators for the following procedures were selected: pedicle screw placement, vertebroplasty, posterior cervical laminectomy and foraminotomy, lumbar puncture, facet joint injection, and spinal needle insertion and placement. These studies had a low-to-medium methodological quality with a MERSQI mean score of 11.47 out of 18 (standard deviation=1.81). CONCLUSIONS: This review described the current state and applications of VR-based simulator training and assessment approaches in spinal procedures. Limitations, strengths, and future advancements of VR-based simulators for training and assessment in spinal surgery were explored. Higher-quality studies with patient-related outcome measures are needed. To establish further adaptation of VR-based simulators in spinal surgery, future evaluations need to improve the study quality, apply long-term study designs, and examine non-technical skills, as well as multidisciplinary team training.