A model for predicting the GEARS score from virtual reality surgical simulator metrics.

BACKGROUND: Surgical education relies heavily upon simulation. Assessment tools include robotic simulator assessments and Global Evaluative Assessment of Robotic Skills (GEARS) metrics, which have been validated. Training programs use GEARS for proficiency testing; however, it requires a trained human evaluator. Due to limited time, learners are reliant on surgical simulator feedback to improve their skills. GEARS and simulator scores have been shown to be correlated but in what capacity is unknown. Our goal is to develop a model for predicting GEARS score using simulator metrics. METHODS: Linear and multivariate logistic regressions were used on previously reported data by this group. Subjects performed simple (Ring and Rail 1) and complex (Suture Sponge 1) tasks on simulators, the dV-Trainer (dVT) and the da Vinci Skills Simulator (dVSS). They were scored via simulator metrics and GEARS. RESULTS: A linear model for each simulator and exercise showed a positive linear correlation. Equations were developed for predicting GEARS Total Score from simulator Overall Score. Next, the effects of each individual simulator metric on the GEARS Total Score for each simulator and exercise were examined. On the dVSS, Excessive Instrument Force was significant for Ring and Rail 1 and Instrument Collision was significant for Suture Sponge 1. On the dVT, Time to Complete was significant for both exercises. Once the significant variables were identified, multivariate models were generated. Comparing the predicted GEARS Total Score from the linear model (using only simulator Overall Score) to that using the multivariate model (using the significant variables for each simulator and exercise), the results were similar. CONCLUSIONS: Our results suggest that trainees can use simulator Overall Score to predict GEARS Total Score using our linear regression equations. This can improve the training process for those preparing for high-stakes assessments.

A Comparison of Robotic Simulation Performance on Basic Virtual Reality Skills: Simulator Subjective Versus Objective Assessment Tools.

STUDY OBJECTIVE: To answer the question of whether there is a difference between robotic virtual reality simulator performance assessment and validated human reviewers. Current surgical education relies heavily on simulation. Several assessment tools are available to the trainee, including the actual robotic simulator assessment metrics and the Global Evaluative Assessment of Robotic Skills (GEARS) metrics, both of which have been independently validated. GEARS is a rating scale through which human evaluators can score trainees' performances on 6 domains: depth perception, bimanual dexterity, efficiency, force sensitivity, autonomy, and robotic control. Each domain is scored on a 5-point Likert scale with anchors. We used 2 common robotic simulators, the dV-Trainer (dVT; Mimic Technologies Inc., Seattle, WA) and the da Vinci Skills Simulator (dVSS; Intuitive Surgical, Sunnyvale, CA), to compare the performance metrics of robotic surgical simulators with the GEARS for a basic robotic task on each simulator. DESIGN: A prospective single-blinded randomized study. SETTING: A surgical education and training center. PARTICIPANTS: Surgeons and surgeons in training. INTERVENTIONS: Demographic information was collected including sex, age, level of training, specialty, and previous surgical and simulator experience. Subjects performed 2 trials of ring and rail 1 (RR1) on each of the 2 simulators (dVSS and dVT) after undergoing randomization and warm-up exercises. The second RR1 trial simulator performance was recorded, and the deidentified videos were sent to human reviewers using GEARS. Eight different simulator assessment metrics were identified and paired with a similar performance metric in the GEARS tool. The GEARS evaluation scores and simulator assessment scores were paired and a Spearman rho calculated for their level of correlation. MEASUREMENTS AND MAIN RESULTS: Seventy-four subjects were enrolled in this randomized study with 9 subjects excluded for missing or incomplete data. There was a strong correlation between the GEARS score and the simulator metric score for time to complete versus efficiency, time to complete versus total score, economy of motion versus depth perception, and overall score versus total score with rho coefficients greater than or equal to 0.70; these were significant (p < .0001). Those with weak correlation (rho ≥0.30) were bimanual dexterity versus economy of motion, efficiency versus master workspace range, bimanual dexterity versus master workspace range, and robotic control versus instrument collisions. CONCLUSION: On basic VR tasks, several simulator metrics are well matched with GEARS scores assigned by human reviewers, but others are not. Identifying these matches/mismatches can improve the training and assessment process when using robotic surgical simulators.

Robotic surgery simulation validity and usability comparative analysis.

BACKGROUND: The introduction of simulation into minimally invasive robotic surgery is relatively recent and has seen rapid advancement; therefore, a need exists to develop training curriculums and identify systems that will be most effective at training surgical skills. Several simulators have been introduced to support these aims-the daVinci skills simulator, Mimic dV-Trainer, Surgical Simulated Systems' RoSS, and Simbionix Robotix Mentor. While multiple studies have been conducted to demonstrate the validity of these systems, studies comparing the perceived value of these devices as tools for education and skills are lacking. METHODS: Subjects who qualified as medical students or physicians (n = 105) were assigned a specific order to use each of the three simulators. After completing a demographic questionnaire, participants performed one exercise on the three simulators and completed a second questionnaire regarding their experience with the device. After using all systems, they completed a final questionnaire, which detailed their comparative preferences. The subject's performance metrics were also collected from each simulator. RESULTS: The data confirmed the face, content, and construct validity for the dV-trainer and skills simulator. Similar validities could not be confirmed for the RoSS. >80 % of the time, participants chose the skills simulator in terms of physical comfort, ergonomics, and overall choice. However, only 55 % thought the skills simulator was worth the cost of the equipment. The dV-Trainer had the highest cost preference scores with 71 % of respondents feeling it was worth the investment. CONCLUSIONS: Usability can affect the consistency and commitment of users of robotic surgical simulators. In a previous study, these simulators were objectively reviewed and compared in terms of their system capabilities. Collectively, this work will offer end-users and potential buyers a comparison of the perceived value and preferences of robotic simulators.

Design methodology for a simulator of a robotic surgical system.

Traditional spinal surgery procedures are completed with limited direct visualization. This imposes limitations on the surgeon's ability to place screws into the spine. The Mazor Renaissance robotic system was developed to improve the accuracy of pedicle screw insertion. Current training for this device comes with significant constraints. This suggests that a simulation-based solution may be valuable to the current training. This paper describes efforts to apply the theories of human-system integration (HSI) and instructional system design to define the requirements for a design of a simulator for specific robotic surgery system. From this, an instructional plan was conducted, to which an HSI-driven design document for a simulation system was developed. This paper describes the efforts to create a design method for a simulator of a specific robotic surgery system and provides a blended design process, which can be used during the early life cycle of any surgical simulation design.

A comparative analysis and guide to virtual reality robotic surgical simulators.

BACKGROUND: Since the US Food and Drug Administration approved robotically assisted surgical devices for human surgery in 2000, the number of surgeries utilizing this innovative technology has risen. In 2015, approximately 650 000 robot-assisted procedures were performed worldwide. Surgeons must be properly trained to safely transition to using such innovative technology. Multiple virtual reality robotic simulators are now commercially available for educational and training purposes. There is a need for comparative evaluations of these simulators to aid users in selecting an appropriate device for their purposes. METHODS: We conducted a comparison of the design and capabilities of all dedicated simulators of the da Vinci robot - the da Vinci Skills Simulator (dVSS), dV-Trainer (dVT), Robotic Skills Simulators (RoSS) and the RobotiX Mentor. This paper provides the base specifications of the hardware and software, with an emphasis on the training capabilities of each system. RESULTS: Each simulator contains a large number of training exercises for skills development: dVSS n = 40, dVT n = 65, RoSS n = 52, RobotiX Mentor n = 31. All four offer 3D visual images but use different display technologies. The dVSS leverages the real robotic surgical console to provide visualization, hand controls and foot pedals. The dVT, RoSS and RobotiX Mentor created simulated versions of all of these control systems. Each includes systems management services that allow instructors to collect, export and analyze the scores of students using the simulators. CONCLUSIONS: This study provides comparative information on the four simulators' functional capabilities. Each device offers unique advantages and capabilities for training robotic surgeons. Each has been the subject of validation experiments, which have been published in the literature. But those do not provide specific details on the capabilities of the simulators, which are necessary for an understanding sufficient to select the one best suited for an organization's needs. This article provides comparative information to assist with that type of selection.

Development and application of a multi-modal task analysis to support intelligent tutoring of complex skills.

BACKGROUND: Contemporary work in the design and development of intelligent training systems employs task analysis (TA) methods for gathering knowledge that is subsequently encoded into task models. These task models form the basis of intelligent interpretation of student performance within education and training systems. Also referred to as expert models, they represent the optimal way(s) of performing a training task. Within Intelligent Tutoring Systems (ITSs), real-time comparison of trainee task performance against the task model drives automated assessment and interactive support (such as immediate feedback) functionality. However, previous task analysis (TA) methods, including various forms of cognitive task analysis (CTA), may not be sufficient to support identification of the detailed design specifications required for the development of an ITS for a complex training task incorporating multiple underlying skill components, as well as multi-modal information presentation, assessment, and feedback modalities. Our current work seeks to develop an ITS for training Robotic Assisted Laparoscopic Surgery (RALS), a complex task domain that requires a coordinated utilization of integrated cognitive, psychomotor, and perceptual skills. RESULTS: In this paper, we describe a methodological extension to CTA, referred to as multi-modal task analysis (MMTA) that elicits and captures the nuances of integrated and isolated cognitive, psychomotor, and perceptual skill modalities as they apply to training and performing complex operational tasks. In the current case, we illustrate the application of the MMTA method described here to RALS training tasks. The products of the analysis are quantitatively summarized, and observations from a preliminary qualitative validation are reported. CONCLUSIONS: We find that iterative use of the described MMTA method leads to sufficiently complete and robust task models to support encoding of cognitive, psychomotor, and perceptual skills requisite to training and performance of complex skills within ITS task models.

Impact of delay on telesurgical performance: study on the robotic simulator dV-Trainer.

PURPOSE: To determine the impact of communication latency on telesurgical performance using the robotic simulator dV-Trainer®. METHODS: Surgeons were enrolled during three robotic congresses. They were randomly assigned to a delay group (ranging from 100 to 1000 ms). Each group performed three times a set of four exercises on the simulator: the first attempt without delay (Base) and the last two attempts with delay (Warm-up and Test). The impact of different levels of latency was evaluated. RESULTS: Thirty-seven surgeons were involved. The different latency groups achieved similar baseline performance with a mean task completion time of 207.2 s (p > 0.05). In the Test stage, the task duration increased gradually from 156.4 to 310.7 s as latency increased from 100 to 500 ms. In separate groups, the task duration deteriorated from Base for latency stages at delays ≥300 ms, and the errors increased at 500 ms and above (p < 0.05). The subjects' performance tended to improve from the Warm-up to the Test period. Few subjects completed the tasks with a delay higher than 700 ms. CONCLUSION: Gradually increasing latency has a growing impact on performances. Measurable deterioration of performance begins at 300 ms. Delays higher than 700 ms are difficult to manage especially in more complex tasks. Surgeons showed the potential to adapt to delay and may be trained to improve their telesurgical performance at lower-latency levels.

Crowd-Sourced Assessment of Technical Skills: Differentiating Animate Surgical Skill Through the Wisdom of Crowds.

BACKGROUND: Objective quantification of surgical skill is imperative as we enter a healthcare environment of quality improvement and performance-based reimbursement. The gold standard tools are infrequently used due to time-intensiveness, cost inefficiency, and lack of standard practices. We hypothesized that valid performance scores of surgical skill can be obtained through crowdsourcing. METHODS: Twelve surgeons of varying robotic surgical experience performed live porcine robot-assisted urinary bladder closures. Blinded video-recorded performances were scored by expert surgeon graders and by Amazon's Mechanical Turk crowdsourcing crowd workers using the Global Evaluative Assessment of Robotic Skills tool assessing five technical skills domains. Seven expert graders and 50 unique Mechanical Turkers (each paid $0.75/survey) evaluated each video. Global assessment scores were analyzed for correlation and agreement. RESULTS: Six hundred Mechanical Turkers completed the surveys in less than 5 hours, while seven surgeon graders took 14 days. The duration of video clips ranged from 2 to 11 minutes. The correlation coefficient between the Turkers' and expert graders' scores was 0.95 and Cronbach's Alpha was 0.93. Inter-rater reliability among the surgeon graders was 0.89. CONCLUSION: Crowdsourcing surgical skills assessment yielded rapid inexpensive agreement with global performance scores given by expert surgeon graders. The crowdsourcing method may provide surgical educators and medical institutions with a boundless number of procedural skills assessors to efficiently quantify technical skills for use in trainee advancement and hospital quality improvement.