Developing an Evidence-Based Surgical Curriculum: Learning from a Randomized Controlled Trial of Surgical Rehearsal in Virtual Reality.

BACKGROUND: Surgical rehearsal - patient-specific preoperative surgical practice - can be provided by virtual reality simulation. This study investigated the effect of surgical rehearsal on cortical mastoidectomy performance and procedure duration. METHODS: University students (n=40) were randomized evenly into a rehearsal and control group. After watching a video tutorial on cortical mastoidectomy, participants completed the procedure on a virtual reality simulator as a pre-test. Participants completed a further 8 cortical mastoidectomies on the virtual reality simulator as training before drilling two 3-dimensional (3D) printed temporal bones. The rehearsal group received 3D printed bones they had previously operated on in virtual reality, while the control group received 2 new bones. Cortical mastoidectomy was assessed by 3 blinded graders using the Melbourne Mastoidectomy Scale. RESULTS: There was high interrater reliability between the 3 graders (intraclass correlation coefficient, r=0.8533, P < .0001). There was no difference in the mean surgical performance on the two 3D printed bones between the control and rehearsal groups (P=.2791). There was no significant difference in the mean procedure duration between the control and rehearsal groups for both 3D printed bones (P=.8709). However, there was a significant decrease in procedure duration between the first and second 3D printed bones (P < .0001). CONCLUSION: In this study, patient-specific virtual reality rehearsal provided no additional advantage to cortical mastoidectomy performance by novice operators compared to generic practice on a virtual reality simulator. Further, virtual reality training did not improve cortical mastoidectomy performance on 3D printed bones, highlighting the impact of anatomical diversity and changing operating modalities on the acquisition of new surgical skills.

Automated assessment of cortical mastoidectomy performance in virtual reality.

INTRODUCTION: Cortical mastoidectomy is a core skill that Otolaryngology trainees must gain competency in. Automated competency assessments have the potential to reduce assessment subjectivity and bias, as well as reducing the workload for surgical trainers. OBJECTIVES: This study aimed to develop and validate an automated competency assessment system for cortical mastoidectomy. PARTICIPANTS: Data from 60 participants (Group 1) were used to develop and validate an automated competency assessment system for cortical mastoidectomy. Data from 14 other participants (Group 2) were used to test the generalisability of the automated assessment. DESIGN: Participants drilled cortical mastoidectomies on a virtual reality temporal bone simulator. Procedures were graded by a blinded expert using the previously validated Melbourne Mastoidectomy Scale: a different expert assessed procedures by Groups 1 and 2. Using data from Group 1, simulator metrics were developed to map directly to the individual items of this scale. Metric value thresholds were calculated by comparing automated simulator metric values to expert scores. Binary scores per item were allocated using these thresholds. Validation was performed using random sub-sampling. The generalisability of the method was investigated by performing the automated assessment on mastoidectomies performed by Group 2, and correlating these with scores of a second blinded expert. RESULTS: The automated binary score compared with the expert score per item had an accuracy, sensitivity and specificity of 0.9450, 0.9547 and 0.9343, respectively, for Group 1; and 0.8614, 0.8579 and 0.8654, respectively, for Group 2. There was a strong correlation between the total scores per participant assigned by the expert and calculated by the automatic assessment method for both Group 1 (r = .9144, P < .0001) and Group 2 (r = .7224, P < .0001). CONCLUSION: This study outlines a virtual reality-based method of automated assessment of competency in cortical mastoidectomy, which proved comparable to the assessment provided by human experts.

The Melbourne Mastoidectomy Scale: Validation of an end-product dissection scale for cortical mastoidectomy.

INTRODUCTION: Cortical mastoidectomy is a common otolaryngology procedure and represents a compulsory part of otolaryngology training. As such, a specific validated assessment score is needed for the progression of competency-based training in this procedure. Although multiple temporal bone dissection scales have been developed, they have all been validated for advanced temporal bone dissection including posterior tympanotomy, rather than the task of cortical mastoidectomy. METHODS: The Melbourne Mastoidectomy Scale, a 20-item end-product dissection scale to assess cortical mastoidectomy, was developed. The scale was validated using dissections by 30 participants (10 novice, 10 intermediate and 10 expert) on a virtual reality temporal bone simulator. All dissections were assessed independently by three blinded graders. Additionally, all procedures were graded with an abbreviated Welling Scale by one grader. RESULTS: There was high inter-rater reliability between the three graders (r = .9210, P < .0001). There was a significant difference in scores between the three groups (P < .0001). Additionally, there was a large effect size between all three groups: the differences between the novice group and both the intermediate group (P = .0119, η2  = 0.2482) and expert group (P < .001, η2  = 0.6356) were significant. The difference between the intermediate group and expert group again had a large effect size (η2  = 0.3217), but was not significant. The Melbourne Mastoidectomy Scale correlated well with an abbreviated Welling Scale (r = .8485, P < .0001). CONCLUSION: The Melbourne Mastoidectomy Scale offers a validated score for use in the assessment of cortical mastoidectomy.