Impact of virtual reality training on mastoidectomy performance: a prospective randomised study.

PURPOSE: The opportunities for surgical training and practice in the operating room are in decline due to limited resources, increased efficiency demands, growing complexity of the cases, and concerns for patient safety. Virtual reality (VR) offers a novel opportunity to enhance surgical training and provide complementary three-dimensional experience that has been usually available in the operating room. Since VR allows viewing and manipulation of realistic 3D models, the VR environment could enhance anatomical and topographical knowledge, in particular. In this study, we explored whether incorporating VR anatomy training improves novices' performance during mastoidectomy over traditional methods. METHODS: Thirty medical students were randomized into two groups and taught mastoidectomy in a structured manner. One group utilized a VR temporal bone model during the training while the other group used more traditional materials such as anatomy books. After the training, all participants completed a mastoidectomy on a 3D-printed temporal bone model under expert supervision. Performance during the mastoidectomy was evaluated with multiple metrics and feedback regarding the two training methods was gathered from the participants. RESULTS: The VR training method was rated better by the participants, and they also needed less guidance during the mastoidectomy. There were no significant differences in operational time, the occurrence of injuries, self-assessment scores, and the surgical outcome between the two groups. CONCLUSION: Our results support the utilization of VR training in complete novices as it has higher trainee satisfaction and leads to at least as good results as the more traditional methods.

A three-dimensionally printed otological model for cholesteatoma mastoidectomy training.

PURPOSE: To relate the creation and expert validation (face and content validity) of an affordable three-dimensional (3-D) printed model of temporal bones with chronic otitis media with cholesteatoma (COMC) as a simulator for mastoidectomy. METHODS: We performed computed tomography (CT) of the temporal bones of a patient with COMC followed at the University of São Paulo (USP) Hospital with 3-D Slicer to create a 3-D model of the affected bone using light-curing resin and silicone (cholesteatoma). The final 3-D printed images were scored by 10 otologists using a customized version of the Michigan Standard Simulation Scale Experience (MiSSES). Internal consistency and inter-rater reliability were assessed using Cronbach's α and intraclass correlations. RESULTS: Otologists consistently scored the model positively for fidelity, educational value, reactions, and the overall model quality. Nine otologists agreed that the model was a good educational device for surgical training of COMC. All experts deemed the model ready-or nearly ready-for use. The final cost of the model, including raw materials and manufacturing, was 120 USD. CONCLUSIONS: Using 3-D printing technology, we created the first anatomically accurate, low-cost, disease-reproducing 3-D model of temporal bones for mastoidectomy training for cholesteatoma.

Am I doing this right? Structured self-assessment during simulation training of mastoidectomy improves cadaver dissection performance: a prospective educational study.

PURPOSE: Temporal bone surgery requires excellent surgical skills and simulation-based training can aid novices' skills acquisition. However, simulation-based training is challenged by early stagnation of performance after few performances. Structured self-assessment during practice might enhance learning by inducing reflection and engagement in the learning task. In this study, structured self-assessment was introduced during virtual reality (VR) simulation of mastoidectomy to investigate the effects on subsequent performance during cadaveric dissection. METHODS: A prospective educational study with comparison with historical controls (reference cohort). At a temporal bone dissection course, eighteen participants performed structured self-assessment during 3 h of VR simulation mastoidectomy training before proceeding to cadaver dissection (intervention cohort). At a previous course, eighteen participants received identical VR simulation training but without the structured self-assessment (reference cohort). Final products from VR simulation and cadaveric dissection were recorded and assessed by two blinded raters using a 19-point modified Welling Scale. RESULTS: The intervention cohort completed fewer procedures (average 4.2) during VR simulation training than the reference cohort (average 5.7). Nevertheless, the intervention cohort achieved a significantly higher average performance score both in VR simulation (11.1 points, 95% CI [10.6-11.5]) and subsequent cadaveric dissection (11.8 points, 95% CI [10.7-12.8]) compared with the reference cohort, who scored 9.1 points (95% CI [8.7-9.5]) during VR simulation and 5.8 points (95% CI [4.8-6.8]) during cadaveric dissection. CONCLUSIONS: Structured self-assessment is a valuable learning support during self-directed VR simulation training of mastoidectomy and the positive effect on performance transfers to subsequent cadaveric dissection performance.

Mastoidectomy Training: Is Anatomical Variation Needed? A Randomized, Controlled Trial on Performance and Skills Transfer From Virtual Reality to a Three-Dimensional Printed Model.

OBJECTIVE: Virtual reality (VR) simulation-based training effectively improves novices' mastoidectomy skills. Unfortunately, learning plateaus at an insufficient level and knowledge on optimizing mastoidectomy training to overcome this plateau is needed. In this study, we aim to investigate how training on anatomically different temporal bone cases affects learning, including the effect on retention and transfer of skills. STUDY DESIGN: Randomized controlled trial of an educational intervention. SETTING: The Simulation Center at Copenhagen Academy for Medical Education and Simulation. PARTICIPANTS: Twenty-four medical students from the University of Copenhagen. INTERVENTION: Participants were randomized to practice mastoidectomy on either 12 anatomically varying (intervention group) or 12 identical (control group) cases in a VR simulator. At the end of training and again ~ 3 weeks after training (retention), learners were tested on a new VR patient case and a three-dimensional printed model. MAIN OUTCOME MEASURE: Mastoidectomy performance evaluated by blinded expert raters using a 26-item modified Welling Scale. RESULTS: The intervention and control groups' performance results were comparable at the end of training. Likewise, retention and transfer performances were similar between groups. The overall mean score at the end of training corresponded to approximately 70% of the possible maximum score. CONCLUSIONS: Simulation-based training using anatomical variation was equivalent to training on a single case with respect to acquisition, retention, and transfer of mastoidectomy skills. This suggests that efforts to expose novices to variation during initial training are unnecessary as this variation has limited effect, and-conversely-that educators can expose novices to naturally different anatomical variations without worry of hindered learning.

Automated summative feedback improves performance and retention in simulation training of mastoidectomy: a randomised controlled trial.

OBJECTIVE: This study aimed to investigate the effects of automated metrics-based summative feedback on performance, retention and cognitive load in distributed virtual reality simulation training of mastoidectomy. METHOD: Twenty-four medical students were randomised in two groups and performed 15 mastoidectomies on a distributed virtual reality simulator as practice. The intervention group received additional summative metrics-based feedback; the control group followed standard instructions. Two to three months after training, participants performed a retention test without learning supports. RESULTS: The intervention group had a better final-product score (mean difference = 1.0 points; p = 0.001) and metrics-based score (mean difference = 12.7; p < 0.001). At retention, the metrics-based score for the intervention group remained superior (mean difference = 6.9 per cent; p = 0.02). Also at the retention, cognitive load was higher in the intervention group (mean difference = 10.0 per cent; p < 0.001). CONCLUSION: Summative metrics-based feedback improved performance and lead to a safer and faster performance compared with standard instructions and seems a valuable educational tool in the early acquisition of temporal bone skills.

Hand Motion Analysis Illustrates Differences When Drilling Cadaveric and Printed Temporal Bone.

OBJECTIVE: Temporal bone simulation is now commonly used to augment cadaveric education. Assessment of these tools is ongoing, with haptic modeling illustrating dissimilar motion patterns compared to cadaveric opportunities. This has the potential to result in maladaptive skill development. It is hypothesized that trainee drill motion patterns during printed model dissection may likewise demonstrate dissimilar hand motion patterns. METHODS: Resident surgeons dissected 3D-printed temporal bones generated from microCT data and cadaveric simulations. A magnetic position tracking system (TrakSTAR Ascension, Yarraville, Australia) captured drill position and orientation. Skill assessment included cortical mastoidectomy, thinning procedures (sigmoid sinus, dural plate, posterior canal wall) and facial recess development. Dissection was performed by 8 trainees (n = 5 < PGY3 > n = 3) using k-cos metrics to analyze drill strokes within position recordings. K-cos metrics define strokes by change in direction, providing metrics for stroke duration, curvature, and length. RESULTS: T-tests between models showed no significant difference in drill stroke frequency (cadaveric = 1.36/s, printed = 1.50/s, P < .40) but demonstrate significantly shorter duration (cadaveric = 0.37 s, printed = 0.16 s, P < .01) and a higher percentage of curved strokes (cadaveric = 31, printed = 67, P < .01) employed in printed bone dissection. Junior staff used a higher number of short strokes (junior = 0.54, senior = 0.38, P < .01) and higher percentage of curved strokes (junior = 35%, senior = 21%, P < .01). CONCLUSIONS: Significant differences in hand motions were present between simulations, however the significance is unclear. This may indicate that printed bone is not best positioned to be the principal training schema.

Automated assessment of cortical mastoidectomy performance in virtual reality.

INTRODUCTION: Cortical mastoidectomy is a core skill that Otolaryngology trainees must gain competency in. Automated competency assessments have the potential to reduce assessment subjectivity and bias, as well as reducing the workload for surgical trainers. OBJECTIVES: This study aimed to develop and validate an automated competency assessment system for cortical mastoidectomy. PARTICIPANTS: Data from 60 participants (Group 1) were used to develop and validate an automated competency assessment system for cortical mastoidectomy. Data from 14 other participants (Group 2) were used to test the generalisability of the automated assessment. DESIGN: Participants drilled cortical mastoidectomies on a virtual reality temporal bone simulator. Procedures were graded by a blinded expert using the previously validated Melbourne Mastoidectomy Scale: a different expert assessed procedures by Groups 1 and 2. Using data from Group 1, simulator metrics were developed to map directly to the individual items of this scale. Metric value thresholds were calculated by comparing automated simulator metric values to expert scores. Binary scores per item were allocated using these thresholds. Validation was performed using random sub-sampling. The generalisability of the method was investigated by performing the automated assessment on mastoidectomies performed by Group 2, and correlating these with scores of a second blinded expert. RESULTS: The automated binary score compared with the expert score per item had an accuracy, sensitivity and specificity of 0.9450, 0.9547 and 0.9343, respectively, for Group 1; and 0.8614, 0.8579 and 0.8654, respectively, for Group 2. There was a strong correlation between the total scores per participant assigned by the expert and calculated by the automatic assessment method for both Group 1 (r = .9144, P < .0001) and Group 2 (r = .7224, P < .0001). CONCLUSION: This study outlines a virtual reality-based method of automated assessment of competency in cortical mastoidectomy, which proved comparable to the assessment provided by human experts.

Patient-specific Virtual Temporal Bone Simulation Based on Clinical Cone-beam Computed Tomography.

OBJECTIVES: Patient-specific surgical simulation allows presurgical planning through three-dimensional (3D) visualization and virtual rehearsal. Virtual reality simulation for otologic surgery can be based on high-resolution cone-beam computed tomography (CBCT). This study aimed to evaluate clinicians' experience with patient-specific simulation of mastoid surgery. METHODS: Prospective, multi-institutional study. Preoperative temporal bone CBCT scans of patients undergoing cochlear implantation (CI) were retrospectively obtained. Automated processing and segmentation routines were used. Otologic surgeons performed a complete mastoidectomy with facial recess approach on the patient-specific virtual cases in the institution's temporal bone simulator. Participants completed surveys regarding the perceived accuracy and utility of the simulation. RESULTS: Twenty-two clinical CBCTs were obtained. Four attending otologic surgeons and 5 otolaryngology trainees enrolled in the study. The mean number of simulations completed by each participant was 16.5 (range 3-22). "Overall experience" and "usefulness for presurgical planning" were rated as "good," "very good," or "excellent" in 84.6% and 71.6% of the simulations, respectively. In 10.7% of simulations, the surgeon reported to have gained a significantly greater understanding of the patient's anatomy compared to standard imaging. Participants were able to better appreciate subtle anatomic findings after using the simulator for 60.4% of cases. Variable CBCT acquisition quality was the most reported limitation. CONCLUSION: Patient-specific simulation using preoperative CBCT is feasible and may provide valuable insights prior to otologic surgery. Establishing a CBCT acquisition protocol that allows for consistent segmentation will be essential for reliable surgical simulation. LEVEL OF EVIDENCE: 3 Laryngoscope, 131:1855-1862, 2021.

Operable, Low-Cost, High-Resolution, Patient-Specific 3D Printed Temporal Bones for Surgical Simulation and Evaluation.

OBJECTIVES: Three-dimensional printed models created on a consumer level printer can be used to practice mastoidectomy and to discern mastoidectomy experience level. Current models in the literature for mastoidectomy are limited by expense or operability. The aims of this study were (1) to investigate the utility of an inexpensive model for mastoidectomy and (2) to assess whether the model can be used as an evaluation tool to discern the experience level of the surgeon performing mastoidectomy. METHODS: Three-dimensional printed temporal bone models from the CT scan of a 7-year old patient were created using a consumer-level stereolithography 3D printer for a raw material cost of $10 each. Mastoidectomy with facial recess approach was performed by 4 PGY-2 residents, 4 PGY-5 residents, and 4 attending surgeons on the models who then filled out an evaluation. The drilled models were collected and then graded in a blinded fashion by 6 attending otolaryngologists. RESULTS: Both residents and faculty felt the model was useful for training (mean score 4.7 out of 5; range: 4-5) and case preparation (mean score: 4.3; range: 3-5). Grading of the drilled models revealed significant differences between junior resident, senior resident, and attending surgeon scores (P = .012) with moderate to excellent interrater agreement (ICC = 0.882). CONCLUSION: The described operable model that is patient-specific was rated favorably for pediatric mastoidectomy case preparation and training by residents and faculty. The model may be used to differentiate between experience levels and has promise for use in formative and summative evaluations.

Using Intraoperative Recordings to Evaluate Surgical Technique and Performance in Mastoidectomy.

Importance: Otolaryngology residency programs currently lack rigorous methods for assessing surgical skill and often rely on biased tools of evaluation. Objectives: To evaluate which techniques used in mastoidectomy can serve as indicators of surgeon level (defined as the level of training) and whether these determinations of technique can be made based solely on the movement of the drill head or suction. Design, Setting, and Participants: In this prospective, observational study conducted from January 1, 2015, to December 31, 2019, at a single tertiary care institution, 3 independent observers made blinded evaluations on 24 intraoperative recordings of surgeons (6 junior residents, 4 senior residents, and 2 attending surgeons) performing mastoidectomies. Main Outcomes and Measures: Observers assessed drill stroke count, drilling efficiency, stroke pattern, use of suction and irrigation, and estimated surgeon level. Assessments were made on both original videos and animated videos that show only the path of the burr head or suction as dots against a white background. Results: Among the 24 recorded mastoidectomies performed by the 12 study surgeons, intraclass correlation was excellent for original video assessment of drill stroke count (0.98 [95% CI, 0.97-1.00]), use of suction (0.75 [95% CI, 0.52-0.89]), use of irrigation (0.83 [95% CI, 0.66-0.92]), and estimated surgeon level (0.82 [95% CI, 0.64-0.92]) and fair for drilling efficiency (0.54 [95% CI, 0.09-0.79]) and stroke pattern (0.49 [95% CI, -0.02 to 0.76]). Intraclass correlation was excellent for animated video assessment of drill stroke count per unit time (0.98 [95% CI, 0.96-0.99]) and drilling efficiency (0.80 [95% CI, 0.60-0.91]), good for stroke pattern (0.68 [95% CI, 0.38-0.85]) and estimated surgeon level (based on path of drill) (0.69 [95% CI, 0.38-0.85]), and fair for use of suction (0.58 [95% CI, 0.16-0.80]) and estimated surgeon level (based on path of suction) (0.58 [95% CI, 0.17-0.80]). On evaluation of original videos, junior residents had lower drill stroke count compared with senior residents and attending surgeons (6.0 [interquartile range (IQR), 3.0-8.0] vs 9.5 [IQR, 5.0-13.0] vs 10.5 [IQR, 5.0-17.8]; η2 = 0.14 [95% CI, 0.01-0.28]). On evaluation of animated videos, junior residents also had lower drill stroke count compared with senior residents and attending surgeons (6.0 [IQR, 4.0-9.0] vs 10.5 [IQR, 10.0-13.8] vs 10.5 [IQR, 4.3-21.0]; η2 = 0.19 [95% CI, 0.04-0.33]). Compared with junior and senior residents, attending surgeons had higher median ratings of drilling efficiency (original videos: junior residents, 4.0 [IQR, 3.0-4.0]; senior residents, 4.0 [IQR, 3.0-4.8]; attending surgeons, 5.0 [IQR, 4.3-5.0]; η2 = 0.23 [95% CI, 0.06-0.37]; animated videos: junior residents, 4.0 [IQR, 3.0-4.0]; senior residents, 3.0 [IQR, 2.0-4.0]; attending surgeons, 5.0 [IQR, 4.0-5.0]; η2 = 0.25 [95% CI, 0.08-0.39]) and stroke pattern (original videos: junior residents, 4.0 [IQR, 3.0-4.0]; senior residents, 4.0 [IQR, 3.0-4.8]; attending surgeons, 5.0 [IQR, 5.0-5.0]; η2 = 0.17 [95% CI, 0.03-0.31]; animated videos: junior residents, 4.0 [IQR, 3.0-4.0]; senior residents, 4.0 [IQR, 2.0-4.0]; attending surgeons, 5.0 [IQR, 5.0-5.0]; η2 = 0.15 [95% CI, 0.02-0.29]). Conclusions and Relevance: This study suggests that observation of intraoperative mastoidectomy recordings is a feasible method of evaluating surgeon level. Reasonable indicators of surgeon level include the drill stroke count, drilling efficiency, stroke pattern, and use of the suction irrigator. Observing the path of the drill alone is sufficient to appreciate differences in drilling technique but not sufficient to accurately determine surgeon level. Intraoperative recordings can serve as a useful addition to resident education and evaluation.

The Melbourne Mastoidectomy Scale: Validation of an end-product dissection scale for cortical mastoidectomy.

INTRODUCTION: Cortical mastoidectomy is a common otolaryngology procedure and represents a compulsory part of otolaryngology training. As such, a specific validated assessment score is needed for the progression of competency-based training in this procedure. Although multiple temporal bone dissection scales have been developed, they have all been validated for advanced temporal bone dissection including posterior tympanotomy, rather than the task of cortical mastoidectomy. METHODS: The Melbourne Mastoidectomy Scale, a 20-item end-product dissection scale to assess cortical mastoidectomy, was developed. The scale was validated using dissections by 30 participants (10 novice, 10 intermediate and 10 expert) on a virtual reality temporal bone simulator. All dissections were assessed independently by three blinded graders. Additionally, all procedures were graded with an abbreviated Welling Scale by one grader. RESULTS: There was high inter-rater reliability between the three graders (r = .9210, P < .0001). There was a significant difference in scores between the three groups (P < .0001). Additionally, there was a large effect size between all three groups: the differences between the novice group and both the intermediate group (P = .0119, η2  = 0.2482) and expert group (P < .001, η2  = 0.6356) were significant. The difference between the intermediate group and expert group again had a large effect size (η2  = 0.3217), but was not significant. The Melbourne Mastoidectomy Scale correlated well with an abbreviated Welling Scale (r = .8485, P < .0001). CONCLUSION: The Melbourne Mastoidectomy Scale offers a validated score for use in the assessment of cortical mastoidectomy.

Comparison of Materials Used for 3D-Printing Temporal Bone Models to Simulate Surgical Dissection.

OBJECTIVE: To identify 3D-printed temporal bone (TB) models that most accurately recreate cortical mastoidectomy for use as a training tool by comparison of different materials and fabrication methods. BACKGROUND: There are several different printers and materials available to create 3D-printed TB models for surgical planning and trainee education. Current reports using Acrylonitrile Butadiene Styrene (ABS) plastic generated via fused deposition modeling (FDM) have validated the capacity for 3D-printed models to serve as accurate surgical simulators. Here, a head-to-head comparison of models produced using different materials and fabrication processes was performed to identify superior models for application in skull base surgical training. METHODS: High-resolution CT scans of normal TBs were used to create stereolithography files with image conversion for application in 3D-printing. The 3D-printed models were constructed using five different materials and four printers, including ABS printed on a MakerBot 2x printer, photopolymerizable polymer (Photo) using the Objet 350 Connex3 Printer, polycarbonate (PC) using the FDM-Fortus 400 mc printer, and two types of photocrosslinkable acrylic resin, white and blue (FLW and FLB, respectively), using the Formlabs Form 2 stereolithography printer. Printed TBs were drilled to assess the haptic experience and recreation of TB anatomy with comparison to the current paradigm of ABS. RESULTS: Surgical drilling demonstrated that FLW models created by FDM as well as PC and Photo models generated using photopolymerization more closely recreated cortical mastoidectomy compared to ABS models. ABS generated odor and did not represent the anatomy accurately. Blue resin performed poorly in simulation, likely due to its dark color and translucent appearance. CONCLUSIONS: PC, Photo, and FLW models best replicated surgical drilling and anatomy as compared to ABS and FLB models. These prototypes are reliable simulators for surgical training.

A time-sensitive rubric for assessing mastoidectomy proficiency.

OBJECTIVE: To develop a time-sensitive, standardized rubric for cadaveric temporal bone dissection for otolaryngology resident education. METHODS: This is a five-year prospective cohort study that evaluated otolaryngology resident performance during sequential cadaveric temporal bone dissection courses at a single otolaryngology residency training program. A canal-wall-up mastoidectomy with a facial recess approach was performed adhering to a 30-minute time limit and graded according to a standardized rubric. Main outcome measures included: (1) correct structure identification and (2) injuries sustained to structures as compared by resident post-graduate year (PGY) level. RESULTS: Thirteen residents were evaluated from October 2012 to March 2017. This included 57 individual graded exercises performed over ten dissection courses. The average score for PGY-2 residents was lowest (68.9), and PGY-5 residents achieved the highest average score (87.7). Junior residents correctly identified fewer structures (77.5%) when compared to senior residents (91.3%), p < 0.0001. Correct performance of a facial recess approach was achieved by 100% of senior residents, but only 59.3% of junior residents (p = 0.0003). The percentage of major injuries, which included the facial nerve, tegmen, labyrinth, and ossicular chain, decreased each PGY-level from a maximum of 17% by PGY-2 residents to a minimum of 5% by PGY-5 residents. CONCLUSION: Senior residents correctly identify more structures and are able to complete a facial recess approach with higher fidelity when subjected to a time-sensitive graded mastoidectomy rubric.

3D printed temporal bone as a tool for otologic surgery simulation.

PURPOSE: In this face validity study, we discuss the fabrication and utility of an affordable, computed tomography (CT)-based, anatomy-accurate, 3-dimensional (3D) printed temporal bone models for junior otolaryngology resident training. MATERIALS AND METHODS: After IRB exemption, patient CT scans were anonymized and downloaded as Digital Imaging and Communications in Medicine (DICOM) files to prepare for conversion. These files were converted to stereolithography format for 3D printing. Important soft tissue structures were identified and labeled to be printed in a separate color than bone. Models were printed using a desktop 3D printer (Ultimaker 3 Extended, Ultimaker BV, Netherlands) and polylactic acid (PLA) filament. 10 junior residents with no previous drilling experience participated in the study. Each resident was asked to drill a simple mastoidectomy on both a cadaveric and 3D printed temporal bone. Following their experience, they were asked to complete a Likert questionnaire. RESULTS: The final result was an anatomically accurate (XYZ accuracy = 12.5, 12.5, 5 µm) 3D model of a temporal bone that was deemed to be appropriate in tactile feedback using the surgical drill. The total cost of the material required to fabricate the model was approximately $1.50. Participants found the 3D models overall to be similar to cadaveric temporal bones, particularly in overall value and safety. CONCLUSIONS: 3D printed temporal bone models can be used as an affordable and inexhaustible alternative, or supplement, to traditional cadaveric surgical simulation.

The Effect of Simulator-Integrated Tutoring for Guidance in Virtual Reality Simulation Training.

INTRODUCTION: Simulation-integrated tutoring in virtual reality (VR) simulation training by green lighting is a common learning support in simulation-based temporal bone surgical training. However, tutoring overreliance can negatively affect learning. We therefore wanted to investigate the effects of simulator-integrated tutoring on performance and learning. METHODS: A prospective, educational cohort study of a learning intervention (simulator-integrated tutoring) during repeated and distributed VR simulation training for directed, self-regulated learning of the mastoidectomy procedure. Two cohorts of novices (medical students) were recruited: 16 participants were trained using the intervention program (intermittent simulator-integrated tutoring) and 14 participants constituted a nontutored reference cohort. Outcomes were final-product performance assessed by 2 blinded raters and simulator-recorded metrics. RESULTS: Simulator-integrated tutoring had a large and positive effect on the final-product performance while turned on (mean difference = 3.8 points, P < 0.0001). However, this did not translate to a better final-product performance in subsequent nontutored procedures. The tutored cohort had a better metrics-based score, reflecting higher efficiency of drilling (mean difference = 3.6%, P = 0.001). For the individual metrics, simulator-integrated tutoring had mixed effects both during procedures and on the tutored cohort in general (learning effect). CONCLUSIONS: Simulator-integrated tutoring by green lighting did not induce a better final-product performance but increased efficiency. The mixed effects on learning could be caused by tutoring overreliance, resulting from a lack of cognitive engagement when the tutor function is on. Further learning strategies such as feedback should be explored to support novice learning and cognitive engagement.

Standard Setting of Competency in Mastoidectomy for the Cross-Institutional Mastoidectomy Assessment Tool.

OBJECTIVE: Competency-based surgical training involves progressive autonomy given to the trainee. This requires systematic and evidence-based assessment with well-defined standards of proficiency. The objective of this study is to develop standards for the cross-institutional mastoidectomy assessment tool to inform decisions regarding whether a resident demonstrates sufficient skill to perform a mastoidectomy with or without supervision. METHODS: A panel of fellowship-trained content experts in mastoidectomy was surveyed in relation to the 16 items of the assessment tool to determine the skills needed for supervised and unsupervised surgery. We examined the consensus score to investigate the degree of agreement among respondents for each survey item as well as additional analyses to determine whether the reported skill level required for each survey item was significantly different for the supervised versus unsupervised level. RESULTS: Ten panelists representing different US training programs responded. There was considerable consensus on cut-off scores for each item and trainee level between panelists, with moderate (0.62) to very high (0.95) consensus scores depending on assessment item. Further analyses demonstrated that the difference between supervised and unsupervised skill levels was significantly meaningful for all items. Finally, minimum-passing scores for each item was established. CONCLUSION: We defined performance standards for the cross-institutional mastoidectomy assessment tool using the Angoff method. These cut-off scores that can be used to determine when trainees can progress from performance under supervision to performance without supervision. This can be used to guide training in a competency-based training curriculum.

The effect of structured self-assessment in virtual reality simulation training of mastoidectomy.

PURPOSE: Virtual reality (VR) simulation surgical skills training is well established, but self-directed practice is often associated with a learning curve plateau. In this study, we investigate the effects of structured self-assessment as a means to improve performance in mastoidectomy training. METHODS: The study was a prospective, educational study. Two cohorts of novices (medical students) were recruited for practice of anatomical mastoidectomy in a training program with five distributed training blocks. Fifteen participants performed structured self-assessment after each procedure (intervention cohort). A reference cohort of another 14 participants served as controls. Performances were assessed by two blinded raters using a modified Welling Scale and simulator-recorded metrics. RESULTS: The self-assessment cohort performed superiorly to the reference cohort (mean difference of final product score 0.87 points, p = 0.001) and substantially reduced the number of repetitions needed. The self-assessment cohort also had more passing performances for the combined metrics-based score reflecting increased efficiency. Finally, the self-assessment cohort made fewer collisions compared with the reference cohort especially with the chorda tympani, the facial nerve, the incus, and the malleus. CONCLUSIONS: VR simulation training of surgical skills benefits from having learners perform structured self-assessment following each procedure as this increases performance, accelerates the learning curve thereby reducing time needed for training, and induces a safer performance with fewer collisions with critical structures. Structured self-assessment was in itself not sufficient to counter the learning curve plateau and for continued skills development additional supports for deliberate practice are needed.

Decentralized virtual reality mastoidectomy simulation training: a prospective, mixed-methods study.

PURPOSE: Virtual reality (VR) training of mastoidectomy is effective in surgical training-particularly if organized as distributed practice. However, centralization of practice facilities is a barrier to implementation of distributed simulation training. Decentralized training could be a potential solution. Here, we aim to assess the feasibility, use, and barriers to decentralized VR mastoidectomy training using a freeware, high-fidelity temporal bone simulator. METHODS: In a prospective, mixed-methods study, 20 otorhinolaryngology residents were given three months of local access to a VR mastoidectomy simulator. Additionally, trainees were provided a range of learning supports for directed, self-regulated learning. Questionnaire data were collected and focus group interviews conducted. The interviews were analyzed using thematic analysis and compared with quantitative findings. RESULTS: Participants trained 48.5 h combined and mainly towards the end of the trial. Most participants used between two and four different learning supports. Qualitative analysis revealed five main themes regarding implementation of decentralized simulation training: convenience, time for training, ease of use, evidence for training, and testing. CONCLUSIONS: Decentralized VR training using a freeware, high-fidelity mastoidectomy simulator is feasible but did not lead to a high training volume or truly distributed practice. Evidence for training was found motivational. Access to training, educational designs, and the role of testing are important for participant motivation and require further evaluation.

Performance metrics in mastoidectomy training: a systematic review.

OBJECTIVE: To investigate validity evidence, and strengths and limitations of performance metrics in mastoidectomy training. METHODS: A systematic review following the PRISMA guidelines. Studies reporting performance metrics in mastoidectomy/temporal bone surgery were included. Data on design, outcomes, and results were extracted by two reviewers. Validity evidence according to Messick's framework and level of evidence were assessed. RESULTS: The search yielded a total of 1085 studies from the years 1947-2018 and 35 studies were included for full data extraction after abstract and full-text screening. 33 different metrics on mastoidectomy performance were identified and ranked according to the number of reports. Most of the 33 metrics identified had some amount of validity evidence. The metrics with most validity evidence were related to drilling time, volume drilled per time, force applied near vital structures, and volume removed. CONCLUSIONS: This review provides an overview of current metrics of mastoidectomy performance, their validity, strengths and limitations, and identifies the gap in validity evidence of some metrics. Evidence-based metrics can be used for performance assessment in temporal bone surgery and for providing integrated and automated feedback in virtual reality simulation training. The use of such metrics in simulation-based mastoidectomy training can potentially address some of the limitations in current temporal bone skill assessment and ease assessment in repeated practice. However, at present, an automated feedback based on metrics in VR simulation does not have sufficient empirical basis and has not been generally accepted for use in training and certification. LEVEL OF EVIDENCE: 2a.

The utility of virtual reality surgical simulation in the undergraduate otorhinolaryngology curriculum.

OBJECTIVE: To examine the impact of temporal bone virtual reality surgical simulator use in the undergraduate otorhinolaryngology curriculum. METHODS: Medical students attended a workshop involving the use of a temporal bone virtual reality surgical simulator. Students completed a pre-workshop questionnaire on career interests. A post-workshop questionnaire evaluated the perceived usefulness and enjoyment of the virtual reality surgical simulator experience, and assessed changes in their interest in ENT. RESULTS: Thirty-two fifth-year University of Auckland medical students were recruited. The majority of students (53.1 per cent) had already chosen their career path. The simulator experience was useful for: stimulating thoughts around career plans (71.9 per cent), providing hands-on experience (93.8 per cent) and teaching disease processes (93.8 per cent). After the workshop, 53.1 per cent of students were more interested in a career in ENT. CONCLUSION: Virtual reality may be a fun and engaging way of teaching ENT. Furthermore, it could help guide student career planning.