RESUMO
STUDY OBJECTIVE: The contribution of emergency medicine clinicians' nontechnical skills in providing safe, high-quality care in the emergency department (ED) is well known. In 2015, the UK Royal College of Emergency Medicine introduced explicit validated descriptors of nontechnical skills needed to function effectively in the ED. A new nontechnical skills assessment tool that provided a score for 12 domains of nontechnical skills and detailed narrative feedback, the Extended Supervised Learning Event (ESLE), was introduced and was mandated as part of the Royal College of Emergency Medicine assessment schedule. We aim to evaluate the psychometric reliability of the ESLE in its first year of use. METHODS: ESLEs were mandated for all UK emergency medicine trainees in the final 4 years of a 6-year national training program from August 2015. The completed assessments were uploaded to the Royal College of Emergency Medicine e-portfolio. All assessments recorded in the Royal College of Emergency Medicine e-portfolio database between August 2015 and August 2016 were anonymized and analyzed for psychometric reliability, using generalizability theory. Decision analysis was used to model the effect of altering the number of episodes and assessors on reliability. RESULTS: A total of 1,390 ESLEs were analyzed. The majority (62%) of the variation in nontechnical skills scores was attributable to the trainee's ability. The circumstances of the event (eg, case complexity, workload) accounted for 21% and the stringency or leniency of assessors the remaining 16%. Decision analysis suggests that 3 ESLEs by 2 or more assessors, as currently recommended in the Royal College of Emergency Medicine curriculum, provide an assessment with a reliability coefficient of 0.8. CONCLUSION: Board-certified-equivalent emergency medicine supervisors are able to provide reliable assessments of emergency medicine trainees' nontechnical skills in the workplace by using the ESLE.