Reliability of simulation-based assessment for practicing physicians: performance is context-specific.

INTRODUCTION: Even physicians who routinely work in complex, dynamic practices may be unprepared to optimally manage challenging critical events. High-fidelity simulation can realistically mimic critical clinically relevant events, however the reliability and validity of simulation-based assessment scores for practicing physicians has not been established. METHODS: Standardised complex simulation scenarios were developed and administered to board-certified, practicing anesthesiologists who volunteered to participate in an assessment study during formative maintenance of certification activities. A subset of the study population agreed to participate as the primary responder in a second scenario for this study. The physicians were assessed independently by trained raters on both teamwork/behavioural and technical performance measures. Analysis using Generalisability and Decision studies were completed for the two scenarios with two raters. RESULTS: The behavioural score was not more reliable than the technical score. With two raters > 20 scenarios would be required to achieve a reliability estimate of 0.7. Increasing the number of raters for a given scenario would have little effect on reliability. CONCLUSIONS: The performance of practicing physicians on simulated critical events may be highly context-specific. Realistic simulation-based assessment for practicing physicians is resource-intensive and may be best-suited for individualized formative feedback. More importantly, aggregate data from a population of participants may have an even higher impact if used to identify skill or knowledge gaps to be addressed by training programs and inform continuing education improvements across the profession.

Simulation-based Assessment of the Management of Critical Events by Board-certified Anesthesiologists.

BACKGROUND: We sought to determine whether mannequin-based simulation can reliably characterize how board-certified anesthesiologists manage simulated medical emergencies. Our primary focus was to identify gaps in performance and to establish psychometric properties of the assessment methods. METHODS: A total of 263 consenting board-certified anesthesiologists participating in existing simulation-based maintenance of certification courses at one of eight simulation centers were video recorded performing simulated emergency scenarios. Each participated in two 20-min, standardized, high-fidelity simulated medical crisis scenarios, once each as primary anesthesiologist and first responder. Via a Delphi technique, an independent panel of expert anesthesiologists identified critical performance elements for each scenario. Trained, blinded anesthesiologists rated video recordings using standardized rating tools. Measures included the percentage of critical performance elements observed and holistic (one to nine ordinal scale) ratings of participant's technical and nontechnical performance. Raters also judged whether the performance was at a level expected of a board-certified anesthesiologist. RESULTS: Rater reliability for most measures was good. In 284 simulated emergencies, participants were rated as successfully completing 81% (interquartile range, 75 to 90%) of the critical performance elements. The median rating of both technical and nontechnical holistic performance was five, distributed across the nine-point scale. Approximately one-quarter of participants received low holistic ratings (i.e., three or less). Higher-rated performances were associated with younger age but not with previous simulation experience or other individual characteristics. Calling for help was associated with better individual and team performance. CONCLUSIONS: Standardized simulation-based assessment identified performance gaps informing opportunities for improvement. If a substantial proportion of experienced anesthesiologists struggle with managing medical emergencies, continuing medical education activities should be reevaluated.

Decision-Making During High-Risk Events: A Systematic Literature Review.

Effective decision-making in crisis events is challenging due to time pressure, uncertainty, and dynamic decisional environments. We conducted a systematic literature review in PubMed and PsycINFO, identifying 32 empiric research papers that examine how trained professionals make naturalistic decisions under pressure. We used structured qualitative analysis methods to extract key themes. The studies explored different aspects of decision-making across multiple domains. The majority (19) focused on healthcare; military, fire and rescue, oil installation, and aviation domains were also represented. We found appreciable variability in research focus, methodology, and decision-making descriptions. We identified five main themes: (1) decision-making strategy, (2) time pressure, (3) stress, (4) uncertainty, and (5) errors. Recognition-primed decision-making (RPD) strategies were reported in all studies that analyzed this aspect. Analytical strategies were also prominent, appearing more frequently in contexts with less time pressure and explicit training to generate multiple explanations. Practitioner experience, time pressure, stress, and uncertainty were major influencing factors. Professionals must adapt to the time available, types of uncertainty, and individual skills when making decisions in high-risk situations. Improved understanding of these decisional factors can inform evidence-based enhancements to training, technology, and process design.

A Taxonomy of Delivery and Documentation Deviations During Delivery of High-Fidelity Simulations.

INTRODUCTION: We developed a taxonomy of simulation delivery and documentation deviations noted during a multicenter, high-fidelity simulation trial that was conducted to assess practicing physicians' performance. Eight simulation centers sought to implement standardized scenarios over 2 years. Rules, guidelines, and detailed scenario scripts were established to facilitate reproducible scenario delivery; however, pilot trials revealed deviations from those rubrics. A taxonomy with hierarchically arranged terms that define a lack of standardization of simulation scenario delivery was then created to aid educators and researchers in assessing and describing their ability to reproducibly conduct simulations. METHODS: Thirty-six types of delivery or documentation deviations were identified from the scenario scripts and study rules. Using a Delphi technique and open card sorting, simulation experts formulated a taxonomy of high-fidelity simulation execution and documentation deviations. The taxonomy was iteratively refined and then tested by 2 investigators not involved with its development. RESULTS: The taxonomy has 2 main classes, simulation center deviation and participant deviation, which are further subdivided into as many as 6 subclasses. Inter-rater classification agreement using the taxonomy was 74% or greater for each of the 7 levels of its hierarchy. Cohen kappa calculations confirmed substantial agreement beyond that expected by chance. All deviations were classified within the taxonomy. CONCLUSIONS: This is a useful taxonomy that standardizes terms for simulation delivery and documentation deviations, facilitates quality assurance in scenario delivery, and enables quantification of the impact of deviations upon simulation-based performance assessment.

Assessing student usage, perception, and the utility of a Web-based simulation in a third-year medical school clerkship.

STUDY OBJECTIVE: The goals of this study were to assess students' usage data of Web-based simulation (WBS), to determine if it can fill gaps in clinical experience-based medical education, and to determine students' perceived value of this kind of simulation during a clinical clerkship. DESIGN: Observational/prospective cohort. SETTING: Medical school affiliated with a large academic hospital. SUBJECTS: A total od 138 medical students. INTERVENTIONS: Web-based simulation. MEASUREMENTS: Medical students in an anesthesiology clerkship were assigned a WBS focusing on the clinical use of pulmonary artery catheters (PACs). Usage data, including day of week and time of day that the simulation was used and total usage time, were collected for 99 students. Eighty voluntary survey responses, which gauged student perception of the simulation and clinical exposure to PACs, were also collected. MAIN RESULTS: Seventy-two percent of attempts were made during nonclinical hours of 5 pm to 7 am. Seventy-seven percent of students spent less than 30 minutes in total using the simulation. Students preferred the simulation (rated 4.1/5) over textbook (3.59) learning to a statistically significant degree (P < .001) and held favorable views toward the simulation. Sixty-seven percent of students had never encountered a patient with a PAC before performing the simulation, and 41% did not discuss this learning objective during their clerkship. Students' self-rated understanding of PACs significantly increased from a presimulation score of 1.8 of 5, to 2.56 (mean difference, 0.760; P < .001) after using the simulation. CONCLUSIONS: WBS in medical school clerkships is accepted by students and can fill gaps in clinical medical school education, without negatively affecting students' workloads or clerkship experiences.

A Clinical Procedures Course for Medical Students.

INTRODUCTION: Doctors perform many clinical procedures throughout their careers. It is important for students to learn these procedures in a nonthreatening environment. This clinical procedures course introduces students to several basic diagnostic and therapeutic procedures, both invasive and noninvasive. These include managing pediatric and adult airways, starting intravenous lines, inserting arterial and central lines, inserting Foley catheters and nasogastric tubes, and performing lumbar punctures and paracentesis. METHODS: Small-group teaching is used to achieve these objectives; over the course of 4 weeks, the medical students meet once a week for 4 hours. Each meeting includes teaching and demonstrations of the procedures by faculty instructors and residents. This is followed by practice of the procedures on mannequin simulators and partial task trainers by the students. Feedback is then given to the students by the instructors. RESULTS: Based on conversations during the feedback sessions, the students feel that the materials used in the course are helpful in learning these clinical procedures. DISCUSSION: The medical students feel that the course familiarizes them with clinical procedures they may be asked to perform on patients during their clinical rotations and postgraduate training.

Simulation for maintenance of certification in anesthesiology: the first two years.

The ultimate goal of physician education is the application of knowledge and skills to patient care. The Maintenance of Certification (MOC) for Anesthesiologists program incorporates mannequin-based simulation to help realize this goal. Results from the first 2 years of experience suggest that 583 physician participants transferred knowledge and skills from their simulated experiences into real-world practice. Participants consistently found the experience educationally valuable and clinically relevant, and reported that it led to changes in practice. This first experience with mannequin-based simulation for MOC indicates that physicians accept this teaching modality, many with enthusiasm. Simulation education addresses many of the identified intentions of current continuing medical education (CME) and can help educators realize goals for educating physician-learners.