Reliability of simulation-based assessment for practicing physicians: performance is context-specific.

INTRODUCTION: Even physicians who routinely work in complex, dynamic practices may be unprepared to optimally manage challenging critical events. High-fidelity simulation can realistically mimic critical clinically relevant events, however the reliability and validity of simulation-based assessment scores for practicing physicians has not been established. METHODS: Standardised complex simulation scenarios were developed and administered to board-certified, practicing anesthesiologists who volunteered to participate in an assessment study during formative maintenance of certification activities. A subset of the study population agreed to participate as the primary responder in a second scenario for this study. The physicians were assessed independently by trained raters on both teamwork/behavioural and technical performance measures. Analysis using Generalisability and Decision studies were completed for the two scenarios with two raters. RESULTS: The behavioural score was not more reliable than the technical score. With two raters > 20 scenarios would be required to achieve a reliability estimate of 0.7. Increasing the number of raters for a given scenario would have little effect on reliability. CONCLUSIONS: The performance of practicing physicians on simulated critical events may be highly context-specific. Realistic simulation-based assessment for practicing physicians is resource-intensive and may be best-suited for individualized formative feedback. More importantly, aggregate data from a population of participants may have an even higher impact if used to identify skill or knowledge gaps to be addressed by training programs and inform continuing education improvements across the profession.

Clinical Uses and Impacts of Emergency Manuals During Perioperative Crises.

BACKGROUND: Performing key actions efficiently during crises can determine clinical outcomes, yet even expert clinicians omit key actions. Simulation-based studies of crises show that correct performance of key actions dramatically increases when emergency manuals (EMs) are used. Despite widespread dissemination of EMs, there is a need to understand in clinical contexts, when, how, and how often EMs are used and not used, along with perceived impacts. METHODS: We conducted interviews with the anesthesia professionals involved in perioperative crises, identified with criterion-based sampling, occurring between October 2014 and May 2016 at 2 large academic medical centers with a history of EM training and implementation. Our convergent, mixed-methods study of the interview data extracted quantitative counts and qualitative themes of EM use and nonuse during clinical crises. RESULTS: Interviews with 53 anesthesia professionals yielded 80 descriptions of applicable clinical crises, with varying durations and event types. Of 69 unique patients whose cases involved crises, the EM was used during 37 (54%; 95% confidence interval [CI], 41-66). Impacts on clinician team members included decreased stress for individual anesthesia professionals (95%), enabled teamwork (73%), and calmed atmosphere (46%). Impacts on delivery of patient care included specific action improvements, including catching errors of omission, for example, turning off anesthetic during cardiac arrest, only after EM use (59%); process improvements, for example, double-checking all actions were completed (41%); and impediments (0%). In 8% of crises, EM use was associated with potential distractions, although none were perceived to harm delivery of patient care. For 32 EM nonuses (46%; 95% CI, 34-59), participants self-identified errors of omission or delays in key actions (56%), all key actions performed (13%), and crisis too brief for EM to be used (31%). CONCLUSIONS: This study provides evidence that EMs in operating rooms are being used during many applicable crises and that clinicians perceive EM use to add value. The reported negative effects were minimal and potentially offset by positive effects.

Use of an Emergency Manual During an Intraoperative Cardiac Arrest by an Interprofessional Team: A Positive-Exemplar Case Study of a New Patient Safety Tool.

BACKGROUND: An emergency manual (EM) is a set of evidence-based crisis checklists, or cognitive aids, that can improve team performance. EMs are used in other safety-critical industries, and health care simulation studies have shown their efficacy, but use in clinical settings is nascent. A case study was conducted on the use of an EM during one intraoperative crisis, which entailed the assessment of the impact of the EM's use on teamwork and patient care and the identification of lessons for effectively using EMs during future clinical crises. METHODS: In a case study of a single crisis, an EM was used during a cardiac arrest at a tertiary care hospital that had systematically implemented perioperative EMs. Semistructured interviews were conducted with all six clinicians present, interview transcripts were iteratively coded, and thematic analysis was performed. RESULTS: All clinician participants stated that EM use enabled effective team functioning via reducing stress of individual clinicians, fostering a calm work environment, and improving teamwork and communication. These impacts in turn improved the delivery of patient care during a clinical crisis and influenced participants' intended EM use during future appropriate crises. CONCLUSION: In this positive-exemplar case study, an EM was used to improve delivery of evidence-based patient care through effective clinical team functioning. EM use must complement rather than replace good clinician education, judgment, and teamwork. More broadly, understanding why and how things go well via analyzing positive-exemplar case studies, as a converse of root cause analyses for negative events, can be used to identify effective applications of safety innovations.

Simulation-based Assessment of the Management of Critical Events by Board-certified Anesthesiologists.

BACKGROUND: We sought to determine whether mannequin-based simulation can reliably characterize how board-certified anesthesiologists manage simulated medical emergencies. Our primary focus was to identify gaps in performance and to establish psychometric properties of the assessment methods. METHODS: A total of 263 consenting board-certified anesthesiologists participating in existing simulation-based maintenance of certification courses at one of eight simulation centers were video recorded performing simulated emergency scenarios. Each participated in two 20-min, standardized, high-fidelity simulated medical crisis scenarios, once each as primary anesthesiologist and first responder. Via a Delphi technique, an independent panel of expert anesthesiologists identified critical performance elements for each scenario. Trained, blinded anesthesiologists rated video recordings using standardized rating tools. Measures included the percentage of critical performance elements observed and holistic (one to nine ordinal scale) ratings of participant's technical and nontechnical performance. Raters also judged whether the performance was at a level expected of a board-certified anesthesiologist. RESULTS: Rater reliability for most measures was good. In 284 simulated emergencies, participants were rated as successfully completing 81% (interquartile range, 75 to 90%) of the critical performance elements. The median rating of both technical and nontechnical holistic performance was five, distributed across the nine-point scale. Approximately one-quarter of participants received low holistic ratings (i.e., three or less). Higher-rated performances were associated with younger age but not with previous simulation experience or other individual characteristics. Calling for help was associated with better individual and team performance. CONCLUSIONS: Standardized simulation-based assessment identified performance gaps informing opportunities for improvement. If a substantial proportion of experienced anesthesiologists struggle with managing medical emergencies, continuing medical education activities should be reevaluated.

Practice improvements based on participation in simulation for the maintenance of certification in anesthesiology program.

BACKGROUND: This study describes anesthesiologists' practice improvements undertaken during the first 3 yr of simulation activities for the Maintenance of Certification in Anesthesiology Program. METHODS: A stratified sampling of 3 yr (2010-2012) of participants' practice improvement plans was coded, categorized, and analyzed. RESULTS: Using the sampling scheme, 634 of 1,275 participants in Maintenance of Certification in Anesthesiology Program simulation courses were evaluated from the following practice settings: 41% (262) academic, 54% (339) community, and 5% (33) military/other. A total of 1,982 plans were analyzed for completion, target audience, and topic. On follow-up, 79% (1,558) were fully completed, 16% (310) were partially completed, and 6% (114) were not completed within the 90-day reporting period. Plans targeted the reporting individual (89% of plans) and others (78% of plans): anesthesia providers (50%), non-anesthesia physicians (16%), and non-anesthesia non-physician providers (26%). From the plans, 2,453 improvements were categorized as work environment or systems changes (33% of improvements), teamwork skills (30%), personal knowledge (29%), handoff (4%), procedural skills (3%), or patient communication (1%). The median word count was 63 (interquartile range, 30 to 126) for each participant's combined plans and 147 (interquartile range, 52 to 257) for improvement follow-up reports. CONCLUSIONS: After making a commitment to change, 94% of anesthesiologists participating in a Maintenance of Certification in Anesthesiology Program simulation course successfully implemented some or all of their planned practice improvements. This compares favorably to rates in other studies. Simulation experiences stimulate active learning and motivate personal and collaborative practice improvement changes. Further evaluation will assess the impact of the improvements and further refine the program.

Evaluation of a Standardized Program for Training Practicing Anesthesiologists in Ultrasound-Guided Regional Anesthesia Skills.

OBJECTIVES: Practicing anesthesiologists have generally not received formal training in ultrasound-guided perineural catheter insertion. We designed this study to determine the efficacy of a standardized teaching program in this population. METHODS: Anesthesiologists in practice for 10 years or more were recruited and enrolled to participate in a 1-day program: lectures and live-model ultrasound scanning (morning) and faculty-led iterative practice and mannequin-based simulation (afternoon). Participants were assessed and recorded while performing ultrasound-guided perineural catheter insertion at baseline, at midday (interval), and after the program (final). Videos were scored by 2 blinded reviewers using a composite tool and global rating scale. Participants were surveyed every 3 months for 1 year to report the number of procedures, efficacy of teaching methods, and implementation obstacles. RESULTS: Thirty-two participants were enrolled and completed the program; 31 of 32 (97%) completed the 1-year follow-up. Final scores [median (10th-90th percentiles)] were 21.5 (14.5-28.0) of 30 points compared to 14.0 (9.0-20.0) at interval (P < .001 versus final) and 12.0 (8.5-17.5) at baseline (P < .001 versus final), with no difference between interval and baseline. The global rating scale showed an identical pattern. Twelve of 26 participants without previous experience performed at least 1 perineural catheter insertion after training (P < .001). However, there were no differences in the monthly average number of procedures or complications after the course when compared to baseline. CONCLUSIONS: Practicing anesthesiologists without previous training in ultrasound-guided regional anesthesia can acquire perineural catheter insertion skills after a 1-day standardized course, but changing clinical practice remains a challenge.

Towards meaningful simulation-based learning with medical students and junior physicians.

BACKGROUND: This research provides an educational perspective on simulation-based medical education by implementing both the characteristics of meaningful learning and the concepts of facilitating, training, and learning processes. AIMS: This study aims to evaluate, from the perspectives of both facilitators and students, the meaningfulness of five different simulation-based courses. METHODS: The courses were implemented in the spring of 2010. The data were collected from facilitators (n = 9) and students (n = 25) using group interviews (one individual interview), observations, video recordings, and pre- and post-questionnaires. The research analyzes qualitative data using the qualitative content analysis method to answer the following research question: From facilitators' and students' perspectives, how does the facilitating and training in simulation-based learning environments (SBLEs) foster the meaningful learning of students? RESULTS: It seems that simulation-based learning is, at its foundation, meaningful since it inherently supports the many characteristics of meaningful learning. However, characteristics also exist that simulation-based learning does not inherently support. In this study, the goal-oriented, self-directed, and individual training characteristics were only somewhat supported during the facilitation and training in SBLEs. CONCLUSIONS: In running these courses in the future, facilitators should concentrate on those characteristics that were only somewhat supported.

Improving patient care through leadership engagement with frontline staff: a Department of Veterans Affairs case study.

BACKGROUND: Leveraging Frontline Expertise (LFLE) is a patient safety intervention for engaging senior managers with the work-systems challenges faced by frontline workers and ensuring follow-up and accountability for systemic change. A study was conducted to assess the ability to refine, implement, and demonstrate the effectiveness of LFLE, which was designed for and tested in private-sector hospitals, in a Department of Veterans Affairs medical center (VAMC), typically a more hierarchical setting. METHODS: LFLE was pilot tested in an urban, East coast-based VAMC, which implemented LFLE in its emergency department and operating room, with the medical/surgical ward and ICU serving as controls. A 20-month multimethod evaluation involved interviews, observation, data-tracking forms, and surveys to measure participant perceptions of the program, operational benchmarks of effectiveness, and longitudinal change in safety climate. RESULTS: Implementation showed fidelity to program design. Participating units identified 22 improvement opportunities, 16 (73%) of which were fully or partially resolved. Senior managers' attitudes toward LFLE were more positive than those of frontline staff, whose attitudes were mixed. Perceptions of safety climate deteriorated during the study period in the implementation units relative to controls. DISCUSSION: LFLE can be implemented in the VA, yield work-system improvements, and increase alignment of improvement aims and actions across hierarchical levels. Yet the results also warn against dangers inherent in adapting improvement programs to new settings. Findings suggest the need for active listening and learning from frontline staff by senior managers and trust building across hierarchical

Collaborative Use of Lung Mechanics Simulation for Testing and Iterative Design for Three Emergency Use Ventilation Device Projects.

SUMMARY STATEMENT: We describe our collaboration with engineering, clinical, and simulation colleagues to use a lung simulator (IngMar Medical ASL 5000) to aid in the development of 3 open-source ventilation devices for patients with COVID-19.Twenty-nine test conditions were created by programming software lung models of varying disease severity in the ASL 5000 to test basic functionality, safety features, and compliance with regulatory requirements for emergency use authorization for the 3 projects' prototypes. More than 200 simulations were performed, with the design team present to enable rapid troubleshooting and design iteration in real time.Working with 3 separate simultaneous ventilation device projects allowed us to rapidly learn from each, improving our ability to successfully collaborate with the different design/build teams.This project illustrates the role of simulation in facilitating collaborative innovation in health care, both in emergency and everyday settings that extend beyond the COVID-19 pandemic.

Decision-Making During High-Risk Events: A Systematic Literature Review.

Effective decision-making in crisis events is challenging due to time pressure, uncertainty, and dynamic decisional environments. We conducted a systematic literature review in PubMed and PsycINFO, identifying 32 empiric research papers that examine how trained professionals make naturalistic decisions under pressure. We used structured qualitative analysis methods to extract key themes. The studies explored different aspects of decision-making across multiple domains. The majority (19) focused on healthcare; military, fire and rescue, oil installation, and aviation domains were also represented. We found appreciable variability in research focus, methodology, and decision-making descriptions. We identified five main themes: (1) decision-making strategy, (2) time pressure, (3) stress, (4) uncertainty, and (5) errors. Recognition-primed decision-making (RPD) strategies were reported in all studies that analyzed this aspect. Analytical strategies were also prominent, appearing more frequently in contexts with less time pressure and explicit training to generate multiple explanations. Practitioner experience, time pressure, stress, and uncertainty were major influencing factors. Professionals must adapt to the time available, types of uncertainty, and individual skills when making decisions in high-risk situations. Improved understanding of these decisional factors can inform evidence-based enhancements to training, technology, and process design.

Adapting Cognitive Task Analysis Methods for Use in a Large Sample Simulation Study of High-Risk Healthcare Events.

Cognitive task analysis (CTA) methods are traditionally used to conduct small-sample, in-depth studies. In this case study, CTA methods were adapted for a large multi-site study in which 102 anesthesiologists worked through four different high-fidelity simulated high-consequence incidents. Cognitive interviews were used to elicit decision processes following each simulated incident. In this paper, we highlight three practical challenges that arose: (1) standardizing the interview techniques for use across a large, distributed team of diverse backgrounds; (2) developing effective training; and (3) developing a strategy to analyze the resulting large amount of qualitative data. We reflect on how we addressed these challenges by increasing standardization, developing focused training, overcoming social norms that hindered interview effectiveness, and conducting a staged analysis. We share findings from a preliminary analysis that provides early validation of the strategy employed. Analysis of a subset of 64 interview transcripts using a decompositional analysis approach suggests that interviewers successfully elicited descriptions of decision processes that varied due to the different challenges presented by the four simulated incidents. A holistic analysis of the same 64 transcripts revealed individual differences in how anesthesiologists interpreted and managed the same case.

Preliminary study of ergonomic behavior during simulated ultrasound-guided regional anesthesia using a head-mounted display.

A head-mounted display provides continuous real-time imaging within the practitioner's visual field. We evaluated the feasibility of using head-mounted display technology to improve ergonomics in ultrasound-guided regional anesthesia in a simulated environment. Two anesthesiologists performed an equal number of ultrasound-guided popliteal-sciatic nerve blocks using the head-mounted display on a porcine hindquarter, and an independent observer assessed each practitioner's ergonomics (eg, head turning, arching, eye movements, and needle manipulation) and the overall block quality based on the injectate spread around the target nerve for each procedure. Both practitioners performed their procedures without directly viewing the ultrasound monitor, and neither practitioner showed poor ergonomic behavior. Head-mounted display technology may offer potential advantages during ultrasound-guided regional anesthesia.

A Simple Ventilator Designed To Be Used in Shortage Crises: Construction and Verification Testing.

BACKGROUND: The COVID-19 pandemic has demonstrated the possibility of severe ventilator shortages in the near future. OBJECTIVE: We aimed to develop an acute shortage ventilator. METHODS: The ventilator was designed to mechanically compress a self-inflating bag resuscitator, using a modified ventilator patient circuit, which is controlled by a microcontroller and an optional laptop. It was designed to operate in both volume-controlled mode and pressure-controlled assist modes. We tested the ventilator in 4 modes using an artificial lung while measuring the volume, flow, and pressure delivered over time by the ventilator. RESULTS: The ventilator was successful in reaching the desired tidal volume and respiratory rates specified in national emergency use resuscitator system guidelines. The ventilator responded to simulated spontaneous breathing. CONCLUSIONS: The key design goals were achieved. We developed a simple device with high performance for short-term use, made primarily from common hospital parts and generally available nonmedical components to avoid any compatibility or safety issues with the patient, and at low cost, with a unit cost per ventilator is less than $400 US excluding the patient circuit parts, that can be easily manufactured.

Use of medical simulation to explore equipment failures and human-machine interactions in anesthesia machine pipeline supply crossover.

BACKGROUND: High-fidelity medical simulation can be used to explore failure modes of technology and equipment and human-machine interactions. We present the use of an equipment malfunction simulation scenario, oxygen (O(2))/nitrous oxide (N(2)O) pipeline crossover, to probe residents' knowledge and their use of anesthetic equipment in a rapidly escalating crisis. METHODS: In this descriptive study, 20 third-year anesthesia residents were paired into 10 two-member teams. The scenario involved an Ohmeda Modulus SE 7500 anesthetic machine with a Datex AS/3 monitor that provided vital signs and gas monitoring. Before the scenario started, we switched pipeline connections so that N(2)O entered through the O(2) pipeline and vice versa. Because of the switched pipeline, the auxiliary O(2) flowmeter delivered N(2)O instead of O(2). Two expert, independent raters reviewed videotaped scenarios and recorded the alarms explicitly noted by participants and methods of ventilation. RESULTS: Nine pairs became aware of the low fraction of inspired O(2) (Fio(2)) alarm. Only 3 pairs recognized the high fraction of inspired N(2)O (Fin(2)o) alarm. One group failed to recognize both the low Fio(2) and the high Fin(2)o alarms. Nine groups took 3 or more steps before instigating a definitive route of oxygenation. Seven groups used the auxiliary O(2) flowmeter at some point during the management steps. CONCLUSIONS: The fact that so many participants used the auxiliary O(2) flowmeter may expose machine factors and related human-machine interactions during an equipment crisis. Use of the auxiliary O(2) flowmeter as a presumed external source of O(2) contributed to delays in definitive treatment. Many participants also failed to notice the presence of high N(2)O. This may have been, in part, attributable to 2 facts that we uncovered during our video review: (a) the transitory nature of the "high N(2)O" alert, and (b) the dominance of the low Fio(2) alarm, which many chose to mute. We suggest that the use of high-fidelity simulations may be a promising avenue to further examine hypotheses related to failure modes of equipment and possible management response strategies of clinicians.

Comparing safety climate in naval aviation and hospitals: implications for improving patient safety.

BACKGROUND: Evidence of variation in safety climate suggests the need for improvement among at least some hospitals. However, comparisons only among hospitals may underestimate the improvement required. Comparison of hospitals with analogous industries may provide a broader perspective on the safety status of our nation's hospitals. PURPOSE: The purpose of this study was to compare safety climate among hospital workers with personnel from naval aviation, an organization that operates with high reliability despite intrinsically hazardous conditions. METHODOLOGY: We surveyed a random sample of health care workers in 67 U.S. hospitals and, for generalizability, 30 veterans affairs hospitals using questions comparable with those posed at approximately the same time (2007) to a census of personnel from 35 squadrons of U.S. naval aviators. We received 13,841 (41%) completed surveys in U.S. hospitals, 5,511 (50%) in veterans affairs hospitals, and 14,854 (82%) among naval aviators. We examined differences in respondents' perceptions of safety climate at their institution overall and for 16 individual items. FINDINGS: Safety climate was three times better on average among naval aviators than among hospital personnel. Naval aviators perceived a safer climate (up to seven times safer) than hospital personnel with respect to each of the 16 survey items. Compared with hospital managers, naval commanders perceived climate more like frontline personnel did. When contrasting naval aviators with hospital personnel working in comparably hazardous areas, safety climate discrepancies increased rather than decreased. One individual hospital performed as well as naval aviation on average, and at least one hospital outperformed the Navy benchmark for all but three individual survey items. PRACTICE IMPLICATIONS: Results suggest that hospitals have not sufficiently created a uniform priority of safety. However, if each hospital performed as well as the top-performing hospital in each area measured, hospitals could achieve safety climate levels comparable with naval aviation. Major interventions to bolster hospital safety climate continue to be required to improve patient safety.

Guidelines for the Responsible Use of Deception in Simulation: Ethical and Educational Considerations.

STATEMENT: Many techniques and modifications commonly used by the simulation community have been identified as deceptive. Deception is an important issue addressed by both the newly adopted Healthcare Simulationist Code of Ethics and the American Psychological Association Code of Conduct. Some view these approaches as essential whereas others question their necessity as well as their untoward psychological effects. In an attempt to offer guidance to simulation-based healthcare educators, we explore educational practices commonly identified as deceptive along with their potential benefits and detriments. We then address important decision points and high-risk situations that should be avoided to uphold ethical boundaries and psychological safety among learners. These are subsequently analyzed in light of the Code of Ethics and used to formulate guidelines for educators that are intended to ensure that deception, when necessary, is implemented in as psychologically safe a manner as possible.