Reliability of simulation-based assessment for practicing physicians: performance is context-specific.

INTRODUCTION: Even physicians who routinely work in complex, dynamic practices may be unprepared to optimally manage challenging critical events. High-fidelity simulation can realistically mimic critical clinically relevant events, however the reliability and validity of simulation-based assessment scores for practicing physicians has not been established. METHODS: Standardised complex simulation scenarios were developed and administered to board-certified, practicing anesthesiologists who volunteered to participate in an assessment study during formative maintenance of certification activities. A subset of the study population agreed to participate as the primary responder in a second scenario for this study. The physicians were assessed independently by trained raters on both teamwork/behavioural and technical performance measures. Analysis using Generalisability and Decision studies were completed for the two scenarios with two raters. RESULTS: The behavioural score was not more reliable than the technical score. With two raters > 20 scenarios would be required to achieve a reliability estimate of 0.7. Increasing the number of raters for a given scenario would have little effect on reliability. CONCLUSIONS: The performance of practicing physicians on simulated critical events may be highly context-specific. Realistic simulation-based assessment for practicing physicians is resource-intensive and may be best-suited for individualized formative feedback. More importantly, aggregate data from a population of participants may have an even higher impact if used to identify skill or knowledge gaps to be addressed by training programs and inform continuing education improvements across the profession.

Clinical Uses and Impacts of Emergency Manuals During Perioperative Crises.

BACKGROUND: Performing key actions efficiently during crises can determine clinical outcomes, yet even expert clinicians omit key actions. Simulation-based studies of crises show that correct performance of key actions dramatically increases when emergency manuals (EMs) are used. Despite widespread dissemination of EMs, there is a need to understand in clinical contexts, when, how, and how often EMs are used and not used, along with perceived impacts. METHODS: We conducted interviews with the anesthesia professionals involved in perioperative crises, identified with criterion-based sampling, occurring between October 2014 and May 2016 at 2 large academic medical centers with a history of EM training and implementation. Our convergent, mixed-methods study of the interview data extracted quantitative counts and qualitative themes of EM use and nonuse during clinical crises. RESULTS: Interviews with 53 anesthesia professionals yielded 80 descriptions of applicable clinical crises, with varying durations and event types. Of 69 unique patients whose cases involved crises, the EM was used during 37 (54%; 95% confidence interval [CI], 41-66). Impacts on clinician team members included decreased stress for individual anesthesia professionals (95%), enabled teamwork (73%), and calmed atmosphere (46%). Impacts on delivery of patient care included specific action improvements, including catching errors of omission, for example, turning off anesthetic during cardiac arrest, only after EM use (59%); process improvements, for example, double-checking all actions were completed (41%); and impediments (0%). In 8% of crises, EM use was associated with potential distractions, although none were perceived to harm delivery of patient care. For 32 EM nonuses (46%; 95% CI, 34-59), participants self-identified errors of omission or delays in key actions (56%), all key actions performed (13%), and crisis too brief for EM to be used (31%). CONCLUSIONS: This study provides evidence that EMs in operating rooms are being used during many applicable crises and that clinicians perceive EM use to add value. The reported negative effects were minimal and potentially offset by positive effects.

Use of an Emergency Manual During an Intraoperative Cardiac Arrest by an Interprofessional Team: A Positive-Exemplar Case Study of a New Patient Safety Tool.

BACKGROUND: An emergency manual (EM) is a set of evidence-based crisis checklists, or cognitive aids, that can improve team performance. EMs are used in other safety-critical industries, and health care simulation studies have shown their efficacy, but use in clinical settings is nascent. A case study was conducted on the use of an EM during one intraoperative crisis, which entailed the assessment of the impact of the EM's use on teamwork and patient care and the identification of lessons for effectively using EMs during future clinical crises. METHODS: In a case study of a single crisis, an EM was used during a cardiac arrest at a tertiary care hospital that had systematically implemented perioperative EMs. Semistructured interviews were conducted with all six clinicians present, interview transcripts were iteratively coded, and thematic analysis was performed. RESULTS: All clinician participants stated that EM use enabled effective team functioning via reducing stress of individual clinicians, fostering a calm work environment, and improving teamwork and communication. These impacts in turn improved the delivery of patient care during a clinical crisis and influenced participants' intended EM use during future appropriate crises. CONCLUSION: In this positive-exemplar case study, an EM was used to improve delivery of evidence-based patient care through effective clinical team functioning. EM use must complement rather than replace good clinician education, judgment, and teamwork. More broadly, understanding why and how things go well via analyzing positive-exemplar case studies, as a converse of root cause analyses for negative events, can be used to identify effective applications of safety innovations.

Simulation-based Assessment of the Management of Critical Events by Board-certified Anesthesiologists.

BACKGROUND: We sought to determine whether mannequin-based simulation can reliably characterize how board-certified anesthesiologists manage simulated medical emergencies. Our primary focus was to identify gaps in performance and to establish psychometric properties of the assessment methods. METHODS: A total of 263 consenting board-certified anesthesiologists participating in existing simulation-based maintenance of certification courses at one of eight simulation centers were video recorded performing simulated emergency scenarios. Each participated in two 20-min, standardized, high-fidelity simulated medical crisis scenarios, once each as primary anesthesiologist and first responder. Via a Delphi technique, an independent panel of expert anesthesiologists identified critical performance elements for each scenario. Trained, blinded anesthesiologists rated video recordings using standardized rating tools. Measures included the percentage of critical performance elements observed and holistic (one to nine ordinal scale) ratings of participant's technical and nontechnical performance. Raters also judged whether the performance was at a level expected of a board-certified anesthesiologist. RESULTS: Rater reliability for most measures was good. In 284 simulated emergencies, participants were rated as successfully completing 81% (interquartile range, 75 to 90%) of the critical performance elements. The median rating of both technical and nontechnical holistic performance was five, distributed across the nine-point scale. Approximately one-quarter of participants received low holistic ratings (i.e., three or less). Higher-rated performances were associated with younger age but not with previous simulation experience or other individual characteristics. Calling for help was associated with better individual and team performance. CONCLUSIONS: Standardized simulation-based assessment identified performance gaps informing opportunities for improvement. If a substantial proportion of experienced anesthesiologists struggle with managing medical emergencies, continuing medical education activities should be reevaluated.

Practice improvements based on participation in simulation for the maintenance of certification in anesthesiology program.

BACKGROUND: This study describes anesthesiologists' practice improvements undertaken during the first 3 yr of simulation activities for the Maintenance of Certification in Anesthesiology Program. METHODS: A stratified sampling of 3 yr (2010-2012) of participants' practice improvement plans was coded, categorized, and analyzed. RESULTS: Using the sampling scheme, 634 of 1,275 participants in Maintenance of Certification in Anesthesiology Program simulation courses were evaluated from the following practice settings: 41% (262) academic, 54% (339) community, and 5% (33) military/other. A total of 1,982 plans were analyzed for completion, target audience, and topic. On follow-up, 79% (1,558) were fully completed, 16% (310) were partially completed, and 6% (114) were not completed within the 90-day reporting period. Plans targeted the reporting individual (89% of plans) and others (78% of plans): anesthesia providers (50%), non-anesthesia physicians (16%), and non-anesthesia non-physician providers (26%). From the plans, 2,453 improvements were categorized as work environment or systems changes (33% of improvements), teamwork skills (30%), personal knowledge (29%), handoff (4%), procedural skills (3%), or patient communication (1%). The median word count was 63 (interquartile range, 30 to 126) for each participant's combined plans and 147 (interquartile range, 52 to 257) for improvement follow-up reports. CONCLUSIONS: After making a commitment to change, 94% of anesthesiologists participating in a Maintenance of Certification in Anesthesiology Program simulation course successfully implemented some or all of their planned practice improvements. This compares favorably to rates in other studies. Simulation experiences stimulate active learning and motivate personal and collaborative practice improvement changes. Further evaluation will assess the impact of the improvements and further refine the program.

Evaluation of a Standardized Program for Training Practicing Anesthesiologists in Ultrasound-Guided Regional Anesthesia Skills.

OBJECTIVES: Practicing anesthesiologists have generally not received formal training in ultrasound-guided perineural catheter insertion. We designed this study to determine the efficacy of a standardized teaching program in this population. METHODS: Anesthesiologists in practice for 10 years or more were recruited and enrolled to participate in a 1-day program: lectures and live-model ultrasound scanning (morning) and faculty-led iterative practice and mannequin-based simulation (afternoon). Participants were assessed and recorded while performing ultrasound-guided perineural catheter insertion at baseline, at midday (interval), and after the program (final). Videos were scored by 2 blinded reviewers using a composite tool and global rating scale. Participants were surveyed every 3 months for 1 year to report the number of procedures, efficacy of teaching methods, and implementation obstacles. RESULTS: Thirty-two participants were enrolled and completed the program; 31 of 32 (97%) completed the 1-year follow-up. Final scores [median (10th-90th percentiles)] were 21.5 (14.5-28.0) of 30 points compared to 14.0 (9.0-20.0) at interval (P < .001 versus final) and 12.0 (8.5-17.5) at baseline (P < .001 versus final), with no difference between interval and baseline. The global rating scale showed an identical pattern. Twelve of 26 participants without previous experience performed at least 1 perineural catheter insertion after training (P < .001). However, there were no differences in the monthly average number of procedures or complications after the course when compared to baseline. CONCLUSIONS: Practicing anesthesiologists without previous training in ultrasound-guided regional anesthesia can acquire perineural catheter insertion skills after a 1-day standardized course, but changing clinical practice remains a challenge.

Collaborative Use of Lung Mechanics Simulation for Testing and Iterative Design for Three Emergency Use Ventilation Device Projects.

SUMMARY STATEMENT: We describe our collaboration with engineering, clinical, and simulation colleagues to use a lung simulator (IngMar Medical ASL 5000) to aid in the development of 3 open-source ventilation devices for patients with COVID-19.Twenty-nine test conditions were created by programming software lung models of varying disease severity in the ASL 5000 to test basic functionality, safety features, and compliance with regulatory requirements for emergency use authorization for the 3 projects' prototypes. More than 200 simulations were performed, with the design team present to enable rapid troubleshooting and design iteration in real time.Working with 3 separate simultaneous ventilation device projects allowed us to rapidly learn from each, improving our ability to successfully collaborate with the different design/build teams.This project illustrates the role of simulation in facilitating collaborative innovation in health care, both in emergency and everyday settings that extend beyond the COVID-19 pandemic.

Decision-Making During High-Risk Events: A Systematic Literature Review.

Effective decision-making in crisis events is challenging due to time pressure, uncertainty, and dynamic decisional environments. We conducted a systematic literature review in PubMed and PsycINFO, identifying 32 empiric research papers that examine how trained professionals make naturalistic decisions under pressure. We used structured qualitative analysis methods to extract key themes. The studies explored different aspects of decision-making across multiple domains. The majority (19) focused on healthcare; military, fire and rescue, oil installation, and aviation domains were also represented. We found appreciable variability in research focus, methodology, and decision-making descriptions. We identified five main themes: (1) decision-making strategy, (2) time pressure, (3) stress, (4) uncertainty, and (5) errors. Recognition-primed decision-making (RPD) strategies were reported in all studies that analyzed this aspect. Analytical strategies were also prominent, appearing more frequently in contexts with less time pressure and explicit training to generate multiple explanations. Practitioner experience, time pressure, stress, and uncertainty were major influencing factors. Professionals must adapt to the time available, types of uncertainty, and individual skills when making decisions in high-risk situations. Improved understanding of these decisional factors can inform evidence-based enhancements to training, technology, and process design.

Preliminary study of ergonomic behavior during simulated ultrasound-guided regional anesthesia using a head-mounted display.

A head-mounted display provides continuous real-time imaging within the practitioner's visual field. We evaluated the feasibility of using head-mounted display technology to improve ergonomics in ultrasound-guided regional anesthesia in a simulated environment. Two anesthesiologists performed an equal number of ultrasound-guided popliteal-sciatic nerve blocks using the head-mounted display on a porcine hindquarter, and an independent observer assessed each practitioner's ergonomics (eg, head turning, arching, eye movements, and needle manipulation) and the overall block quality based on the injectate spread around the target nerve for each procedure. Both practitioners performed their procedures without directly viewing the ultrasound monitor, and neither practitioner showed poor ergonomic behavior. Head-mounted display technology may offer potential advantages during ultrasound-guided regional anesthesia.

A Simple Ventilator Designed To Be Used in Shortage Crises: Construction and Verification Testing.

BACKGROUND: The COVID-19 pandemic has demonstrated the possibility of severe ventilator shortages in the near future. OBJECTIVE: We aimed to develop an acute shortage ventilator. METHODS: The ventilator was designed to mechanically compress a self-inflating bag resuscitator, using a modified ventilator patient circuit, which is controlled by a microcontroller and an optional laptop. It was designed to operate in both volume-controlled mode and pressure-controlled assist modes. We tested the ventilator in 4 modes using an artificial lung while measuring the volume, flow, and pressure delivered over time by the ventilator. RESULTS: The ventilator was successful in reaching the desired tidal volume and respiratory rates specified in national emergency use resuscitator system guidelines. The ventilator responded to simulated spontaneous breathing. CONCLUSIONS: The key design goals were achieved. We developed a simple device with high performance for short-term use, made primarily from common hospital parts and generally available nonmedical components to avoid any compatibility or safety issues with the patient, and at low cost, with a unit cost per ventilator is less than $400 US excluding the patient circuit parts, that can be easily manufactured.

Use of medical simulation to explore equipment failures and human-machine interactions in anesthesia machine pipeline supply crossover.

BACKGROUND: High-fidelity medical simulation can be used to explore failure modes of technology and equipment and human-machine interactions. We present the use of an equipment malfunction simulation scenario, oxygen (O(2))/nitrous oxide (N(2)O) pipeline crossover, to probe residents' knowledge and their use of anesthetic equipment in a rapidly escalating crisis. METHODS: In this descriptive study, 20 third-year anesthesia residents were paired into 10 two-member teams. The scenario involved an Ohmeda Modulus SE 7500 anesthetic machine with a Datex AS/3 monitor that provided vital signs and gas monitoring. Before the scenario started, we switched pipeline connections so that N(2)O entered through the O(2) pipeline and vice versa. Because of the switched pipeline, the auxiliary O(2) flowmeter delivered N(2)O instead of O(2). Two expert, independent raters reviewed videotaped scenarios and recorded the alarms explicitly noted by participants and methods of ventilation. RESULTS: Nine pairs became aware of the low fraction of inspired O(2) (Fio(2)) alarm. Only 3 pairs recognized the high fraction of inspired N(2)O (Fin(2)o) alarm. One group failed to recognize both the low Fio(2) and the high Fin(2)o alarms. Nine groups took 3 or more steps before instigating a definitive route of oxygenation. Seven groups used the auxiliary O(2) flowmeter at some point during the management steps. CONCLUSIONS: The fact that so many participants used the auxiliary O(2) flowmeter may expose machine factors and related human-machine interactions during an equipment crisis. Use of the auxiliary O(2) flowmeter as a presumed external source of O(2) contributed to delays in definitive treatment. Many participants also failed to notice the presence of high N(2)O. This may have been, in part, attributable to 2 facts that we uncovered during our video review: (a) the transitory nature of the "high N(2)O" alert, and (b) the dominance of the low Fio(2) alarm, which many chose to mute. We suggest that the use of high-fidelity simulations may be a promising avenue to further examine hypotheses related to failure modes of equipment and possible management response strategies of clinicians.

Comparing safety climate in naval aviation and hospitals: implications for improving patient safety.

BACKGROUND: Evidence of variation in safety climate suggests the need for improvement among at least some hospitals. However, comparisons only among hospitals may underestimate the improvement required. Comparison of hospitals with analogous industries may provide a broader perspective on the safety status of our nation's hospitals. PURPOSE: The purpose of this study was to compare safety climate among hospital workers with personnel from naval aviation, an organization that operates with high reliability despite intrinsically hazardous conditions. METHODOLOGY: We surveyed a random sample of health care workers in 67 U.S. hospitals and, for generalizability, 30 veterans affairs hospitals using questions comparable with those posed at approximately the same time (2007) to a census of personnel from 35 squadrons of U.S. naval aviators. We received 13,841 (41%) completed surveys in U.S. hospitals, 5,511 (50%) in veterans affairs hospitals, and 14,854 (82%) among naval aviators. We examined differences in respondents' perceptions of safety climate at their institution overall and for 16 individual items. FINDINGS: Safety climate was three times better on average among naval aviators than among hospital personnel. Naval aviators perceived a safer climate (up to seven times safer) than hospital personnel with respect to each of the 16 survey items. Compared with hospital managers, naval commanders perceived climate more like frontline personnel did. When contrasting naval aviators with hospital personnel working in comparably hazardous areas, safety climate discrepancies increased rather than decreased. One individual hospital performed as well as naval aviation on average, and at least one hospital outperformed the Navy benchmark for all but three individual survey items. PRACTICE IMPLICATIONS: Results suggest that hospitals have not sufficiently created a uniform priority of safety. However, if each hospital performed as well as the top-performing hospital in each area measured, hospitals could achieve safety climate levels comparable with naval aviation. Major interventions to bolster hospital safety climate continue to be required to improve patient safety.

Patient safety climate in 92 US hospitals: differences by work area and discipline.

BACKGROUND: Concern about patient safety has promoted efforts to improve safety climate. A better understanding of how patient safety climate differs among distinct work areas and disciplines in hospitals would facilitate the design and implementation of interventions. OBJECTIVES: To understand workers' perceptions of safety climate and ways in which climate varies among hospitals and by work area and discipline. RESEARCH DESIGN: We administered the Patient Safety Climate in Healthcare Organizations survey in 2004-2005 to personnel in a stratified random sample of 92 US hospitals. SUBJECTS: We sampled 100% of senior managers and physicians and 10% of all other workers. We received 18,361 completed surveys (52% response). MEASURES: The survey measured safety climate perceptions and worker and job characteristics of hospital personnel. We calculated and compared the percent of responses inconsistent with a climate of safety among hospitals, work areas, and disciplines. RESULTS: Overall, 17% of responses were inconsistent with a safety climate. Patient safety climate differed by hospital and among and within work areas and disciplines. Emergency department personnel perceived worse safety climate and personnel in nonclinical areas perceived better safety climate than workers in other areas. Nurses were more negative than physicians regarding their work unit's support and recognition of safety efforts, and physicians showed marginally more fear of shame than nurses. For other dimensions of safety climate, physician-nurse differences depended on their work area. CONCLUSIONS: Differences among and within hospitals suggest that strategies for improving safety climate and patient safety should be tailored for work areas and disciplines.

Relationship of hospital organizational culture to patient safety climate in the Veterans Health Administration.

Improving safety climate could enhance patient safety, yet little evidence exists regarding the relationship between hospital characteristics and safety climate. This study assessed the relationship between hospitals' organizational culture and safety climate in Veterans Health Administration (VA) hospitals nationally. Data were collected from a sample of employees in a stratified random sample of 30 VA hospitals over a 6-month period (response rate = 50%; n = 4,625). The Patient Safety Climate in Healthcare Organizations (PSCHO) and the Zammuto and Krakower surveys were used to measure safety climate and organizational culture, respectively. Higher levels of safety climate were significantly associated with higher levels of group and entrepreneurial cultures, while lower levels of safety climate were associated with higher levels of hierarchical culture. Hospitals could use these results to design specific interventions aimed at improving safety climate.