Measuring deliberate reflection in residents: validation and psychometric properties of a measurement tool.

PURPOSE: Reflective capacity is "the ability to understand critical analysis of knowledge and experience to achieve deeper meaning." In medicine, there is little provision for post-graduate medical education to teach deliberate reflection. The feasibility, scoring characteristics, reliability, validation, and adaptability of a modified previously validated instrument was examined for its usefulness assessing reflective capacity in residents as a step toward developing interventions for improvement. METHODS: Third-year residents and fellows from four anesthesia training programs were administered a slightly modified version of the Reflection Evaluation for Learners' Enhanced Competencies Tool (REFLECT) in a prospective, observational study at the end of the 2019 academic year. Six written vignettes of imperfect anesthesia situations were created. Subjects recorded their perspectives on two randomly assigned vignettes. Responses were scored using a 5-element rubric; average scores were analyzed for psychometric properties. An independent self-report assessment method, the Cognitive Behavior Survey: Residency Level (rCBS) was used to examine construct validity. Internal consistency (ICR, Cronbach's alpha) and interrater reliability (weighted kappa) were examined. Pearson correlations were used between the two measures of reflective capacity. RESULTS: 46/136 invited subjects completed 2/6 randomly assigned vignettes. Interrater agreement was high (k = 0.85). The overall average REFLECT score was 1.8 (1-4 scale) with good distribution across the range of scores. ICR for both the REFLECT score (mean 1.8, sd 0.5; α = 0.92) and the reflection scale of the rCBS (mean 4.5, sd 1.1; α = 0.94) were excellent. There was a significant correlation between REFLECT score and the rCBS reflection scale (r = .44, p < 0.01). CONCLUSIONS: This study demonstrates feasibility, reliability, and sufficiently robust psychometric properties of a modified REFLECT rubric to assess graduate medical trainees' reflective capacity and established construct/convergent validity to an independent measure. The instrument has the potential to assess the effectiveness of interventions intended to improve reflective capacity.

Intraoperative Deaths: Who, Why, and Can We Prevent Them?

INTRODUCTION: Intraoperative deaths (IODs) are rare but catastrophic. We systematically analyzed IODs to identify clinical and patient safety patterns. METHODS: IODs in a large academic center between 2015 and 2019 were included. Perioperative details were systematically reviewed, focusing on (1) identifying phenotypes of IOD, (2) describing emerging themes immediately preceding cardiac arrest, and (3) suggesting interventions to mitigate IOD in each phenotype. RESULTS: Forty-one patients were included. Three IOD phenotypes were identified: trauma (T), nontrauma emergency (NT), and elective (EL) surgery patients, each with 2 sub-phenotypes (e.g., ELm and ELv for elective surgery with medical arrests or vascular injury and bleeding, respectively). In phenotype T, cardiopulmonary resuscitation was initiated before incision in 42%, resuscitative thoracotomy was performed in 33%, and transient return of spontaneous circulation was achieved in 30% of patients. In phenotype NT, ruptured aortic aneurysms accounted for half the cases, and median blood product utilization was 2,694 mL. In phenotype ELm, preoperative evaluation did not include electrocardiogram in 12%, cardiac consultation in 62%, stress test in 87%, and chest x-ray in 37% of patients. In phenotype ELv, 83% had a single peripheral intravenous line, and vascular injury was almost always followed by escalation in monitoring (e.g., central/arterial line), alert to the blood bank, and call for surgical backup. CONCLUSIONS: We have created a framework for IOD that can help with intraoperative safety and quality analysis. Focusing on interventions that address appropriateness versus futility in care in phenotypes T and NT, and on prevention and mitigation of intraoperative vessel injury (e.g., intraoperative rescue team) or preoperative optimization in phenotype EL may help prevent IODs.

Reliability of simulation-based assessment for practicing physicians: performance is context-specific.

INTRODUCTION: Even physicians who routinely work in complex, dynamic practices may be unprepared to optimally manage challenging critical events. High-fidelity simulation can realistically mimic critical clinically relevant events, however the reliability and validity of simulation-based assessment scores for practicing physicians has not been established. METHODS: Standardised complex simulation scenarios were developed and administered to board-certified, practicing anesthesiologists who volunteered to participate in an assessment study during formative maintenance of certification activities. A subset of the study population agreed to participate as the primary responder in a second scenario for this study. The physicians were assessed independently by trained raters on both teamwork/behavioural and technical performance measures. Analysis using Generalisability and Decision studies were completed for the two scenarios with two raters. RESULTS: The behavioural score was not more reliable than the technical score. With two raters > 20 scenarios would be required to achieve a reliability estimate of 0.7. Increasing the number of raters for a given scenario would have little effect on reliability. CONCLUSIONS: The performance of practicing physicians on simulated critical events may be highly context-specific. Realistic simulation-based assessment for practicing physicians is resource-intensive and may be best-suited for individualized formative feedback. More importantly, aggregate data from a population of participants may have an even higher impact if used to identify skill or knowledge gaps to be addressed by training programs and inform continuing education improvements across the profession.

Tribute to Dr. Takuo Aoyagi, inventor of pulse oximetry.

INTRODUCTION: Dr. Takuo Aoyagi invented pulse oximetry in 1974. Pulse oximeters are widely used worldwide, most recently making headlines during the COVID-19 pandemic. Dr. Aoyagi passed away on April 18, 2020, aware of the significance of his invention, but still actively searching for the theory that would take his invention to new heights. METHOD: Many people who knew Dr. Aoyagi, or knew of him and his invention, agreed to participate in this tribute to his work. The authors, from Japan and around the world, represent all aspects of the development of medical devices, including scientists and engineers, clinicians, academics, business people, and clinical practitioners. RESULTS: While the idea of pulse oximetry originated in Japan, device development lagged in Japan due to a lack of business, clinical, and academic interest. Awareness of the importance of anesthesia safety in the US, due to academic foresight and media attention, in combination with excellence in technological innovation, led to widespread use of pulse oximetry around the world. CONCLUSION: Dr. Aoyagi's final wish was to find a theory of pulse oximetry. We hope this tribute to him and his invention will inspire a new generation of scientists, clinicians, and related organizations to secure the foundation of the theory.

Clinical Uses and Impacts of Emergency Manuals During Perioperative Crises.

BACKGROUND: Performing key actions efficiently during crises can determine clinical outcomes, yet even expert clinicians omit key actions. Simulation-based studies of crises show that correct performance of key actions dramatically increases when emergency manuals (EMs) are used. Despite widespread dissemination of EMs, there is a need to understand in clinical contexts, when, how, and how often EMs are used and not used, along with perceived impacts. METHODS: We conducted interviews with the anesthesia professionals involved in perioperative crises, identified with criterion-based sampling, occurring between October 2014 and May 2016 at 2 large academic medical centers with a history of EM training and implementation. Our convergent, mixed-methods study of the interview data extracted quantitative counts and qualitative themes of EM use and nonuse during clinical crises. RESULTS: Interviews with 53 anesthesia professionals yielded 80 descriptions of applicable clinical crises, with varying durations and event types. Of 69 unique patients whose cases involved crises, the EM was used during 37 (54%; 95% confidence interval [CI], 41-66). Impacts on clinician team members included decreased stress for individual anesthesia professionals (95%), enabled teamwork (73%), and calmed atmosphere (46%). Impacts on delivery of patient care included specific action improvements, including catching errors of omission, for example, turning off anesthetic during cardiac arrest, only after EM use (59%); process improvements, for example, double-checking all actions were completed (41%); and impediments (0%). In 8% of crises, EM use was associated with potential distractions, although none were perceived to harm delivery of patient care. For 32 EM nonuses (46%; 95% CI, 34-59), participants self-identified errors of omission or delays in key actions (56%), all key actions performed (13%), and crisis too brief for EM to be used (31%). CONCLUSIONS: This study provides evidence that EMs in operating rooms are being used during many applicable crises and that clinicians perceive EM use to add value. The reported negative effects were minimal and potentially offset by positive effects.

Critical Role of the Surgeon-Anesthesiologist Relationship for Patient Safety.

Teamwork is now recognized as important for safe, high-quality perioperative care. The relationship in each surgeon-anesthesiologist dyad is perhaps the most critical element of overall team performance. A well-functioning relationship is conducive to safe, effective care. A dysfunctional relationship can promote unsafe conditions and contribute to an adverse outcome. Yet, there is little research about this relationship, about what works well or not well, what can be done to optimize it. This article explores functional and dysfunctional aspects of the relationship, identifies some negative stereotypes each profession has of the other and calls for research to better characterize and understand how to improve working relationships. Suggestions are given for what an ideal relationship might be and actions that surgeons and anesthesiologists can take to improve how they work together. The goal is safer care for patients, and more joy and meaning in work for surgeons and anesthesiologists.

Simulation-based Assessment to Reliably Identify Key Resident Performance Attributes.

BACKGROUND: Obtaining reliable and valid information on resident performance is critical to patient safety and training program improvement. The goals were to characterize important anesthesia resident performance gaps that are not typically evaluated, and to further validate scores from a multiscenario simulation-based assessment. METHODS: Seven high-fidelity scenarios reflecting core anesthesiology skills were administered to 51 first-year residents (CA-1s) and 16 third-year residents (CA-3s) from three residency programs. Twenty trained attending anesthesiologists rated resident performances using a seven-point behaviorally anchored rating scale for five domains: (1) formulate a clear plan, (2) modify the plan under changing conditions, (3) communicate effectively, (4) identify performance improvement opportunities, and (5) recognize limits. A second rater assessed 10% of encounters. Scores and variances for each domain, each scenario, and the total were compared. Low domain ratings (1, 2) were examined in detail. RESULTS: Interrater agreement was 0.76; reliability of the seven-scenario assessment was r = 0.70. CA-3s had a significantly higher average total score (4.9 ± 1.1 vs. 4.6 ± 1.1, P = 0.01, effect size = 0.33). CA-3s significantly outscored CA-1s for five of seven scenarios and domains 1, 2, and 3. CA-1s had a significantly higher proportion of worrisome ratings than CA-3s (chi-square = 24.1, P < 0.01, effect size = 1.50). Ninety-eight percent of residents rated the simulations more educational than an average day in the operating room. CONCLUSIONS: Sensitivity of the assessment to CA-1 versus CA-3 performance differences for most scenarios and domains supports validity. No differences, by experience level, were detected for two domains associated with reflective practice. Smaller score variances for CA-3s likely reflect a training effect; however, worrisome performance scores for both CA-1s and CA-3s suggest room for improvement.

Use of an Emergency Manual During an Intraoperative Cardiac Arrest by an Interprofessional Team: A Positive-Exemplar Case Study of a New Patient Safety Tool.

BACKGROUND: An emergency manual (EM) is a set of evidence-based crisis checklists, or cognitive aids, that can improve team performance. EMs are used in other safety-critical industries, and health care simulation studies have shown their efficacy, but use in clinical settings is nascent. A case study was conducted on the use of an EM during one intraoperative crisis, which entailed the assessment of the impact of the EM's use on teamwork and patient care and the identification of lessons for effectively using EMs during future clinical crises. METHODS: In a case study of a single crisis, an EM was used during a cardiac arrest at a tertiary care hospital that had systematically implemented perioperative EMs. Semistructured interviews were conducted with all six clinicians present, interview transcripts were iteratively coded, and thematic analysis was performed. RESULTS: All clinician participants stated that EM use enabled effective team functioning via reducing stress of individual clinicians, fostering a calm work environment, and improving teamwork and communication. These impacts in turn improved the delivery of patient care during a clinical crisis and influenced participants' intended EM use during future appropriate crises. CONCLUSION: In this positive-exemplar case study, an EM was used to improve delivery of evidence-based patient care through effective clinical team functioning. EM use must complement rather than replace good clinician education, judgment, and teamwork. More broadly, understanding why and how things go well via analyzing positive-exemplar case studies, as a converse of root cause analyses for negative events, can be used to identify effective applications of safety innovations.

Simulation-based Assessment of the Management of Critical Events by Board-certified Anesthesiologists.

BACKGROUND: We sought to determine whether mannequin-based simulation can reliably characterize how board-certified anesthesiologists manage simulated medical emergencies. Our primary focus was to identify gaps in performance and to establish psychometric properties of the assessment methods. METHODS: A total of 263 consenting board-certified anesthesiologists participating in existing simulation-based maintenance of certification courses at one of eight simulation centers were video recorded performing simulated emergency scenarios. Each participated in two 20-min, standardized, high-fidelity simulated medical crisis scenarios, once each as primary anesthesiologist and first responder. Via a Delphi technique, an independent panel of expert anesthesiologists identified critical performance elements for each scenario. Trained, blinded anesthesiologists rated video recordings using standardized rating tools. Measures included the percentage of critical performance elements observed and holistic (one to nine ordinal scale) ratings of participant's technical and nontechnical performance. Raters also judged whether the performance was at a level expected of a board-certified anesthesiologist. RESULTS: Rater reliability for most measures was good. In 284 simulated emergencies, participants were rated as successfully completing 81% (interquartile range, 75 to 90%) of the critical performance elements. The median rating of both technical and nontechnical holistic performance was five, distributed across the nine-point scale. Approximately one-quarter of participants received low holistic ratings (i.e., three or less). Higher-rated performances were associated with younger age but not with previous simulation experience or other individual characteristics. Calling for help was associated with better individual and team performance. CONCLUSIONS: Standardized simulation-based assessment identified performance gaps informing opportunities for improvement. If a substantial proportion of experienced anesthesiologists struggle with managing medical emergencies, continuing medical education activities should be reevaluated.

Practice improvements based on participation in simulation for the maintenance of certification in anesthesiology program.

BACKGROUND: This study describes anesthesiologists' practice improvements undertaken during the first 3 yr of simulation activities for the Maintenance of Certification in Anesthesiology Program. METHODS: A stratified sampling of 3 yr (2010-2012) of participants' practice improvement plans was coded, categorized, and analyzed. RESULTS: Using the sampling scheme, 634 of 1,275 participants in Maintenance of Certification in Anesthesiology Program simulation courses were evaluated from the following practice settings: 41% (262) academic, 54% (339) community, and 5% (33) military/other. A total of 1,982 plans were analyzed for completion, target audience, and topic. On follow-up, 79% (1,558) were fully completed, 16% (310) were partially completed, and 6% (114) were not completed within the 90-day reporting period. Plans targeted the reporting individual (89% of plans) and others (78% of plans): anesthesia providers (50%), non-anesthesia physicians (16%), and non-anesthesia non-physician providers (26%). From the plans, 2,453 improvements were categorized as work environment or systems changes (33% of improvements), teamwork skills (30%), personal knowledge (29%), handoff (4%), procedural skills (3%), or patient communication (1%). The median word count was 63 (interquartile range, 30 to 126) for each participant's combined plans and 147 (interquartile range, 52 to 257) for improvement follow-up reports. CONCLUSIONS: After making a commitment to change, 94% of anesthesiologists participating in a Maintenance of Certification in Anesthesiology Program simulation course successfully implemented some or all of their planned practice improvements. This compares favorably to rates in other studies. Simulation experiences stimulate active learning and motivate personal and collaborative practice improvement changes. Further evaluation will assess the impact of the improvements and further refine the program.

Pilot testing of a model for insurer-driven, large-scale multicenter simulation training for operating room teams.

OBJECTIVE: To test the feasibility of implementing a standardized teamwork training program with full operating room teams in multiple institutions, driven by malpractice insurer support and incentives. BACKGROUND: Failures in intraoperative teamwork are among the leading causes of preventable patient injury and death in surgical patients. Teamwork training, particularly using simulation, can be an effective intervention but is difficult to scale. METHODS: A malpractice insurer convened a collaborative with 4 Harvard-affiliated simulation programs to develop a standardized operating room teamwork training curriculum, including principles of communication, assertiveness, and use of the World Health Organization Surgical Safety Checklist. Participant teams were compensated for lost operative time via malpractice premium discounts, continuing education credits, and compensation for lost wages. The course was delivered through a simulation program involving the management of intraoperative emergency scenarios. Participants were surveyed for their perceptions of the program and of its impact on clinical practice. RESULTS: A total of 221 active operating room staff members participated in the program. Each team contained at least 1 attending surgeon, 1 attending anesthesiologist, and 1 operating room nurse (mean size per team: 7 ± 2 participants). No study dates were cancelled because of lack of attendance. The survey response rate was 99% (218/221). Overall, the vast majority of participants found the scenarios realistic [94% (95% confidence interval: 90.9%, 97.2%)], appropriately challenging [95.4% (92.6%, 98.2%)], relevant to their practice [96.3% (93.8%, 98.8%)], and found the training would help them provide safer patient care [92.6% (89.1%, 96.1%)]. Surgeons reported their greatest personal deficit as communication skills. Operating room nurses and anesthesiologists reported a greater need than surgeons to work on personal assertiveness. CONCLUSIONS: A standardized multicenter team training program involving full operative teams is feasible with high-fidelity simulation and modest compensation for lost time. The vast majority of the multidisciplinary participants believed the course to have had a meaningful impact on their approach to clinical practice.

Simulation-based assessment to identify critical gaps in safe anesthesia resident performance.

BACKGROUND: Valid methods are needed to identify anesthesia resident performance gaps early in training. However, many assessment tools in medicine have not been properly validated. The authors designed and tested use of a behaviorally anchored scale, as part of a multiscenario simulation-based assessment system, to identify high- and low-performing residents with regard to domains of greatest concern to expert anesthesiology faculty. METHODS: An expert faculty panel derived five key behavioral domains of interest by using a Delphi process (1) Synthesizes information to formulate a clear anesthetic plan; (2) Implements a plan based on changing conditions; (3) Demonstrates effective interpersonal and communication skills with patients and staff; (4) Identifies ways to improve performance; and (5) Recognizes own limits. Seven simulation scenarios spanning pre-to-postoperative encounters were used to assess performances of 22 first-year residents and 8 fellows from two institutions. Two of 10 trained faculty raters blinded to trainee program and training level scored each performance independently by using a behaviorally anchored rating scale. Residents, fellows, facilitators, and raters completed surveys. RESULTS: Evidence supporting the reliability and validity of the assessment scores was procured, including a high generalizability coefficient (ρ = 0.81) and expected performance differences between first-year resident and fellow participants. A majority of trainees, facilitators, and raters judged the assessment to be useful, realistic, and representative of critical skills required for safe practice. CONCLUSION: The study provides initial evidence to support the validity of a simulation-based performance assessment system for identifying critical gaps in safe anesthesia resident performance early in training.

Introducing standardized "readbacks" to improve patient safety in surgery: a prospective survey in 92 providers at a public safety-net hospital.

BACKGROUND: Communication breakdowns represent the main root cause of preventable complications which lead to harm to surgical patients. Standardized readbacks have been successfully implemented as a main pillar of professional aviation safety for decades, to ensure a safe closed-loop communication between air traffic control and individual pilots. The present study was designed to determine the perception of staff in perioperative services regarding the role of standardized readbacks for improving patient safety in surgery at a single public safety-net hospital and level 1 trauma center. METHODS: A 12-item questionnaire was sent to 180 providers in perioperative services at Denver Health Medical Center. The survey was designed to determine the individual participants' perception of (1) appropriateness of current readback processes; (2) willingness to attend a future training module on this topic; (3) specific scenarios in which readbacks may be effective; and (4) perceived major barriers to the implementation of standardized readbacks. Survey results were compared between departments (surgery versus anesthesia) and between specific staff roles (attending or midlevel provider, resident physician, nursing staff), using non-parametric tests. RESULTS: The response rate to the survey was 50.1% (n=92). Respondents overwhelmingly recognized the role of readbacks in reducing communication errors and improving patient safety. There was a strong agreement among respondents to support participation in a readbacks training program. There was no difference in the responses between the surgery and anesthesia departments.There was a statistically significant difference in the healthcare providers willingness to attend a short training module on readbacks (p<0.001). Resident physicians were less likely to endorse the importance of readbacks in reducing communication errors (p=0.01) and less willing to attend a short training module on readbacks (p<0.001), as compared to staff providers and nursing staff.The main challenge for respondents, which emanated from their responses, appeared to relate to determining the ideal scenarios in which readbacks may be most appropriately used. Overall, respondents strongly felt that readbacks had an important role in patient handoffs, patient orders regarding critical results, counting and verifying surgical instruments, and delegating multiple perioperative tasks. CONCLUSION: The majority of all respondents appear to perceive standardized readbacks as an effective tool for reducing and/or preventing adverse events in the care of surgical patients, derived from a breakdown in communication among perioperative caregivers. Further work needs to be done to define the exact clinical scenarios in which readbacks may be most efficiently implemented, including the definition of a uniform set of scripted quotes and phrases, which should likely be standardized in concert with the aviation safety model.