Measuring deliberate reflection in residents: validation and psychometric properties of a measurement tool.

PURPOSE: Reflective capacity is "the ability to understand critical analysis of knowledge and experience to achieve deeper meaning." In medicine, there is little provision for post-graduate medical education to teach deliberate reflection. The feasibility, scoring characteristics, reliability, validation, and adaptability of a modified previously validated instrument was examined for its usefulness assessing reflective capacity in residents as a step toward developing interventions for improvement. METHODS: Third-year residents and fellows from four anesthesia training programs were administered a slightly modified version of the Reflection Evaluation for Learners' Enhanced Competencies Tool (REFLECT) in a prospective, observational study at the end of the 2019 academic year. Six written vignettes of imperfect anesthesia situations were created. Subjects recorded their perspectives on two randomly assigned vignettes. Responses were scored using a 5-element rubric; average scores were analyzed for psychometric properties. An independent self-report assessment method, the Cognitive Behavior Survey: Residency Level (rCBS) was used to examine construct validity. Internal consistency (ICR, Cronbach's alpha) and interrater reliability (weighted kappa) were examined. Pearson correlations were used between the two measures of reflective capacity. RESULTS: 46/136 invited subjects completed 2/6 randomly assigned vignettes. Interrater agreement was high (k = 0.85). The overall average REFLECT score was 1.8 (1-4 scale) with good distribution across the range of scores. ICR for both the REFLECT score (mean 1.8, sd 0.5; α = 0.92) and the reflection scale of the rCBS (mean 4.5, sd 1.1; α = 0.94) were excellent. There was a significant correlation between REFLECT score and the rCBS reflection scale (r = .44, p < 0.01). CONCLUSIONS: This study demonstrates feasibility, reliability, and sufficiently robust psychometric properties of a modified REFLECT rubric to assess graduate medical trainees' reflective capacity and established construct/convergent validity to an independent measure. The instrument has the potential to assess the effectiveness of interventions intended to improve reflective capacity.

Characterization of Reflective Capacity of Anesthesiology Trainees in an Irish Tertiary Referral Teaching Hospital.

Background: Reflective practice is associated with improved accuracy of medical diagnosis and superior performance in complex situations. Systematic observation of trainees' reflective capacities constitutes a basis for an effective support of reflective practice within the training paradigm. We set out to examine the reflective capacity among anesthesiology trainees in a tertiary referral hospital. Methods: We invited 61 anesthesiology trainees in Cork University Hospitals, Ireland, to participate. Each trainee was invited to respond to 2 investigator-written vignettes prepared by the investigators and suitable for evaluation using the Reflection Evaluation for Learners' Enhanced Competencies Tool (REFLECT) and to produce and then respond to a written vignette based on their own experience. All responses were assessed by 2 independent assessors who had undergone training in the application of the REFLECT rubric, which gives quantifiable scores. Interrater reliability was assessed by weighted kappa coefficient. Association between years of training in medicine and level of reflective capacity was examined using correlation and multiple regression analyses, controlling for age. Results: Twenty-nine trainees agreed to participate, the overall REFLECT Level was 2.16 (SD 0.7), corresponding to "thoughtful action," indicating low to moderate reflective ability. Cronbach's alpha for the 5 items of the REFLECT scale was excellent (r = 0.92). Weighted kappa was very satisfactory (k = 0.81). A strong association was demonstrated between years in medicine and scores on REFLECT, controlling for age of participant (F = -2.57, Beta coefficient = -0.30). Respondents with less experience had greater mean REFLECT scores than respondents with more experience (F = 5.5, P = .02; post hoc mean difference = 0.7, P = .03 for ≤32 months vs ≥99 months). There was a significant effect for gender (t = -4.3, P = .001), with women's responses receiving greater REFLECT scores than men's responses (mean difference = 0.67, P = .001). Conclusions: Overall, participants demonstrated low to moderate reflective capacity, as assessed by the REFLECT rubric. Reflective capacity of the anesthesiology trainees appears to decrease as years of medical training progress. However, our respondents were not sampled over time to fully support this conclusion. Further research is needed on the psychometric properties of the REFLECT rubric and the generalizability of our findings.

Intraoperative Deaths: Who, Why, and Can We Prevent Them?

INTRODUCTION: Intraoperative deaths (IODs) are rare but catastrophic. We systematically analyzed IODs to identify clinical and patient safety patterns. METHODS: IODs in a large academic center between 2015 and 2019 were included. Perioperative details were systematically reviewed, focusing on (1) identifying phenotypes of IOD, (2) describing emerging themes immediately preceding cardiac arrest, and (3) suggesting interventions to mitigate IOD in each phenotype. RESULTS: Forty-one patients were included. Three IOD phenotypes were identified: trauma (T), nontrauma emergency (NT), and elective (EL) surgery patients, each with 2 sub-phenotypes (e.g., ELm and ELv for elective surgery with medical arrests or vascular injury and bleeding, respectively). In phenotype T, cardiopulmonary resuscitation was initiated before incision in 42%, resuscitative thoracotomy was performed in 33%, and transient return of spontaneous circulation was achieved in 30% of patients. In phenotype NT, ruptured aortic aneurysms accounted for half the cases, and median blood product utilization was 2,694 mL. In phenotype ELm, preoperative evaluation did not include electrocardiogram in 12%, cardiac consultation in 62%, stress test in 87%, and chest x-ray in 37% of patients. In phenotype ELv, 83% had a single peripheral intravenous line, and vascular injury was almost always followed by escalation in monitoring (e.g., central/arterial line), alert to the blood bank, and call for surgical backup. CONCLUSIONS: We have created a framework for IOD that can help with intraoperative safety and quality analysis. Focusing on interventions that address appropriateness versus futility in care in phenotypes T and NT, and on prevention and mitigation of intraoperative vessel injury (e.g., intraoperative rescue team) or preoperative optimization in phenotype EL may help prevent IODs.

Tribute to Dr. Takuo Aoyagi, inventor of pulse oximetry.

INTRODUCTION: Dr. Takuo Aoyagi invented pulse oximetry in 1974. Pulse oximeters are widely used worldwide, most recently making headlines during the COVID-19 pandemic. Dr. Aoyagi passed away on April 18, 2020, aware of the significance of his invention, but still actively searching for the theory that would take his invention to new heights. METHOD: Many people who knew Dr. Aoyagi, or knew of him and his invention, agreed to participate in this tribute to his work. The authors, from Japan and around the world, represent all aspects of the development of medical devices, including scientists and engineers, clinicians, academics, business people, and clinical practitioners. RESULTS: While the idea of pulse oximetry originated in Japan, device development lagged in Japan due to a lack of business, clinical, and academic interest. Awareness of the importance of anesthesia safety in the US, due to academic foresight and media attention, in combination with excellence in technological innovation, led to widespread use of pulse oximetry around the world. CONCLUSION: Dr. Aoyagi's final wish was to find a theory of pulse oximetry. We hope this tribute to him and his invention will inspire a new generation of scientists, clinicians, and related organizations to secure the foundation of the theory.

Reliability of simulation-based assessment for practicing physicians: performance is context-specific.

INTRODUCTION: Even physicians who routinely work in complex, dynamic practices may be unprepared to optimally manage challenging critical events. High-fidelity simulation can realistically mimic critical clinically relevant events, however the reliability and validity of simulation-based assessment scores for practicing physicians has not been established. METHODS: Standardised complex simulation scenarios were developed and administered to board-certified, practicing anesthesiologists who volunteered to participate in an assessment study during formative maintenance of certification activities. A subset of the study population agreed to participate as the primary responder in a second scenario for this study. The physicians were assessed independently by trained raters on both teamwork/behavioural and technical performance measures. Analysis using Generalisability and Decision studies were completed for the two scenarios with two raters. RESULTS: The behavioural score was not more reliable than the technical score. With two raters > 20 scenarios would be required to achieve a reliability estimate of 0.7. Increasing the number of raters for a given scenario would have little effect on reliability. CONCLUSIONS: The performance of practicing physicians on simulated critical events may be highly context-specific. Realistic simulation-based assessment for practicing physicians is resource-intensive and may be best-suited for individualized formative feedback. More importantly, aggregate data from a population of participants may have an even higher impact if used to identify skill or knowledge gaps to be addressed by training programs and inform continuing education improvements across the profession.

The Case of the Inadvertently Triggered Laser: An Historical Example of Simulation-Enhanced Adverse Event Investigation.

SUMMARY STATEMENT: This report describes an historical case example that illustrates how simulation used in investigating adverse events can identify underlying, remediable causes that otherwise may not be discovered. The event involved an operating room fire caused by inadvertent triggering of a laser. Guidelines for action after an adverse event were used to guide the postevent actions. An expert in operating room fires was engaged to conduct the investigation. Recommendations and considerations for conducting such investigations with simulation are offered. Simulation is likely underused for healthcare adverse event investigations; this case example may encourage more widespread application.

Clinical Uses and Impacts of Emergency Manuals During Perioperative Crises.

BACKGROUND: Performing key actions efficiently during crises can determine clinical outcomes, yet even expert clinicians omit key actions. Simulation-based studies of crises show that correct performance of key actions dramatically increases when emergency manuals (EMs) are used. Despite widespread dissemination of EMs, there is a need to understand in clinical contexts, when, how, and how often EMs are used and not used, along with perceived impacts. METHODS: We conducted interviews with the anesthesia professionals involved in perioperative crises, identified with criterion-based sampling, occurring between October 2014 and May 2016 at 2 large academic medical centers with a history of EM training and implementation. Our convergent, mixed-methods study of the interview data extracted quantitative counts and qualitative themes of EM use and nonuse during clinical crises. RESULTS: Interviews with 53 anesthesia professionals yielded 80 descriptions of applicable clinical crises, with varying durations and event types. Of 69 unique patients whose cases involved crises, the EM was used during 37 (54%; 95% confidence interval [CI], 41-66). Impacts on clinician team members included decreased stress for individual anesthesia professionals (95%), enabled teamwork (73%), and calmed atmosphere (46%). Impacts on delivery of patient care included specific action improvements, including catching errors of omission, for example, turning off anesthetic during cardiac arrest, only after EM use (59%); process improvements, for example, double-checking all actions were completed (41%); and impediments (0%). In 8% of crises, EM use was associated with potential distractions, although none were perceived to harm delivery of patient care. For 32 EM nonuses (46%; 95% CI, 34-59), participants self-identified errors of omission or delays in key actions (56%), all key actions performed (13%), and crisis too brief for EM to be used (31%). CONCLUSIONS: This study provides evidence that EMs in operating rooms are being used during many applicable crises and that clinicians perceive EM use to add value. The reported negative effects were minimal and potentially offset by positive effects.

Use of an Emergency Manual During an Intraoperative Cardiac Arrest by an Interprofessional Team: A Positive-Exemplar Case Study of a New Patient Safety Tool.

BACKGROUND: An emergency manual (EM) is a set of evidence-based crisis checklists, or cognitive aids, that can improve team performance. EMs are used in other safety-critical industries, and health care simulation studies have shown their efficacy, but use in clinical settings is nascent. A case study was conducted on the use of an EM during one intraoperative crisis, which entailed the assessment of the impact of the EM's use on teamwork and patient care and the identification of lessons for effectively using EMs during future clinical crises. METHODS: In a case study of a single crisis, an EM was used during a cardiac arrest at a tertiary care hospital that had systematically implemented perioperative EMs. Semistructured interviews were conducted with all six clinicians present, interview transcripts were iteratively coded, and thematic analysis was performed. RESULTS: All clinician participants stated that EM use enabled effective team functioning via reducing stress of individual clinicians, fostering a calm work environment, and improving teamwork and communication. These impacts in turn improved the delivery of patient care during a clinical crisis and influenced participants' intended EM use during future appropriate crises. CONCLUSION: In this positive-exemplar case study, an EM was used to improve delivery of evidence-based patient care through effective clinical team functioning. EM use must complement rather than replace good clinician education, judgment, and teamwork. More broadly, understanding why and how things go well via analyzing positive-exemplar case studies, as a converse of root cause analyses for negative events, can be used to identify effective applications of safety innovations.

Critical Role of the Surgeon-Anesthesiologist Relationship for Patient Safety.

Teamwork is now recognized as important for safe, high-quality perioperative care. The relationship in each surgeon-anesthesiologist dyad is perhaps the most critical element of overall team performance. A well-functioning relationship is conducive to safe, effective care. A dysfunctional relationship can promote unsafe conditions and contribute to an adverse outcome. Yet, there is little research about this relationship, about what works well or not well, what can be done to optimize it. This article explores functional and dysfunctional aspects of the relationship, identifies some negative stereotypes each profession has of the other and calls for research to better characterize and understand how to improve working relationships. Suggestions are given for what an ideal relationship might be and actions that surgeons and anesthesiologists can take to improve how they work together. The goal is safer care for patients, and more joy and meaning in work for surgeons and anesthesiologists.

Critical Role of the Surgeon-Anesthesiologist Relationship for Patient Safety.

Teamwork is now recognized as important for safe, high-quality perioperative care. The relationship in each surgeon-anesthesiologist dyad is perhaps the most critical element of overall team performance. A well-functioning relationship is conducive to safe, effective care. A dysfunctional relationship can promote unsafe conditions and contribute to an adverse outcome. Yet, there is little research about this relationship, about what works well or not well, what can be done to optimize it. This article explores functional and dysfunctional aspects of the relationship, identifies some negative stereotypes each profession has of the other and calls for research to better characterize and understand how to improve working relationships. Suggestions are given for what an ideal relationship might be and actions that surgeons and anesthesiologists can take to improve how they work together. The goal is safer care for patients, and more joy and meaning in work for surgeons and anesthesiologists.

Simulation-based Assessment to Reliably Identify Key Resident Performance Attributes.

BACKGROUND: Obtaining reliable and valid information on resident performance is critical to patient safety and training program improvement. The goals were to characterize important anesthesia resident performance gaps that are not typically evaluated, and to further validate scores from a multiscenario simulation-based assessment. METHODS: Seven high-fidelity scenarios reflecting core anesthesiology skills were administered to 51 first-year residents (CA-1s) and 16 third-year residents (CA-3s) from three residency programs. Twenty trained attending anesthesiologists rated resident performances using a seven-point behaviorally anchored rating scale for five domains: (1) formulate a clear plan, (2) modify the plan under changing conditions, (3) communicate effectively, (4) identify performance improvement opportunities, and (5) recognize limits. A second rater assessed 10% of encounters. Scores and variances for each domain, each scenario, and the total were compared. Low domain ratings (1, 2) were examined in detail. RESULTS: Interrater agreement was 0.76; reliability of the seven-scenario assessment was r = 0.70. CA-3s had a significantly higher average total score (4.9 ± 1.1 vs. 4.6 ± 1.1, P = 0.01, effect size = 0.33). CA-3s significantly outscored CA-1s for five of seven scenarios and domains 1, 2, and 3. CA-1s had a significantly higher proportion of worrisome ratings than CA-3s (chi-square = 24.1, P < 0.01, effect size = 1.50). Ninety-eight percent of residents rated the simulations more educational than an average day in the operating room. CONCLUSIONS: Sensitivity of the assessment to CA-1 versus CA-3 performance differences for most scenarios and domains supports validity. No differences, by experience level, were detected for two domains associated with reflective practice. Smaller score variances for CA-3s likely reflect a training effect; however, worrisome performance scores for both CA-1s and CA-3s suggest room for improvement.

Learning how to "teach one": A needs assessment of the state of faculty development within the Consortium of the American College of Surgeons Accredited Education Institutes.

BACKGROUND: Developing faculty competencies in curriculum development, teaching, and assessment using simulation is critical for the success of the Consortium of the American College of Surgeons Accredited Education Institutes program. The state of and needs for faculty development in the Accredited Education Institute community are unknown currently. The Faculty Development Committee of the Consortium of the Accredited Education Institutes conducted a survey of Accredited Education Institutes to ascertain what types of practices are used currently, with what frequency, and what needs are perceived for further programs and courses to guide the plan of action for the Faculty Development Committee. METHODS: The Faculty Development Committee created a 20-question survey with quantitative and qualitative items aimed at gathering data about practices of faculty development and needs within the Consortium of Accredited Education Institutes. The survey was sent to all 83 Accredited Education Institutes program leaders via Survey Monkey in January 2015 with 2 follow-up reminders. Quantitative data were compiled and analyzed using descriptive statistics, and qualitative data were interpreted for common themes. RESULTS: Fifty-four out of the 83 programs (65%) responded to the survey. Two-thirds of the programs had from 1 to 30 faculty teaching at their Accredited Education Institutes. More than three-quarters of the programs taught general surgery, emergency medicine, or obstetrics/gynecology. More than 60% of programs had some form of faculty development, but 91% reported a need to expand their offerings for faculty development with "extreme value" for debriefing skills (70%), assessment (47%), feedback (40%), and curriculum development (40%). Accredited Education Institutes felt that the Consortium could assist with faculty development through such activities as the provision of online resources, sharing of best practices, provision of a blueprint for development of a faculty curriculum and information related to available, credible master programs of faculty development and health professions education. CONCLUSION: Many Accredited Education Institutes programs are engaged in faculty development activities, but almost all see great needs in faculty development related to debriefing, assessment, and curricular development. These results should help to guide the action and decision-making of the Consortium Faculty Development Committee to improve teaching within the American College of Surgeons Accredited Education Institutes.