Using Concurrent Complication Reporting to Evaluate Resident Critical Thinking and Enhance Adult Learning.

OBJECTIVE: Critical thinking and accurate case analysis is difficult to quantify even within the context of routine morbidity and mortality reporting. We designed and implemented a HIPAA-compliant adverse outcome reporting system that collects weekly resident assessments of clinical care across multiple domains (case summary, complications, error analysis, Clavien-Dindo Harm, cognitive bias, standard of care, and ACGME core competencies). We hypothesized that incorporation of this system into the residency program's core curriculum would allow for identification of areas of cognitive weakness or strength and provide a longitudinal evaluation of critical thinking development. DESIGN: A validated, password-protected electronic platform linked to our electronic medical record was used to collect cases weekly in which surgical adverse events occurred. General surgery residents critiqued 1932 cases over a 4-year period from 3 major medical centers within our system. These data were reviewed by teaching faculty, corrected for accuracy and graded utilizing the software's critique algorithm. Grades were emailed to the residents at the time of the review, collected prospectively, stratified, and analyzed by post-graduate year (PGY). Evaluation of the resident scores for each domain and the resultant composite scores allowed for comparison of critical thinking skills across post-graduate year (PGY) over time. SETTING: Data was collected from 3 independently ACGME-accredited surgery residency programs over 3 tertiary hospitals within our health system. PARTICIPANTS: General surgery residents in clinical PGY 1-5. RESULTS: Residents scored highest in properly identifying ACGME core competencies and determining Clavien-Dindo scores (p < 0.006) with no improvement in providing accurate and concise clinical summaries. However, residents improved in recording data sufficient to identify error (p < 0.00001). A positive linear trend in median scores for all remaining domains except for cognitive bias was demonstrated (p < 0.001). Senior residents scored significantly higher than junior residents in all domains. Scores > 90% were never achieved. CONCLUSIONS: The use of an electronic standardized critique algorithm in the evaluation and assessment of adverse surgical case outcomes enabled the measure of residents' critical thinking skills. Feedback in the form of teaching faculty-facilitated discussion and emailed grades enhanced adult learning with a steady improvement in performance over PGY. Although residents improved with PGY, the data suggest that further improvement in all categories is possible. Implementing this standardized critique algorithm across PGY allows for evaluation of areas of individual resident weakness vs. strength, progression over time, and comparisons to peers. These data suggest that routine complication reporting may be enhanced as a critical thinking assessment tool and that improvement in critical thinking can be quantified. Incorporation of this platform into M&M conference has the potential to augment executive function and professional identity development.

Practice-Based Learning and Improvement: Improving Morbidity and Mortality Review Using Natural Language Processing.

INTRODUCTION: Practice-Based Learning and Improvement, a core competency identified by the Accreditation Council for Graduate Medical Education, carries importance throughout a physician's career. Practice-Based Learning and Improvement is cultivated by a critical review of complications, yet methods to accurately identify complications are inadequate. Machine-learning algorithms show promise in improving identification of complications. We compare a manual-supplemented natural language processing (ms-NLP) methodology against a validated electronic morbidity and mortality (MM) database, the Morbidity and Mortality Adverse Event Reporting System (MARS) to understand the utility of NLP in MM review. METHODS: The number and severity of complications were compared between MARS and ms-NLP of surgical hospitalization discharge summaries among three academic medical centers. Clavien-Dindo (CD) scores were assigned to cases with identified complications and classified into minor (CD I-II) or major (CD III-IV) harm. RESULTS: Of 7774 admissions, 987 cases were identified to have 1659 complications by MARS and 1296 by ms-NLP. MARS identified 611 (62%) cases, whereas ms-NLP identified 670 (68%) cases. Less than one-third of cases (299, 30.3%) were detected by both methods. MARS identified a greater number of complications with major harm (457, 46.30%) than did ms-NLP (P < 0.0001). CONCLUSIONS: Both a prospectively maintained MM database and ms-NLP review of discharge summaries fail to identify a significant proportion of postoperative complications and overlap 1/3 of the time. ms-NLP more frequently identifies cases with minor complications, whereas prospective voluntary reporting more frequently identifies major complications. The educational benefit of reporting and analysis of complication data may be supplemented by ms-NLP but not replaced by it at this time.

Reducing surgical site infections after colectomy: bundle item compliance, process, and outlier identification.

BACKGROUND: The purpose of this study was to implement a checklist monitoring system and identify critical surgical checklist items associated with post-colectomy surgical site infections (SSI). The relationship between checklist compliance, infection rates, and identification of non-compliant surgeons was explored. MATERIALS AND METHODS: National Health Safety Network (NHSN) data were imported annually to establish baseline incidence of post-colectomy SSI from 2016 to 2019. A colectomy checklist was used to monitor compliance for 1694 random colectomies (1274 elective; 420 emergency). Reports were generated monthly to profile system, hospital, surgeon-specific infection, and checklist compliance rates. RESULTS: Checklist compliance improved in elective and emergent colectomies to > 90% for all items except oral antibiotic and mechanical bowel prep in elective cases. Annualized total SSI and organ space infection rates in elective cases decreased by 33% and 45%, respectively. Elective and emergency SSI's were reduced for Superficial Incisional Primary (SIP), Deep Incisional Primary (DIP), and Intra-Abdominal Abscess (IAB) by 66%, 60.4%, and 78.3%, respectively. Checklist compliance between low (< 3%) and high (> 3%) infection rate surgeons demonstrated significantly lower utilization of oral antibiotic prep (p < 0.03) and mechanical bowel prep (p < 0.02) in high infection rate surgeons. CONCLUSION: Surgeons compliant with colectomy checklists decreased elective and emergency colectomy infection rates. Ceiling compliance rates > 95% for bundle items are suggested to achieve optimal reductions in SSIs and efforts should be focused on surgeons with NHSN infection rates > 3%. Oral antibiotic prep and mechanical bowel prep compliance rates in elective colectomy appeared to differentiate high infection rate surgeons from low infection rate surgeons.

Cognitive bias and severity of harm following surgery: Plan for workflow debiasing strategy.

INTRODUCTION: This study analyzes the relationship between cognitive bias (CB) and harm severity as measured by Clavien-Dindo Scores (CD). METHODS: A prospectively collected series of 655 severity matched general surgical cases with complications were analyzed. Cases were evaluated for CB and assigned harm scores as defined by CD grade. Potentially mitigating "debiasing" strategies were identified for each bias attribution. RESULTS: Among cases with CB, 24% (55/232) were CD(I-II) and 76% (177/232) were CD(III-V). Odds ratio suggests that serious complications occur nearly 60% more frequently when CB is identified. The CBs identified with severe harm were Overconfidence, Commission, Anchoring, Confirmation, and Diagnosis Momentum. Preliminary data on debiasing strategies suggest diagnosis review, linear reasoning and Type II thinking may be relevant in over 85% of complications. CONCLUSION: The incidence of CB is increased in patients sustaining severe harm. Understanding the specific CBs identified and their mitigating debiasing strategies may improve outcomes.

Impact of Timing of Tocilizumab Use in Hospitalized Patients With SARS-CoV-2 Infection.

BACKGROUND: SARS-CoV-2 (severe acute respiratory syndrome coronavirus 2) continues to be a global challenge due to the lack of definitive treatment strategies. We sought to determine the efficacy of early administration of anti-interleukin 6 therapy in reducing hospital mortality and progression to mechanical ventilation. METHODS: This was a retrospective chart review of 11,512 patients infected with SARS-CoV-2 who were admitted to a New York health system from March to May 2020. Tocilizumab was administered to subjects at the nasal cannula level of oxygen support to maintain an oxygen saturation of >88%. The Charlson comorbidity index was used as an objective assessment of the burden of comorbidities to predict 10-year mortality. The primary outcome of interest was hospital mortality. Secondary outcomes were progression to mechanical ventilation; the prevalence of venous thromboembolism and renal failure; and the change in C-reactive protein, D-dimer, and ferritin levels after tocilizumab administration. Propensity score matching by using a 1:2 protocol was used to match the tocilizumab and non-tocilizumab groups to minimize selection bias. The groups were matched on baseline demographic characteristics, including age, sex, and body mass index; Charlson comorbidity index score; laboratory markers, including ferritin, D-dimer, lactate dehydrogenase, and C-reactive protein values; and the maximum oxygen requirement at the time of tocilizumab administration. Mortality outcomes were evaluated based on the level of oxygen requirement and the day of hospitalization at the time of tocilizumab administration. RESULTS: The overall hospital mortality was significantly reduced in the tocilizumab group when tocilizumab was administered at the nasal cannula level (10.4% vs 22.0%; P = .002). In subjects who received tocilizumab at the nasal cannula level, the progression to mechanical ventilation was reduced versus subjects who were initially on higher levels of oxygen support (6.3% vs 18.7%; P < .001). There was no improvement in mortality when tocilizumab was given at the time of requiring non-rebreather, high-flow nasal cannula, noninvasive ventilator, or invasive ventilator. CONCLUSIONS: Early use of anti-interleukin 6 therapy may be associated with improved hospital mortality and reduction in progression to more severe coronavirus disease 2019.

Core Competency Self-Assessment Enhances Critical Review of Complications and Entrustable Activities.

BACKGROUND: The Accreditation Council for Graduate Medical Education has defined six core competencies (CCs) that every successful physician should possess. However, the assessment of CC achievement among trainees is difficult. This project was designed to prospectively evaluate the impact of resident identification of CC as a component of morbidity review on error identification and standard of care (SOC) assessments. The platform was assessed for its reliability as a measure of resident critical analysis of complication causality across postgraduate year (PGY). MATERIALS AND METHODS: A total of 1945 general surgery cases with complications were assessed for error identification and SOC management between January 1, 2016, and December 31, 2018. CC identification was additionally assessed between January 1, 2019, and December 31, 2019, and included 708 general surgery cases. Data were evaluated for error assessments and overall SOC management. PGY4 and 5 residents were compared for number of cases and complications reviewed, severity, error causation, and CC relevance. RESULTS: Study groups were equivalent by Clavien-Dindo scores. Error identification significantly increased in all categories: diagnostic (P < 0.001), technical (P < 0.05), judgment (P < 0.001), system (P < 0.001), and communication (P < 0.001). Overall SOC assessments validated by a supervising surgical quality officer were unchanged. An increased exposure to cases with severe complications, error causation, and CC relevance was noted across PGY. CONCLUSIONS: The addition of CC assessment into morbidity review appears to improve the critical thinking of evaluating residents by increasing the identification of management errors. Used as an element of prospective self-assessment, teaching residents to identify CC principles in cases with complications may assist in learner progression toward clinical competence and critical thinking.

Cognitive Bias Impact on Management of Postoperative Complications, Medical Error, and Standard of Care.

BACKGROUND: Cognitive bias (CB) is increasingly recognized as an important source of medical error and up to 75% of errors in internal medicine are thought to be cognitive in origin (O'Sullivan ED, Schofield SJ. Cognitive bias in clinical medicine. J R Coll Physicans Edinb. 2018;48;225-232). However, primary data regarding the true incidence of bias is lacking. A prospective evaluation of CB in the management of surgical cases with complications has not been reported. This study reports the incidence and distribution of various types of CBs, and evaluates their impact on management errors and standard of care (SOC). METHODS: A prospectively collected series of 736 general surgical cases with complications from three university hospitals was analyzed. Surgical residents evaluated cases for 22 types of CBs (Croskerry P. The importance of cognitive errors in diagnosis and strategies to minimize them. Acad Med. 2003;78:775-780). Supervising quality officers validated all quality assessments. Data were assessed for the incidence of CBs, error assessments (diagnostic, technical, judgment, system, communication, therapeutic, and professionalism), and SOC. RESULTS: CB was attributed in 32.7% (241/736) of all cases with complications. The most common CBs identified, both singly and in groups, were anchoring, confirmation, omission, commission, overconfidence, premature closure, hindsight, diagnosis momentum, outcome, and ascertainment bias. The attribution of CB was correlated to a statistically significant increase in the incidence of management errors by the surgical team and lower SOC assessments. CONCLUSIONS: CBs are identified in the management of cases with complications and are associated with an increase in management errors and a degradation in SOC. Insight into the types of CBs and their association with the type and severity of management errors may prove useful in improving quality care.

A report card system using error profile analysis and concurrent morbidity and mortality review: surgical outcome analysis, part II.

BACKGROUND: An effective report card system for adverse outcome error analysis following surgery is lacking. We hypothesized that a memorialized database could be used in conjunction with error analysis and management evaluation at Morbidity & Mortality conference to generate individualized report cards for Attending Surgeon and System performance. STUDY DESIGN: Prospectively collected data from September 2000 through April 2005 were reported following Morbidity & Mortality review on 1618 adverse outcomes, including 219 deaths, following 29,237 operative procedures, in a complete loop to approximately 60 individual surgeons and responsible system personnel. RESULTS: A 40% reduction of gross mortality (P < 0.001) and 43% reduction of age-adjusted mortality were achieved over 4 years at the Academic Center. Quality issues were identified at a rate three times greater than required by New York State regulations and increased from a baseline 4.96% to 32.7% (odds ratio 1.94; P < 0.03) in cases associated with mortality. A detailed review demonstrated a significant increase (P < 0.001) in system errors and physician-related diagnostic and judgment errors associated with mortality highlighted those practices and processes involved, and contrasted the results between academic (43% mortality improvement) and community (no improvement) hospitals. CONCLUSIONS: The findings suggest that structured concurrent data collection combined with non-punitive error-based case review and individualized report cards can be used to provide detailed feedback on surgical performance to individual surgeons and possibly improve clinical outcomes.

Benchmarking surgical incident reports using a database and a triage system to reduce adverse outcomes.

OBJECTIVE: To study the profile of incidents affecting quality outcomes after surgery by developing a usable operating room and perioperative clinical incident report database and a functional electronic classification, triage, and reporting system. Previously, incident reports after surgery were handled on an individual, episodic basis, which limited the ability to perceive actuarial patterns and meaningfully improve outcomes. DESIGN, SETTING, AND PARTICIPANTS: Clinical incident reports were experientially generated in the second largest health care system in New York City. Data were entered into a functional classification system organized into 16 categories, and weekly triage meetings were held to electronically review and report summaries on 40 to 60 incident reports per week. System development and deployment reviewed 1041 reports after 19,693 operative procedures. During the next 4 years, 3819 additional reports were generated from 83,988 operative procedures and were reported electronically to the appropriate departments. MAIN OUTCOME MEASURES: Number of incident reports generated annually. RESULTS: A significant decrease in volume-adjusted clinical incident reports occurred (from 53 to 39 reports per 1000 procedures) from 2001 to 2005 (P < .001). Reductions in incident reports were observed for ambulatory conversions (74% reduction), wasted implants (65%), skin breakdown (64%), complications in the operating room (42%), laparoscopic conversions (32%), and cancellations (23%) as a result of data-focused process and clinical interventions. Six of 16 categories of incident reports accounted for more than 88% of all incident reports. CONCLUSION: These data suggest that effective review, communication, and summary feedback of clinical incident reports can produce a statistically significant decrease in adverse outcomes.

A morbidity and mortality conference-based classification system for adverse events: surgical outcome analysis: part I.

BACKGROUND: We hypothesized that an archive database in conjunction with Morbidity and Mortality (M&M) review could be used to define a systematic list of post-surgical adverse events and identify areas for performance improvement. STUDY DESIGN: Adverse event data following surgery were prospectively collected at the Beth Israel Medical Center in NYC from academic, specialty, community hospital, and ambulatory care settings over a 5-year period from September 2000 through April 2005. A classification system and analysis methodology was developed to guide and maximize the effectiveness of M&M review. RESULTS: A total of 1618 adverse events, including 219 deaths, were analyzed following 29,237 operative procedures according to the analysis method described. A list of 245 adverse events was classified among 15 groups, and a subgroup of 25 adverse events accounted for over 80% of total adverse events. Five categories of adverse events were associated with death in surgical patients and 4 of 5 categories were post-operative events. Used in conjunction with M&M review, data derived from this analysis highlighted those adverse events with the greatest clinical frequency to the department's quality profile. CONCLUSIONS: We present a classification system for surgical adverse events and propose a specific analysis method which may be used in conjunction with Morbidity and Mortality Conference to standardize the profiling of surgical performance.

Assessment of medical devices: how to conduct comparative technology evaluations of product performance.

OBJECTIVES: U.S. expenditures on medical devices (US dollars 70 billion in 2003) are one of the fastest growing components of hospital costs. Physicians' selection of medical devices lacks an evidence base on the comparative clinical effectiveness of these products. Comparative studies (e.g., vendor 1 versus vendor 2, technology A versus technology B) are increasingly promoted in the public sector as a means of cost containment, value-based purchasing, and quality improvement. This study illustrates how hospitals and physicians can conduct comparative technology assessments of product performance. METHODS: Surgeons evaluated comparable medical devices manufactured by eight different vendors in standardized surgical procedures. Devices included sutures and endomechanical products, which account for US dollars 2.5 billion of total device spending. Evaluations covered multiple performance dimensions, including ergonomics, functionality, clinical acceptability, and vendor preference. RESULTS: One vendor's products garnered consistently high ratings from surgeons, while two other vendors garnered consistently low ratings. Differences in ratings were statistically significant and persist when controlling for physician background characteristics and prior experience. Study results were used by a large hospital group purchasing organization to select which vendors to contract with for these products. CONCLUSIONS: Comparative technology evaluations assist physicians and hospitals in making cost-effective purchases of devices. These evaluations provide robust information on the performance of products routinely used by clinicians. Such evaluations can be carefully designed to have scientific rigor and clinical credibility.

The 80-hour work guidelines and resident survey perceptions of quality.

OBJECTIVE: We hypothesized that resident fatigue error should improve, related to well-rested trainees as a direct cause/effect benefit. However, patient hospital care quality is multifactorial, so impact on patient care quality by changing only one variable for a single caregiver group was unknown. DESIGN AND PARTICIPANTS: Convenience samples of 156 residents from three surgical specialties were administered a questionnaire in early 2004 addressing perceptions of patient care quality before and after the 80-h workweek. Additionally, residents recently under work-hour restrictions (Newly Restricted, NR) were compared to New York state trainees already regulated by work-hour restrictions (Previously Restricted, PR). SETTING: Surgical residency training venues. MAIN OUTCOME MEASURE: Survey results; the level of significance for all tests was 0.05. RESULTS: The participation response rate was 94.5%. Eighty-eight percent of respondents indicated by survey subjective impression that patient care quality was either unchanged (63%) or worse (26%) due to work-hour restrictions (P = 0.003). PR residents were more likely than NR residents to report unchanged or worse quality of care (P = 0.015). Residents overall did perceive improvement in some types of error with fewer fatigue-related errors (P < 0.001), e.g., medication (P < 0.001), judgment (P = 0.001), and dexterity (P = 0.013), subsequent to work-hour restrictions. However, more errors were perceived related to continuity of care (P < 0.001), miscommunication (P = 0.001), and cross-coverage availability (P = 0.001). CONCLUSIONS: Despite an expected perception of improvement in fatigue-related errors, most participants (particularly PR residents) reported impressions that patient care quality had remained unchanged or had declined under the work-hour restrictions. Unresolved challenges with continuity of care, miscommunication, and cross-coverage availability are possible explanations. Mere work-hour reduction does not appear to improve patient care quality automatically nor to decrease the possibility for some types of error. Process interventions that specifically target trainee sign-out coverage constraints as part of a global reassessment will be important for future attempts to enhance quality hospital patient care.

Barriers to implementing a surgical beta-blocker protocol.

BACKGROUND: Experience with a quality improvement (QI) program undertaken to increase the use of beta-adrenergic blockade in at-risk patients at both a major academic medical center and a community hospital suggests barriers to implementation. METHODS: A retrospective and prospective cohort study was performed to establish the incidence and effectiveness of beta-blockade use pre- and postimplementation of a standardized screening tool and a major education program as part of a QI project. Data gathering involved a baseline phase pre-intervention; 6 weeks postintervention; and 3-6 months postintervention. RESULTS: During phase I (baseline) 56% of eligible received beta-blockers, but targeted measures (a pre-induction heart rate < 70 or a systolic blood pressure [BP] < 110 mmHg) were achieved in only 11% of patients. Phase II saw a significant overall increase in beta-blocker administration (79%) and efficacy (50%). However, during phase III (3-6 months postimplementation), the rate of beta-blocker administration fell to 61% overall, while overall efficacy remained stable at 52%. Significant differences between the academic and community hospitals were observed throughout the study. CONCLUSION: Implementation of a quality program for beta-blockade is significantly affected by the presence or absence of ongoing physician and staff education beyond the study period.