How Reliable are Single-Question Workplace-Based Assessments in Surgery?

OBJECTIVE: Workplace-based assessments (WBAs) play an important role in the assessment of surgical trainees. Because these assessment tools are utilized by a multitude of faculty, inter-rater reliability is important to consider when interpreting WBA data. Although there is evidence supporting the validity of many of these tools, inter-reliability evidence is lacking. This study aimed to evaluate the inter-rater reliability of multiple operative WBA tools utilized in general surgery residency. DESIGN: General surgery residents and teaching faculty were recorded during 6 general surgery operations. Nine faculty raters each reviewed 6 videos and rated each resident on performance (using the Society for Improving Medical Professional Learning, or SIMPL, Performance Scale as well as the operative performance rating system (OPRS) Scale), entrustment (using the ten Cate Entrustment-Supervision Scale), and autonomy (using the Zwisch Scale). The ratings were reviewed for inter-rater reliability using percent agreement and intraclass correlations. PARTICIPANTS: Nine faculty members viewed the videos and assigned ratings for multiple WBAs. RESULTS: Absolute intraclass correlation coefficients for each scale ranged from 0.33 to 0.47. CONCLUSIONS: All single-item WBA scales had low to moderate inter-rater reliability. While rater training may improve inter-rater reliability for single observations, many observations by many raters are needed to reliably assess trainee performance in the workplace.

Improving Graduate Medical Education by Aggregating Data Across the Medical Education Continuum.

ABSTRACT: Meaningful improvements to graduate medical education (GME) have been achieved in recent decades, yet many GME improvement pilots have been small trials without rigorous outcome measures and with limited generalizability. Thus, lack of access to large-scale data is a key barrier to generating empiric evidence to improve GME. In this article, the authors examine the potential of a national GME data infrastructure to improve GME, review the output of 2 national workshops on this topic, and propose a path toward achieving this goal.The authors envision a future where medical education is shaped by evidence from rigorous research powered by comprehensive, multi-institutional data. To achieve this goal, premedical education, undergraduate medical education, GME, and practicing physician data must be collected using a common data dictionary and standards and longitudinally linked using unique individual identifiers. The envisioned data infrastructure could provide a foundation for evidence-based decisions across all aspects of GME and help optimize the education of individual residents.Two workshops hosted by the National Academies of Sciences, Engineering, and Medicine Board on Health Care Services explored the prospect of better using GME data to improve education and its outcomes. There was broad consensus about the potential value of a longitudinal data infrastructure to improve GME. Significant obstacles were also noted.Suggested next steps outlined by the authors include producing a more complete inventory of data already being collected and managed by key medical education leadership organizations, pursuing a grass-roots data sharing pilot among GME-sponsoring institutions, and formulating the technical and governance frameworks needed to aggregate data across organizations.The power and potential of big data is evident across many disciplines, and the authors believe that harnessing the power of big data in GME is the best next step toward advancing evidence-based physician education.

Variation in Competence of Graduating General Surgery Trainees.

OBJECTIVE: To examine the readiness of general surgery residents in their final year of training to perform 5 common surgical procedures based on their documented performance during training. DESIGN: Intraoperative performance ratings were analyzed using a Bayesian mixed effects approach, adjusting for rater, trainee, procedure, case complexity, and postgraduate year (PGY) as random effects as well as month in academic year and cumulative, procedure-specific performance per trainee as fixed effects. This model was then used to estimate each PGY 5 trainee's final probability of being able to independently perform each procedure. The actual, documented competency rates for individual trainees were then identified across each of the 5 most common general surgery procedures: appendectomy, cholecystectomy, ventral hernia repair, groin hernia repair, and partial colectomy. SETTING: This study was conducted using data from members of the SIMPL collaborative. PARTICIPANTS: A total of 17,248 evaluations of 927 PGY5 general surgery residents were analyzed from 2015 to 2021. RESULTS: The percentage of residents who requested a SIMPL rating during their PGY5 year and achieved a ≥90% probability of being rated as independent, or "Practice-Ready," was 97.4% for appendectomy, 82.4% for cholecystectomy, 43.5% for ventral hernia repair, 24% for groin hernia repair, and 5.3% for partial colectomy. CONCLUSIONS: There is substantial variation in the demonstrated competency of general surgery residents to perform several common surgical procedures at the end of their training. This variation in readiness calls for careful study of how surgical residents can become more adequately prepared to enter independent practice.

Mortality and Severe Complications Among Newly Graduated Surgeons in the United States.

OBJECTIVE: To evaluate severe complications and mortality over years of independent practice among general surgeons. BACKGROUND: Despite concerns that newly graduated general surgeons may be unprepared for independent practice, it is unclear whether patient outcomes differ between early and later career surgeons. METHODS: We used Medicare claims for patients discharged between July 1, 2007 and December 31, 2019 to evaluate 30-day severe complications and mortality for 26 operations defined as core procedures by the American Board of Surgery. Generalized additive mixed models were used to assess the association between surgeon years in practice and 30-day outcomes while adjusting for differences in patient, hospital, and surgeon characteristics. RESULTS: The cohort included 1,329,358 operations performed by 14,399 surgeons. In generalized mixed models, the relative risk (RR) of mortality was higher among surgeons in their first year of practice compared with surgeons in their 15th year of practice [5.5% (95% CI: 4.1%-7.3%) vs 4.7% (95% CI: 3.5%-6.3%), RR: 1.17 (95% CI: 1.11-1.22)]. Similarly, the RR of severe complications was higher among surgeons in their first year of practice compared with surgeons in their 15th year of practice [7.5% (95% CI: 6.6%-8.5%) versus 6.9% (95% CI: 6.1%-7.9%), RR: 1.08 (95% CI: 1.03-1.14)]. When stratified by individual operation, 21 operations had a significantly higher RR of mortality and all 26 operations had a significantly higher RR of severe complications in the first compared with the 15th year of practice. CONCLUSIONS: Among general surgeons performing common operations, rates of mortality and severe complications were higher among newly graduated surgeons compared with later career surgeons.

Evaluating Educational Outcomes Using Patient Outcomes of New Surgeons Performing Partial Colectomy Compared to Cholecystectomy.

PURPOSE: Despite ongoing efforts to improve surgical education, surgical residents face gaps in their training. However, it is unknown if differences in the training of surgeons are reflected in the patient outcomes of those surgeons once they enter practice. This study aimed to compare the patient outcomes among new surgeons performing partial colectomy-a common procedure for which training is limited-and cholecystectomy-a common procedure for which training is robust. METHOD: The authors retrospectively analyzed all adult Medicare claims data for patients undergoing inpatient partial colectomy and inpatient cholecystectomy between 2007 and 2018. Generalized additive mixed models were used to investigate the associations between surgeon years in practice and risk-adjusted rates of 30-day serious complications and death for patients undergoing partial colectomy and cholecystectomy. RESULTS: A total of 14,449 surgeons at 4,011 hospitals performed 340,114 partial colectomy and 355,923 cholecystectomy inpatient operations during the study period. Patients undergoing a partial colectomy by a surgeon in their 1st vs 15th year of practice had higher rates of serious complications (5.22% [95% CI, 4.85%-5.60%] vs 4.37% [95% CI, 4.22%-4.52%]; P < .01) and death (3.05% [95% CI, 2.92%-3.17%] vs 2.83% [95% CI, 2.75%-2.91%]; P < .01). Patients undergoing a cholecystectomy by a surgeon in their 1st vs 15th year of practice had similar rates of 30-day serious complications (4.11% vs 3.89%; P = .11) and death (1.71% vs 1.70%; P = .93). CONCLUSIONS: Patients undergoing partial colectomy faced a higher risk of serious complications and death when the operation was performed by a new surgeon compared to an experienced surgeon. Conversely, patient outcomes following cholecystectomy were similar for new and experienced surgeons. More attention to partial colectomy during residency training may benefit patients.

Association of Surgical Resident Competency Ratings With Patient Outcomes.

PURPOSE: Accurate assessment of clinical performance is essential to ensure graduating residents are competent for unsupervised practice. The Accreditation Council for Graduate Medical Education milestones framework is the most widely used competency-based framework in the United States. However, the relationship between residents' milestones competency ratings and their subsequent early career clinical outcomes has not been established. It is important to examine the association between milestones competency ratings of U.S. general surgical residents and those surgeons' patient outcomes in early career practice. METHOD: A retrospective, cross-sectional study was conducted using a sample of national Medicare claims for 23 common, high-risk inpatient general surgical procedures performed between July 1, 2015, and November 30, 2018 (n = 12,400 cases) by nonfellowship-trained U.S. general surgeons. Milestone ratings collected during those surgeons' last year of residency (n = 701 residents) were compared with their risk-adjusted rates of mortality, any complication, or severe complication within 30 days of index operation during their first 2 years of practice. RESULTS: There were no associations between mean milestone competency ratings of graduating general surgery residents and their subsequent early career patient outcomes, including any complication (23% proficient vs 22% not yet proficient; relative risk [RR], 0.97, [95% CI, 0.88-1.08]); severe complication (9% vs 9%, respectively; RR, 1.01, [95% CI, 0.86-1.19]); and mortality (5% vs 5%; RR, 1.07, [95% CI, 0.88-1.30]). Secondary analyses yielded no associations between patient outcomes and milestone ratings specific to technical performance, or between patient outcomes and composites of operative performance, professionalism, or leadership milestones ratings ( P ranged .32-.97). CONCLUSIONS: Milestone ratings of graduating general surgery residents were not associated with the patient outcomes of those surgeons when they performed common, higher-risk procedures in a Medicare population. Efforts to improve how milestones ratings are generated might strengthen their association with early career outcomes.

Laparoscopic transversus abdominis plane block reduces postoperative opioid requirements after laparoscopic cholecystectomy.

BACKGROUND: Surgeons directly contribute to the over-prescription of opioids. Alternative postoperative pain management strategies are necessary to reduce opioid dispensation and combat the opioid epidemic. We set out to examine the effectiveness of a laparoscopic transversus abdominis plane block on reducing opioid requirements after laparoscopic cholecystectomy. METHODS: In a retrospective cohort analysis, we compared opioid naïve patients who underwent an elective, outpatient laparoscopic cholecystectomy with a transversus abdominis plane block with patients who underwent a laparoscopic cholecystectomy alone between January 2018 and June 2021 at a single institution. Patient characteristics, perioperative pain scores, and postoperative analgesic requirements were compared between cohorts. RESULTS: There were 200 patients included in the study (laparoscopic cholecystectomy with a transversus abdominis plane block, n = 100; laparoscopic cholecystectomy alone, n = 100). The average postoperative pain scores in the postanesthesia care unit were equivalent between the groups (laparoscopic cholecystectomy with a transversus abdominis plane block = 3.39 versus laparoscopic cholecystectomy alone = 4.17, P = .12), with the mean postanesthesia care unit opioid requirements significantly lower in patients receiving laparoscopic cholecystectomy with a transversus abdominis plane block (12.1 vs 20.4 oral morphine equivalents, P < .001). Patients receiving laparoscopic cholecystectomy with a transversus abdominis plane block were prescribed fewer opioids on discharge (mean 77.5 vs 92.9 oral morphine equivalents, P < .05) and reported using a lower proportion of their opioid prescription at follow-up (83.2% vs 100%, P < .001). Of the patients receiving laparoscopic cholecystectomy with a transversus abdominis plane block, 65% reported using over-the-counter pain medications compared with 82% of patients receiving laparoscopic cholecystectomy alone (P < .001). CONCLUSION: Performing a laparoscopic transversus abdominis plane block during elective laparoscopic cholecystectomy is a safe and effective strategy to reduce postoperative opioid requirements for the treatment of acute postoperative pain.

Opportunities to Increase Workplace Feedback: A Multi-institutional Examination.

OBJECTIVE: While feedback is an essential component of resident education, there are few large-scale studies examining when and under what conditions formative feedback is provided. Workplace-based assessment systems offer an opportunity to identify factors influencing when faculty provides feedback to trainees. Influential factors affecting feedback may provide targets for increasing and improving feedback in resident education. DESIGN: Data on whether dictated feedback was provided were obtained from the Society for Improving Medical Professional Learning (SIMPL) mobile application. We used generalized linear mixed effects models to identify the degree to which faculty members, procedures, surgical case characteristics, and trainee performance were associated with whether narrative feedback was provided using SIMPL. SETTING: This study was conducted using data from members of the SIMPL collaborative. PARTICIPANTS: 67,434 evaluations from 70 general surgery programs were included from 2015 to 2021. Of these, 25,355 evaluations included dictated feedback. RESULTS: Approximately 61% of the variation in whether dictated feedback was provided was attributable to the individual faculty member. Compared to residents who achieved autonomy ratings of "Active Help," residents who achieved ratings of "Supervision Only" (odds ratio (OR) = 0.80, 95% confidence interval (CI) = 0.72, 0.88) had a lower likelihood of receiving dictated feedback. Residents who achieved ratings of "Intermediate" (OR = 0.81, CI = 0.74, 0.89), "Practice-Ready" (OR = 0.50, CI = 0.45, 0.57), or "Exceptional (OR = 0.64, CI = 0.54, 0.76) showed a lower likelihood of receiving dictated feedback compared to those rated as "Inexperienced." Cases rated as "High" in terms of complexity were associated with an increased likelihood of having dictation (OR = 1.35, CI = 1.26, 1.44). CONCLUSIONS: The largest contributing factor for whether dictated feedback is included in a SIMPL evaluation are factors specific to the attending surgeon. Resident performance, resident autonomy, and case complexity had only modest associations with feedback decisions. Efforts to improve the amount of formative feedback for trainees should be directed towards reducing the variation in which attending surgeons elect to provide feedback.

How Do Resident Surgeons Identify Operative Case Complexity? An Analysis of Resident versus Attending Perceptions.

OBJECTIVE: Accurate recognition of patient-related complexity of an operation is critical for appropriate surgical decision making. It is not yet understood whether general surgery residents are able to accurately assess the relative complexity of a given operative case. This study investigates the agreement of case complexity ratings between residents and attending surgeons and explores whether resident-related factors correlate with any discordance in perception of patient-related operative complexity. DESIGN: Residents and attending surgeons rated the relative complexity of completed cases on a 3 point scale via the SIMPL (Society for Improving Medical Professional Learning) operative assessment smartphone app. Additional trainee demographic data, autonomy ratings, and performance ratings were also obtained from the SIMPL registry for each rated case. Complexity agreement was defined as an equal rating between the resident and attending and assigned a value of zero. Over-estimate ratings were assigned a positive value and under-estimate ratings were assigned a negative value. Trends in complexity agreement were analyzed using descriptive statistics and mixed-effects models. RESULTS: A total of 43,179 general surgery cases were rated by 1946 categorical general surgery residents and 1520 attending surgeons between 2015 and 2020. Residents and attendings agreed on case complexity in 63.23% of cases, while the residents overestimated complexity in 13.37% of cases and underestimated complexity in 23.40% of cases. Every level of resident except post-graduate year 2 had similar rates of agreement about the complexity of a procedure, while residents who received a higher autonomy rating were more likely to be in agreement with the faculty raters (OR 1.12, 95% CI 1.06-1.19). CONCLUSIONS: The results of this study suggest that general surgery residents inaccurately perceive the patient-related complexity of a given case approximately one third of the time. Greater experience and operative autonomy appear to be associated with higher complexity agreement. Future research into factors influencing perceived case complexity may provide insight into how to best implement new teaching for surgical residents regarding the concept of case complexity.

Novel method to link surgical trainee performance data to patient outcomes.

BACKGROUND: A significant roadblock in surgical education research has been the inability to compare trainee performance to the outcomes of those surgeons after they enter independent practice. We describe the feasibility of an innovative method to link trainee performance data with patient outcomes. METHODS: We extracted surgeon NPI numbers from Medicare claims data for common general surgery procedures between 2007 and 2017. Next, American Board of Surgery (ABS) trainee performance data was cross-referenced with additional resources to supplement NPI data. The patient and trainee datasets were linked using NPI number and a linkage rate was calculated. RESULTS: We identified 12,952 unique surgeons in the Medicare file. Medicare surgeons were matched with ABS records by NPI number, with 96.2% (n = 12,460) of surgeons linked successfully. CONCLUSIONS: We demonstrated a novel process to link patient outcomes to trainee performance. This innovation can enable future research investigating the relationship between surgical trainee performance and patient outcomes in independent practice.

Children With Arachnoid Cysts Who Sustain Blunt Head Trauma: Injury Mechanisms and Outcomes.

OBJECTIVES: Arachnoid cysts are abnormal intracranial fluid collections, and there is concern that these cysts may bleed or rupture following blunt head trauma. Our objective was to determine the risk of cyst-related complications in a cohort of children with arachnoid cysts who were evaluated for head trauma. METHODS: We analyzed the Pediatric Emergency Care Applied Research Network (PECARN) head trauma public use data set, which was the product of a study that enrolled children with blunt head trauma from June 2004 to September 2006. We identified children with arachnoid cysts on cranial computed tomography (CT) and described the patient demographics, mechanisms of injury, clinical presentations, CT evidence of traumatic brain injury (TBI), and clinical outcomes. Clinically important TBI was defined as TBI leading to: 1) death from TBI, 2) neurosurgical intervention, 3) intubation for > 24 hours for the TBI, or 4) hospitalization for 2 or more nights for the head injury in association with TBI on CT. RESULTS: Data were available for 43,399 children who sustained blunt head trauma, of whom 15,899 had cranial CT scans obtained and 68 (0.4%) had arachnoid cysts. Falls were the most common mechanisms of injury (47%) and 87% of children had either moderate or severe injury mechanisms. Glasgow Coma Scale (GCS) scores ranged from 6 to 15, with 61 (90%) having GCS scores of 15. Two of the children with arachnoid cysts had TBIs on CT, one of which was clinically important. There were no identified cases of arachnoid cyst-related bleeding or complications. CONCLUSIONS: In this cohort of 68 children with arachnoid cysts who sustained head trauma, none demonstrated cyst-related bleeding or complications. This suggests the risk of arachnoid cyst-related complications in children following blunt head trauma is low and evaluation should align with existing clinical decision rules.

Quality of life in liver transplant candidates: frailty is a better indicator than severity of liver disease.

BACKGROUND: In an effort to understand the diminished quality of life (QoL) exhibited by patients with end-stage liver disease (ESLD), we studied the association of frailty and severity of liver disease with quality of life in this patient population. METHODS: In a prospective, single-center cohort study (N=487), we assessed frailty and QoL in patients with ESLD referred for liver transplant. Frailty was measured on a scale from 0 to 5 by grip strength, gait speed, exhaustion, shrinkage, and physical activity, with scores of 3 or higher characterized as frail. Physical, mental, and combined overall quality of life scores ranging from 0 to 100 were assessed using Short Form 36. Pearson correlation and multiple linear regression were used to identify variables associated with QoL. RESULTS: Quality of life was notably low in the study cohort (mean: physical, 42.9±24.1; mental, 58.3±23.2). In multivariate analysis adjusted for demographic and clinical characteristics, frailty was significantly negative associated with physical (slope, -22.55, 95% confidence interval, -26.39 to -18.71; P<0.001) and mental QoL (slope, -17.59, 95% confidence interval, -21.47 to -13.71; P<0.001). Model for ESLD (MELD) was not associated with QoL. CONCLUSION: In ESLD patient referred for liver transplant, diminished QoL appears to be significantly negatively associated with frailty and not with severity of liver disease as measured MELD. With further study, if frailty is shown to be a remediable condition, targeted programs may help decrease frailty and improve quality of life in ESLD patients.