RESUMO
INTRODUCTION: Surgery residents who complete a nondesignated preliminary year have an additional year of training compared to those who begin as categorical residents. While this additional year is sometimes perceived negatively, these more experienced residents may outperform traditional categorical (TC) residents in their first year of training. METHODS: Operative assessment ratings were recorded for first year categorical general surgery residents in the United States between 2015 and 2023 using the Society for Improving Medical and Professional Learning assessment platform. Residents were categorized based on the completion of a nondesignated preliminary year ("Previous Prelim" [PP]) or not ("Traditional Categorical"). Ratings were analyzed using generalized mixed effects models. Performance and autonomy outcomes were dichotomized: "less experience" or "more experience" and "no autonomy" or "some autonomy", respectively. Fixed effects included academic month and case complexity, while random effects included resident, faculty, program, and procedure. RESULTS: A total of 34,353 evaluations from 86 general surgery programs were collected. Of these, 829 were evaluations from PP residents. Faculty ratings of PP versus TC revealed no differences in adjusted probabilities of achieving a "more experience" rating (82% versus 76%, P = 0.26) but a higher adjusted probability of achieving a "some autonomy" rating (88% versus 80%, P = 0.04) for PP compared to TC. Analysis of resident self-reported ratings revealed higher adjusted probabilities of a "more experience" rating (77% versus 50%, P = 0.01) and "some autonomy" rating (87% versus 73%; P = 0.02) for PP compared to TC. CONCLUSIONS: First year general surgery residents who previously completed a preliminary year have similar operative performance faculty ratings when compared to their peers.
RESUMO
OBJECTIVE: To provide an overview of the current use of Entrustable Professional Activities (EPAs) in postgraduate general surgery training internationally. BACKGROUND: Entrustable Professional Activities (EPAs) were introduced to connect clinical competencies and the professional activities to be entrusted to trainees on graduation. The popularity of EPAs as a framework for assessment is growing globally, including in general surgery. Anecdotally, there appears to be substantial variation in how they are implemented, yet a formal comparison of their use in postgraduate general surgery training is lacking. METHODS: A scoping review was performed, based on the original five-stage approach described by Arksey and O'Malley with the addition of protocol-specific items from the Preferred Reporting Items for Systematic Reviews and Meta-Analysis Protocols extension for scoping reviews (PRISMA-ScR). RESULTS: Twenty-nine published and grey literature sources were included in the review. Entrustable Professional Activity use in postgraduate general surgery training was identified in 11 unique contexts, including from North America, South America, Europe, Asia, Africa, and Australia. There were substantial differences in the scope and number of EPAs, tools used for EPA assessment, and how EPAs were sequenced through training. Despite wide variation, eight distinct EPAs were common to the majority (>80%) of countries. Several articles described findings of EPA use in postgraduate general surgery training, allowing identification of multiple barriers and facilitators to integration. CONCLUSIONS: This review provides guidance for certification and regulatory bodies, program directors, and institutions with ambitions to implement EPAs for assessment and curricular design. In settings where EPAs are already used, the data may facilitate refinement of programs and strategies.
RESUMO
OBJECTIVE: Workplace-based assessments (WBAs) play an important role in the assessment of surgical trainees. Because these assessment tools are utilized by a multitude of faculty, inter-rater reliability is important to consider when interpreting WBA data. Although there is evidence supporting the validity of many of these tools, inter-reliability evidence is lacking. This study aimed to evaluate the inter-rater reliability of multiple operative WBA tools utilized in general surgery residency. DESIGN: General surgery residents and teaching faculty were recorded during 6 general surgery operations. Nine faculty raters each reviewed 6 videos and rated each resident on performance (using the Society for Improving Medical Professional Learning, or SIMPL, Performance Scale as well as the operative performance rating system (OPRS) Scale), entrustment (using the ten Cate Entrustment-Supervision Scale), and autonomy (using the Zwisch Scale). The ratings were reviewed for inter-rater reliability using percent agreement and intraclass correlations. PARTICIPANTS: Nine faculty members viewed the videos and assigned ratings for multiple WBAs. RESULTS: Absolute intraclass correlation coefficients for each scale ranged from 0.33 to 0.47. CONCLUSIONS: All single-item WBA scales had low to moderate inter-rater reliability. While rater training may improve inter-rater reliability for single observations, many observations by many raters are needed to reliably assess trainee performance in the workplace.
Assuntos
Competência Clínica , Avaliação Educacional , Cirurgia Geral , Internato e Residência , Local de Trabalho , Cirurgia Geral/educação , Reprodutibilidade dos Testes , Humanos , Avaliação Educacional/métodos , Educação de Pós-Graduação em Medicina/métodos , Gravação em Vídeo , Docentes de Medicina , Masculino , FemininoRESUMO
OBJECTIVE: The objective of this study is to explore the patient characteristics and practice patterns of non-certified surgeons who treat Medicare patients in the United States. SUMMARY BACKGROUND DATA: While most surgeons in the United States are board-certified, non-certified surgeons are permitted to practice in many locations. At the same time, surgical workforce shortages threaten access to surgical care for many patients. It is possible that non-certified surgeons may be able to help fill these access gaps. However, little is known about the practice patterns of non-certified surgeons. METHODS: A 100% sample of Medicare claims data from 2014-2019 were used to identify practicing general surgeons. Surgeons were categorized as certified or non-certified in general surgeryââ based on data from the American Board of Surgery. Surgeon practice patterns and patient characteristics were analyzed. RESULTS: A total of 2,097,206 patient cases were included in the study. These patients were treated by 16,076 surgeons, of which 6% were identified as non-certified surgeons. Compared to certified surgeons, non-certified surgeons were less frequently fellowship-trained (20.5% vs. 24.2%, P=0.008) and more likely to be a foreign medical graduate (14.5% vs. 9.2%, P<0.001). Non-certified surgeons were more frequently practicing in for-profit hospitals (21.2% vs. 14.2%, P<0.001) and critical access hospitals (2.2% vs. 1.3%, P<0.001), and were less likely to practice in a teaching hospital (63.2% vs. 72.4%, P<0.001). Compared to certified surgeons, non-certified surgeons treated more non-White patients (19.6% vs. 14%, P<0.001) as well as a higher percentage of patients in the two lowest socioeconomic status (SES) quintiles (36.2% vs. 29.2%, P<0.001). Operations related to emergency admissions were more common amongst non-certified surgeons (68.8% vs. 55.7%, P<0.001). There were no differences in gender or age of the patients treated by certified and non-certified surgeons. CONCLUSION: For Medicare patients, non-certified surgeons treated more patients who are non-White, of lower SES, and in more rural, critical-access hospitals.
RESUMO
PURPOSE: The era of precision education is increasingly leveraging electronic health record (EHR) data to assess residents' clinical performance. But precision in what the EHR-based resident performance metrics are truly assessing is not fully understood. For instance, there is limited understanding of how EHR-based measures account for the influence of the team on an individual's performance-or conversely how an individual contributes to team performances. This study aims to elaborate on how the theoretical understandings of supportive and collaborative interdependence are captured in residents' EHR-based metrics. METHOD: Using a mixed methods study design, the authors conducted a secondary analysis of 5 existing quantitative and qualitative datasets used in previous EHR studies to investigate how aspects of interdependence shape the ways that team-based care is provided to patients. RESULTS: Quantitative analyses of 16 EHR-based metrics found variability in faculty and resident performance (both between and within resident). Qualitative analyses revealed that faculty lack awareness of their own EHR-based performance metrics, which limits their ability to act interdependently with residents in an evidence-informed fashion. The lens of interdependence elucidates how resident practice patterns develop across residency training, shifting from supportive to collaborative interdependence over time. Joint displays merging the quantitative and qualitative analyses showed that residents are aware of variability in faculty's practice patterns and that viewing resident EHR-based measures without accounting for the interdependence of residents with faculty is problematic, particularly within the framework of precision education. CONCLUSIONS: To prepare for this new paradigm of precision education, educators need to develop and evaluate theoretically robust models that measure interdependence in EHR-based metrics, affording more nuanced interpretation of such metrics when assessing residents throughout training.
Assuntos
Registros Eletrônicos de Saúde , Internato e Residência , Humanos , Competência Clínica , EscolaridadeRESUMO
OBJECTIVE: To evaluate severe complications and mortality over years of independent practice among general surgeons. BACKGROUND: Despite concerns that newly graduated general surgeons may be unprepared for independent practice, it is unclear whether patient outcomes differ between early and later career surgeons. METHODS: We used Medicare claims for patients discharged between July 1, 2007 and December 31, 2019 to evaluate 30-day severe complications and mortality for 26 operations defined as core procedures by the American Board of Surgery. Generalized additive mixed models were used to assess the association between surgeon years in practice and 30-day outcomes while adjusting for differences in patient, hospital, and surgeon characteristics. RESULTS: The cohort included 1,329,358 operations performed by 14,399 surgeons. In generalized mixed models, the relative risk (RR) of mortality was higher among surgeons in their first year of practice compared with surgeons in their 15th year of practice [5.5% (95% CI: 4.1%-7.3%) vs 4.7% (95% CI: 3.5%-6.3%), RR: 1.17 (95% CI: 1.11-1.22)]. Similarly, the RR of severe complications was higher among surgeons in their first year of practice compared with surgeons in their 15th year of practice [7.5% (95% CI: 6.6%-8.5%) versus 6.9% (95% CI: 6.1%-7.9%), RR: 1.08 (95% CI: 1.03-1.14)]. When stratified by individual operation, 21 operations had a significantly higher RR of mortality and all 26 operations had a significantly higher RR of severe complications in the first compared with the 15th year of practice. CONCLUSIONS: Among general surgeons performing common operations, rates of mortality and severe complications were higher among newly graduated surgeons compared with later career surgeons.
Assuntos
Medicare , Cirurgiões , Humanos , Estados Unidos/epidemiologia , Idoso , Hospitais , Mortalidade Hospitalar , Competência Clínica , Complicações Pós-Operatórias/epidemiologia , Estudos RetrospectivosRESUMO
OBJECTIVE: To determine the relationship between race/ethnicity and case volume among graduating surgical residents. BACKGROUND: Racial/ethnic minority individuals face barriers to entry and advancement in surgery; however, no large-scale investigations of the operative experience of racial/ethnic minority residents have been performed. METHODS: A multi-institutional retrospective analysis of the Accreditation Council for Graduate Medical Education case logs of categorical general surgery residents at 20 programs in the US Resident OPerative Experience Consortium database was performed. All residents graduating between 2010 and 2020 were included. The total, surgeon chief, surgeon junior, and teaching assistant case volumes were compared between racial/ethnic groups. RESULTS: The cohort included 1343 residents. There were 211 (15.7%) Asian, 65 (4.8%) Black, 73 (5.4%) Hispanic, 71 (5.3%) "Other" (Native American or Multiple Race), and 923 (68.7%) White residents. On adjusted analysis, Black residents performed 76 fewer total cases (95% CI, -109 to -43, P <0.001) and 69 fewer surgeon junior cases (-98 to -40, P <0.001) than White residents. Comparing adjusted total case volume by graduation year, both Black residents and White residents performed more cases over time; however, there was no difference in the rates of annual increase (10 versus 12 cases per year increase, respectively, P =0.769). Thus, differences in total case volume persisted over the study period. CONCLUSIONS: In this multi-institutional study, Black residents graduated with lower case volume than non-minority residents throughout the previous decade. Reduced operative learning opportunities may negatively impact professional advancement. Systemic interventions are needed to promote equitable operative experience and positive culture change.
Assuntos
Cirurgia Geral , Internato e Residência , Humanos , Estudos Retrospectivos , Etnicidade , Competência Clínica , Grupos Minoritários , Educação de Pós-Graduação em Medicina , Cirurgia Geral/educaçãoRESUMO
ABSTRACT: Meaningful improvements to graduate medical education (GME) have been achieved in recent decades, yet many GME improvement pilots have been small trials without rigorous outcome measures and with limited generalizability. Thus, lack of access to large-scale data is a key barrier to generating empiric evidence to improve GME. In this article, the authors examine the potential of a national GME data infrastructure to improve GME, review the output of 2 national workshops on this topic, and propose a path toward achieving this goal.The authors envision a future where medical education is shaped by evidence from rigorous research powered by comprehensive, multi-institutional data. To achieve this goal, premedical education, undergraduate medical education, GME, and practicing physician data must be collected using a common data dictionary and standards and longitudinally linked using unique individual identifiers. The envisioned data infrastructure could provide a foundation for evidence-based decisions across all aspects of GME and help optimize the education of individual residents.Two workshops hosted by the National Academies of Sciences, Engineering, and Medicine Board on Health Care Services explored the prospect of better using GME data to improve education and its outcomes. There was broad consensus about the potential value of a longitudinal data infrastructure to improve GME. Significant obstacles were also noted.Suggested next steps outlined by the authors include producing a more complete inventory of data already being collected and managed by key medical education leadership organizations, pursuing a grass-roots data sharing pilot among GME-sponsoring institutions, and formulating the technical and governance frameworks needed to aggregate data across organizations.The power and potential of big data is evident across many disciplines, and the authors believe that harnessing the power of big data in GME is the best next step toward advancing evidence-based physician education.
Assuntos
Educação Médica , Internato e Residência , Medicina , Humanos , Agregação de Dados , Educação de Pós-Graduação em Medicina , EscolaridadeRESUMO
OBJECTIVE: To examine the readiness of general surgery residents in their final year of training to perform 5 common surgical procedures based on their documented performance during training. DESIGN: Intraoperative performance ratings were analyzed using a Bayesian mixed effects approach, adjusting for rater, trainee, procedure, case complexity, and postgraduate year (PGY) as random effects as well as month in academic year and cumulative, procedure-specific performance per trainee as fixed effects. This model was then used to estimate each PGY 5 trainee's final probability of being able to independently perform each procedure. The actual, documented competency rates for individual trainees were then identified across each of the 5 most common general surgery procedures: appendectomy, cholecystectomy, ventral hernia repair, groin hernia repair, and partial colectomy. SETTING: This study was conducted using data from members of the SIMPL collaborative. PARTICIPANTS: A total of 17,248 evaluations of 927 PGY5 general surgery residents were analyzed from 2015 to 2021. RESULTS: The percentage of residents who requested a SIMPL rating during their PGY5 year and achieved a ≥90% probability of being rated as independent, or "Practice-Ready," was 97.4% for appendectomy, 82.4% for cholecystectomy, 43.5% for ventral hernia repair, 24% for groin hernia repair, and 5.3% for partial colectomy. CONCLUSIONS: There is substantial variation in the demonstrated competency of general surgery residents to perform several common surgical procedures at the end of their training. This variation in readiness calls for careful study of how surgical residents can become more adequately prepared to enter independent practice.
Assuntos
Cirurgia Geral , Hérnia Inguinal , Hérnia Ventral , Internato e Residência , Humanos , Teorema de Bayes , Competência Clínica , Educação de Pós-Graduação em Medicina/métodos , Hérnia Inguinal/cirurgia , Hérnia Ventral/cirurgia , Cirurgia Geral/educaçãoRESUMO
Competency-based medical education (CBME) is the future of medical education and relies heavily on high quality assessment. However, the current assessment practices employed by many general surgery graduate medical education training programs are subpar. Assessments often lack reliability and validity evidence, have low faculty engagement, and differ from program to program. Given the importance of assessment in CBME, it is critical that we build a better assessment system for measuring trainee competency. We propose that an ideal system of assessment is standardized, evidence-based, comprehensive, integrated, and continuously improving. In this article, we explore these characteristics and propose next steps to achieve such a system of assessment in general surgery.
Assuntos
Educação de Pós-Graduação em Medicina , Educação Médica , Humanos , Reprodutibilidade dos Testes , Educação Baseada em Competências , Docentes de Medicina , Competência ClínicaRESUMO
ABSTRACT: Medical training programs and health care systems collect ever-increasing amounts of educational and clinical data. These data are collected with the primary purpose of supporting either trainee learning or patient care. Well-established principles guide the secondary use of these data for program evaluation and quality improvement initiatives. More recently, however, these clinical and educational data are also increasingly being used to train artificial intelligence (AI) models. The implications of this relatively unique secondary use of data have not been well explored. These models can support the development of sophisticated AI products that can be commercialized. While these products have the potential to support and improve the educational system, there are challenges related to validity, patient and learner consent, and biased or discriminatory outputs. The authors consider the implications of developing AI models and products using educational and clinical data from learners, discuss the uses of these products within medical education, and outline considerations that should guide the appropriate use of data for this purpose. These issues are further explored by examining how they have been navigated in an educational collaborative.
Assuntos
Inteligência Artificial , Educação Médica , Humanos , Escolaridade , Aprendizagem , Avaliação de Programas e Projetos de SaúdeRESUMO
Over the past decade, medical education has shifted from a time-based approach to a competency-based approach for surgical training. This transition presents many new systemic challenges. The Society for Improving Medical Professional Learning (SIMPL) was created to respond to these challenges through coordinated collaboration across an international network of medical educators. The primary goal of the SIMPL network was to implement a workplace-based assessment and feedback platform. To date, SIMPL has developed, implemented, and sustained a platform that represents the earliest and largest effort to support workplace-based assessment at scale. The SIMPL model for collaborative improvement demonstrates a potential approach to addressing other complex systemic problems in medical education.
RESUMO
PURPOSE: Despite ongoing efforts to improve surgical education, surgical residents face gaps in their training. However, it is unknown if differences in the training of surgeons are reflected in the patient outcomes of those surgeons once they enter practice. This study aimed to compare the patient outcomes among new surgeons performing partial colectomy-a common procedure for which training is limited-and cholecystectomy-a common procedure for which training is robust. METHOD: The authors retrospectively analyzed all adult Medicare claims data for patients undergoing inpatient partial colectomy and inpatient cholecystectomy between 2007 and 2018. Generalized additive mixed models were used to investigate the associations between surgeon years in practice and risk-adjusted rates of 30-day serious complications and death for patients undergoing partial colectomy and cholecystectomy. RESULTS: A total of 14,449 surgeons at 4,011 hospitals performed 340,114 partial colectomy and 355,923 cholecystectomy inpatient operations during the study period. Patients undergoing a partial colectomy by a surgeon in their 1st vs 15th year of practice had higher rates of serious complications (5.22% [95% CI, 4.85%-5.60%] vs 4.37% [95% CI, 4.22%-4.52%]; P < .01) and death (3.05% [95% CI, 2.92%-3.17%] vs 2.83% [95% CI, 2.75%-2.91%]; P < .01). Patients undergoing a cholecystectomy by a surgeon in their 1st vs 15th year of practice had similar rates of 30-day serious complications (4.11% vs 3.89%; P = .11) and death (1.71% vs 1.70%; P = .93). CONCLUSIONS: Patients undergoing partial colectomy faced a higher risk of serious complications and death when the operation was performed by a new surgeon compared to an experienced surgeon. Conversely, patient outcomes following cholecystectomy were similar for new and experienced surgeons. More attention to partial colectomy during residency training may benefit patients.
Assuntos
Medicare , Cirurgiões , Adulto , Humanos , Idoso , Estados Unidos/epidemiologia , Estudos Retrospectivos , Colecistectomia/efeitos adversos , Colectomia/efeitos adversos , Colectomia/educação , Colectomia/métodosRESUMO
BACKGROUND: The SIMPL operative feedback tool is used in many U.S. surgical residency programs. However, the challenges of implementation and benefits of the web-based platform in low- and middle-income countries are unknown. The aim of this study was to evaluate implementation of SIMPL in a general surgery residency training program in Kenya. METHODS: SIMPL was pilot tested at Tenwek Hospital from January through December 2021. Participant perspectives of SIMPL were elicited through a survey and semi-structured interviews. Descriptive statistics were used to analyze survey data. Inductive qualitative content analysis of interview responses was performed by two independent researchers. RESULTS: Fourteen residents and six faculty (100% response rate) were included in the study and completed over 600 operative assessments. All respondents reported numerical evaluations and dictated feedback were useful. Respondents felt that SIMPL was easy to use, improved quality and frequency of feedback, helped refine surgical skills, and increased resident autonomy. Barriers to use included participants forgetting to complete evaluations, junior residents not submitting evaluations when minimally involved in cases, and technological challenges. Suggestions for improvement included expansion of SIMPL to surgical subspecialties and allowing senior residents to provide feedback to juniors. All respondents wanted to continue using SIMPL, and 90% recommended use at other programs. CONCLUSION: Residents and faculty at Tenwek Hospital believed SIMPL were a positive addition to their training program. There were a few barriers to use and suggestions for improvement specific to the training environment in Kenya, but this study demonstrates it is feasible to use SIMPL in settings outside the U.S. with the appropriate resources.
Assuntos
Cirurgia Geral , Internato e Residência , Humanos , Smartphone , Retroalimentação , Quênia , Competência Clínica , Educação de Pós-Graduação em Medicina/métodos , Hospitais , Cirurgia Geral/educaçãoRESUMO
Objectives: This study tests the null hypotheses that overall sentiment and gendered words in verbal feedback and resident operative autonomy relative to performance are similar for female and male residents. Background: Female and male surgical residents may experience training differently, affecting the quality of learning and graduated autonomy. Methods: A longitudinal, observational study using a Society for Improving Medical Professional Learning collaborative dataset describing resident and attending evaluations of resident operative performance and autonomy and recordings of verbal feedback from attendings from surgical procedures performed at 54 US general surgery residency training programs from 2016 to 2021. Overall sentiment, adjectives, and gendered words in verbal feedback were quantified by natural language processing. Resident operative autonomy and performance, as evaluated by attendings, were reported on 5-point ordinal scales. Performance-adjusted autonomy was calculated as autonomy minus performance. Results: The final dataset included objective assessments and dictated feedback for 2683 surgical procedures. Sentiment scores were higher for female residents (95 [interquartile range (IQR), 4-100] vs 86 [IQR 2-100]; P < 0.001). Gendered words were present in a greater proportion of dictations for female residents (29% vs 25%; P = 0.04) due to male attendings disproportionately using male-associated words in feedback for female residents (28% vs 23%; P = 0.01). Overall, attendings reported that male residents received greater performance-adjusted autonomy compared with female residents (P < 0.001). Conclusions: Sentiment and gendered words in verbal feedback and performance-adjusted operative autonomy differed for female and male general surgery residents. These findings suggest a need to ensure that trainees are given appropriate and equitable operative autonomy and feedback.
RESUMO
INTRODUCTION: Efforts to improve surgical resident well-being could be accelerated with an improved understanding of resident job demands and resources. In this study, we sought to obtain a clearer picture of surgery resident job demands by assessing how residents distribute their time both inside and outside of the hospital. Furthermore, we aimed to elucidate residents' perceptions about current duty hour regulations. METHODS: A cross-sectional survey was sent to 1098 surgical residents at 27 US programs. Responses regarding work hours, demographics, well-being (utilizing the physician well-being index), and perceptions of duty hours in relation to education and rest, were collected. Data were evaluated using descriptive statistics and content analysis. RESULTS: A total of 163 residents (14.8% response rate) were included in the study. Residents reported a median total patient care hours per week of 78.0 h. Trainees spent 12.5 h on other professional activities. Greater than 40% of residents were "at risk" for depression and suicide based on physician well-being index scores. Four major themes associated with education and rest were identified: 1) duty hour definitions and reporting mechanisms do not completely reflect the amount of work residents perform, 2) quality patient care and educational opportunities do not fit neatly within the duty hour framework, 3) resident perceptions of duty hours are impacted the educational environment, and 4) long work hours and lack of adequate rest negatively affect well-being. CONCLUSIONS: The breadth and depth of trainee job demands are not accurately captured by current duty hour reporting mechanisms, and residents do not believe that their current work hours allow for adequate rest or even completion of other clinical or academic tasks outside of the hospital. Many residents are unwell. Duty hour policies and resident well-being may be improved with a more holistic accounting of resident job demands and greater attention to the resources that residents have to offset those demands.
Assuntos
Cirurgia Geral , Internato e Residência , Humanos , Admissão e Escalonamento de Pessoal , Carga de Trabalho , Estudos Transversais , Qualidade da Assistência à Saúde , Cirurgia Geral/educação , Tolerância ao Trabalho ProgramadoRESUMO
OBJECTIVE: Assessing surgical trainee operative performance is time- and resource-intensive. To maximize the utility of each assessment, it is important to understand which assessment activities provide the most information about a trainee's performance. The objective of this study is to identify the procedures that best differentiate performance for each general surgery postgraduate year (PGY)-level, leading to recommendations for targeted assessment. DESIGN: The Society for Improving Medical Professional Learning (SIMPL) operative performance ratings were modeled using a multilevel Rasch model which identified the highest and lowest performing trainees for each PGY-level. For each procedure within each PGY-level, a procedural performance discrimination index was calculated by subtracting the proportion of "practice-ready" ratings of the lowest performing trainees from the proportion of "practice-ready" ratings of the highest performing trainees. Four-quadrant plots were created using the median procedure volume and median discrimination index for each PGY-level. All procedures within the upper right quadrant were considered "highly differentiating, high volume" procedures. SETTING: This study was conducted across 70 general surgical residency programs who are members of the SIMPL collaborative. PARTICIPANTS: A total of 54,790 operative performance evaluations of categorical general surgery trainees were collected between 2015 and 2021. Trainees who had at least 1 procedure in common were included. Procedures with less than 25 evaluations per training year were excluded. RESULTS: The total number of evaluations per procedure ranged from 25 to 2,131. Discrimination values were generated for 51 (PGY1), 54 (PGY2), 92 (PGY3), 105 (PGY4), and 103 (PGY5) procedures. Using the above criteria, a total of 12 (PGY1), 15 (PGY2), 22 (PGY3), 21 (PGY4), and 28 (PGY5) procedures were identified as highly differentiating, high volume procedures. CONCLUSIONS: Our study draws on national data to identify procedures which are most useful in differentiating trainee operative performance at each PGY-level. This list of procedures can be used to guide targeted assessment and improve assessment efficiency.
Assuntos
Cirurgia Geral , Internato e Residência , Humanos , Educação de Pós-Graduação em Medicina/métodos , Competência Clínica , Avaliação Educacional/métodos , Cirurgia Geral/educaçãoRESUMO
OBJECTIVE: Feedback is critical for learning, however, gender differences exist in the quality of feedback that trainees receive. For example, narrative feedback on surgical trainees' end-of-block rotations differs based on trainee-faculty gender dyads, with female faculty giving higher quality feedback and male trainees receiving higher quality feedback. Though this represents evidence of gender bias in global evaluations, there is limited understanding of how much bias might be present in operative workplace-based assessments (WBAs). In this study, we explore the quality of narrative feedback among trainee-faculty gender dyads in an operative WBA. DESIGN: A previously validated natural language processing model was used to examine instances of narrative feedback and assign a probability of being characterized as high quality feedback (defined as feedback which was relevant as well as corrective and/or specific). A linear mixed model was performed, with probability of high quality feedback as the outcome, and resident gender, faculty gender, PGY, case complexity, autonomy rating, and operative performance rating as explanatory variables. PARTICIPANTS: Analyses included 67,434 SIMPL operative performance evaluations (2,319 general surgery residents, 70 institutions) collected from September 2015 through September 2021. RESULTS: Of 36.3% evaluations included narrative feedback. Male faculty were more likely to provide narrative feedback compared to female faculty. Mean probabilities of receiving high quality feedback ranged from 81.6 (female faculty-male resident) to 84.7 (male faculty-female resident). Model-based results demonstrated that female residents were more likely to receive high quality feedback (p < 0.01), however, there was no significant difference in probability of high quality narrative feedback based on faculty-resident gender dyad (pâ¯=â¯0.77). CONCLUSIONS: Our study revealed resident gender differences in the probability of receiving high-quality narrative feedback following a general surgery operation. However, we found no significant differences based on faculty-resident gender dyad. Male faculty were more likely to provide narrative feedback compared to their female colleagues. Further research using general surgery resident-specific feedback quality models may be warranted.
Assuntos
Cirurgia Geral , Internato e Residência , Humanos , Masculino , Feminino , Retroalimentação , Competência Clínica , Sexismo , Educação de Pós-Graduação em Medicina/métodos , Cirurgia Geral/educaçãoRESUMO
OBJECTIVE: As medical education systems increasingly move toward competency-based training, it is important to understand the tools available to assess competency and how these tools are utilized. The Society for Improving Medical Professional Learning (SIMPL) offers a smart phone-based assessment system that supports workplace-based assessment of residents' and fellows' operative autonomy, performance, and case complexity. The purpose of this study was to characterize implementation of the SIMPL app within vascular surgery integrated residency (0+5) and fellowship (5+2) training programs. METHODS: SIMPL operative ratings recorded between 2018 and 2022 were collected from all participating vascular surgery training institutions (n = 9 institutions with 5+2 and 0+5 programs; n = 4 institutions with 5+2 program only). The characteristics of programs, trainees, faculty, and SIMPL operative assessments were evaluated using descriptive statistics. RESULTS: Operative assessments were completed for 2457 cases by 85 attendings and 86 trainees, totaling 4615 unique operative assessment ratings. Attendings included dictated feedback in 52% of assessments. Senior-level residents received more assessments than junior-level residents (postgraduate year [PGY]1-3, n = 439; PGY4-5, n = 551). Performance ratings demonstrated increases from junior to senior trainees for both resident and fellow cohorts with "performance-ready" or "exceptional performance" ratings increasing by nearly two-fold for PGY1 to PGY5 residents (28.1% vs 40.6%), and from first- to second-year fellows (PGY6, 46.7%; PGY7, 60.3%). Similar gains in autonomy were demonstrated as trainees progressed through training. Senior residents were more frequently granted autonomy with "supervision only" than junior residents (PGY1, 8.7%; PGY5, 21.6%). "Supervision only" autonomy ratings were granted to 21.8% of graduating fellows. Assessment data included a greater proportion of complex cases for senior compared with junior fellows (PGY6, 20.9% vs PGY7, 26.5%). Program Directors felt that faculty and trainee buy-in were the main barriers to implementation of the SIMPL assessment app. CONCLUSIONS: This is the first description of the SIMPL app as an operative assessment tool within vascular surgery that has been successfully implemented in both residency and fellowship programs. The assessment data demonstrates expected progressive gains in trainees' autonomy and performance, as well as increasing case complexity, across PGY years. Given the selection of SIMPL as the assessment platform for required American Board of Surgery and Vascular Surgery Board Entrustable Professional Activities assessments, understanding facilitators and barriers to implementation of workplace-based assessments using this app is imperative, particularly as we move toward competency-based medical education.
Assuntos
Cirurgia Geral , Internato e Residência , Humanos , Bolsas de Estudo , Educação de Pós-Graduação em Medicina , Competência Clínica , Procedimentos Cirúrgicos Vasculares , Local de Trabalho , Cirurgia Geral/educaçãoRESUMO
INTRODUCTION: The competency-based medical education (CBME) movement continues to gain momentum in postgraduate physician training, resulting in increasing interest among surgical training programmes on how to implement it effectively. Entrustable professional activities (EPAs) were introduced to connect competencies (characteristics/abilities of learners) and the professional activities to be entrusted to them on qualification/graduation. Although reviews related to the field of general surgery have been published on specific aspects of CBME, for example, workplace-based assessment, there is a paucity of published guidance available for surgeon-educators to plan and implement CBME using an EPA framework. This scoping review aims to provide an overview of the current use of EPAs in general surgery. METHODS AND ANALYSIS: This scoping review will be based on the original five-stage approach described by Arksey and O'Malley with the addition of protocol-specific items from the Preferred Reporting Items for Systematic Reviews and Meta-Analysis Protocols extension for scoping reviews. The primary research question focuses on describing how EPAs have been used in general surgery and where potential gaps remain that warrant further study. The review will include peer-reviewed journal publications and an extensive review of grey literature sources. The following databases will be searched for published literature from 1 January 2005 to 31 December 2022: PubMed; CINAHL, Africa-Wide Information, PsycInfo and ERIC (via EBSCOhost); Scopus; and SciELO (via Web of Science). Studies that describe the use of EPAs for curriculum design, teaching and/or assessment of competence in postgraduate general surgery training will be eligible for inclusion. ETHICS AND DISSEMINATION: The institutional ethics board of the University of Cape Town has granted a waiver of formal approval requirement. The dissemination strategy includes publication of results in peer-reviewed journals, presentation at international conferences and presentation to relevant stakeholders as deemed appropriate.