AI Versus MD: Evaluating the surgical decision-making accuracy of ChatGPT-4.

BACKGROUND: ChatGPT-4 is a large language model with possible applications to surgery education The aim of this study was to investigate the accuracy of ChatGPT-4's surgical decision-making compared with general surgery residents and attending surgeons. METHODS: Five clinical scenarios were created from actual patient data based on common general surgery diagnoses. Scripts were developed to sequentially provide clinical information and ask decision-making questions. Responses to the prompts were scored based on a standardized rubric for a total of 50 points. Each clinical scenario was run through Chat GPT-4 and sent electronically to all general surgery residents and attendings at a single institution. Scores were compared using Wilcoxon rank sum tests. RESULTS: On average, ChatGPT-4 scored 39.6 points (79.2%, standard deviation ± 0.89 points). A total of five junior residents, 12 senior residents, and five attendings completed the clinical scenarios (resident response rate = 15.9%; attending response rate = 13.8%). On average, the junior residents scored a total of 33.4 (66.8%, standard deviation ± 3.29), senior residents 38.0 (76.0%, standard deviation ± 4.75), and attendings 38.8 (77.6%, standard deviation ± 5.45). ChatGPT-4 scored significantly better than junior residents (P = .009) but was not significantly different from senior residents or attendings. ChatGPT-4 was significantly better than junior residents at identifying the correct operation to perform (P = .0182) and recommending additional workup for postoperative complications (P = .012). CONCLUSION: ChatGPT-4 performed superior to junior residents and equivalent to senior residents and attendings when faced with surgical patient scenarios. Large language models, such as ChatGPT, may have the potential to be an educational resource for junior residents to develop surgical decision-making skills.

National Profile of the ACGME Milestones 1.0 and 2.0 within General Surgery: A Seven-Year National Study from 2014 to 2021.

PURPOSE: The Accreditation Council for Graduate Medical Education (ACGME) introduced General Surgery Milestones 1.0 in 2014 and Milestones 2.0 in 2020 as steps toward competency-based training. Analysis will inform residency programs on curriculum development, assessment, feedback, and faculty development. This study describes the distributions and trends for Milestones 1.0 and 2.0 ratings and proportion of residents not achieving the level 4.0 graduation target. METHODS: A deidentified dataset of milestone ratings for all ACGME-accredited General Surgery residency programs in the United States was used. Medians and interquartile ranges (IQR) were reported for milestone ratings at each PGY level. Percentages of PGY-5s receiving final year ratings of less than 4.0 were calculated. Wilcoxon rank sum tests were used to compare 1.0 and 2.0 median ratings. Kruskal-Wallis and Bonferroni post-hoc tests were used to compare median ratings across time periods and PGY levels. Chi-squared tests were used to compare the proportion of level 4.0 nonachievement under both systems. RESULTS: Milestones 1.0 data consisted of 13,866 residents and Milestones 2.0 data consisted of 7,633 residents. For 1.0 and 2.0, all competency domain median ratings were higher for subsequent years of training. Milestones 2.0 had significantly higher median ratings at all PGY levels for all competency domains except Medical Knowledge. Percentages of PGY-5 residents not achieving the graduation target in Milestones 1.0 ranged from 27% to 42% and in 2.0 from 5% to 13%. For Milestones 1.0, all subcompetencies showed an increased number of residents achieving the graduation target from 2014 to 2019. CONCLUSIONS: This study of General Surgery Milestones 1.0 and 2.0 data uncovered significant increases in average ratings and significantly fewer residents not achieving the graduation target under the 2.0 system. We hypothesize that these findings may be related more to rating bias given the change in rating scales, rather than a true increase in resident ability.

Practical Guide to Experimental and Quasi-Experimental Research in Surgical Education.

This Guide to Statistics and Methods describes the methods and pitfalls of experimental and quasi-experimental study designs in surgical education.

Virtual training for small group facilitators.

BACKGROUND: In addition to providing patient care, interprofessional health care teams work collaboratively on a variety of projects. These projects often benefit from using facilitated small group project discussion sessions, such as the Harvard Macy Institute's (HMI) Step Back Process (SBP). Although having a trained facilitator is an important component of the SBP, only a limited number of health care professionals can attend HMI courses in person or virtually, limiting its impact. APPROACH: We developed three video-based education (VBE) modules to deliver facilitator training on the SBP, informed by principles of Mayer's cognitive theory of multimedia learning. For module development, we used a five-step approach. We evaluated effectiveness of the modules as a self-directed method to enhance SBP facilitator training. An initial survey collected demographic data and module feedback, a follow-up survey collected feedback on the modules' impact on facilitation and interviews focused on the participants' overall experience. EVALUATION: Survey results indicated that the modules were positively received and helped to improve participant confidence in facilitating. We identified four themes from the interviews: challenges of facilitating, value of group feedback, value of modules for experienced facilitators and the modules as part of a multi-modal approach to train new facilitators. IMPLICATIONS: This innovation provides insight on delivering facilitator training on the SBP using VBE. Health professions educators developing online facilitator training could adapt our development process and modify implementation guided by our results. Future work should evaluate the best methods to integrate VBE modules into a longitudinal virtual community and assess facilitation techniques.

Trend of minimally invasive and open surgery experience of general surgery residents: Accreditation Council for Graduate Medical Education general surgery case log in Thailand.

Purpose: Minimally invasive surgery (MIS) offers patients several benefits, such as smaller incisions, and fast recovery times. General surgery residents should be trained in both open and MIS. We aimed to examine the trends of minimally invasive and open procedures performed by general surgery residents in Thailand. Methods: A retrospective review of the Royal College of Surgeons of Thailand and Accreditation Council for Graduate Medical Education general surgery case logs from 2007 to 2018 was performed for common open and laparoscopic general surgery operations. The data were grouped by three time periods, which were 2007-2010, 2011-2014, and 2015-2018, and analyzed to explore changes in the operative trends. Results: For Thai residents, the mean number of laparoscopic operations per person per year increased from 5.97 to 9.36 (56.78% increase) and open increased from 20.02 to 27.16 (35.67% increase). There was a significant increase in the average number of minimally invasive procedures performed among cholecystectomy (5.83, 6.57, 8.10; p < 0.001) and inguinal hernia repair (0.33, 0.35, 0.66; p < 0.001). Compared to general surgery residents in the United States, Thai residents had more experience with open appendectomy, but significantly less experience with all other operations/procedures. Conclusion: The number of open and minimally invasive procedures performed or assisted by Thai general surgery residents has slowly increased, but generally lags behind residents in the United States. The Thai education program must be updated to improve residents' technical skills in open and laparoscopic surgery to remain competitive with their global partners.

Preclinical Immersion Experiences Improve Medical Student Perceptions of Surgery.

INTRODUCTION: Medical students historically receive little to no preclinical exposure to surgery and surgical subspecialties. As a result, by the time they reach their clinical clerkship time, students often have already found interest in other specialties. The goal of this study is to utilize the knowledge to action (KTA) implementation framework to design and refine a clinical immersion experience during the second year of medical school. METHODS: A total of 94 second-year Harvard Medical School students underwent the surgical immersion experience between 2019 and 2022 (the program was postponed in 2020 due to COVID). The development and refinement of the curriculum were nicely modeled by the KTA implementation framework. We identified a gap in medical student preclinical education, adapted a curriculum for preclinical medical students at Massachusetts General Hospital , selected the curriculum components to provide a high-level overview of surgery, monitored the student experience, and evaluated outcomes using the student surveys. Based on the survey results, inductive thematic analysis was utilized to identify prominent positive and negative themes. The feedback was then used to tailor subsequent iterations of the immersion experience. RESULTS: Eighty-eight medical students completed the survey (RR = 93.6%), and 85% rated the immersion experience as "excellent", 11% "very good", 4% "good", and 0% "fair" or "poor". There was no significant difference in ratings between sessions. Several key themes were identified, including changed perceptions, diversity of surgical fields, teamwork, surgery clerkship preparedness, and the need for more preclinical exposure. CONCLUSIONS: Preclinical medical students gave overwhelmingly positive reviews of the surgical immersion experience. A half-day intervention is sufficient to begin changing students' views toward surgery, disproving stereotypes, and even inspiring some to consider a surgical field themselves. In addition, the KTA implementation framework is a useful model for the development and refinement of medical education curricula.

Incorporating Well-Being into Mentorship Meetings: A Case Demonstration at Massachusetts General Hospital Department of Surgery a Harvard Medical School Affiliate.

Surgeons have been under great pressure during the COVID pandemic. Their careers are filled with fast paced decisions, life and death situations, and long hours at work. The COVID pandemic created more tasks and even new responsibilities at times, but when the operating rooms were closed down, there was less work. The COVID experience invited the opportunity to rethink mentoring in the surgery department at the Massachusetts General Hospital. The leadership experimented with a new style of mentoring which involved a team approach. In addition, they tried something else that was new: adding a lifestyle medicine expert and wellness coach to the mentoring team. The program was tested on 13 early stage surgeons who found the experience to be beneficial, and they commented that they wished they had it even earlier in their careers. Including a non-surgeon who was a lifestyle medicine physician and wellness coach added an element of whole person health that was acceptable to the surgeons and even embraced as the majority of them elected to follow up with one on one coaching after the mentoring meeting. This team mentoring program with senior surgeons and a lifestyle medicine expert is one that can be explored by other departments and other hospitals given its success at the department of surgery at Massachusetts General Hospital.

Multidisciplinary perceptions on robotic surgical training: the robot is a stimulus for surgical education change.

BACKGROUND: It is unclear how to best establish successful robotic training programs or if subspecialty robotic program principles can be adapted for general surgery practice. The objective of this study is to understand the perspectives of high-volume robotic surgical educators on best practices in robotic surgery training and to provide recommendations transferable across surgical disciplines. METHODS: This multi-institutional qualitative analysis involved semi-structured interviews with high-volume robotic educators from academic general surgery (AGS), community general surgery (CGS), urology (URO), and gynecology (GYN). Purposeful sampling and snowballing ensured high-volume status and geographically balanced representation across four strata. Interviews were transcribed, deidentified, and independently, inductively coded. A codebook was developed and refined using constant comparative method until interrater reliability kappa reached 0.95. A qualitative thematic, framework analysis was completed. RESULTS: Thirty-four interviews were completed: AGS (n = 9), CGS (n = 8), URO (n = 9), and GYN (n = 8) resulting in 40 codes and four themes. Theme 1: intangibles of culture, resident engagement, and faculty and administrative buy-in are as important as tangibles of robot and simulator access, online modules, and case volumes. Theme 2: robotic OR integration stresses the trainee-autonomy versus patient-safety balance. Theme 3: trainees acquire robotic skills along individual learning curves; benchmark assessments track progress. Theme 4: AGS can learn from URO and GYN through multidisciplinary collaboration but must balance pre-existing training program use with context-specific curricular needs. CONCLUSIONS: Robotic surgical experts emphasize the importance of universal training paradigms, such as a strong educational culture that balances autonomy and patient safety, collaboration between disciplines, and routine assessments for continuous growth. Often, introduction and acceptance of the robot serves as a stimulus to discuss broader surgical education change.

The Big Sibling program: Impact of a medical student-resident mentorship program during the surgery clerkship.

BACKGROUND: We created a Big Sibling mentorship program for medical students and studied the program effects. METHODS: Between July 2019 to December 2020, students completing their surgery clerkship were paired with a Big Sibling surgical research resident. Participation in and perceptions of the program were assessed by survey. RESULTS: 81 medical students and 25 residents participated with a 79% and 95% survey response rate, respectively. The most valuable topics discussed included ward skills, personal development and career advising. Students who interacted >2 times with their Big Sibling were more likely to perceive the operating room as a positive learning environment, view attendings as role models, and receive mentoring and feedback from residents and attendings (p = 0.03, 0.02, 0.01 respectively). 78% of residents thought the program was a positive experience and no residents found it burdensome. CONCLUSION: The Big Siblings program enhances the surgery clerkship learning environment. Students who engaged with their Big Sibling had a more positive view of the clerkship and the mentorship provided by residents and attendings.

Inpatient Satisfaction With Surgical Resident Care After Elective General and Oncologic Surgery.

OBJECTIVE: To investigate inpatient satisfaction with surgical resident care. BACKGROUND: Surgical trainees are often the primary providers of care to surgical inpatients, yet patient satisfaction with surgical resident care is not well characterized or routinely assessed. METHODS: English-speaking, general surgery inpatients recovering from elective gastrointestinal and oncologic surgery were invited to complete a survey addressing their satisfaction with surgical resident care. Patients positively identified photos of surgical senior residents and interns before completing a modified version of the Consumer Assessment of Healthcare Providers and Systems Surgical Care Survey (S-CAHPS). Adapted S-CAHPS items were scored using the "top-box" method. RESULTS: Ninety percent of recruited patients agreed to participate (324/359, mean age=62.2, 50.3% male). Patients were able to correctly identify their seniors and interns 85% and 83% of the time, respectively ( P =0.14). On a 10-point scale, seniors had a mean rating of 9.23±1.27 and interns had a mean rating of 9.01±1.49 ( P =0.14). Ninety-nine percent of patients agreed it was important to help in the education of future surgeons. CONCLUSIONS: Surgical inpatients were able to recognize their resident physicians with high frequency and rated resident care highly overall, suggesting that they may serve as a willing source of feedback regarding residents' development of core competencies such as interpersonal skills, communication, professionalism, and patient care. Future work should investigate how to best incorporate patient evaluation of surgical resident care routinely into trainee assessment to support resident development.

Neurosurgeons' Perspectives on Vascular Entrustable Professional Activities.

This article discusses subspecialty Canadian neurosurgeons' perceptions of entrustable professional activities (EPAs) assessments and variabilities prior to the implementation of the Competence by Design (CBD) system in Canada. Vascular neurosurgeons were asked to reflect on how they would evaluate and give feedback to neurosurgery residents concerning the EPA "Performing surgery for patients with an intracranial aneurysm." Interviews were transcribed and analyzed using a deductive approach. Themes were derived from these interviews and reflected on the subjectivity and biases present in the EPA assessment forms. Indeed, faculty may require more training in the transitioning to a CBD evaluation system.

Effective Senior Surgical Residents as Defined by Their Peers: A Qualitative Content Analysis of Nontechnical Skills Development.

OBJECTIVE: This study aims to define an effective senior resident and understand the process of leadership and nontechnical skill development in the transition from junior to senior surgery resident. SUMMARY BACKGROUND: General surgery residents are responsible for patient care, technically demanding operations, and diverse care team management. However, leadership skill development for the transition from junior to senior resident roles is often overlooked. METHODS: We conducted 15 semi-structured focus groups with surgery residents from an urban, academic institution. Focus group transcripts were inductively coded. Using content analysis and constant comparative methodology, primary codes were refined into categories and organized into higher-level themes. RESULTS: Thirty-three general surgery residents completed fifteen focus groups. Six themes were identified. Three themes describe the process of becoming an effective senior resident: how to define a senior resident's scope of practice, the transition process, and the importance of personal investment. Three themes were identified regarding effective seniors: ideal traits, teachable skills, and the team and patient impact. CONCLUSIONS: Surgery residents define an effective senior resident as the team member with the highest level of experience who manages the big picture of patient care. The transition is improved by personal engagement and acknowledgement of the transition. Ideal traits of effective seniors, including emotional intelligence and inherent personality traits, allow a resident to more naturally assume this role; however, teachable skills, such as communication, expectation setting and competence, can be taught to improve one's effectiveness. The actions of a senior resident impact the team and patient care, underscoring the importance of understanding this role.

Impact of Rater Training on Residents Technical Skill Assessments: A Randomized Trial.

OBJECTIVE: The ACS/APDS Resident Skills Curriculum's Objective Structured Assessment of Technical Skills (OSATS) consists of task-specific checklists and a global rating scale (GRS) completed by raters. Prior work demonstrated a need for rater training. This study evaluates the impact of a rater-training curriculum on scoring discrimination, consistency, and validity for handsewn bowel anastomosis (HBA) and vascular anastomosis (VA). DESIGN/ METHODS: A rater training video model was developed, which included a GRS orientation and anchoring performances representing the range of potential scores. Faculty raters were randomized to rater training or no rater training and were asked to score videos of resident HBA/VA. Consensus scores were assigned to each video using a modified Delphi process (Gold Score). Trained and untrained scores were analyzed for discrimination and score spread and compared to the Gold Score for relative agreement. RESULTS: Eight general and eight vascular surgery faculty were randomized to score 24 HBA/VA videos. Rater training increased rater discrimination and decreased rating scale shrinkage for both VA (mean trained score: 2.83, variance 1.88; mean untrained score: 3.1, variance 1.14, p = 0.007) and HBA (mean trained score: 3.52, variance 1.44; mean untrained score: 3.42, variance 0.96, p = 0.033). On validity analyses, a comparison between each rater group vs Gold Score revealed a moderate training impact for VA, trained κ=0.65 vs untrained κ=0.57 and no impact for HBA, R1 κ = 0.71 vs R2 κ = 0.73. CONCLUSION: A rater-training curriculum improved raters' ability to differentiate performance levels and use a wider range of the scoring scale. However, despite rater training, there was persistent disagreement between faculty GRS scores with no groups reaching the agreement threshold for formative assessment. If technical skill exams are incorporated into high stakes assessments, consensus ratings via a standard setting process are likely a more valid option than individual faculty ratings.

Gaps in Practice Management Skills After Training: A Qualitative Needs Assessment of Early Career Surgeons.

PURPOSE: Shifts in American healthcare delivery mechanisms pose significant hurdles to new physicians. Surgeons are particularly susceptible to these changes, but surgical residency educational efforts primarily focus on technical and clinical training to the exclusion of business and management practices. This study conducted a needs assessment of perceived gaps in practice management skills among early career surgeons to guide future training curricula. METHODS: This study was an exploratory qualitative study following the Consolidated Criteria for Reporting Qualitative Research. Purposive sampling was used to identify early career (<5 years following fellowship completion) surgeons across the United States. A semi-structured interview guide was created from interviews with surgical administrators and physician administrative curricula. Transcripts were de-identified and analyzed using a constructivist grounded theory approach. RESULTS: Ten surgeons from 6 specialties and 6 institutions were interviewed along with 3 surgeon administrators. Three major domains of need were identified: (1) fundamentals of procedural coding, clinical billing, & compliance, (2) finding/building a practice, and (3) navigating organizational challenges. First, surgeons thought trainees would benefit from a better understanding of reimbursement schema and the basics of health policy. They also thought that more exposure to malpractice litigation, especially for handling case review or expert witness requests, would be helpful for discerning how to handle such issues early in their career. In addition, early career surgeons expressed a desire to have dedicated mentorship time, a primer on evaluating job offers with simulated contract negotiation, and guidance regarding administrative roles. Finally, surgeons requested training in change management techniques, care pathway construction, and the basics of staffing decisions. CONCLUSIONS: There are significant practice management gaps in surgical training which may be amenable to targeted educational efforts during a residency or fellowship program. Future research will test the generalizability of these findings as well as build curricula that adequately meet these needs.

Finding my people: Effects of student identity and vulnerability to Stereotype Threat on sense of belonging in surgery.

BACKGROUND: Diversity in surgery lags behind the medical student population. We documented first-year medical students' vulnerability to stereotype threat (VST) and its impact on a sense of belonging in surgery. METHODS: All first-year medical students at a single academic institution were surveyed. Demographics, VST, anticipated clerkship experience, and sense of belonging were assessed. RESULTS: 44% of students were vulnerable to ST in upcoming clerkships, with the majority worried about surgical clerkships. More student from races/ethnicities underrepresented in medicine (URM; 74%) and sexual minorities (62%) were vulnerable than white (30%) and heterosexual (38%) students respectively (p = 0.001 and p = 0.017). Knowing a surgeon with a shared identity would enhance belonging for most students (84%). VST was higher for those who do not anticipate working with (p < 0.001) or do not know a surgeon (p = 0.0001) who shares their identity. CONCLUSION: VST significantly influences a student's sense of belonging in surgery. More research is needed to promote inclusivity in surgery.

A multisite study of medical student perspectives on the core surgical clerkship.

BACKGROUND: The surgical clerkship is the primary surgical learning experience for medical students. This study aims to understand student perspectives on the surgery clerkship both before and after the core surgical rotation. METHODS: Medical students at 4 academic hospitals completed pre and postclerkship surveys that included open-ended questions regarding (1) student learning goals and concerns and (2) how surgical clerkship learning could be enhanced. Thematic analysis was performed, and interrater reliability was calculated. RESULTS: Ninety-one percent of students completed both a pre and postclerkship survey (n =162 of 179), generating 320 preclerkship and 270 postclerkship responses. Mean kappa coefficients were 0.83 and 0.82 for pre and postclerkship primary themes, respectively. Thematic analysis identified 5 broad themes: (1) core learning expectations, (2) understanding surgical careers, culture, and work, (3) inhabiting the role of a surgeon, (4) inclusion in the surgical team, and (5) the unique role of the medical student on clinical clerkships. Based on these themes, we propose a learner-centered model of a successful surgical clerkship that satisfies discrete student learning and goals and career objectives while ameliorating the challenges of high-stakes clinical surgical environments such as the operating room. CONCLUSION: Understanding student perspectives on the surgery clerkship, including preclerkship motivations and concerns and postclerkship reflections on surgical learning, revealed potential targets of intervention to improve the surgery clerkship. Future investigation may elucidate whether the proposed model of the elements of a successful surgery clerkship learning facilitates improvement of the surgical learning environment and enhanced surgical learning.

Eyes of the Beholders: First-year medical students' perceptions of surgeons and the field of surgery.

BACKGROUND: Medical students have negative perceptions of surgery prior to their clerkships. To explore possible explanations, we examined the association between these perceptions, individual identity and vulnerability to stereotype threat (ST). METHODS: All first-year medical students at a single school received an electronic survey which assessed identity groups, vulnerability to ST and perceptions of surgeons/surgery. Multi-method analyses examined these associations. RESULTS: Women held more negative than positive views about the field of surgery (p = 0.007) but not surgeons. Students vulnerable to ST had negative views about both surgeons (p < 0.0001) and surgery (p = 0.007). They were also less interested in pursuing a surgical career compared to non-vulnerable students (56% vs. 80% p = 0.03). CONCLUSION: For some students, negative views of surgeons and surgery appear to be associated with individual identity and ST. Future research should aim to confirm these findings and identify strategies to develop positive perceptions for these populations.

Challenges of virtual interviewing for surgical fellowships: a qualitative analysis of applicant experiences.

BACKGROUND: The COVID-19 pandemic forced surgical fellowship programs to transition from in-person to remote applicant interviews; the virtual interviewing format presented new and unique challenges. We sought to understand applicants' perceived challenges to virtual interviewing for a surgical fellowship program. METHOD: A grounded theory-based qualitative study was performed utilizing semi-structured interviews with fellowship applicants from the 2020 fellowship match. All Fellowship Council-registered applicants were eligible. We purposefully sampled participants to balance across gender, specialty-choice, and academic versus community-program affiliation. Interviews were inductively analyzed by two researchers for prominent themes. RESULTS: Fifteen interviews were conducted. Participants were 60% male (n = 9), with 33% (n = 5) from non-academic institutions. They applied for the following fellowships: Advanced Gastrointestinal/Minimal Invasive (55%), Bariatric (30%), Hepatopancreatobiliary (10%) and Surgical Oncology (5%). Four main themes emerged to describe virtual interview process challenges: (1) perceived data deficiency, (2) superficial personal connections, (3) magnification of non-professionalism, and (4) logistical frustrations. Applicants recommend program directors provide more information about the fellowship prior to interview day and offer informal independent interactions with current and previous fellows. CONCLUSIONS: According to fellowship applicants, virtual interviews resulted in a lack of information for rank-list decision making ultimately requiring them to rely on other information avenues to base their decisions. These applicants have offered advice to fellowship program directors and future applicants to better optimize this process.