Practical Guide to Assessment Tool Development for Surgical Education Research.

This Guide to Statistics and Methods describes the process of validation and gathering validity evidence for assessment tool development for surgical education research.

Sex influences how attendings evaluate surgical residents in the operating room.

BACKGROUND: The aim of this study was to evaluate the influence of sex on facultys' perception of resident autonomy and performance. METHODS: Autonomy/performance/complexity evaluations performed by faculty of categorical general surgery residents (2015-2021) were analyzed. Comparisons of scores by faculty and resident sex were performed. RESULTS: A total of 10967 paper/electronic evaluations were collected. Female attendings rated female residents significantly lower in autonomy when compared to males (2.75 vs 2.91, p â€‹= â€‹0.0037). There was no significant difference in autonomy ratings for male versus female residents when evaluated by a male attending (2.93 vs 2.96, p â€‹= â€‹0.054) but male attendings did rate female residents significantly lower in autonomy at the highest complexities (2.37 vs 2.50, p â€‹= â€‹0.012). CONCLUSION: The data suggests a unique interaction between attending and resident sex. A periodic evaluation of evaluations within one's program may provide invaluable implicit bias insight and should be considered.

Surgery Clerkship Directors' Perceptions of the COVID-19 Pandemic's Impact on Medical Student Education.

BACKGROUND: This study assessed the national impact of the COVID-19 pandemic on the education of medical students assigned to surgery clerkship rotations, as reported by surgery clerkship directors(CDs). STUDY DESIGN: In the spring of 2020 and 2021, the authors surveyed 164 CDs from 144 LCME-accredited US medical schools regarding their views of the pandemic's impact on the surgery clerkship curriculum, students' experiences, outcomes, and institutional responses. RESULTS: Overall survey response rates, calculated as no. respondents/no. surveyed were 44.5%(73/164) and 50.6%(83/164) for the spring 2020 and 2021 surveys, respectively. Nearly all CDs(>95%) pivoted to virtual platforms and solutions. Most returned to some form of in-person learning by winter 2020, and pre-pandemic status by spring 2021(46%, 38/83). Students' progression to the next year was delayed by 12%(9/73), and preparation was negatively impacted by 45%(37/83). Despite these data, CDs perceived students' interest in surgical careers was not significantly affected(89% vs. 77.0%, p=0.09). Over the one-year study, the proportion of CDs reporting a severe negative impact on the curriculum dropped significantly(p<0.0001) for most parameters assessed except summative evaluations(40.3% vs. 45.7%,p=0.53). CDs(n=83) also noted the pandemic's positive impact with respect to virtual patient encounters(21.7%), didactics(16.9%), student test performance(16.9%), continuous personal learning(14.5%), engagement in the clerkship(9.6%) and student interest in surgery as a career(7.2%). CONCLUSION: During the pandemic, the severe negative impact on student educational programs lessened and novel virtual curricular solutions emerged. Student interest in surgery as a career was sustained. Measures of student competency and effectiveness of new curriculum, including telehealth, remain areas for future investigation.

One-Year Follow-Up of COVID-19 Impact on Surgical Education: Clinical Training Restored but Surgical Trainee Emotional Well-Being Still at Risk.

BACKGROUND: A previous survey documented the severe disruption of the coronavirus disease 2019 pandemic on surgical education and trainee well-being during the initial surge and systemic lockdowns. Herein, we report the results of a follow-up survey inclusive of the 2020 to 2021 academic year. STUDY DESIGN: A survey was distributed to education leaders across all surgical specialties in summer 2021. We compared the proportion of participants reporting severe disruption in key areas with those of the spring 2020 survey. Aggregated differences by year were assessed using chi-square analysis. RESULTS: In 2021, severe disruption of education programs was reported by 14% compared with 32% in 2020 (p < 0.0001). Severe reductions in nonemergency surgery were reported by 38% compared with 87% of respondents in 2020. Severe disruption of expected progression of surgical trainee autonomy by rank also significantly decreased to 5% to 8% in 2021 from 15% to 23% in 2020 among respondent programs (p < 0.001). In 2021 clinical remediation was reported for postgraduate year 1 to 2 and postgraduate year 3 to 4, typically through revised rotations (19% and 26%) and additional use of simulation (20% and 19%) maintaining trainee promotion and job placement. In 2021, surgical trainees' physical safety and health were reported as less severely impacted compared with 2020; however, negative effects of isolation (77%), burnout (75%), and the severe impact on emotional well-being (17%) were prevalent. CONCLUSIONS: One year after the initial coronavirus disease 2019 outbreak, clinical training and surgical trainee health were less negatively impacted. Disruption of emotional well-being remained high. Future needs include better objective measures of clinical competence beyond case numbers and the implementation of novel programs to promote surgical trainee health and well-being.

American College of Surgeons Objective Assessment of Skills in Surgery (ACS OASIS): A Formative Assessment of Junior Residents' Technical Skills.

OBJECTIVE: The objective assessment of technical skills of junior residents is essential in implementing competency-based training and providing specific feedback regarding areas for improvement. An innovative assessment that can be easily implemented by training programs nationwide has been developed by expert surgeon educators under the aegis of the American College of Surgeons (ACS) Division of Education. This assessment, ACS Objective Assessment of Skills in Surgery (ACS OASIS) uses eight stations to address technical skills important for junior residents within the domains of laparoscopic appendectomy, excision of lipoma, central line placement, laparoscopic cholecystectomy, trocar placement, exploratory laparotomy, repair of enterotomy, and tube thoracostomy. The purpose of this study was to implement ACS OASIS at a number of sites to study its psychometric rigor. DESIGN: The ACS OASIS was pre-piloted at two programs to establish feasibility and to gather information regarding implementation. Each skills station was 12 minutes long, and the faculty completed a checklist with 5 to 15 items, and a global assessment scale. The study was then repeated at three pilot sites and included 29 junior residents who were assessed by a total of 44 faculty. Psychometric data for the stations and checklists were collected and analyzed. SETTING: The pre-pilot sites were Geisinger and University of Tennessee Knoxville.Data were gathered from pilot sites that included Wellspan Health, Duke University, and University of California Los Angeles. RESULTS: The mean checklist score for all learners was 76% (IQR of 66%-85%). The average global rating was 3.36 on a 5-point scale with a standard deviation of 0.56. The overall cut score derived using the borderline group method was at 68% with 34% of performances requiring remediation. Using this criterion, the average number of stations that were completed by each learner without need for remediation was five.The station discrimination index ranged from 0.27 to 0.65 (all above the threshold of 0.25), demonstrating solid psychometric characteristics at the station level. The internal-consistency reliability was 0.76 with SEM of 5.8%. The inter-rater reliability (intraclass correlation) was high at 0.73 with general agreement of 79% between the two raters. The station discrimination was at 0.45 (range of 0.27 to 0.65) indicating a high level of differentiation between high and low performers. Using the generalizability theory, the G-coefficient reliability was at 0.72 with the reliability projection flattening after 8 stations. Overall, 75% to 82% the faculty and learners rated ACS OASIS as realistic and beneficial. CONCLUSIONS: ACS OASIS is a psychometrically sound technical skills assessment tool that can provide useful information for feedback to junior residents and support efforts to remediate gaps in performance.

Impact of the COVID-19 pandemic on surgical trainee education and well-being spring 2020-winter 2020: A path forward.

BACKGROUND: The time course and longitudinal impact of the COVID -19 pandemic on surgical education(SE) and learner well-being (LWB)is unknown. MATERIAL AND METHODS: Check-in surveys were distributed to Surgery Program Directors and Department Chairs, including general surgery and surgical specialties, in the summer and winter of 2020 and compared to a survey from spring 2020. Statistical associations for items with self-reported ACGME Stage and the survey period were assessed using categorical analysis. RESULTS: Stage 3 institutions were reported in spring (30%), summer (4%) [p < 0.0001] and increased in the winter (18%). Severe disruption (SD) was stage dependent (Stage 3; 45% (83/184) vs. Stages 1 and 2; 26% (206/801)[p < 0.0001]). This lessened in the winter (23%) vs. spring (32%) p = 0.02. LWB severe disruption was similar in spring 27%, summer 22%, winter 25% and was associated with Stage 3. CONCLUSIONS: Steps taken during the pandemic reduced SD but did not improve LWB. Systemic efforts are needed to protect learners and combat isolation pervasive in a pandemic.

Non-technical skill assessments across levels of US surgical training.

BACKGROUND: To ensure safe patient care, regulatory bodies worldwide have incorporated non-technical skills proficiency in core competencies for graduation from surgical residency. We describe normative data on non-technical skill ratings of surgical residents across training levels using the US-adapted Non-Technical Skills for Surgeons (NOTSS-US) assessment tool. METHODS: We undertook an exploratory, prospective cohort study of 32 residents-interns (postgraduate year 1), junior residents (postgraduate years 2-3), and senior residents (postgraduate years 4-5)-across 3 US academic surgery residency programs. Faculty went through online training to rate residents, directly observed residents while operating together, then submitted NOTSS-US ratings on specific resident's intraoperative performance. Mean NOTSS-US ratings (total range 4-20, sum of category scores; situation awareness, decision-making, communication/teamwork, leadership each ranged 1-5, with 1=poor, 3=average, 5=excellent) were stratified by residents' training level and adjusted for resident-, rater-, and case-level variables, using mixed-effects linear regression. RESULTS: For 80 operations, the overall mean total NOTSS-US rating was 12.9 (standard deviation, 3.5). The adjusted mean total NOTSS-US rating was 16.0 for senior residents, 11.6 for junior residents, and 9.5 for interns. Adjusted differences for total NOTSS-US ratings were statistically significant across the following training levels: senior residents to interns (6.5; 95% confidence interval, 4.3-8.7; P < .001), senior to junior residents (4.4; 95% confidence interval, 2.5-6.2; P < .001), and junior residents to interns (2.1; 95% confidence interval, 0.3-3.9; P = .017). Differences in adjusted NOTSS-US ratings across residents' training levels persisted for individual NOTSS-US behavior categories. CONCLUSION: These data and online training materials can support US residency programs in determining competency-based performance milestones to develop surgical trainees' non-technical skills.

Identifying Outlier Hospitals in Gastric Cancer Lymph Node Yield Using the National Cancer Database.

BACKGROUND: Lymph node (LN) yield is a key quality indicator that is associated with improved staging in surgically resected gastric cancer. The National Comprehensive Cancer Network recommends a yield of ≥15 LNs for proper staging, yet most facilities in the United States fail to achieve this number. The present study aimed to identify factors that could affect LN yield on a facility level and identify outlier hospitals. METHODS: This was a retrospective review of adults (aged ≥18 y) with gastric cancer (Tumor-Node-Metastasis Stages I-III) who underwent gastrectomy. Data were analyzed from the National Cancer Database (2004-2016). Multivariate analysis identified patient and tumor characteristics, whereas an observed-to-expected ratio of identified outlier hospitals. Facility factors were compared between high and low outliers. RESULTS: A total of 26,590 patients were included in this study. Of these patients, only 50.3% had an LN yield ≥15. The multivariate model of patient and tumor characteristics demonstrated a concordance index was 0.684. A total of 1245 facilities were included. There were 198 low outlier LN yield hospitals and 135 high outlier LN yield hospitals (observed-to-expected ratio of 0.42 ± 0.24 versus 1.38 ± 0.19, P < 0.0001). There was a difference in facility type between low and high outliers (P < 0.0001). High LN yield hospitals had a larger surgical volume than low LN yield hospitals (median 8.4 [4.9, 13.5] versus 3.5 [2.4, 5.2]; P < 0.0001). CONCLUSIONS: Nearly half of the population exhibited low compliance to National Comprehensive Cancer Network recommendations. Facility-level disparities exist as high yearly surgical volume and academic facility status distinguished high-performing outlier hospitals.

Expansion of Surgical Graduate Medical Education Training Programs: A Return on Investment Analysis.

BACKGROUND: The productivity of surgical departments is limited by the staffing of attending surgeons as well as surgical residents. Despite ongoing surgeon shortages, many health care organizations have been reluctant to expand training programs because of concerns about cost. We sought to determine the return on investment for the expansion of surgical training programs within our health system. METHODS: This study was completed as a retrospective review comparing two independent surgical departments at separate hospitals within a single integrated health system, including complete fiscal information from 2012 to 2019. Hospital A is a 594-bed hospital with large growth in its graduate surgical training programs over the study's period, whereas Hospital B is a 320-bed hospital where there was no expansion in surgical education initiatives. Case volumes, the number of full-time employees (FTE), and revenue data were obtained from our health systems business office. The number of surgical trainees, including general surgery residents and vascular surgery fellows, was provided by our office of Graduate Medical Education. The average yearly net revenue per surgeon was calculated for each training program and hospital location. RESULTS: Our results indicate a positive association between the number of surgical trainees and departmental net revenue, as well as the annual revenue generated per physician FTE. Each additional ancillary provider per physician FTE resulted in a positive impact of $112,552-$264,003 (R2 of 0.69 to 0.051). CONCLUSIONS: Regardless of hospital location or surgical specialty, our results demonstrate a positive association between the average net revenue generated per surgeon and the number of surgical trainees supporting the department. These findings are novel and provide evidence of a positive return on investment when surgical training programs are expanded.

Have the Characteristics for Success as a Surgeon Changed? A Century of Perspective Through the American College of Surgeons.

BACKGROUND: The Presidential Address of the American College of Surgeons (ACS) is an influential platform during the convocation for new Fellows every year. Recent work reported that most ACS presidents primarily discuss personal characteristics for success; however, these qualities were never specified. Therefore, this study aimed to identify the personal characteristics that are espoused in ACS presidential addresses as essential for success as a surgeon. STUDY DESIGN: Thematic analysis was completed for every ACS presidential address (98 addresses between 1913 and 2019). Full-text addresses were reviewed (2 team members), personal characteristics were coded (1 team member) and then assembled into patterns and themes (3 team-members' consensus). A temporal frame was adopted in grouping these themes in that personal qualities that appeared consistently throughout this period were classified as Enduring Characteristics and those that emerged only in later years were classified as Recent Characteristics. RESULTS: Enduring Characteristics that were present throughout the century included sincere compassion for patients; integrity; engagement (willingness to help shape the changing field at the institutional or national level); and commitment to lifelong learning. Recent Characteristics included humility and the interpersonal attributes of inclusivity and the ability to be a collaborative team leader. CONCLUSIONS: Surgery has experienced countless paradigm shifts since 1913, and the perceived characteristics for success have similarly evolved to include more interpersonal abilities. The importance of sincere compassion for patients, integrity, engagement, and commitment to lifelong learning remained consistent for more than a century.

Disparate opinions on the value of Vice Chairs of education in Departments of Surgery: A national survey of Department Chairs and other surgical education stakeholders.

BACKGROUND: The position of Vice Chair of Education (VCE) is increasingly common in Surgery Departments. The role remains ill-defined. The purpose of this study was to explore perceptions of Department Chairs (DCs) and Other Education Stakeholders (OESs) regarding the VCE role. METHODS: DCs and OESs at institutions with a VCE were surveyed. Descriptive statistics and cross-tabulations were calculated (SAS V9.4). RESULTS: The overall response rate was 25% (166/666). There were significant differences in whether DCs and OESs agree that the VCE supports others in fulfilling educational roles (95.2% vs 49.5%, p = 0.0002), is critical in achieving education missions (90.5% vs 56.6%, p = 0.0032), enhances the quality of education (95.3% vs 65.7%, p = 0.0174), and is important to education teams (95.0% vs 68.7%, p = 0.0464). CONCLUSIONS: DCs value the VCE role more so than OESs, whom VCEs support. In order for VCEs to be effective educational leaders in Departments of Surgery, the needs of key stakeholders deserve further clarification.

Quality Improvement for Surgical Resection of Pancreatic Head Adenocarcinoma : Hospital and Surgeon Predictors of Higher Than Expected R1 Resection Using the National Cancer Database.

BACKGROUND: The mainstay of treatment for pancreatic cancer is surgical resection; however, positive surgical margins remain commonplace. We identified hospitals with higher than predicted rates of positive margins and isolated factors that caused this discordance. METHODS: This is a retrospective review of patients with head of the pancreas adenocarcinoma in the National Cancer Database between 2004 and 2015. A nomogram was used to calculate the observed to expected positive margin rates (O/E) for facilities. If the O/E differed significantly (P < .05), it was considered an outlier. RESULTS: Among a total of 19 968 patients, 24.3% had positive margins. Among hospitals with lower than expected positive margin rates, 73.6% were academic or research programs, 17% were comprehensive community cancer programs, and none were community cancer programs (P = .0002). Within the group with higher than expected positive margin rates, 47% were comprehensive community cancer programs and 38.6% were academic or research programs (P = .0002). The mean hospital volume was higher in the low positive margin group (110.4 vs 48.8, P < .0001). CONCLUSIONS: Facility type and hospital volume can predict improvement in the O/E ratio for margin positivity in pancreatic adenocarcinoma resection. Surgeons should consider referral to academic or research facilities with higher case volumes for improved surgical resection.

Robotic Inguinal Hernia Repair Outcomes: Operative Time and Cost Analysis.

BACKGROUND: Robotic inguinal hernia repair is the latest iteration of minimally invasive herniorrhaphy. Previous studies have shown expedited learning curves compared to traditional laparoscopy, which may be offset by higher cost and longer operative time. We sought to compare operative time and direct cost across the evolving surgical practice of 10 surgeons in our healthcare system. METHODS: This is a retrospective review of all transabdominal preperitoneal robotic inguinal hernia repairs performed by 10 general surgeons from July 2015 to September 2018. Patients requiring conversion to an open procedure or undergoing simultaneous procedures were excluded. The data was divided to compare each surgeon's initial 20 cases to their subsequent cases. Direct operative cost was calculated based on the sum of supplies used intra-operatively. Multivariate analysis, using a generalized estimating equation, was adjusted for laterality and resident involvement to evaluate outcomes. RESULTS: Robotic inguinal hernia repairs were divided into two groups: early experience (n = 167) and late experience (n = 262). The late experience had a shorter mean operative time by 17.6 min (confidence interval: 4.06 - 31.13, p = 0.011), a lower mean direct operative cost by $538.17 (confidence interval: 307.14 - 769.20, p < 0.0001), and fewer postoperative complications (p = 0.030) on multivariate analysis. Thirty-day readmission rates were similar between both groups. CONCLUSION: Increasing surgeon experience with robotic inguinal hernia repair is associated with a predictable reduction in operative time, complication rates, and direct operative cost per case. Thirty-day readmission rates are not affected by the learning curve.

Impact of the COVID-19 Pandemic on Surgical Training and Learner Well-Being: Report of a Survey of General Surgery and Other Surgical Specialty Educators.

BACKGROUND: The COVID-19 pandemic disrupted the delivery of surgical services. The purpose of this communication was to report the impact of the pandemic on surgical training and learner well-being and to document adaptations made by surgery departments. STUDY DESIGN: A 37-item survey was distributed to educational leaders in general surgery and other surgical specialty training programs. It included both closed- and open-ended questions and the self-reported stages of GME during the COVID-19 pandemic, as defined by the ACGME. Statistical associations for items with stage were assessed using categorical analysis. RESULTS: The response rate was 21% (472 of 2,196). US stage distribution (n = 447) was as follows: stage 1, 22%; stage 2, 48%; and stage 3, 30%. Impact on clinical education significantly increased by stage, with severe reductions in nonemergency operations (73% and 86% vs 98%) and emergency operations (8% and 16% vs 34%). Variable effects were reported on minimal expected case numbers across all stages. Reductions were reported in outpatient experience (83%), in-hospital experience (70%), and outside rotations (57%). Increases in ICU rotations were reported with advancing stage (7% and 13% vs 37%). Severity of impact on didactic education increased with stage (14% and 30% vs 46%). Virtual conferences were adopted by 97% across all stages. Severity of impact on learner well-being increased by stage-physical safety (6% and 9% vs 31%), physical health (0% and 7% vs 17%), and emotional health (11% and 24% vs 42%). Regardless of stage, most but not all made adaptations to support trainees' well-being. CONCLUSIONS: The pandemic adversely impacted surgical training and the well-being of learners across all surgical specialties proportional to increasing ACGME stage. There is a need to develop education disaster plans to support technical competency and learner well-being. Careful assessment for program advancement will also be necessary. The experience during this pandemic shows that virtual learning and telemedicine will have a considerable impact on the future of surgical education.

Application Factors May Not Be Predictors of Success Among General Surgery Residents as Measured by ACGME Milestones.

BACKGROUND: Can factors within the Electronic Residency Application Service application be used to predict the success of general surgery residents as measured by the Accreditation Council for Graduate Medical Education (ACGME) general surgery milestones? METHODS: This is a retrospective study of 21 residents who completed training at a single general surgery residency program. Electronic Residency Application Service applications were reviewed for objective data, such as age, US Medical Licensing Examination scores, and authorship of academic publications as well as for letters of recommendation, which were scored using a standardized grading system. These factors were correlated to resident success as measured by ACGME general surgery milestone outcomes using univariate and multivariate analyses. This study was conducted at a single academic tertiary care and level 1 trauma facility. Residents who completed general surgery residency training from the years of 2012-2018 were included in the study. RESULTS: There were few correlations between application factors and resident success determined by the ACGME milestones. CONCLUSIONS: Application factors alone do not account for ongoing growth and development throughout residency. Unlike the results presented in the literature for other surgical subspecialties, predicting general surgery resident success based on application factors is not straightforward.

MAGIC versus MacDonald treatment regimens for gastric cancer: Trends and predictors of multimodal therapy for gastric cancer using the National Cancer Database.

BACKGROUND: Multimodal therapy is beneficial in gastric cancer, however this practice is not universal. This study examines trends, identifies associative factors, and examines overall survival (OS) benefit from multimodal therapy in gastric cancer. METHODS: Gastric cancer patients staged IB-III from 2005 to 2014, identified using the National Cancer Database, were categorized by treatment: surgery alone, perioperative chemotherapy, and adjuvant chemoradiation. Groups were analyzed to identify associative factors of perioperative therapy. RESULTS: We examined 9243 patients, with the majority receiving multimodal therapy (57%). The proportion of those receiving perioperative chemotherapy rose dramatically from 7.5% in 2006 to 46% in 2013. Academic center treatment was strongly associated with perioperative over adjuvant therapy (p < 0.0001). An OS advantage was clearly seen in those receiving multimodal therapy versus surgery alone (p < 0.0001), with no difference between perioperative and adjuvant therapies. CONCLUSIONS: Treatment of gastric cancer with multimodal therapy has risen significantly since 2005, largely due to increasing use of perioperative chemotherapy. As perioperative therapy becomes more prevalent, more patients will have the opportunity for the improved survival benefit of multimodal therapy.

Meaningful autonomy in general surgery training: Exploring for gender bias.

BACKGROUND: Resident autonomy is essential to the development of a surgical resident. This study aims to analyze gender differences in meaningful autonomy (MA) given to general surgery trainees intraoperatively. METHODS: This is a retrospective study of general surgery residents at an academic-affiliated tertiary care facility. Attending surgeons completed post-operative evaluations based on the Zwisch model (4-point scale, ≥3 indicating MA). RESULTS: Attending faculty members (37 males, 15 females) completed evaluations of 35 residents (18 males, 17 females). A total of 3574 evaluations were analyzed (1380 female, 2194 male residents) over 28 months. Multivariate analysis revealed case complexity, post graduate year level and rater gender were significantly associated with MA. Resident gender and faculty experience did not impact MA. CONCLUSIONS: In contrast to published literature, resident gender did not influence MA. This may be encouraging to surgical programs seeking strategies to address gender bias.