Impact of the COVID-19 pandemic on American College of Surgeons-Accredited Education Institutes & American Society of Anesthesiologists-Simulation Education Network: Opportunities for interdisciplinary collaboration.

BACKGROUND: The COVID-19 pandemic presented challenges for simulation programs including American College of Surgeons Accredited Education Institutes and American Society of Anesthesiologists Simulation Education Network. American College of Surgeons Accredited Education Institutes and American Society of Anesthesiologists Simulation Education Network leadership were surveyed to identify opportunities to enhance patient safety through simulation. METHODS: Between January and June 2021, surveys consisting of 3 targeted domains: (I) Changing practice; (II) Contributions and recognition; and (III) Moving ahead were distributed to 100 American College of Surgeons Accredited Education Institutes and 54 American Society of Anesthesiologists Simulation Education Network centers. Responses were combined and percent frequencies reported. RESULTS: Ninety-six respondents, representing 51 (51%) American College of Surgeons Accredited Education Institutes, 17 (31.5%) American Society of Anesthesiologists Simulation Education Network, and 28 dually accredited centers, completed the survey. Change of practice. Although 20.3% of centers stayed fully operational at the COVID-19 onset, 82% of all centers closed: 32% were closed less than 3 months, 28% were closed 3 to 6 months, 8% were closed 7 to 9 months, and 32% remained closed as of June 6, 2021. Most impacted activities were large-group instruction and team training. Sixty-nine percent of programs converted in-person to virtual programs. Contributions. The top reported innovative contributions included policies (80%), curricula (80%), and scholarly work (74%), Moving ahead. The respondents' top concerns were returning to high-quality training to best address learners' deficiencies and re-engagement of re-directed training programs. When asked "How the American College of Surgeons/American Society of Anesthesiologists Programs could best assist your simulation center goals?" the top responses were "facilitate collaboration" and "publish best practices from this work." CONCLUSION: The Pandemic presented multiple challenges and opportunities for simulation centers. Opportunities included collaboration between American College of Surgeons Accredited Education Institutes and the American Society of Anesthesiologists Simulation Education Network to identify best practices and resources needed to enhance patient safety through simulation.

American College of Surgeons Objective Assessment of Skills in Surgery (ACS OASIS): A Formative Assessment of Junior Residents' Technical Skills.

OBJECTIVE: The objective assessment of technical skills of junior residents is essential in implementing competency-based training and providing specific feedback regarding areas for improvement. An innovative assessment that can be easily implemented by training programs nationwide has been developed by expert surgeon educators under the aegis of the American College of Surgeons (ACS) Division of Education. This assessment, ACS Objective Assessment of Skills in Surgery (ACS OASIS) uses eight stations to address technical skills important for junior residents within the domains of laparoscopic appendectomy, excision of lipoma, central line placement, laparoscopic cholecystectomy, trocar placement, exploratory laparotomy, repair of enterotomy, and tube thoracostomy. The purpose of this study was to implement ACS OASIS at a number of sites to study its psychometric rigor. DESIGN: The ACS OASIS was pre-piloted at two programs to establish feasibility and to gather information regarding implementation. Each skills station was 12 minutes long, and the faculty completed a checklist with 5 to 15 items, and a global assessment scale. The study was then repeated at three pilot sites and included 29 junior residents who were assessed by a total of 44 faculty. Psychometric data for the stations and checklists were collected and analyzed. SETTING: The pre-pilot sites were Geisinger and University of Tennessee Knoxville.Data were gathered from pilot sites that included Wellspan Health, Duke University, and University of California Los Angeles. RESULTS: The mean checklist score for all learners was 76% (IQR of 66%-85%). The average global rating was 3.36 on a 5-point scale with a standard deviation of 0.56. The overall cut score derived using the borderline group method was at 68% with 34% of performances requiring remediation. Using this criterion, the average number of stations that were completed by each learner without need for remediation was five.The station discrimination index ranged from 0.27 to 0.65 (all above the threshold of 0.25), demonstrating solid psychometric characteristics at the station level. The internal-consistency reliability was 0.76 with SEM of 5.8%. The inter-rater reliability (intraclass correlation) was high at 0.73 with general agreement of 79% between the two raters. The station discrimination was at 0.45 (range of 0.27 to 0.65) indicating a high level of differentiation between high and low performers. Using the generalizability theory, the G-coefficient reliability was at 0.72 with the reliability projection flattening after 8 stations. Overall, 75% to 82% the faculty and learners rated ACS OASIS as realistic and beneficial. CONCLUSIONS: ACS OASIS is a psychometrically sound technical skills assessment tool that can provide useful information for feedback to junior residents and support efforts to remediate gaps in performance.

What are the Top Research Priorities in Surgical Simulation and How Can They Be Best Addressed? Results From a Multidisciplinary Consensus Conference.

OBJECTIVES: To define the top priorities in simulation-based surgical education where additional research would have the highest potential to advance the field and develop proposals that would address the identified research priorities. SUMMARY AND BACKGROUND DATA: Simulation has become integral part of surgical training but there are a number of outstanding questions that have slowed advances in this field. METHODS: The Delphi methodology was used to define the top priorities in simulation-based surgical education. A research summit was held with multiple stakeholders under the auspices of the American College of Surgeons Division of Education to develop proposals to address these priorities. RESULTS: Consensus was achieved after the first round of voting on the following 3 most important topics: (1) impact of simulation training on patient safety and outcomes, (2) the value proposition of simulation, and (3) the use of simulation for physician certification and credentialing. Knowledge gaps, challenges and opportunities, and research questions to address these topics were defined by summit participants. CONCLUSIONS: The top 3 priorities in surgical simulation research were defined and project outlines were developed for impactful projects on these topics. Successful completion of such projects is expected to advance the field of simulation-based surgical education.

Advanced Modular Manikin and Surgical Team Experience During a Trauma Simulation: Results of a Single-Blinded Randomized Trial.

BACKGROUND: Our aim was assess whether an integrated Advanced Modular Manikin (AMM) provides improved participant experience compared with use of peripheral simulators alone during a standardized trauma team scenario. Simulation-based team training has been shown to improve team performance. To address limitations of existing manikin simulators, the AMM platform was created that enables interconnectedness, interoperability, and integration of multiple simulators ("peripherals") into an adaptable, comprehensive training system. METHODS: A randomized single-blinded, crossover study with 2 conditions was used to assess learner experience differences when using the integrated AMM platform vs peripheral simulators. First responders, anesthesiologists, and surgeons rated their experience and workload with the conditions in a 3-scene standardized trauma scenario. Participant ratings were compared and focus groups conducted to obtain insight into participant experience. RESULTS: Fourteen teams (n = 42) participated. Team experience ratings were higher for the integrated AMM condition compared with peripherals (Cohen's d = .25, p = 0.016). Participant experience varied by background with surgeons and first responders rating their experience significantly higher compared with anesthesiologists (p < 0.001). Higher workload ratings were observed with the integrated AMM condition (Cohen's d = .35, p = 0.014) driven primarily by anesthesiologist ratings. Focus groups revealed that participants preferred the integrated AMM condition based on its increased realism, physiologic responsiveness, and feedback provided on their interventions. CONCLUSIONS: This first comprehensive evaluation suggests that integration with the AMM platform provides benefits over individual peripheral simulators and has the potential to expand simulation-based learning opportunities and enhance learner experience, especially for surgeons.

American College of Surgeons/Association for Surgical Education medical student simulation-based surgical skills curriculum needs assessment.

BACKGROUND: Simulation can enhance learning effectiveness, efficiency, and patient safety and is engaging for learners. METHODS: A survey was conducted of surgical clerkship directors nationally and medical students at 5 medical schools to rank and stratify simulation-based educational topics. Students applying to surgery were compared with others using Wilcoxon's rank-sum tests. RESULTS: Seventy-three of 163 clerkship directors (45%) and 231 of 872 students (26.5%) completed the survey. Of students, 28.6% were applying for surgical residency training. Clerkship directors and students generally agreed on the importance and timing of specific educational topics. Clerkship directors tended to rank basic skills, such as examination skills, higher than medical students. Students ranked procedural skills, such as lumbar puncture, more highly than clerkship directors. CONCLUSIONS: Surgery clerkship directors and 4th-year medical students agree substantially about the content of a simulation-based curriculum, although 4th-year medical students recommended that some topics be taught earlier than the clerkship directors recommended. Students planning to apply to surgical residencies did not differ significantly in their scoring from students pursuing nonsurgical specialties.

American College of Surgeons Resident Objective Structured Clinical Examination: a national program to assess clinical readiness of entering postgraduate year 1 surgery residents.

OBJECTIVE: This report describes the development, initial implementation, and reliability of American College of Surgeons Resident Objective Structured Clinical Examination (ACS OSCE). BACKGROUND: Variability in clinical knowledge and skills of entering surgery residents has been demonstrated. The ACS OSCE was developed to evaluate and help remediate residents' knowledge and skills in managing patients with life-threatening conditions. METHODS: A task force of surgeons and professional educators developed 10 standardized clinical case stations, evaluation checklists, and rating scales. Standardized patients (SPs) evaluated each resident's clinical skills (history taking, physical examination, communication, and SP-global scores). Residents completed checklists on diagnosis and management. Coefficient alpha and item-total correlations were used, respectively, to assess internal consistency of metrics and station validity. The resident's overall performance for each station was calculated by combining scores of the individual skills. Analysis of variance compared performance across different institutions. RESULTS: A total of 103 postgraduate year 1 residents from 7 institutions completed the OSCE. Reliability coefficients of skills ranged from 0.38 for diagnosis to 0.68 for global scores. For overall performance on individual stations, the reliability coefficients ranged from 0.51 to 0.82. Using total percent correct scores from highly reliable stations (α > 0.8), wide variability in resident performance was demonstrated within and between the 7 institutions. CONCLUSIONS: The ACS OSCE was successfully implemented across diverse institutions. It had moderate reliability and demonstrated variability among entering surgery residents. The ACS OSCE is now available for broader implementation. It should help reduce resident variability and address the requirements of Accreditation Council for Graduate Medical Education for resident supervision.

Women in surgery residency programs: evolving trends from a national perspective.

BACKGROUND: Similar numbers of men and women are currently graduating from United States (US) medical schools; therefore, surgery residency programs need to attract graduates of both genders. This study compared gender distributions of allopathic US medical graduates (USMG) from academic years 1999-2000 through 2004-2005. In addition, the gender distributions of USMG and international medical graduates (IMG; analyzed separately) entering accredited general surgery (GS) programs and USMG entering other surgical specialty programs were compared across academic years 2000-2001 through 2005-2006. STUDY DESIGN: Data were extracted from the American College of Surgeons Resident Master File and the Association of American Medical Colleges FACTS Website and Data Warehouse. Chi-square statistics compared gender distributions across years for all USMG graduating and applying to GS programs each year between 1999-2000 and 2004-2005 and for USMG and IMG entering training between 2000-2001 and 2005-2006. RESULTS: During the study period, the proportion of women increased significantly (p < 0.001) among USMG (43% to 47%), USMG applying to GS programs (27% to 33%), and USMG entering GS residencies (32% to 40%); the percentages of women among IMG entering GS residencies ranged from 11% to 18%, with no apparent linear increase. Proportions of women among USMG entering training increased in most surgical specialties examined. CONCLUSIONS: The gender gap among USMG entering GS training appears to be closing, concurrent with that of USMG overall during the study period. Surgery programs must continue to recruit and retain women to attract the best and brightest trainees.

Multi-institutional validation of a web-based core competency assessment system.

OBJECTIVE: The Association of Program Directors in Surgery and the Division of Education of the American College of Surgeons developed and implemented a web-based system for end-of-rotation faculty assessment of ACGME core competencies of residents. This study assesses its reliability and validity across multiple programs. METHODS: Each assessment included ratings (1-5 scale) on 23 items reflecting the 6 core competencies. A total of 4241 end-of-rotation assessments were completed for 332 general surgery residents (> or =5 evaluations each) at 5 sites during the 2004-2005 and 2005-2006 academic years. The mean rating for each resident on each item was computed for each academic year. The mean rating of items representing each competency was computed for each resident. Additional data included USMLE and ABSITE scores, PGY, and status in program (categorical, designated preliminary, and undesignated preliminary). RESULTS: Coefficient alpha was greater than 0.90 for each competency score. Mean ratings for each competency increased significantly (p < 0.01) as a function of PGY. Mean ratings for professionalism and interpersonal/communication skills (IPC) were significantly higher than all other competencies at all PGY levels. Competency ratings of PGY 1 residents correlated significantly with USMLE Step I, ranging from (r = 0.26, p < 0.01) for Professionalism to (r = 0.41, p < 0.001) for Systems-Based Practice. Ratings of Knowledge (r = 0.31, p < 0.01), Practice-Based Learning & Improvement (PBLI; r = 0.22, p < 0.05), and Systems-Based Practice (r = 0.20, p < 0.05) correlated significantly with 2005 ABSITE Total Percentile. Ratings of all competencies correlated significantly with the 2006 ABSITE Total Percentile Score (range: r = 0.20, p < 0.05 for professionalism to r = 0.35, p < 0.001 for knowledge). Categorical and designated preliminary residents received significantly higher ratings (p < 0.05) than nondesignated preliminaries for knowledge, patient care, PBLI, and systems-based practice only. CONCLUSIONS: Faculty ratings of core competencies are internally consistent. The pattern of statistically significant correlations between competency ratings and USMLE and ABSITE scores supports the postdictive and concurrent validity, respectively, of faculty perceptions of resident knowledge. The pattern of increased ratings as a function of PGY supports the construct validity of faculty ratings of resident core competencies.