A review of the American College of Surgeons accredited education institutes' assessment practices for learners, faculty, and continuous program improvement.

Purpose: Simulation is an instructional modality that offers opportunities for assessment across many domains. The American College of Surgeons created the Accredited Education Institutes (AEIs) to build a community of high-quality simulation centers focused around improving surgical education and training. The goals of this project were to identify assessment methods used by AEIs, discuss how these methods align with established assessment frameworks, identify best practices, and provide guidance on best practice implementation. Methods: The authors analyzed responses regarding learner assessment, faculty assessment, and continuous program improvement from AEI accreditations surveys using deductive qualitative analysis. Results: Data from ninety-six centers were reviewed. Codes for each category were organized into formal and informal themes. For learner assessment, examinations and checklists identified as the most common types of formal assessment used and oral feedback as the most common type of informal assessment. For faculty assessment, written evaluations were the most common formal type and debriefs were the most common informal type. For continuous program improvement, written evaluations were the most common formal type and oral feedback was the most frequent informal type. Discussion: The goal of assessment should be to encourage learning through feedback and to ensure the attainment of educational competencies. The data revealed a variety of assessment modalities used to accomplish this goal with AEIs frequently utilizing some of the most reliable forms of assessment. We discuss how these forms of assessment can be integrated with best practices to develop assessment portfolios for learners and faculty, performance improvement reports for faculty, and assessments of clinical impact. Supplementary Information: The online version contains supplementary material available at 10.1007/s44186-023-00132-6.

Team FIRST framework: Identifying core teamwork competencies critical to interprofessional healthcare curricula.

Interprofessional healthcare team function is critical to the effective delivery of patient care. Team members must possess teamwork competencies, as team function impacts patient, staff, team, and healthcare organizational outcomes. There is evidence that team training is beneficial; however, consensus on the optimal training content, methods, and evaluation is lacking. This manuscript will focus on training content. Team science and training research indicates that an effective team training program must be founded upon teamwork competencies. The Team FIRST framework asserts there are 10 teamwork competencies essential for healthcare providers: recognizing criticality of teamwork, creating a psychologically safe environment, structured communication, closed-loop communication, asking clarifying questions, sharing unique information, optimizing team mental models, mutual trust, mutual performance monitoring, and reflection/debriefing. The Team FIRST framework was conceptualized to instill these evidence-based teamwork competencies in healthcare professionals to improve interprofessional collaboration. This framework is founded in validated team science research and serves future efforts to develop and pilot educational strategies that educate healthcare workers on these competencies.

At-home medical student simulation: achieving knot-tying proficiency using video-based assessment.

Purpose: Due to the pandemic, we restructured our medical student knot-tying simulation to a virtual format. This study evaluated curriculum feasibility and effectiveness. Methods: Over 4 weeks, second-year medical students (n = 229) viewed a video tutorial (task demonstration, errors, scoring) and self-practiced to proficiency (no critical errors, < 2 min) using at-home suture kits (simple interrupted suture, instrument tie, penrose drain model). Optional virtual tutoring sessions were offered. Students submitted video performance for proficiency verification. Two sets of 14 videos were viewed by two surgeons until inter-rater reliability (IRR) was established. Students scoring "needs remediation" attended virtual remediation sessions. Non-parametric statistics were performed using RStudio. Results: All 229 medical students completed the curriculum within 1-4 h; 1.3% attended an optional tutorial. All videos were assessed: 4.8% "exceeds expectations", 60.7% "meets expectations", and 34.5% "needs remediation." All 79 needing remediation due to critical errors achieved proficiency during 1-h group sessions. IRR Cohen's κ was 0.69 (initial) and 1.0 (ultimate). Task completion time was 56 (47-68) s (median [IQR]); p < 0.01 between all pairs. Students rated the overall curriculum (79.2%) and overall curriculum and video tutorial effectiveness (92.7%) as "agree" or "strongly agree". No definitive preference emerged regarding virtual versus in-person formats; however, 80.2% affirmed wanting other at-home skills curricula. Comments supported home practice as lower stress; remediation students valued direct formative feedback. Conclusions: A completely virtual 1-month knot-tying simulation is feasible and effective in achieving proficiency using video-based assessment and as-needed remediation strategies for a large student class.

Optimizing Medical Student Clerkship Schedules Using a Novel Application of the Hungarian Algorithm.

PROBLEM: Medical students often have preferences regarding the order of their clinical rotations, but assigning rotations fairly and efficiently can be challenging. To achieve a solution that optimizes assignments (i.e., maximizes student satisfaction), the authors present a novel application of the Hungarian algorithm, designed at the University of Texas Southwestern Medical Center (UTSW), to assign student schedules. APPROACH: Possible schedules were divided into distinct pathway options with k total number of seats. Each of n students submitted a ranked list of their top 5 pathway choices. An n × k matrix was formed, where the location (i, j) represented the cost associated with student i being placed in seat j. Progressively higher costs were assigned to students receiving less desired pathways. The Hungarian algorithm was then used to find the assignments that minimize total cost. The authors compared the performance of the Hungarian algorithm against 2 alternative algorithms (i.e., the rank and lottery algorithms). To evaluate the 3 algorithms, 4 simulations were conducted with different popularity weights for different pathways and were run across 1,000 trials. The algorithms were also compared using 3 years of UTSW student preference data for the classes of 2019, 2020, and 2021. OUTCOMES: In all 4 computer simulations, the Hungarian algorithm resulted in more students receiving 1 of their top 3 choices and fewer students receiving none of their preferences. Similarly, for UTSW student preference data, the Hungarian algorithm resulted in more students receiving 1 of their top 3 preferences and fewer students receiving none of their ranked preferences. NEXT STEPS: This approach may be broadly applied to scheduling challenges in undergraduate and graduate medical education. Furthermore, by manipulating cost values, additional constraints can be enforced (e.g., requiring certain seats to be filled, attempting to avoid schedules that begin with a student's desired specialty).

Adaptive Surgical Robotic Training Using Real-Time Stylistic Behavior Feedback Through Haptic Cues.

Surgical skill directly affects surgical procedure outcomes; thus, effective training is needed to ensure satisfactory results. Many objective assessment metrics have been developed that provide the trainee with descriptive feedback about their performance however, often lack feedback on how to improve performance. The most effective training method is one that is intuitive, easy to understand, personalized to the user,and provided in a timely manner. We propose a framework to enable user-adaptive training using near real-time detection of performance, based on intuitive styles of surgical movements, and design a haptic feedback framework to assist with correcting styles of movement. We evaluate the ability of three types of force feedback (spring, damping, and spring plus damping feedback), computed based on prior user positions, to improve different stylistic behaviors of the user during kinematically constrained reaching movement tasks. The results indicate that five out of six styles studied here were improved using at least one of the three types of force feedback. Task performance metrics were compared in the presence of the three types of feedback. Task time was statistically significantly lower when applying spring feedback, compared to the other two types of feedback. Path straightness and targeting error were statistically significantly improved when using spring-damping feedback compared to the other two types of feedback. This study presents a groundwork for adaptive training in robotic surgery based on near real-time human-centric models of surgical behavior.

Overarching themes from ACS-AEI accreditation survey best practices 2011-2019.

The American College of Surgeons launched the Accredited Education Institutes (ACS-AEIs) in 2005 to create a network of simulation centers to offer the highest-quality surgical education and training and to pursue cutting-edge scholarship. Specific standards and criteria were developed to accredit simulation centers, and decisions made based on the application, site surveyor visit, and ACS-AEI Accreditation Committee review. In addition to granting accreditation and providing recommendations for improvement, site surveyors and the Accreditation Committee identified best practices defined as areas far exceeding the accreditation standards or novel methods of advancing high-quality, impactful education. Best practices were compiled into a list starting in 2011 for dissemination to all members of the ACS-AEI Consortium through on-line videos, newsletters, and workshops at the Annual ACS Surgical Simulation Summit. Using thematic analysis, the authors classified common themes from the 337 best practices identified during 247 accreditation reviews between June 2011 and June 2019. As best practices were extracted, the authors compiled them into a single database. Eight overarching themes were identified, including approaches to faculty development, scholarly activity, development of curricula, use of resources, delivery of educational content, assessment of learners, and collaboration between centers. The authors discuss themes in the context of their practical relevance and highlight strategies used at different centers, with an emphasis on the importance of each theme in developing a successful simulation program.

Assessing quality and resources during campus-wide simulation integration.

INTRODUCTION: Our simulation center, supported by four departments (Surgery, OB/GYN, Urology, and Anesthesiology), is accredited as a comprehensive Accredited Educational Institute (AEI) and is now expanding to accommodate all departments on campus. METHODS: A 61-point questionnaire was administered to 44 stakeholders, representing all of UME and GME. Data were compared for AEI vs. non-AEI activities. RESULTS: Responses were collected from all 44 groups (100% response rate). Overall, 43 simulation activities were hosted within the AEI and 40 were hosted by non-AEI stakeholders. AEI activities were more likely to be mandatory (93% vs. 75%, p = 0.02), have written learning objectives (79% vs 43%, p < 0.001), and use validated assessment metrics (33% vs. 13%, p = 0.03). CONCLUSION: These data suggest that the AEI courses are more robust in terms of structured learning and assessment compared to non-AEI courses. Campus-wide application of uniform quality standards is anticipated to require significant faculty, course, and program development.

No Medical Student Left Behind: A Study to Examine the Effects of Structured Step 1 Support within our Curriculum.

This article was migrated. The article was marked as recommended. Introduction:The University of Texas Southwestern Medical School (UT Southwestern) developed a required, pass/fail Step 1 preparation course in conjunction with their new curriculum to support students with structure, encouragement, and accountability. This study examined the Step 1 performance of students who have taken the prep course after completing pre-clinical coursework and factors which predicted outcomes. Methods: Data were collected between January 2017 and July 2018 from 453 2 nd-year medical students enrolled in the six week course. Multilinear regression and Chi-squared analysis were performed to compare pre-clinical course performance with Step 1 readiness and outcomes. Results: Average course final exam scores and CBSSA results were significantly and directly associated with Step 1 scores. Pre-clinical exam performance, practice exam scores at the beginning of the Step 1 preparation period and an increase in practice exam scores over the study period correlated with Step 1 performance. The percent pass rates and mean scores in 2017 were at 97% and 235, but in 2018, no student failed Step 1 (100% pass rate), and the mean score rose to 239. Discussion and Conclusion: A structured Step 1 Prep Course was well-accepted by students. Exam performance can be predicted from pre-clerkship performance and progress made during the course on practice exams and early intervention for at-risk students can improve performance of at-risk students.

Automatic Surgical Skill Rating Using Stylistic Behavior Components.

A gold standard in surgical skill rating and evaluation is direct observation, which a group of experts rate trainees based on a likert scale, by observing their performance during a surgical task. This method is time and resource intensive. To alleviate this burden, many studies have focused on automatic surgical skill assessment; however, the metrics suggested by the literature for automatic evaluation do not capture the stylistic behavior of the user. In addition very few studies focus on automatic rating of surgical skills based on available likert scales. In a previous study we presented a stylistic behavior lexicon for surgical skill. In this study we evaluate the lexicon's ability to automatically rate robotic surgical skill, based on the 6 domains in the Global Evaluative Assessment of Robotic Skills (GEARS). 14 subjects of different skill levels performed two surgical tasks on da Vinci surgical simulator. Different measurements were acquired as subjects performed the tasks, including limb (hand and arm) kinematics and joint (shoulder, elbow, wrist) positions. Posture videos of the subjects performing the task, as well as videos of the task being performed were viewed and rated by faculty experts based on the 6 domains in GEARS. The paired videos were also rated via crowd-sourcing based on our stylistic behavior lexicon. Two separate regression learner models, one using the sensor measurements and the other using crowd ratings for our proposed lexicon, were trained for each domain in GEARS. The results indicate that the scores predicted from both prediction models are in agreement with the gold standard faculty ratings.

Developing a comprehensive, proficiency-based training program for robotic surgery.

INTRODUCTION: Robotically assisted surgery has become very popular for numerous surgical disciplines, yet training practices remain variable with little to no validation. The purpose of this study was to develop a comprehensive, proficiency-based robotic training program. METHODS: A skill deconstruction list was generated by observation of robotic operations and interviews with experts. Available resources were used, and other components were developed as needed to develop a comprehensive, proficiency-based curriculum to teach all deconstructed skills. Preliminary construct and content validity and curriculum feasibility were evaluated. RESULTS: The skill deconstruction list contained 23 items. Curricular components included an online tutorial, a half-day interactive session, and 9 inanimate exercises with objective metrics. Novice (546 ± 26) and expert (923 ± 60) inanimate composite scores were different (P < .001), supporting construct validity, and substantial pre-test to post-test improvement was noted after successful training completion. All 23 deconstructed skills were rated as highly relevant (4.9 ± 0.5; 5-point scale), and no skills were absent from the curriculum, supporting content validity. CONCLUSION: These data suggest that this proficiency-based training curriculum comprehensively addresses the skills necessary to perform robotic operations with early construct and content validity and feasibility demonstrated. Further validation is encouraged.

Comprehensive proficiency-based inanimate training for robotic surgery: reliability, feasibility, and educational benefit.

BACKGROUND: We previously developed a comprehensive proficiency-based robotic training curriculum demonstrating construct, content, and face validity. This study aimed to assess reliability, feasibility, and educational benefit associated with curricular implementation. METHODS: Over an 11-month period, 55 residents, fellows, and faculty (robotic novices) from general surgery, urology, and gynecology were enrolled in a 2-month curriculum: online didactics, half-day hands-on tutorial, and self-practice using nine inanimate exercises. Each trainee completed a questionnaire and performed a single proctored repetition of each task before (pretest) and after (post-test) training. Tasks were scored for time and errors using modified FLS metrics. For inter-rater reliability (IRR), three trainees were scored by two raters and analyzed using intraclass correlation coefficients (ICC). Data from eight experts were analyzed using ICC and Cronbach's α to determine test-retest reliability and internal consistency, respectively. Educational benefit was assessed by comparing baseline (pretest) and final (post-test) trainee performance; comparisons used Wilcoxon signed-rank test. RESULTS: Of the 55 trainees that pretested, 53 (96 %) completed all curricular components in 9-17 h and reached proficiency after completing an average of 72 ± 28 repetitions over 5 ± 1 h. Trainees indicated minimal prior robotic experience and "poor comfort" with robotic skills at baseline (1.8 ± 0.9) compared to final testing (3.1 ± 0.8, p < 0.001). IRR data for the composite score revealed an ICC of 0.96 (p < 0.001). Test-retest reliability was 0.91 (p < 0.001) and internal consistency was 0.81. Performance improved significantly after training for all nine tasks and according to composite scores (548 ± 176 vs. 914 ± 81, p < 0.001), demonstrating educational benefit. CONCLUSION: This curriculum is associated with high reliability measures, demonstrated feasibility for a large cohort of trainees, and yielded significant educational benefit. Further studies and adoption of this curriculum are encouraged.

Proficiency-based training for robotic surgery: construct validity, workload, and expert levels for nine inanimate exercises.

BACKGROUND: We previously developed nine inanimate training exercises as part of a comprehensive, proficiency-based robotic training curriculum that addressed 23 unique skills identified via task deconstruction of robotic operations. The purpose of this study was to evaluate construct validity, workload, and expert levels for the nine exercises. METHODS: Expert robotic surgeons (n = 8, fellows and faculty) and novice trainees (n = 4, medical students) each performed three to five consecutive repetitions of nine previously reported exercises (five FLS models with or without modifications and four custom-made models). Each task was scored for time and accuracy using modified FLS metrics; task scores were normalized to a previously established (preliminary) proficiency level and a composite score equaled the sum of the nine normalized task scores. Questionnaires were administered regarding prior experience. After each exercise, participants completed a validated NASA-TLX Workload Scale to rate the mental, physical, temporal, performance, effort, and frustration levels of each task. RESULTS: Experts had performed 119 (range = 15-600) robotic operations; novices had observed ≤ 1 robotic operation. For all nine tasks and the composite score, experts achieved significantly better performance than novices (932 ± 67 vs. 618 ± 111, respectively; P < 0.001). No significant differences in workload between experts and novices were detected (32.9 ± 3.5 vs. 32.0 ± 9.1, respectively; n.s.). Importantly, frustration ratings were relatively low for both groups (4.0 ± 0.7 vs. 3.8 ± 1.6, n.s.). The mean performance of the eight experts was deemed suitable as a revised proficiency level for each task. CONCLUSION: Using objective performance metrics, all nine exercises demonstrated construct validity. Workload was similar between experts and novices and frustration was low for both groups. These data suggest that the nine structured exercises are suitable for proficiency-based robotic training.

Content and face validity of a comprehensive robotic skills training program for general surgery, urology, and gynecology.

BACKGROUND: The authors previously developed a comprehensive, proficiency-based robotic training curriculum that aimed to address 23 unique skills identified via task deconstruction of robotic operations. The purpose of this study was to determine the content and face validity of this curriculum. METHODS: Expert robotic surgeons (n = 12) rated each deconstructed skill regarding relevance to robotic operations, were oriented to the curricular components, performed 3 to 5 repetitions on the 9 exercises, and rated each exercise. RESULTS: In terms of content validity, experts rated all 23 deconstructed skills as highly relevant (4.5 on a 5-point scale). Ratings for the 9 inanimate exercises indicated moderate to thorough measurement of designated skills. For face validity, experts indicated that each exercise effectively measured relevant skills (100% agreement) and was highly effective for training and assessment (4.5 on a 5-point scale). CONCLUSIONS: These data indicate that the 23 deconstructed skills accurately represent the appropriate content for robotic skills training and strongly support content and face validity for this curriculum.