Expanded Access to Video-Based Laparoscopic Skills Assessments: Ease, Reliability, and Accuracy.

OBJECTIVE: Video-based performance assessments provide essential feedback to surgical residents, but in-person and remote video-based assessment by trained proctors incurs significant cost. We aimed to determine the reliability, accuracy, and difficulty of untrained attending staff surgeon raters completing video-based assessments of a basic laparoscopic skill. Secondarily, we aimed to compare reliability and accuracy between 2 different types of assessment tools. DESIGN: An anonymous survey was distributed electronically to surgical attendings via a national organizational listserv. Survey items included demographics, rating of video-based assessment experience (1 = have never completed video-based assessments, 5 = often complete video-based assessments), and rating of favorability toward video-based and in-person assessments (0 = not favorable, 100 = favorable). Participants watched 2 laparoscopic peg transfer performances, then rated each performance using an Objective Structured Assessment of Technical Skill (OSATS) form and the McGill Inanimate System for Training and Evaluation of Laparoscopic Skills (MISTELS). Participants then rated assessment completion ease (1 = Very Easy, 5 = Very Difficult). SETTING: National survey of practicing surgeons. PARTICIPANTS: Sixty-one surgery attendings with experience in laparoscopic surgery from 10 institutions participated as untrained raters. Six experienced laparoscopic skills proctors participated as expert raters. RESULTS: Inter-rater reliability was substantial for both OSATS (k = 0.75) and MISTELS (k = 0.85). MISTELS accuracy was significantly higher than that of OSATS (κ: MISTELS = 0.18, 95%CI = [0.06,0.29]; OSATS = 0.02, 95%CI = [-0.01,0.04]). While participants were inexperienced with completing video-based assessments (median = 1/5), they perceived video-based assessments favorably (mean = 73.4) and felt assessment completion was "Easy" on average. CONCLUSIONS: We demonstrate that faculty raters untrained in simulation-based assessments can successfully complete video-based assessments of basic laparoscopic skills with substantial inter-rater reliability without marked difficulty. These findings suggest an opportunity to increase access to feedback for trainees using video-based assessment of fundamental skills in laparoscopic surgery.

National Profile of the ACGME Milestones 1.0 and 2.0 within General Surgery: A Seven-Year National Study from 2014 to 2021.

PURPOSE: The Accreditation Council for Graduate Medical Education (ACGME) introduced General Surgery Milestones 1.0 in 2014 and Milestones 2.0 in 2020 as steps toward competency-based training. Analysis will inform residency programs on curriculum development, assessment, feedback, and faculty development. This study describes the distributions and trends for Milestones 1.0 and 2.0 ratings and proportion of residents not achieving the level 4.0 graduation target. METHODS: A deidentified dataset of milestone ratings for all ACGME-accredited General Surgery residency programs in the United States was used. Medians and interquartile ranges (IQR) were reported for milestone ratings at each PGY level. Percentages of PGY-5s receiving final year ratings of less than 4.0 were calculated. Wilcoxon rank sum tests were used to compare 1.0 and 2.0 median ratings. Kruskal-Wallis and Bonferroni post-hoc tests were used to compare median ratings across time periods and PGY levels. Chi-squared tests were used to compare the proportion of level 4.0 nonachievement under both systems. RESULTS: Milestones 1.0 data consisted of 13,866 residents and Milestones 2.0 data consisted of 7,633 residents. For 1.0 and 2.0, all competency domain median ratings were higher for subsequent years of training. Milestones 2.0 had significantly higher median ratings at all PGY levels for all competency domains except Medical Knowledge. Percentages of PGY-5 residents not achieving the graduation target in Milestones 1.0 ranged from 27% to 42% and in 2.0 from 5% to 13%. For Milestones 1.0, all subcompetencies showed an increased number of residents achieving the graduation target from 2014 to 2019. CONCLUSIONS: This study of General Surgery Milestones 1.0 and 2.0 data uncovered significant increases in average ratings and significantly fewer residents not achieving the graduation target under the 2.0 system. We hypothesize that these findings may be related more to rating bias given the change in rating scales, rather than a true increase in resident ability.

Accreditation Council for Graduate Medical Education Milestone Training Ratings and Surgeons' Early Outcomes.

Importance: National data on the development of competence during training have been reported using the Accreditation Council for Graduate Medical Education (ACGME) Milestones system. It is now possible to consider longitudinal analyses that link Milestone ratings during training to patient outcomes data of recent graduates. Objective: To evaluate the association of in-training ACGME Milestone ratings in a surgical specialty with subsequent complication rates following a commonly performed operation, endovascular aortic aneurysm repair (EVAR). Design, Setting, and Participants: This study of patient outcomes followed EVAR in the Vascular Quality Initiative (VQI) registry (4213 admissions from 208 hospitals treated by 327 surgeons). All surgeons included in this study graduated from ACGME-accredited training programs from 2015 through 2019 and had Milestone ratings 6 months prior to graduation. Data were analyzed from December 1, 2021, through September 15, 2023. Because Milestone ratings can vary with program, they were corrected for program effect using a deviation score from the program mean. Exposure: Milestone ratings assigned to individual trainees 6 months prior to graduation, based on judgments of surgical competence. Main Outcomes and Measures: Surgical complications following EVAR for patients treated by recent graduates during the index hospitalization, obtained using the nationwide Society for Vascular Surgery Patient Safety Organization's VQI registry, which includes 929 participating centers in 49 US states. Results: The study included outcomes for 4213 patients (mean [SD] age, 73.25 [8.74] years; 3379 male participants [80.2%]). Postoperative complications included 9.5% major (400 of 4213 cases) and 30.2% minor (1274 of 4213 cases) complications. After adjusting for patient risk factors and site of training, a significant association was identified between individual Milestone ratings of surgical trainees and major complications in early surgical practice in programs with lower mean Milestone ratings (odds ratio, 0.50; 95% CI; 0.27-0.95). Conclusions and Relevance: In this study, Milestone assessments of surgical trainees were associated with subsequent clinical outcomes in their early career. Although these findings represent one surgical specialty, they suggest Milestone ratings can be used in any specialty to identify trainees at risk for future adverse patient outcomes when applying the same theory and methodology. Milestones data should inform data-driven educational interventions and trainee remediation to optimize future patient outcomes.

The Association of ACGME Milestones With Performance on American Board of Surgery Assessments: A National Investigation of Surgical Trainees.

OBJECTIVE: To determine the relationship between, and predictive utility of, milestone ratings and subsequent American Board of Surgery (ABS) vascular surgery in-training examination (VSITE), vascular qualifying examination (VQE), and vascular certifying examination (VCE) performance in a national cohort of vascular surgery trainees. BACKGROUND: Specialty board certification is an important indicator of physician competence. However, predicting future board certification examination performance during training continues to be challenging. METHODS: This is a national longitudinal cohort study examining relational and predictive associations between Accreditation Council for Graduate Medical Education (ACGME) Milestone ratings and performance on VSITE, VQE, and VCE for all vascular surgery trainees from 2015 to 2021. Predictive associations between milestone ratings and VSITE were conducted using cross-classified random-effects regression. Cross-classified random-effects logistic regression was used to identify predictive associations between milestone ratings and VQE and VCE. RESULTS: Milestone ratings were obtained for all residents and fellows(n=1,118) from 164 programs during the study period (from July 2015 to June 2021), including 145,959 total trainee assessments. Medical knowledge (MK) and patient care (PC) milestone ratings were strongly predictive of VSITE performance across all postgraduate years (PGYs) of training, with MK ratings demonstrating a slightly stronger predictive association overall (MK coefficient 17.26 to 35.76, ß = 0.15 to 0.23). All core competency ratings were predictive of VSITE performance in PGYs 4 and 5. PGY 5 MK was highly predictive of VQE performance [OR 4.73, (95% CI, 3.87-5.78), P <0.001]. PC subcompetencies were also highly predictive of VQE performance in the final year of training [OR 4.14, (95% CI, 3.17-5.41), P <0.001]. All other competencies were also significantly predictive of first-attempt VQE pass with ORs of 1.53 and higher. PGY 4 ICS ratings [OR 4.0, (95% CI, 3.06-5.21), P <0.001] emerged as the strongest predictor of VCE first-attempt pass. Again, all subcompetency ratings remained significant predictors of first-attempt pass on CE with ORs of 1.48 and higher. CONCLUSIONS: ACGME Milestone ratings are highly predictive of future VSITE performance, and first-attempt pass achievement on VQE and VCE in a national cohort of surgical trainees.

Principles of Justice, Equity, Diversity, and Inclusion in Health Care Distance Simulation Education: Consensus Building via the Nominal Group Technique.

PURPOSE: Health care distance simulation (HCDS) replicates professional encounters through an immersive experience overseen by experts and in which technological infrastructure enriches the learning activity. As HCDS has gained traction, so has the movement to provide inclusive and accessible simulation experiences for all participants. However, established guidelines for best practices in HCDS regarding justice, equity, diversity, and inclusion (JEDI) are lacking. This study aimed to generate consensus statements on JEDI principles in synchronous HCDS education using the nominal group technique (NGT). METHOD: Professionals with experience in HCDS education were invited to generate, record, discuss, and vote on ideas that they considered best practices for JEDI. This process was followed by a thematic analysis of the NGT discussion to provide a deeper understanding of the final consensus statements. An independent group of HCDS educators individually reviewed and recorded their agreement or disagreement with the consensus statements created by the NGT process. RESULTS: Eleven independent experts agreed on 6 key practices for JEDI in HCDS. Educators need to (1) be aware of JEDI principles, (2) be able to define and differentiate JEDI, (3) model JEDI in their environment, (4) have expertise and comfort facilitating conversations and debriefing around JEDI issues, (5) be advocates within their organizations to ensure equitable educational experiences, and (6) achieve JEDI without compromising educational objectives. Experts were divided on the approach to technology to ensure equitable learning experiences: some believed that the most basic technology accessible to all learners should be used, and some believed that the technology used should be determined by the competency of the students or faculty. CONCLUSIONS: Structural and institutional barriers in HCDS education persist despite agreement on key JEDI practices. Conclusive research is needed to guide the optimal policy in HCDS toward creating equitable learning experiences while bridging the digital divide.

Patient handoffs and multi-specialty trainee perspectives across an institution: informing recommendations for health systems and an expanded conceptual framework for handoffs.

BACKGROUND: Safe and effective physician-to-physician patient handoffs are integral to patient safety. Unfortunately, poor handoffs continue to be a major cause of medical errors. Developing a better understanding of challenges faced by health care providers is critical to address this continued patient safety threat. This study addresses the gap in the literature exploring broad, cross-specialty trainee perspectives around handoffs and provides a set of trainee-informed recommendations for both training programs and institutions. METHODS: Using a constructivist paradigm, the authors conducted a concurrent/embedded mixed method study to investigate trainees' experiences with patient handoffs across Stanford University Hospital, a large academic medical center. The authors designed and administered a survey instrument including Likert-style and open-ended questions to solicit information about trainee experiences from multiple specialties. The authors performed a thematic analysis of open-ended responses. RESULTS: 687/1138 (60.4%) of residents and fellows responded to the survey, representing 46 training programs and over 30 specialties. There was wide variability in handoff content and process, most notably code status not being consistently mentioned a third of the time for patients who were not full code. Supervision and feedback about handoffs were inconsistently provided. Trainees identified multiple health-systems level issues that complicated handoffs and suggested solutions to these threats. Our thematic analysis identified five important aspects of handoffs: (1) handoff elements, (2) health-systems-level factors, (3) impact of the handoff, (4) agency (duty), and (5) blame and shame. CONCLUSIONS: Health systems, interpersonal, and intrapersonal issues affect handoff communication. The authors propose an expanded theoretical framework for effective patient handoffs and provide a set of trainee-informed recommendations for training programs and sponsoring institutions. Cultural and health-systems issues must be prioritized and addressed, as an undercurrent of blame and shame permeates the clinical environment.

Impact of a Virtual Professional Development Coaching Program on the Professional Fulfillment and Well-Being of Women Surgery Residents: A Randomized Controlled Trial.

OBJECTIVE: Evaluate the effect of a virtual coaching program offered to women surgery residents in a surgical society. BACKGROUND: Randomized controlled experiments evaluating the effect of coaching on trainee well-being and burnout is lacking. METHODS: Women surgery residents in the Association of Women Surgeons were recruited to participate in a randomized controlled trial of the effects of a virtual coaching program on trainee well-being. Attending surgeons served as coaches after completing in-person training. Residents (n=237) were randomized to intervention (three 1:1 coaching sessions over 9 mo) or control (e-mailed wellness resources). Participants were surveyed at baseline and postintervention using validated measures of well-being, burnout, and resilience. Changes in outcome measures between presurvey and postsurvey were compared between study arms. RESULTS: Survey response rates were 56.9% (n=66) in the control group and 69.4% (n=84) in the intervention group ( P =0.05). The intervention group showed significant improvement in professional fulfillment ( P =0.021), burnout (0.026), work exhaustion (0.017), self-valuation (0.003), and well-being ( P =0.002); whereas the control group showed significant improvement in self-valuation ( P =0.015) and significant decline in resilience ( P =0.025). The intervention group had a significant improvement in well-being ( P =0.015) and intolerance of uncertainty ( P =0.015) compared to controls. CONCLUSIONS: Women surgery residents who participated in a remote coaching program offered by a surgical society demonstrated improvement in aspects of well-being relative to peers who did not receive coaching. Therefore, remote coaching offered by a professional society may be a useful component of initiatives directed at trainee well-being.

The Use of Learning Analytics to Enable Detection of Underperforming Trainees: An Analysis of National Vascular Surgery Trainee ACGME Milestones Assessment Data.

OBJECTIVE: This study aims to investigate at-risk scores of semiannual Accreditation Council for Graduate Medical Education (ACGME) Milestone ratings for vascular surgical trainees' final achievement of competency targets. SUMMARY BACKGROUND DATA: ACGME Milestones assessments have been collected since 2015 for Vascular Surgery. It is unclear whether milestone ratings throughout training predict achievement of recommended performance targets upon graduation. METHODS: National ACGME Milestones data were utilized for analyses. All trainees completing 2-year vascular surgery fellowships in June 2018 and 5-year integrated vascular surgery residencies in June 2019 were included. A generalized estimating equations model was used to obtain at-risk scores for each of the 31 subcompetencies by semiannual review periods, to estimate the probability of trainees achieving the recommended graduation target based on their previous ratings. RESULTS: A total of 122 vascular surgery fellows (VSFs) (95.3%) and 52 integrated vascular surgery residents (IVSRs) (100%) were included. VSFs and IVSRs did not achieve level 4.0 competency targets at a rate of 1.6% to 25.4% across subcompetencies, which was not significantly different between the 2 groups for any of the subcompetencies ( P = 0.161-0.999). Trainees were found to be at greater risk of not achieving competency targets when lower milestone ratings were assigned, and at later time-points in training. At a milestone rating of 2.5, with 1 year remaining before graduation, the at-risk score for not achieving the target level 4.0 milestone ranged from 2.9% to 77.9% for VSFs and 33.3% to 75.0% for IVSRs. CONCLUSION: The ACGME Milestones provide early diagnostic and predictive information for vascular surgery trainees' achievement of competence at completion of training.

The American College of Surgeons Entering Resident Readiness Assessment (ACS ERRA): Validity Evidence From a Three-Year National Study.

OBJECTIVE: To gather validity evidence supporting the use and interpretation of scores from the American College of Surgeons Entering Resident Readiness Assessment (ACS ERRA) Program. SUMMARY AND BACKGROUND DATA: ACS ERRA is an online formative assessment program developed to assess entering surgery residents' ability to make critical clinical decisions, and includes 12 clinical areas and 20 topics identified by a national panel of surgeon educators and residency program directors. METHODS: Data from 3 national testing administrations of ACS ERRA (2018-2020) were used to gather validity evidence regarding content, response process, internal structure (reliability), relations to other variables, and consequences. RESULTS: Over the 3 administrations, 1975 surgery residents participated from 125 distinct residency programs. Overall scores [Mean = 64% (SD = 7%)] remained consistent across the 3 years ( P = 0.670). There were no significant differences among resident characteristics (gender, age, international medical graduate status). The mean case discrimination index was 0.54 [SD = 0.15]. Kappa inter-rater reliability for scoring was 0.87; the overall test score reliability (G-coefficient) was 0.86 (Ф-coefficient = 0.83). Residents who completed residency readiness programs had higher ACS ERRA scores (66% versus 63%, Cohen's d = 0.23, P < 0.001). On average, 15% of decisions made (21/140 per test) involved potentially harmful actions. Variability in scores from graduating medical schools (7%) carried over twice as much weight than from matched residency programs (3%). CONCLUSIONS: ACS ERRA scores provide valuable information to entering surgery residents and surgery program directors to aid in development of individual and group learning plans.

The Impact of Simulation Based Training on the Fundamentals of Endoscopic Surgery Performance Examination.

OBJECTIVE: To determine if simulation training is required to pass the FES skills test and assess the relationship between simulation training, clinical training, and FES skills test performance. SUMMARY OF BACKGROUND DATA: The ABS began requiring completion of the Flexible Endoscopy Curriculum for all applicants beginning in 2018. The role of simulation-based training in FES skills test performance after this requirement has not been evaluated. METHODS: De-identified data from the initial FES skills tests after the Flexible Endoscopy Curriculum requirement was reviewed, and 731 unique participants with reported simulation experience demographics were identified. Self-reported data included sex, upper (UE) and lower (LE) endoscopy experience, and simulator training hours (SE). Final FES skills exam scores and pass/fail designations for each participant were reported by the FES program staff. RESULTS: There was a statistically discernible difference in mean FES total scores between those reporting no SE and more experienced groups ( P = 0.002), and between less and more experienced UE and LE groups ( P < 0.001). There was no statistically discernible difference in FES skills exam pass rates between SE groups ( P = 0.2), but there was a strong relationship between clinical experience (UE & LE) and pass rate ( P < 0.001). Finally, on logistic regression analysis, LE was a discernible predictor of passing [odds ratio (OR) = 1.4, 95% confidence interval (CI) 1.1-1.8, P = 0.02], while UE [odds ratio (OR) = 1, 95% CI 0.8-1.3, P = 0.9] and SE (OR = 1,95% CI 0.9- 1.3, P = 0.7) were not. CONCLUSIONS: There is no threat to the validity of the FES skills test from a need for simulation training to pass the FES skills test. Similarly, the amount of simulation practice is not predictive of passing, but can improve performance on certain FES tasks.

Predicting Performance at Graduation From Early ACGME Milestone Ratings: Longitudinal Learning Analytics in Professionalism and Communication in Vascular Surgery.

OBJECTIVE: Program directors in surgical disciplines need more tools from the ACGME to help them use Milestone ratings to improve trainees' performance. This is especially true in competencies that are notoriously difficult to measure, such as professionalism (PROF) and interpersonal and communication skills (ICS). It is now widely understood that skills in these two areas have direct impact on patient care outcomes. This study investigated the potential for generating early predictors of final Milestone ratings within the PROF and ICS competency categories. DESIGN: This retrospective cohort study utilized Milestone ratings from all ACGME-accredited vascular surgery training programs, covering residents and fellows who completed training in June 2019. The outcome measure studied was the rate of achieving the recommended graduation target of Milestone Level 4 (possible range: 1-5), while the predictors were the Milestone ratings attained at earlier stages of training. Predictive probability values (PPVs) were calculated for each of the 3 PROF and two ICS sub-competencies to estimate the probability of trainees not reaching the recommended graduation target based on their previous Milestone ratings. SETTING: All ACGME-accredited vascular surgery training programs within the United States. PARTICIPANTS: All trainees completing a 2 year vascular surgery fellowship (VSF) in June 2019 (n = 119) or a 5 year integrated vascular surgery residency (IVSR) in June 2019 (n = 52) were included in the analyses. RESULTS: The overall rate of failing to achieve the recommended graduation target across all PROF and ICS sub-competencies ranged from 7.7% to 21.8% of all trainees. For trainees with a Milestone rating at ≤ 2.5 with 1 year remaining in their training program, the predictive probability of not achieving the recommended graduation target ranged from 37.0% to 71.5% across sub-competencies, with the highest risks observed under PROF for "Administrative Tasks" (71.5%) and under ICS for "Communication with the Healthcare Team" (56.7%). CONCLUSIONS: As many as 1 in 4 vascular surgery trainees did not achieve the ACGME vascular surgery Milestones targets for graduation in at least one of the PROF and ICS sub-competencies. Biannual ACGME Milestone assessment ratings of PROF and ICS during early training can be used to predict achievement of competency targets at time of graduation. Early clues to problems in PROF and ICS enable programs to address potential deficits early in training to ensure competency in these essential non-technical skills prior to entering unsupervised practice.

When Pulmonologists Are Novice to Navigational Bronchoscopy, What Predicts Diagnostic Yield?

Predicting factors of diagnostic yield in electromagnetic navigation bronchoscopy (ENB) have been explored in a number of previous studies based on data from experienced operators. However, little is known about predicting factors when the procedure is carried out by operators in the beginning of their learning curve. We here aim to identify the role of operators' experience as well as lesion- and procedure characteristics on diagnostic yield of ENB procedures in the hands of novice ENB operators. Four operators from three centers without prior ENB experience were enrolled. The outcome of consecutive ENB procedures was assessed and classified as either diagnostic or non-diagnostic and predicting factors of diagnostic yield were assessed. A total of 215 procedures were assessed. A total of 122 (57%) of the ENB procedures resulted in diagnostic biopsies. Diagnostic ENB procedures were associated with a minor yet significant difference in tumor size compared to non-diagnostic/inconclusive ENB procedures (28 mm vs. 24 mm; p = 0.03). Diagnostic ENB procedures were associated with visible lesions at either fluoroscopy (p = 0.003) or radial endobronchial ultrasound (rEBUS), (p = 0.001). In the logistic regression model, lesion visibility on fluoroscopy, but none of operator experience, the presence of a bronchus sign, lesion size, or location nor visibility on rEBUS significantly impacted the diagnostic yield. In novice ENB operators, lesion visibility on fluoroscopy was the only factor found to increase the chance of obtaining a diagnostic sample.

Impact of Rater Training on Residents Technical Skill Assessments: A Randomized Trial.

OBJECTIVE: The ACS/APDS Resident Skills Curriculum's Objective Structured Assessment of Technical Skills (OSATS) consists of task-specific checklists and a global rating scale (GRS) completed by raters. Prior work demonstrated a need for rater training. This study evaluates the impact of a rater-training curriculum on scoring discrimination, consistency, and validity for handsewn bowel anastomosis (HBA) and vascular anastomosis (VA). DESIGN/ METHODS: A rater training video model was developed, which included a GRS orientation and anchoring performances representing the range of potential scores. Faculty raters were randomized to rater training or no rater training and were asked to score videos of resident HBA/VA. Consensus scores were assigned to each video using a modified Delphi process (Gold Score). Trained and untrained scores were analyzed for discrimination and score spread and compared to the Gold Score for relative agreement. RESULTS: Eight general and eight vascular surgery faculty were randomized to score 24 HBA/VA videos. Rater training increased rater discrimination and decreased rating scale shrinkage for both VA (mean trained score: 2.83, variance 1.88; mean untrained score: 3.1, variance 1.14, p = 0.007) and HBA (mean trained score: 3.52, variance 1.44; mean untrained score: 3.42, variance 0.96, p = 0.033). On validity analyses, a comparison between each rater group vs Gold Score revealed a moderate training impact for VA, trained κ=0.65 vs untrained κ=0.57 and no impact for HBA, R1 κ = 0.71 vs R2 κ = 0.73. CONCLUSION: A rater-training curriculum improved raters' ability to differentiate performance levels and use a wider range of the scoring scale. However, despite rater training, there was persistent disagreement between faculty GRS scores with no groups reaching the agreement threshold for formative assessment. If technical skill exams are incorporated into high stakes assessments, consensus ratings via a standard setting process are likely a more valid option than individual faculty ratings.

Needs assessment in head and neck surgical oncology training: A qualitative study of expert opinions.

BACKGROUND: Few studies have investigated the needs of head and neck surgery trainees and areas for improvement of fellowship programs. METHODS: We conducted a qualitative study by interviewing a nationally representative sample of program directors and national leaders in head and neck surgery. We asked about the current state and strengths of training; and areas for further improvement. All interviews were independently coded and analyzed by two reviewers. RESULTS: All experts (100%) believed that the current training provides a strong foundation and furthermore, a standardized curriculum is beneficial. Multidisciplinary training (80%), participation in tumor boards (75%), and a syllabus (60%) were the most frequently mentioned components. Most believed that a formal certification process would be beneficial (73%), though there was no consensus on the format. CONCLUSION: Experts in head and neck surgery are generally in favor of a standardized curriculum. Further discussions of a formal certification process might be warranted.

American College of Surgeons Objective Assessment of Skills in Surgery (ACS OASIS): A Formative Assessment of Junior Residents' Technical Skills.

OBJECTIVE: The objective assessment of technical skills of junior residents is essential in implementing competency-based training and providing specific feedback regarding areas for improvement. An innovative assessment that can be easily implemented by training programs nationwide has been developed by expert surgeon educators under the aegis of the American College of Surgeons (ACS) Division of Education. This assessment, ACS Objective Assessment of Skills in Surgery (ACS OASIS) uses eight stations to address technical skills important for junior residents within the domains of laparoscopic appendectomy, excision of lipoma, central line placement, laparoscopic cholecystectomy, trocar placement, exploratory laparotomy, repair of enterotomy, and tube thoracostomy. The purpose of this study was to implement ACS OASIS at a number of sites to study its psychometric rigor. DESIGN: The ACS OASIS was pre-piloted at two programs to establish feasibility and to gather information regarding implementation. Each skills station was 12 minutes long, and the faculty completed a checklist with 5 to 15 items, and a global assessment scale. The study was then repeated at three pilot sites and included 29 junior residents who were assessed by a total of 44 faculty. Psychometric data for the stations and checklists were collected and analyzed. SETTING: The pre-pilot sites were Geisinger and University of Tennessee Knoxville.Data were gathered from pilot sites that included Wellspan Health, Duke University, and University of California Los Angeles. RESULTS: The mean checklist score for all learners was 76% (IQR of 66%-85%). The average global rating was 3.36 on a 5-point scale with a standard deviation of 0.56. The overall cut score derived using the borderline group method was at 68% with 34% of performances requiring remediation. Using this criterion, the average number of stations that were completed by each learner without need for remediation was five.The station discrimination index ranged from 0.27 to 0.65 (all above the threshold of 0.25), demonstrating solid psychometric characteristics at the station level. The internal-consistency reliability was 0.76 with SEM of 5.8%. The inter-rater reliability (intraclass correlation) was high at 0.73 with general agreement of 79% between the two raters. The station discrimination was at 0.45 (range of 0.27 to 0.65) indicating a high level of differentiation between high and low performers. Using the generalizability theory, the G-coefficient reliability was at 0.72 with the reliability projection flattening after 8 stations. Overall, 75% to 82% the faculty and learners rated ACS OASIS as realistic and beneficial. CONCLUSIONS: ACS OASIS is a psychometrically sound technical skills assessment tool that can provide useful information for feedback to junior residents and support efforts to remediate gaps in performance.

Expert consensus on the conceptual alignment of Accreditation Council for Graduate Medical Education competencies with patient outcomes after common vascular surgical procedures.

BACKGROUND: The quality and effectiveness of vascular surgery education should be evaluated based on patient care outcomes. To investigate predictive associations between trainee performance and subsequent patient outcomes, a critical first step is to determine the conceptual alignment of educational competencies with clinical outcomes in practice. We sought to generate expert consensus on the conceptual alignment of the Accreditation Council for Graduate Medical Education (ACGME) Vascular Surgery subcompetencies with patient care outcomes across different Vascular Quality Initiative (VQI) registries. METHODS: A national panel of vascular surgeons with expertise in both clinical care and education were recruited to participate in a modified Delphi expert consensus building process to map ACGME Vascular Surgery subcompetencies (educational markers of resident performance) to VQI clinical modules (patient outcomes). A master list of items for rating was created, including the 31 ACGME Vascular Surgery subcompetencies and 8 VQI clinical registries (endovascular abdominal aortic aneurysm repair, open abdominal aortic aneurysm, thoracic endovascular aortic repair, carotid endarterectomy, carotid artery stent, infrainguinal, suprainguinal, and peripheral vascular intervention). These items were entered into an iterative Delphi process. Positive consensus was reached when 75% or more of the participants ranked an item as mandatory. Intraclass correlations (ICCs) were used to evaluate consistency between experts for each Delphi round. RESULTS: A total of 13 experts who contributed to the development of the Vascular Surgery Milestones participated; 12 experts (92%) participated in both rounds of the Delphi process. Two rounds of Delphi were conducted, as suggested by excellent expert agreement (round 1, ICC = 0.79 [95% confidence interval, 0.74-0.84]; round 2, ICC = 0.97 [95% confidence interval, 0.960-.98]). Using the predetermined consensus cutoff threshold, the Delphi process reduced the number of subcompetencies mapped to patient care outcomes from 31 to a range of 9 to 15 across the 8 VQI clinical registries. Practice-based learning and improvement, and professionalism subcompetencies were identified as less relevant to patient outcome variables captured by the VQI registries after the final round, and the only the systems-based practice subcompetency that was identified as relevant was radiation safety in two of the endovascular registries. CONCLUSIONS: A national panel of vascular surgeon experts reported a high degree of agreement on the relevance of ACGME subcompetencies to patient care outcomes as captured in the VQI clinical registry. Systems-based practice, practice-based learning and improvement, and professionalism competencies were identified as less relevant to patient outcomes after specific surgical procedures.

Resident Readiness for Senior Level Decision Making: Identifying the Domains for Formative Assessment and Feedback.

OBJECTIVE: To establish expert consensus regarding the domains and topics for senior surgery residents (PGY-4) to make critical decisions and assume senior-level responsibilities, and to develop the formative American College of Surgeons Senior Resident Readiness Assessment (ACS SRRA) Program. DESIGN: The American College of Surgeons (ACS) education leadership team conducted a focus group with surgical experts to identify the content for an assessment tool to evaluate senior residents' readiness for their increased levels of responsibility. After the focus group, national experts were recruited to develop consensus on the topics through three rounds of surveys using Delphi methodology. The Delphi participants rated topics using Likert-type scales and their comments were incorporated into subsequent rounds. Consensus was defined as ≥ 80% agreement with internal-consistency reliability (Cronbach's alpha) ≥ 0.8. In a stepwise fashion, topics that did not achieve consensus for inclusion were removed from subsequent survey rounds. SETTING: The surveys were administered via an online questionnaire. PARTICIPANTS: Twelve program directors and assistant program directors made up the focus group. The 39 Delphi participants represented seven different surgical subspecialties and were from diverse practice settings. The median length of experience in general surgery resident education was 20 years (IQR 14.3-30.0) with 64% of the experts being either current or past general surgery residency program directors. RESULTS: The response rate was 100% and Cronbach's alpha was ≥ 0.9 for each round. The Delphi participants contributed a large number of comments. Of the 201 topics that were evaluated initially, 120 topics in 25 core clinical areas were included to create the final domains of ACS SRRA. CONCLUSIONS: National consensus on the domain of the ACS SRRA has been achieved via the modified Delphi method among expert surgeon educators. ACS SRRA will identify clinical topics and areas in which each senior resident needs improvement and provide data to residents and residency programs to develop individualized learning plans. This would help in preparing the senior residents to assume their responsibilities and support their readiness for future fellowship training or surgical practice.

Medical Spanish Endocrinology Educational Module.

Introduction: Hispanic/Latinx patients experience health disparities in endocrine disease, such as higher rates of mortality due to diabetes mellitus, worse outcomes in the surgical treatment of thyroid cancer, and lack of knowledge about bone health and osteoporotic fracture prevention. Educational strategies to teach cultural and linguistic medical Spanish knowledge to medical students have the potential to improve Hispanic/Latinx endocrine health. Methods: We created an 8-hour medical Spanish endocrine module targeting language and cultural skills acquisition. Specifically, students practiced obtaining a past medical history, obtaining a medications history, providing and explaining a diagnosis, explaining discharge instructions, and discussing sociocultural aspects of endocrine health. We divided the module into four 2-hour sessions: (1) preclass self-study, (2) class period, (3) role-play/interviewing practice session, and (4) case/cultural activity assignment. Participants completed a pre- and postassessment to measure student confidence level and knowledge. Results: We implemented the module four times at one medical school, with 47 fourth-year medical students with intermediate or higher general Spanish skills. Confidence increased for all learners in the endocrine-focused interview and exam in Spanish. Heritage Spanish learners, who were exposed to Spanish at home as children, reported higher postmodule confidence in eliciting the past medical history of endocrine problems. Discussion: The medical Spanish endocrine module improved the communication skills of medical students with intermediate through advanced Spanish proficiency. Future study should evaluate learner clinical performance metrics and applications to other groups of learners, such as resident physicians or health professions involved in the care of patients with endocrine disease.

Transfer of virtual reality endoscopy training to live animal colonoscopy: a randomized control trial of proficiency vs. repetition-based training.

BACKGROUND: Low first-time pass rates of the Fundamentals of Endoscopic Surgery (FES) exam stimulated development of virtual reality (VR) simulation curricula for test preparation. This study evaluates the transfer of VR endoscopy training to live porcine endoscopy performance and compares the relative effectiveness of a proficiency-based vs repetition-based VR training curriculum. METHODS: Novice endoscopists completed pretesting including the FES manual skills examination and Global Assessment of GI Endoscopic Skills (GAGES) assessment of porcine upper and lower endoscopy. Participants were randomly assigned one of two curricula: proficiency-based or repetition-based. Following curriculum completion, participants post-tested via repeat FES examination and GAGES porcine endoscopy assessments. The two cohorts pre-to-post-test differences were compared using ANCOVA. RESULTS: Twenty-two residents completed the curricula. There were no differences in demographics or clinical endoscopy experience between the groups. The repetition group spent significantly more time on the simulator (repetition: 242.2 min, SD 48.6) compared to the proficiency group (proficiency: 170.0 min, SD 66.3; p = 0.013). There was a significant improvement in porcine endoscopy (pre: 10.6, SD 2.8, post: 16.6, SD 3.4; p < 0.001) and colonoscopy (pre: 10.4, SD 2.7, post: 16.4, SD 4.2; p < 0.001) GAGES scores as well as FES manual skills performance (pre: 270.9, SD 105.5, post: 477.4, SD 68.9; p < 0.001) for the total cohort. There was no difference in post-test GAGES performance or FES manual skills exam performance between the two groups. Both the proficiency and repetition group had a 100% pass rate on the FES skills exam following VR curriculum completion. CONCLUSION: A VR endoscopy curriculum translates to improved performance in upper and lower endoscopy in a live animal model. VR curricula type did not affect FES manual skills examination or live colonoscopy outcomes; however, a proficiency curriculum is less time-consuming and can provide a structured approach to prepare for both the FES exam and clinical endoscopy.

Colonoscope retraction technique and predicting adenoma detection rate: a multicenter study.

BACKGROUND AND AIMS: The success of preventing colorectal cancer relies on the expertise of the colonoscopists. Studies suggest that the retraction technique is a powerful indicator of expertise in distinguishing endoscopists with various adenoma detection rates (ADRs). We aimed to develop a retraction technique score and explore the correlation between endoscopists' retraction technique and their ADRs. METHODS: In a prospective, multicenter study, 8 colonoscopist nurses and physicians with various ADRs were included. Data from patients admitted for a colonoscopy, as part of the Danish nationwide screening program, were gathered directly from the Olympus ScopeGuide system (UPD-3; Olympus Optical, Tokyo, Japan) providing XYZ-coordinates from the coils along the length of the colonoscope. Motor skill measures were developed based on tip retraction, retraction efficiency, and retraction distance. The principal component analysis was used to study the association among the 3 measures and the historical ADR to create a combined score, the colonoscopy retraction score (CoRS). RESULTS: Three hundred thirty-three recordings were analyzed. We demonstrated a significant and strong correlation between CoRS and ADR (.90, P < .01). Conversely, withdrawal time did not correlate significantly with ADR (.33, P = .42). In procedures without polypectomies or biopsy sampling, a significant and strong correlation was found between CoRS and ADR (.88, P < .01) and between withdrawal time and ADR (.75, P = .03). CONCLUSIONS: This study presents a novel, real-time computerized and unbiased assessment tool for colonoscopy withdrawal. CoRS strongly correlated with ADR with and without therapeutic interventions during withdrawal and could be used to ensure quality instead of minimal withdrawal time. (Clinical trial registration number: NCT03587935.).