Evaluating the Validity of National Multiassessment System in Postgraduate Surgical Training: A Retrospective Cohort Study.

OBJECTIVE: The objective of this study was to evaluate the validity evidence supporting the use and interpretation of a multifaceted assessment system in the early years of surgical training. DESIGN: This was a national retrospective cohort study analyzing the validity and reliability of an assessment process for surgical residents over a 2-year period. Data from all elements of the assessment process was evaluated using Messick's unified validity framework. Assessments were categorized as Workplace-based, Structured assessment performed in the academic center and Multiple Mini Interview. SETTING: Our Institution is a health sciences university and the body responsible for the training and certification of all surgeons in our national program. Residents on the Core Surgical Training program undergo multiple assessments over the first 2 years of postgraduate training, both in the workplace and the academic training center, which inform their progression into higher surgical training in their chosen specialty. PARTICIPANTS: Data was collected from 2 cohorts of the entire population of postgraduate trainees nationally (N = 114). RESULTS: Best practice standards for educational testing aligned with the results supporting the use of this assessment process. Findings indicate a robust assessment system, demonstrating validity evidence in content, response process, internal structure, relations to other variables, and consequences. Composite score reliability of the assessment was 0.89 which demonstrates a highly reliable process. Correlation between workplace-based assessments and standardized tests performed in the simulation setting was also very high (0.93). CONCLUSIONS: The Core Training Assessment System (CTAS) provides a psychometrically rigorous system to measure trainee competence during the initial years of training. Residency programs of all sizes can replicate the methods described here to demonstrate the validity of their assessment processes, thereby being able to stand over decisions on surgical competency.

Flexible endoscopic intubation in emergency medicine: A mixed-methods needs assessment.

Objectives: This needs assessment aimed to improve understanding of flexible endoscopic intubation training and practice in emergency medicine (EM), providing insights to educators and practice leaders seeking to improve education and practices. Methods: We conducted a multicenter, mixed-methods needs assessment of emergency physicians (EPs) incorporating focus groups and a survey. Focus groups comprised community EPs, academic EPs, and resident EPs. We analyzed focus group transcripts using grounded theory, qualitatively describing EM endoscopic intubation. The qualitative analysis shaped our survey instrument, which we deployed in cross-sectional fashion. We report survey data with descriptive statistics. Results: Focus groups with 13 EPs identified three themes: indications for use of endoscopic intubation, factors impacting a physician's decision to endoscopically intubate, and attaining and maintaining endoscopic intubation competency. Of 257 surveyed EPs (33% response rate), 79% had received endoscopic intubation training during residency, though 82% had performed this procedure 10 or fewer times in their career. Despite 97% acknowledging the necessity of competency, only 23% felt highly confident in their ability to perform endoscopic intubation. Participants (93%) reported scarce opportunities to perform the procedure and identified factors believed to facilitate competency acquisition and maintenance, including opportunities to perform endoscopic intubation in practice (98%), local champions (93%), and performing nasopharyngoscopy (87%). Conclusions: While most EPs acknowledged the importance of competency in endoscopic intubation, they reported scarce procedural opportunities and commonly expressed low confidence. Further research is needed on this topic, and we propose avenues to enhance education and practices related to endoscopic intubation. These include development of robust procedural curricula, support of local champions, and incorporating nasopharyngoscopy into EM practice.

Accreditation Council for Graduate Medical Education Milestone Training Ratings and Surgeons' Early Outcomes.

Importance: National data on the development of competence during training have been reported using the Accreditation Council for Graduate Medical Education (ACGME) Milestones system. It is now possible to consider longitudinal analyses that link Milestone ratings during training to patient outcomes data of recent graduates. Objective: To evaluate the association of in-training ACGME Milestone ratings in a surgical specialty with subsequent complication rates following a commonly performed operation, endovascular aortic aneurysm repair (EVAR). Design, Setting, and Participants: This study of patient outcomes followed EVAR in the Vascular Quality Initiative (VQI) registry (4213 admissions from 208 hospitals treated by 327 surgeons). All surgeons included in this study graduated from ACGME-accredited training programs from 2015 through 2019 and had Milestone ratings 6 months prior to graduation. Data were analyzed from December 1, 2021, through September 15, 2023. Because Milestone ratings can vary with program, they were corrected for program effect using a deviation score from the program mean. Exposure: Milestone ratings assigned to individual trainees 6 months prior to graduation, based on judgments of surgical competence. Main Outcomes and Measures: Surgical complications following EVAR for patients treated by recent graduates during the index hospitalization, obtained using the nationwide Society for Vascular Surgery Patient Safety Organization's VQI registry, which includes 929 participating centers in 49 US states. Results: The study included outcomes for 4213 patients (mean [SD] age, 73.25 [8.74] years; 3379 male participants [80.2%]). Postoperative complications included 9.5% major (400 of 4213 cases) and 30.2% minor (1274 of 4213 cases) complications. After adjusting for patient risk factors and site of training, a significant association was identified between individual Milestone ratings of surgical trainees and major complications in early surgical practice in programs with lower mean Milestone ratings (odds ratio, 0.50; 95% CI; 0.27-0.95). Conclusions and Relevance: In this study, Milestone assessments of surgical trainees were associated with subsequent clinical outcomes in their early career. Although these findings represent one surgical specialty, they suggest Milestone ratings can be used in any specialty to identify trainees at risk for future adverse patient outcomes when applying the same theory and methodology. Milestones data should inform data-driven educational interventions and trainee remediation to optimize future patient outcomes.

Assessing methodological quality in dental education research using MERSQI: Analysis of publications from two journals.

PURPOSE: The Medical Education Research Study Quality Instrument (MERSQI) has been used frequently to assess the methodological quality of medical education but not for dental education. The present study aimed to assess the methodological quality using MERSQI scores of articles published in the Journal of Dental Education (JDE) and the European Journal of Dental Education (EJDE). METHODS: A cross-sectional assessment of the quality of manuscripts published in 2012, 2017, and 2022 JDE and EJDE was conducted. MERSQI data, numbers of authors, first and corresponding author degrees, geographic origins, and funding information were also extracted for each included study. Descriptive and analytical statistics were conducted, and significance level was set at α < 0.05. RESULTS: Four hundred ninety-five articles met the inclusion criteria. The most common study design was a single-group cross-sectional or single-group posttest and conducted in one institution for all studied years. In all journals and years, studies were assessed mainly by participants. The study outcome was mostly satisfaction, attitudes, perceptions, opinions, and general facts. The total mean MERSQI score for each journal and year varied. Year and geographic origin significantly affected the total MERSQI score. Papers originating from Asia had the highest score, followed by South America, Europe, North America, Oceania, and Africa. CONCLUSION: MERSQI score is applicable to the assessment of the methodological quality of dental educational research. The MERSQI score for most of the domains was similar for both journals. The MERSQI score was affected by publication years and geographic origins.

Competencies of undergraduate physiotherapy education: A scoping review.

Background: In recent years, the need for competency-based medical education has been emphasised. Each country needs a defined set of physiotherapy competencies from the associations and governing bodies. Objectives: Our review aimed to map competencies of undergraduate physiotherapy education and propose a context-specific competency framework for Namibia. Method: This scoping review was conducted following the Joanna Briggs Institute framework and was reported using the Preferred Reporting for Systematic Reviews and Meta-analysis Extension for Scoping Reviews. Qualitative direct content analysis utilising the five main competency domains from the WHO Rehabilitation Competency Framework was adapted. Results: Five main competency domains were proposed: practice, professional growth and involvement, learning and development, management and leadership, and research. Nineteen potential competencies were identified, and each competency has a set of knowledge and skills activities that is expected of each student. Conclusion: The proposed competencies still need to undergo expert consensus and content validation before they can be adopted and implemented in Namibia. Future studies can explore the perspectives and experiences of the faculty, students and clinicians on the current status of competency-based education of undergraduate physiotherapy programme in Namibia. Similarly, future studies can focus on possible assessment strategies that can be used for each competency and an evaluation framework for assessing milestones in student competencies from entry into clinical education to graduation. Clinical implications: The review proposed a context-specific competency framework for Namibia with a set of knowledge and skills activities that is expected of each student. The faculty can adopt these competencies and improve on their competency-based physiotherapy education.

The Inconspicuous Learner Handover: An Exploratory Study of U.S. Emergency Medicine Program Directors' Perceptions of Learner Handovers from Medical School to Residency.

Phenomenon: Central to competency-based medical education is the need for a seamless developmental continuum of training and practice. Trainees currently experience significant discontinuity in the transition from undergraduate (UME) to graduate medical education (GME). The learner handover is intended to smooth this transition, but little is known about how well this is working from the GME perspective. In an attempt to gather preliminary evidence, this study explores U.S. program directors (PDs) perspective of the learner handover from UME to GME. Approach: Using exploratory qualitative methodology, we conducted semi-structured interviews with 12 Emergency Medicine PDs within the U.S. from October to November, 2020. We asked participants to describe their current perception of the learner handover from UME to GME. Then we performed thematic analysis using an inductive approach. Findings: We identified two main themes: The inconspicuous learner handover and barrier to creating a successful UME to GME learner handover. PDs described the current state of the learner handover as "nonexistent," yet acknowledged that information is transmitted from UME to GME. Participants also highlighted key challenges preventing a successful learner handover from UME to GME. These included: conflicting expectations, issues of trust and transparency, and a dearth of assessment data to actually hand over. Insights: PDs highlight the inconspicuous nature of learner handovers, suggesting that assessment information is not shared in the way it should be in the transition from UME to GME. Challenges with the learner handover demonstrate a lack of trust, transparency, and explicit communication between UME and GME. Our findings can inform how national organizations establish a unified approach to transmitting growth-oriented assessment data and formalize transparent learner handovers from UME to GME.

The rapid growth and expansion of Master of health professions education (MHPE) programs: A mixed-methods study of international curricular trends and guidelines for programs.

PURPOSE: This study explores international trends and standards of Master's degree programs through a comprehensive environmental scan and focus group interviews to understand curricular structure, content, program director expectations, educational context, and future directions. METHOD: Authors conducted a two-phase mixed-methods sequential explanatory design to conduct the environmental scan (phase 1), and subsequently conducting focus groups (phase 2) with program directors. A population list of Master's programs was used to generate a sampling frame, considering the geographic region (continent) and institution type (university, organization, public institution). Qualitative data were coded to analyze the breadth and depth of courses. Three one-hour virtual focus group interviews were conducted with ten program directors. RESULTS: The population list of 159 Masters programs worldwide was used to create a sample for analysis in the environmental scan (n = 46 Masters programs), representing programs from North America, Europe, Australia, and South Africa. Most programs (39%) delivered their courses online, with 20% exclusively offering an in-person program. Focus group participants indicated expectations of graduates, context in which they learn, as well as future directions for improving health professions education graduate programs. CONCLUSION: Program directors should consider programmatic aims, localized needs, and quality/standard of the program in designing Masters programs, with individualized growth opportunities for learners.

An Analysis of Written and Numeric Scores in End-of-Rotation Forms from Three Residency Programs.

Introduction: End-of-Rotation Forms (EORFs) assess resident progress in graduate medical education and are a major component of Clinical Competency Committee (CCC) discussion. Single-institution studies suggest EORFs can detect deficiencies, but both grades and comments skew positive. In this study, we sought to determine whether the EORFs from three programs, including multiple specialties and institutions, produced useful information for residents, program directors, and CCCs. Methods: Evaluations from three programs were included (Program 1, Institution A, Internal Medicine: n = 38; Program 2, Institution A, Anesthesia: n = 9; Program 3, Institution B, Anesthesia: n = 11). Two independent researchers coded each written comment for relevance (specificity and actionability) and orientation (praise or critical) using a standardized rubric. Numeric scores were analyzed using descriptive statistics. Results: 4869 evaluations were collected from the programs. Of the 77,434 discrete numeric scores, 691 (0.89%) were considered "below expected level." 71.2% (2683/3767) of the total written comments were scored as irrelevant, while 3217 (85.4%) of total comments were scored positive and 550 (14.6%) were critical. When combined, 63.2% (n = 2379) of comments were scored positive and irrelevant while 6.5% (n = 246) were scored critical and relevant. Discussion: <1% of comments indicated below average performance; >70% of comments scored irrelevant. Critical, relevant comments were least frequently observed, consistent across all 3 programs. The low rate of constructive feedback and the high rate of irrelevant comments are inadequate for a CCC to make informed decisions. The consistency of these findings across programs, specialties, and institutions suggests both local and systemic changes should be considered.

Surgical skills training for practicing surgeons founded on established educational theories and frameworks.

Following completion of structured training in residency or fellowship programs, surgeons need to acquire new skills throughout their careers to address changes in their surgical practices, and to perform new procedures and use new technologies as these are developed. The opportunities available to surgeons to acquire the new skills, safely introduce these skills into surgical practice, and then maintain the skills, vary greatly and may not fully support a surgeon's needs. The article shares background information relating to the vexing problems, and describes established educational theories and frameworks as well as specific frameworks relating to surgical skills training and verification that may be used in designing and implementing comprehensive and impactful surgical skills training programs aimed at practicing surgeons. A number of practical strategies based on these theories and frameworks are highlighted. National professional organizations and academic medical centers need to play a critical role in pursuing such efforts, which will remain pivotal in supporting the professional work of surgeons and in providing optimal surgical care well into the future.

Evaluating the Quality of the Core Entrustable Professional Activities for New Pharmacy Graduates.

This study aimed to evaluate the quality of the American Association of Colleges of Pharmacy Core Entrustable Professional Activities (Core EPAs) for New Pharmacy Graduates according to standards outlined in competency-based education literature utilizing the Queen's EPA Quality (EQual) rubric. A cohort of pharmacists with EPA expertise rated Core EPA quality with the EQual rubric and provided recommendations for revisions. A generalizability study determined the reliability of the EQual ratings with pharmacist users. Nine pharmacists responded (4.4%). Most EPAs (9/15) did not reach the overall cut-off score, indicating low quality. EPAs 1 through 5 and EPA 14 (fulfill a medication order) were deemed high quality. EPA 12 (use evidence-based information to advance patient care) scored the lowest at 3.47 (SEM 0.29). EPA 14 scored the highest at 4.60 (SEM 0.14). EPA 15 (create a written plan for continuous professional development) was the only EPA to fail to reach the cut-off across all EQual domains. EPAs in the Patient Care Provider Domain received significantly higher ratings than other EPAs. On average, three respondents recommended revision for each. Most comments aligned with the EPA's EQual rubric performance. The generalizability study analysis revealed excellent reliability (G = 0.80). Determining EPA quality utilizing objective measurement tools should drive EPA development and revisions to more accurately reflect the roles, responsibilities, and expectations of pharmacists on the healthcare team.

Patient handoffs and multi-specialty trainee perspectives across an institution: informing recommendations for health systems and an expanded conceptual framework for handoffs.

BACKGROUND: Safe and effective physician-to-physician patient handoffs are integral to patient safety. Unfortunately, poor handoffs continue to be a major cause of medical errors. Developing a better understanding of challenges faced by health care providers is critical to address this continued patient safety threat. This study addresses the gap in the literature exploring broad, cross-specialty trainee perspectives around handoffs and provides a set of trainee-informed recommendations for both training programs and institutions. METHODS: Using a constructivist paradigm, the authors conducted a concurrent/embedded mixed method study to investigate trainees' experiences with patient handoffs across Stanford University Hospital, a large academic medical center. The authors designed and administered a survey instrument including Likert-style and open-ended questions to solicit information about trainee experiences from multiple specialties. The authors performed a thematic analysis of open-ended responses. RESULTS: 687/1138 (60.4%) of residents and fellows responded to the survey, representing 46 training programs and over 30 specialties. There was wide variability in handoff content and process, most notably code status not being consistently mentioned a third of the time for patients who were not full code. Supervision and feedback about handoffs were inconsistently provided. Trainees identified multiple health-systems level issues that complicated handoffs and suggested solutions to these threats. Our thematic analysis identified five important aspects of handoffs: (1) handoff elements, (2) health-systems-level factors, (3) impact of the handoff, (4) agency (duty), and (5) blame and shame. CONCLUSIONS: Health systems, interpersonal, and intrapersonal issues affect handoff communication. The authors propose an expanded theoretical framework for effective patient handoffs and provide a set of trainee-informed recommendations for training programs and sponsoring institutions. Cultural and health-systems issues must be prioritized and addressed, as an undercurrent of blame and shame permeates the clinical environment.

Tuition Costs of Master's of Health Professions Education Programs: A Cross-Sectional Analysis.

PURPOSE: The increasing professionalization of medical education during the past 2 decades has ushered in an era in which formal degrees, particularly master's of health professions education (MHPE), have become important for career advancement in medical education. Although tuition costs can pose a substantial barrier for many seeking advanced degrees in health professions education, data on tuition associated with these programs are lacking. This study examines the accessibility of pertinent cost-related information available to prospective students and the variability of costs among programs worldwide. METHOD: The authors conducted an Internet-based, cross-sectional study, augmented with emails and direct contact with educators, to extract tuition-related data for MHPE programs between March 29, 2022, and September 20, 2022. Costs were converted to an annual total within each jurisdiction's currency and converted to U.S. dollars on August 18, 2022. RESULTS: Of the 121 programs included in the final cost analysis, only 56 had publicly available cost information. Excluding programs free to local students, the mean (SD) total tuition cost was $19,169 ($16,649), and the median (interquartile range) cost was $13,784 ($9,401- $22,650) (n = 109). North America had the highest mean (SD) tuition for local students ($26,751 [$22,538]), followed by Australia and New Zealand ($19,778 [$10,514]) and Europe ($14,872 [$7,731]), whereas Africa had the lowest ($2,598 [$1,650]). The region with the highest mean (SD) tuition for international students was North America ($38,217 [$19,500]), followed by Australia and New Zealand ($36,891 [$10,397]) and Europe ($22,677 [$10,010]), whereas Africa had the lowest ($3,237 [$1,189]). CONCLUSIONS: There is substantial variability in the geographic distribution of MHPE programs and marked differences in tuition. Incomplete program websites and limited responsiveness from many programs contributed to a lack of transparency regarding potential financial implications. Greater efforts are necessary to ensure equitable access to health professions education.

Exploring knowledge, attitudes and perceived myths towards foetal programming and factors affecting it among the public and health professionals in Pakistan: a needs assessment study.

OBJECTIVE: To explore knowledge, attitudes and practices of laypersons and health professionals towards foetal programming, and factors affecting it. METHODS: The mixed methods study was conducted at the Aga Khan University, Karachi, from January 20, 2021 to May 13, 2022, and comprised adults of either gender with access to social media platforms. Data was collected using an online survey questionnaire in English and Urdu developed to capture responses from a diverse pool of participants. The survey tool was circulated through WhatsApp, Facebook and Instagram. Two focus group discussions were conducted; one with laypersons in group A and the other with health and allied professionals in group B. Data was analysed using SPSS 21, while data related to focus group discussions was subjected to thematic analysis. RESULTS: Of the 358 participants, 173(48.3%) were in group A and 185(51.7%) were in group B. There were 34(18.4) subjects in group A and 27(15.6) in group B who had knowledge of foetal programming (p>0.05). Only factors related to father's health and dietary elements on the foetus were significantly different between the groups (p<0.05). Thematic analysis led to the formation of 3 overarching themes: parent's lifestyle, comorbidity and diet on foetal health; myths and cultural beliefs regarding foetal development; and the need for training / awareness for practitioners and community. CONCLUSIONS: Lack of knowledge and misinformation about foetal programming and development was common among health professionals and laypersons.

"My Assessments Are Biased!" Measurement and Sociocultural Approaches to Achieve Fairness in Assessment in Medical Education.

Assessing learners is foundational to their training and developmental growth throughout the medical education continuum. However, growing evidence shows the prevalence and impact of harmful bias in assessments in medical education, accelerating the urgency to identify solutions. Assessment bias presents a critical problem for all stages of learning and the broader educational system. Bias poses significant challenges to learners, disrupts the learning environment, and threatens the pathway and transition of learners into health professionals. While the topic of assessment bias has been examined within the context of measurement literature, limited guidance and solutions exist for learners in medical education, particularly in the clinical environment. This article presents an overview of assessment bias, focusing on clinical learners. A definition of bias and its manifestations in assessments are presented. Consequences of assessment bias are discussed within the contexts of validity and fairness and their impact on learners, patients/caregivers, and the broader field of medicine. Messick's unified validity framework is used to contextualize assessment bias; in addition, perspectives from sociocultural contexts are incorporated into the discussion to elaborate the nuanced implications in the clinical training environment. Discussions of these topics are conceptualized within the literature and the interventions used to date. The article concludes with practical recommendations to overcome bias and to develop an ideal assessment system. Recommendations address articulating values to guide assessment, designing assessment to foster learning and outcomes, attending to assessment procedures, promoting continuous quality improvement of assessment, and fostering equitable learning and assessment environments.

The Role of Religious Culture in Medical Professionalism in a Muslim Arab Society.

Purpose: Calls have been made to integrate concepts and practices derived from Muslim culture into medical professionalism in Muslim societies. Little is known about how these religious cultural concepts (RCCs) influence medical practice and education. This study explored the influence of RCCs on medical professionalism in Saudi Arabia. Methods: This was a qualitative study that implemented a constructivist, grounded theory approach. Semi-structured interviews about RCCs and medical professionalism were conducted with 15 Saudi physicians at a single academic medical center. Purposive sampling was used to recruit participants of different genders, generations, and specialties. Data collection and analysis were iterative. A theoretical framework was formulated. Results: Key findings: (i) the role of RCCs in medical professionalism is perceived to be constantly evolving due to the evolution of societal interpretations of RCCs; (ii) participants described applying two standards to judge what is professional: a medical standard and a religious cultural standard. Participants shifted between these two standards variably and non-linearly. This variable shifting altered the values shaping medical professionalism, at times unpredictably. Discussion: Academic Saudi physicians argued against assuming a stable traditional interpretation of RCCs, emphasized the evolving contribution of RCCs to medical professionalism, and indicated that the process of merging religious cultural and medical standards in medical practice is variable and may alter medical practice values. Therefore, these physicians perceived RCCs to be useful as supplements to but not as a backbone for medical professionalism. Careful consideration of the potential impact of RCCs on the values of medical professionalism is warranted.

Comparative evaluation of weighted mean and composite score for combining assessments.

OBJECTIVES: Approaches for combining scores have been based on weighted mean (WM) without consideration for psychometric characteristics of each individual assessments. This study evaluates consequences of WM and composite score (CS) approach. METHODS: Data from two longitudinal cohorts (n = 219) were utilized for performance in three Operative Dentistry courses as basis to compare the two score-combining methods. Four assessments (two written and two practical exams) from each course were combined using WM and CS approaches. WM scores were calculated by multiplying the score by its weight and summing across assessments. The CS approach follow a modification of the Kane and Case method, by standardizing scores, taking into account the reliability and associations between each assessment score. t-Tests and Pearson's correlation were used to evaluate the consequences of the WM and CS approaches. In addition, changes in each student's rank across WM and CS were determined. RESULTS: Combining scores using CS method produced lower scores and higher percentage failure in all courses compared to WM. Students ranks were changed significantly when CS was used with only 15% of the cohorts retained their ranks. CONCLUSIONS: CS produced a composite that is correlated with WM but still being substantively different providing meaningful and psychometrically rigorous information.

Learning transfusion medicine through scoring objective structured clinical examination.

BACKGROUND: Transfusion medicine education at the undergraduate level is typically limited in duration. In view of limitations of traditional teaching methods, we explore effectiveness of scoring (Objective Structured Clinical Examination) OSCE as an educational method. MATERIALS AND METHODS: The study was of a randomized interventional three group pre-test-post-test design. Participants were undergraduate medical students in their two final years. The intervention was watching and scoring 2 videotaped OSCE stations about obtaining consent for blood transfusions and assessing the ability to explain risks, benefits, and alternatives of blood transfusion. Participants were asked to assess the performance of the videotaped actor using checklists. Participants were randomized to watch and evaluate one set of videos at either the highest, intermediate, or lowest compliance with required consent elements. Main measure was performance in a knowledge test containing multiple-choice and true/false questions. This was given before (pre-test), immediately after the intervention (post-test 1), and after 8 weeks (post-test 2). Student perceptions regarding the intervention was assessed immediately after the session. RESULTS: Sixty-nine students were randomized. Post-test 1 results (mean 16.52, SD 1.88) were significantly greater than pre-test results (mean 11.83, SD 2.13) by group and across all groups (p < 0.001). Post-test 2 results for the complete cohort showed maintenance of significant improvement in comparison with the pre-test. The majority of students agreed that learning through scoring OSCE was an effective educational experience. CONCLUSIONS: In the undergraduate medical setting, scoring OSCE stations may enhance learning of content discussed and evaluated in the stations.

Implementation of the AAMC's Holistic Review Model for Psychiatry Resident Recruitment.

Introduction: In psychiatry, several converging factors are impacting the recruitment of residents: the increased competitiveness of the specialty, the national trend to take active steps to improve diversity and inclusion, and the decision from USMLE to change Step 1 to a pass/fail result. Methods: We developed a workshop for psychiatry residency program directors to meet these challenges and transition into using a holistic review model during recruitment. The workshop included (1) a didactic session providing background on the AAMC holistic review model; (2) a small-group exercise to determine and prioritize experiences, attributes, competencies, and metrics (EACMs) aligned with the program's mission and aims; (3) a review of the rankings from the previous exercise, selection of two "very important" criteria for each of the four domains of the EACM model, and operationalization of these criteria based on the recruitment process; and (4) a discussion focused on application of program criteria with example applicants. Results: The holistic review workshop was conducted at the American Association of Psychiatry Residency Directors conference in 2021 with 48 self-selected attendees. Following the workshop, 74% of attendees reported a likelihood of implementing holistic applications during their next application cycle, 78% were able to leave with at least one actionable item, 100% thought that the session was interactive, and 78% felt that the session met their expectations. Discussion: Implementing a holistic review for psychiatry residency recruitment can assist programs in responding to the rapidly changing landscape and achieve aims for improving diversity and inclusion.

Development of a United States Radiation Oncology Curricular Framework: A Stakeholder Delphi Consensus.

PURPOSE: A United States (US) radiation oncology curriculum, developed using best practices for curriculum inquiry, is needed to guide residency education and qualifying examinations. Competency-based training, including entrustable professional activities (EPAs), provides an outcomes-based approach to modern graduate medical education. This study aimed to define US radiation oncology EPAs and curricular content domains using a deliberative process with input from multiple stakeholder groups. METHODS AND MATERIALS: The Radiation Oncology Education Collaborative Study Group Core Curriculum Project Leadership Committee developed initial content domains and EPAs. Following recruitment of stakeholders, a Delphi process was used to achieve consensus. In the first round, content domains and EPAs were reviewed for inclusion and exclusion, clarity, time allocation (content domains), and level of training (EPAs). Participants submitted additional content domains and EPAs for consideration. Any content domains or EPAs 1 standard deviation below the median for inclusion and exclusion underwent Leadership Committee review. All participants completing the first Delphi round were invited to the second round. Percent curriculum time allocated for content domains and a single subdomain were finalized. New EPAs or EPAs undergoing major revisions were reviewed. RESULTS: A total of 186 participants representing diverse stakeholder groups participated. One hundred fourteen completed the first Delphi round (61.3%). Of 114 invited, 77 participants completed the second round of the Delphi process (67.5%). Overall, 6 of 9 content domains met consensus, 1 content domain was removed, and 2 content domains were combined. Four subdomains of a single content domain were reviewed and met consensus. Consensus on percent time allocated per content domain and subdomain was reached. Of 55 initial EPAs, 52 final EPAs met consensus. CONCLUSIONS: Deliberative curriculum inquiry was successfully used to develop a consensus on US radiation oncology content domains and EPAs. These data can guide the allocation of educational time in training programs, help inform weighting for qualifying examinations, and help guide clinical training and resident assessment.