Evaluating a Situational Judgment Test for Use in Medical School Admissions: Two Years of AAMC PREview Exam Administration Data.

PURPOSE: To examine the relationship between the Association of American Medical Colleges (AAMC) Professional Readiness Exam (PREview) scores and other admissions data, group differences in mean PREview scores, and whether adding a new assessment tool affected the volume and composition of applicant pools. METHOD: Data from the 2020 and 2021 PREview exam administrations were analyzed. Two U.S. schools participated in the PREview pilot in 2020 and 6 U.S. schools participated in 2021. PREview scores were paired with data from the American Medical College Application Service (undergraduate grade point averages [GPAs], Medical College Admission Test [MCAT] scores, race, and ethnicity) and participating schools (interview ratings). RESULTS: Data included 19,525 PREview scores from 18,549 unique PREview examinees. Correlations between PREview scores and undergraduate GPAs ( r = .16) and MCAT scores ( r = .29) were small and positive. Correlations between PREview scores and interview ratings were also small and positive, ranging between .09 and .14 after correcting for range restriction. Small group differences in mean PREview scores were observed between White and Black or African American and White and Hispanic, Latino, or of Spanish origin examinees. The addition of the PREview exam did not substantially change the volume or composition of participating schools' applicant pools. CONCLUSIONS: Results suggest the PREview exam measures knowledge of competencies that are distinct from those measured by other measures used in medical school admissions. Observed group differences were smaller than group differences observed with traditional academic assessments and evaluations. The addition of the PREview exam did not substantially change the overall volume of applications or the proportions of out-of-state, underrepresented in medicine, or lower socioeconomic status applicants. While more research is needed, these results suggest the PREview exam may provide unique information to the admissions process without adversely affecting applicant pools.

Promising Practices in US Sponsoring Institutions to Advance Diversity, Equity, and Inclusion in Graduate Medical Education.

Background Best practices to improve diversity, equity, and inclusion (DEI) in the biomedical workforce remain poorly understood. The Accreditation Council for Graduate Medical Education launched the Barbara Ross-Lee, DO, Diversity, Equity, and Inclusion award for sponsoring institutions to celebrate efforts to improve DEI in graduate medical education (GME). Objective To identify themes in practices used by award applicants to improve DEI efforts at their institutions, using a qualitative design. Methods This qualitative study employed an exploratory, inductive approach and constant comparative method to analyze award applications from 2 submission cycles (2020, 2021). Data analysis involved the use of a preliminary codebook of 29 program applications used in a previous study, which was modified and expanded, to perform a subsequent analysis of 12 sponsoring institution applications. Seven adjudication sessions were conducted to ensure coding consistency and resolve disagreements, resulting in the identification of final themes. Results Institutions' approaches to advancing DEI resulted from work within 5 themes and 10 subthemes. The themes encompassed organizational commitment (policies that reflect DEI mission), data infrastructure (tracking recruitment, retention, and inclusion efforts), community connection (service-learning opportunities), diverse team engagement (coproduction with residents), and systematic strategies for DEI support throughout the educational continuum. Consistent across themes was the importance of collaboration, avoiding silos, and the need for a comprehensive longitudinal approach to DEI to achieve a diverse GME workforce. Conclusions This qualitative study identified 5 themes that can inform and guide sponsoring institutions in promoting DEI.

Transition to Residency: National Study of Factors Contributing to Variability in Learner Milestones Ratings in Emergency Medicine and Family Medicine.

PURPOSE: The developmental trajectory of learning during residency may be attributed to multiple factors, including variation in individual trainee performance, program-level factors, graduating medical school effects, and the learning environment. Understanding the relationship between medical school and learner performance during residency is important in prioritizing undergraduate curricular strategies and educational approaches for effective transition to residency and postgraduate training. This study explores factors contributing to longitudinal and developmental variability in resident Milestones ratings, focusing on variability due to graduating medical school, training program, and learners using national cohort data from emergency medicine (EM) and family medicine (FM). METHOD: Data from programs with residents entering training in July 2016 were used (EM: n=1,645 residents, 178 residency programs; FM: n=3,997 residents, 487 residency programs). Descriptive statistics were used to examine data trends. Cross-classified mixed-effects regression were used to decompose variance components in Milestones ratings. RESULTS: During postgraduate year (PGY)-1, graduating medical school accounted for 5% and 6% of the variability in Milestones ratings, decreasing to 2% and 5% by PGY-3 for EM and FM, respectively. Residency program accounted for substantial variability during PGY-1 (EM=70%, FM=53%) but decreased during PGY-3 (EM=62%, FM=44%), with greater variability across training period in patient care (PC), medical knowledge (MK), and systems-based practice (SBP). Learner variance increased significantly between PGY-1 (EM=23%, FM=34%) and PGY-3 (EM=34%, FM=44%), with greater variability in practice-based learning and improvement (PBLI), professionalism (PROF), and interpersonal communication skills (ICS). CONCLUSIONS: The greatest variance in Milestone ratings can be attributed to the residency program and to a lesser degree, learners, and medical school. The dynamic impact of program-level factors on learners shifts during the first year and across the duration of residency training, highlighting the influence of curricular, instructional, and programmatic factors on resident performance throughout residency.

Strategies and Best Practices to Improve Diversity, Equity, and Inclusion Among US Graduate Medical Education Programs.

Importance: Closing the diversity gap is critical to ensure equity in medical education and health care quality. Nevertheless, evidence-based strategies and best practices to improve diversity, equity, and inclusion (DEI) in the biomedical workforce remain poorly understood and underused. To improve the culture of DEI in graduate medical education (GME), in 2020 the Accreditation Council of Graduate Medical Education (ACGME) launched the Barbara Ross-Lee, DO, Diversity, Equity, and Inclusion Award to recognize exceptional DEI efforts in US residency programs. Objective: To identify strategies and best practices that exemplary US GME programs use to improve DEI. Design and Setting: This qualitative study performed an exploratory content analysis of award applications submitted to the ACGME over 2 cycles in 2020 and 2021, using the constant comparative method. The research team first acknowledged their own biases related to DEI, used caution to not overinterpret the data, and performed several cross-checks during data analysis to ensure confirmability of the results. A preliminary codebook was developed and used during regular adjudication sessions. Disagreements were discussed until agreements were reached. Main Outcomes and Measures: Foundational (ie, commonly cited, high-impact, and small-effort strategies considered achievable by all programs) and aspirational (ie, potential for high impact but requiring greater effort and investment) DEI strategies used by exemplary GME programs. Results: This qualitative study included 29 award applications submitted between August 17, 2020, and January 11, 2022. Strategies spanned the education continuum from premedical students through faculty. Foundational strategies included working with schools, community colleges, and 4-year college campuses; providing structured support for visiting students; mission-driven holistic review for admissions and selection; interviewer trainings on implicit bias mitigation and on how racism and discrimination impact admission processes and advancement; interview-day DEI strategies; inclusive selection and DEI committees; mission statements that include DEI; and retention efforts to improve faculty diversity. Aspirational strategies included development of longitudinal bidirectional collaborations (eg, articulation agreements, annual workshops, funded rotations and/or research) with organizations working with applicants who were historically excluded and underrepresented in medicine, blinding metrics in residency applications, longitudinal curricula on DEI and health equity, and faculty mentoring such as affinity groups, mentored research, and joint academic-community recruitments. Findings provide residency program leadership with a menu of options at various inflection points to foster DEI within their programs. Conclusions and Relevance: The findings of this qualitative study suggest that GME programs might adopt strategies of exemplary programs to improve DEI in residency, ensure compliance with accreditation standards, and improve health outcomes for all.

Using learning analytics in clinical competency committees: Increasing the impact of competency-based medical education.

Graduate medical education (GME) and Clinical Competency Committees (CCC) have been evolving to monitor trainee progression using competency-based medical education principles and outcomes, though evidence suggests CCCs fall short of this goal. Challenges include that evaluation data are often incomplete, insufficient, poorly aligned with performance, conflicting or of unknown quality, and CCCs struggle to organize, analyze, visualize, and integrate data elements across sources, collection methods, contexts, and time-periods, which makes advancement decisions difficult. Learning analytics have significant potential to improve competence committee decision making, yet their use is not yet commonplace. Learning analytics (LA) is the interpretation of multiple data sources gathered on trainees to assess academic progress, predict future performance, and identify potential issues to be addressed with feedback and individualized learning plans. What distinguishes LA from other educational approaches is systematic data collection and advanced digital interpretation and visualization to inform educational systems. These data are necessary to: 1) fully understand educational contexts and guide improvements; 2) advance proficiency among stakeholders to make ethical and accurate summative decisions; and 3) clearly communicate methods, findings, and actionable recommendations for a range of educational stakeholders. The ACGME released the third edition CCC Guidebook for Programs in 2020 and the 2021 Milestones 2.0 supplement of the Journal of Graduate Medical Education (JGME Supplement) presented important papers that describe evaluation and implementation features of effective CCCs. Principles of LA underpin national GME outcomes data and training across specialties; however, little guidance currently exists on how GME programs can use LA to improve the CCC process. Here we outline recommendations for implementing learning analytics for supporting decision making on trainee progress in two areas: 1) Data Quality and Decision Making, and 2) Educator Development.

Association of Marginalized Identities With Alpha Omega Alpha Honor Society and Gold Humanism Honor Society Membership Among Medical Students.

Importance: Disparities in medical student membership in Alpha Omega Alpha (AOA) are well documented. Less is known about Gold Humanism Honor Society (GHHS) membership and it remains unknown how the intersection of different identities is associated with membership in these honor societies. Objective: To examine the association between honor society membership and medical student race and ethnicity, sex, sexual orientation, socioeconomic status, and intersection of identities. Design, Setting, and Participants: This cross-sectional study analyzed data from Association of American Medical Colleges data collection instruments. The study included all students who graduated from Liaison Committee on Medical Education-accredited US medical schools from 2016 to 2019 and completed the Graduation Questionnaire. Data analysis was conducted from January 12 to July 12, 2022. Main Outcomes and Measures: Likelihood of AOA and GHHS membership by student race and ethnicity, sex, sexual orientation, childhood family income, and intersection of identities. Results: The sample of 50 384 individuals comprised 82 (0.2%) American Indian or Alaska Native, 10 601 (21.0%) Asian, 2464 (4.9%) Black, 3291 (6.5%) Hispanic, 25 (0.1%) Native Hawaiian or Pacific Islander, 30 610 (60.8%) White, 2476 (4.9%) multiracial students, and 834 (1.7%) students of other races or ethnicities. Sex and sexual orientation included 25 672 (51.0%) men and 3078 (6.1%) lesbian, gay, and bisexual (LGB). Childhood family income comprised 31 758 (60.0%) individuals with $75 000 per year or greater, 8160 (16.2%) with $50 000 to $74 999 per year, 6864 (13.6%) with $25 000 to $49 999 per year, and 3612 (7.2%) with less than $25 000 per year. The sample included 7303 (14.5%) AOA members only, 4925 (9.8%) GHHS members only, and 2384 (4.7%) members of both societies. In AOA, American Indian or Alaska Native (OR, 0.49; 95% CI, 0.25-0.96), Asian (OR, 0.49; 95% CI, 0.45-0.53), Black (OR, 0.25; 95% CI, 0.20-0.30), Hispanic (OR, 0.53; 95% CI, 0.47-0.59), multiracial (OR, 0.69; 95% CI, 0.62-0.77), and other race and ethnicity (OR, 0.73; 95% CI, 0.60-0.88) were underrepresented compared with White students; LGB students (OR, 0.75; 95% CI, 0.67-0.83) were underrepresented compared with heterosexual students; and childhood family income $50 000 to $74 999 (OR, 0.81; 95% CI, 0.75-0.86), $25 000 to $49 999 (OR, 0.68; 95% CI, 0.62-0.74), and less than $25 000 (OR, 0.60; 95% CI, 0.53-0.69) were underrepresented compared with greater than or equal to $75 000. In GHHS, Asian students (OR, 0.80; 95% CI, 0.73-0.87) were underrepresented compared with White students, female students (OR, 1.55; 95% CI, 1.45-1.65) were overrepresented compared with male students, LGB students (OR, 1.36; 95% CI, 1.23-1.51) were overrepresented compared with heterosexual students, and students with childhood family income $25 000 to $49 999 (OR, 0.85; 95% CI, 0.78-0.94) and less than $25 000 (OR, 0.75; 95% CI, 0.66-0.86) were underrepresented compared with those with greater than or equal to $75 000. Likelihood of AOA, but not GHHS, membership decreased as number of marginalized identities increased. Conclusions and Relevance: In this cross-sectional study of US medical students, membership disparities were noted in both AOA and GHHS. However, differences in GHHS existed across fewer identities, sometimes favored the marginalized group, and were not cumulative.

The Performance and Trajectory of Medical Students With Disabilities: Results From a Multisite, Multicohort Study.

PURPOSE: To conduct a post-Americans with Disabilities Act Amendments Act of 2008 multisite, multicohort study called the Pathways Project to assess the performance and trajectory of medical students with disabilities (SWDs). METHOD: From June to December 2020, the authors conducted a matched cohort study of SWDs and nondisabled controls from 2 graduating cohorts (2018 and 2019) across 11 U.S. MD-granting medical schools. Each SWD was matched with 2 controls, one from their institution and, whenever possible, one from their cohort for Medical College Admission Test score and self-reported gender. Outcome measures included final attempt Step 1 and Step 2 Clinical Knowledge scores, time to graduation, leave of absence, matching on first attempt, and matching to primary care. RESULTS: A total of 171 SWDs and 341 controls were included; the majority of SWDs had cognitive/learning disabilities (118/171, 69.0%). Compared with controls, SWDs with physical/sensory disabilities had similar times to graduation (88.6%, 95% confidence interval [CI]: 77.0, 100.0 vs 95.1%, 95% CI: 90.3, 99.8; P = .20), Step 1 scores (229.6 vs 233.4; P = .118), and match on first attempt (93.9%, 95% CI: 86.9, 100.0 vs 94.6%, 95% CI: 91.8, 97.4; P = .842), while SWDs with cognitive/learning disabilities had lower Step 1 scores (219.4; P < .001) and were less likely to graduate on time (81.2%, 95% CI: 69.2, 93.2; P = .003) and match on first attempt (85.3%, 95% CI: 78.0, 92.7; P = .009). Accommodated SWDs had Step 1 scores that were 5.9 points higher than nonaccommodated SWDs (95% CI: -0.7, 12.5; P = .08). CONCLUSIONS: Structural barriers remain for SWDs with cognitive/learning disabilities, which could be partially mitigated by accommodations on high-stakes exams.

Responding to the Opioid Epidemic: Educational Competencies for Pain and Substance Use Disorder from the Medical Schools of the University of California.

OBJECTIVE: The University of California (UC) leadership sought to develop a robust educational response to the epidemic of opioid-related deaths. Because the contributors to this current crisis are multifactorial, a comprehensive response requires educating future physicians about safe and effective management of pain, safer opioid prescribing, and identification and treatment of substance use disorder (SUD). METHODS: The six UC medical schools appointed an opioid crisis workgroup to develop educational strategies and a coordinated response to the opioid epidemic. The workgroup had diverse specialty and disciplinary representation. This workgroup focused on developing a foundational set of educational competencies for adoption across all UC medical schools that address pain, SUD, and public health concerns related to the opioid crisis. RESULTS: The UC pain and SUD competencies were either newly created or adapted from existing competencies that addressed pain, SUD, and opioid and other prescription drug misuse. The final competencies covered three domains: pain, SUD, and public health issues related to the opioid crisis. CONCLUSIONS: The authors present a novel set of educational competencies as a response to the opioid crisis. These competencies emphasize the subject areas that are fundamental to the opioid crisis: pain management, the safe use of opioids, and understanding and treating SUD.

Three Things Schools Should Do to Make Advancement Assessment Just.

This article considers how student advancement assessment in American medical schools undermines equity. Although much attention is paid to admissions processes' capacity to diversify the physician workforce, students' advancement has been neglected as the next key step along their journeys toward graduation and residency training. This article canvasses common ways advancement undermines equity and suggests 3 areas of focus. In particular, it suggests that retention, student progression, and career advancement milestones are at least as important as admissions-based efforts to promote justice in medical education opportunity.