Revolutionizing Health Professions Admissions to Achieve an Inclusive Workforce.

This article describes the "The Admissions Revolution: Bold Strategies for Diversifying the Healthcare Workforce" conference, which preceded the 2022 Beyond Flexner Alliance Conference and called for health professions institutions to boldly reimagine the admission process to diversify the health care workforce. Proposed strategies encompassed 4 key themes: admission metrics, aligning admission practices with institutional mission, community partnerships to fulfill social mission, and student support and retention. Transformation of the health professions admission process requires broad institutional and individual effort. Careful consideration and implementation of these practices will help institutions achieve greater workforce diversity and catalyze progress toward health equity.

Does Masked Interviewing Encourage Holistic Review in Residency Selection? A Mixed-Methods Study.

Problem: Medical educators increasingly champion holistic review. However, in U.S. residency selection, holistic review has been difficult to implement, hindered by a reliance on standardized academic criteria such as board scores. Masking faculty interviewers to applicants' academic files is a potential means of promoting holistic residency selection by increasing the interview's ability to make a discrete contribution to evaluation. However, little research has directly analyzed the effects of masking on how residency selection committees evaluate applicants. This mixed-methods study examined how masking interviews altered residency selection in an anesthesiology program at a large U.S. academic medical center. Intervention: During the 2019-2020 residency selection season in the University of Pennsylvania's Department of Anesthesiology & Critical Care, we masked interviewers to the major academic components of candidates' application files (board scores, transcripts, letters) on approximately half of interview days. The intent of the masking intervention was to mitigate the tendency of interviewers to form predispositions about candidates based on standardized academic criteria and thereby allow the interview to make a more independent contribution to candidate evaluation. Context: Our examination of the masking intervention used a concurrent, partially mixed, equal-status mixed-methods design guided by a pragmatist approach. We audio-recorded selection committee meetings and qualitatively analyzed them to explore how masking affected the process of candidate evaluation. We also collected independent candidate ratings from interviewers and consensus committee ratings and statistically compared ratings of candidates interviewed on masked days to ratings from conventional days. Impact: In conventional committee meetings, interviewers focused on how to reconcile academic metrics and interviews, and their evaluations of interviews were framed according to predispositions about candidates formed through perusal of application files. In masked meetings, members instead spent considerable effort evaluating candidates' "fit" and whether they came off as tactful. Masked interviewers gave halting opinions of candidates and sometimes pushed for committee leaders to reveal academic information, leading to masking breaches. Higher USMLE Step 1 score and higher medical school ranking were statistically associated with more favorable consensus rating. We found no significant differences in rating outcomes between masked and conventional interview days. Lessons learned: Elimination of academic metrics during the residency interview phase does not straightforwardly promote holistic review. While critical reflection among medical educators about the fairness and utility of such metrics has been productive, research and intervention should focus on the more proximate topic of how programs apply academic and other criteria to evaluate applicants.

The Impact of COVID-19 on Pre-Veterinary Opportunities and Recommendations for DVM Admissions.

During the summer of 2020, a survey-based study was conducted at North Carolina State University, a land-grant university, to evaluate the impact of COVID-19 on pre-veterinary students' ability to gain experience hours for Doctor of Veterinary Medicine (DVM) admissions. Of the 286 respondents (47% of the respondent pool), 92% reported losing at least one animal, veterinary, research, extracurricular, or work opportunity due to COVID-19, and 59% were not able to find a replacement. Of the lost experiences, 74 (20.8%) were for academic credit, resulting in 131 total academic credit hours lost, while only 12 credit hours were gained via alternative experiences. Of respondents, 30% (29.7%) identified as applicants of the 2020-2021 Veterinary Medical College Application Service (VMCAS) cycle. More than half (52.6%) of the sample identified being concerned about the strength of their VMCAS experiences due to these lost opportunities. Many respondents reported considering delaying application submissions by taking a gap year (17.5%) or having had their intended graduation timeline affected (14.8%). Since the majority of veterinary colleges utilize a holistic review process, this study provides a basis for understanding the effects of COVID-19 on the duration, depth, and diversity of experiences gained by future DVM applicants. This article also provides recommendations for DVM admissions adaptations based on the outcomes of the data.

How organizational culture influences holistic review: a qualitative multiple case study.

Studies primarily involving single health professions programs suggest that holistic review in admissions can increase underrepresented minority (URM) representation among admitted students. However, data showing little improvement in the overall proportion of URMs in many health professions, despite widespread use of holistic review, suggest that relatively few programs using holistic review admit substantial proportions of underrepresented minorities. Therefore, more research is needed to understand factors that facilitate holistic review practices that successfully promote diverse student enrollment. The literature suggests that a supportive organizational culture is necessary for holistic review to be effective; yet, the influence of culture on admissions has not been directly studied. This study employs a qualitative, multiple case study approach to explore the influence of a culture that values diversity and inclusion ('diversity culture') on holistic review practices in two physician assistant educational programs that met criteria consistent with a proposed conceptual framework linking diversity culture to holistic admissions associated with high URM student enrollment (relative to other similar programs). Data from multiple sources were collected at each program during the 2018-2019 admissions cycle, and a coding manual derived from the conceptual framework facilitated directed content analysis and comparison of program similarities and differences. Consistent with the conceptual framework, diversity culture appeared to be a strong driver of holistic admissions practices that support enrolling diverse classes of students. Additional insights emerged that may serve as propositions for further testing and include the finding that URM faculty 'champions for diversity' appeared to strongly influence the admissions process.

Reimagining Merit and Representation: Promoting Equity and Reducing Bias in GME Through Holistic Review.

OBJECTIVE: This study aims to evaluate the capacity of a holistic review process in comparison with non-holistic approaches to facilitate mission-driven recruitment in residency interview screening and selection, with particular attention to the promotion of race equity for applicants underrepresented in medicine (URM). METHODS: Five hundred forty-seven applicants to a psychiatry residency program from US allopathic medical schools were evaluated for interview selection via three distinct screening rubrics-one holistic approach (Holistic Review; HR) and two non-holistic processes: Traditional (TR) and Traditional Modified (TM). Each applicant was assigned a composite score corresponding to each rubric, and the top 100 applicants in each rubric were identified as selected for interview. Odds ratios (OR) of selection for interview according to URM status and secondary outcomes, including clinical performance and lived experience, were measured by analysis of group composition via univariate logistic regression. RESULTS: Relative to Traditional, Holistic Review significantly increased the odds of URM applicant selection for interview (TR-OR: 0.35 vs HR-OR: 0.84, p < 0.01). Assigning value to lived experience and de-emphasizing USMLE STEP1 scores contributed to the significant changes in odds ratio of interview selection for URM applicants. CONCLUSIONS: Traditional interview selection methods systematically exclude URM applicants from consideration without due attention to applicant strengths or potential contribution to clinical care. Conversely, holistic screening represents a structural intervention capable of critically examining measures of merit, reducing bias, and increasing URM representation in residency recruitment, screening, and selection.

An equivalence study of interview platform: Does videoconference technology impact medical school acceptance rates of different groups?

Web-based interviewing may be an effective element of a medical school's larger approach to promotion of holistic review, as recommended by the Association of American Medical Colleges, by facilitating the feasibility of including rural and community physicians in the interview process. Only 10% of medical schools offer videoconference interviews to applicants and little is known about the impact of this interview modality on the admissions process. This study investigated the impact of overall acceptance rates using videoconference interviews and face-to-face interviews in the medical school selection process using an equivalence trial design. The University of New Mexico School of Medicine integrated a videoconferencing interview option for community and rural physician interviewers in a pseudo-random fashion during the 2014-2016 admissions cycles. Logistic regression was conducted to examine whether videoconference interviews impacted acceptance rates or the characteristics of accepted students. Demographic, admissions and diversity factors were analyzed that included applicant age, MCAT score, cumulative GPA, gender, underrepresented in medicine, socioeconomic status and geographic residency. Data from 752 interviews were analyzed. Adjusted rates of acceptance for face-to-face (37.0%; 95% CI 28.2, 46.7%) and videoconference (36.1%; 95% CI 17.8, 59.5%) interviews were within an a priori ± 5% margin of equivalence. Both interview conditions yielded highly diverse groups of admitted students. Having a higher medical college admission test score, grade point average, and self-identifying as disadvantaged increased odds of admission in both interview modalities. Integration of the videoconference interview did not impact the overall acceptance of a highly diverse and qualified group of applicants, and allowed rural and community physicians to participate in the medical school interview process as well as allowed campus faculty and medical student committee members to interview remotely.

Impact of holistic review on student interview pool diversity.

Diversity in the physician workforce lags behind the rapidly changing US population. Since the gateway to becoming a physician is medical school, diversity must be addressed in the admissions process. The Association of American Medical Colleges has implemented a Holistic Review Initiative aimed at assisting medical schools with broadening admission criteria to include relevant, mission-driven attributes and experiences in addition to academic preparation to identify applicants poised to meet the needs of a diverse patient population. More evidence is needed to determine whether holistic review results in a more diverse selection process. One of the keys to holistic review is to apply holistic principles in all stages of the selection process to ensure qualified applicants are not overlooked. This study examines whether the use of holistic review during application screening at a new medical school increased the diversity of applicants selected for interview. Using retrospective data from the first five application cycles at the Oakland University William Beaumont School of Medicine (OUWB), the author compared demographic and experiential differences between the applicants selected using holistic review, including experiences, attributes and academic metrics, to a test sample selected solely using academic metrics. The dataset consisted of the total group of applicants selected for interview in 2011 through 2015 using holistic review (n = 2773) and the same number of applicants who would have been selected for an interview using an academic-only selection model (n = 2773), which included 1204 applicants who were selected using both methods (final n = 4342). The author used a combination of cross-tabulation and analysis of variance to identify differences between applicants selected using holistic review and applicants in the test sample selected using only academics. The holistic review process yielded a significantly higher than expected percent of female (adj. resid. = 13.2, p < .01), traditionally underrepresented in medicine (adj. resid. = 15.8, p < .01), first generation (adj. resid. = 5.8, p < .01), and self-identified disadvantaged (adj resid. = 11.5, p < .01) applicants in the interview pool than selected using academic metrics alone. In addition, holistically selected applicants averaged significantly more hours than academically selected students in the areas of pre-medical school paid employment (F = 10.99, mean difference = 657.99, p < .01) and community service (F = 15.36, mean difference = 475.58, p < .01). Using mission-driven, holistic admissions criteria comprised of applicant attributes and experiences in addition to academic metrics resulted in a more diverse interview pool than using academic metrics alone. These findings add support for the use of holistic review in the application screening process as a means for increasing diversity in medical school interview pools.

The influence of MCAT and GPA preadmission academic metrics on interview scores.

Medical school admissions interviews are used to assess applicants' nonacademic characteristics as advocated by the Association of American Medical Colleges' Advancing Holistic Review Initiative. The objective of this study is to determine whether academic metrics continue to significantly influence interviewers' scores in holistic processes by blinding interviewers to applicants' undergraduate grade point averages (uGPA) and Medical College Admission Test (MCAT). This study examines academic and demographic predictors of interview scores for two applicant cohorts at the University of Michigan Medical School. In 2012, interviewers were provided applicants' uGPA and MCAT scores; in 2013, these academic metrics were withheld from interviewers' files. Hierarchical regression analysis was conducted to examine the influence of academic and demographic variables on overall cohort interview scores. When interviewers were provided uGPA and MCAT scores, academic metrics explained more variation in interview scores (7.9%) than when interviewers were blinded to these metrics (4.1%). Further analysis showed a statistically significant interaction between cohort and uGPA, indicating that the association between uGPA and interview scores was significantly stronger for the 2012 unblinded cohort compared to the 2013 blinded cohort (ß = .573, P < .05). By contrast, MCAT scores had no interactive effects on interviewer scores. While MCAT scores accounted for some variation in interview scores for both cohorts, only access to uGPA significantly influenced interviewers' scores when looking at interaction effects. Withholding academic metrics from interviewers' files may promote assessment of nonacademic characteristics independently from academic metrics.

Impact of Holistic Application Scoring on Interview Offers.

OBJECTIVE: We aimed to develop a holistic screening tool for surgical residency application processing to capture the diverse skills and attributes of the applicant, based on characteristics most commonly associated with success in our residency program. DESIGN: We developed an application-scoring rubric with 4 domains based on ACGME Holistic reviewing criteria: academic potential, experiences, personal attributes, and clinical competency. We scored academic potential based on a composite of Step 2 score, MSPE tier rank, surgery clerkship grade, college honors, publications, and presentations. An additional score accounted for personal adversity overcome or "distance travelled". This included previous homelessness, first-generation college student or immigrant status, noted socioeconomic hardship, disability overcome, or other stated personal experience of discrimination including underrepresented in medicine status. We sorted the list of top 200 candidates by adversity score, Step 2 score, academic potential score, and total overall score to compare the groups in terms of interview offers. SETTING: We are an academic surgical residency program housed within a private medical college in the Northeast region. PARTICIPANTS: All categorical applicants to our program were managed with our holistic screening approach. RESULTS: There were 29 students with the highest adversity score (AS) of 4 and 26 (90%) were selected to interview based on holistic overall score and reviewer comments. Fourteen students had an AS of 3, and 12 (86%) were selected to interview. Twenty-five students had an AS of 2, and 23 (92%) were selected to interview. If Step 2 score alone had been used to determine which applicants should be interviewed, only 11 students (38%) with an AS of 4 would have been interviewed. If the academic potential alone was used for screening, only 10 (35%) of students with an AS of 4 would have been interviewed. Taking all students with any adversity score into account (n = 70), when screened with only Step 2 scores, just 31(44%) would have been interviewed. When ranked by academic potential score, 35 (50%) would have been interviewed. When applying our holistic overall score alone, 56 (80%) would have been interviewed. CONCLUSIONS: Performing a holistic application review and ranking students not only by standardized exam scores, but also considering other history of academic achievement, personal attributes, experiences in leadership or service, and clinical competency can allow for mitigation of implicit bias. Allowance for an adversity score can help programs recognize students who have significantly longer "distance traveled" to make it through medical education and who have the potential to be fantastic residents.

From Athlete to Surgeon: Examining Sports and Grit Among General Surgery Residents.

OBJECTIVE: The transition of Step 1 to pass/fail has generated concerns over selecting promising candidates. Holistic reviews integrate other proficiencies, including extracurriculars such as sports. Grit - defined as perseverance and passion for long-term goals - has been positively associated with competitive activities and is predictive of academic success. The prevalence and impact of sports participation and its relationship to grit in the general surgery resident population has not been described and was investigated in this study. DESIGN: Surveys measuring sports participation and grit were distributed after the 2021 ABSITE. Grit was assessed through the short grit scale. Inferential statistics were performed. SETTING/PARTICIPANTS: General surgery residents in all US training programs who completed the 2021 ABSITE. RESULTS: Of 5468 respondents (response rate 59.6%), 2,548 (46.7%) were female, 917 (17.4%) URiM, 2171 (39.8%) married, and 1,069 (19.6%) parents. About 4284 (83.8%) residents reported being involved in competitive sports. Grit was higher in residents with a competitive sports history (3.67 ± 0.58 versus 3.60 ± 0.61, p = 0.0022). Greater time commitment and being part of a team was positively correlated to grit (both p < 0.0001). Individuals that self-identified as underrepresented in medicine (URiM) had higher grit (3.71 ± 0.59 versus 3.65 ± 0.58 for non-URiM, p < 0.0001) as did female (p = 0.0016), married residents (p < 0.0001), and parents (p < 0.0001). Being an athlete was associated with significantly higher grit for nearly all demographic subgroups, including URiM (p = 0.0068), married (p = 0.0175), and parents (p = 0.0487). CONCLUSIONS: Higher grit was found in athletes and marginalized groups including females, URiM, and residents that were married or parents. Our data suggests that recruiting applicants of diverse backgrounds and experiences will result in a grittier cohort; a group potentially equipped to weather the arduous surgical residency training path. Recruiting residents with characteristics associated with higher grit can potentially impact diversity of the surgical workforce.

Holistic Review of Applicants by the United States Postgraduate Training Programs Is Not Associated With the Percentage of Female Residents.

Introduction Given the underrepresentation of female physicians in most specialties and the aim of holistic review in residency applications to improve the diversity of matriculating resident physicians in the United States (US) postgraduate medical training programs, we examined the association between holistic review and female resident representation among US postgraduate training programs. Methods We conducted a cross-sectional survey of US postgraduate training programs to inquire about their use of holistic review for resident applications (independent variable). The primary outcome was the percentage of female residents in each program, which was obtained along with other program-level characteristics from the Fellowship and Residency Electronic Interactive Database Access (FREIDA) catalog in April 2023. We limited the analysis to the 10 specialties with the most training spots in 2022, including anesthesiology, emergency medicine, family medicine, internal medicine, neurology, obstetrics and gynecology, orthopedic surgery, pediatrics, psychiatry, and surgery (general). We also examined the interactions between holistic review and specialty and the percentage of female faculty using model comparison and simple slopes analyses.  Results Of the 3,364 total programs surveyed from the 10 specialties, 222 (6.6%) responded. Responders and nonresponders had similar program-level characteristics, including program type (e.g., university, community), specialty, and reported minimum board examination scores. Of the 222 responders, 179 (80.6%) reported performing holistic review. The percentage of female residents was 49.0% (interquartile range 37.5 to 66.7) in the no holistic review group and 47.8% (35.4 to 65.0) in the holistic review group (median difference 0.9%, 95% confidence interval -6.7 to 8.3). Furthermore, there was no evidence of interaction between holistic review and either the specialty or the percentage of female faculty on the outcome of the percentage of female residents. Conclusions Holistic review of residency applications in this limited sample of US postgraduate training programs was not associated with the percentage of female residents. The role of holistic review in addressing the imbalance of male and female physicians in the healthcare workforce, particularly between specialties, remains unknown.

Introductory Video as a Supplemental Application for General Surgery Residency Candidates: Concept, Implementation, Preliminary Outcomes, and Future Directions.

OBJECTIVE: The residency selection process has changed dramatically over the past several years, primarily due to the COVID-19 pandemic causing a transition to virtual interviews and limiting sub-internship opportunities. Another new major change is the transition of the USMLE Step 1 examination from a numerical score to pass/fail as of January 2022. In anticipation of this major change, our program asked applicants to submit a short video clip expressing their interest. PARTICIPANTS: Review of each of the videos was completed by three members of the admission committee and each reviewer was required to complete formal implicit bias training prior to reviewing the videos. RESULTS: Over 1200 general surgery residency applications were received via the Electronic Residency Application Service (ERAS) for the 2022-2023 application cycle. 286 applicants were selected for a formal interview invitation and asked to submit a 2-minute introductory video. In response, 222 (77.62%) of the 286 cohort submitted introductory videos. 210 of these videos (94.59%) were submitted within the first week. The average length of the videos submitted was 105.10 seconds. CONCLUSION: Using optional introductory videos as a supplemental application has helped to narrow the applicant pool and identify individuals to offer interviews.

Creating a Diverse Physician Workforce Through Training in Diversity, Bias, and Selection: An Innovative Approach to Preparing Medical Students as Future Academic Leaders.

Medical students are the future of academic medicine. They will serve as admissions committee members, deans, and program directors with responsibility for selecting future physicians. At the same time, schools and programs are working to diversify the physician workforce to care for diverse patient populations. At UAB Heersink School of Medicine, we developed an elective course, Holistic Review: Creating a Mission-Driven Physician Workforce, to train learners in bias, diversity, and data-informed decision-making. Students applied lessons to the admissions process by participating in application screening and multiple-mini interview rating. Student reflections demonstrated course applicability for careers in medicine and beyond.

Epistemic Oppression, Construct Validity, and Scientific Rigor: Commentary on Woo et al. (2022).

In this commentary, I highlight flaws in the article by Woo and colleagues (this issue) that undermine its credibility and utility as rigorous science that contributes to the field. I do so by discussing (a) the concept of epistemic oppression regarding the glaring exclusion of multiple germane bodies of research and (b) the importance of including construct validity within a psychometric article regarding the Graduate Record Examination (GRE). I conclude with a plea to the authors that the matter of anti-Black violence and murder, which they reference, is one to be taken respectfully, seriously, and somberly.

Steps Leaders Can Take to Increase Diversity, Enhance Inclusion, and Achieve Equity.

The importance of skin of color and diversity, equity, and inclusion (DEI) started to be recognized in the late 1990s. Since then, because of the advocacy and effort of several highly visible leaders in dermatology, noticeable progress has been achieved. Leadership lessons learned for successful implementation of DEI include the following: (1) commitment by and continued engagement of highly visible leaders; (2) engagement of other societies in dermatology; (3) engagement of dermatology department leaders and educators; (4) education of the next generation of dermatologists; (5) inclusivity in DEI to include gender and sexual orientation; and (6) cultivation of allies and allyship.

Implementation of the AAMC's Holistic Review Model for Psychiatry Resident Recruitment.

Introduction: In psychiatry, several converging factors are impacting the recruitment of residents: the increased competitiveness of the specialty, the national trend to take active steps to improve diversity and inclusion, and the decision from USMLE to change Step 1 to a pass/fail result. Methods: We developed a workshop for psychiatry residency program directors to meet these challenges and transition into using a holistic review model during recruitment. The workshop included (1) a didactic session providing background on the AAMC holistic review model; (2) a small-group exercise to determine and prioritize experiences, attributes, competencies, and metrics (EACMs) aligned with the program's mission and aims; (3) a review of the rankings from the previous exercise, selection of two "very important" criteria for each of the four domains of the EACM model, and operationalization of these criteria based on the recruitment process; and (4) a discussion focused on application of program criteria with example applicants. Results: The holistic review workshop was conducted at the American Association of Psychiatry Residency Directors conference in 2021 with 48 self-selected attendees. Following the workshop, 74% of attendees reported a likelihood of implementing holistic applications during their next application cycle, 78% were able to leave with at least one actionable item, 100% thought that the session was interactive, and 78% felt that the session met their expectations. Discussion: Implementing a holistic review for psychiatry residency recruitment can assist programs in responding to the rapidly changing landscape and achieve aims for improving diversity and inclusion.

Implementation of a Holistic Review Process of US Allopathic Medical Students Eliminates Non-Comparable Metrics and Bias in General Surgery Residency Interview Invitations.

OBJECTIVE: Use of traditional scoring metrics for residency recruitment creates racial and gender bias. In addition, widespread use of pass/fail grading has led to noncomparable data. To adjust to these challenges, we developed a holistic review (HR) rubric for scoring residency applicants for interview selection. DESIGN: Single-center observational study comparing the proportion of underrepresented in medicine (URM) students and their United States Medical Licensing Exam (USMLE) scores who were invited for interview before (2015-2020) and after (2022) implementation of a holistic review process. SETTING: General surgery residency program at a tertiary academic center. PARTICIPANTS: US allopathic medical students applying for general surgery residency. RESULTS: After initial screening, a total of 1514 allopathic applicants were narrowed down to 586 (38.7%) for HR. A total of 52% were female and 17% identified as URM. Based on HR score, 20% (118/586) of applicants were invited for an interview. The median HR score was 11 (range 4-19). There was a fourfold higher coefficient of variation of HR scores (22.3; 95% CI 21.0-23.7) compared to USMLE scores (5.1; 95% Cl 4.8-5.3), resulting in greater spread and distinction among applicants. There were no significant differences in HR scores between genders (p = 0.60) or URM vs non-URM (p = 0.08). There were no significant differences in Step 1 (p = 0.60) and 2CK (p = 0.30) scores between those who were invited to interview or not. On multivariable analysis, USMLE scores (OR 1.01; 95% CI 0.98-1.03), URM status (OR 1.71 95% CI 0.98-2.92), and gender (OR 0.94, 95% CI 0.60-1.45) did not predict interview selection (all p > 0.05). There was a meaningful increase in the percentage of URM interviewed after HR implementation (12.9% vs 23.1%, p = 0.016). CONCLUSION: The holistic review process is feasible and eliminates the use of noncomparable metrics for surgical applicant interview invitations and increases the percentage of URM applicants invited to interview.