Designing a Situational Judgment Test for Use in Medical School Admissions.

ABSTRACT: It has long been acknowledged that professional competencies are required for success in medical school, residency training, and medical practice. Over the last decade, medical schools have begun to introduce standardized assessments of professional competencies, but many still rely on interviews to assess these competencies, which occur after about half of the applicant pool has already been screened out. In this article, the authors discuss the development, evaluation, and launch of the Association of American Medical Colleges (AAMC) situational judgment test (SJT) for use in medical school admissions. The AAMC SJT is designed to assess an examinee's understanding of effective and ineffective behaviors related to the core competencies for entering medical students, including service orientation, social skills, cultural competence, teamwork, ethical responsibility to self and others, reliability and dependability, resilience and adaptability, and capacity for improvement. The authors evaluate the evidence for the need for SJTs in medical school admissions by exploring common derailers in medical school, gaps in the admissions process regarding information about professional competencies, and the challenge of conducting holistic review in a high-volume context. They summarize existing research from the employment, international medical education, and residency selection contexts suggesting that SJT scores are positively associated with subsequent performance and may add value to the admissions process. The authors discuss 5 goals that were the foundation for developing the AAMC SJT: (1) assess the professional competencies needed for success in medical school using a proven method, (2) enable holistic review in a high-volume admissions context, (3) create and share a program of research to support the appropriate use of SJT scores, (4) signal the need for preparation in professionalism to learners, and (5) balance the need for a new assessment with minimizing the burden and risk for applicants.

Self-Reported Disadvantage in Medical School Admissions: A Call to Review, Revise, and Further Advance Holistic Review.

PURPOSE: This study examined how applicants interpret the self-reported disadvantaged (SRD) question in the American Medical College Application Service (AMCAS) application. METHOD: Data from 129,262 applicants who applied through AMCAS from 2017 through 2019 were used, including financial and family history, demographic characteristics, and work status and residence. Fifteen applicants from the 2020 and 2021 AMCAS cycles were interviewed about their experiences with the SRD question. RESULTS: Large effects were found for SRD applicants with fee assistance waivers ( h = 0.89), Pell grants ( h = 1.21), state or federal aid ( h = 1.10), and parents with less education ( h = 0.98) and non-SRD applicants with a large proportion of their education paid by family ( d = 1.03). Another large difference was found for reported family income distribution (73% of SRD applicants reporting family income < $50,000 vs 15% of non-SRD applicants). More SRD applicants were Black or Hispanic (26% vs 16% and 5% vs 5%), Deferred Action for Childhood Arrivals recipients (11% vs 2%), born outside the United States (32% vs 16%), and raised in a medically underserved area (60% vs 14%). There was a moderate effect for first-generation to college SRD applicants ( h = 0.61). SRD applicants had lower Medical College Admission Test scores ( d = 0.62) and overall and science grade point averages ( d = 0.50 and 0.49, respectively) but no meaningful differences in acceptance or matriculation rates. The interviews identified 5 themes: (1) unclear disadvantage definition; (2) different perceptions of disadvantage and overcoming challenges or obstacles ; (3) identification as disadvantaged or not; (4) SRD essay content; and (5) concerns about lack of transparency in how the SRD question is used in admissions. CONCLUSIONS: Revising the SRD question by including context, phrasing, and instructions for broader experience categories might be beneficial because of lack of transparency and understanding.