Predicting Medical Student Performance With a Situational Judgment Test in Admissions.

PURPOSE: To explore correlations between AAMC situational judgment test (SJT) scores, other admissions data, and learners' medical school performance. METHOD: First- and second-year medical students from 8 U.S. MD-granting medical schools completed a prototype version of the AAMC SJT in 2017. Outcomes included research-only faculty ratings of student performance, final course grades, and faculty evaluations of student performance, 2017-2018 and 2018-2019 academic years. Bivariate correlations were used to investigate the relationship between SJT scores and student performance outcomes and hierarchical regressions to investigate whether SJT scores provided incremental validity over MCAT total scores and cumulative undergraduate grade point averages (UGPAs) for predicting student performance outcomes. RESULTS: In general, there were small positive correlations with research-only faculty ratings from the first year of medical school, with the highest for social skills/service orientation ( rcorrected = .33, P < .05). Correlations were higher, with the highest for social skills/service orientation and cultural competence ( rcorrected = .33 and .36, respectively, P < .05) in the second year in medical school. SJT scores improved prediction of research-only faculty ratings over MCAT total scores and UGPAs for reliability and dependability/capacity for improvement, cultural competence, social skills/service orientation, and the overall composite score in the first year and for resilience and adaptability, social skills/service orientation, cultural competence, and the overall composite score in the second year. SJT scores demonstrated small correlations with course grades ( rsample-weighted = .10, P = ns) and faculty evaluations related to professionalism skills ( rsample-weighted = .14, P < .05); however, MCAT total scores explained most of the variance associated with course outcomes. CONCLUSIONS: These studies provide initial evidence that SJT scores may add value to the medical school admissions process because scores were related to faculty ratings of professional behaviors and provided unique information relative to MCAT scores and UGPAs.

Designing a Situational Judgment Test for Use in Medical School Admissions.

ABSTRACT: It has long been acknowledged that professional competencies are required for success in medical school, residency training, and medical practice. Over the last decade, medical schools have begun to introduce standardized assessments of professional competencies, but many still rely on interviews to assess these competencies, which occur after about half of the applicant pool has already been screened out. In this article, the authors discuss the development, evaluation, and launch of the Association of American Medical Colleges (AAMC) situational judgment test (SJT) for use in medical school admissions. The AAMC SJT is designed to assess an examinee's understanding of effective and ineffective behaviors related to the core competencies for entering medical students, including service orientation, social skills, cultural competence, teamwork, ethical responsibility to self and others, reliability and dependability, resilience and adaptability, and capacity for improvement. The authors evaluate the evidence for the need for SJTs in medical school admissions by exploring common derailers in medical school, gaps in the admissions process regarding information about professional competencies, and the challenge of conducting holistic review in a high-volume context. They summarize existing research from the employment, international medical education, and residency selection contexts suggesting that SJT scores are positively associated with subsequent performance and may add value to the admissions process. The authors discuss 5 goals that were the foundation for developing the AAMC SJT: (1) assess the professional competencies needed for success in medical school using a proven method, (2) enable holistic review in a high-volume admissions context, (3) create and share a program of research to support the appropriate use of SJT scores, (4) signal the need for preparation in professionalism to learners, and (5) balance the need for a new assessment with minimizing the burden and risk for applicants.

The predictive validity of the MCAT exam in relation to academic performance through medical school: a national cohort study of 2001-2004 matriculants.

PURPOSE: Most research examining the predictive validity of the Medical College Admission Test (MCAT) has focused on the relationship between MCAT scores and scores on the United States Medical Licensing Examination Step exams. This study examined whether MCAT scores predict students' unimpeded progress toward graduation (UP), which the authors defined as not withdrawing or being dismissed for academic reasons, graduating within five years of matriculation, and passing the Step 1, Step 2 Clinical Knowledge, and Step 2 Clinical Skills exams on the first attempt. METHOD: Students who matriculated during 2001-2004 at 119 U.S. medical schools were included in the analyses. Logistic regression analyses were used to estimate the relationships between UP and MCAT total scores alone, undergraduate grade point averages (UGPAs) alone, and UGPAs and MCAT total scores together. All analyses were conducted at the school level and were considered together to evaluate relationships across schools. RESULTS: The majority of matriculants experienced UP. Together, UGPAs and MCAT total scores predicted UP well. MCAT total scores alone were a better predictor than UGPAs alone. Relationships were similar across schools; however, there was more variability across schools in the relationship between UP and UGPAs than between UP and MCAT total scores. CONCLUSIONS: The combination of UGPAs and MCAT total scores performs well as a predictor of UP. Both UGPAs and MCAT total scores are strong predictors of academic performance in medical school through graduation, not just the first two years. Further, these relationships generalize across medical schools.

An overview of the medical school admission process and use of applicant data in decision making: what has changed since the 1980s?

PURPOSE: To investigate current medical school admission processes and whether they differ from those in 1986 when they were last reviewed by the Association of American Medical Colleges (AAMC). METHOD: In spring 2008, admission deans from all MD-granting U.S. and Canadian medical schools using the Medical College Admission Test (MCAT) were invited to complete an online survey that asked participants to describe their institution's admission process and to report the use and rate the importance of applicant data in making decisions at each stage. RESULTS: The 120 responding admission officers reported using a variety of data to make decisions. Most indicated using interviews to assess applicants' personal characteristics. Compared with 1986, there was an increase in the emphasis placed on academic data during pre-interview screening. While GPA data were among the most important data in decision making at all stages in 1986, data use and importance varied by the stage of the process in 2008: MCAT scores and undergraduate GPAs were rated as the most important data for deciding whom to invite to submit secondary applications and interview, whereas interview recommendations and letters of recommendation were rated as the most important data in deciding whom to accept. CONCLUSIONS: This study underscores the complexity of the medical school admission process and suggests increased use of a holistic approach that considers the whole applicant when making admission decisions. Findings will inform AAMC initiatives focused on transforming admission processes.

Core personal competencies important to entering students' success in medical school: what are they and how could they be assessed early in the admission process?

Assessing applicants' personal competencies in the admission process has proven difficult because there is not an agreed-on set of personal competencies for entering medical students. In addition, there are questions about the measurement properties and costs of currently available assessment tools. The Association of American Medical College's Innovation Lab Working Group (ILWG) and Admissions Initiative therefore engaged in a multistep, multiyear process to identify personal competencies important to entering students' success in medical school as well as ways to measure them early in the admission process. To identify core personal competencies, they conducted literature reviews, surveyed U.S and Canadian medical school admission officers, and solicited input from the admission community. To identify tools with the potential to provide data in time for pre-interview screening, they reviewed the higher education and employment literature and evaluated tools' psychometric properties, group differences, risk of coaching/faking, likely applicant and admission officer reactions, costs, and scalability. This process resulted in a list of nine core personal competencies rated by stakeholders as very or extremely important for entering medical students: ethical responsibility to self and others; reliability and dependability; service orientation; social skills; capacity for improvement; resilience and adaptability; cultural competence; oral communication; and teamwork. The ILWG's research suggests that some tools hold promise for assessing personal competencies, but the authors caution that none are perfect for all situations. They recommend that multiple tools be used to evaluate information about applicants' personal competencies in deciding whom to interview.