Insights from an updated personnel selection meta-analytic matrix: Revisiting general mental ability tests' role in the validity-diversity trade-off.

General mental ability (GMA) tests have long been at the heart of the validity-diversity trade-off, with conventional wisdom being that reducing their weight in personnel selection can improve adverse impact, but that this results in steep costs to criterion-related validity. However, Sackett et al. (2022) revealed that the criterion-related validity of GMA tests has been considerably overestimated due to inappropriate range restriction corrections. Thus, we revisit the role of GMA tests in the validity-diversity trade-off using an updated meta-analytic correlation matrix of the relationships six selection methods (biodata, GMA tests, conscientiousness tests, structured interviews, integrity tests, and situational judgment tests) have with job performance, along with their Black-White mean differences. Our results lead to the conclusion that excluding GMA tests generally has little to no effect on validity, but substantially decreases adverse impact. Contrary to popular belief, GMA tests are not a driving factor in the validity-diversity trade-off. This does not fully resolve the validity-diversity trade-off, though: Our results show there is still some validity reduction required to get to an adverse impact ratio of .80, although the validity reduction is less than previously thought. Instead, it shows that the validity-diversity trade-off conversation should shift from the role of GMA tests to that of other selection methods. The present study also addresses which selection methods now emerge as most valid and whether composites of selection methods can result in validities similar to those expected prior to Sackett et al. (2022). (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Examining the Existence of Cognitive Thresholds in Highly Quantitative College Courses.

While the dominant finding indicates a monotonic relationship between cognitive ability and academic performance, some researchers have suggested the existence of cognitive thresholds for challenging coursework, such that a certain level of cognitive ability is required for reaching a satisfactory level of academic achievement. Given the significance of finding a threshold for understanding the relationship between cognitive ability and academic performance, and the limited studies on the topic, it is worth further investigating the possibility of cognitive thresholds. Using a multi-institutional dataset and the necessary condition analysis (NCA), we attempted to replicate previous findings of cognitive thresholds on the major GPA of mathematics and physics-majored students, as well as the course grade of organic chemistry, to examine whether high SAT math scores constitute a necessary condition for obtaining satisfactory grades in these courses. The results from the two studies do not indicate an absolute cognitive threshold point below which students are doomed to fail regardless of the amount of effort they devote into learning. However, we did find that the chance of students with a low level of quantitative ability to succeed in highly quantitative courses is very small, which qualifies for the virtually necessary condition.

Perceptions of Justice in Clinical Learning Environments: Development and Validation of an Organizational Justice Measure for Medical Trainees.

PURPOSE: This study aimed to develop an instrument to measure medical trainees' perceptions of justice in clinical learning environments. METHOD: Between 2019 and 2023, the authors conducted a multiyear, multi-institutional, multiphase study to develop a 16-item justice measure with 4 dimensions: interpersonal, informational, procedural, and distributive. The authors gathered validity evidence based on test content, internal structure, and relationships with other variables across 3 phases. Phase 1 involved drafting items and gathering evidence that items measured intended dimensions. Phase 2 involved analyzing relevance of items for target groups, examining interitem correlations and factor loadings in a preliminary analysis, and obtaining reliability estimates. Phase 3 involved a confirmatory factor analysis and collecting convergent and discriminant validity evidence. RESULTS: In phase 1, 63 of 91 draft items were retained following a content validation exercise gauging how well items measured targeted dimensions (mean [SD] item ratings within dimensions, 4.16 [0.36] to 4.39 [0.34]) on a 5-point Likert scale (with 1 indicating not at all well and 5 indicating extremely well). In phase 2, 30 items were removed due to low factor loadings (i.e., < 0.40), and 4 items per dimension were selected (factor loadings, 0.42-0.89). In phase 3, a confirmatory factor analysis supported the 4-dimension model (χ2 = 610.14, P < .001; comparative fit index = 0.90, Tucker-Lewis Index = 0.87, root mean squared error of approximation = 0.11, standardized root mean squared residual = 0.06), with convergent and discriminant validity evidence showing hypothesized positive correlations with a justice measure (r = 0.93, P < .001), trait positive affect (r = 0.46, P < .001), and emotional stability (r = 0.33, P < .001) and negative correlations with trait negative affect (r = -0.39, P < .001). CONCLUSIONS: Results indicate the measure's potential utility in understanding justice perceptions and designing targeted interventions.

Stereotype lift and stereotype threat effects on subgroup mean differences for cognitive tests: A meta-analysis of adult samples.

A large body of literature has studied the effect of stereotype threat and stereotype lift on cognitive test performance. Research on stereotype threat (ST) examines whether the awareness of a negative stereotype can decrease stereotyped group members' test performance. A less commonly studied influence of stereotypes is stereotype lift (SL), defined as an increase in a group's test performance due to not being part of a negative stereotype. For example, men might perform better on math tests if they are primed on the stereotype that men are better than women at math. Walton and Cohen (2003) previously meta-analyzed the impact of SL on cognitive tests, finding an overall d = 0.24. We report an updated meta-analysis on SL with more samples and moderator analyses. We then meta-analyzed between-group effects (majority-minority group differences both in the presence and absence of SL and ST) to compare their relative contributions to subgroup mean differences on cognitive tests. Our results indicate that SL has a small influence on cognitive test performance (d = 0.09, SDres = 0.19), and that subgroup mean differences result largely from between-group effects rather than from the effects of ST and SL. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

An updated meta-analysis of the interrater reliability of supervisory performance ratings.

Given the centrality of the job performance construct to organizational researchers, it is critical to understand the reliability of the most common way it is operationalized in the literature. To this end, we conducted an updated meta-analysis on the interrater reliability of supervisory ratings of job performance (k = 132 independent samples) using a new meta-analytic procedure (i.e., the Morris estimator), which includes both within- and between-study variance in the calculation of study weights. An important benefit of this approach is that it prevents large-sample studies from dominating the results. In this investigation, we also examined different factors that may affect interrater reliability, including job complexity, managerial level, rating purpose, performance measure, and rater perspective. We found a higher interrater reliability estimate (r = .65) compared to previous meta-analyses on the topic, and our results converged with an important, but often neglected, finding from a previous meta-analysis by Conway and Huffcutt (1997), such that interrater reliability varies meaningfully by job type (r = .57 for managerial positions vs. r = .68 for nonmanagerial positions). Given this finding, we advise against the use of an overall grand mean of interrater reliability. Instead, we recommend using job-specific or local reliabilities for making corrections for attenuation. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Designing pareto-optimal selection systems for multiple minority subgroups and multiple criteria.

Currently used Pareto-optimal (PO) approaches for balancing diversity and validity goals in selection can deal only with one minority group and one criterion. These are key limitations because the workplace and society at large are getting increasingly diverse and because selection system designers often have interest in multiple criteria. Therefore, the article extends existing methods for designing PO selection systems to situations involving multiple criteria and multiple minority groups (i.e., multiobjective PO selection systems). We first present a hybrid multiobjective PO approach for computing selection systems that are PO with respect to (a) a set of quality objectives (i.e., criteria) and (b) a set of diversity objectives where each diversity objective relates to a different minority group. Next, we propose three two-dimensional subspace procedures that aid selection designers in choosing between the PO systems in case of a high number of quality and diversity objectives. We illustrate our novel multiobjective PO approaches via several example applications, thereby demonstrating that they are the first to reveal the complete gamut of eligible PO selection designs and to faithfully capture the Pareto trade-off front in case of more than two objectives. In addition, a small-scale cross-validation study confirms that the resulting PO selection designs retain an advantage over alternative designs when applied in new validation samples. Finally, the article provides a link to an executable code to perform the new multiobjective PO approaches. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

A contemporary look at the relationship between general cognitive ability and job performance.

The relationship between general cognitive ability (GCA) and overall job performance has been a long-accepted fact in industrial and organizational psychology. However, the most prominent data on this relationship date back more than 50 years. This meta-analysis examines the relationship between GCA and overall job performance using studies from the current century. Results across 153 samples and a total sample size of 40,740 show a mean observed validity of .16, with a residual SD of .09. Correcting for unreliability in the criterion and correcting predictive studies for range restriction produces a mean corrected validity of .22 and a residual SD of .11. While this is a much smaller estimate than the .51 value offered by Schmidt and Hunter (1998), that value has been critiqued by Sackett et al. (2022), who offered a mean corrected validity of .31 based on integrating findings from prior meta-analyses of 20th century data. We obtain a lower value (.22) for 21st century data. We conclude that GCA is related to job performance, but our estimate of the magnitude of the relationship is lower than prior estimates. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Correcting for range restriction in meta-analysis: A reply to Oh et al. (2023).

Oh et al. (2023) question a number of choices made in our article (Sackett et al., 2022); here we respond. They interpret our article as recommending against correcting for range restriction in general in concurrent validation studies; yet, we emphasize that we endorse correction when one has access to the information needed to do so. Our focus was on making range restriction corrections when conducting meta-analyses, where it is common for primary studies to be silent as to the prior basis for selection of the employees later participating in the concurrent validation study. As such, the applicant pool information needed for correction is typically not available. Sackett et al. (2022) highlighted that in many situations, range restriction will be small; so, the inability to correct for it results in only a modest underestimate of validity. Oh et al. mention settings that would result in substantial range restriction; here, we present our rationale as to why we view such settings as uncommon rather than as making up the bulk of the studies contributing to meta-analyses. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Differences in Trainee Evaluations of Faculty by Rater and Ratee Gender.

PURPOSE: To examine whether gender differences exist in medical trainees' (residents' and fellows') evaluations of faculty at a number of clinical departments. METHOD: The authors conducted a single-institution (University of Minnesota Medical School) retrospective cohort analysis of 5,071 trainee evaluations of 447 faculty (for which trainee and faculty gender information was available) completed between July 1, 2019, and June 30, 2022. The authors developed and employed a 17-item measure of clinical teaching effectiveness, with 4 dimensions: overall teaching effectiveness, role modeling, facilitating knowledge acquisition, and teaching procedures. Using both between- and within-subject samples, they conducted analyses to examine gender differences among the trainees making ratings (rater effects), the faculty receiving ratings (ratee effects), and whether faculty ratings differed by trainee gender (interaction effects). RESULTS: There was a statistically significant rater effect for the overall teaching effectiveness and facilitating knowledge acquisition dimensions (B = -0.28 and -0.14, 95% CI: [-0.35, -0.21] and [-0.20, -0.09], respectively, P < .001, medium corrected effect sizes between -0.34 and -0.54); female trainees rated male and female faculty lower than male trainees on both dimensions. There also was a statistically significant ratee effect for the overall teaching effectiveness and role modeling dimensions (B = -0.09 and -0.08, 95% CI: [-0.16, -0.02] and [-0.13, -0.04], P = .01 and < .001, respectively, small to medium corrected effect sizes between -0.16 and -0.44); female faculty were rated lower than male faculty on both dimensions. There was not a statistically significant interaction effect. CONCLUSIONS: Female trainees rated faculty lower than male trainees and female faculty were rated lower than male faculty on 2 teaching dimensions each. The authors encourage researchers to continue to examine the reasons for the evaluation differences observed and how implicit bias interventions might help to address them.

Challenging conclusions about predictive bias against Hispanic test takers in personnel selection.

Berry et al. (2020) noted that predictive bias is a function of three factors: subgroup mean difference on the predictor (dx), subgroup mean difference on the criterion (dy), and test validity (rxy). They used meta-analytic estimates of each of these three to examine predictive bias against Hispanic test takers when cognitive tests are used in personnel selection. They found that tests underpredict Hispanic job performance by an average of .21 SDs, which would call into question the fairness of cognitive test use in personnel selection. We located 119 studies in which all three parameters-dy, dx, and rxy-could be obtained, thus holding sample, setting, and operationalization constant in estimating the three parameters within each study. This produced a substantially different conclusion: We find that tests overpredict Hispanic performance by .04-.20 SDs, depending on assumptions made about artifact corrections. Factors contributing to differences between the two studies include differences in range restriction corrections, sample incomparability, and Berry et al.'s use of rxy estimated from the total sample rather than within the majority subgroup. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Can a Situational Judgment Test Identify Trainees at Risk of Professionalism Issues? A Multi-Institutional, Prospective Cohort Study.

PURPOSE: To determine whether overall situational judgment test (SJT) scores are associated with programs' clinical competency committee (CCC) ratings of trainee professionalism, any concerning behavior, and concerning behavior requiring active remediation at 2 time periods. METHOD: In fall 2019, trainees from 17 U.S. programs (16 residency, 1 fellowship) took an online 15-scenario SJT developed to measure 7 dimensions of professionalism. CCC midyear and year-end (6 months and 1 year following SJT completion, respectively) professionalism scores and concern ratings were gathered for academic year 2019-2020. Analyses were conducted to determine whether overall SJT scores related to overall professionalism ratings, trainees displaying any concerns, and trainees requiring active remediation at both time periods. RESULTS: Overall SJT scores correlated positively with midyear and year-end overall professionalism ratings ( r = .21 and .14, P < .001 and = .03, respectively). Holding gender and race/ethnicity constant, a 1 standard deviation (SD) increase in overall SJT score was associated with a .20 SD increase in overall professionalism ratings at midyear ( P = .005) and a .22 SD increase at year-end ( P = .001). Holding gender and race/ethnicity constant, a 1 SD increase in overall SJT score decreased the odds of a trainee displaying any concerns by 37% (odds ratio [OR] 95% confidence interval [CI]: [.44, .87], P = .006) at midyear and 34% (OR 95% CI: [.46, .95], P = .025) at year-end and decreased the odds of a trainee requiring active remediation by 51% (OR 95% CI: [.25, .90], P = .02) at midyear. CONCLUSIONS: Overall SJT scores correlated positively with midyear and year-end overall professionalism ratings and were associated with whether trainees exhibited any concerning behavior at midyear and year-end and whether trainees needed active remediation at midyear. Future research should investigate whether other potential professionalism measures are associated with concerning trainee behavior.

Familial resemblance, citizenship, and counterproductive work behavior: A combined twin, adoption, parent-offspring, and spouse approach.

Given the well-documented importance of counterproductive workplace behavior and organizational citizenship behavior (together nontask performance), it is important to clarify the degree to which these behaviors are attributable to organizational climate versus preexisting individual differences. Such clarification informs where these behaviors stem from, and consequently has practical implications for organizations (e.g., guiding prioritization of selection criteria). We investigated familial resemblance for nontask performance among twins, nontwin and adoptive siblings, parents and offspring, and midlife and late-life couples drawn from two, large-scale studies: the Minnesota Twin Family Study and the Sibling Interaction Behavior Study. Similarity among family members' (e.g., parents-offspring, siblings) engagement in nontask performance was assessed to estimate the degree to which preexisting individual differences (i.e., genetic variability) and the environment (i.e., environmentality) accounted for variation in counterproductive and citizenship behavior. We found that degree of familial resemblance for nontask performance increased with increasing genetic relationship. Nonetheless, genetically identical individuals correlated only moderately in their workplace behavior (r = .29-.40), highlighting the importance of environmental differences. Notably, family members were more similar in their counterproductive than citizenship behavior, suggesting citizenship behavior is comparatively more environmentally influenced. Spouse/partner similarity for nontask behavior was modest and did not vary between midlife and late-life couples, suggesting spousal influence on nontask performance is limited. These findings offer insight to organizations regarding the degree of nature (individual differences) and nurture (including organizational factors) influences on nontask performance, which has implications for the selection of interventions (e.g., relative value of applicant selection or incumbent interventions). (PsycInfo Database Record (c) 2022 APA, all rights reserved).

A comprehensive examination of the cross-validity of pareto-optimal versus fixed-weight selection systems in the biobjective selection context.

The article presents evidence for the cross-validity potential of fixed-weight (FW) versus Pareto-Optimal (PO) selection systems in biobjective selection situations where both the goals of diversity and quality are valued and the importance of the goals is undecided a priori. The article extends previous research by also studying the cross-validity potential of selection systems in the practically most important sample-to-sample cross-validity scenario. We address three research questions: (a) Do different PO systems show comparable levels of relative (i.e., proportional) achievement upon cross-validation? (b) Do PO systems achieve higher levels of relative achievement upon cross-validation than FW selection systems?, and (c) How does the achievement of PO and FW systems, in terms of adverse impact ratios and average performance of the selected applicants, evolve under cross-validation? As a key result, in case of sufficiently large applicant pools (typically 100 applicants or more), PO systems had on average a higher cross-validity potential than the corresponding FW systems. Yet, even for applicant pools as large as 500, FW systems may match the merits of PO systems and we present a straightforward procedure to decide which FW systems may offer a comparable cross-validation potential than the PO systems. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Weighting admission scores to balance predictiveness-diversity: The Pareto-optimization approach.

CONTEXT: Although many medical schools seek to improve diversity, they grapple with the challenge of how to weight the scores of different admission methods to achieve a balance between obtaining high predictiveness and ensuring diversity in the selected student pool. Yet, in large-scale employment settings, substantial progress has been made on this front: Pareto-optimization has been introduced as an elegant statistical tool to assist decision makers in determining the weights assigned to selection methods in advance (before the selection has taken place) so that a selection system is designed to achieve an optimal balance as reflected by the trade-off that one outcome (e.g., predictiveness) cannot be improved without harm to the other outcome (e.g., diversity). AIMS: This paper reviews the theory and research evidence about Pareto-optimization and explains how Pareto-optimization permits medical schools to better balance predictiveness and diversity in medical admission systems. METHODS: After reviewing common weighting schemes (unit, regression-based and ad hoc weighting) and their drawbacks, we introduce the theory and logic of Pareto-optimization for better balancing predictiveness and diversity. To this end, we also offer an illustrative example. Next, we review the mathematical basis and available research evidence regarding Pareto-optimization. Finally, we discuss potential criticisms (i.e., complexity and legal concerns). CONCLUSIONS: Compared to traditional unit weighting, regression-based weighting and ad hoc weighting, Pareto-optimization leads to substantial increases in diversity intake (up to three times more), while keeping the predictiveness of the selection methods at the same level. Moreover, the Pareto-optimization is robust to sampling variability and variability of the input selection parameters.

On the assessment of predictive bias in selection systems with multiple predictors.

There is a long history of examining assessments used in college admissions or personnel selection for predictive bias, also called differential prediction, to determine whether a selection system predicts comparable levels of performance for individuals from different demographic groups who have the same assessment scores. We expand on previous research that has considered predictive bias in individual predictor variables to (a) examine magnitudes of differential prediction in multipredictor selection systems and (b) explore how differences in prediction generalize across samples. We also share updated methods for computing standardized effect sizes for categorically moderated regression models that facilitate the meta-analysis of differential prediction effects. Our findings highlight the importance of analyzing composite predictors when testing for predictive bias in compensatory selection systems and demonstrate the generalizability of long-observed differential prediction trends by race/ethnicity. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Revisiting meta-analytic estimates of validity in personnel selection: Addressing systematic overcorrection for restriction of range.

This paper systematically revisits prior meta-analytic conclusions about the criterion-related validity of personnel selection procedures, and particularly the effect of range restriction corrections on those validity estimates. Corrections for range restriction in meta-analyses of predictor-criterion relationships in personnel selection contexts typically involve the use of an artifact distribution. After outlining and critiquing five approaches that have commonly been used to create and apply range restriction artifact distributions, we conclude that each has significant issues that often result in substantial overcorrection and that therefore the validity of many selection procedures for predicting job performance has been substantially overestimated. Revisiting prior meta-analytic conclusions produces revised validity estimates. Key findings are that most of the same selection procedures that ranked high in prior summaries remain high in rank, but with mean validity estimates reduced by .10-.20 points. Structured interviews emerged as the top-ranked selection procedure. We also pair validity estimates with information about mean Black-White subgroup differences per selection procedure, providing information about validity-diversity tradeoffs. We conclude that our selection procedures remain useful, but selection predictor-criterion relationships are considerably lower than previously thought. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Development of a taxonomy of unprofessional behavior in clinical learning environments using learner-generated critical incidents.

PURPOSE: Few studies have examined medical residents' and fellows' (trainees) direct experience of unprofessional behavior in clinical learning environments (CLE). The purpose of this study was to create a taxonomy of unprofessional behavior in CLEs using critical incidents gathered from trainees. METHOD: In step 1 (data collection), the authors collected 382 critical incidents from trainees at more than a dozen CLEs over a six-year period (2013-2019). In step 2 (model generation), nine subject matter experts (SMEs) sorted the incidents into homogenous clusters and this structure was tested with principal components analysis (PCA). In step 3 (model evaluation), two new groups of SMEs each re-sorted half of the incidents into the PCA-derived categories. RESULTS: A 13-component solution accounted for 62.46% of the variance in the critical incidents collected. The SMEs who re-sorted the critical incidents demonstrated good agreement with each other and with the 13-component PCA solution. The resulting taxonomy included 13 dimensions, with 48.7% of behaviors focused on displays of aggression or discriminatory conduct. CONCLUSIONS: Critical incident methodology can provide unique insights into the dimensionality of unprofessional behavior in the CLE. Future research should leverage the taxonomy created to inform professionalism assessment development in the CLE.

The process of attrition in pre-medical studies: A large-scale analysis across 102 schools.

The important but difficult choice of vocational trajectory often takes place in college, beginning with majoring in a subject and taking relevant coursework. Of all possible disciplines, pre-medical studies are often not a formally defined major but pursued by a substantial proportion of the college population. Understanding students' experiences with pre-med coursework is valuable and understudied, as most research on medical education focuses on the later medical school and residency. We examined the pattern and predictors of attrition at various milestones along the pre-med coursework track during college. Using a College Board dataset, we analyzed a sample of 15,442 students spanning 102 institutions who began their post-secondary education in years between 2006 and 2009. We examined whether students fulfilled the required coursework to remain eligible for medical schools at several milestones: 1) one semester of general chemistry, biology, physics, 2) two semesters of general chemistry, biology, physics, 3) one semester of organic chemistry, and 4) either the second semester of organic chemistry or one semester of biochemistry, and predictors of persistence at each milestone. Only 16.5% of students who intended to major in pre-med graduate college with the required coursework for medical schools. Attrition rates are highest initially but drop as students take more advanced courses. Predictors of persistence include academic preparedness before college (e.g., SAT scores, high school GPA) and college performance (e.g., grades in pre-med courses). Students who perform better academically both in high school and in college courses are more likely to remain eligible for medical school.

Improving Our Ability to Predict Resident Applicant Performance: Validity Evidence for a Situational Judgment Test.

Construct: We investigated whether a situational judgment test (SJT) designed to measure professionalism in physicians predicts residents' performance on (a) Accreditation Council for Graduate Medical Education (ACGME) competencies and (b) a multisource professionalism assessment (MPA). Background: There is a consensus regarding the importance of assessing professionalism and interpersonal and communication skills in medical students, residents, and practicing physicians. Nonetheless, these noncognitive competencies are not well measured during medical education selection processes. One promising method for measuring these noncognitive competencies is the SJT. In a typical SJT, respondents are presented with written or video-based scenarios and asked to make choices from a set of alternative courses of action. Interpersonally oriented SJTs are commonly used for selection to medical schools in the United Kingdom and Belgium and for postgraduate selection of trainees to medical practice in Belgium, Singapore, Canada, and Australia. However, despite international evidence suggesting that SJTs are useful predictors of in-training performance, end-of-training performance, supervisory ratings of performance, and clinical skills licensing objective structured clinical examinations, the use of interpersonally oriented SJTs in residency settings in the United States has been infrequently investigated. The purpose of this study was to investigate whether residents' performance on an SJT designed to measure professionalism-related competencies-conscientiousness, integrity, accountability, aspiring to excellence, teamwork, stress tolerance, and patient-centered care-predicts both their current and future performance as residents on two important but conceptually distinct criteria: ACGME competencies and the MPA. Approach: We developed an SJT to measure seven dimensions of professionalism. During calendar year 2017, 21 residency programs from 2 institutions administered the SJT. We conducted analyses to determine the validity of SJT and USMLE scores in predicting milestone performance in ACGME core competency domains and the MPA in June 2017 and 3 months later in September 2017 for the MPA and 1 year later, in June 2018, for ACGME domains. Results: At both periods, the SJT score predicted overall ACGME milestone performance (r = .13 and .17, respectively; p < .05) and MPA performance (r = .19 and .21, respectively; p < .05). In addition, the SJT predicted ACGME patient care, systems-based practice, practice-based learning and improvement, interpersonal and communication skills, and professionalism competencies (r = .16, .15, .15, .17, and .16, respectively; p < .05) 1 year later. The SJT score contributed incremental validity over USMLE scores in predicting overall ACGME milestone performance (ΔR = .07) 1 year later and MPA performance (ΔR = .05) 3 months later. Conclusions: SJTs show promise as a method for assessing noncognitive attributes in residency program applicants. The SJT's incremental validity to the USMLE series in this study underscores the importance of moving beyond these standardized tests to a more holistic review of candidates that includes both cognitive and noncognitive measures.

Some Roads Lead to Psychology, Some Lead Away: College Student Characteristics and Psychology Major Choice.

Understanding which students enter and leave psychology majors in college is critical to understanding the pipeline into the field. In this study, we compared psychology majors with nonpsychology majors on the basis of demographic, degree planning, academic preparedness, and academic performance variables using a unique longitudinal sample of nearly a million college students at 249 colleges and universities. Guided by prior research, we examined which students would persist in psychology, enter psychology from another major, or leave psychology for another major between three points in time: intended major before entering college, second-year college major, and fourth-year college major. Critically, most students who majored in psychology did not initially express interest in the field, but entering and persisting in the field was strongly associated with high school exposure and performance in psychology. Students with poorer performance in college often transfer into psychology from majors that may be perceived as difficult (e.g., science, technology, engineering, and math), whereas higher-performing students appear to leave psychology for these same majors, which may also be perceived as more lucrative. These results are concerning for the field of psychology if individuals with high potential who are originally interested in the field eventually leave.