An updated meta-analysis of the interrater reliability of supervisory performance ratings.

Given the centrality of the job performance construct to organizational researchers, it is critical to understand the reliability of the most common way it is operationalized in the literature. To this end, we conducted an updated meta-analysis on the interrater reliability of supervisory ratings of job performance (k = 132 independent samples) using a new meta-analytic procedure (i.e., the Morris estimator), which includes both within- and between-study variance in the calculation of study weights. An important benefit of this approach is that it prevents large-sample studies from dominating the results. In this investigation, we also examined different factors that may affect interrater reliability, including job complexity, managerial level, rating purpose, performance measure, and rater perspective. We found a higher interrater reliability estimate (r = .65) compared to previous meta-analyses on the topic, and our results converged with an important, but often neglected, finding from a previous meta-analysis by Conway and Huffcutt (1997), such that interrater reliability varies meaningfully by job type (r = .57 for managerial positions vs. r = .68 for nonmanagerial positions). Given this finding, we advise against the use of an overall grand mean of interrater reliability. Instead, we recommend using job-specific or local reliabilities for making corrections for attenuation. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

The role of socioeconomic status in SAT-grade relationships and in college admissions decisions.

This article examines the role of socioeconomic status (SES) in the relationships among college admissions-test scores, secondary school grades, and subsequent academic performance. Scores on the SAT (a test widely used in the admissions process in the United States), secondary school grades, college grades, and SES measures from 143,606 students at 110 colleges and universities were examined, and results of these analyses were compared with results obtained using a 41-school data set including scores from the prior version of the SAT and using University of California data from prior research on the role of SES. In all the data sets, the SAT showed incremental validity over secondary school grades in predicting subsequent academic performance, and this incremental relationship was not substantially affected by controlling for SES. The SES of enrolled students was very similar to that of specific schools' applicant pools, which suggests that the barrier to college for low-SES students in the United States is a lower rate of entering the college admissions process, rather than exclusion on the part of colleges.

Ability-performance relationships in education and employment settings: critical tests of the more-is-better and the good-enough hypotheses.

The nature of the relationship between ability and performance is of critical importance for admission decisions in the context of higher education and for personnel selection. Although previous research has supported the more-is-better hypothesis by documenting linearity of ability-performance relationships, such research has not been sensitive enough to detect deviations at the top ends of the score distributions. An alternative position receiving considerable attention is the good-enough hypothesis, which suggests that although higher levels of ability may result in better performance up to a threshold, above this threshold greater ability does not translate to better performance. In this study, the nature of the relationship between cognitive ability and performance was examined throughout the score range in four large-scale data sets. Monotonicity was maintained in all instances. Contrary to the good-enough hypothesis, the ability-performance relationship was commonly stronger at the top end of the score distribution than at the bottom end.

Testing the generalizability of indirect range restriction corrections.

Range restriction is a common problem in personnel selection and other contexts in applied psychology. For many years researchers have used corrections that assume range restriction was direct, even when it was known that range restriction was indirect. Hunter, Schmidt, and Le (2006) proposed a new correction for cases of indirect range restriction that greatly increases its potential usefulness due to its reduced information requirements compared to alternatives. The current study examines the applicability of Hunter et al.'s correction to settings where its assumed structural model is violated by including the measures that are to be involved in corrections in the original selection composite. We conclude that Hunter et al.'s correction should generally be preferred when compared to its common alternative, Thorndike's Case II correction for direct range restriction. However, this is due to the likely violation of one of the other assumptions of the Hunter et al. correction in most applied settings. Correction mechanisms and practical implications are discussed.