Weighting admission scores to balance predictiveness-diversity: The Pareto-optimization approach.

CONTEXT: Although many medical schools seek to improve diversity, they grapple with the challenge of how to weight the scores of different admission methods to achieve a balance between obtaining high predictiveness and ensuring diversity in the selected student pool. Yet, in large-scale employment settings, substantial progress has been made on this front: Pareto-optimization has been introduced as an elegant statistical tool to assist decision makers in determining the weights assigned to selection methods in advance (before the selection has taken place) so that a selection system is designed to achieve an optimal balance as reflected by the trade-off that one outcome (e.g., predictiveness) cannot be improved without harm to the other outcome (e.g., diversity). AIMS: This paper reviews the theory and research evidence about Pareto-optimization and explains how Pareto-optimization permits medical schools to better balance predictiveness and diversity in medical admission systems. METHODS: After reviewing common weighting schemes (unit, regression-based and ad hoc weighting) and their drawbacks, we introduce the theory and logic of Pareto-optimization for better balancing predictiveness and diversity. To this end, we also offer an illustrative example. Next, we review the mathematical basis and available research evidence regarding Pareto-optimization. Finally, we discuss potential criticisms (i.e., complexity and legal concerns). CONCLUSIONS: Compared to traditional unit weighting, regression-based weighting and ad hoc weighting, Pareto-optimization leads to substantial increases in diversity intake (up to three times more), while keeping the predictiveness of the selection methods at the same level. Moreover, the Pareto-optimization is robust to sampling variability and variability of the input selection parameters.

Situational judgement tests for selection: Traditional vs construct-driven approaches.

CONTEXT: Historically, situational judgement tests (SJTs) have been widely used for personnel selection. Their use in medical selection in Europe is growing, with plans for further expansion into North America and Australasia, in an attempt to measure and select on 'non-academic' personal attributes. However, there is a lack of clarity regarding what such tests actually measure and how they should be designed, scored and implemented within the medical and health education selection process. In particular, the theoretical basis from which such tests are developed will determine the scoring options available, influencing their psychometric properties and, ultimately, their validity. METHODS: The aim of this article is to create an awareness of the previous theory and practice that has informed SJT development. We describe the emerging interest in the use of the SJT format to measure specific constructs (eg 'resilience', 'dependability', etc.), drawing on the tradition of 'individual differences' psychology. We compare and contrast this newer 'construct-driven' method with the traditional, pragmatic approach to SJT creation, often employed by organisational psychologists. Making reference to measurement theory, we highlight how the anticipated psychometric properties of traditional vs construct-driven SJTs are likely to differ. CONCLUSIONS: Compared to traditional SJTs, construct-driven SJTs have a strong theoretical basis, are uni- rather than multidimensional, and may behave more like personality self-report instruments. Emerging evidence also suggests that construct-driven SJTs have comparable predictive validity for workplace performance, although they may be more prone to 'faking' effects. It is possible that construct-driven approaches prove more appropriate at early stages of medical selection, where candidates have little or no health care work experience. Conversely, traditional SJTs may be more suitable for specialty recruitment, where a range of hypothetical workplace scenarios can be sampled in assessments.

Response distortion on personality tests in applicants: comparing high-stakes to low-stakes medical settings.

The current study examined the degree to which applicants applying for medical internships distort their responses to personality tests and assessed whether this response distortion led to reduced predictive validity. The applicant sample (n = 530) completed the NEO Personality Inventory whilst applying for one of 60 positions as first-year post-graduate medical interns. Predictive validity was assessed using university grades, averaged over the entire medical degree. Applicant responses for the Big Five (i.e., neuroticism, extraversion, openness, conscientiousness, and agreeableness) and 30 facets of personality were compared to a range of normative samples where personality was measured in standard research settings including medical students, role model physicians, current interns, and standard young-adult test norms. Applicants had substantially higher scores on conscientiousness, openness, agreeableness, and extraversion and lower scores on neuroticism with an average absolute standardized difference of 1.03, when averaged over the normative samples. While current interns, medical students, and especially role model physicians do show a more socially desirable personality profile than standard test norms, applicants provided responses that were substantially more socially desirable. Of the Big Five, conscientiousness was the strongest predictor of academic performance in both applicants (r = .11) and medical students (r = .21). Findings suggest that applicants engage in substantial response distortion, and that the predictive validity of personality is modest and may be reduced in an applicant setting.

Future directions in personality, occupational and medical selection: myths, misunderstandings, measurement, and suggestions.

This paper has two objectives: (1) presenting recent advances in personality theory whereby personality traits are conceptualized within a framework that focuses on the dynamic interactions of behaviour, biology, context, and states, and (2) discussing the implications of these developments for measurement and medical selection. We start by presenting evidence that traits are no longer regarded as stable deterministic predictors of behaviour. Instead, traits are found to change across generations, the life span, and in response to environmental contingencies. Thus, there is an urgent need to explore how traits change as function of medical education. Second, drawing on recent theory and research (behavioural reaction norms and the density distribution model) we highlight evidence to show how the expression of trait relevant behaviour is dependent on context, and is distributed with an average (typical behaviour or personality) and a variance (plasticity or adaptability), with traditional personality measure associated with typical responding. Third, we demystify that some traits are better than others showing that so-called "good" traits have a dark-side. Fourth, we show how these developments impact on how personality might be assessed, thereby presenting recent evidence on the use of contextualized personality measures, situational judgment tests, other reports, and implicit measures. Throughout the paper, we outline the key implications of these developments for medical selection practices.

Widening access in selection using situational judgement tests: evidence from the UKCAT.

CONTEXT: Widening access promotes student diversity and the appropriate representation of all demographic groups. This study aims to examine diversity-related benefits of the use of situational judgement tests (SJTs) in the UK Clinical Aptitude Test (UKCAT) in terms of three demographic variables: (i) socio-economic status (SES); (ii) ethnicity, and (iii) gender. METHODS: Outcomes in medical and dental school applicant cohorts for the years 2012 (n = 15 581) and 2013 (n = 15 454) were studied. Applicants' scores on cognitive tests and an SJT were linked to SES (parents' occupational status), ethnicity (White versus Black and other minority ethnic candidates), and gender. RESULTS: Firstly, the effect size for SES was lower for the SJT (d = 0.13-0.20 in favour of the higher SES group) than it was for the cognitive tests (d = 0.38-0.35). Secondly, effect sizes for ethnicity of the SJT and cognitive tests were similar (d = ~ 0.50 in favour of White candidates). Thirdly, males outperformed females on cognitive tests, whereas the reverse was true for SJTs. When equal weight was given to the SJT and the cognitive tests in the admission decision and when the selection ratio was stringent, simulated scenarios showed that using an SJT in addition to cognitive tests might enable admissions boards to select more students from lower SES backgrounds and more female students. CONCLUSIONS: The SJT has the potential to appropriately complement cognitive tests in the selection of doctors and dentists. It may also put candidates of lower SES backgrounds at less of a disadvantage and may potentially diversify the student intake. However, use of the SJT applied in this study did not diminish the role of ethnicity. Future research should examine these findings with other SJTs and other tests internationally and scrutinise the causes underlying the role of ethnicity.

Differences between multimedia and text-based assessments of emotion management: An exploration with the multimedia emotion management assessment (MEMA).

People process emotional information using visual, vocal, and verbal cues. However, emotion management is typically assessed with text based rather than multimedia stimuli. This study (N = 427) presents the new multimedia emotion management assessment (MEMA) and compares it to the text-based assessment of emotion management used in the MSCEIT. The text-based and multimedia assessment showed similar levels of cognitive saturation and similar prediction of relevant criteria. Results demonstrate that the MEMA scores have equivalent evidence of validity to the text-based MSCEIT test scores, demonstrating that multimedia assessment of emotion management is viable. Furthermore, our results inform the debate as to whether cognitive saturation in emotional intelligence (EI) measures represents "noise" or "substance". We find that cognitive ability associations with EI represent substantive variance rather than construct-irrelevant shared variance due to reading comprehension ability required for text-based items.

The image of psychology programs: the value of the instrumental-symbolic framework.

As competition for funding and students intensifies, it becomes increasingly important for psychology programs to have an image that is attractive and makes them stand out from other programs. The current study uses the instrumental-symbolic framework from the marketing domain to determine the image of different master's programs in psychology and examines how these image dimensions relate to student attraction and competitor differentiation. The samples consist of both potential students (N = 114) and current students (N = 68) of three psychology programs at a Belgian university: industrial and organizational psychology, clinical psychology, and experimental psychology. The results demonstrate that both instrumental attributes (e.g., interpersonal activities) and symbolic trait inferences (e.g., sincerity) are key components of the image of psychology programs and predict attractiveness as well as differentiation. In addition, symbolic image dimensions seem more important for current students of psychology programs than for potential students.

The chemistry between us: Illuminating complementarity patterns in interpersonal role-play assessment via moment-to-moment analyses.

In assessment and selection, organizations often include interpersonal interactions because they provide insights into candidates' interpersonal skills. These skills are then typically assessed via one-shot, retrospective assessor ratings. Unfortunately, the assessment of interpersonal skills at such a trait-like level fails to capture the richness of how the interaction unfolds at the behavioral exchange level within a role-play assessment. This study uses the lens of interpersonal complementarity theory to advance our understanding of interpersonal dynamics in role-play assessment and their effects on assessor ratings. Ninety-six MBA students participated in four different flash role-plays as part of diagnosing their strengths and weaknesses. Apart from gathering assessor ratings and criterion measures, coders also conducted a fine-grained examination of how the behavior of the two interaction partners (i.e., MBA students and role-players) unfolded at the moment-to-moment level via the Continuous Assessment of Interpersonal Dynamics (CAID) measurement tool. In all role-plays, candidates consistently showed mutual adaptations in line with complementarity principles: Affiliative behavior led to affiliative behavior, whereas dominant behavior resulted in docile, following behavior and vice versa. For affiliation, mutual influence also occurred in that both interaction partners' temporal trends in affiliation were entrained over time. Complementarity patterns were significantly related to ratings of in situ (role-playing) assessors but not to ratings of ex situ (remote) assessors. The effect of complementarity on validity was mixed. Overall, this study highlights the importance of going beyond overall ratings to capture behavioral contingencies such as complementarity patterns in interpersonal role-play assessment. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Politics speak louder than skills: Political similarity effects in hireability judgments in multiparty contexts and the role of political interest.

Recruiters increasingly cybervet job applicants by checking their social media profiles. Theory (i.e., the political affiliation model, PAM) and research show that during cybervetting, recruiters are exposed to job-unrelated information such as political affiliation, which might trigger similarity-attraction effects and bias hireability judgments. However, as the PAM was developed in a more polarized two-party political system, it is pivotal to test and refine the PAM in a multiparty context. Therefore, we asked working professionals from the United States (two-party context, N = 266) and Germany (multiparty context, N = 747) to rate an applicant's hireability after cybervetting a LinkedIn profile that was manipulated in a between-subjects design (party affiliation by individuating information). Key tenets of the PAM could be transferred to multiparty contexts: The political similarity-attraction effect predicted hireability judgments beyond job-related individuating information, especially regarding organizational citizenship behavior. In addition, in a multiparty context, these biasing effects of political similarity and liking were not attenuated. Yet, there were also differences: In a multiparty context, political similarity had to be operationalized in terms of political value similarity and recruiters' political interest emerged as a significant moderator of the effects. So, this study refines the PAM by showing in multiparty contexts the importance of (a) a values-based perspective (instead of a behavioral political affiliation perspective) and (b) political interest (instead of identification). Accordingly, we provide a more nuanced understanding of when political affiliation similarity contributes to perceived overall similarity in affecting liking and hireability judgments in cybervetting. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Designing pareto-optimal selection systems for multiple minority subgroups and multiple criteria.

Currently used Pareto-optimal (PO) approaches for balancing diversity and validity goals in selection can deal only with one minority group and one criterion. These are key limitations because the workplace and society at large are getting increasingly diverse and because selection system designers often have interest in multiple criteria. Therefore, the article extends existing methods for designing PO selection systems to situations involving multiple criteria and multiple minority groups (i.e., multiobjective PO selection systems). We first present a hybrid multiobjective PO approach for computing selection systems that are PO with respect to (a) a set of quality objectives (i.e., criteria) and (b) a set of diversity objectives where each diversity objective relates to a different minority group. Next, we propose three two-dimensional subspace procedures that aid selection designers in choosing between the PO systems in case of a high number of quality and diversity objectives. We illustrate our novel multiobjective PO approaches via several example applications, thereby demonstrating that they are the first to reveal the complete gamut of eligible PO selection designs and to faithfully capture the Pareto trade-off front in case of more than two objectives. In addition, a small-scale cross-validation study confirms that the resulting PO selection designs retain an advantage over alternative designs when applied in new validation samples. Finally, the article provides a link to an executable code to perform the new multiobjective PO approaches. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Insights from an updated personnel selection meta-analytic matrix: Revisiting general mental ability tests' role in the validity-diversity trade-off.

General mental ability (GMA) tests have long been at the heart of the validity-diversity trade-off, with conventional wisdom being that reducing their weight in personnel selection can improve adverse impact, but that this results in steep costs to criterion-related validity. However, Sackett et al. (2022) revealed that the criterion-related validity of GMA tests has been considerably overestimated due to inappropriate range restriction corrections. Thus, we revisit the role of GMA tests in the validity-diversity trade-off using an updated meta-analytic correlation matrix of the relationships six selection methods (biodata, GMA tests, conscientiousness tests, structured interviews, integrity tests, and situational judgment tests) have with job performance, along with their Black-White mean differences. Our results lead to the conclusion that excluding GMA tests generally has little to no effect on validity, but substantially decreases adverse impact. Contrary to popular belief, GMA tests are not a driving factor in the validity-diversity trade-off. This does not fully resolve the validity-diversity trade-off, though: Our results show there is still some validity reduction required to get to an adverse impact ratio of .80, although the validity reduction is less than previously thought. Instead, it shows that the validity-diversity trade-off conversation should shift from the role of GMA tests to that of other selection methods. The present study also addresses which selection methods now emerge as most valid and whether composites of selection methods can result in validities similar to those expected prior to Sackett et al. (2022). (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Adjusting medical school admission: assessing interpersonal skills using situational judgement tests.

CONTEXT: Today's formal medical school admission systems often include only cognitively oriented tests, although most medical school curricula emphasise both cognitive and non-cognitive factors. Situational judgement tests (SJTs) may represent an innovative approach to the formal measurement of interpersonal skills in large groups of candidates in medical school admission processes. This study examined the validity of interpersonal video-based SJTs in relation to a variety of outcome measures. METHODS: This study used a longitudinal and multiple-cohort design to examine anonymised medical school admissions and medical education data. It focused on data for the Flemish medical school admission examination between 1999 and 2002. Participants were 5444 candidates taking the medical school admission examination. Outcome measures were first-year grade point average (GPA), GPA in interpersonal communication courses, GPA in non-interpersonal courses, Bachelor's degree GPA, Master's degree GPA and final-year GPA (after 7 years). For students pursuing careers in general practice, additional outcome measures (9 years after sitting examinations) included supervisor ratings and the results of an interpersonal objective structured clinical examination (OSCE), a general practice knowledge test and a case-based interview. RESULTS: Interpersonal skills assessment carried out using SJTs had significant added value over cognitive tests for predicting interpersonal GPA throughout the curriculum, doctor performance, and performance on an OSCE and in a case-based interview. For the other outcomes, cognitive tests emerged as the better predictors. Females significantly outperformed males on the SJT (d = -0.26). The interpersonal SJT was perceived as significantly more job-related than the cognitive tests (d = 0.55). CONCLUSIONS: Video-based SJTs as measures of procedural knowledge about interpersonal behaviour show promise as complements to cognitive examination components. The interpersonal skills training received during medical education does not negate the selection of students on the basis of interpersonal skills. Future research is needed to examine the use of SJTs in other cultures and student populations.

Multiple, speeded assessments under scrutiny: Underlying theory, design considerations, reliability, and validity.

Recently, multiple, speeded assessments (e.g., "speeded" or "flash" role-plays) have made rapid inroads into the selection domain. So far, however, the conceptual underpinning and empirical evidence related to these short, fast-paced assessment approaches has been lacking. This raises questions whether these speeded assessments can serve as reliable and valid indicators of future performance. This article uses the notions of stimulus and response domain sampling to conceptualize multiple, speeded behavioral job simulations as a hybrid of established simulation-based selection methods. Next, we draw upon the thin slices of behavior paradigm to theorize about the quality of ratings made in multiple, speeded behavioral simulations. In two studies, various assessor pools assessed a sample of 96 MBA students in 18 3-min role-plays designed to capture situations in the junior management domain. At the individual speeded role-play level, reliability and validity were not ensured. Yet, aggregated across all assessors' ratings of all speeded role-plays, the overall score for predicting future performance was high (.54). Validities remained high when assessors evaluated only the first minute (vs. full 3 min) or received only a control training (vs. traditional assessor training). Aggregating ratings of performance in multiple, heterogeneous situations that elicit a variety of domain-relevant behavior emerged as key requirement to obtain adequate domain coverage, capture both ability and personality (extraversion and agreeableness), and achieve substantial validities. Overall, these results show the importance of the stimulus and response domain sampling logic and send a strong warning to using "single" speeded behavioral simulations in practice. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Arousing Motives or Eliciting Stories? On the Role of Pictures in a Picture-Story Exercise.

Picture-story exercises (PSE) form a popular measurement approach that has been widely used for the assessment of implicit motives. However, current theorizing offers two diverging perspectives on the role of pictures in PSEs: either to elicit stories or to arouse motives. In the current study, we tested these perspectives in an experimental design. We administered a PSE either with or without pictures. Results from N = 281 participants revealed that the experimental manipulation had a medium to large effect for the affiliation and power motive domains, but no effect for the achievement motive domain. We conclude that the herein chosen pictures cues function differentially across motives, as they aroused the affiliation and power motives, but not the achievement motive.

Correcting for range restriction in meta-analysis: A reply to Oh et al. (2023).

Oh et al. (2023) question a number of choices made in our article (Sackett et al., 2022); here we respond. They interpret our article as recommending against correcting for range restriction in general in concurrent validation studies; yet, we emphasize that we endorse correction when one has access to the information needed to do so. Our focus was on making range restriction corrections when conducting meta-analyses, where it is common for primary studies to be silent as to the prior basis for selection of the employees later participating in the concurrent validation study. As such, the applicant pool information needed for correction is typically not available. Sackett et al. (2022) highlighted that in many situations, range restriction will be small; so, the inability to correct for it results in only a modest underestimate of validity. Oh et al. mention settings that would result in substantial range restriction; here, we present our rationale as to why we view such settings as uncommon rather than as making up the bulk of the studies contributing to meta-analyses. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Validity evidence for the situational judgment test paradigm in emotional intelligence measurement.

To date, various measurement approaches have been proposed to assess emotional intelligence (EI). Recently, two new EI tests have been developed based on the situational judgment test (SJT) paradigm: the Situational Test of Emotional Understanding (STEU) and the Situational Test of Emotion Management (STEM). Initial attempts have been made to examine the construct-related validity of these new tests; we extend these findings by placing the tests in a broad nomological network. To this end, 850 undergraduate students completed a personality inventory, a cognitive ability test, a self-report EI test, a performance-based EI measure, the STEU, and the STEM. The SJT-based EI tests were not strongly correlated with personality and fluid cognitive ability. Regarding their relation with existing EI measures, the tests did not capture the same construct as self-report EI measures, but corresponded rather to performance-based EI measures. Overall, these results lend support for the SJT paradigm for measuring EI as an ability.

A comprehensive examination of the cross-validity of pareto-optimal versus fixed-weight selection systems in the biobjective selection context.

The article presents evidence for the cross-validity potential of fixed-weight (FW) versus Pareto-Optimal (PO) selection systems in biobjective selection situations where both the goals of diversity and quality are valued and the importance of the goals is undecided a priori. The article extends previous research by also studying the cross-validity potential of selection systems in the practically most important sample-to-sample cross-validity scenario. We address three research questions: (a) Do different PO systems show comparable levels of relative (i.e., proportional) achievement upon cross-validation? (b) Do PO systems achieve higher levels of relative achievement upon cross-validation than FW selection systems?, and (c) How does the achievement of PO and FW systems, in terms of adverse impact ratios and average performance of the selected applicants, evolve under cross-validation? As a key result, in case of sufficiently large applicant pools (typically 100 applicants or more), PO systems had on average a higher cross-validity potential than the corresponding FW systems. Yet, even for applicant pools as large as 500, FW systems may match the merits of PO systems and we present a straightforward procedure to decide which FW systems may offer a comparable cross-validation potential than the PO systems. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Revisiting meta-analytic estimates of validity in personnel selection: Addressing systematic overcorrection for restriction of range.

This paper systematically revisits prior meta-analytic conclusions about the criterion-related validity of personnel selection procedures, and particularly the effect of range restriction corrections on those validity estimates. Corrections for range restriction in meta-analyses of predictor-criterion relationships in personnel selection contexts typically involve the use of an artifact distribution. After outlining and critiquing five approaches that have commonly been used to create and apply range restriction artifact distributions, we conclude that each has significant issues that often result in substantial overcorrection and that therefore the validity of many selection procedures for predicting job performance has been substantially overestimated. Revisiting prior meta-analytic conclusions produces revised validity estimates. Key findings are that most of the same selection procedures that ranked high in prior summaries remain high in rank, but with mean validity estimates reduced by .10-.20 points. Structured interviews emerged as the top-ranked selection procedure. We also pair validity estimates with information about mean Black-White subgroup differences per selection procedure, providing information about validity-diversity tradeoffs. We conclude that our selection procedures remain useful, but selection predictor-criterion relationships are considerably lower than previously thought. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

The role of emotions as mechanisms of mid-test warning messages during personality testing: A field experiment.

This study focuses on the role of emotions in personnel selection and faking research. In particular, we posit that emotions are likely to be activated when applicants receive warning messages from organizations. Drawing on Nabi (Nabi, Communication Theory, 9, 1999, 292) cognitive-functional model of discrete negative emotions, we propose and empirically test the effects of three discrete negative emotions (guilt, fear, and anger) triggered by a warning message during a personality test on personality score accuracy and perceived test fairness. Participants in this within-subjects field experiment were 1,447 applicants for graduate school at a large public university in China. They completed two parallel forms of a personality test: one within a selection context, and another within a developmental context 6 months later as a baseline measure. In the selection context, a warning (or a control) message was randomly assigned to participants during the personality test. Emotions and perceived test fairness were measured after the test was completed. Results indicated that guilt, fear, and anger each played a unique role. Guilt explained how mid-test warnings improved personality score accuracy among fakers, whereas fear accounted for why nonfakers over-corrected their personality scores. Finally, anger explained why the mid-test warnings reduced perceived test fairness for both fakers and nonfakers. Theoretical and practical implications are discussed. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Prompt-specificity in scenario-based assessments: Associations with personality versus knowledge and effects on predictive validity.

Many scenario-based assessments (e.g., interviews, assessment center exercises, work samples, simulations, and situational judgment tests) use prompts (i.e., cues provided to respondents to increase the likelihood that the information received from them is clear, sufficient, and job-related). However, a dilemma for practitioners and researchers is how general or specific one should prompt people's answers. We posit that such differences in prompt-specificity (i.e., extent to which prompts cue performance criteria) have important implications for the predictive validity of scenario-based assessment scores. Drawing on the interplay of situation construal and situational strength theory, we propose that prompt-specificity leads to differential relationships between scenario-based scores and external constructs (personality traits vs. knowledge), which in turn affects the predictive validity of scenario-based assessments. We tested this general hypothesis using intercultural scenarios for predicting effectiveness in multicultural teams. Using a randomized predictive validation design, we contrast scores on these scenarios with general (N = 157) versus specific (N = 158) prompts. As a general conclusion, prompt-specificity mattered: Lesser prompt-specificity augmented the role of perspective taking and openness-to-experience in the intercultural scenario scores and their validity for predicting intercultural performance, whereas greater prompt-specificity increased the role of knowledge in these scores and their validity for predicting in-role performance. This study's theoretical and practical implications go beyond a specific assessment procedure and apply to a broad array of assessment and training approaches that rely on scenarios. (PsycInfo Database Record (c) 2021 APA, all rights reserved).