Inferring signs from purposeful samples: The role of context in competency assessment.

CONTEXT: Medical students' clinical competencies are customarily assessed using convenience samples of performance from real practice. The question is how these convenience samples can be turned into purposeful samples to extrapolate students' overall competency profile at the end of medical school, particularly given the context specificity of clinical performance. In this paper, we will address this issue of inferring signs from samples using insights from the discipline of psychology. THEORETICAL PERSPECTIVE: We adapted Smith's theory of predictor validity of universals, occupationals and relationals to the context of clinical competency assessment. Universals are characteristics required by all working individuals and therefore not context dependent. Occupationals refer to characteristics required by certain jobs but not others and therefore are dependent on task-related features of an occupation. Relationals are required in a specific organisational context with habitual ways of working together. APPLICATION: Through seven propositions, we assert that generalising from samples of assessed clinical competencies during clerkships to generic competencies (i.e., signs) is dependent on whether characteristics are universals, occupationals and relationals, with universals most and relationals least generalisable. CONCLUSION: When determining what types of ratings to use to evaluate medical student competence, medical education has shown too little nuance in considering the degree to which particular characteristics are likely to be generalisable, approaching the issue in an all-or-none manner. Smith's distinction between universals, occupationals and relationals offers a promising way forward that has implications for assessment, student selection and career choice.

Broadly sampled assessment reduces ethnicity-related differences in clinical grades.

CONTEXT: Ethnicity-related differences in clinical grades exist. Broad sampling in assessment of clinical competencies involves multiple assessments used by multiple assessors across multiple moments. Broad sampling in assessment potentially reduces irrelevant variances and may therefore mitigate ethnic disparities in clinical grades. OBJECTIVES: Research question 1 (RQ1): to assess whether the relationship between students' ethnicity and clinical grades is weaker in a broadly sampled versus a global assessment. Research question 2 (RQ2): to assess whether larger ethnicity-related differences in grades occur when supervisors are given the opportunity to deviate from the broadly sampled assessment score. METHODS: Students' ethnicity was classified as Turkish/Moroccan/African, Surinamese/Antillean, Asian, Western, and native Dutch. RQ1: 1667 students (74.3% native Dutch students) were included, who entered medical school between 2002 and 2004 (global assessment, 818 students) and between 2008 and 2010 (broadly sampled assessment, 849 students). The main outcome measure was whether or not students received ≥3 times a grade of 8 or higher on a scale from 1 to 10 in five clerkships. RQ2: 849 students (72.4% native Dutch students) were included, who were assessed by broad sampling. The main outcome measure was the number of grade points by which supervisors had deviated from broadly sampled scores. Both analyses were adjusted for gender, age, (im)migration status and average bachelor grade. RESULTS: Research question 1: ethnicity-related differences in clinical grades were smaller in broadly sampled than in global assessment, and this was also seen after adjustments. More specifically, native Dutch students had reduced probabilities (0.87-0.65) in broadly sampled as compared with global assessment, whereas Surinamese (0.03-0.51) and Asian students (0.21-0.30) had increased probabilities of having ≥3 times a grade of 8 or higher in five clerkships. Research question 2: when supervisors were allowed to deviate from original grades, ethnicity-related differences in clinical grades were reintroduced. CONCLUSIONS: Broadly sampled assessment reduces ethnicity-related differences in grades.

Integrity situational judgement test for medical school selection: judging 'what to do' versus 'what not to do'.

CONTEXT: Despite their widespread use in medical school selection, there remains a lack of clarity on exactly what situational judgement tests (SJTs) measure. OBJECTIVES: We aimed to develop an SJT that measures integrity by combining critical incident interviews (inductive approach) with an innovative deductive approach. The deductive approach guided the development of the SJT according to two established theoretical models, of which one was positively related to integrity (honesty-humility [HH]) and one was negatively related to integrity (cognitive distortions [CD]). The Integrity SJT covered desirable (HH-based) and undesirable (CD-based) response options. We examined the convergent and discriminant validity of the Integrity SJT and compared the validity of the HH-based and CD-based subscores. METHODS: The Integrity SJT was administered to 402 prospective applicants at a Dutch medical school. The Integrity SJT consisted of 57 scenarios, each followed by four response options, of which two represented HH facets and two represented CD categories. Three SJT scores were computed, including a total, an HH-based and a CD-based score. The validity of these scores was examined according to their relationships with external integrity-related measures (convergent validity) and self-efficacy (discriminant validity). RESULTS: The three SJT scores correlated significantly with all integrity-related measures and not with self-efficacy, indicating convergent and discriminant validity. In addition, the CD-based SJT score correlated significantly more strongly than the HH-based SJT score with two of the four integrity-related measures. CONCLUSIONS: An SJT that assesses the ability to correctly recognise CD-based response options as inappropriate (i.e. what one should not do) seems to have stronger convergent validity than an SJT that assesses the ability to correctly recognise HH-based response options as appropriate (i.e. what one should do). This finding might be explained by the larger consensus on what is considered inappropriate than on what is considered appropriate in a challenging situation. It may be promising to focus an SJT on the ability to recognise what one should not do.

Influence of response instructions and response format on applicant perceptions of a situational judgement test for medical school selection.

BACKGROUND: This study examined the influence of two Situational Judgement Test (SJT) design features (response instructions and response format) on applicant perceptions. Additionally, we investigated demographic subgroup differences in applicant perceptions of an SJT. METHODS: Medical school applicants (N = 372) responded to an online survey on applicant perceptions, including a description and two example items of an SJT. Respondents randomly received one of four SJT versions (should do-rating, should do-pick-one, would do-rating, would do-pick-one). They rated overall favourability and items on four procedural justice factors (face validity, applicant differentiation, study relatedness and chance to perform) and ease-of-cheating. Additionally, applicant perceptions were compared for subgroups based on gender, ethnic background and first-generation university status. RESULTS: Applicants rated would-do instructions as easier to cheat than should-do instructions. Rating formats received more favourable judgements than pick-one formats on applicant differentiation, study-relatedness, chance to perform and ease of cheating. No significant main effect for demographic subgroup on applicant perceptions was found, but significant interaction effects showed that certain subgroups might have more pronounced preferences for certain SJT design features. Specifically, ethnic minority applicants - but not ethnic majority applicants - showed greater preference for should-do than would-do instructions. Additionally, first-generation university students - but not non-first-generation university students - were more favourable of rating formats than of pick-one formats. CONCLUSIONS: Findings indicate that changing SJT design features may positively affect applicant perceptions by promoting procedural justice factors and reducing perceived ease of cheating and that response instructions and response format can increase the attractiveness of SJTs for minority applicants.

Participation in a scientific pre-university program and medical students' interest in an academic career.

BACKGROUND: The proportion of medical doctors involved in research activities is declining. Undergraduate medical research programs are positively associated with medical students' research interest. Scientific pre-university programs (SPUPs) outside the medical domain are also positively associated with research interest, but have not been related to the shortage of clinician-scientists. This study examined the effect of an SPUP on medical students' research interest. METHODS: This study was conducted at a Dutch medical school. Medical students in all years who had participated in an SPUP and first-year master students who had not participated in an SPUP were invited to fill out an online survey on extracurricular activities and future career interests. SPUP participants were compared with three groups of non-participants: (i) an unmatched group, (ii) a group matched on gender and pre-university Grade Point Average (pu-GPA) and (iii) a group matched on gender and first-year GPA, one to five years after finishing the SPUP. Participants evaluated the SPUP through ratings of statements about the program. RESULTS: Two-hundred forty medical students, including 71 SPUP participants responded to the survey. SPUP participants participated significantly more often in the Honors class (i.e., extracurricular educational program for high-performing students), gained significantly more often extracurricular research experience, enrolled significantly more often in the Research master (i.e., research training program parallel to the clinical master program) and obtained significantly more often a scholarship than unmatched non-SPUP participants. Using a non-SPUP group matched on gender and pu-GPA reduced the effect size of the significant differences in Honors class participation, Research master participation and scholarship obtainment. Using a non-SPUP group matched on gender and first-year GPA rendered the significant difference in Research master participation and scholarship obtainment insignificant. Significantly more SPUP participants than unmatched non-SPUP participants preferred a combination of clinical care and research in their future career. Using a non-SPUP group matched on gender and either pu-GPA or first-year GPA did not change the effect size of this significant difference. CONCLUSIONS: These findings demonstrate the potential value of an SPUP in increasing the number of medical students with research interest and as a policy measure to help to alleviate the shortage of clinician-scientists.

Psychological predictors of cultural diversity support at work.

OBJECTIVES: As diversity management activities become more prominent worldwide it is important to understand psychological reactions to them to ensure success, but empirical evidence is lacking. This study investigated employees' and managers' intentions and behavior to promote cultural diversity at work in a variety of organizations in the Netherlands, using Ajzen's theory of planned behavior. METHOD: Predictors of intentions to promote cultural diversity at work (N = 670) and actual behavior after 6 months were assessed among managers and employees using self-reports in a 2-wave survey design. Participants' average age at Time 1 was 38.26 years (SD = 11.86), 56% was female, and there were 78.1% Dutch ethnic majority and 21.9% ethnic minority participants. RESULTS: Attitude to cultural diversity promotion at work and perceived behavioral control (PBC) related positively to both individuals' intentions to promote cultural diversity at work, which in turn predicted behavior. The strongest driver, however, was attitude. Managers' reported PBC and behavior were higher compared to employees. CONCLUSIONS: This study supported the applicability of the theory of planned behavior to predict intentions and behavior to promote cultural diversity at work. With an increasingly diverse workforce, this study aimed to advance our understanding of drivers of individual reactions and behavior to support cultural diversity at work. (PsycINFO Database Record

The family-study interface and academic outcomes: differences and similarities between ethnic minority and ethnic majority students.

The present study investigated possible differences in the family-study interface between ethnic minority and ethnic majority students as an explanation for the poorer study results of ethnic minority students compared with those of majority students. We used a model for family-study conflict and facilitation derived from family-work and work-study models. This model held true for the full sample and both non-Western ethnic minority students (N = 342) and ethnic majority students (N = 1314) separately at a major Dutch university. Multivariate analyses of variance revealed that ethnic minority students reported less study effort and earned lower grades compared with ethnic majority students. Regarding the family-study interface, ethnic minority students reported more family-study conflict than did ethnic majority students. No differences were found between the 2 groups in family-study facilitation. Ethnic minority students participated more in family activities and were more involved with their family than ethnic majority students. Levels of experienced family support were equal for both groups of students. Students who received more family social support reported less conflict and more facilitation. This latter finding held more strongly for majority students, resulting in more study effort and higher grades for this group. The results demonstrated the explanatory power of the family-study conflict and facilitation model for both groups.

[Minority GP in training more likely to 'underperform': who is to blame?] / Aios met migratieachtergrond vaker 'onderpresteerder'.

Recent work revealed that ethnic minority gp-trainees are more at risk of underperformance than their majority peers. We argue that causes for underperformance can be identified from two perspectives, namely that of the trainees ('them') and that of the assessors and the institution ('us'). Potential impeding factors from the student perspective include lack of practical clinical skills and differences in communication styles. At the level of the assessors, it is important to consider in-group bias and individual assessors' perceptions and preferences. Finally, possible factors at the institutional level are type of evaluation system used and organizational culture. Questions are raised regarding the existence of one golden rule for doctor-patient communication and the role of the hidden curriculum in negatively influencing social and academic outcomes for minority trainees. The urge is for creating an inclusive learning environment ensuring psychological safety and talent usage for all our future doctors.

The predictive validity of criminogenic needs for male and female offenders: comparing the relative impact of needs in predicting recidivism.

Most instruments used to assess offenders' risk of recidivism were developed and validated on male samples. Use of these instruments with female offenders is, however, common practice. This use with female offenders implies the assumption that the risk of recidivism can be predicted on the basis of the same risk factors for women as for men. Yet, this implied gender-neutrality of offender risk instruments has been the topic of much debate. This study compared criminogenic needs in male and female offenders and their relevance in predicting recidivism. A large sample of male and female offenders (N = 16,239) charged with a range of index offenses was studied. Results mainly support the gender neutrality of existing offender risk and needs assessment. However, results do suggest that some criminogenic needs may indeed have a different impact on recidivism for men and women. Problems with accommodation, education and work, and relationships with friends were more strongly correlated to general recidivism in men than in women. For women, difficulties with emotional well-being had a stronger correlation with recidivism than for men. In addition, relative to all other criminogenic needs, problems with emotional well-being were more important for women than for men in predicting general as well as violent recidivism. However, because the bivariate correlation for female offenders between emotional difficulties and recidivism is weak (as it is for male offenders), the question remains whether the relative importance of emotional difficulties in predicting recidivism in women actually has clinical relevance.

The Motivation and Opportunity for Socially Desirable Responding Does Not Alter the General Factor of Personality.

Socially desirable responding may affect the factor structure of personality questionnaires and may be one of the reasons for the common variance among personality traits. In this study, we test this hypothesis by investigating the influence of the motivational test-taking context (development vs. selection) and the opportunity to distort responses (forced-choice vs. Likert response format) on personality questionnaire scores. Data from real selection and assessment candidates (total N = 3,980) matched on gender, age, and educational level were used. Mean score differences were found between the selection and development groups, with smaller differences for the FC version. Yet, exploratory structural equation models showed that the overall factor structures as well as the general factor were highly similar across the four groups. Thus, although socially desirable responding may affect mean scores on personality traits, it does not appear to affect factor structures. This study further suggests that the common variance in personality questionnaires is consistent and appears to be little influenced by motivational pressures for response distortion.

The consequences of self- and other-focused emotional intelligence: Not all sunshine and roses.

Emotional intelligence (EI) contributes to good performance and well-being in jobs that involve frequent interpersonal contact. However, as EI is composed of self- and other-focused dimensions, it remains unclear which dimensions are responsible for better performance and well-being. We hypothesized that other-focused EI dimensions in particular relate to task performance, whereas self-focused EI dimensions relate to employees' subjective stress and physiological responses to emotional job demands. We asked Dutch secretaries (N = 110) to professionally respond to five emotionally demanding work-related phone calls. The secretaries' skin conductance levels were recorded during the calls, and the secretaries had to indicate their stress levels after each call. Two independent raters coded the secretaries' effectiveness and the number of emotion regulation attempts during the phone calls. The results showed that other-focused emotion regulation was positively related to only one of the task performance indicators during three phone calls. In line with the hypotheses, self-focused emotion appraisal was negatively related to the secretaries' subjective stress levels after all the phone calls. Self-focused emotion regulation was positively related to the secretaries' skin conductance levels during all but one of the phone calls. This outcome suggests that self-focused EI dimensions decrease the subjective experience of stress but are accompanied by physiological costs, whereas other-focused emotion regulation may be positively but weakly related to task performance in emotionally demanding contexts. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Intentional response distortion on personality tests: using eye-tracking to understand response processes when faking.

Intentional response distortion or faking among job applicants completing measures such as personality and integrity tests is a concern in personnel selection. The present study aimed to investigate whether eye-tracking technology can improve our understanding of the response process when faking. In an experimental within-participants design, a Big Five personality test and an integrity measure were administered to 129 university students in 2 conditions: a respond honestly and a faking good instruction. Item responses, response latencies, and eye movements were measured. Results demonstrated that all personality dimensions were fakeable. In support of the theoretical position that faking involves a less cognitively demanding process than responding honestly, we found that response times were on average 0.25 s lower and participants had less eye fixations in the fake good condition [corrected]. However, in the fake good condition, participants had more fixations on the 2 extreme response options of the 5-point answering scale, and they fixated on these more directly after having read the question. These findings support the idea that faking leads to semantic rather than self-referenced item interpretations. Eye-tracking was demonstrated to be potentially useful in detecting faking behavior, improving detecting rates over and beyond response extremity and latency metrics.

The maladaptive personality traits of the Personality Inventory for DSM-5 (PID-5) in relation to the HEXACO personality factors and schizotypy/dissociation.

The Personality Inventory for DSM-5 (PID-5), a new measure of maladaptive personality traits, has recently been developed by the DSM-5 Personality and Personality Disorders Workgroup. The PID-5 variables were examined within the seven-factor space defined by the six HEXACO factors and the Schizotypy/Dissociation factor (Ashton & Lee, 2012) using participant samples from Canada (N = 378) and the Netherlands (N = 476). Extension analyses showed that several PID-5 facet-level scales represented each of the Honesty-Humility, Emotionality, Extraversion, Conscientiousness, and Schizotypy/Dissociation factors. In contrast, only one PID-5 scale loaded strongly on HEXACO Agreeableness, and no PID-5 scales loaded strongly on Openness to Experience. In addition, a joint factor analysis involving the PID-5 variables and facets of the Five-Factor Model was conducted in the Canadian sample and recovered a set of seven factors corresponding rather closely to the HEXACO factors plus Schizotypy/Dissociation. The authors discuss implications for the assessment and structure of normal and abnormal personality.