Development of a taxonomy of unprofessional behavior in clinical learning environments using learner-generated critical incidents.

PURPOSE: Few studies have examined medical residents' and fellows' (trainees) direct experience of unprofessional behavior in clinical learning environments (CLE). The purpose of this study was to create a taxonomy of unprofessional behavior in CLEs using critical incidents gathered from trainees. METHOD: In step 1 (data collection), the authors collected 382 critical incidents from trainees at more than a dozen CLEs over a six-year period (2013-2019). In step 2 (model generation), nine subject matter experts (SMEs) sorted the incidents into homogenous clusters and this structure was tested with principal components analysis (PCA). In step 3 (model evaluation), two new groups of SMEs each re-sorted half of the incidents into the PCA-derived categories. RESULTS: A 13-component solution accounted for 62.46% of the variance in the critical incidents collected. The SMEs who re-sorted the critical incidents demonstrated good agreement with each other and with the 13-component PCA solution. The resulting taxonomy included 13 dimensions, with 48.7% of behaviors focused on displays of aggression or discriminatory conduct. CONCLUSIONS: Critical incident methodology can provide unique insights into the dimensionality of unprofessional behavior in the CLE. Future research should leverage the taxonomy created to inform professionalism assessment development in the CLE.

Improving Our Ability to Predict Resident Applicant Performance: Validity Evidence for a Situational Judgment Test.

Construct: We investigated whether a situational judgment test (SJT) designed to measure professionalism in physicians predicts residents' performance on (a) Accreditation Council for Graduate Medical Education (ACGME) competencies and (b) a multisource professionalism assessment (MPA). Background: There is a consensus regarding the importance of assessing professionalism and interpersonal and communication skills in medical students, residents, and practicing physicians. Nonetheless, these noncognitive competencies are not well measured during medical education selection processes. One promising method for measuring these noncognitive competencies is the SJT. In a typical SJT, respondents are presented with written or video-based scenarios and asked to make choices from a set of alternative courses of action. Interpersonally oriented SJTs are commonly used for selection to medical schools in the United Kingdom and Belgium and for postgraduate selection of trainees to medical practice in Belgium, Singapore, Canada, and Australia. However, despite international evidence suggesting that SJTs are useful predictors of in-training performance, end-of-training performance, supervisory ratings of performance, and clinical skills licensing objective structured clinical examinations, the use of interpersonally oriented SJTs in residency settings in the United States has been infrequently investigated. The purpose of this study was to investigate whether residents' performance on an SJT designed to measure professionalism-related competencies-conscientiousness, integrity, accountability, aspiring to excellence, teamwork, stress tolerance, and patient-centered care-predicts both their current and future performance as residents on two important but conceptually distinct criteria: ACGME competencies and the MPA. Approach: We developed an SJT to measure seven dimensions of professionalism. During calendar year 2017, 21 residency programs from 2 institutions administered the SJT. We conducted analyses to determine the validity of SJT and USMLE scores in predicting milestone performance in ACGME core competency domains and the MPA in June 2017 and 3 months later in September 2017 for the MPA and 1 year later, in June 2018, for ACGME domains. Results: At both periods, the SJT score predicted overall ACGME milestone performance (r = .13 and .17, respectively; p < .05) and MPA performance (r = .19 and .21, respectively; p < .05). In addition, the SJT predicted ACGME patient care, systems-based practice, practice-based learning and improvement, interpersonal and communication skills, and professionalism competencies (r = .16, .15, .15, .17, and .16, respectively; p < .05) 1 year later. The SJT score contributed incremental validity over USMLE scores in predicting overall ACGME milestone performance (ΔR = .07) 1 year later and MPA performance (ΔR = .05) 3 months later. Conclusions: SJTs show promise as a method for assessing noncognitive attributes in residency program applicants. The SJT's incremental validity to the USMLE series in this study underscores the importance of moving beyond these standardized tests to a more holistic review of candidates that includes both cognitive and noncognitive measures.

Insights from an updated personnel selection meta-analytic matrix: Revisiting general mental ability tests' role in the validity-diversity trade-off.

General mental ability (GMA) tests have long been at the heart of the validity-diversity trade-off, with conventional wisdom being that reducing their weight in personnel selection can improve adverse impact, but that this results in steep costs to criterion-related validity. However, Sackett et al. (2022) revealed that the criterion-related validity of GMA tests has been considerably overestimated due to inappropriate range restriction corrections. Thus, we revisit the role of GMA tests in the validity-diversity trade-off using an updated meta-analytic correlation matrix of the relationships six selection methods (biodata, GMA tests, conscientiousness tests, structured interviews, integrity tests, and situational judgment tests) have with job performance, along with their Black-White mean differences. Our results lead to the conclusion that excluding GMA tests generally has little to no effect on validity, but substantially decreases adverse impact. Contrary to popular belief, GMA tests are not a driving factor in the validity-diversity trade-off. This does not fully resolve the validity-diversity trade-off, though: Our results show there is still some validity reduction required to get to an adverse impact ratio of .80, although the validity reduction is less than previously thought. Instead, it shows that the validity-diversity trade-off conversation should shift from the role of GMA tests to that of other selection methods. The present study also addresses which selection methods now emerge as most valid and whether composites of selection methods can result in validities similar to those expected prior to Sackett et al. (2022). (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Challenging conclusions about predictive bias against Hispanic test takers in personnel selection.

Berry et al. (2020) noted that predictive bias is a function of three factors: subgroup mean difference on the predictor (dx), subgroup mean difference on the criterion (dy), and test validity (rxy). They used meta-analytic estimates of each of these three to examine predictive bias against Hispanic test takers when cognitive tests are used in personnel selection. They found that tests underpredict Hispanic job performance by an average of .21 SDs, which would call into question the fairness of cognitive test use in personnel selection. We located 119 studies in which all three parameters-dy, dx, and rxy-could be obtained, thus holding sample, setting, and operationalization constant in estimating the three parameters within each study. This produced a substantially different conclusion: We find that tests overpredict Hispanic performance by .04-.20 SDs, depending on assumptions made about artifact corrections. Factors contributing to differences between the two studies include differences in range restriction corrections, sample incomparability, and Berry et al.'s use of rxy estimated from the total sample rather than within the majority subgroup. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Correcting for range restriction in meta-analysis: A reply to Oh et al. (2023).

Oh et al. (2023) question a number of choices made in our article (Sackett et al., 2022); here we respond. They interpret our article as recommending against correcting for range restriction in general in concurrent validation studies; yet, we emphasize that we endorse correction when one has access to the information needed to do so. Our focus was on making range restriction corrections when conducting meta-analyses, where it is common for primary studies to be silent as to the prior basis for selection of the employees later participating in the concurrent validation study. As such, the applicant pool information needed for correction is typically not available. Sackett et al. (2022) highlighted that in many situations, range restriction will be small; so, the inability to correct for it results in only a modest underestimate of validity. Oh et al. mention settings that would result in substantial range restriction; here, we present our rationale as to why we view such settings as uncommon rather than as making up the bulk of the studies contributing to meta-analyses. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Revisiting meta-analytic estimates of validity in personnel selection: Addressing systematic overcorrection for restriction of range.

This paper systematically revisits prior meta-analytic conclusions about the criterion-related validity of personnel selection procedures, and particularly the effect of range restriction corrections on those validity estimates. Corrections for range restriction in meta-analyses of predictor-criterion relationships in personnel selection contexts typically involve the use of an artifact distribution. After outlining and critiquing five approaches that have commonly been used to create and apply range restriction artifact distributions, we conclude that each has significant issues that often result in substantial overcorrection and that therefore the validity of many selection procedures for predicting job performance has been substantially overestimated. Revisiting prior meta-analytic conclusions produces revised validity estimates. Key findings are that most of the same selection procedures that ranked high in prior summaries remain high in rank, but with mean validity estimates reduced by .10-.20 points. Structured interviews emerged as the top-ranked selection procedure. We also pair validity estimates with information about mean Black-White subgroup differences per selection procedure, providing information about validity-diversity tradeoffs. We conclude that our selection procedures remain useful, but selection predictor-criterion relationships are considerably lower than previously thought. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Can a Situational Judgment Test Identify Trainees at Risk of Professionalism Issues? A Multi-Institutional, Prospective Cohort Study.

PURPOSE: To determine whether overall situational judgment test (SJT) scores are associated with programs' clinical competency committee (CCC) ratings of trainee professionalism, any concerning behavior, and concerning behavior requiring active remediation at 2 time periods. METHOD: In fall 2019, trainees from 17 U.S. programs (16 residency, 1 fellowship) took an online 15-scenario SJT developed to measure 7 dimensions of professionalism. CCC midyear and year-end (6 months and 1 year following SJT completion, respectively) professionalism scores and concern ratings were gathered for academic year 2019-2020. Analyses were conducted to determine whether overall SJT scores related to overall professionalism ratings, trainees displaying any concerns, and trainees requiring active remediation at both time periods. RESULTS: Overall SJT scores correlated positively with midyear and year-end overall professionalism ratings ( r = .21 and .14, P < .001 and = .03, respectively). Holding gender and race/ethnicity constant, a 1 standard deviation (SD) increase in overall SJT score was associated with a .20 SD increase in overall professionalism ratings at midyear ( P = .005) and a .22 SD increase at year-end ( P = .001). Holding gender and race/ethnicity constant, a 1 SD increase in overall SJT score decreased the odds of a trainee displaying any concerns by 37% (odds ratio [OR] 95% confidence interval [CI]: [.44, .87], P = .006) at midyear and 34% (OR 95% CI: [.46, .95], P = .025) at year-end and decreased the odds of a trainee requiring active remediation by 51% (OR 95% CI: [.25, .90], P = .02) at midyear. CONCLUSIONS: Overall SJT scores correlated positively with midyear and year-end overall professionalism ratings and were associated with whether trainees exhibited any concerning behavior at midyear and year-end and whether trainees needed active remediation at midyear. Future research should investigate whether other potential professionalism measures are associated with concerning trainee behavior.

Exploring public sentiment on enforced remote work during COVID-19.

Due to the coronavirus disease 2019 (COVID-19) pandemic, many employees have been strongly encouraged or mandated to work from home. The present study sought to understand the attitudes and experiences of the general public toward remote work by analyzing Twitter data from March 30 to July 5 of 2020. We web scraped over 1 million tweets using keywords such as "telework," "work from home," "remote work," and so forth, and analyzed the content using natural language processing (NLP) techniques. Sentiment analysis results show generally positive attitudes expressed by remote work-related tweets, with minor dips during the weekend. Topic modeling results uncovered themes among tweets including home office, cybersecurity, mental health, work-life balance, teamwork, and leadership, with minor changes in topics revealed over the 14-week period. Findings point to topics of particular concern regarding working from home and can help guide hypothesis generation for future research. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

The process of attrition in pre-medical studies: A large-scale analysis across 102 schools.

The important but difficult choice of vocational trajectory often takes place in college, beginning with majoring in a subject and taking relevant coursework. Of all possible disciplines, pre-medical studies are often not a formally defined major but pursued by a substantial proportion of the college population. Understanding students' experiences with pre-med coursework is valuable and understudied, as most research on medical education focuses on the later medical school and residency. We examined the pattern and predictors of attrition at various milestones along the pre-med coursework track during college. Using a College Board dataset, we analyzed a sample of 15,442 students spanning 102 institutions who began their post-secondary education in years between 2006 and 2009. We examined whether students fulfilled the required coursework to remain eligible for medical schools at several milestones: 1) one semester of general chemistry, biology, physics, 2) two semesters of general chemistry, biology, physics, 3) one semester of organic chemistry, and 4) either the second semester of organic chemistry or one semester of biochemistry, and predictors of persistence at each milestone. Only 16.5% of students who intended to major in pre-med graduate college with the required coursework for medical schools. Attrition rates are highest initially but drop as students take more advanced courses. Predictors of persistence include academic preparedness before college (e.g., SAT scores, high school GPA) and college performance (e.g., grades in pre-med courses). Students who perform better academically both in high school and in college courses are more likely to remain eligible for medical school.