Building reliable and generalizable clerkship competency assessments: Impact of 'hawk-dove' correction.

PURPOSE: Systematic differences among raters' approaches to student assessment may result in leniency or stringency of assessment scores. This study examines the generalizability of medical student workplace-based competency assessments including the impact of rater-adjusted scores for leniency and stringency. METHODS: Data were collected from summative clerkship assessments completed for 204 students during 2017-2018 the clerkship at a single institution. Generalizability theory was used to explore variance attributed to different facets (rater, learner, item, and competency domain) through three unbalanced random-effects models by clerkship including applying assessor stringency-leniency adjustments. RESULTS: In the original assessments, only 4-8% of the variance was attributed to the student with the remainder being rater variance and error. Aggregating items to create a composite score increased variability attributable to the student (5-13% of variance). Applying a stringency-leniency ('hawk-dove') correction substantially increased the variance attributed to the student (14.8-17.8%) and reliability. Controlling for assessor leniency/stringency reduced measurement error, decreasing the number of assessments required for generalizability from 16-50 to 11-14. CONCLUSIONS: Similar to prior research, most of the variance in competency assessment scores was attributable to raters, with only a small proportion attributed to the student. Making stringency-leniency corrections using rater-adjusted scores improved the psychometric characteristics of assessment scores.

Burnout in OB/GYN Students and Residents.

In this article we address the concept of burnout, first in the medical student setting, and then in the residency setting. We will review the prevalence followed by a discussion of risk factors, consequences, and finally thoughts on prevention and intervention.

Evaluating the Reliability and Validity Evidence of the RIME (Reporter-Interpreter-Manager-Educator) Framework for Summative Assessments Across Clerkships.

PURPOSE: The ability of medical schools to accurately and reliably assess medical student clinical performance is paramount. The RIME (reporter-interpreter-manager-educator) schema was originally developed as a synthetic and intuitive assessment framework for internal medicine clerkships. Validity evidence of this framework has not been rigorously evaluated outside of internal medicine. This study examined factors contributing to variability in RIME assessment scores using generalizability theory and decision studies across multiple clerkships, thereby contributing to its internal structure validity evidence. METHOD: Data were collected from RIME-based summative clerkship assessments during 2018-2019 at Virginia Commonwealth University. Generalizability theory was used to explore variance attributed to different facets through a series of unbalanced random-effects models by clerkship. For all analyses, decision (D-) studies were conducted to estimate the effects of increasing the number of assessments. RESULTS: From 231 students, 6,915 observations were analyzed. Interpreter was the most common RIME designation (44.5%-46.8%) across all clerkships. Variability attributable to students ranged from 16.7% in neurology to 25.4% in surgery. D-studies showed the number of assessments needed to achieve an acceptable reliability (0.7) ranged from 7 in pediatrics and surgery to 11 in internal medicine and 12 in neurology. However, depending on the clerkship each student received between 3 and 8 assessments. CONCLUSIONS: This study conducted generalizability- and D-studies to examine the internal structure validity evidence of RIME clinical performance assessments across clinical clerkships. Substantial proportion of variance in RIME assessment scores was attributable to the rater, with less attributed to the student. However, the proportion of variance attributed to the student was greater than what has been demonstrated in other generalizability studies of summative clinical assessments. Overall, these findings support the use of RIME as a framework for assessment across clerkships and demonstrate the number of assessments required to obtain sufficient reliability.

Reassessing the impact of smoking on preeclampsia/eclampsia: are there age and racial differences?

OBJECTIVE: To investigate the association between cigarette use during pregnancy and pregnancy-induced hypertension/preeclampsia/eclampsia (PIH) by maternal race/ethnicity and age. METHODS: This retrospective cohort study was based on the U.S. 2010 natality data. Our study sample included U.S. women who delivered singleton pregnancies between 20 and 44 weeks of gestation without major fetal anomalies in 2010 (n = 3,113,164). Multivariate logistic regression models were fit to estimate crude and adjusted odds ratios and the corresponding 95% confidence intervals. RESULTS: We observed that the association between maternal smoking and PIH varied by maternal race/ethnicity and age. Compared with non-smokers, reduced odds of PIH among pregnant smokers was only evident for non-Hispanic white and non-Hispanic American Indian women aged less than 35 years. Non-Hispanic Asian/Pacific Islander women who smoked during pregnancy had increased odds of PIH regardless of maternal age. Non-Hispanic white and non-Hispanic black women 35 years or older who smoked during pregnancy also had increased odds of PIH. CONCLUSION: Our study findings suggest important differences by maternal race/ethnicity and age in the association between cigarette use during pregnancy and PIH. More research is needed to establish the biologic and social mechanisms that might explain the variations with maternal age and race/ethnicity that were observed in our study.