Building reliable and generalizable clerkship competency assessments: Impact of 'hawk-dove' correction.

PURPOSE: Systematic differences among raters' approaches to student assessment may result in leniency or stringency of assessment scores. This study examines the generalizability of medical student workplace-based competency assessments including the impact of rater-adjusted scores for leniency and stringency. METHODS: Data were collected from summative clerkship assessments completed for 204 students during 2017-2018 the clerkship at a single institution. Generalizability theory was used to explore variance attributed to different facets (rater, learner, item, and competency domain) through three unbalanced random-effects models by clerkship including applying assessor stringency-leniency adjustments. RESULTS: In the original assessments, only 4-8% of the variance was attributed to the student with the remainder being rater variance and error. Aggregating items to create a composite score increased variability attributable to the student (5-13% of variance). Applying a stringency-leniency ('hawk-dove') correction substantially increased the variance attributed to the student (14.8-17.8%) and reliability. Controlling for assessor leniency/stringency reduced measurement error, decreasing the number of assessments required for generalizability from 16-50 to 11-14. CONCLUSIONS: Similar to prior research, most of the variance in competency assessment scores was attributable to raters, with only a small proportion attributed to the student. Making stringency-leniency corrections using rater-adjusted scores improved the psychometric characteristics of assessment scores.

Uncertainty in Decision Making in Medicine: A Scoping Review and Thematic Analysis of Conceptual Models.

PURPOSE: The practice of medicine is rarely straightforward. Data used to facilitate medical decision making may be conflicting, ambiguous, or scarce, and providing optimal care requires balancing clinicians' expertise and available evidence with patients' preferences. To explore uncertainty in decision making across disciplines, the authors performed a scoping review and thematic analysis of the literature to formulate a model describing the decision-making process in medicine under uncertain conditions. METHOD: In 2016, the authors performed a comprehensive search of key databases using a combination of keywords and controlled vocabulary. They identified and reviewed 3,398 records. After applying their inclusion and exclusion criteria to the titles and abstracts and then full texts, 19 articles were selected. The authors applied a qualitative thematic analysis to these articles, using codes to extract themes related to uncertainty in decision making. RESULTS: The 19 articles spanned 6 fields of study and 5 disciplines within the health sciences. The thematic analysis revealed 6 main themes: recognition of uncertainty, classification of uncertainty, stakeholder perspectives, knowledge acquisition, decision-making approach, and evaluation of the decision-making process. CONCLUSIONS: Based on the themes that emerged from their thematic analysis of the literature characterizing the effects of uncertainty and ambiguity on the decision-making process, the authors developed a framework depicting the interplay between these themes with a visual representation of the decision-making process under uncertain conditions. Future research includes further development and validation of this framework to inform medical school curricula.

Student well-being and the learning environment.

BACKGROUND: Medical student well-being is a critical issue in medical education and is linked to burnout and resilience. Understanding the impact of the clinical learning environment may be crucial to developing effective curricular interventions. Medical student well-being is a critical issue in medical education OBJECTIVE: To determine factors affecting medical student well-being and perceived stress during clinical clerkships and describe any associations with the learning environment, resilience, and performance. METHODS: This was a retrospective study of one cohort of medical students surveyed at the conclusion of third-year clinical clerkships using the Medical School Learning Environment Scale, Brief Resilience Scale, Perceived Stress Scale, and the World Health Organization Well-Being Index. The United States Medical Licensing Examination (USMLE), Step 1 scores and clerkship performance were also collected. Analysis included Pearson correlations and linear regression. RESULTS: Of the 204 students completing the clerkships, 146 students completed the survey (72%). Well-being correlated positively with learning environment (r = 0.43) and resilience (r = 0.35, p < 0.01). Stress was negatively correlated with learning environment (r = -0.34) and resilience (r = -0.53, p < 0.01). Well-being was not significantly correlated with performance measures. However, stress demonstrated a weak positive correlation with clerkship performance (r = 0.17, p < 0.05). The overall regression models for well-being and stress were statistically significant (p < 0.001) and explained 23% and 26% of the variance, respectively. CONCLUSIONS: Our study found that well-being and stress are associated with students' resilience and perception of the learning environment in the clinical phase of medical school. However, we found no relationship between well-being and performance and a modest positive relationship between stress and performance. This data helps to illustrate the complex relationship between these factors.