Nurturing Untapped Integration Expertise of MS4 Assessment Writers.

Creating original, integrated multiple-choice questions (MCQs) is time-consuming and onerous for basic science and clinical faculty. We demonstrate that medical students are co-experts to overcome assessment challenges of the faculty. We recruited, trained, and motivated medical students to write 10,000 high-quality MCQs for use in the foundational courses of medical education. These students were ideal because they possessed integrated knowledge (basic sciences and clinical experience). We taught them how to write high-quality MCQs using a writing template and continuous monitoring and support by an item bank curator. The students themselves also benefitted personally and pedagogically from the experience.

Assessing clinical competence: a multitrait-multimethod matrix construct validity study.

Education in Doctor of Medicine programs has moved towards an emphasis on clinical competency, with entrustable professional activities providing a framework of learning objectives and outcomes to be assessed within the clinical environment. While the identification and structured definition of objectives and outcomes have evolved, many methods employed to assess clerkship students' clinical skills remain relatively unchanged. There is a paucity of medical education research applying advanced statistical design and analytic techniques to investigate the validity of clinical skills assessment. One robust statistical method, multitrait-multimethod matrix analysis, can be applied to investigate construct validity across multiple assessment instruments and settings. Four traits were operationalized to represent the construct of critical clinical skills (professionalism, data gathering, data synthesis, and data delivery). The traits were assessed using three methods (direct observations by faculty coaches, clinical workplace-based evaluations, and objective structured clinical examination type clinical practice examinations). The four traits and three methods were intercorrelated for the multitrait-multimethod matrix analysis. The results indicated reliability values in the adequate to good range across the three methods with the majority of the validity coefficients demonstrating statistical significance. The clearest evidence for convergent and divergent validity was with the professionalism trait. The correlations on the same method/different traits analyses indicated substantial method effect; particularly on clinical workplace-based assessments. The multitrait-multimethod matrix approach, currently underutilized in medical education, could be employed to explore validity evidence of complex constructs such as clinical skills. These results can inform faculty development programs to improve the reliability and validity of assessments within the clinical environment.

Race and Gender Bias in Clerkship Grading.

Phenomenon: Existing literature, as well as anecdotal evidence, suggests that tiered clinical grading systems may display systematic demographic biases. This study aimed to investigate these potential inequities in-depth. Specifically, this study attempted to address the following gaps in the literature: (1) studying grades actually assigned to students (as opposed to self-reported ones), (2) using longitudinal data over an 8-year period, providing stability of data, (3) analyzing three important, potentially confounding covariates, (4) using a comprehensive multivariate statistical design, and (5) investigating not just the main effects of gender and race, but also their potential interaction. Approach: Participants included 1,905 graduates (985 women, 51.7%) who received the Doctor of Medicine degree between 2014 and 2021. Most of the participants were white (n = 1,310, 68.8%) and about one-fifth were nonwhite (n = 397, 20.8%). There were no reported race data for 10.4% (n = 198). To explore potential differential grading, a two-way multivariate analysis of covariance was employed to examine the impact of race and gender on grades in eight required clerkships, adjusting for prior academic performance. Findings: There were two significant main effects, race and gender, but no interaction effect between gender and race. Women received higher grades on average on all eight clerkships, and white students received higher grades on average on four of the eight clerkships (Medicine, Pediatrics, Surgery, Obstetrics/Gynecology). These relationships held even when accounting for prior performance covariates. Insights: These findings provide additional evidence that tiered grading systems may be subject to systematic demographic biases. It is difficult to tease apart the contributions of various factors to the observed differences in gender and race on clerkship grades, and the interactions that produce these biases may be quite complex. The simplest solution to cut through the tangled web of grading biases may be to move away from a tiered grading system altogether.

Stability and Differences in Empathy Between Men and Women Medical Students: a Panel Design Study.

PHENOMENON: Empathy is central to the physician-patient relationship, and affects clinical outcomes. There is uncertainty about the stability of empathy in medical students over the course of medical school, as well as differences in empathy between men and women. APPROACH: A panel study design was used to follow first year through fourth year medical students (MS1-4) during the 2018-2019 school year. Empathy was measured using the interpersonal reactivity index (IRI), a self-report scale that separates empathy into a cognitive perspective taking (PT) and affective empathic concern (EC) component. FINDINGS: A total of 631 (359 women and 272 men) from 970 students (65% response rate) responded to a baseline survey, and a total of 536 students (300 women and 236 men) from 970 students (55% response rate) responded to surveys throughout the year. At baseline, women had significantly higher EC scores than men (p < 0.0001), with no significant PT difference between men and women (p > 0.05). These differences were stable for all MS cohorts. INSIGHTS: Women had self-reported higher affective empathy (EC component) than men, while there were no differences in cognitive empathy (PT component). We discuss these data in the context of defining gender vs. sex, socialized gender stereotypes, and implications for future research.

Impact of the stringency of lockdown measures on covid-19: A theoretical model of a pandemic.

BACKGROUND: How effective have lockdowns been at reducing the covid-19 infection and mortality rates? Lockdowns influence contact among persons within or between populations including restricting travel, closing schools, prohibiting public gatherings, requiring workplace closures, all designed to slow the contagion of the virus. The purpose of the present study was to assess the impact of lockdown measures on the spread of covid-19 and test a theoretical model of the covid-19 pandemic employing structural equation modelling. METHODS: Lockdown variables, population demographics, mortality rates, infection rates, and health were obtained for eight countries: Austria, Belgium, France, Germany, Italy, Netherlands, Spain, and the United Kingdom. The dataset, owid-covid-data.csv, was downloaded on 06/01/2020 from: https://github.com/owid/covid-19-data/tree/master/public/data. Infection spread and mortality data were depicted as logistic growth and analyzed with stepwise multiple regression. The overall structure of the covid-19 data was explored through factor analyses leading to a theoretical model that was tested using latent variable path analysis. RESULTS: Multiple regression indicated that the time from lockdown had a small but significant effect (ß = 0.112, p< 0.01) on reducing the number of cases per million. The stringency index produced the most important effect for mortality and infection rates (ß = 0.588,ß = 0.702, ß = 0.518, ß = 0.681; p< 0.01). Exploratory and confirmatory analyses resulted in meaningful and cohesive latent variables: 1) Mortality, 2) Infection Spread, 3) Pop Health Risk, and 4) Health Vulnerability (Comparative Fit Index = 0.91; Standardized Root Mean Square Residual = 0.08). DISCUSSION: The stringency index had a large impact on the growth of covid-19 infection and mortality rates as did percentage of population aged over 65, median age, per capita GDP, diabetes prevalence, cardiovascular death rates, and ICU hospital beds per 100K. The overall Latent Variable Path Analysis is theoretically meaningful and coherent with acceptable fit indices as a model of the covid-19 pandemic.

Validity Evidence for Assessing Entrustable Professional Activities During Undergraduate Medical Education.

PURPOSE: To explore validity evidence for the use of entrustable professional activities (EPAs) as an assessment framework in medical education. METHOD: Formative assessments on the 13 Core EPAs for entering residency were collected for 4 cohorts of students over a 9- to 12-month longitudinal integrated clerkship as part of the Education in Pediatrics Across the Continuum pilot at the University of Minnesota Medical School. The students requested assessments from clinical supervisors based on direct observation while engaging in patient care together. Based on each observation, the faculty member rated the student on a 9-point scale corresponding to levels of supervision required. Six EPAs were included in the present analyses. Student ratings were depicted as curves describing their performance over time; regression models were employed to fit the curves. The unit of analyses for the learning curves was observations rather than individual students. RESULTS: (1) Frequent assessments on EPAs provided a developmental picture of competence consistent with the negative exponential learning curve theory; (2) This finding was true across a variety of EPAs and across students; and (3) The time to attain the threshold level of performance on the EPA for entrustment varied by student and EPA. CONCLUSIONS: The results provide validity evidence for an EPA-based program of assessment. Students assessed using multiple observations performing the Core EPAs for entering residency demonstrate classic developmental progression toward the desired level of competence resulting in entrustment decisions. Future work with larger data samples will allow further psychometric analyses of assessment of EPAs.

Assessing anaesthesiology and intensive care specialty physicians: An Italian language multisource feedback system.

BACKGROUND: Physician professionalism, including anaesthesiologists and intensive care doctors, should be continuously assessed during training and subsequent clinical practice. Multi-source feedback (MSF) is an assessment system in which healthcare professionals are assessed on several constructs (e.g., communication, professionalism, etc.) by multiple people (medical colleagues, coworkers, patients, self) in their sphere of influence. MSF has gained widespread acceptance for both formative and summative assessment of professionalism for reflecting on how to improve clinical practice. METHODS: Instrument development and psychometric analysis (feasibility, reliability, construct validity via exploratory factor analysis) for MSF questionnaires in a postgraduate specialty training in Anaesthesiology and intensive care in Italy. Sixty-four residents at the Università del Piemonte Orientale (Italy) Anesthesiology Residency Program. Main outcomes assessed were: development and psychometric testing of 4 questionnaires: self, medical colleague, coworker and patient assessment. RESULTS: Overall 605 medical colleague questionnaires (mean of 9.3 ±1.9) and 543 coworker surveys (mean 8.4 ±1.4) were collected providing high mean ratings for all items (> 4.0 /5.0). The self-assessment item mean score ranged from 3.1 to 4.3. Patient questionnaires (n = 308) were returned from 31 residents (40%; mean 9.9 ± 6.2). Three items had high percentages of "unable to assess" (> 15%) in coworker questionnaires. Factor analyses resulted in a two-factor solution: clinical management with leadership and accountability accounting for at least 75% of the total variance for the medical colleague and coworker's survey with high internal consistency reliability (Cronbach's α > 0.9). Patient's questionnaires had a low return rate, a limited exploratory analysis was performed. CONCLUSIONS: We provide a feasible and reliable Italian language MSF instrument with evidence of construct validity for the self, coworkers and medical colleague. Patient feedback was difficult to collect in our setting.

A Study of the Validity of the New MCAT Exam.

PURPOSE: To conduct a study of the validity of the new Medical College Admission Test (MCAT). METHOD: Deidentified data for first- and second-year medical students (185 women, 54.3%; 156 men, 45.7%) who matriculated in 2016 and 2017 to the University of Minnesota Medical School-Twin Cities were included. Of those students, 220 (64.5%) had taken the new MCAT exam and 182 (53.4%) had taken the old MCAT exam (61 [17.9%] had taken both). The authors calculated descriptive statistics and Pearson product moment correlations (r) between new and old MCAT section scores. They conducteda regression analysis of MCAT section scores with Step 1 scores and with preclerkship course performance. They also conducted an exploratory factor analysis (principal component analysis with varimax rotation) of MCAT scores, undergraduate grade point average, Step 1 scores, and course performance. RESULTS: The new MCAT exam section mean score percentiles ranged from 72 to 78 (mean composite score percentile of 80). The old MCAT exam section mean score percentiles ranged from 84 to 88 (mean composite score percentile of 83). The pattern of correlations among and between new and old MCAT exam section scores (range of r: 0.03-0.67; P < .01) provided evidence of both divergent and convergent validities. Backward multiple regression of new MCAT exam section scores and Step 1 scores resulted in a multiple R of .440; the same analysis with Human Behavior course performance as the dependent variable provided a similar solution with the expected sections of the new MCAT exam (multiple R = .502). The factor analysis resulted in 4 cohesive, theoretically meaningful factors: biomedical knowledge, basic science concepts, cognitive reasoning, and general achievement. CONCLUSIONS: This study provided empirical evidence of multiple types of validity for the new MCAT exam.