Assessment of Clinical Reasoning in Undergraduate Medical Education: A Pragmatic Approach to Programmatic Assessment.

PURPOSE: Clinical reasoning, a complex construct integral to the practice of medicine, has been challenging to define, teach, and assess. Programmatic assessment purports to overcome validity limitations of judgments made from individual assessments through proportionality and triangulation processes. This study explored a pragmatic approach to the programmatic assessment of clinical reasoning. METHOD: The study analyzed data from 2 student cohorts from the University of Utah School of Medicine (UUSOM) (n = 113 in cohort 1 and 119 in cohort 2) and 1 cohort from the University of Colorado School of Medicine (CUSOM) using assessment data that spanned from 2017 to 2021 (n = 199). The study methods included the following: (1) asking faculty judges to categorize student clinical reasoning skills, (2) selecting institution-specific assessment data conceptually aligned with clinical reasoning, (3) calculating correlations between assessment data and faculty judgments, and (4) developing regression models between assessment data and faculty judgments. RESULTS: Faculty judgments of student clinical reasoning skills were converted to a continuous variable of clinical reasoning struggles, with mean (SD) ratings of 2.93 (0.27) for the 232 UUSOM students and 2.96 (0.17) for the 199 CUSOM students. A total of 67 and 32 discrete assessment variables were included from the UUSOM and CUSOM, respectively. Pearson r correlations were moderate to strong between many individual and composite assessment variables and faculty judgments. Regression models demonstrated an overall adjusted R2 (standard error of the estimate) of 0.50 (0.19) for UUSOM cohort 1, 0.28 (0.15) for UUSOM cohort 2, and 0.30 (0.14) for CUSOM. CONCLUSIONS: This study represents an early pragmatic exploration of regression analysis as a potential tool for operationalizing the proportionality and triangulation principles of programmatic assessment. The study found that programmatic assessment may be a useful framework for longitudinal assessment of complicated constructs, such as clinical reasoning.

Virtual Learning and Assessment in Rheumatology Fellowship Training: Objective Structured Clinical Examination Revisited.

OBJECTIVE: With the onset of the COVID-19 pandemic, an annual multi-institutional face-to-face rheumatology objective structured clinical examination (ROSCE) was transformed into a virtual format. The educational goals of the virtual ROSCE (vROSCE) were to reproduce the educational value of the previous in-person ROSCE, providing a valuable formative assessment of rheumatology training activities encompassing the 6 Accreditation Council for Graduate Medical Education (ACGME) core competencies for fellows-in-training (FITs). This article describes the novel design, feasibility, and stakeholder value of a vROSCE. METHODS: Through an established collaboration of 5 rheumatology fellowship training programs, in February 2021, a vROSCE was created and conducted using a Zoom platform. Station development included learning objectives, FIT instructions, faculty proctor instructions, and a checklist by which to provide structured formative feedback. An anonymous, optional web-based survey was sent to FIT participants to evaluate the experience. RESULTS: Twenty-three rheumatology FITs from 5 institutions successfully rotated through 6 stations in the vROSCE. Immediate feedback was given to each FIT using standardized rubrics structured around ACGME core competencies. A total of 65% of FITs (15 of 23) responded to the survey, and 93% of survey respondents agreed or strongly agreed that the vROSCE was a helpful educational activity and identified individualized opportunities for improvement. CONCLUSION: A vROSCE is an innovative, feasible, valuable, and well-received educational technology tool. The vROSCE enriched rheumatology FITs' education and offered collaborative learning experiences across institutions.

Incorporating Telemedicine in Rheumatology Fellowship Training Programs: Needs Assessment, Curricular Intervention, and Evaluation.

OBJECTIVE: To increase the confidence of rheumatology fellows in training (FITs) in delivering virtual care (VC) and prepare them for independent practice, we developed educational materials addressing gaps in their skills. METHODS: We identified gaps in telemedicine skills based on FIT performance in a virtual rheumatology objective structured clinical examination (vROSCE) station on VC delivery using video teleconference technology and survey (survey 1) responses. We created educational materials including videos of "mediocre" and "excellent" VC examples, discussion/reflection questions, and a document summarizing key practices. We measured change in the confidence levels of FITs for delivering VC with a post-intervention survey (survey 2). RESULTS: Thirty-seven FITs (19 first-year, 18 second- plus third-year fellows) from 7 rheumatology fellowship training programs participated in a vROSCE and demonstrated gaps in skills mapping to several Rheumatology Telehealth Competency domains. Confidence levels of FITs improved significantly from survey 1 to survey 2 for 22 of 34 (65%) questions. All participating FITs found the educational materials helpful for learning and reflecting on their own VC practice; 18 FITs (64%) qualified usefulness as "moderately" or "a lot." Through surveying, 17 FITs (61%) reported implementing skills from the instructional videos into VC visits. CONCLUSION: Continually assessing our learners' needs and creating educational materials addressing gaps in training are requisite. Using a vROSCE station, needs assessments, and targeted learning with videos and discussion-guidance materials enhanced the confidence level of FITs in VC delivery. It is imperative to incorporate VC delivery into fellowship training program curricula to ensure breadth in skills, attitudes, and knowledge of new entrants into the rheumatology workforce.