Validity of a cardiology fellow performance assessment: reliability and associations with standardized examinations and awards.

BACKGROUND: Most work on the validity of clinical assessments for measuring learner performance in graduate medical education has occurred at the residency level. Minimal research exists on the validity of clinical assessments for measuring learner performance in advanced subspecialties. We sought to determine validity characteristics of cardiology fellows' assessment scores during subspecialty training, which represents the largest subspecialty of internal medicine. Validity evidence included item content, internal consistency reliability, and associations between faculty-of-fellow clinical assessments and other pertinent variables. METHODS: This was a retrospective validation study exploring the domains of content, internal structure, and relations to other variables validity evidence for scores on faculty-of-fellow clinical assessments that include the 10-item Mayo Cardiology Fellows Assessment (MCFA-10). Participants included 7 cardiology fellowship classes. The MCFA-10 item content included questions previously validated in the assessment of internal medicine residents. Internal structure evidence was assessed through Cronbach's α. The outcome for relations to other variables evidence was overall mean of faculty-of-fellow assessment score (scale 1-5). Independent variables included common measures of fellow performance. FINDINGS: Participants included 65 cardiology fellows. The overall mean ± standard deviation faculty-of-fellow assessment score was 4.07 ± 0.18. Content evidence for the MCFA-10 scores was based on published literature and core competencies. Cronbach's α was 0.98, suggesting high internal consistency reliability and offering evidence for internal structure validity. In multivariable analysis to provide relations to other variables evidence, mean assessment scores were independently associated with in-training examination scores (beta = 0.088 per 10-point increase; p = 0.05) and receiving a departmental or institutional award (beta = 0.152; p = 0.001). Assessment scores were not associated with educational conference attendance, compliance with completion of required evaluations, faculty appointment upon completion of training, or performance on the board certification exam. R2 for the multivariable model was 0.25. CONCLUSIONS: These findings provide sound validity evidence establishing item content, internal consistency reliability, and associations with other variables for faculty-of-fellow clinical assessment scores that include MCFA-10 items during cardiology fellowship. Relations to other variables evidence included associations of assessment scores with performance on the in-training examination and receipt of competitive awards. These data support the utility of the MCFA-10 as a measure of performance during cardiology training and could serve as the foundation for future research on the assessment of subspecialty learners.

Predicting Quality of Clinical Performance From Cardiology Fellowship Applications.

Variables in cardiology fellowship applications have not been objectively analyzed against applicants' subsequent clinical performance. We investigated possible correlations in a retrospective cohort study of 65 cardiology fellows at the Mayo Clinic (Rochester, Minn) who began 2 years of clinical training from July 2007 through July 2013. Application variables included the strength of comparative statements in recommendation letters and the authors' academic ranks, membership status in the Alpha Omega Alpha Honor Medical Society, awards earned, volunteer activities, United States Medical Licensing Examination (USMLE) scores, advanced degrees, publications, and completion of a residency program ranked in the top 6 in the United States. The outcome was clinical performance as measured by a mean of faculty evaluation scores during clinical training. The overall mean evaluation score was 4.07 ± 0.18 (scale, 1-5). After multivariable analysis, evaluation scores were associated with Alpha Omega Alpha designation (ß=0.13; 95% CI, 0.01-0.25; P=0.03), residency program reputation (ß=0.13; 95% CI, 0.05-0.21; P=0.004), and strength of comparative statements in recommendation letters (ß=0.08; 95% CI, 0.01-0.15; P=0.02), particularly in letters from residency program directors (ß=0.05; 95% CI, 0.01-0.08; P=0.009). Objective factors to consider in the cardiology fellowship application include Alpha Omega Alpha membership, residency program reputation, and comparative statements from residency program directors.