Assessment of resident knowledge: subjective assessment versus performance on the ACR in-training examination.

ABSTRACT

RATIONALE AND OBJECTIVES: The authors assessed the ability of faculty and residents to predict the ranked performance of residents on the American College of Radiology (ACR) In-Training Examination. MATERIALS AND METHODS: Radiology faculty at Penn State Geisinger Health System (PSGHS), the Medical College of Virginia (MCV), and the University of Virginia (UVA) and residents at PSGHS and MCV ranked the expected performances of residents taking the 1997 ACR In-Training Examination. Surveyed faculty and residents were blinded to the actual performances on the examination. Forty-nine residents took the examination (21 at PSGHS, 22 at MCV, six at UVA), and 37 faculty members (11 at PSGHS, 11 at MCV, 15 at UVA) participated in the study. Correlation analysis was performed to assess the agreement between the subjective and actual ranking of residents in each residency class. RESULTS: Faculty were moderately accurate in the overall ranking of resident performances (r = 0.34). High levels of concordance for ranking individual residents correlated with accuracy in only certain cases. Differences in agreement and accuracy of the respondents existed between PSGHS and MCV (P = .0001 and .0014, respectively). The concordance of respondents increased significantly from the 1st- to the 2nd-year class at MCV (P = .0002), whereas accuracy increased significantly between these classes for the PSGHS (P = .042). CONCLUSION: Faculty are only moderately successful in ranking resident performances on the ACR In-Training Examination, and a high level of agreement is not necessarily indicative of increased accuracy. The concordance and accuracy of subjective rankings differ among residency programs and classes.