Limitations of the Inter-Unit Reliability: A Set of Practical Examples.

Healthcare quality measures are statistics that serve to evaluate healthcare providers and identify those that need to improve their care. Before using these measures in clinical practice, developers and reviewers assess measure reliability, which describes the degree to which differences in the measure values reflect actual variation in healthcare quality, as opposed to random noise. The Inter-Unit Reliability (IUR) is a popular statistic for assessing reliability, and it describes the proportion of total variation in a measure that is attributable to between-provider variation. However, Kalbfleisch, He, Xia, and Li (2018) [Health Services and Outcomes Research Methodology, 18, 215-225] have argued that the IUR has a severe limitation in that some of the between-provider variation may be unrelated to quality of care. In this paper, we illustrate the practical implications of this limitation through several concrete examples. We show that certain best-practices in measure development, such as careful risk adjustment and exclusion of unstable measure values, can decrease the sample IUR value. These findings uncover potential negative consequences of discarding measures with IUR values below some arbitrary threshold.

Integrating Concepts in Biology Textbook Increases Learning: Assessment Triangulation Using Concept Inventory, Card Sorting, and MCAT Instruments, Followed by Longitudinal Tracking.

The purpose of this study was to examine the educational impact of an intervention, the inquiry-focused textbook Integrating Concepts in Biology (ICB), when used in a yearlong introductory biology course sequence. Student learning was evaluated using three published instruments: 1) The Biology Concept Inventory probed depth of student mastery of fundamental concepts in organismal and cellular topics when confronting misconceptions as distractors. ICB students had higher gains in all six topic categories (+43% vs. peers overall, p < 0.01). 2) The Biology Card Sorting Task assessed whether students organized biological ideas more superficially, as novices do, or based on deeper concepts, like experts. The frequency with which ICB students connected deep-concept pairs, or triplets, was similar to peers; but deep understanding of structure/function was much higher (for pairs: 77% vs. 25%, p < 0.01). 3) A content-focused Medical College Admission Test (MCAT) posttest compared ICB student content knowledge with that of peers from 15 prior years. Historically, MCAT performance for each semester ranged from 53% to 64%; the ICB cohort scored 62%, in the top quintile. Longitudinal tracking in five upper-level science courses the following year found ICB students outperformed peers in physiology (85% vs. 80%, p < 0.01).