Comparing computer adaptive testing stopping rules under the generalized partial-credit model.

An important consideration of any computer adaptive testing (CAT) program is the criterion used for ending item administration-the stopping rule, which ensures that all examinees are assessed to the same standard. Although various stopping rules exist, none of them have been compared under the generalized partial-credit model (Muraki in Applied Psychological Measurement, 16, 159-176, 1992). In this simulation study we compared the performance of three variable-length stopping rules-standard error (SE), minimum information (MI), and change in theta (CT)-both in isolation and in combination with requirements of minimum and maximum numbers of items, as well as a fixed-length stopping rule. Each stopping rule was examined under two termination criteria-one a more lenient requirement (SE = 0.35, MI = 0.56, CT = 0.05), and one more stringent (SE = 0.30, MI = 0.42, CT = 0.02). The simulation design also included content-balancing and exposure controls, aspects of CAT that have been excluded in previous research comparing variable-length stopping rules. The minimum-information stopping rule produced biased theta estimates and varied greatly in measurement quality across the theta distribution. The absolute-change-in-theta stopping rule had strong performance when paired with a lower criterion and a minimum test length. The standard error stopping rule consistently provided the best balance of measurement precision and operational efficiency and was based on the fewest number of administered items necessary to obtain accurate and precise theta estimates, particularly when it was paired with a maximum-number-of-items stopping rule.

A Hierarchical Rater Model for Longitudinal Data.

Research studies in psychology and education often seek to detect changes or growth in an outcome over a duration of time. This research provides a solution to those interested in estimating latent traits from psychological measures that rely on human raters. Rater effects potentially degrade the quality of scores in constructed response and performance assessments. We develop an extension of the hierarchical rater model (HRM), which yields estimates of latent traits that have been corrected for individual rater bias and variability, for ratings that come from longitudinal designs. The parameterization, called the longitudinal HRM (L-HRM), includes an autoregressive time series process to permit serial dependence between latent traits at adjacent timepoints, as well as a parameter for overall growth. We evaluate and demonstrate the feasibility and performance of the L-HRM using simulation studies. Parameter recovery results reveal predictable amounts and patterns of bias and error for most parameters across conditions. An application to ratings from a study of character strength demonstrates the model. We discuss limitations and future research directions to improve the L-HRM.

Comparing Imputation Methods for Trait Estimation Using the Rating Scale Model.

This study examined the performance of four methods of handling missing data for discrete response options on a questionnaire: (1) ignoring the missingness (using only the observed items to estimate trait levels); (2) nearest-neighbor hot deck imputation; (3) multiple hot deck imputation; and (4) semi-parametric multiple imputation. A simulation study examining three questionnaire lengths (41-, 20-, and 10-item) crossed with three levels of missingness (10, 25, and 40 percent) was conducted to see which methods best recovered trait estimates when data were missing completely at random and the polytomous items were scored with Andrich's (1978) rating scale model. The results showed that ignoring the missingness and semi-parametric imputation best recovered known trait levels across all conditions, with the semi-parametric technique providing the most precise trait estimates. This study demonstrates the power of specific objectivity in Rasch measurement, as ignoring the missingness leads to generally unbiased trait estimates.

Nurse-led disease management for hypertension control in a diverse urban community: a randomized trial.

BACKGROUND: Treated but uncontrolled hypertension is highly prevalent in African American and Hispanic communities. OBJECTIVE: To test the effectiveness on blood pressure of home blood pressure monitors alone or in combination with follow-up by a nurse manager. DESIGN: Randomized controlled effectiveness trial. PATIENTS: Four hundred and sixteen African American or Hispanic patients with a history of uncontrolled hypertension. Patients with blood pressure ≥150/95, or ≥140/85 for patients with diabetes or renal disease, at enrollment were recruited from one community clinic and four hospital outpatient clinics in East and Central Harlem, New York City. INTERVENTION: Patients were randomized to receive usual care or a home blood pressure monitor plus one in-person counseling session and 9 months of telephone follow-up with a registered nurse. During the trial, the home monitor alone arm was added. MAIN MEASURES: Change in systolic and diastolic blood pressure at 9 and 18 months. KEY RESULTS: Changes from baseline to 9 months in systolic blood pressure relative to usual care was -7.0 mm Hg (Confidence Interval [CI], -13.4 to -0.6) in the nurse management plus home blood pressure monitor arm, and +1.1 mm Hg (95% CI, -5.5 to 7.8) in the home blood pressure monitor only arm. No statistically significant differences in systolic blood pressure were observed among treatment arms at 18 months. No statistically significant improvements in diastolic blood pressure were found across treatment arms at 9 or 18 months. Changes in prescribing practices did not explain the decrease in blood pressure in the nurse management arm. CONCLUSIONS: A nurse management intervention combining an in-person visit, periodic phone calls, and home blood pressure monitoring over 9 months was associated with a statistically significant reduction in systolic, but not diastolic, blood pressure compared to usual care in a high risk population. Home blood pressure monitoring alone was no more effective than usual care.

Cost-effectiveness of nurse-led disease management for heart failure in an ethnically diverse urban community.

BACKGROUND: Randomized, controlled trials have shown that nurse-led disease management for patients with heart failure can reduce hospitalizations. Less is known about the cost-effectiveness of these interventions. OBJECTIVE: To estimate the cost-effectiveness of a nurse-led disease management intervention over 12 months, implemented in a randomized, controlled effectiveness trial. DESIGN: Cost-effectiveness analysis conducted alongside a randomized trial. DATA SOURCES: Medical costs from administrative records, and self-reported quality of life and nonmedical costs from patient surveys. PARTICIPANTS: Patients with systolic dysfunction recruited from ambulatory clinics in Harlem, New York. TIME HORIZON: 12 months. PERSPECTIVE: Societal and payer. INTERVENTION: 12-month program that involved 1 face-to-face encounter with a nurse and regular telephone follow-up. OUTCOME MEASURES: Quality of life as measured by the Health Utilities Index Mark 3 and EuroQol-5D and cost-effectiveness as measured by the incremental cost-effectiveness ratio (ICER). RESULTS OF BASE-CASE ANALYSIS: Costs and quality of life were higher in the nurse-managed group than the usual care group. The ICERs over 12 months were $17,543 per EuroQol-5D-based quality-adjusted life-year (QALY) and $15,169 per Health Utilities Index Mark 3-based QALY (in 2001 U.S. dollars). RESULTS OF SENSITIVITY ANALYSIS: From a payer perspective, the ICER ranged from $3673 to $4495 per QALY. Applying national prices in place of New York City prices yielded a societal ICER of $13,460 to $15,556 per QALY. Cost-effectiveness acceptability curves suggest that the intervention was most likely cost-effective for patients with less severe (New York Heart Association classes I to II) heart failure. LIMITATION: The trial was conducted in an ethnically diverse, inner-city neighborhood; thus, results may not be generalizable to other communities. CONCLUSION: Over 12 months, the nurse-led disease management program was a reasonably cost-effective way to reduce the burden of heart failure in this community.

Trends in Classroom Observation Scores.

Observations and ratings of classroom teaching and interactions collected over time are susceptible to trends in both the quality of instruction and rater behavior. These trends have potential implications for inferences about teaching and for study design. We use scores on the Classroom Assessment Scoring System-Secondary (CLASS-S) protocol from 458 middle school teachers over a 2-year period to study changes over time in (a) the average quality of teaching for the population of teachers, (b) the average severity of the population of raters, and (c) the severity of individual raters. To obtain these estimates and assess them in the context of other factors that contribute to the variability in scores, we develop an augmented G study model that is broadly applicable for modeling sources of variability in classroom observation ratings data collected over time. In our data, we found that trends in teaching quality were small. Rater drift was very large during raters' initial days of observation and persisted throughout nearly 2 years of scoring. Raters did not converge to a common level of severity; using our model we estimate that variability among raters actually increases over the course of the study. Variance decompositions based on the model find that trends are a modest source of variance relative to overall rater effects, rater errors on specific lessons, and residual error. The discussion provides possible explanations for trends and rater divergence as well as implications for designs collecting ratings over time.