RESUMO
Differential item functioning (DIF) is often used to examine validity evidence of alternate form test accommodations. Unfortunately, traditional approaches for evaluating DIF are prone to selection bias. This article proposes a novel DIF framework that capitalizes on regression discontinuity design analysis to control for selection bias. A simulation study was performed to compare the new framework with traditional logistic regression, with respect to Type I error and power rates of the uniform DIF test statistics and bias and root mean square error of the corresponding effect size estimators. The new framework better controlled the Type I error rate and demonstrated minimal bias but suffered from low power and lack of precision. Implications for practice are discussed.
RESUMO
Prior research suggests that subscores from a single achievement test seldom add value over a single total score. Such scores typically correspond to subcontent areas in the total content domain, but content subdomains might not provide a sound basis for subscores. Using scores on an inferential reading comprehension test from 625 third, fourth, and fifth graders, two new methods of creating subscores were explored. Three subscores were based on the types of incorrect answers given by students. The fourth was based on temporal efficiency in giving correct answers. All four scores were reliable. The three subscores based on incorrect answers added value and validity. In logistic regression analyses predicting failure to reach proficiency on a statewide test, models including subscores fit better than the model with a single total score. Including the pattern of incorrect responses improved fit in all three grades, whereas including the comprehension efficiency score only modestly improved fit in fourth and fifth grades, but not third grade. Area under the curve (AUC) statistics from receiver operating characteristic (ROC) curves based on the various models were higher for models including subscores than those without subscores. Implications for using models with and without subscores are illustrated and discussed.
RESUMO
Purpose: There is a lack of quick, reliable, and valid standardized reading comprehension assessments appropriate for postsecondary readers. We attempted to address this gap by designing Read, Understand, Learn, & Excel (RULE), a reading comprehension measure that employs sentence verification and recall tasks to assess reading comprehension. This article describes the exploratory study undertaken to construct RULE and then examines the preliminary concurrent validity and alternate form reliability of this measure. Method: The RULE measure was first developed by designing reading stimuli, test items for the sentence verification task, and directions for the recall test for 2 forms based on previous work (Griffiths, Sohlberg, Kirk, Fickas, & Biancarosa, 2016). Thirty undergraduate students who identified themselves as typical readers were administered the RULE measure as well as the Nelson-Denny Reading Test (Brown, Fishco, & Hanna, 1993). Students also completed questionnaires and participated in informal interviews to provide information regarding study and learning habits and academic background. Results: There was preliminary evidence of alternate form reliability between the sentence verification task sets of 2 chapters of RULE (r = .38, p < .05). Preliminary evidence for concurrent validity between RULE and the Nelson-Denny Reading Test was provided by correlation coefficients in the low to moderate range (.03-.38). Conclusion: RULE design and preliminary findings of concurrent validity and alternate form reliability provide "proof of concept" for an ecologically valid testing format that assesses comprehension skills appropriate for the postsecondary level. Suggestions for strengthening validity and reliability of the tool are provided, and clinical contributions of RULE are discussed. Supplemental Material: https://doi.org/10.23641/asha.6987371.
