Identifying Ability and Nonability Groups: Incorporating Response Times Using Mixture Modeling.

The goal of the present study was to address the analytical complexity of incorporating responses and response times through applying the Jeon and De Boeck mixture item response theory model in Mplus 8.7. Using both simulated and real data, we attempt to identify subgroups of responders that are rapid guessers or engage knowledge retrieval strategies. When applying the mixture model to a measure of contextual error in linguistics results pointed to the presence of a knowledge retrieval strategy. That is, a participant either knows the content (morphology, grammar rules) and can identify the error, or lacks the requisite knowledge and cannot benefit from spending more time on an item. In contrast, as item difficulty progressed, the high-ability group utilized the additional time to make informed guesses. The methodology is illustrated using annotated code in Mplus 8.7.

Assessing Language Skills Using Diagnostic Classification Models: An Example Using a Language Instrument.

The primary purpose of the present study is to inform and illustrate, using examples, the use of Diagnostic Classification Models (DCMs) for the assessment of skills and competencies in cognition and academic achievement. A secondary purpose is to compare and contrast traditional and contemporary psychometrics for the measurement of skills and competencies. DCMs are described along the lines of other psychometric models within the Confirmatory Factor Analysis tradition such as the bifactor model and the known mixture models that are utilized to classify individuals into subgroups. The inclusion of interaction terms and constraints along with its confirmatory nature enables DCMs to accurately assess the possession of skills and competencies. The above is illustrated using an empirical dataset from Saudi Arabia (n = 2642), in which language skills are evaluated on how they conform to known levels of competency based on the CEFR (Council of Europe, 2001) using the English Proficiency Test (EPT).

El propósito principal del presente estudio fue informar e ilustrar, mediante ejemplos, el uso de Modelos de Clasificación Diagnóstica (DCM) para la evaluación de habilidades y competencias en cognición y rendimiento académico. Un propósito secundario fue comparar y contrastar la psicometría tradicional y contemporánea para la medición de habilidades y competencias. Los DCM se describen siguiendo las líneas de otros modelos psicométricos dentro de la tradición del Análisis Factorial Confirmatorio, como el modelo bifactor y los conocidos modelos mixtos que se utilizan para clasificar a los individuos en subgrupos. La inclusión de términos y restricciones de interacción junto con su naturaleza confirmatoria permite a los DCM evaluar con precisión la posesión de habilidades y competencias. Lo anterior se ilustra utilizando un conjunto de datos empíricos de Arabia Saudita (n = 2642), que evalúan cómo las habilidades lingüísticas se ajustan a los niveles de competencia conocidos, basados en el MCER (Council of Europe, 2001).

Identifying Student Subgroups as a Function of School Level Attributes: A Multilevel Latent Class Analysis.

The purpose of the present study was to profile high school students' achievement as a function of their demographic characteristics, parent attributes (e.g., education), and school behaviors (e.g., number of absences). Students were nested within schools in the Saudi Arabia Kingdom. Out of a large sample of 500k, participants involved 3 random samples of 2,000 students measured during the years 2016, 2017, and 2018. Randomization was conducted at the student level to ensure that all school units will be represented and at their respective frequency. Students were nested within 50 high schools. We adopted the multilevel latent profile analysis protocol put forth by Schmiege et al. (2018) and Mäkikangas et al. (2018) that account for nested data and tested latent class structure invariance over time. Results pointed to the presence of a 4-profile solution based on BIC, the Bayes factor, and several information criteria put forth by Masyn (2013). Latent profile separation was mostly guided by parents' education and the number of student absences (being positive and negative predictors of high achievement classes, respectively). Two models tested whether the proportions of level 1 profiles to level 2 units are variable and whether level 2 profiles vary as a function of level 1 profiles. Results pointed to the presence of significant variability due to schools.

Improving Measures via Examining the Behavior of Distractors in Multiple-Choice Tests: Assessment and Remediation.

The purpose of the present article was to illustrate, using an example from a national assessment, the value from analyzing the behavior of distractors in measures that engage the multiple-choice format. A secondary purpose of the present article was to illustrate four remedial actions that can potentially improve the measurement of the construct(s) under study. Participants were 2,248 individuals who took a national examination of chemistry. The behavior of the distractors was analyzed by modeling their behavior within the Rasch model. Potentially informative distractors were (a) further modeled using the partial credit model, (b) split onto separate items and retested for model fit and parsimony, (c) combined to form a "super" item or testlet, and (d) reexamined after deleting low-ability individuals who likely guessed on those informative, albeit erroneous, distractors. Results indicated that all but the item split strategies were associated with better model fit compared with the original model. The best fitted model, however, involved modeling and crediting informative distractors via the partial credit model or eliminating the responses of low-ability individuals who likely guessed on informative distractors. The implications, advantages, and disadvantages of modeling informative distractors for measurement purposes are discussed.

The Impact of Non-attempted and Dually-Attempted Items on Person Abilities Using Item Response Theory.

The purpose of the present study was to relate response strategy with person ability estimates. Two behavioral strategies were examined: (a) the strategy to skip items in order to save time on timed tests, and, (b) the strategy to select two responses on an item, with the hope that one of them may be considered correct. Participants were 4,422 individuals who were administered a standardized achievement measure related to math, biology, chemistry, and physics. In the present evaluation, only the physics subscale was employed. Two analyses were conducted: (a) a person-based one to identify differences between groups and potential correlates of those differences, and, (b) a measure-based analysis in order to identify the parts of the measure that were responsible for potential group differentiation. For (a) person abilities the 2-PL model was employed and later the 3-PL and 4-PL models in order to estimate upper and lower asymptotes of person abilities. For (b) differential item functioning, differential test functioning, and differential distractor functioning were investigated. Results indicated that there were significant differences between groups with completers having the highest ability compared to both non-attempters and dual responders. There were no significant differences between no-attempters and dual responders. The present findings have implications for response strategy efficacy and measure evaluation, revision, and construction.