Testlet-Based Multidimensional Adaptive Testing.

Multidimensional adaptive testing (MAT) is a highly efficient method for the simultaneous measurement of several latent traits. Currently, no psychometrically sound approach is available for the use of MAT in testlet-based tests. Testlets are sets of items sharing a common stimulus such as a graph or a text. They are frequently used in large operational testing programs like TOEFL, PISA, PIRLS, or NAEP. To make MAT accessible for such testing programs, we present a novel combination of MAT with a multidimensional generalization of the random effects testlet model (MAT-MTIRT). MAT-MTIRT compared to non-adaptive testing is examined for several combinations of testlet effect variances (0.0, 0.5, 1.0, and 1.5) and testlet sizes (3, 6, and 9 items) with a simulation study considering three ability dimensions with simple loading structure. MAT-MTIRT outperformed non-adaptive testing regarding the measurement precision of the ability estimates. Further, the measurement precision decreased when testlet effect variances and testlet sizes increased. The suggested combination of the MTIRT model therefore provides a solution to the substantial problems of testlet-based tests while keeping the length of the test within an acceptable range.

Estimating tests including subtests.

Current assessment studies often face a dilemma that arises from the necessity of a simultaneous unidimensional and multidimensional interpretation of a test. When, additional to the measurement of a single domain, subdomains of this domain are to be measured, one and the same data set will have to be analyzed unidimensionaly and multidimensionaly at the same time even though it is a contradiction of the theoretical assumptions underlying the analysis. This article, at first, describes the psychometric deficiencies typically going along with current analysis approaches. Subsequently, a subdimension model is proposed that explicitly allows for the existence of subdomains, or subdimensions, in a measured domain. Thereby, the model provides a means to obtain calibration results that suffer less from the deficiencies described and also allows for an item selection in the test development process that already considers the multidimensional structure of the test.