An exploratory Q-matrix estimation method based on sparse non-negative matrix factorization.

Cognitive diagnostic assessment (CDA) is widely used because it can provide refined diagnostic information. The Q-matrix is the basis of CDA, and can be specified by domain experts or by data-driven estimation methods based on observed response data. The data-driven Q-matrix estimation methods have become a research hotspot because of their objectivity, accuracy, and low calibration cost. However, most of the existing data-driven methods require known prior knowledge, such as initial Q-matrix, partial q-vector, or the number of attributes. Under the G-DINA model, we propose to estimate the number of attributes and Q-matrix elements simultaneously without any prior knowledge by the sparse non-negative matrix factorization (SNMF) method, which has the advantage of high scalability and universality. Simulation studies are carried out to investigate the performance of the SNMF. The results under a wide variety of simulation conditions indicate that the SNMF has good performance in the accuracy of attribute number and Q-matrix elements estimation. In addition, a set of real data is taken as an example to illustrate its application. Finally, we discuss the limitations of the current study and directions for future research.

Data-driven Q-matrix learning based on Boolean matrix factorization in cognitive diagnostic assessment.

Attributes and the Q-matrix are the central components for cognitive diagnostic assessment, and are usually defined by domain experts. However, it is challenging and time consuming for experts to specify the attributes and Q-matrix manually. Thus, there is an urgent need for an automatic and intelligent means to address this concern. This paper presents a new data-driven approach for learning the Q-matrix from response data. By constructing a statistical index and a heuristic algorithm based on Boolean matrix factorization, the response matrix is decomposed into the Boolean product of the Q-matrix and the attribute mastery patterns. The feasibility of the proposed approach is evaluated using simulated data generated under various conditions. A real data example is also presented to demonstrate the usefulness of the proposed approach.

A Speed-Accuracy Tradeoff Hierarchical Model Based on Cognitive Experiment.

Most tests are administered within an allocated time. Due to the time limit, examinees might have different trade-offs on different items. In educational testing, the traditional hierarchical model cannot adequately account for the tradeoffs between response time and accuracy. Because of this, some joint models were developed as an extension of the traditional hierarchical model based on covariance. However, they cannot directly reflect the dynamic relationship between response time and accuracy. In contrast, response moderation models took the residual response time as the independent variable of the response model. Nevertheless, the models enlarge the time effect. Alternatively, the speed-accuracy tradeoff (SAT) model is superior to other experimental models in the SAT experiment. Therefore, this paper incorporates the SAT model with the traditional hierarchical model to establish a SAT hierarchical model. The results demonstrated that the Bayesian Markov chain Monte Carlo (MCMC) algorithm performed well in the SAT hierarchical model of parameters by using simulation. Finally, the deviance information criterion (DIC) more preferred the SAT hierarchical model than other models in empirical data. This means that it is indispensable to add the effect of response time on accuracy, but likewise should limit the effect on the empirical data.

Online Calibration of Polytomous Items Under the Graded Response Model.

Computerized adaptive testing (CAT) is an efficient testing mode, which allows each examinee to answer appropriate items according his or her latent trait level. The implementation of CAT requires a large-scale item pool, and item pool needs to be frequently replenished with new items to ensure test validity and security. Online calibration is a technique to calibrate the parameters of new items in CAT, which seeds new items in the process of answering operational items, and estimates the parameters of new items through the response data of examinees on new items. The most popular estimation methods include one EM cycle method (OEM) and multiple EM cycle method (MEM) under dichotomous item response theory models. This paper extends OEM and MEM to the graded response model (GRM), a popular model for polytomous data with ordered categories. Two simulation studies were carried out to explore online calibration under a variety of conditions, including calibration design, initial item parameter calculation methods, calibration methods, calibration sample size and the number of categories. Results show that the calibration accuracy of new items were acceptable, and which were affected by the interaction of some factors, therefore some conclusions were given.