An Item Response Theory Model for Incorporating Response Times in Forced-Choice Measures.

Forced-choice (FC) measures have been widely used in many personality or attitude tests as an alternative to rating scales, which employ comparative rather than absolute judgments. Several response biases, such as social desirability, response styles, and acquiescence bias, can be reduced effectively. Another type of data linked with comparative judgments is response time (RT), which contains potential information concerning respondents' decision-making process. It would be challenging but exciting to combine RT into FC measures better to reveal respondents' behaviors or preferences in personality measurement. Given this situation, this study aims to propose a new item response theory (IRT) model that incorporates RT into FC measures to improve personality assessment. Simulation studies show that the proposed model can effectively improve the estimation accuracy of personality traits with the ancillary information contained in RT. Also, an application on a real data set reveals that the proposed model estimates similar but different parameter values compared with the conventional Thurstonian IRT model. The RT information can explain these differences.

Methods for online calibration of Q-matrix and item parameters for polytomous responses in cognitive diagnostic computerized adaptive testing.

The ability to rapidly provide examinees with detailed and effective diagnostic information is a critical topic in psychology. Knowing what diagnostic criteria the examinees have met enables the practitioner to seek the solution to help them in a timely manner, and this can be achieved by cognitive diagnostic computerized adaptive testing (CD-CAT). However, the pervasive challenge of replenishing items in the CD-CAT item bank limits its practical application. Online calibration is a means to address item replenishment, but in CD-CAT, most existing online calibration methods that jointly calibrate the Q-matrix and item parameters of the new items are developed only for dichotomous responses and are time-consuming. Notably, previous studies pay no attention to polytomously scored items that are frequently observed in testing, even though they can offer additional evidence for the examinees' diagnosis. To fill this gap, we propose a SCAD-based method (SCAD-EM) to calibrate the Q-matrix and item parameters of the new items with polytomous response data in order to promote the application of CD-CAT in practice. The performance of the SCAD-EM was investigated in two comprehensive simulation studies and compared against the revised single-item estimation method (SIE-BIC). Results indicated that the SCAD-EM produces a higher calibration accuracy for the category-level Q-matrix and is computationally more efficient across all conditions, but it produces a lower calibration accuracy for the item-level Q-matrix. An empirical study further demonstrated the utility of the SCAD-EM and the SIE-BIC methods in calibrating new items with a real dataset. The advantages of the proposed method, its limitations, and possible future research directions are offered at the end.

A general nonparametric classification method for multiple strategies in cognitive diagnostic assessment.

Cognitive diagnosis models (CDMs) have been used as psychometric tools in educational assessments to estimate students' strengths and weaknesses in terms of cognitive skills learned and skills that need study. In practice, it is not uncommon that questions can often be solved using more than one strategy, which requires CDMs capable of accommodating multiple strategies. However, existing parametric multi-strategy CDMs need a large sample size to produce a reliable estimation of item parameters and examinees' proficiency class memberships, which obstructs their practical applications. This article proposes a general nonparametric multi-strategy classification method with promising classification accuracy in small samples for dichotomous response data. The method can accommodate different strategy selection approaches and different condensation rules. Simulation studies showed that the proposed method outperformed the parametric CDMs when sample sizes were small. A set of real data was analyzed as well to illustrate the application of the proposed method in practice.

Item selection methods in multidimensional computerized adaptive testing for forced-choice items using Thurstonian IRT model.

Multidimensional computerized adaptive testing for forced-choice items (MFC-CAT) combines the benefits of multidimensional forced-choice (MFC) items and computerized adaptive testing (CAT) in that it eliminates response biases and reduces administration time. Previous studies that explored designs of MFC-CAT only discussed item selection methods based on the Fisher information (FI), which is known to perform unstably at early stages of CAT. This study proposes a set of new item selection methods based on the KL information for MFC-CAT (namely MFC-KI, MFC-KB, and MFC-KLP) based on the Thurstonian IRT (TIRT) model. Three simulation studies, including one based on real data, were conducted to compare the performance of the proposed KL-based item selection methods against the existing FI-based methods in three- and five-dimensional MFC-CAT scenarios with various test lengths and inter-trait correlations. Results demonstrate that the proposed KL-based item selection methods are feasible for MFC-CAT and generate acceptable trait estimation accuracy and uniformity of item pool usage. Among the three proposed methods, MFC-KB and MFC-KLP outperformed the existing FI-based item selection methods and resulted in the most accurate trait estimation and relatively even utilization of the item pool.

Measurement invariance and latent mean differences of the morbid curiosity scale (MCS) across the United States and China.

In recent years, there has been a growing body of research focusing on morbid curiosity. However, the development of measurement tools has been slow, with only two scales available. Compared to the unidimensional scale of Curiosity About Morbid Events (CAME) proposed by Zuckerman and Little (1986), the recently developed four-factor Morbid Curiosity Scale (MCS) by Scrivner (2021) demonstrates a stable factor structure and good reliability and validity. As the time since the development of this scale is relatively short, its measurement properties have not been widely evaluated. Therefore, this study used exploratory and confirmatory factor analyses to validate the factor structure of the MCS in the Chinese cultural context, and the results supported the four-factor structure of the MCS. Additionally, we established partial scalar invariance of the MCS between Chinese (N = 663) and American (N = 330) cultures, and further analyzed cultural differences in morbid curiosity using latent mean comparison. The results revealed that Chinese individuals had a lower motivation for understanding the minds of dangerous people. This study validated the four-factor Morbid Curiosity Scale across different cultures for the first time, promoting the generalizability of the four-factor MCS and suggesting its potential for use in a wide range of cultural backgrounds. These findings contribute to enriching cross-cultural research on morbid curiosity and its associated psychological factors.

A Mixed Sequential IRT Model for Mixed-Format Items.

To provide more insight into an individual's response process and cognitive process, this study proposed three mixed sequential item response models (MS-IRMs) for mixed-format items consisting of a mixture of a multiple-choice item and an open-ended item that emphasize a sequential response process and are scored sequentially. Relative to existing polytomous models such as the graded response model (GRM), generalized partial credit model (GPCM), or traditional sequential Rasch model (SRM), the proposed models employ an appropriate processing function for each task to improve conventional polytomous models. Simulation studies were carried out to investigate the performance of the proposed models, and the results indicated that all proposed models outperformed the SRM, GRM, and GPCM in terms of parameter recovery and model fit. An application illustration of the MS-IRMs in comparison with traditional models was demonstrated by using real data from TIMSS 2007.

Psychometric Properties and Measurement Invariance of the LIVES Daily Hassles Scale in Chinese Samples.

Daily hassles have a larger effect on our health and well-being than those major events in daily life. The present study aimed to evaluate the psychometric properties and measurement invariance of the LIVES Daily Hassles Scale (LIVES-DHS) in Chinese samples, which consisted of 815 people at work aged between 20 and 60 years old. The results of both Explanatory Factor Analysis and Confirmatory Factor Analysis showed that the five-factor model solution was better than other solutions, which supported the original structure of LIVES-DHS. The Cronbach's alpha coefficients of the five subdimensions varied between.721 and.818, with the entire scale of.920, and McDonald's ω values of the five subdimensions varied between.716 and.821, with the entire scale of.936. The results also showed the support for measurement invariance of the five-factor model across different groups, which is the first to offer evidence for configural, metric, scalar and strict invariance of LIVES-DHS across gender, age and educational groups.

Using Process Data to Improve Classification Accuracy of Cognitive Diagnosis Model.

With the advance of computer-based assessments, many process data, such as response times (RTs), action sequences, Eye-tracking data, the log data for collaborative problem-solving (CPS) and mouse click/drag becomes readily available. Findings from previous studies (e.g., Peng et al., Multivariate Behavioral Research, 1-20, 2021; Xu, The British Journal of Mathematical and Statistical Psychology, 73(3), 474-505, 2020; He & von Davier, Handbook of research on technology tools for real-world skill development (pp. 750-777). IGI Global, 2016; Man & Harring, Educational and Psychological Measurement, 81(3), 441-465, 2021) suggest a substantial relationship between this human-computer interactive process information and proficiency, which means these process data were potentially useful variables for psychological and educational measurement. To make full use of the process data, this paper aims to combine two useful and easily available types of process data, including the mouse click/drag traces and the response times, to the conventional cognitive diagnostic model (CDM) to better understand individual's response behavior and improve the classification accuracy of existing CDM. Then the full Bayesian analysis using Markov chain Monte Carlo (MCMC) was employed to estimate the proposed model parameters. The viability of the proposed model was investigated by an empirical data and two simulation studies. Results indicated the proposed model combing both types of process data could not only improve the attribute classification reliability in real data analysis, but also provide an improvement on item parameters recovery and person classification accuracy.

A multiple logistic regression-based (MLR-B) Q-matrix validation method for cognitive diagnosis models:A confirmatory approach.

Q-matrix is an essential component specifying the relationship between attributes and items, which plays a key role in cognitive diagnosis assessment. The Q-matrix is usually developed by domain experts and its specifications tend to be subjective and might have misspecifications. Many existing pieces of research concentrate on the validation of Q-matrix; however, few of them can be applied to saturated cognitive diagnosis models. This paper proposes a general and effective Q-matrix validation method by employing multiple logistic regression model. Simulation studies are carried out to investigate the performance of the proposed method and compare it with four existing methods. Simulation results indicate the proposed method outperforms the existing methods in terms of validation accuracy. In addition, a set of real data is used as an example to illustrate its application. Finally, we discuss the limitations of the current study and the directions of future studies.

A new perspective on detecting performance decline: A change-point analysis based on Jensen-Shannon divergence.

A common observation in ability assessment is that the probability of an examinee giving a correct response drops for end-of-test items due to low motivation, time limits or other factors. On the test-takers' side, this change can be considered performance decline (PD), which can strongly affect test validity and bias respondents' ability estimators. Currently, there is an increasing interest in the detection of PD among researchers and practitioners. Researchers and practitioners found that PD detection fails to achieve acceptable power, which is typically below 0.55. Change-point analysis (CPA), a well-developed statistical method, can be applied to item response sequences to identify whether an abrupt change exists. Existing CPA methods cannot be directly used to detect PD because they are appropriate for two-sided alternative hypotheses. To address these issues, this research firstly develops a CPA method based on Jensen-Shannon divergence to detect PD. Additionally, existing CPA statistics were converted into one-sided statistics to accommodate PD detection. Then, a simulation study was conducted to investigate the performance of the proposed method and compare it with modified CPA statistics. Results show that the proposed CPA method can detect PD with higher power while generating a well-controlled Type-I error rate. Compared against modified CPA statistics, the proposed method exhibits an augmentation in power from 1.0% to 8.2%, with average of 5.7% and higher accuracy in locating the change point. Finally, the proposed method was applied to two real datasets to demonstrate its utility.

Two efficient selection methods for high-dimensional CD-CAT utilizing max-marginals factor from MAP query and ensemble learning approach.

Computerized adaptive testing for cognitive diagnosis (CD-CAT) needs to be efficient and responsive in real time to meet practical applications' requirements. For high-dimensional data, the number of categories to be recognized in a test grows exponentially as the number of attributes increases, which can easily cause system reaction time to be too long such that it adversely affects the examinees and thus seriously impacts the measurement efficiency. More importantly, the long-time CPU operations and memory usage of item selection in CD-CAT due to intensive computation are impractical and cannot wholly meet practice needs. This paper proposed two new efficient selection strategies (HIA and CEL) for high-dimensional CD-CAT to address this issue by incorporating the max-marginals from the maximum a posteriori query and integrating the ensemble learning approach into the previous efficient selection methods, respectively. The performance of the proposed selection method was compared with the conventional selection method using simulated and real item pools. The results showed that the proposed methods could significantly improve the measurement efficiency with about 1/2-1/200 of the conventional methods' computation time while retaining similar measurement accuracy. With increasing number of attributes and size of the item pool, the computation time advantage of the proposed methods becomes more significant.

Dual-Objective Item Selection Methods in Computerized Adaptive Test Using the Higher-Order Cognitive Diagnostic Models.

To efficiently obtain information about both the general abilities and detailed cognitive profiles of examinees from a single model that uses a single-calibration process, higher-order cognitive diagnostic computerized adaptive testing (CD-CAT) that employ higher-order cognitive diagnostic models have been developed. However, the current item selection methods used in higher-order CD-CAT adaptively select items according to only the attribute profiles, which might lead to low precision regarding general abilities; hence, an appropriate method was proposed for this CAT system in this study. Under the framework of the higher-order models, the responses were affected by attribute profiles, which were governed by general abilities. It is reasonable to hold that the item responses were affected by a combination of general abilities and attribute profiles. Based on the logic of Shannon entropy and the generalized deterministic, inputs, noisy "and" gate (G-DINA) model discrimination index (GDI), two new item selection methods were proposed for higher-order CD-CAT by considering the above combination in this study. The simulation results demonstrated that the new methods achieved more accurate estimations of both general abilities and cognitive profiles than the existing methods and maintained distinct advantages in terms of item pool usage.

Combining Cognitive Diagnostic Computerized Adaptive Testing With Multidimensional Item Response Theory.

The new generation of tests not only focuses on the general ability but also the process of finer-grained skills. Under the guidance of this thought, researchers have developed a dual-purpose CD-CAT (Dual-CAT). In the existing Dual-CAT, the models used in overall ability estimation are unidimensional IRT models, which cannot apply to the multidimensional tests. This article intends to develop a multidimensional Dual-CAT to improve its applicability. To achieve this goal, this article firstly proposes some item selection methods for the multidimensional Dual-CAT, and then verifies the estimation accuracy and exposure rate of these methods through both simulation study and a real item bank study. The results show that the established multidimensional Dual-CAT is effective and the new proposed methods outperform the traditional methods. Finally, this article discusses the future direction of the Dual-CAT.

A Generalized Diagnostic Classification Modeling Framework Integrating Differential Speediness: Advantages and Illustrations in Psychological and Educational Testing.

To advance the theoretical foundation of incorporating response times (RTs) into diagnostic classification models (DCMs), this study attempts to further derive, test and illustrate a generalized modeling framework (known as the JVRT-LCDM) that can simultaneously analyze response accuracy and differential speediness based on an existing method (Zhan et al., British Journal of Mathematical and Statistical Psychology, 71(2), 262-286, 2018). The JVRT-LCDM not only provides fine-grained diagnostic feedback without strict model constraints but also clarifies the specific speed trajectory of individuals. Moreover, some existing models from psychometric literatures are included in the JVRT-LCDM as special cases. The feasibility of the JVRT-LCDM is investigated via a Monte Carlo simulation study using a Bayesian estimation scheme, and two empirical datasets are then analyzed to illustrate the applicability of the JVRT-LCDM in practice. The results indicate that (1) as a generalized and flexible model, the JVRT-LCDM realizes high correct classification rates and accurate speed parameter recovery; (2) the JVRT-LCDM outperforms the existing models in terms of model-data fit, diagnostic consistency, and estimation of specific individuals in practical cognitive diagnosis assessments; and (3) the JVRT-LCDM provides reliable evidence for nonconstant speed modeling.

Comparison of Psychometric Characteristics for Five Versions of the Interpersonal Needs Questionnaire in Teenagers Sample.

Interpersonal Needs Questionnaire (INQ) is a self-report measure of perceived burdensomeness and thwarted belongingness with five versions in recent studies. There are five versions of INQ. But results from studies using different versions are quite different. Current suicide behavior among teenagers has attracted much attention. But which version is more suitable for teenage samples is still uncertain. It is important to compare the potential differences in different versions of INQ to identify the most psychometrically available version to predict teenagers' acquired capability for suicide and provide them with timely help to reduce teenagers' suicide rates. This study compared the construct validity, internal consistency, validity, and average test information of each version in the sample of teenagers. Results showed the 10-item version provided the most average test information in both thwarted belongingness subscale and perceived burdensomeness subscale, and the INQ-10 is more suitable for teenage samples.

Automated Test Assembly for Multistage Testing With Cognitive Diagnosis.

Computer multistage adaptive test (MST) combines the advantages of paper and pencil-based test (P&P) and computer-adaptive test (CAT). As CAT, MST is adaptive based on modules; as P&P, MST can meet the need of test developers to manage test forms and keep test forms parallel. Cognitive diagnosis (CD) can accurately measure students' knowledge states (KSs) and provide diagnostic information, which is conducive to student's self-learning and teacher's targeted teaching. Although MST and CD have a lot of advantages, many factors prevent MST from applying to CD. In this study, we first attempt to employ automated test assembly (ATA) to achieve the objectives of MST in the application of CD (called CD-MST) via heuristic algorithms. The mean correct response probability of all KSs for each item is used to describe the item difficulty of CD. The attribute reliability in CD is defined as the test quantitative target. A simulation study with the G-DINA model (generalized deterministic input noisy "and" gate model) was carried out to investigate the proposed CD-MST, and the results showed that the assembled panels of CD-MST satisfied the statistical and the non-statistical constraints.

Psychometric Properties of the Short Forms of the Social Interaction Anxiety Scale and the Social Phobia Scale in a Chinese College Sample.

This study was carried out to examine the factor structure and psychometric properties of the Social Interaction Anxiety Scale (SIAS) and Social Phobia Scale (SPS) in a sample of 1,001 Chinese college students (male: 34%; female: 66%). Confirmatory factor analysis (CFA) indicated that the two-factor shortened version of the SIAS-6/SPS-6 fit the data well. In addition, the item response theory (IRT) method confirmed the construct and items for the 12 items of the SIAS-6/SPS-6 with satisfactory discrimination, threshold parameters, and test information curve. It was concluded that the factor structure and psychometric properties of the SIAS-6/SPS-6 support their use for such assessment in a Chinese college sample.

Development and Validation of an Item Bank for Drug Dependence Measurement Using Computer Adaptive Testing.

BACKGROUND: Currently, measurement tools to assess patient-reported outcomes for drug dependence are limited in their latent trait to adapt to the needs of individual patients while also maintaining comparability of scores across patients. Purpose/Objectives: To develop an item bank for computer adaptive testing (CAT) to measure severity of drug dependence. Methods: There were four phases: (1) review the literature of drug dependence measurement; (2) formulate an item list to be assessed by experts; (3) pretest our item list in two substance dependence treatment centers; and (4) field-test and conduct psychometric performance analysis with the final item bank. Additionally, based on our response data, a CAT simulation was used to validate the item bank, Drug Dependence CAT (DD-CAT). Results: The final drug dependence item bank - with a unidimensional configuration - contained 56 items with good item-fit, high discrimination, no differential item functioning, and covered all symptoms of diagnostic criteria for drug dependence. These results revealed that the final item bank was of good quality. Additionally, the results of a simulation CAT procedure with real response data indicated that the DD-CAT item bank exhibited acceptable and reasonable test reliability, content validity, and criterion-related validity. Conclusions/Importance: The proposed item bank for DD-CAT contained acceptable reliability and validity, and exhibited a shorter but efficient assessment of drug dependence. These psychometric properties can result in shorter test times, less information loss, and a reduction in the testing burden of patients.

Developing and Validating an Item Bank for Alcohol Use Disorder Screening in the Chinese Population by Using the Computerized Adaptive Testing.

OBJECTIVE: To detect the individual's severity of alcohol use disorder (AUD) in an effective and accurate manner, this study aimed to build an item bank for AUD screening and derive the computerized adaptive testing (CAT) version of AUD (CAT-AUD). METHODS: The initial CAT-AUD item bank was selected from the Chinese version of the questionnaires related to AUD according to the DSM-5 criteria. Then 915 valid Chinese samples, covering the healthy individuals and the AUD high-risk individuals, completed the initial CAT-AUD item bank. By testing the unidimensionality, test fit, item fit, discrimination parameter and differential item functioning of the initial item bank, the final CAT-AUD item bank confirming to the requirements of the item response theory (IRT) were obtained. Subsequently, the CAT-AUD simulation study based on the real data of the final item bank conducted to detect characteristics, reliability, validity, and predictive utility (sensitivity and specificity) of CAT-AUD. RESULTS: The CAT-AUD item bank meeting the IRT psychometric measurement requirements could be well geared into the graded response model. The Pearson's correlation between the estimated theta via CAT-AUD and the estimated theta via the full-length item bank reached 0.95, and the criterion-related validity was 0.63. CAT-AUD can provide highly reliable test results for subjects whose theta above -0.8 with an average of 16 items. Besides, the predictive utility of CAT-AUD was better than AUDIT and AUDIT-C. CONCLUSION: In brief, the CAT-AUD developed in this study can effectively screen the AUD high-risk group and accurately measure the AUD severity of individuals.

Development of a Computerized Adaptive Test for Separation Anxiety Disorder Among Adolescents.

BACKGROUND: Separation anxiety disorder (SAD) is one of the most common mental disorders among children and adolescents, and it may seriously affect their growth, daily life, and learning. Self-report scales have been used for diagnosis, which require lengthy testing and personnel. METHODS: A total of 1,241 adolescents were recruited from 16 junior- and senior-high schools in China. The initial item bank was selected from classical SAD scales according to the DSM-5. First, the optimal model was selected using item response theory (IRT) according to data fit. Then, per the IRT analysis, items that did not meet the psychometric requirements were deleted (e.g., discriminating values < 0.2). Consequently, a computerized adaptive test (CAT) for SAD was formed (CAT-SAD). RESULTS: An average of 17 items per participant was required to achieve and maintain a 0.3 standard error of measurement in the SAD severity estimate. The estimated correlation of the CAT-SAD with the total 68-item test score was 0.955. CAT-SAD scores were strongly related to the probability of a SAD diagnosis with the Separation Anxiety Assessment Scale-Child and Adolescent Version. Therefore, SAD could be accurately predicted by the CAT-SAD. CONCLUSIONS: Exploratory factor analyses revealed that SAD was unidimensional. The CAT-SAD, which has good reliability and validity and high sensitivity and specificity, provides an efficient test for adolescents with SAD as compared to standard paper-and-pencil tests. It can be used to diagnose varying degrees of SAD quickly and reliably and ease the burden on adolescents. Potential applications for inexpensive, efficient, and accurate screening of SAD are discussed.