The Accuracy of Bayesian Model Fit Indices in Selecting Among Multidimensional Item Response Theory Models.

Item response theory (IRT) models are often compared with respect to predictive performance to determine the dimensionality of rating scale data. However, such model comparisons could be biased toward nested-dimensionality IRT models (e.g., the bifactor model) when comparing those models with non-nested-dimensionality IRT models (e.g., a unidimensional or a between-item-dimensionality model). The reason is that, compared with non-nested-dimensionality models, nested-dimensionality models could have a greater propensity to fit data that do not represent a specific dimensional structure. However, it is unclear as to what degree model comparison results are biased toward nested-dimensionality IRT models when the data represent specific dimensional structures and when Bayesian estimation and model comparison indices are used. We conducted a simulation study to add clarity to this issue. We examined the accuracy of four Bayesian predictive performance indices at differentiating among non-nested- and nested-dimensionality IRT models. The deviance information criterion (DIC), a commonly used index to compare Bayesian models, was extremely biased toward nested-dimensionality IRT models, favoring them even when non-nested-dimensionality models were the correct models. The Pareto-smoothed importance sampling approximation of the leave-one-out cross-validation was the least biased, with the Watanabe information criterion and the log-predicted marginal likelihood closely following. The findings demonstrate that nested-dimensionality IRT models are not automatically favored when the data represent specific dimensional structures as long as an appropriate predictive performance index is used.

Multilevel mediation analysis in R: A comparison of bootstrap and Bayesian approaches.

Mediation analysis in repeated measures studies can shed light on the mechanisms through which experimental manipulations change the outcome variable. However, the literature on interval estimation for the indirect effect in the 1-1-1 single mediator model is sparse. Most simulation studies to date evaluating mediation analysis in multilevel data considered scenarios that do not match the expected numbers of level 1 and level 2 units typically encountered in experimental studies, and no study to date has compared resampling and Bayesian methods for constructing intervals for the indirect effect in this context. We conducted a simulation study to compare statistical properties of interval estimates of the indirect effect obtained using four bootstrap and two Bayesian methods in the 1-1-1 mediation model with and without random effects. Bayesian credibility intervals had coverage closest to the nominal value and no instances of excessive Type I error rates, but lower power than resampling methods. Findings indicated that the pattern of performance for resampling methods often depended on the presence of random effects. We provide suggestions for selecting an interval estimator for the indirect effect depending on the most important statistical property for a given study, as well as code in R for implementing all methods evaluated in the simulation study. Findings and code from this project will hopefully support the use of mediation analysis in experimental research with repeated measures.

Model-agnostic unsupervised detection of bots in a Likert-type questionnaire.

To detect bots in online survey data, there is a wealth of literature on statistical detection using only responses to Likert-type items. There are two traditions in the literature. One tradition requires labeled data, forgoing strong model assumptions. The other tradition requires a measurement model, forgoing collection of labeled data. In the present article, we consider the problem where neither requirement is available, for an inventory that has the same number of Likert-type categories for all items. We propose a bot detection algorithm that is both model-agnostic and unsupervised. Our proposed algorithm involves a permutation test with leave-one-out calculations of outlier statistics. For each respondent, it outputs a p value for the null hypothesis that the respondent is a bot. Such an algorithm offers nominal sensitivity calibration that is robust to the bot response distribution. In a simulation study, we found our proposed algorithm to improve upon naive alternatives in terms of 95% sensitivity calibration and, in many scenarios, in terms of classification accuracy.

Comparison of latent variable and psychological network models in PROMIS data: output metrics and factor structure.

PURPOSE: Much research is still needed to compare traditional latent variable models such as confirmatory factor analysis (CFA) to emerging psychometric models such as the Gaussian graphical model (GGM). Previous comparisons of GGM centrality indices with factor loadings from CFA have discovered redundancies, and investigations into how well a GGM-based alternative to exploratory factor analysis (i.e., exploratory graph analysis, or EGA) is able to recover the hypothesized factor structure show mixed results. Importantly, such comparisons have not typically been examined in real mental and physical health symptom data, despite such data being an excellent candidate for the GGM. Our goal was to extend previous work by comparing the GGM and CFA using data from Wave 1 of the Patient Reported Outcomes Measurement Information System (PROMIS). METHODS: Models were fit to PROMIS data based on 16 test forms designed to measure 9 mental and physical health domains. Our analyses borrowed a two-stage approach for handling missing data from the structural equation modeling literature. RESULTS: We found weaker correspondence between centrality indices and factor loadings than found by previous research, but in a similar pattern of correspondence. EGA recommended a factor structure discrepant with PROMIS domains in most cases yet may be taken to provide substantive insight into the dimensionality of PROMIS domains. CONCLUSION: In real mental and physical health data, the GGM and EGA may provide complementary information to traditional CFA metrics.

Cognitive fatigue weakens the effects of distraction on pain.

ABSTRACT: Perceived pain can be viewed because of a competition between nociceptive inputs and other competing goals, such as performing a demanding cognitive task. Task performance, however, suffers when cognitively fatigued. We therefore predicted that cognitive fatigue would weaken the pain-reducing effects of performing a concurrent cognitive task, which would indicate a causal link between fatigue and heightened pain sensitivity. In this study, 2 groups of pain-free adults performed cognitive tasks while receiving painful heat stimuli. In 1 group, we induced cognitive fatigue before performing the tasks. We found that fatigue led to more pain and worse performance when the task was demanding, suggesting that fatigue weakens one's ability to distract from pain. These findings show that cognitive fatigue can impair performance on subsequent tasks and that this impairment can lower a person's ability to distract from and reduce their pain.

Supervised Classes, Unsupervised Mixing Proportions: Detection of Bots in a Likert-Type Questionnaire.

Administering Likert-type questionnaires to online samples risks contamination of the data by malicious computer-generated random responses, also known as bots. Although nonresponsivity indices (NRIs) such as person-total correlations or Mahalanobis distance have shown great promise to detect bots, universal cutoff values are elusive. An initial calibration sample constructed via stratified sampling of bots and humans-real or simulated under a measurement model-has been used to empirically choose cutoffs with a high nominal specificity. However, a high-specificity cutoff is less accurate when the target sample has a high contamination rate. In the present article, we propose the supervised classes, unsupervised mixing proportions (SCUMP) algorithm that chooses a cutoff to maximize accuracy. SCUMP uses a Gaussian mixture model to estimate, unsupervised, the contamination rate in the sample of interest. A simulation study found that, in the absence of model misspecification on the bots, our cutoffs maintained accuracy across varying contamination rates.

Parsimony in model selection: Tools for assessing fit propensity.

Theories can be represented as statistical models for empirical testing. There is a vast literature on model selection and multimodel inference that focuses on how to assess which statistical model, and therefore which theory, best fits the available data. For example, given some data, one can compare models on various information criterion or other fit statistics. However, what these indices fail to capture is the full range of counterfactuals. That is, some models may fit the given data better not because they represent a more correct theory, but simply because these models have more fit propensity-a tendency to fit a wider range of data, even nonsensical data, better. Current approaches fall short in considering the principle of parsimony (Occam's Razor), often equating it with the number of model parameters. Here we offer a toolkit for researchers to better study and understand parsimony through the fit propensity of structural equation models. We provide an R package (ockhamSEM) built on the popular lavaan package. To illustrate the importance of evaluating fit propensity, we use ockhamSEM to investigate the factor structure of the Rosenberg Self-Esteem Scale. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Validating the German version of the Personality Disorder Severity-ICD-11 Scale using nominal response models.

The International Classification of Diseases (ICD-11) features a new classification of personality disorders (PD), focusing on the severity of PD. Although there are numerous self-report measures that assess PD severity, to date only the Personality Disorder Severity-ICD-11 (PDS-ICD-11) is based on ICD-11's operationalization of PD. Initial results indicated that the PDS-ICD-11 measures a unidimensional construct, but the assumptions made for scoring its bipolar items had not been fully examined. The aim of this study is to fill this gap and investigate the latent structure of the German version of the PDS-ICD-11 using nominal response models (NRM), which allow for testing these assumptions. We applied the PDS-ICD-11 together with other self-report measures in a sample of 1,228 individuals from the general population. NRM indicated an acceptable fit of a unidimensional model, with only few deviations from the theoretically imposed scoring scheme. The total score was sufficiently reliable and correlated meaningfully with other self-report measures of PD severity. Regarding Diagnostic and Statistical Manual of Mental Disorders (DSM-5) and ICD-11 maladaptive trait domains, the total score was found to be most strongly associated with negative affectivity, whereas associations with antagonism and anankastia were small or nonsignificant. We conclude that the proposed scoring scheme of the PDS-ICD-11 items is acceptable, and the examined psychometric properties of the German version largely correspond to the results from the English-language development study. The total score, however, depicts more internalizing than externalizing personality pathology. Future studies should investigate the diagnostic efficiency of the PDS-ICD-11 scale using multiple methods and time points as well as clinical and forensic samples. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Distraction from pain depends on task demands and motivation.

Introduction: Pain captures attention automatically, yet we can inhibit pain when we are motivated to perform other tasks. Previous studies show that engaging in a cognitively demanding task reduces pain compared with a task that is minimally demanding, yet the effects of motivation on this pain-reducing effect remain largely unexplored. Objectives: In this study, we hypothesized that motivating people to engage in a task with high demands would lead to more cognitive resources directed toward the task, thereby amplifying its pain-reducing effects. Methods: On different trials, participants performed an easy (left-right arrow discrimination) or demanding (2-back) cognitive task while receiving nonpainful or painful heat stimuli. In half of the trials, monetary rewards were offered to motivate participants to engage and perform well in the task. Results: Results showed an interaction between task demands and rewards, whereby offering rewards strengthened the pain-reducing effect of a distracting task when demands were high. This effect was reinforced by increased 2-back performance when rewards were offered, indicating that both task demands and motivation are necessary to inhibit pain. Conclusions: When task demands are low, motivation to engage in the task will have little impact on pain because performance cannot further increase. When motivation is low, participants will spend minimal effort to perform well in the task, thus hindering the pain-reducing effects of higher task demands. These findings suggest that the pain-reducing properties of distraction can be optimized by carefully calibrating the demands and motivational value of the task.

A Comparison of Modern and Popular Approaches to Calculating Reliability for Dichotomously Scored Items.

Recent work on reliability coefficients has largely focused on continuous items, including critiques of Cronbach's alpha. Although two new model-based reliability coefficients have been proposed for dichotomous items (Dimitrov, 2003a,b; Green & Yang, 2009a), these approaches have yet to be compared to each other or other popular estimates of reliability such as omega, alpha, and the greatest lower bound. We seek computational improvements to one of these model-based reliability coefficients and, in addition, conduct initial Monte Carlo simulations to compare coefficients using dichotomous data. Our results suggest that such improvements to the model-based approach are warranted, while model-based approaches were generally superior.

On the Performance of Semi- and Nonparametric Item Response Functions in Computer Adaptive Tests.

Large-scale assessments often use a computer adaptive test (CAT) for selection of items and for scoring respondents. Such tests often assume a parametric form for the relationship between item responses and the underlying construct. Although semi- and nonparametric response functions could be used, there is scant research on their performance in a CAT. In this work, we compare parametric response functions versus those estimated using kernel smoothing and a logistic function of a monotonic polynomial. Monotonic polynomial items can be used with traditional CAT item selection algorithms that use analytical derivatives. We compared these approaches in CAT simulations with a variety of item selection algorithms. Our simulations also varied the features of the calibration and item pool: sample size, the presence of missing data, and the percentage of nonstandard items. In general, the results support the use of semi- and nonparametric item response functions in a CAT.

More flexible response functions for the PROMIS physical functioning item bank by application of a monotonic polynomial approach.

PURPOSE: In developing item banks for patient reported outcomes (PROs), nonparametric techniques are often used for investigating empirical item response curves, whereas final banks usually use parsimonious parametric models. A flexible approach based on monotonic polynomials (MP) provides a compromise by modeling items with both complex and simpler response curves. This paper investigates the suitability of MPs to PRO data. METHOD: Using PROMIS Wave 1 data (N = 15,725) for Physical Function, we fitted an MP model and the graded response model (GRM). We compared both models in terms of overall model fit, latent trait estimates, and item/test information. We quantified possible GRM item misfit using approaches that compute discrepancies with the MP. Through simulations, we investigated the ability of the MP to perform well versus the GRM under identical data collection conditions. RESULTS: A likelihood ratio test (p < 0.001) and AIC (but not BIC) indicated better fit for the MP. Latent trait estimates and expected test scores were comparable between models, but we observed higher information for the MP in the lower range of physical functioning. Many items were flagged as possibly misfitting and simulations supported the performance of the MP. Yet discrepancies between the MP and GRM were small. CONCLUSION: The MP approach allows inclusion of items with complex response curves into PRO item banks. Information for the physical functioning item bank may be greater than originally thought for low levels of physical functioning. This may translate into small improvements if an MP approach is used.

An approach to structural equation modeling with both factors and components: Integrated generalized structured component analysis.

In this article, we propose integrated generalized structured component analysis (IGSCA), which is a general statistical approach for analyzing data with both components and factors in the same model, simultaneously. This approach combines generalized structured component analysis (GSCA) and generalized structured component analysis with measurement errors incorporated (GSCAM) in a unified manner and can estimate both factor- and component-model parameters, including component and factor loadings, component and factor path coefficients, and path coefficients connecting factors and components. We conduct 2 simulation studies to investigate the performance of IGSCA under models with both factors and components. The first simulation study assesses how existing approaches for structural equation modeling and IGSCA recover parameters. This study shows that only consistent partial least squares (PLSc) and IGSCA yield unbiased estimates of all parameters, whereas the other approaches always provided biased estimates of several parameters. As such, we conduct a second, extensive simulation study to evaluate the relative performance of the 2 competitors (PLSc and IGSCA), considering a variety of experimental factors (model specification, sample size, the number of indicators per factor/component, and exogenous factor/component correlation). IGSCA exhibits better performance than PLSc under most conditions. We also present a real data application of IGSCA to the study of genes and their influence on depression. Finally, we discuss the implications and limitations of this approach, and recommendations for future research. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

A Comparison of Limited-Information Test Statistics for a Response Style MIRT Model.

An increased use of models for measuring response styles is apparent in recent years with the multidimensional nominal response model (MNRM) as one prominent example. Inclusion of latent constructs representing extreme (ERS) or midpoint response style (MRS) often improves model fit according to information criteria. However, a test of absolute model fit is often not reported even though it could comprise an important piece of validity evidence. Limited information test statistics are candidates for this task, including the full (M2), ordinal (M2*), and mixed (C2) statistics, which differ in whether additional collapsing of univariate or bivariate contingency tables is conducted. Such collapsing makes sense when item categories are ordinal, which may not hold under the MNRM. More generally, limited information test statistics have gone unevaluated under nominal data and non-ordinal latent trait models. We present a simulation study evaluating the performance of M2, M2*, and C2 with the MNRM. Manipulated conditions included sample size, presence and type of response style, and strength of item slopes on substantive and style dimensions. We found that M2 sometimes had inflated Type I error rates, M2* always had little power, and C2 lacked power under some conditions. M2 and C2 may provide complementary and valuable information regarding model fit.

The Monotonic Polynomial Graded Response Model: Implementation and a Comparative Study.

We present a monotonic polynomial graded response (GRMP) model that subsumes the unidimensional graded response model for ordered categorical responses and results in flexible category response functions. We suggest improvements in the parameterization of the polynomial underlying similar models, expand upon an underlying response variable derivation of the model, and in lieu of an overall discrimination parameter we propose an index to aid in interpreting the strength of relationship between the latent variable and underlying item responses. In applications, the GRMP is compared to two approaches: (a) a previously developed monotonic polynomial generalized partial credit (GPCMP) model; and (b) logistic and probit variants of the heteroscedastic graded response (HGR) model that we estimate using maximum marginal likelihood with the expectation-maximization algorithm. Results suggest that the GRMP can fit real data better than the GPCMP and the probit variant of the HGR, but is slightly outperformed by the logistic HGR. Two simulation studies compared the ability of the GRMP and logistic HGR to recover category response functions. While the GRMP showed some ability to recover HGR response functions and those based on kernel smoothing, the HGR was more specific in the types of response functions it could recover. In general, the GRMP and HGR make different assumptions regarding the underlying response variables, and can result in different category response function shapes.

Estimation of Response Styles Using the Multidimensional Nominal Response Model: A Tutorial and Comparison With Sum Scores.

Recent years have seen a dramatic increase in item response models for measuring response styles on Likert-type items. These model-based approaches stand in contrast to traditional sum-score-based methods where researchers count the number of times that participants selected certain response options. The multidimensional nominal response model (MNRM) offers a flexible model-based approach that may be intuitive to those familiar with sum score approaches. This paper presents a tutorial on the model along with code for estimating it using three different software packages: flexMIRT®, mirt, and Mplus. We focus on specification and interpretation of response functions. In addition, we provide analytical details on how sum score to scale score conversion can be done with the MNRM. In the context of a real data example, three different scoring approaches are then compared. This example illustrates how sum-score-based approaches can sometimes yield scores that are confounded with substantive content. We expect that the current paper will facilitate further investigations as to whether different substantive conclusions are reached under alternative approaches to measuring response styles.

OpenMx: A Modular Research Environment for Item Response Theory Method Development.

There are many item response theory software packages designed for users. Here, the authors introduce an environment tailored to method development and simulation. Implementations of a selection of classic algorithms are available as well as some recently developed methods. Source code is developed in public repositories on GitHub; your collaboration is welcome.

On Lagrange Multiplier Tests in Multidimensional Item Response Theory: Information Matrices and Model Misspecification.

Lagrange multiplier (LM) or score tests have seen renewed interest for the purpose of diagnosing misspecification in item response theory (IRT) models. LM tests can also be used to test whether parameters differ from a fixed value. We argue that the utility of LM tests depends on both the method used to compute the test and the degree of misspecification in the initially fitted model. We demonstrate both of these points in the context of a multidimensional IRT framework. Through an extensive Monte Carlo simulation study, we examine the performance of LM tests under varying degrees of model misspecification, model size, and different information matrix approximations. A generalized LM test designed specifically for use under misspecification, which has apparently not been previously studied in an IRT framework, performed the best in our simulations. Finally, we reemphasize caution in using LM tests for model specification searches.

A flexible full-information approach to the modeling of response styles.

We present a flexible full-information approach to modeling multiple user-defined response styles across multiple constructs of interest. The model is based on a novel parameterization of the multidimensional nominal response model that separates estimation of overall item slopes from the scoring functions (indicating the order of categories) for each item and latent trait. This feature allows the definition of response styles to vary across items as well as overall item slopes that vary across items for both substantive and response style dimensions. We compared the model with similar approaches using examples from the smoking initiative of the Patient-Reported Outcomes Measurement Information System. A small set of simulations showed that the estimation approach is able to recover model parameters, factor scores, and reasonable estimates of standard errors. Furthermore, these simulations suggest that failing to include response style factors (when present in the data generating model) has adverse consequences for substantive trait factor score recovery. (PsycINFO Database Record

Maximum Marginal Likelihood Estimation of a Monotonic Polynomial Generalized Partial Credit Model with Applications to Multiple Group Analysis.

We present a semi-parametric approach to estimating item response functions (IRF) useful when the true IRF does not strictly follow commonly used functions. Our approach replaces the linear predictor of the generalized partial credit model with a monotonic polynomial. The model includes the regular generalized partial credit model at the lowest order polynomial. Our approach extends Liang's (A semi-parametric approach to estimate IRFs, Unpublished doctoral dissertation, 2007) method for dichotomous item responses to the case of polytomous data. Furthermore, item parameter estimation is implemented with maximum marginal likelihood using the Bock-Aitkin EM algorithm, thereby facilitating multiple group analyses useful in operational settings. Our approach is demonstrated on both educational and psychological data. We present simulation results comparing our approach to more standard IRF estimation approaches and other non-parametric and semi-parametric alternatives.