A Model-Based Approach to the Disentanglement and Differential Treatment of Engaged and Disengaged Item Omissions.

Item omissions in large-scale assessments may occur for various reasons, ranging from disengagement to not being capable of solving the item and giving up. Current response-time-based classification approaches allow researchers to implement different treatments of item omissions presumably going back to different mechanisms. These approaches, however, are limited in that they require a clear-cut decision on the underlying missingness mechanism and do not allow to take the uncertainty in classification into account. We present a response-time-based model-based mixture modeling approach that overcomes this limitation. The approach (a) facilitates disentangling item omissions stemming from disengagement from those going back to solution behavior, (b) considers the uncertainty in omission classification, (c) allows for omission mechanisms to vary on the item-by-examinee level, (d) supports investigating person and item characteristics associated with different types of omission behavior, and (e) gives researchers flexibility in deciding on how to handle different types of omissions. The approach exhibits good parameter recovery under realistic research conditions. We illustrate the approach on data from the Programme for the International Assessment of Adult Competencies 2012 and compare it against previous classification approaches for item omissions.

Disentangling Different Aspects of Change in Tests with the D-Diffusion Model.

Diffusion-based item response theory models are measurement models that link parameters of the diffusion model (drift rate, boundary separation) to latent traits of test takers. Similar to standard latent trait models, they assume the invariance of the test takers' latent traits during a test. Previous research, however, suggests that traits change as test takers learn or decrease their effort. In this paper, we combine the diffusion-based item response theory model with a latent growth curve model. In the model, the latent traits of each test taker are allowed to change during the test until a stable level is reached. As different change processes are assumed for different traits, different aspects of change can be separated. We discuss different versions of the model that make different assumptions about the form (linear versus quadratic) and rate (fixed versus individual-specific) of change. In order to fit the model to data, we propose a Bayes estimator. Parameter recovery is investigated in a simulation study. The study suggests that parameter recovery is good under certain conditions. We illustrate the application of the model to data measuring visuo-spatial perspective-taking.

Using Response Times for Joint Modeling of Response and Omission Behavior.

For adequate modeling of missing responses, a thorough understanding of the nonresponse mechanisms is vital. As a large number of major testing programs are in the process or already have been moving to computer-based assessment, a rich body of additional data on examinee behavior becomes easily accessible. These additional data may contain valuable information on the processes associated with nonresponse. Bringing together research on item omissions with approaches for modeling response time data, we propose a framework for simultaneously modeling response behavior and omission behavior utilizing timing information for both. As such, the proposed model allows (a) to gain a deeper understanding of response and nonresponse behavior in general and, in particular, of the processes underlying item omissions in LSAs, (b) to model the processes determining the time examinees require to generate a response or to omit an item, and (c) to account for nonignorable item omissions. Parameter recovery of the proposed model is studied within a simulation study. An illustration of the model by means of an application to real data is provided.

Smartphone-based intervention for postpartum depressive symptoms (Smart-e-Moms): study protocol for a randomized controlled trial.

BACKGROUND: Postpartum depression constitutes a significant public health issue, with prevalence rates ranging between 8 and 19% in high-income nations. Nevertheless, numerous barriers, including time constraints, societal stigmatization, and feelings of shame, contribute to the limited utilization of healthcare services during the postpartum period. Digital interventions offer an opportunity to enhance care for women experiencing postpartum depressive symptoms. METHODS: We will conduct a two-arm randomized controlled trial to assess the effectiveness of a smartphone-based intervention in comparison to a treatment-as-usual control group in Germany. Our aim is to randomize 556 participants in a 1:1 ratio. Participants in the intervention group will be provided access to a preventive smartphone-based intervention called "Smart-e-Moms," which incorporates therapeutic support and comprises 10 concise modules rooted in cognitive-behavioral therapy. For the intervention group, evaluations will take place at baseline (t0), prior to sessions 4 and 8 (intermediate assessments), and upon completing the intervention 6 weeks after baseline (t1). The control group's assessments will be at baseline (t0) and 6 weeks after baseline. Follow-up assessments are scheduled at 12 and 24 weeks from baseline to examine the short-term stability of any observed effects. We anticipate that participants in the intervention group will exhibit improvements in their postpartum depressive symptoms (as measured with the Edinburgh Postnatal Depression Scale). Additionally, we will analyze secondary outcomes, including maternal bonding, stress levels, self-efficacy, satisfaction with the intervention, and healthcare utilization. DISCUSSION: If Smart-e-Moms proves to be effective, it has the potential to play a significant role in postpartum depression care within German-speaking regions. Ideally, this intervention could not only benefit maternal well-being but also improve the prospects for healthy child development. TRIAL REGISTRATION: German clinical trials registry DRKS00032324. Registered on January 26, 2024.

The InterModel Vigorish as a Lens for Understanding (and Quantifying) the Value of Item Response Models for Dichotomously Coded Items.

The deployment of statistical models-such as those used in item response theory-necessitates the use of indices that are informative about the degree to which a given model is appropriate for a specific data context. We introduce the InterModel Vigorish (IMV) as an index that can be used to quantify accuracy for models of dichotomous item responses based on the improvement across two sets of predictions (i.e., predictions from two item response models or predictions from a single such model relative to prediction based on the mean). This index has a range of desirable features: It can be used for the comparison of non-nested models and its values are highly portable and generalizable. We use this fact to compare predictive performance across a variety of simulated data contexts and also demonstrate qualitative differences in behavior between the IMV and other common indices (e.g., the AIC and RMSEA). We also illustrate the utility of the IMV in empirical applications with data from 89 dichotomous item response datasets. These empirical applications help illustrate how the IMV can be used in practice and substantiate our claims regarding various aspects of model performance. These findings indicate that the IMV may be a useful indicator in psychometrics, especially as it allows for easy comparison of predictions across a variety of contexts.

Internet-based transdiagnostic treatment for emotional disorders in Arabic- and Farsi-speaking refugees: study protocol of a randomized controlled trial.

BACKGROUND: Refugee populations have an increased risk for mental disorders, such as depression, anxiety, and posttraumatic stress disorders. Comorbidity is common. At the same time, refugees face multiple barriers to accessing mental health treatment. Only a minority of them receive adequate help. The planned trial evaluates a low-threshold, transdiagnostic Internet-based treatment. The trial aims at establishing its efficacy and cost-effectiveness compared with no treatment. METHODS: N = 131 treatment-seeking Arabic- or Farsi-speaking patients, meeting diagnostic criteria for a depressive, anxiety, and/or posttraumatic stress disorder will be randomized to either the intervention or the waitlist control group. The intervention group receives an Internet-based treatment with weekly written guidance provided by Arabic- or Farsi-speaking professionals. The treatment is based on the Common Elements Treatment Approach (CETA), is tailored to the individual patient, and takes 6-16 weeks. The control group will wait for 3 months and then receive the Internet-based treatment. DISCUSSION: The planned trial will result in an estimate of the efficacy of a low-threshold and scalable treatment option for the most common mental disorders in refugees. TRIAL REGISTRATION: German Registry for Clinical Trials DRKS00024154. Registered on February 1, 2021.

A Response-Time-Based Latent Response Mixture Model for Identifying and Modeling Careless and Insufficient Effort Responding in Survey Data.

Careless and insufficient effort responding (C/IER) can pose a major threat to data quality and, as such, to validity of inferences drawn from questionnaire data. A rich body of methods aiming at its detection has been developed. Most of these methods can detect only specific types of C/IER patterns. However, typically different types of C/IER patterns occur within one data set and need to be accounted for. We present a model-based approach for detecting manifold manifestations of C/IER at once. This is achieved by leveraging response time (RT) information available from computer-administered questionnaires and integrating theoretical considerations on C/IER with recent psychometric modeling approaches. The approach a) takes the specifics of attentive response behavior on questionnaires into account by incorporating the distance-difficulty hypothesis, b) allows for attentiveness to vary on the screen-by-respondent level, c) allows for respondents with different trait and speed levels to differ in their attentiveness, and d) at once deals with various response patterns arising from C/IER. The approach makes use of item-level RTs. An adapted version for aggregated RTs is presented that supports screening for C/IER behavior on the respondent level. Parameter recovery is investigated in a simulation study. The approach is illustrated in an empirical example, comparing different RT measures and contrasting the proposed model-based procedure against indicator-based multiple-hurdle approaches.

Increasing the effectiveness of psychotherapy in routine care through blended therapy with transdiagnostic online modules (PsyTOM): study protocol for a randomized controlled trial.

BACKGROUND: In blended therapy, face-to-face psychotherapy and Internet-based interventions are combined. Blended therapy may be advantageous for patients and psychotherapists. However, most blended interventions focus on cognitive behavioral therapy or single disorders, making them less suitable for routine care settings. METHODS: In a randomized controlled trial, we will compare blended therapy and face-to-face therapy in routine care. We intend to randomize 1152 patients nested in 231 psychotherapists in a 1:1 ratio. Patients in the blended therapy group will receive access to a therapeutic online intervention (TONI). TONI contains 12 transdiagnostic online modules suited for psychodynamic, cognitive behavioral, and systemic therapy. Psychotherapists decide which modules to assign and how to integrate TONI components into the psychotherapeutic process to tailor treatment to their patients' specific needs. We will assess patients at baseline, 6 weeks, 12 weeks, and 6 months. Patients enrolled early in the trial will also complete assessments at 12 months. The primary outcomes are depression and anxiety at 6-month post-randomization, as measured by PHQ-8 and GAD-7. The secondary outcomes include satisfaction with life, level of functioning, personality traits and functioning, eating pathology, sexual problems, alcohol/drug use, satisfaction with treatment, negative effects, and mental health care utilization. In addition, we will collect several potential moderators and mediators, including therapeutic alliance, agency, and self-efficacy. Psychotherapists will also report on changes in symptom severity and therapeutic alliance. Qualitative interviews with psychotherapists and patients will shed light on the barriers and benefits of the blended intervention. Furthermore, we will assess significant others of enrolled patients in a sub-study. DISCUSSION: The integration of online modules which use a common therapeutic language and address therapeutic principles shared across therapeutic approaches into regular psychotherapy has the potential to improve the effectiveness of psychotherapy and transfer it into everyday life as well help save therapists' resources and close treatment gaps. A modular and transdiagnostic setup of the blended intervention also enables psychotherapists to tailor their treatment optimally to the needs of their patients. TRIAL REGISTRATION: German Clinical Trials Register (DRKS) DRKS00028536. Registered on 07.06.2022.

Partial Measurement Invariance: Extending and Evaluating the Cluster Approach for Identifying Anchor Items.

When measurement invariance does not hold, researchers aim for partial measurement invariance by identifying anchor items that are assumed to be measurement invariant. In this paper, we build on Bechger and Maris's approach for identification of anchor items. Instead of identifying differential item functioning (DIF)-free items, they propose to identify different sets of items that are invariant in item parameters within the same item set. We extend their approach by an additional step in order to allow for identification of homogeneously functioning item sets. We evaluate the performance of the extended cluster approach under various conditions and compare its performance to that of previous approaches, that are the equal-mean difficulty (EMD) approach and the iterative forward approach. We show that the EMD and the iterative forward approaches perform well in conditions with balanced DIF or when DIF is small. In conditions with large and unbalanced DIF, they fail to recover the true group mean differences. With appropriate threshold settings, the cluster approach identified a cluster that resulted in unbiased mean difference estimates in all conditions. Compared to previous approaches, the cluster approach allows for a variety of different assumptions as well as for depicting the uncertainty in the results that stem from the choice of the assumption. Using a real data set, we illustrate how the assumptions of the previous approaches may be incorporated in the cluster approach and how the chosen assumption impacts the results.

Combining Clickstream Analyses and Graph-Modeled Data Clustering for Identifying Common Response Processes.

Complex interactive test items are becoming more widely used in assessments. Being computer-administered, assessments using interactive items allow logging time-stamped action sequences. These sequences pose a rich source of information that may facilitate investigating how examinees approach an item and arrive at their given response. There is a rich body of research leveraging action sequence data for investigating examinees' behavior. However, the associated timing data have been considered mainly on the item-level, if at all. Considering timing data on the action-level in addition to action sequences, however, has vast potential to support a more fine-grained assessment of examinees' behavior. We provide an approach that jointly considers action sequences and action-level times for identifying common response processes. In doing so, we integrate tools from clickstream analyses and graph-modeled data clustering with psychometrics. In our approach, we (a) provide similarity measures that are based on both actions and the associated action-level timing data and (b) subsequently employ cluster edge deletion for identifying homogeneous, interpretable, well-separated groups of action patterns, each describing a common response process. Guidelines on how to apply the approach are provided. The approach and its utility are illustrated on a complex problem-solving item from PIAAC 2012.

A Multiprocess Item Response Model for Not-Reached Items due to Time Limits and Quitting.

So far, modeling approaches for not-reached items have considered one single underlying process. However, missing values at the end of a test can occur for a variety of reasons. On the one hand, examinees may not reach the end of a test due to time limits and lack of working speed. On the other hand, examinees may not attempt all items and quit responding due to, for example, fatigue or lack of motivation. We use response times retrieved from computerized testing to distinguish missing data due to lack of speed from missingness due to quitting. On the basis of this information, we present a new model that allows to disentangle and simultaneously model different missing data mechanisms underlying not-reached items. The model (a) supports a more fine-grained understanding of the processes underlying not-reached items and (b) allows to disentangle different sources describing test performance. In a simulation study, we evaluate estimation of the proposed model. In an empirical study, we show what insights can be gained regarding test-taking behavior using this model.

A hierarchical latent response model for inferences about examinee engagement in terms of guessing and item-level non-response.

In low-stakes assessments, test performance has few or no consequences for examinees themselves, so that examinees may not be fully engaged when answering the items. Instead of engaging in solution behaviour, disengaged examinees might randomly guess or generate no response at all. When ignored, examinee disengagement poses a severe threat to the validity of results obtained from low-stakes assessments. Statistical modelling approaches in educational measurement have been proposed that account for non-response or for guessing, but do not consider both types of disengaged behaviour simultaneously. We bring together research on modelling examinee engagement and research on missing values and present a hierarchical latent response model for identifying and modelling the processes associated with examinee disengagement jointly with the processes associated with engaged responses. To that end, we employ a mixture model that identifies disengagement at the item-by-examinee level by assuming different data-generating processes underlying item responses and omissions, respectively, as well as response times associated with engaged and disengaged behaviour. By modelling examinee engagement with a latent response framework, the model allows assessing how examinee engagement relates to ability and speed as well as to identify items that are likely to evoke disengaged test-taking behaviour. An illustration of the model by means of an application to real data is presented.

Do balanced scales assess bipolar constructs? The case of the STAI scales.

Balanced scales, that is, scales based on items whose content is either negatively or positively polarized, are often used in the hope of measuring a bipolar construct. Research has shown that usually balanced scales do not yield 1-dimensional measurements. This threatens their construct validity. The authors show how to test bipolarity while accounting for method effects. This is demonstrated on a data set of state and trait anxiety measured with the State-Trait Anxiety Inventory (STAI; C. D. Spielberger, R. L. Gorsuch, R. Lushene, P. R. Vagg, & G. A. Jacobs, 1983) scales. Taking a test-retest perspective, assuming temporally stable method effects, the authors tested the bipolarity of the temporal change through suitable constraints specified in a structural equation model adapted from S. Vautier, R. Steyer, and A. Boomsma (2008). The model fit the data closely, chi(2)(13, N = 888) = 20.75, p = .07. Thus, the state and trait scales seem to measure bipolar constructs plus temporally stable method effects. Parameter estimates suggest reliable change scores for the state anxiety scale (rho = .90) and specific method effects for the state and trait scales of the STAI.

Compensation and Amplification of Attenuation Bias in Causal Effect Estimates.

Covariate-adjusted treatment effects are commonly estimated in non-randomized studies. It has been shown that measurement error in covariates can bias treatment effect estimates when not appropriately accounted for. So far, these delineations primarily assumed a true data generating model that included just one single covariate. It is, however, more plausible that the true model consists of more than one covariate. We evaluate when a further covariate may reduce bias due to measurement error in another covariate and in which cases it is not recommended to include a further covariate. We analytically derive the amount of bias related to the fallible covariate's reliability and systematically disentangle bias compensation and amplification due to an additional covariate. With a fallible covariate, it is not always beneficial to include an additional covariate for adjustment, as the additional covariate can extensively increase the bias. The mechanisms for an increased bias due to an additional covariate can be complex, even in a simple setting of just two covariates. A high reliability of the fallible covariate or a high correlation between the covariates cannot in general prevent from substantial bias. We show distorting effects of a fallible covariate in an empirical example and discuss adjustment for latent covariates as a possible solution.

Using Response Times to Model Not-Reached Items due to Time Limits.

Missing values at the end of a test typically are the result of test takers running out of time and can as such be understood by studying test takers' working speed. As testing moves to computer-based assessment, response times become available allowing to simulatenously model speed and ability. Integrating research on response time modeling with research on modeling missing responses, we propose using response times to model missing values due to time limits. We identify similarities between approaches used to account for not-reached items (Rose et al. in ETS Res Rep Ser 2010:i-53, 2010) and the speed-accuracy (SA) model for joint modeling of effective speed and effective ability as proposed by van der Linden (Psychometrika 72(3):287-308, 2007). In a simulation, we show (a) that the SA model can recover parameters in the presence of missing values due to time limits and (b) that the response time model, using item-level timing information rather than a count of not-reached items, results in person parameter estimates that differ from missing data IRT models applied to not-reached items. We propose using the SA model to model the missing data process and to use both, ability and speed, to describe the performance of test takers. We illustrate the application of the model in an empirical analysis.

When Nonresponse Mechanisms Change: Effects on Trends and Group Comparisons in International Large-Scale Assessments.

Mechanisms causing item nonresponses in large-scale assessments are often said to be nonignorable. Parameter estimates can be biased if nonignorable missing data mechanisms are not adequately modeled. In trend analyses, it is plausible for the missing data mechanism and the percentage of missing values to change over time. In this article, we investigated (a) the extent to which the missing data mechanism and the percentage of missing values changed over time in real large-scale assessment data, (b) how different approaches for dealing with missing data performed under such conditions, and (c) the practical implications for trend estimates. These issues are highly relevant because the conclusions hold for all kinds of group mean differences in large-scale assessments. In a reanalysis of PISA (Programme for International Student Assessment) data from 35 OECD countries, we found that missing data mechanisms and numbers of missing values varied considerably across time points, countries, and domains. In a simulation study, we generated data in which we allowed the missing data mechanism and the amount of missing data to change over time. We showed that the trend estimates were biased if differences in the missing-data mechanisms were not taken into account, in our case, when omissions were scored as wrong, when omissions were ignored, or when model-based approaches assuming a constant missing data mechanism over time were used. The results suggest that the most accurate estimates can be obtained from the application of multiple group models for nonignorable missing values when the amounts of missing data and the missing data mechanisms changed over time. In an empirical example, we furthermore showed that the large decline in PISA reading literacy in Ireland in 2009 was reduced when we estimated trends using missing data treatments that accounted for changes in missing data mechanisms.

Psychometric Modelling of Longitudinal Genetically Informative Twin Data.

The often-used A(C)E model that decomposes phenotypic variance into parts due to additive genetic and environmental influences can be extended to a longitudinal model when the trait has been assessed at multiple occasions. This enables inference about the nature (e.g., genetic or environmental) of the covariance among the different measurement points. In the case that the measurement of the phenotype relies on self-report data (e.g., questionnaire data), often, aggregated scores (e.g., sum-scores) are used as a proxy for the phenotype. However, earlier research based on the univariate ACE model that concerns a single measurement occasion has shown that this can lead to an underestimation of heritability and that instead, one should prefer to model the raw item data by integrating an explicit measurement model into the analysis. This has, however, not been translated to the more complex longitudinal case. In this paper, we first present a latent state twin A(C)E model that combines the genetic twin model with an item response theory (IRT) model as well as its specification in a Bayesian framework. Two simulation studies were conducted to investigate 1) how large the bias is when sum-scores are used in the longitudinal A(C)E model and 2) if using the latent twin model can overcome the potential bias. Results of the first simulation study (e.g., AE model) demonstrated that using a sum-score approach leads to underestimated heritability estimates and biased covariance estimates. Surprisingly, the IRT approach also lead to bias, but to a much lesser degree. The amount of bias increased in the second simulation study (e.g., ACE model) under both frameworks, with the IRT approach still being the less biased approach. Since the bias was less severe under the IRT approach than under the sum-score approach and due to other advantages of latent variable modelling, we still advise researcher to adopt the IRT approach. We further illustrate differences between the traditional sum-score approach and the latent state twin A(C)E model by analyzing data of a two-wave twin study, consisting of the answers of 8,016 twins on a scale developed to measure social attitudes related to conservatism.

When does measurement error in covariates impact causal effect estimates? Analytic derivations of different scenarios and an empirical illustration.

The average causal treatment effect (ATE) can be estimated from observational data based on covariate adjustment. Even if all confounding covariates are observed, they might not necessarily be reliably measured and may fail to obtain an unbiased ATE estimate. Instead of fallible covariates, the respective latent covariates can be used for covariate adjustment. But is it always necessary to use latent covariates? How well do analysis of covariance (ANCOVA) or propensity score (PS) methods estimate the ATE when latent covariates are used? We first analytically delineate the conditions under which latent instead of fallible covariates are necessary to obtain the ATE. Then we empirically examine the difference between ATE estimates when adjusting for fallible or latent covariates in an applied example. We discuss the issue of fallible covariates within a stochastic theory of causal effects and analyse data of a within-study comparison with recently developed ANCOVA and PS procedures that allow for latent covariates. We show that fallible covariates do not necessarily bias ATE estimates, but point out different scenarios in which adjusting for latent covariates is required. In our empirical application, we demonstrate how latent covariates can be incorporated for ATE estimation in ANCOVA and in PS analysis.