A screen-time-based mixture model for identifying and monitoring careless and insufficient effort responding in ecological momentary assessment data.

Ecological momentary assessment (EMA) involves repeated real-time sampling of respondents' current behaviors and experiences. The intensive repeated assessment imposes an increased burden on respondents, rendering EMAs vulnerable to respondent noncompliance and/or careless and insufficient effort responding (C/IER). We developed a mixture modeling approach that equips researchers with a tool for (a) gauging the degree of C/IER contamination of their EMA data and (b) studying the trajectory of C/IER across the study. For separating attentive from C/IER behavior, the approach leverages collateral information from screen times, which are routinely recorded in electronically administered EMAs, and translates theoretical considerations on respondents' behavior into component models for attentive and careless screen times as well as for the functional form of C/IER trajectories. We show how a sensible choice of component models (a) allows disentangling short screen times due to C/IER from familiarity effects due to repeated exposure to the same measures, (b) aids in gaining a fine-grained understanding of C/IER trajectories by distinguishing within-day from between-day effects, and (c) allows investigating interindividual differences in attentiveness. The approach shows good parameter recovery when attentive and C/IER screen time distributions exhibit sufficient separation and yields valid conclusions even in scenarios of uncontaminated data. The approach is illustrated on EMA data from the German Socio-Economic Panel innovation sample. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Accounting for careless and insufficient effort responding in large-scale survey data-development, evaluation, and application of a screen-time-based weighting procedure.

Careless and insufficient effort responding (C/IER) poses a major threat to the quality of large-scale survey data. Traditional indicator-based procedures for its detection are limited in that they are only sensitive to specific types of C/IER behavior, such as straight lining or rapid responding, rely on arbitrary threshold settings, and do not allow taking the uncertainty of C/IER classification into account. Overcoming these limitations, we develop a two-step screen-time-based weighting procedure for computer-administered surveys. The procedure allows considering the uncertainty in C/IER identification, is agnostic towards the specific types of C/IE response patterns, and can feasibly be integrated with common analysis workflows for large-scale survey data. In Step 1, we draw on mixture modeling to identify subcomponents of log screen time distributions presumably stemming from C/IER. In Step 2, the analysis model of choice is applied to item response data, with respondents' posterior class probabilities being employed to downweigh response patterns according to their probability of stemming from C/IER. We illustrate the approach on a sample of more than 400,000 respondents being administered 48 scales of the PISA 2018 background questionnaire. We gather supporting validity evidence by investigating relationships between C/IER proportions and screen characteristics that entail higher cognitive burden, such as screen position and text length, relating identified C/IER proportions to other indicators of C/IER as well as by investigating rank-order consistency in C/IER behavior across screens. Finally, in a re-analysis of the PISA 2018 background questionnaire data, we investigate the impact of the C/IER adjustments on country-level comparisons.

Reciprocal relationships between parental and scholastic homework assistance and students' academic functioning at elementary school.

Homework assistance is provided both by parents and by institutions, for example, full-day schools. Previous research found evidence that the quality of homework assistance - measured by three dimensions derived from the self-determination theory, namely, responsiveness, structure, and control - is reciprocally related with students' academic functioning (i.e., achievement and homework behavior). However, findings on parental homework assistance have been consistent only for the secondary level, whereas elementary school students have been studied less and previous results obtained for this population are inconclusive. Moreover, research on homework assistance that is given at school is scarce. Therefore, the present study aimed to investigate whether reciprocal associations between the quality of homework assistance and students' academic functioning in elementary school can be found (1) for parental homework assistance and (2) for scholastic homework assistance. We calculated cross-lagged panel models based on longitudinal data from 335 German fourth graders collected in autumn 2019 (September and October) and winter 2020 (February and March). The analyses for scholastic homework assistance were based on a subsample of 112 students. Whereas responsiveness and structure did not predict students' outcomes in the way we expected, control had unfavorable relationships in both homework settings. Moreover, parental control was reciprocally related with students' mean grades in three subjects. The positive forms of homework assistance (responsiveness and structure) were predicted by different indicators of academic functioning in the two homework settings.

Pooling methods for likelihood ratio tests in multiply imputed data sets.

Likelihood ratio tests (LRTs) are a popular tool for comparing statistical models. However, missing data are also common in empirical research, and multiple imputation (MI) is often used to deal with them. In multiply imputed data, there are multiple options for conducting LRTs, and new methods are still being proposed. In this article, we compare all available methods in multiple simulations covering applications in linear regression, generalized linear models, and structural equation modeling. In addition, we implemented these methods in an R package, and we illustrate its application in an example analysis concerned with the investigation of measurement invariance. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

The Dimensionality of Reading Self-Concept: Examining Its Stability Using Local Structural Equation Models.

For results from large-scale surveys to inform policy and practice appropriately, all participants must interpret and respond to items similarly. While organizers of surveys assessing student outcomes often ensure this for achievement measures, doing so for psychological questionnaires is also critical. We demonstrate this by examining the dimensionality of reading self-concept-a crucial psychological construct for several outcomes-across reading achievement levels. We use Programme for International Student Assessment 2018 data (N = 529,966) and local structural equation models (LSEMs) to do so. Results reveal that reading self-concept dimensions (assessed through reading competence and difficulty) vary across reading achievement levels. Students with low reading achievement show differentiated responses to the two item sets (high competence-high difficulty). In contrast, students with high reading achievement have reconciled responses (high competence-low difficulty). Our results highlight the value of LSEMs in examining factor structure generalizability of constructs in large-scale surveys and call for greater cognitive testing during item development.

Alleviating estimation problems in small sample structural equation modeling-A comparison of constrained maximum likelihood, Bayesian estimation, and fixed reliability approaches.

Small sample structural equation modeling (SEM) may exhibit serious estimation problems, such as failure to converge, inadmissible solutions, and unstable parameter estimates. A vast literature has compared the performance of different solutions for small sample SEM in contrast to unconstrained maximum likelihood (ML) estimation. Less is known, however, on the gains and pitfalls of different solutions in contrast to each other. Focusing on three current solutions-constrained ML, Bayesian methods using Markov chain Monte Carlo techniques, and fixed reliability single indicator (SI) approaches-we bridge this gap. When doing so, we evaluate the potential and boundaries of different parameterizations, constraints, and weakly informative prior distributions for improving the quality of the estimation procedure and stabilizing parameter estimates. The performance of all approaches is compared in a simulation study. Under conditions with low reliabilities, Bayesian methods without additional prior information by far outperform constrained ML in terms of accuracy of parameter estimates as well as the worst-performing fixed reliability SI approach and do not perform worse than the best-performing fixed reliability SI approach. Under conditions with high reliabilities, constrained ML shows good performance. Both constrained ML and Bayesian methods exhibit conservative to acceptable Type I error rates. Fixed reliability SI approaches are prone to undercoverage and severe inflation of Type I error rates. Stabilizing effects on Bayesian parameter estimates can be achieved even with mildly incorrect prior information. In an empirical example, we illustrate the practical importance of carefully choosing the method of analysis for small sample SEM. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

A Bayesian Approach to Estimating Reciprocal Effects with the Bivariate STARTS Model.

The bivariate Stable Trait, AutoRegressive Trait, and State (STARTS) model provides a general approach for estimating reciprocal effects between constructs over time. However, previous research has shown that this model is difficult to estimate using the maximum likelihood (ML) method (e.g., nonconvergence). In this article, we introduce a Bayesian approach for estimating the bivariate STARTS model and implement it in the software Stan. We discuss issues of model parameterization and show how appropriate prior distributions for model parameters can be selected. Specifically, we propose the four-parameter beta distribution as a flexible prior distribution for the autoregressive and cross-lagged effects. Using a simulation study, we show that the proposed Bayesian approach provides more accurate estimates than ML estimation in challenging data constellations. An example is presented to illustrate how the Bayesian approach can be used to stabilize the parameter estimates of the bivariate STARTS model.

Meta-analyzing individual participant data from studies with complex survey designs: A tutorial on using the two-stage approach for data from educational large-scale assessments.

Descriptive analyses of socially important or theoretically interesting phenomena and trends are a vital component of research in the behavioral, social, economic, and health sciences. Such analyses yield reliable results when using representative individual participant data (IPD) from studies with complex survey designs, including educational large-scale assessments (ELSAs) or social, health, and economic survey and panel studies. The meta-analytic integration of these results offers unique and novel research opportunities to provide strong empirical evidence of the consistency and generalizability of important phenomena and trends. Using ELSAs as an example, this tutorial offers methodological guidance on how to use the two-stage approach to IPD meta-analysis to account for the statistical challenges of complex survey designs (e.g., sampling weights, clustered and missing IPD), first, to conduct descriptive analyses (Stage 1), and second, to integrate results with three-level meta-analytic and meta-regression models to take into account dependencies among effect sizes (Stage 2). The two-stage approach is illustrated with IPD on reading achievement from the Programme for International Student Assessment (PISA). We demonstrate how to analyze and integrate standardized mean differences (e.g., gender differences), correlations (e.g., with students' socioeconomic status [SES]), and interactions between individual characteristics at the participant level (e.g., the interaction between gender and SES) across several PISA cycles. All the datafiles and R scripts we used are available online. Because complex social, health, or economic survey and panel studies share many methodological features with ELSAs, the guidance offered in this tutorial is also helpful for synthesizing research evidence from these studies.

A machine learning-based procedure for leveraging clickstream data to investigate early predictability of failure on interactive tasks.

Early detection of risk of failure on interactive tasks comes with great potential for better understanding how examinees differ in their initial behavior as well as for adaptively tailoring interactive tasks to examinees' competence levels. Drawing on procedures originating in shopper intent prediction on e-commerce platforms, we introduce and showcase a machine learning-based procedure that leverages early-window clickstream data for systematically investigating early predictability of behavioral outcomes on interactive tasks. We derive features related to the occurrence, frequency, sequentiality, and timing of performed actions from early-window clickstreams and use extreme gradient boosting for classification. Multiple measures are suggested to evaluate the quality and utility of early predictions. The procedure is outlined by investigating early predictability of failure on two PIAAC 2012 Problem Solving in Technology Rich Environments (PSTRE) tasks. We investigated early windows of varying size in terms of time and in terms of actions. We achieved good prediction performance at stages where examinees had, on average, at least two thirds of their solution process ahead of them, and the vast majority of examinees who failed could potentially be detected to be at risk before completing the task. In-depth analyses revealed different features to be indicative of success and failure at different stages of the solution process, thereby highlighting the potential of the applied procedure for gaining a finer-grained understanding of the trajectories of behavioral patterns on interactive tasks.

A three-dimensional taxonomy of achievement emotions.

We present a three-dimensional taxonomy of achievement emotions that considers valence, arousal, and object focus as core features of these emotions. By distinguishing between positive and negative emotions (valence), activating and deactivating emotions (arousal), and activity emotions, prospective outcome emotions, and retrospective outcome emotions (object focus), the taxonomy has a 2 × 2 × 3 structure representing 12 groups of achievement emotions. In four studies across different countries (N = 330, 235, 323, and 269 participants in Canada, the United States, Germany, and the U.K., respectively), we investigated the empirical robustness of the taxonomy in educational (Studies 1-3) and work settings (Study 4). An expanded version of the Achievement Emotions Questionnaire was used to assess 12 key emotions representing the taxonomy. Consistently across the four studies, findings from multilevel facet analysis and structural equation modeling documented the importance of the three dimensions for explaining achievement emotions. In addition, based on hypotheses about relations with external variables, the findings show clear links of the emotions with important antecedents and outcomes. The Big Five personality traits, appraisals of control and value, and context perceptions were predictors of the emotions. The 12 emotions, in turn, were related to participants' use of strategies, cognitive performance, and self-reported health problems. Taken together, the findings provide robust evidence for the unique positions of different achievement emotions in the proposed taxonomy, as well as unique patterns of relations with external variables. Directions for future research and implications for policy and practice are discussed. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Using synthetic data to improve the reproducibility of statistical results in psychological research.

In recent years, psychological research has faced a credibility crisis, and open data are often regarded as an important step toward a more reproducible psychological science. However, privacy concerns are among the main reasons that prevent data sharing. Synthetic data procedures, which are based on the multiple imputation (MI) approach to missing data, can be used to replace sensitive data with simulated values, which can be analyzed in place of the original data. One crucial requirement of this approach is that the synthesis model is correctly specified. In this article, we investigated the statistical properties of synthetic data with a particular emphasis on the reproducibility of statistical results. To this end, we compared conventional approaches to synthetic data based on MI with a data-augmented approach (DA-MI) that attempts to combine the advantages of masking methods and synthetic data, thus making the procedure more robust to misspecification. In multiple simulation studies, we found that the good properties of the MI approach strongly depend on the correct specification of the synthesis model, whereas the DA-MI approach can provide useful results even under various types of misspecification. This suggests that the DA-MI approach to synthetic data can provide an important tool that can be used to facilitate data sharing and improve reproducibility in psychological research. In a working example, we also demonstrate the implementation of these approaches in widely available software, and we provide recommendations for practice. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Causal Inference with Multilevel Data: A Comparison of Different Propensity Score Weighting Approaches.

Propensity score methods are a widely recommended approach to adjust for confounding and to recover treatment effects with non-experimental, single-level data. This article reviews propensity score weighting estimators for multilevel data in which individuals (level 1) are nested in clusters (level 2) and nonrandomly assigned to either a treatment or control condition at level 1. We address the choice of a weighting strategy (inverse probability weights, trimming, overlap weights, calibration weights) and discuss key issues related to the specification of the propensity score model (fixed-effects model, multilevel random-effects model) in the context of multilevel data. In three simulation studies, we show that estimates based on calibration weights, which prioritize balancing the sample distribution of level-1 and (unmeasured) level-2 covariates, should be preferred under many scenarios (i.e., treatment effect heterogeneity, presence of strong level-2 confounding) and can accommodate covariate-by-cluster interactions. However, when level-1 covariate effects vary strongly across clusters (i.e., under random slopes), and this variation is present in both the treatment and outcome data-generating mechanisms, large cluster sizes are needed to obtain accurate estimates of the treatment effect. We also discuss the implementation of survey weights and present a real-data example that illustrates the different methods.

A Comparison of Penalized Maximum Likelihood Estimation and Markov Chain Monte Carlo Techniques for Estimating Confirmatory Factor Analysis Models With Small Sample Sizes.

With small to modest sample sizes and complex models, maximum likelihood (ML) estimation of confirmatory factor analysis (CFA) models can show serious estimation problems such as non-convergence or parameter estimates outside the admissible parameter space. In this article, we distinguish different Bayesian estimators that can be used to stabilize the parameter estimates of a CFA: the mode of the joint posterior distribution that is obtained from penalized maximum likelihood (PML) estimation, and the mean (EAP), median (Med), or mode (MAP) of the marginal posterior distribution that are calculated by using Markov Chain Monte Carlo (MCMC) methods. In two simulation studies, we evaluated the performance of the Bayesian estimators from a frequentist point of view. The results show that the EAP produced more accurate estimates of the latent correlation in many conditions and outperformed the other Bayesian estimators in terms of root mean squared error (RMSE). We also argue that it is often advantageous to choose a parameterization in which the main parameters of interest are bounded, and we suggest the four-parameter beta distribution as a prior distribution for loadings and correlations. Using simulated data, we show that selecting weakly informative four-parameter beta priors can further stabilize parameter estimates, even in cases when the priors were mildly misspecified. Finally, we derive recommendations and propose directions for further research.

Multiple imputation of missing data in multilevel models with the R package mdmb: a flexible sequential modeling approach.

Multilevel models often include nonlinear effects, such as random slopes or interaction effects. The estimation of these models can be difficult when the underlying variables contain missing data. Although several methods for handling missing data such as multiple imputation (MI) can be used with multilevel data, conventional methods for multilevel MI often do not properly take the nonlinear associations between the variables into account. In the present paper, we propose a sequential modeling approach based on Bayesian estimation techniques that can be used to handle missing data in a variety of multilevel models that involve nonlinear effects. The main idea of this approach is to decompose the joint distribution of the data into several parts that correspond to the outcome and explanatory variables in the intended analysis, thus generating imputations in a manner that is compatible with the substantive analysis model. In three simulation studies, we evaluate the sequential modeling approach and compare it with conventional as well as other substantive-model-compatible approaches to multilevel MI. We implemented the sequential modeling approach in the R package mdmb and provide a worked example to illustrate its application.

Is emotional exhaustion only the result of work experiences? A diary study on daily hassles and uplifts in different life domains.

BACKGROUND AND OBJECTIVES: Emotional exhaustion is considered to be the key symptom of burnout. Although it has been defined that emotional exhaustion rather results from work-related experience than from other life domains, this has rarely been studied empirically. The study aimed to investigate the role of different life domains in predicting emotional exhaustion. More precisely, we examined whether daily uplifts and hassles from different life domains were related to emotional exhaustion beyond work-related uplifts and hassles. DESIGN: A diary study was conducted over the course of 14 consecutive days. METHOD: 141 beginning teachers provided information about their daily hassles and uplifts as well as their daily emotional exhaustion. RESULT: Results of multilevel analyses showed that work-related uplifts were negatively and work-related hassles were positively related to emotional exhaustion. Additionally, private uplifts were associated with a statistically significant decrease and private hassles with an increase in emotional exhaustion beyond work-related events. CONCLUSION: Although the variance in emotional exhaustion that was explained by private events was small, the present study suggests that burnout symptoms might not be completely independent from individuals' daily lives outside work.

Maximum likelihood estimation of a social relations structural equation model.

The social relations model (SRM) is widely used in psychology to investigate the components that underlie interpersonal perceptions, behaviors, and judgments. SRM researchers are often interested in investigating the multivariate relations between SRM effects. However, at present, it is not possible to investigate such relations without relying on a two-step approach that depends on potentially unreliable estimates of the true SRM effects. Here, we introduce a way to combine the SRM with the structural equation modeling (SEM) framework and show how the parameters of our combination can be estimated with a maximum likelihood (ML) approach. We illustrate the model with an example from personality psychology. We also investigate the statistical properties of the model in a small simulation study showing that our approach performs well in most simulation conditions. An R package (called srm) is available implementing the proposed methods.

Reanalysis of the German PISA Data: A Comparison of Different Approaches for Trend Estimation With a Particular Emphasis on Mode Effects.

International large-scale assessments, such as the Program for International Student Assessment (PISA), are conducted to provide information on the effectiveness of education systems. In PISA, the target population of 15-year-old students is assessed every 3 years. Trends show whether competencies have changed in the countries between PISA cycles. In order to provide valid trend estimates, it is desirable to retain the same test conditions and statistical methods in all PISA cycles. In PISA 2015, however, the test mode changed from paper-based to computer-based tests, and the scaling method was changed. In this paper, we investigate the effects of these changes on trend estimation in PISA using German data from all PISA cycles (2000-2015). Our findings suggest that the change from paper-based to computer-based tests could have a severe impact on trend estimation but that the change of the scaling model did not substantially change the trend estimates.

Analysis of Interactions and Nonlinear Effects with Missing Data: A Factored Regression Modeling Approach Using Maximum Likelihood Estimation.

When estimating multiple regression models with incomplete predictor variables, it is necessary to specify a joint distribution for the predictor variables. A convenient assumption is that this distribution is a multivariate normal distribution, which is also the default in many statistical software packages. This distribution will in general be misspecified if predictors with missing data have nonlinear effects (e.g., x2) or are included in interaction terms (e.g., x·z). In the present article, we introduce a factored regression modeling approach for estimating regression models with missing data that is based on maximum likelihood estimation. In this approach, the model likelihood is factorized into a part that is due to the model of interest and a part that is due to the model for the incomplete predictors. In three simulation studies, we showed that the factored regression modeling approach produced valid estimates of interaction and nonlinear effects in regression models with missing values on categorical or continuous predictor variables under a broad range of conditions. We developed the R package mdmb, which facilitates a user-friendly application of the factored regression modeling approach, and present a real-data example that illustrates the flexibility of the software.

Assessing the instructional quality of private tutoring and its effects on student outcomes: Analyses from the German National Educational Panel Study.

BACKGROUND: A considerable proportion of secondary school students gets additional support from private tutoring lessons. Empirical findings on the effectiveness of private tutoring are inconclusive. The instructional quality of the tutoring lessons might influence their effectiveness. AIMS: We analysed (1) whether students privately tutored in the subjects German or mathematics outperformed their non-tutored counterparts; (2) whether the instructional quality of the tutoring lessons can be described with the three dimensions, structure, challenge, and support; and (3) the effects of these quality dimensions on the students' academic achievement and their global satisfaction with their school and family situation. SAMPLE: We used data from the German National Educational Panel Study (NEPS). From a total sample of N = 11,358 10th-grade secondary school students, subsamples provided information about tutoring quality in mathematics (nM  = 1,714), German (nG  = 148), and English (nE  = 490). METHODS: The model of three correlated dimensions of tutoring quality was evaluated with confirmatory factor analyses, and the effects on student outcomes were tested in a structural equation modelling framework. RESULTS: Analyses revealed neither global effects of private tutoring nor effects of its instructional quality on students' grades in mathematics or German. However, the support dimension was positively related to students' satisfaction with their school situation. CONCLUSIONS: Private tutoring cannot be recommended as a generally effective strategy to improve academic achievement in school, but might be helpful to relieve the stress level induced by insufficient achievement in school.