Many nonnormalities, one simulation: Do different data generation algorithms affect study results?

Monte Carlo simulation studies are among the primary scientific outputs contributed by methodologists, guiding application of various statistical tools in practice. Although methodological researchers routinely extend simulation study findings through follow-up work, few studies are ever replicated. Simulation studies are susceptible to factors that can contribute to replicability failures, however. This paper sought to conduct a meta-scientific study by replicating one highly cited simulation study (Curran et al., Psychological Methods, 1, 16-29, 1996) that investigated the robustness of normal theory maximum likelihood (ML)-based chi-square fit statistics under multivariate nonnormality. We further examined the generalizability of the original study findings across different nonnormal data generation algorithms. Our replication results were generally consistent with original findings, but we discerned several differences. Our generalizability results were more mixed. Only two results observed under the original data generation algorithm held completely across other algorithms examined. One of the most striking findings we observed was that results associated with the independent generator (IG) data generation algorithm vastly differed from other procedures examined and suggested that ML was robust to nonnormality for the particular factor model used in the simulation. Findings point to the reality that extant methodological recommendations may not be universally valid in contexts where multiple data generation algorithms exist for a given data characteristic. We recommend that researchers consider multiple approaches to generating a specific data or model characteristic (when more than one is available) to optimize the generalizability of simulation results.

The Pull-to-Sit Task: Examining Infant Postural Development in Autism Spectrum Disorder.

OBJECTIVE: To evaluate the predictive relationship between early trajectories of postural and head control during a pull-to-sit task and later autism diagnostic and developmental outcomes. STUDY DESIGN: Using a prospective longitudinal design, postural skills of 100 infants at elevated and low familial likelihood of autism spectrum disorder (ASD) were evaluated using a pull-to-sit task monthly from age 1 month to 6 months. At age 24 months, infants were seen for a developmental and diagnostic evaluation completed by examiners masked to participant group. Latent growth curve models were used to compare early trajectories of pull-to-sit performance in infants later diagnosed with ASD and typically developing infants and to predict developmental outcomes. RESULTS: Pull-to-sit trajectories did not differ in infants with an elevated likelihood of ASD or infants with ASD compared with low-likelihood and typically developing infants, but infants with ASD were more likely to exhibit a head lag by age 4 months. In addition, pull-to-sit trajectories were predictive of social and speech skills 2 years later. CONCLUSIONS: These findings highlight the link between very early pull-to-sit skills and later social and language outcomes. Atypical postural development and persistent presence of head lag may be important early indicators of social and language vulnerabilities, including ASD.

Detecting heterogeneity in the causal direction of dependence: A model-based recursive partitioning approach.

Methods of causal discovery and direction of dependence to evaluate causal properties of variable relations have experienced rapid development. The majority of causal discovery methods, however, relies on the assumption of causal effect homogeneity, that is, the identified causal structure is expected to hold for the entire population. Because causal mechanisms can vary across subpopulations, we propose combining methods of model-based recursive partitioning and non-Gaussian causal discovery to identify such subpopulations. The resulting algorithm can discover subpopulations with potentially varying magnitude and causal direction of effects under mild parameter inequality assumptions. Feasibility conditions are described and results from synthetic data experiments are presented suggesting that large effects and large sample sizes are beneficial for detecting causally competing subgroups with acceptable statistical performance. In a real-world data example, the extraction of meaningful subgroups that differ in the causal mechanism underlying the development of numerical cognition is illustrated. Potential extensions and recommendations for best practice applications are discussed.

Neonatal neurobehavior in infants with autism spectrum disorder.

AIM: To investigate neurobehavioral maturation for neonates who are later diagnosed with autism spectrum disorder (ASD). METHOD: In a prospective longitudinal design, neonatal neurobehavior was examined monthly in 1- to 3-month-old infants at elevated and low familial likelihood of ASD (n=60). At 2 years, infants were seen for a clinical best-estimate evaluation, resulting in 18 infants with ASD and 36 typically developing infants. Repeated-measures analysis of variance models were conducted to examine the effects of age, diagnostic group, and their interactions. RESULTS: Neurobehavioral maturation of infants diagnosed with ASD was largely comparable to typically developing infants from 1 to 3 months, with the exception of the development of attention. Object-focused attention was significantly attenuated for infants with ASD beginning at 2 to 3 months and was predictive of social-communication skills 2 years later. INTERPRETATION: This is the first study to prospectively examine neonatal neurobehavior of infants at an elevated familial likelihood of ASD who later received a diagnosis. Despite relatively intact neurological and behavioral maturation in the neonatal period, attention to objects emerged as a key early indicator of ASD. This suggests a complex attentional vulnerability within the first 3 months of life that may be associated with cascading sequelae of social-communication challenges and the emergence of ASD.

Evaluating SEM Model Fit with Small Degrees of Freedom.

Research has revealed that the performance of root mean square error of approximation (RMSEA) in assessing structural equation models with small degrees of freedom (df) is suboptimal, often resulting in the rejection of correctly specified or closely fitted models. This study investigates the performance of standardized root mean square residual (SRMR) and comparative fit index (CFI) in small df models with various levels of factor loadings, sample sizes, and model misspecifications. We find that, in comparison with RMSEA, population SRMR and CFI are less susceptible to the effects of df. In small df models, the sample SRMR and CFI could provide more useful information to differentiate models with various levels of misfit. The confidence intervals and p-values of a close fit were generally accurate for all three fit indices. We recommend researchers use caution when interpreting RMSEA for models with small df and to rely more on SRMR and CFI.

Temperamental Shyness and Anger/Frustration in Childhood: Normative Development, Individual Differences, and the Impacts of Maternal Intrusiveness and Frontal Electroencephalogram Asymmetry.

This study used latent growth curve modeling to identify normative development and individual differences in the developmental patterns of shyness and anger/frustration across childhood. This study also examined the impacts of maternal intrusiveness and frontal electroencephalogram (EEG) asymmetry at age 4 on the developmental patterns of shyness and anger/frustration. 180 children (92 boys, 88 girls; Mage = 4.07 years at baseline; 75.6% White, 18.3% Black, 6.1% multiracial/other) participated in the study. Normative development included significant linear decreases in shyness and anger/frustration. Individual variation existed in the developmental patterns. Children with left frontal EEG asymmetry showed a faster decreasing pattern of shyness. Children who experienced higher maternal intrusiveness and had left frontal EEG asymmetry showed a slower decreasing pattern of anger/frustration.

Using a linear mixed-effect model framework to estimate multivariate generalizability theory parameters in R.

Multivariate generalizability theory (mG-theory) is an important framework in many behavioral and educational studies, as it describes useful psychometric properties of multidimensional assessments. Nevertheless, the use of mG-theory estimation is limited due to the lack of available software for carrying out the necessary calculations: users rely heavily on independent software programs such as mGENOVA and the BUGS/JAGS suite of programs. Considering the prevalence of R software, this paper presents a solution using the glmmTMB package to accomplish the estimation task. Users adopting the proposed method may find it more convenient for conducting both applied investigation and simulation studies without the need to switch between different software programs.

Evaluating Factorial Invariance: An Interval Estimation Approach Using Bayesian Structural Equation Modeling.

In this study, we introduce an interval estimation approach based on Bayesian structural equation modeling to evaluate factorial invariance. For each tested parameter, the size of noninvariance with an uncertainty interval (i.e. highest density interval [HDI]) is assessed via Bayesian parameter estimation. By comparing the most credible values (i.e. 95% HDI) with a region of practical equivalence (ROPE), the Bayesian approach allows researchers to (1) support the null hypothesis of practical invariance, and (2) examine the practical importance of the noninvariant parameter. Compared to the traditional likelihood ratio test, simulation results suggested that the proposed Bayesian approach could offer additional insight into evaluating factorial invariance, thus, leading to more informative conclusions. We provide an empirical example to demonstrate the procedures necessary to implement the proposed method in applied research. The importance of and influences on the choice of an appropriate ROPE are discussed.

The Relationship Between the Standardized Root Mean Square Residual and Model Misspecification in Factor Analysis Models.

We argue that the definition of close fitting models should embody the notion of substantially ignorable misspecifications (SIM). A SIM model is a misspecified model that might be selected, based on parsimony, over the true model should knowledge of the true model be available. Because in applications the true model (i.e., the data generating mechanism) is unknown, we investigate the relationship between the population standardized root mean square residual (SRMR) values and various model misspecifications in factor analysis models to better understand the magnitudes of the SRMR. Summary effect sizes of misfit such as the SRMR are necessarily insensitive to some non-ignorable localized misspecifications (i.e., the presence of a few large residual correlations in large models). Localized misspecifications may be identified by examining the largest standardized residual covariance. Based on the findings, our population reference values for close fit are based on a two-index strategy: (1) largest absolute value of standardized residual covariance ≤0.10, and (2) SRMR ≤0.05× R¯2 the average R2 of the manifest variables; for acceptable fit our values are 0.15 and 0.10× R¯2 , respectively.

Bayesian SEM for Specification Search Problems in Testing Factorial Invariance.

Specification search problems refer to two important but under-addressed issues in testing for factorial invariance: how to select proper reference indicators and how to locate specific non-invariant parameters. In this study, we propose a two-step procedure to solve these issues. Step 1 is to identify a proper reference indicator using the Bayesian structural equation modeling approach. An item is selected if it is associated with the highest likelihood to be invariant across groups. Step 2 is to locate specific non-invariant parameters, given that a proper reference indicator has already been selected in Step 1. A series of simulation analyses show that the proposed method performs well under a variety of data conditions, and optimal performance is observed under conditions of large magnitude of non-invariance, low proportion of non-invariance, and large sample sizes. We also provide an empirical example to demonstrate the specific procedures to implement the proposed method in applied research. The importance and influences are discussed regarding the choices of informative priors with zero mean and small variances. Extensions and limitations are also pointed out.

Evaluating Close Fit in Ordinal Factor Analysis Models With Multiply Imputed Data.

Multiple imputation (MI) is one of the recommended techniques for handling missing data in ordinal factor analysis models. However, methods for computing MI-based fit indices under ordinal factor analysis models have yet to be developed. In this short note, we introduced the methods of using the standardized root mean squared residual (SRMR) and the root mean square error of approximation (RMSEA) to assess the fit of ordinal factor analysis models with multiply imputed data. Specifically, we described the procedure for computing the MI-based sample estimates and constructing the confidence intervals. Simulation results showed that the proposed methods could yield sufficiently accurate point and interval estimates for both SRMR and RMSEA, especially in conditions with larger sample sizes, less missing data, more response categories, and higher degrees of misfit. Based on the findings, implications and recommendations were discussed.

Factor Structure of the Teacher Strengths and Difficulties Questionnaire in a Large Community-Based Sample: An Investigation of Alternative Measurement Models.

The Strengths and Difficulties Questionnaire (SDQ) is a screening measure commonly used to assess behavioral and emotional symptoms and strengths among children and adolescents. However, despite its frequent use, its underlying factor structure remains an important area of inquiry. Whereas the original five-factor structure has often been supported through exploratory factor analysis, results from confirmatory analyses continue to yield mixed results. We analyzed data from youth in Grades K through 12 from a large epidemiologic study in the Southeastern United States. Teacher-report SDQ data were used to test three confirmatory factor models by school level (i.e., elementary [Grades K-5] and secondary [Grades 6-12]): The original five-factor model, a three-factor model, and a bifactor model. Model fit indices and reliability measures supported the original five-factor model as the preferred model when using the teacher-reported SDQ with both elementary and secondary school children. Implications for using the SDQ in applied research and predictive modeling are discussed.

Is the Area Under Curve Appropriate for Evaluating the Fit of Psychometric Models?

In the literature of modern psychometric modeling, mostly related to item response theory (IRT), the fit of model is evaluated through known indices, such as χ2, M2, and root mean square error of approximation (RMSEA) for absolute assessments as well as Akaike information criterion (AIC), consistent AIC (CAIC), and Bayesian information criterion (BIC) for relative comparisons. Recent developments show a merging trend of psychometric and machine learnings, yet there remains a gap in the model fit evaluation, specifically the use of the area under curve (AUC). This study focuses on the behaviors of AUC in fitting IRT models. Rounds of simulations were conducted to investigate AUC's appropriateness (e.g., power and Type I error rate) under various conditions. The results show that AUC possessed certain advantages under certain conditions such as high-dimensional structure with two-parameter logistic (2PL) and some three-parameter logistic (3PL) models, while disadvantages were also obvious when the true model is unidimensional. It cautions researchers about the dangers of using AUC solely in evaluating psychometric models.

One step at a time: A statistical approach for distinguishing mediators, confounders, and colliders using direction dependence analysis.

In observational data, understanding the causal link when estimating the causal effect of an independent variable (x) on a dependent variable (y) often requires researchers to identify the role of a third variable in the x → y relationship. Mediation, confounding, and colliding are three key third-variable effects that yield different theoretical and methodological implications for drawing causal conclusions. Commonly used covariance-based statistical methods, such as linear regression and structural equation modeling, cannot distinguish these effects in practice, however. In this study, we introduce a statistical approach for distinguishing mediators, confounders, colliders, and potential M-bias structures that uses higher-order moment information from the data. We propose a two-step procedure that uses the Hilbert-Schmidt independence criterion within the direction dependence analysis framework. Results from Monte Carlo simulations show that our proposed approach accurately recovers the true data-generating process of the third variable. We provide an empirical example to demonstrate the application of our proposed approach in psychological research. Finally, we discuss implications and future directions of our work. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Applying Negative Binomial Distribution in Diagnostic Classification Models for Analyzing Count Data.

Diagnostic classification models (DCMs) have been used to classify examinees into groups based on their possession status of a set of latent traits. In addition to traditional item-based scoring approaches, examinees may be scored based on their completion of a series of small and similar tasks. Those scores are usually considered as count variables. To model count scores, this study proposes a new class of DCMs that uses the negative binomial distribution at its core. We explained the proposed model framework and demonstrated its use through an operational example. Simulation studies were conducted to evaluate the performance of the proposed model and compare it with the Poisson-based DCM.

The NEAT Equating Via Chaining Random Forests in the Context of Small Sample Sizes: A Machine-Learning Method.

The part of responses that is absent in the nonequivalent groups with anchor test (NEAT) design can be managed to a planned missing scenario. In the context of small sample sizes, we present a machine learning (ML)-based imputation technique called chaining random forests (CRF) to perform equating tasks within the NEAT design. Specifically, seven CRF-based imputation equating methods are proposed based on different data augmentation methods. The equating performance of the proposed methods is examined through a simulation study. Five factors are considered: (a) test length (20, 30, 40, 50), (b) sample size per test form (50 versus 100), (c) ratio of common/anchor items (0.2 versus 0.3), and (d) equivalent versus nonequivalent groups taking the two forms (no mean difference versus a mean difference of 0.5), and (e) three different types of anchors (random, easy, and hard), resulting in 96 conditions. In addition, five traditional equating methods, (1) Tucker method; (2) Levine observed score method; (3) equipercentile equating method; (4) circle-arc method; and (5) concurrent calibration based on Rasch model, were also considered, plus seven CRF-based imputation equating methods for a total of 12 methods in this study. The findings suggest that benefiting from the advantages of ML techniques, CRF-based methods that incorporate the equating result of the Tucker method, such as IMP_total_Tucker, IMP_pair_Tucker, and IMP_Tucker_cirlce methods, can yield more robust and trustable estimates for the "missingness" in an equating task and therefore result in more accurate equated scores than other counterparts in short-length tests with small samples.

Teacher-student relationships and adolescents' school satisfaction: Behavioural engagement as a mechanism of change.

BACKGROUND: Teacher-student relationships have been linked to various aspects of students' school functioning, including social-emotional well-being in school, but the underlying mechanisms need more investigation. AIMS: In this study, we analysed longitudinal data to test if students' classroom behavioural engagement was a potential mechanism of change that explained how teacher-student relationships affect student school satisfaction. SAMPLE: We used an archival dataset with a sample of seventh graders (ages 11-14, Mage  = 12.7 year) in a middle school in the Southeastern United States. METHODS: Adolescents completed self-report surveys across three waves over the course of 18 months. RESULTS: Longitudinal structural equation modelling analyses revealed that teacher-student relationships were positively associated with positive classroom engagement behaviours and school satisfaction, respectively, at each time, and positive classroom behaviours at Time 2 fully mediated the longitudinal association between teacher-student relationships (Time 1) and school satisfaction (Time 3). CONCLUSIONS: Taken together, results suggested that fostering positive teacher-student relationships to increase students' positive classroom behaviours could be an effective pathway to promote students' satisfaction with school. The applications of the results in educators' and psychologists' work, such as consultation and trainings with teachers, are discussed.

Poisson Diagnostic Classification Models: A Framework and an Exploratory Example.

Assessments with a large amount of small, similar, or often repetitive tasks are being used in educational, neurocognitive, and psychological contexts. For example, respondents are asked to recognize numbers or letters from a large pool of those and the number of correct answers is a count variable. In 1960, George Rasch developed the Rasch Poisson counts model (RPCM) to handle that type of assessment. This article extends the RPCM into the world of diagnostic classification models (DCMs) where a Poisson distribution is applied to traditional DCMs. A framework of Poisson DCMs is proposed and demonstrated through an operational dataset. This study aims to be exploratory with recommendations for future research given in the end.

Examining the validity and measurement invariance across gender and race of the Implicit Thoughts, Emotion, and Behavior Questionnaire.

Implicit theory has been relatively well-studied in the areas of intelligence and personality but remains less investigated in mental health. This article aims to analyze the psychometric properties of the Implicit Thoughts, Emotion, and Behavior Questionnaire (ITEB-Q; Schleider & Weisz, 2016a). We tested its factorial validity, measurement invariance across gender and two racial groups, as well as criterion validity in a large, diverse sample of adolescents. The sample consists of 817 adolescents aged between 14 and 19 and in 9th-12th grade from a high school in a Southern city in the United States. The confirmatory factorial analysis results corroborate the three-factor structures (thoughts, emotion, behavior) of the ITEB-Q. Multigroup analyses established the scalar invariance across gender as well as across Black and White adolescents. Further, latent mean comparison showed that female adolescents reported lower means on the emotion and thought subscales than male counterparts, and Black adolescents reported higher means across three subscales than White adolescents. The ITEB-Q and its subscales demonstrate good internal reliability and concurrent validity with measures of life satisfaction, resilience, emotional regulation, and emotional problems. Overall, our findings suggest that the ITEB-Q produces valid and reliable scores to assess implicit theory of thoughts, emotion, and behavior among adolescents. (PsycInfo Database Record (c) 2022 APA, all rights reserved).