Signal-to-Noise Ratio in Estimating and Testing the Mediation Effect: Structural Equation Modeling versus Path Analysis with Weighted Composites.

Mediation analysis plays an important role in understanding causal processes in social and behavioral sciences. While path analysis with composite scores was criticized to yield biased parameter estimates when variables contain measurement errors, recent literature has pointed out that the population values of parameters of latent-variable models are determined by the subjectively assigned scales of the latent variables. Thus, conclusions in existing studies comparing structural equation modeling (SEM) and path analysis with weighted composites (PAWC) on the accuracy and precision of the estimates of the indirect effect in mediation analysis have little validity. Instead of comparing the size on estimates of the indirect effect between SEM and PAWC, this article compares parameter estimates by signal-to-noise ratio (SNR), which does not depend on the metrics of the latent variables once the anchors of the latent variables are determined. Results show that PAWC yields greater SNR than SEM in estimating and testing the indirect effect even when measurement errors exist. In particular, path analysis via factor scores almost always yields greater SNRs than SEM. Mediation analysis with equally weighted composites (EWCs) also more likely yields greater SNRs than SEM. Consequently, PAWC is statistically more efficient and more powerful than SEM in conducting mediation analysis in empirical research. The article also further studies conditions that cause SEM to have smaller SNRs, and results indicate that the advantage of PAWC becomes more obvious when there is a strong relationship between the predictor and the mediator, whereas the size of the prediction error in the mediator adversely affects the performance of the PAWC methodology. Results of a real-data example also support the conclusions.

Which method delivers greater signal-to-noise ratio: Structural equation modelling or regression analysis with weighted composites?

Observational data typically contain measurement errors. Covariance-based structural equation modelling (CB-SEM) is capable of modelling measurement errors and yields consistent parameter estimates. In contrast, methods of regression analysis using weighted composites as well as a partial least squares approach to SEM facilitate the prediction and diagnosis of individuals/participants. But regression analysis with weighted composites has been known to yield attenuated regression coefficients when predictors contain errors. Contrary to the common belief that CB-SEM is the preferred method for the analysis of observational data, this article shows that regression analysis via weighted composites yields parameter estimates with much smaller standard errors, and thus corresponds to greater values of the signal-to-noise ratio (SNR). In particular, the SNR for the regression coefficient via the least squares (LS) method with equally weighted composites is mathematically greater than that by CB-SEM if the items for each factor are parallel, even when the SEM model is correctly specified and estimated by an efficient method. Analytical, numerical and empirical results also show that LS regression using weighted composites performs as well as or better than the normal maximum likelihood method for CB-SEM under many conditions even when the population distribution is multivariate normal. Results also show that the LS regression coefficients become more efficient when considering the sampling errors in the weights of composites than those that are conditional on weights.

A systematic framework for defining R-squared measures in mediation analysis.

R-squared measures of explained variance are easy to understand, naturally interpretable, and widely used by substantive researchers. In mediation analysis, however, despite recent advances in measures of mediation effect, few effect sizes have good statistical properties. Also, most of these measures are only available for the simplest three-variable mediation model, especially for R²-type measures. By decomposing the mediator into two parts (i.e., the part related to the predictor and the part unrelated to the predictor), this article proposes a systematic framework to develop new effect-size measures of explained variance in mediation analysis. The framework can be easily extended to more complex mediation models and provides more delicate R² measures for empirical researchers. A Monte Carlo simulation study is conducted to examine the statistical properties of the proposed R² effect-size measure. Results show that the new R2 measure performs well in approximating the true value of the explained variance of the mediation effect. The use of the proposed measure is illustrated with empirical examples together with program code for its implementation. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

A Realistic Evaluation of Methods for Handling Missing Data When There is a Mixture of MCAR, MAR, and MNAR Mechanisms in the Same Dataset.

The impact of missing data on statistical inference varies depending on several factors such as the proportion of missingness, missing-data mechanism, and method employed to handle missing values. While these topics have been extensively studied, most recommendations have been made assuming that all missing values are from the same missing-data mechanism. In reality, it is very likely that a mixture of missing-data mechanisms is responsible for missing values in a dataset and even within the same pattern of missingness. Although a mixture of missing-data mechanisms and causes within a dataset is a likely scenario, the performance of popular missing-data methods under these circumstances is unknown. This study provides a realistic evaluation of methods for handling missing data in this setting using Monte Carlo simulation in the context of regression. This study also seeks to identify acceptable proportions of missing values that violate the missing-data mechanism assumed by the method used to handle missing values. Results indicate that multiple imputation (MI) performs better than other principled or ad-hoc methods. Different missing-data methods are also compared via the analysis of a real dataset in which mixtures of missingness mechanisms are created. Recommendations are provided for the use of different methods in practice.

Which method is more powerful in testing the relationship of theoretical constructs? A meta comparison of structural equation modeling and path analysis with weighted composites.

Structural equation modeling (SEM) has been deemed as a proper method when variables contain measurement errors. In contrast, path analysis with composite scores is preferred for prediction and diagnosis of individuals. While path analysis with composite scores has been criticized for yielding biased parameter estimates, recent literature pointed out that the population values of parameters in a latent-variable model depend on artificially assigned scales. Consequently, bias in parameter estimates is not a well-grounded concept for models involving latent constructs. This article compares path analysis with composite scores against SEM with respect to effect size and statistical power in testing the significance of the path coefficients, via the z- or t-statistics. The data come from many sources with various models that are substantively determined. Results show that SEM is not as powerful as path analysis even with equally weighted composites. However, path analysis with Bartlett-factor scores and the partial least-squares approach to SEM perform the best with respect to effect size and power.

Smoothed Quantiles for χ2 Type Test Statistics with Applications.

Chi-square type test statistics are widely used in assessing the goodness-of-fit of a theoretical model. The exact distributions of such statistics can be quite different from the nominal chi-square distribution due to violation of conditions encountered with real data. In such instances, the bootstrap or Monte Carlo methodology might be used to approximate the distribution of the statistic. However, the sample quantile may be a poor estimate of the population counterpart when either the sample size is small or the number of different values of the replicated statistic is limited. Using statistical learning, this article develops a method that yields more accurate quantiles for chi-square type test statistics. Formulas for smoothing the quantiles of chi-square type statistics are obtained. Combined with the bootstrap methodology, the smoothed quantiles are further used to conduct equivalence testing in mean and covariance structure analysis. Two real data examples illustrate the applications of the developed formulas in quantifying the size of model misspecification under equivalence testing. The idea developed in the article can also be used to develop formulas for smoothing the quantiles of other types of test statistics or parameter estimates.

Two-level moderated mediation models with single-level data and new measures of effect sizes.

This article proposes a two-level moderated mediation (2moME) model with single level data, and develops measures to quantify the moderated mediation (moME) effect sizes for both the conventional moME model and the 2moME model. A Bayesian approach is developed to estimate and test moME effects and the corresponding effect sizes (ES). Monte Carlo results indicate that (1) the 2moME model yields more accurate estimates of the parameters than the conventional moME model; (2) the 95% credibility interval following the 2moME model covers the moME effects and the ESs more accurately than that following the conventional moME model; and (3) statistical tests for the existence of the moME effects with the 2moME model are more reliable in controlling type I errors than those with the conventional moME model, especially under heteroscedasticity conditions. In addition, the developed measures of ES are more interpretable, and directly answer the questions regarding the extent to which a moderator can account for the change of the mediation effect between the predictor and the outcome variable through the mediator variable. An empirical example illustrates the application of the 2moME model and the ES measures.

Subtypes of the missing not at random missing data mechanism.

issing values that are missing not at random (MNAR) can result from a variety of missingness processes. However, two fundamental subtypes of MNAR values can be obtained from the definition of the MNAR mechanism itself. The distinction between them deserves consideration because they have characteristic differences in how they distort relationships in the data. This has implications for the validity of statistical results and generalizability of methodological findings that are based on data (empirical or generated) with MNAR values. However, these MNAR subtypes have largely gone unnoticed by the literature. As few studies have considered both subtypes, their relevance to methodological and substantive research has been overlooked. This article systematically introduces the two MNAR subtypes and gives them descriptive names. A case study demonstrates they are mechanically distinct from each other and from other missing-data mechanisms. Applied examples are given to help researchers conceptually identify MNAR subtypes in real data. Methods are provided to generate missing values from both subtypes in simulation studies. Simulation studies for regression and growth curve modeling contexts show MNAR subtypes consistently differ in the severity of their impact on statistical inference. This behavior is examined in light of how relationships in the data become characteristically distorted. The contents of this article are intended to provide a foundation and tools for organized consideration of MNAR subtypes. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Differential Item Functioning Analysis Without A Priori Information on Anchor Items: QQ Plots and Graphical Test.

Differential item functioning (DIF) analysis is an important step in establishing the validity of measurements. Most traditional methods for DIF analysis use an item-by-item strategy via anchor items that are assumed DIF-free. If anchor items are flawed, these methods will yield misleading results due to biased scales. In this article, based on the fact that the item's relative change of difficulty difference (RCD) does not depend on the mean ability of individual groups, a new DIF detection method (RCD-DIF) is proposed by comparing the observed differences against those with simulated data that are known DIF-free. The RCD-DIF method consists of a D-QQ (quantile quantile) plot that permits the identification of internal references points (similar to anchor items), a RCD-QQ plot that facilitates visual examination of DIF, and a RCD graphical test that synchronizes DIF analysis at the test level with that at the item level via confidence intervals on individual items. The RCD procedure visually reveals the overall pattern of DIF in the test and the size of DIF for each item and is expected to work properly even when the majority of the items possess DIF and the DIF pattern is unbalanced. Results of two simulation studies indicate that the RCD graphical test has Type I error rate comparable to those of existing methods but with greater power.

An overview of applied robust methods.

Data in social sciences are typically non-normally distributed and characterized by heavy tails. However, most widely used methods in social sciences are still based on the analyses of sample means and sample covariances. While these conventional methods continue to be used to address new substantive issues, conclusions reached can be inaccurate or misleading. Although there is no 'best method' in practice, robust methods that consider the distribution of the data can perform substantially better than the conventional methods. This article gives an overview of robust procedures, emphasizing a few that have been repeatedly shown to work well for models that are widely used in social and behavioural sciences. Real data examples show how to use the robust methods for latent variable models and for moderated mediation analysis when a regression model contains categorical covariates and product terms. Results and logical analyses indicate that robust methods yield more efficient parameter estimates, more reliable model evaluation, more reliable model/data diagnostics, and more trustworthy conclusions when conducting replication studies. R and SAS programs are provided for routine applications of the recommended robust method.

New measures of effect size in moderation analysis.

Measures of explained variance, ΔR2 and f,2 are routinely used to evaluate the size of moderation effects. However, they suffer from several drawbacks: (a) Not all the variance components of the outcome variable Y are related to the effect of moderation, and so an effect size with the total variance of Y as the denominator cannot accurately characterize the moderation effect; (b) moderation and interaction are conflated; and (c) the assumption of homoscedasticity might be violated when moderation exists. By arguing that measures for the size of moderation effect should be based on the variance of the outcome Y via the predictor variable X (i.e., X→Y), this article develops a new conceptualization of moderation effects that leads to 2 ways of defining new measures of moderation effects size. One is by using regression models that include the moderator, the predictor, and the product term sequentially. The other is based on a variance decomposition of the outcome variable Y. These new effect size measures effectively differentiate the role of the predictor variable from that of the moderator variable. Two empirical examples are provided to contrast the new measures against the traditional ΔR2 and f2, and to illustrate the applications of the new ones. R code is also provided for researchers to compute the new effect size measures. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

A Two-level Moderated Latent Variable Model with Single Level Data.

With single-level data, Yuan, Cheng and Maxwell developed a two-level regression model for more accurate moderation analysis. This article extends the two-level regression model to a two-level moderated latent variable (2MLV) model, and uses a Bayesian approach to estimate and test the moderation effects. Monte Carlo results indicate that: 1) the new method yields more accurate estimate of the interaction effect than those via the product-indicator (PI) approach and latent variable interaction (LVI) with single-level model, both are also estimated via Bayesian method; 2) the coverage rates of the credibility interval following the 2MLV model are closer to the nominal 95% than those following the other methods; 3) the test for the existence of the moderation effect is more reliable in controlling Type I errors than both PI and LVI, especially under heteroscedasticity conditions. Moreover, a more interpretable measure of effect size is developed based on the 2MLV model, which directly answers the question as to what extent a moderator can account for the change of the coefficient between the predictor and the outcome variable. A real data example illustrates the application of the new method.

What Causes the Mean Bias of the Likelihood Ratio Statistic with Many Variables?

Survey data often contain many variables. Structural equation modeling (SEM) is commonly used in analyzing such data. However, conventional SEM methods are not crafted to handle data with a large number of variables (p). A large p can cause Tml, the most widely used likelihood ratio statistic, to depart drastically from the assumed chi-square distribution even with normally distributed data and a relatively large sample size N. A key element affecting this behavior of Tml is its mean bias. The focus of this article is to determine the cause of the bias. To this end, empirical means of Tml via Monte Carlo simulation are used to obtain the empirical bias. The most effective predictors of the mean bias are subsequently identified and their predictive utility examined. The results are further used to predict type I errors of Tml. The article also illustrates how to use the obtained results to determine the required sample size for Tml to behave reasonably well. A real data example is presented to show the effect of the mean bias on model inference as well as how to correct the bias in practice.

Asymptotic bias of normal-distribution-based maximum likelihood estimates of moderation effects with data missing at random.

Moderation analysis is useful for addressing interesting research questions in social sciences and behavioural research. In practice, moderated multiple regression (MMR) models have been most widely used. However, missing data pose a challenge, mainly because the interaction term is a product of two or more variables and thus is a non-linear function of the involved variables. Normal-distribution-based maximum likelihood (NML) has been proposed and applied for estimating MMR models with incomplete data. When data are missing completely at random, moderation effect estimates are consistent. However, simulation results have found that when data in the predictor are missing at random (MAR), NML can yield inaccurate estimates of moderation effects when the moderation effects are non-null. Simulation studies are subject to the limitation of confounding systematic bias with sampling errors. Thus, the purpose of this paper is to analytically derive asymptotic bias of NML estimates of moderation effects with MAR data. Results show that when the moderation effect is zero, there is no asymptotic bias in moderation effect estimates with either normal or non-normal data. When the moderation effect is non-zero, however, asymptotic bias may exist and is determined by factors such as the moderation effect size, missing-data proportion, and type of missingness dependence. Our analytical results suggest that researchers should apply NML to MMR models with caution when missing data exist. Suggestions are given regarding moderation analysis with missing data.

Fit indices for mean structures with growth curve models.

Motivated by the need to effectively evaluate the quality of the mean structure in growth curve modeling (GCM), this article proposes to separately evaluate the goodness of fit of the mean structure from that of the covariance structure. Several fit indices are defined, and rationales are discussed. Particular considerations are given for polynomial and piecewise polynomial models because fit indices for them are valid regardless of the underlying population distribution of the data. Examples indicate that the newly defined fit indices remove the confounding issues with indices jointly evaluating mean and covariance structure models and provide much more reliable evaluation of the mean structure in GCM. Examples also show that pseudo R-squares and concordance correlations are unable to reflect the goodness of mean structures in GCM. Proper use of the fit indices for the purpose of model diagnostics is discussed. A window-based program, WebSEM, is also introduced for easily computing these fit indices by applied researchers. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Missing Data Mechanisms and Homogeneity of Means and Variances-Covariances.

Unless data are missing completely at random (MCAR), proper methodology is crucial for the analysis of incomplete data. Consequently, methods for effectively testing the MCAR mechanism become important, and procedures were developed via testing the homogeneity of means and variances-covariances across the observed patterns (e.g., Kim & Bentler in Psychometrika 67:609-624, 2002; Little in J Am Stat Assoc 83:1198-1202, 1988). The current article shows that the population counterparts of the sample means and covariances of a given pattern of the observed data depend on the underlying structure that generates the data, and the normal-distribution-based maximum likelihood estimates for different patterns of the observed sample can converge to the same values even when data are missing at random or missing not at random, although the values may not equal those of the underlying population distribution. The results imply that statistics developed for testing the homogeneity of means and covariances cannot be safely used for testing the MCAR mechanism even when the population distribution is multivariate normal.

Advances in Measurement Invariance and Mean Comparison of Latent Variables: Equivalence Testing and A Projection-Based Approach.

Measurement invariance (MI) entails that measurements in different groups are comparable, and is a logical prerequisite when studying difference or change across groups. MI is commonly evaluated using multi-group structural equation modeling through a sequence of chi-square and chi-square-difference tests. However, under the conventional null hypothesis testing (NHT) one can never be confident enough to claim MI even when all test statistics are not significant. Equivalence testing (ET) has been recently proposed to replace NHT for studying MI. ET informs researchers a size of possible misspecification and allows them to claim that measurements are practically equivalent across groups if the size of misspecification is smaller than a tolerable value. Another recent advancement in studying MI is a projection-based method under which testing the cross-group equality of means of latent traits does not require the intercepts equal across groups. The purpose of this article is to introduce the key ideas of the two advancements in MI and present a newly developed R package equaltestMI for researchers to easily apply the two methods. A real data example is provided to illustrate the use of the package. It is advocated that researchers should always consider using the two methods whenever MI needs to be examined.

Empirically Corrected Rescaled Statistics for SEM with Small N and Large p.

Survey data often contain many variables. Structural equation modeling (SEM) is commonly used in analyzing such data. With typical nonnormally distributed data in practice, a rescaled statistic Trml proposed by Satorra and Bentler was recommended in the literature of SEM. However, Trml has been shown to be problematic when the sample size N is small and/or the number of variables p is large. There does not exist a reliable test statistic for SEM with small N or large p, especially with nonnormally distributed data. Following the principle of Bartlett correction, this article develops empirical corrections to Trml so that the mean of the empirically corrected statistics approximately equals the degrees of freedom of the nominal chi-square distribution. Results show that empirically corrected statistics control type I errors reasonably well even when N is smaller than 2p, where Trml may reject the correct model 100% even for normally distributed data. The application of the empirically corrected statistics is illustrated via a real data example.

More efficient parameter estimates for factor analysis of ordinal variables by ridge generalized least squares.

Data in psychology are often collected using Likert-type scales, and it has been shown that factor analysis of Likert-type data is better performed on the polychoric correlation matrix than on the product-moment covariance matrix, especially when the distributions of the observed variables are skewed. In theory, factor analysis of the polychoric correlation matrix is best conducted using generalized least squares with an asymptotically correct weight matrix (AGLS). However, simulation studies showed that both least squares (LS) and diagonally weighted least squares (DWLS) perform better than AGLS, and thus LS or DWLS is routinely used in practice. In either LS or DWLS, the associations among the polychoric correlation coefficients are completely ignored. To mend such a gap between statistical theory and empirical work, this paper proposes new methods, called ridge GLS, for factor analysis of ordinal data. Monte Carlo results show that, for a wide range of sample sizes, ridge GLS methods yield uniformly more accurate parameter estimates than existing methods (LS, DWLS, AGLS). A real-data example indicates that estimates by ridge GLS are 9-20% more efficient than those by existing methods. Rescaled and adjusted test statistics as well as sandwich-type standard errors following the ridge GLS methods also perform reasonably well.