Detecting Heterogeneity of Intervention Effects in Comparative Judgments.

Comparative measures such as paired comparisons and rankings are frequently used to evaluate health states and quality of life. The present article introduces log-linear Bradley-Terry (LLBT) models to evaluate intervention effectiveness when outcomes are measured as paired comparisons or rankings and presents a combination of the LLBT model and model-based recursive partitioning (MOB) to detect treatment effect heterogeneity. The MOB LLBT approach enables researchers to identify subgroups that differ in the preference order and in the effect an intervention has on choice behavior. Applicability of MOB LLBT models is demonstrated using an artificial data example with known data-generating mechanism and a real-world data example focusing on drug-harm perception among music festival visitors. In the artificial data example, the MOB LLBT model is able to adequately recover the "true" (population) model. In the real-world data example, the standard LLBT model confirms the existence of a situational willingness among festival visitors to trivialize drug harm when peer consumption behavior is made cognitively accessible. In addition, MOB LLBT results suggest that this trivialization effect is highly context-dependent and most pronounced for participants with low-to-moderate alcohol intoxication who also proactively contacted a substance counselor at the festival venue. Both data examples suggest that MOB LLBT models allow for more nuanced statements about the effectiveness of interventions. We provide R code examples to implement MOB LLBT models for paired comparisons, rankings, and rating (Likert-type) data.

Advanced Categorical Data Analysis in Prevention Science.

A variety of health and social problems are routinely measured in the form of categorical outcome data (such as presence/absence of a problem behavior or stages of disease progression). Therefore, proper quantitative analysis of categorical data lies at the heart of the empirical work conducted in prevention science. Categorical data analysis constitutes a broad dynamic field of methods research and data analysts in prevention science can benefit from incorporating recent advances and developments in the statistical evaluation of categorical outcomes in their methodological repertoire. The present Special Issue, Advanced Categorical Data Analysis in Prevention Science, highlights recent methods developments and illustrates their application in the context of prevention science. Contributions of the Special Issue cover a wide variety of areas ranging from statistical models for binary as well as multi-categorical data, advances in the statistical evaluation of moderation and mediation effects for categorical data, developments in model evaluation and measurement, as well as methods that integrate variable- and person-oriented categorical data analysis. The articles of this Special issue make methodological advances in these areas accessible to the audience of prevention scientists to maintain rigorous statistical practice and decision making. The current paper provides background and rationale for this Special Issue, an overview of the articles, and a brief discussion of some potential future directions for prevention research involving categorical data analysis.

Local Effects of Intervention: a Configural Analysis.

In standard statistical data analysis, the effects of intervention or prevention efforts are evaluated in terms of variable relations. Results from application of regression-type methods suggest whether, overall, intervention is successful. In this article, we propose using configural frequency analysis (CFA) either in tandem with regression-type methods or by itself. CFA allows one to adopt a person-oriented perspective in which individuals are targeted that can be characterized by particular profiles. The questions asked in CFA concern these individuals instead of variables. In prevention research, one can ask whether, for particular profiles, the preventive measures are successful. In three real-world data examples, CFA is applied and compared to standard log-linear modeling. Examples consider non-randomized (observational) and randomized intervention settings. The results of these analyses suggest that person-oriented CFA and standard variable-oriented methods of analysis respond to different questions. We show that integrating person- and variable-oriented perspectives can help researchers obtain a fuller picture of intervention effectiveness. Extensions of the CFA approach are discussed.

Accounting for Heteroskedasticity Resulting from Between-Group Differences in Multilevel Models.

Homogeneity of variance (HOV) is a well-known but often untested assumption in the context of multilevel models (MLMs). However, depending on how large the violation is, how different group sizes are, and the variance pairing, standard errors can be over or underestimated even when using MLMs, resulting in questionable inferential tests. We evaluate several tests (e.g., the H statistic, Breusch Pagan, Levene's test) that can be used with MLMs to assess violations of HOV. Although the traditional robust standard errors used with MLMs require at least 50 clusters to be effective, we assess a robust standard error adjustment (i.e., the CR2 estimator) that can be used even with a few clusters. Findings are assessed using a Monte Carlo simulation and are further illustrated using an applied example. We show that explicitly modeling the heterogenous variance structures or using the CR2 estimator are both effective at ameliorating the issues associated with the fixed effects of the regression model related to violations of HOV resulting from between-group differences.

Covariate selection in causal learning under non-Gaussianity.

Understanding causal mechanisms is a central goal in the behavioral, developmental, and social sciences. When estimating and probing causal effects using observational data, covariate adjustment is a crucial element to remove dependencies between focal predictors and the error term. Covariate selection, however, constitutes a challenging task because availability alone is not an adequate criterion to decide whether a covariate should be included in the statistical model. The present study introduces a non-Gaussian method for covariate selection and provides a forward selection algorithm for linear models (i.e., non-Gaussian forward selection; nGFS) to select appropriate covariates from a set of potential control variables to avoid inconsistent and biased estimators of the causal effect of interest. Further, we demonstrate that the forward selection algorithm has properties compatible with principles of direction of dependence, i.e., probing whether the causal target model is correctly specified with respect to the causal direction of effects. Results of a Monte Carlo simulation study suggest that the selection algorithm performs well, in particular when sample sizes are large (i.e., n ≥ 250) and data strongly deviate from Gaussianity (e.g., distributions with skewness beyond 1.5). An empirical example is given for illustrative purposes.

Parsimonious asymmetric item response theory modeling with the complementary log-log link.

Traditional item response theory (IRT) models assume a symmetric error distribution and rely on symmetric (logit or probit) link functions to model the response probabilities. As an alternative, we investigated the one-parameter complementary log-log model (CLLM), which is founded on an asymmetric error distribution and results in an asymmetric item response function with important psychometric properties. In a series of simulation studies, we demonstrate that the CLLM (a) is estimable in small sample sizes, (b) facilitates item-weighted scoring, and (c) accounts for the effect of guessing, despite the presence of a single parameter. We then provide further evidence for these claims by applying the CLLM to empirical data. Finally, we discuss how this work contributes to the growing psychometric literature on model complexity.

Parsimonious item response theory modeling with the negative log-log link: The role of inflection point shift.

In item response theory (IRT) modeling, the magnitude of the lower and upper asymptote parameters determines the degree to which the inflection point shifts above or below P = 0.50. The current study examines the one-parameter negative log-log model (NLLM), which is characterized by a downward shift in the inflection point, among other distinctive psychometric properties. After detailing the statistical foundations of the NLLM, we present a series of simulation studies to establish item and person parameter estimation accuracy and to demonstrate that this parsimonious model addresses the "slipping" effect (i.e., unexpectedly incorrect answers) via an inflection point < 0.50 rather than through computationally difficult estimation of the upper asymptote. We then provide further support for these simulation results through empirical data analysis. Finally, we discuss how the NLLM contributes to recent methodological literature on the utility of asymmetric IRT models.

Detecting heterogeneity in the causal direction of dependence: A model-based recursive partitioning approach.

Methods of causal discovery and direction of dependence to evaluate causal properties of variable relations have experienced rapid development. The majority of causal discovery methods, however, relies on the assumption of causal effect homogeneity, that is, the identified causal structure is expected to hold for the entire population. Because causal mechanisms can vary across subpopulations, we propose combining methods of model-based recursive partitioning and non-Gaussian causal discovery to identify such subpopulations. The resulting algorithm can discover subpopulations with potentially varying magnitude and causal direction of effects under mild parameter inequality assumptions. Feasibility conditions are described and results from synthetic data experiments are presented suggesting that large effects and large sample sizes are beneficial for detecting causally competing subgroups with acceptable statistical performance. In a real-world data example, the extraction of meaningful subgroups that differ in the causal mechanism underlying the development of numerical cognition is illustrated. Potential extensions and recommendations for best practice applications are discussed.

Configural frequency trees.

Although variable-oriented analyses are dominant in developmental psychopathology, researchers have championed a person-oriented approach that focuses on the individual as a totality. This view has methodological implications and various person-oriented methods have been developed to test person-oriented hypotheses. Configural frequency analysis (CFA) has been identified as a prime method for a person-oriented analysis of categorical data. CFA searches for configurations in cross-classifications and asks whether the number of observed cases is larger (CFA type) or smaller (CFA antitype) than expected under a probability model. The present study introduces a combination of CFA and model-based recursive partitioning (MOB) to test for type/antitype heterogeneity in the population. MOB CFA is well suited to detect complex moderation processes and can distinguish between subpopulation and population types/antitypes. Model specifications are discussed for first-order CFA and prediction CFA. Results from two simulation studies suggest that MOB CFA is able to detect moderation processes with high accuracy. Two empirical examples are given from school mental health research for illustrative purposes. The first example evaluates heterogeneity in student behavior types/antitypes, the second example focuses on the effect of a teacher classroom management intervention on student behavior. An implementation of the approach is provided in R.

Effect of Daily Teacher Feedback on Subsequent Motivation and Mental Health Outcomes in Fifth Grade Students: a Person-Centered Analysis.

Prevention scientists recognize that implementing effective prevention practices and programs responsive to the needs of individuals but based solely upon the findings from variable-centered methods presents several limitations due to numerous risk factors, pathways, and unobserved influences. One such understudied influence that is masked by variable-centered methods, motivation, is a person-level characteristic that influences treatment outcomes. The purpose of this paper is to demonstrate the use of an alternative person-centered approach, group iterative multiple model estimation (GIMME), to model change over time that focuses on the interdependence of daily student motivation levels and teacher feedback and their relations to student outcomes over time. Specifically, we used GIMME to model person level responses to negative teacher feedback regarding students' next day motivational ratings using data from 58 5th grade students participating in a study of the impact of the self-monitoring and regulation training strategy (SMARTS). Results identified a set of SMARTS students whose daily readiness aligned with high rates of self and teacher agreement regarding ongoing performance ratings. However, results identified a group of students whose daily motivation and readiness for change was adversely impacted by negative teacher feedback the day before. For these students, they were more likely than their peers to experience high levels of depression and internalization scores. Motivationally oriented practice suggestions for providing feedback to students who may be sensitive to this type of feedback and research implications of these findings are discussed.

Sensitivity Analysis and Extensions of Testing the Causal Direction of Dependence: A Rejoinder to Thoemmes (2019).

A commentary by Thoemmes on Wiedermann and Sebastian's introductory article on Direction Dependence Analysis (DDA) is responded to in the interest of elaborating and extending direction dependence principles to evaluate causal effect directionality. Considering Thoemmes' observation that some DDA principles are already well-established in machine learning, we argue that several other connections between DDA and research lines in theoretical statistics, econometrics, and quantitative psychology exist, suggesting that DDA is best conceptualized as a framework that summarizes and extends existing knowledge on properties of linear models under non-normality. Further, Thoemmes articulates concerns about assumptions of error distributions used in our main article and presents an artificial data example in which some DDA tests have suboptimal statistical power. We present extensions of DDA to entirely relax distributional assumptions about errors and describe two sensitivity analysis approaches to critically evaluate DDA results. Both sensitivity approaches are illustrated using Thoemmes' artificial data example. Incorporating DDA sensitivity results yields correct causal conclusions. Thus, overall, we stay with our initial conclusion that the use of higher moments in causal inference constitutes an exciting open research area.

Direction Dependence Analysis in the Presence of Confounders: Applications to Linear Mediation Models Using Observational Data.

Statistical methods to identify mis-specifications of linear regression models with respect to the direction of dependence (i.e. whether x→y or y→x better approximates the data-generating mechanism) have received considerable attention. Direction dependence analysis (DDA) constitutes such a statistical tool and makes use of higher-moment information of variables to derive statements concerning directional model mis-specifications in observational data. Previous studies on direction of dependence mainly focused on statistical inference and guidelines for the selection from the two directionally competing candidate models (x→y versus y→x) while assuming the absence of unobserved common causes. The present study describes properties of DDA when confounders are present and extends existing DDA methodology by incorporating the confounder model as a possible explanation. We show that all three explanatory models can be uniquely identified under standard DDA assumptions. Further, we discuss the proposed approach in the context of testing competing mediation models and evaluate an organizational model proposing a mediational relation between school leadership and student achievement via school safety using observational data from an urban school district. Overall, DDA provides strong empirical support that school safety has indeed a causal effect on student achievement but suggests that important confounders are present in the school leadership-safety relation.

Conditional Direction Dependence Analysis: Evaluating the Causal Direction of Effects in Linear Models with Interaction Terms.

Direction dependence analysis (DDA) makes use of higher than second moment information of variables (x and y) to detect potential confounding and to probe the causal direction of linear variable relations (i.e., whether x → y or y → x better approximates the underlying causal mechanism). The "true" predictor is assumed to be a continuous nonnormal exogenous variable. Existing methods compatible with DDA, however, are of limited use when the relation of a focal predictor and an outcome is affected by a moderator. This study presents a conditional direction dependence analysis (CDDA) framework which enables researchers to evaluate the causal direction of conditional regression effects. Monte-Carlo simulations were used to evaluate two different moderation scenarios: Study 1 evaluates the performance of CDDA tests when a moderator affects the strength of the causal effect x → y. Study 2 evaluates cases in which the causal direction itself (x → y vs y → x) depends on moderator values. Study 3 evaluates the robustness of DDA tests in the presence of functional model misspecifications. Results suggest that significance tests compatible with CDDA are suitable in both moderation scenarios, i.e., CDDA allows one to discern regions of a moderator in which the causal direction is uniquely identifiable. An empirical example is provided to illustrate the approach.

Confounder detection in linear mediation models: Performance of kernel-based tests of independence.

It is well-known that the identification of direct and indirect effects in mediation analysis requires strong unconfoundedness assumptions. Even when the predictor is under experimental control, unconfoundedness assumptions must be imposed on the mediator-outcome relation in order to guarantee valid indirect-effect identification. Researchers are therefore advised to test for unconfoundedness when estimating mediation effects. Significance tests to evaluate unconfoundedness usually rely on an instrumental variable (IV)-that is, a variable that is nonindependent of the explanatory variable and, at the same time, independent of all exogenous factors that affect the outcome when the explanatory variable is held constant. Because IVs may be hard to come by, the present study shows that confounders of the mediator-outcome relation can be detected without making use of IVs when variables are nonnormal. We show that kernel-based tests of independence are able to detect confounding under nonnormality. Results from a simulation study are presented that suggest that these tests perform well in terms of Type I error protection and statistical power, independent of the distribution or measurement level of the confounder. A real-world data example from the Job Search Intervention Study (JOBS II) illustrates how the presented approach can be used to minimize the risk of obtaining biased indirect-effect estimates. The data requirements and role of unconfoundedness tests as diagnostic tools are discussed. A Monte Carlo-based power analysis tool for sample size planning is also provided.

Testing the Causal Direction of Mediation Effects in Randomized Intervention Studies.

In a recent update of the standards for evidence in research on prevention interventions, the Society of Prevention Research emphasizes the importance of evaluating and testing the causal mechanism through which an intervention is expected to have an effect on an outcome. Mediation analysis is commonly applied to study such causal processes. However, these analytic tools are limited in their potential to fully understand the role of theorized mediators. For example, in a design where the treatment x is randomized and the mediator (m) and the outcome (y) are measured cross-sectionally, the causal direction of the hypothesized mediator-outcome relation is not uniquely identified. That is, both mediation models, x → m → y or x → y → m, may be plausible candidates to describe the underlying intervention theory. As a third explanation, unobserved confounders can still be responsible for the mediator-outcome association. The present study introduces principles of direction dependence which can be used to empirically evaluate these competing explanatory theories. We show that, under certain conditions, third higher moments of variables (i.e., skewness and co-skewness) can be used to uniquely identify the direction of a mediator-outcome relation. Significance procedures compatible with direction dependence are introduced and results of a simulation study are reported that demonstrate the performance of the tests. An empirical example is given for illustrative purposes and a software implementation of the proposed method is provided in SPSS.

Advances in Statistical Methods for Causal Inference in Prevention Science: Introduction to the Special Section.

The board of the Society for Prevention Research noted recently that extant methods for the analysis of causality mechanisms in prevention may still be too rudimentary for detailed and sophisticated analysis of causality hypotheses. This Special Section aims to fill some of the current voids, in particular in the domain of statistical methods of the analysis of causal inference. In the first article, Bray et al. propose a novel methodological approach in which they link propensity score techniques and Latent Class Analysis. In the second article, Kelcey et al. discuss power analysis tools for the study of causal mediation effects in cluster-randomized interventions. Wiedermann et al. present, in the third article, methods of Direction Dependence Analysis for the identification of confounders and for inference concerning the direction of causal effects in mediation models. A more general approach to the identification of causal structures in non-experimental data is presented by Shimizu in the fourth article. This approach is based on linear non-Gaussian acyclic models. Molenaar introduces vector-autoregressive methods for the optimal representation of Granger causality in time-dependent data. The Special Section concludes with a commentary by Musci and Stuart. In this commentary, the contributions of the articles in the Special Section are highlighted from the perspective of the experimental causal research tradition.

Direction dependence analysis: A framework to test the direction of effects in linear models with an implementation in SPSS.

In nonexperimental data, at least three possible explanations exist for the association of two variables x and y: (1) x is the cause of y, (2) y is the cause of x, or (3) an unmeasured confounder is present. Statistical tests that identify which of the three explanatory models fits best would be a useful adjunct to the use of theory alone. The present article introduces one such statistical method, direction dependence analysis (DDA), which assesses the relative plausibility of the three explanatory models on the basis of higher-moment information about the variables (i.e., skewness and kurtosis). DDA involves the evaluation of three properties of the data: (1) the observed distributions of the variables, (2) the residual distributions of the competing models, and (3) the independence properties of the predictors and residuals of the competing models. When the observed variables are nonnormally distributed, we show that DDA components can be used to uniquely identify each explanatory model. Statistical inference methods for model selection are presented, and macros to implement DDA in SPSS are provided. An empirical example is given to illustrate the approach. Conceptual and empirical considerations are discussed for best-practice applications in psychological data, and sample size recommendations based on previous simulation studies are provided.

Heteroscedasticity as a Basis of Direction Dependence in Reversible Linear Regression Models.

Heteroscedasticity is a well-known issue in linear regression modeling. When heteroscedasticity is observed, researchers are advised to remedy possible model misspecification of the explanatory part of the model (e.g., considering alternative functional forms and/or omitted variables). The present contribution discusses another source of heteroscedasticity in observational data: Directional model misspecifications in the case of nonnormal variables. Directional misspecification refers to situations where alternative models are equally likely to explain the data-generating process (e.g., x → y versus y → x). It is shown that the homoscedasticity assumption is likely to be violated in models that erroneously treat true nonnormal predictors as response variables. Recently, Direction Dependence Analysis (DDA) has been proposed as a framework to empirically evaluate the direction of effects in linear models. The present study links the phenomenon of heteroscedasticity with DDA and describes visual diagnostics and nine homoscedasticity tests that can be used to make decisions concerning the direction of effects in linear models. Results of a Monte Carlo simulation that demonstrate the adequacy of the approach are presented. An empirical example is provided, and applicability of the methodology in cases of violated assumptions is discussed.

Direction of Effects in Multiple Linear Regression Models.

Previous studies analyzed asymmetric properties of the Pearson correlation coefficient using higher than second order moments. These asymmetric properties can be used to determine the direction of dependence in a linear regression setting (i.e., establish which of two variables is more likely to be on the outcome side) within the framework of cross-sectional observational data. Extant approaches are restricted to the bivariate regression case. The present contribution extends the direction of dependence methodology to a multiple linear regression setting by analyzing distributional properties of residuals of competing multiple regression models. It is shown that, under certain conditions, the third central moments of estimated regression residuals can be used to decide upon direction of effects. In addition, three different approaches for statistical inference are discussed: a combined D'Agostino normality test, a skewness difference test, and a bootstrap difference test. Type I error and power of the procedures are assessed using Monte Carlo simulations, and an empirical example is provided for illustrative purposes. In the discussion, issues concerning the quality of psychological data, possible extensions of the proposed methods to the fourth central moment of regression residuals, and potential applications are addressed.