Identifying and Estimating Causal Moderation for Treated and Targeted Subgroups.

Extant literature on moderation effects narrowly focuses on the average moderated treatment effect across the entire sample (AMTE). Missing is the average moderated treatment effect on the treated (AMTT) and other targeted subgroups (AMTS). Much like the average treatment effect on the treated (ATT) for main effects, the AMTS changes the target of inferences from the entire sample to targeted subgroups. Relative to the AMTE, the AMTS is identified under weaker assumptions and often captures more policy-relevant effects. We present a theoretical framework that introduces the AMTS under the potential outcomes framework and delineates the assumptions for causal identification. We then propose a generalized propensity score method as a tool to estimate the AMTS using weights derived with Bayes Theorem. We illustrate the results and differences among the estimands using data from the Early Childhood Longitudinal Study. We conclude with suggestions for future research.

Gender, Racial, and Socioeconomic Disparities on Social and Behavioral Skills for K-8 Students With and Without Interventions: An Integrative Data Analysis of Eight Cluster Randomized Trials.

Despite decades of concern about disparities in educational outcomes for low SES students and students of color, there has been limited rigorous study of programmatic approaches for reducing these disparities in elementary or middle schools. We conducted integrative data analysis (IDA) of the combined data from eight Institute of Education Sciences funded cluster randomized trials to address the research gaps on social and behavioral outcome disparities. The final analytic sample includes 90,880 students in varying grade levels from kindergarten to grade 8 in 387 schools in 4 states (Maryland, Missouri, Virginia, and Texas). Two-level hierarchical linear modeling was used for multilevel moderation analysis. This study provided empirical evidence that there were significant gender, racial, and socioeconomic disparities on social and behavioral outcome measures for elementary and middle school students, the disparities significantly varied across schools, and the disparities could be reduced by interventions. We discussed our findings, implications for interpreting effect sizes of interventions using disparities as empirical benchmarks, and study limitations. We concluded with suggestions for future research.

Power and Sample Size Determination for Multilevel Mediation in Three-Level Cluster-Randomized Trials.

Mediation analyses supply a principal lens to probe the pathways through which a treatment acts upon an outcome because they can dismantle and test the core components of treatments and test how these components function as a coordinated system or theory of action. Experimental evaluation of mediation effects in addition to total effects has become increasingly common but literature has developed only limited guidance on how to plan mediation studies with multi-tiered hierarchical or clustered structures. In this study, we provide methods for computing the power to detect mediation effects in three-level cluster-randomized designs that examine individual- (level one), intermediate- (level two) or cluster-level (level three) mediators. We assess the methods using a simulation and provide examples of a three-level clinic-randomized study (individuals nested within therapists nested within clinics) probing an individual-, intermediate- or cluster-level mediator using the R package PowerUpR and its Shiny application.

Using Propensity Score Analysis of Survey Data to Estimate Population Average Treatment Effects: A Case Study Comparing Different Methods.

BACKGROUND: Many studies in psychological and educational research aim to estimate population average treatment effects (PATE) using data from large complex survey samples, and many of these studies use propensity score methods. Recent advances have investigated how to incorporate survey weights with propensity score methods. However, to this point, that work had not been well summarized, and it was not clear how much difference the different PATE estimation methods would make empirically. PURPOSE: The purpose of this study is to systematically summarize the appropriate use of survey weights in propensity score analysis of complex survey data and use a case study to empirically compare the PATE estimates using multiple analysis methods that include ordinary least squares regression, weighted least squares regression, and various propensity score applications. METHODS: We first summarize various propensity score methods that handle survey weights. We then demonstrate the performance of various analysis methods using a nationally representative data set, the Early Childhood Longitudinal Study-Kindergarten to estimate the effects of preschool on children's academic achievement. The correspondence of the results was evaluated using multiple criteria. RESULTS AND CONCLUSIONS: It is important for researchers to think carefully about their estimand of interest and use methods appropriate for that estimand. If interest is in drawing inferences to the survey target population, it is important to take the survey weights into account, particularly in the outcome analysis stage for estimating the PATE. The case study shows, however, not much difference among various analysis methods in one applied example.

Advances in Statistical Methods for Causal Inference in Prevention Science: Introduction to the Special Section.

The board of the Society for Prevention Research noted recently that extant methods for the analysis of causality mechanisms in prevention may still be too rudimentary for detailed and sophisticated analysis of causality hypotheses. This Special Section aims to fill some of the current voids, in particular in the domain of statistical methods of the analysis of causal inference. In the first article, Bray et al. propose a novel methodological approach in which they link propensity score techniques and Latent Class Analysis. In the second article, Kelcey et al. discuss power analysis tools for the study of causal mediation effects in cluster-randomized interventions. Wiedermann et al. present, in the third article, methods of Direction Dependence Analysis for the identification of confounders and for inference concerning the direction of causal effects in mediation models. A more general approach to the identification of causal structures in non-experimental data is presented by Shimizu in the fourth article. This approach is based on linear non-Gaussian acyclic models. Molenaar introduces vector-autoregressive methods for the optimal representation of Granger causality in time-dependent data. The Special Section concludes with a commentary by Musci and Stuart. In this commentary, the contributions of the articles in the Special Section are highlighted from the perspective of the experimental causal research tradition.

It's all about balance: propensity score matching in the context of complex survey data.

Many research studies aim to draw causal inferences using data from large, nationally representative survey samples, and many of these studies use propensity score matching to make those causal inferences as rigorous as possible given the non-experimental nature of the data. However, very few applied studies are careful about incorporating the survey design with the propensity score analysis, which may mean that the results do not generate population inferences. This may be because few methodological studies examine how to best combine these methods. Furthermore, even fewer of them investigate different non-response mechanisms. This study examines methods for handling survey weights in propensity score matching analyses of survey data under different non-response mechanisms. Our main conclusions are: (i) whether the survey weights are incorporated in the estimation of the propensity score does not impact estimation of the population treatment effect, as long as good population treated-comparison balance is achieved on confounders, (ii) survey weights must be used in the outcome analysis, and (iii) the transferring of survey weights (i.e., assigning the weights of the treated units to the comparison units matched to them) can be beneficial under certain non-response mechanisms.

Sample Size Planning for Cluster-Randomized Interventions Probing Multilevel Mediation.

Multilevel mediation analyses play an essential role in helping researchers develop, probe, and refine theories of action underlying interventions and document how interventions impact outcomes. However, little is known about how to plan studies with sufficient power to detect such multilevel mediation effects. In this study, we describe how to prospectively estimate power and identify sufficient sample sizes for experiments intended to detect multilevel mediation effects. We outline a simple approach to estimate the power to detect mediation effects with individual- or cluster-level mediators using summary statistics easily obtained from empirical literature and the anticipated magnitude of the mediation effect. We draw on a running example to illustrate several different types of mediation and provide an accessible introduction to the design of multilevel mediation studies. The power formulas are implemented in the R package PowerUpR and the PowerUp software ( causalevaluation.org ).

The Incredible Years Teacher Classroom Management Program: Outcomes from a Group Randomized Trial.

This group randomized controlled trial (RCT) evaluated the efficacy of the Incredible Years Teacher Classroom Management Program (IY TCM) on student social behavioral and academic outcomes among a large diverse sample of students within an urban context. Participants included 105 teachers and 1817 students in kindergarten to third grade. Three-level hierarchical linear models (HLM) were conducted to examine the overall treatment effects on teacher-reported student behavior and academic outcomes. In addition, multi-level moderation analyses were conducted to examine whether the treatment effects on student outcomes differed by demographic variables and pretest measures of social emotional and disruptive behavior and academics. Findings indicate that IY TCM reduced student emotional dysregulation (d = - 0.14) and increased prosocial behavior (d = 0.13) and social competence (d = 0.13). In addition, students initially lower on measures of social and academic competence demonstrated significant improvements on the same measure at outcome in comparison to similar peers in control classrooms. Practical significance of the findings and implications for schools and policy makers are discussed.

Can Propensity Score Analysis Approximate Randomized Experiments Using Pretest and Demographic Information in Pre-K Intervention Research?

BACKGROUND: It is unclear whether propensity score analysis (PSA) based on pretest and demographic covariates will meet the ignorability assumption for replicating the results of randomized experiments. PURPOSE: This study applies within-study comparisons to assess whether pre-Kindergarten (pre-K) treatment effects on achievement outcomes estimated using PSA based on a pretest and demographic covariates can approximate those found in a randomized experiment. METHODS: Data-Four studies with samples of pre-K children each provided data on two math achievement outcome measures with baseline pretests and child demographic variables that included race, gender, age, language spoken at home, and mother's highest education. Research Design and Data Analysis-A randomized study of a pre-K math curriculum provided benchmark estimates of effects on achievement measures. Comparison samples from other pre-K studies were then substituted for the original randomized control and the effects were reestimated using PSA. The correspondence was evaluated using multiple criteria. RESULTS AND CONCLUSIONS: The effect estimates using PSA were in the same direction as the benchmark estimates, had similar but not identical statistical significance, and did not differ from the benchmarks at statistically significant levels. However, the magnitude of the effect sizes differed and displayed both absolute and relative bias larger than required to show statistical equivalence with formal tests, but those results were not definitive because of the limited statistical power. We conclude that treatment effect estimates based on a single pretest and demographic covariates in PSA correspond to those from a randomized experiment on the most general criteria for equivalence.

Experimental Power for Indirect Effects in Group-randomized Studies with Group-level Mediators.

Mediation analyses have provided a critical platform to assess the validity of theories of action across a wide range of disciplines. Despite widespread interest and development in these analyses, literature guiding the design of mediation studies has been largely unavailable. Like studies focused on the detection of a total or main effect, an important design consideration is the statistical power to detect indirect effects if they exist. Understanding the sensitivity to detect indirect effects is exceptionally important because it directly influences the scale of data collection and ultimately governs the types of evidence group-randomized studies can bring to bear on theories of action. However, unlike studies concerned with the detection of total effects, literature has not established power formulas for detecting multilevel indirect effects in group-randomized designs. In this study, we develop closed-form expressions to estimate the variance of and the power to detect indirect effects in group-randomized studies with a group-level mediator using two-level linear models (i.e., 2-2-1 mediation). The results suggest that when carefully planned, group-randomized designs may frequently be well positioned to detect mediation effects with typical sample sizes. The resulting power formulas are implemented in the R package PowerUpR and the PowerUp!-Mediator software (causalevaluation.org).

Meaningful Effect Sizes, Intraclass Correlations, and Proportions of Variance Explained by Covariates for Planning Two- and Three-Level Cluster Randomized Trials of Social and Behavioral Outcomes.

BACKGROUND: There is a need for greater guidance regarding design parameters and empirical benchmarks for social and behavioral outcomes to inform assumptions in the design and interpretation of cluster randomized trials (CRTs). OBJECTIVES: We calculated the empirical reference values on critical research design parameters associated with statistical power for children's social and behavioral outcomes, including effect sizes, intraclass correlations (ICCs), and proportions of variance explained by a covariate at different levels (R 2). SUBJECTS: Children from kindergarten to Grade 5 in the samples from four large CRTs evaluating the effectiveness of two classroom- and two school-level preventive interventions. MEASURES: Teacher ratings of students' social and behavioral outcomes using the Teacher Observation of Classroom Adaptation-Checklist and the Social Competence Scale-Teacher. RESEARCH DESIGN: Two types of effect size benchmarks were calculated: (1) normative expectations for change and (2) policy-relevant demographic performance gaps. The ICCs and R 2 were calculated using two-level hierarchical linear modeling (HLM), where students are nested within schools, and three-level HLM, where students were nested within classrooms, and classrooms were nested within schools. RESULTS AND CONCLUSIONS: Comprehensive tables of benchmarks and ICC values are provided to inform prevention researchers in interpreting the effect size of interventions and conduct power analyses for designing CRTs of children's social and behavioral outcomes. The discussion also provides a demonstration for how to use the parameter reference values provided in this article to calculate the sample size for two- and three-level CRTs designs.

Does aggregate school-wide achievement mediate fifth grade outcomes for former early childhood education participants?

This study used a multilevel mediation model to test the theory that former early childhood education (ECE) attendees' 5th grade achievement is mediated by the aggregate school-wide achievement of their elementary school. Aggregate school-wide achievement was defined as the percentage of 5th graders in a school who were at/above academic proficiency in reading or math. Research questions were: (a) Do ECE program participants have better achievement at 5th grade compared with their matched peers who did not participate in an ECE program?; and (b) Is the association between ECE attendance and 5th grade academic performance mediated by school-wide achievement? Results indicated that children who attended prekindergarten (pre-K) and child care outperformed their matched peers who had not attended ECE programs; conversely, those children who did not attend ECE actually outperformed their Head Start counterparts. Mediation analyses indicated that aggregate school-wide achievement at 5th grade partially mediated the association between former ECE attendance and 5th grade performance; however, these mediated effects were small. Overall, the size of the total effects of ECE and the 5th grade academic outcomes were consistent with prior studies. This research confirms the long-term effects of pre-K and child care until 5th grade.

Longitudinal associations between executive functioning and academic skills across content areas.

This study assessed 562 four-year-old children at the beginning and end of their prekindergarten (pre-k) year and followed them to the end of kindergarten. At each time point children were assessed on 6 measures of executive function (EF) and 5 subtests of the Woodcock-Johnson III academic achievement battery. Exploratory factor analyses yielded EF and achievement factor scores. We examined the longitudinal bidirectional associations between these domains as well as the bidirectional associations among the separate content areas and the EF factor. In the pre-k year, strong bidirectional associations were found for EF skills and mathematics and oral comprehension skills but not for literacy skills. After controlling for pre-k gains in both EF and achievement, EF skills continued to be strong predictors of gains in mathematics in kindergarten and a more moderate predictor of kindergarten language gains. These results provide important information on the interrelationship of the developmental domains of EF and achievement as well as support for efforts to determine effective pre-k activities and/or curricula that can improve children's EF skills. They also suggest that mathematics activities may be a possible avenue for improving EF skills in young children.