Gender, Racial, and Socioeconomic Disparities on Social and Behavioral Skills for K-8 Students With and Without Interventions: An Integrative Data Analysis of Eight Cluster Randomized Trials.

Despite decades of concern about disparities in educational outcomes for low SES students and students of color, there has been limited rigorous study of programmatic approaches for reducing these disparities in elementary or middle schools. We conducted integrative data analysis (IDA) of the combined data from eight Institute of Education Sciences funded cluster randomized trials to address the research gaps on social and behavioral outcome disparities. The final analytic sample includes 90,880 students in varying grade levels from kindergarten to grade 8 in 387 schools in 4 states (Maryland, Missouri, Virginia, and Texas). Two-level hierarchical linear modeling was used for multilevel moderation analysis. This study provided empirical evidence that there were significant gender, racial, and socioeconomic disparities on social and behavioral outcome measures for elementary and middle school students, the disparities significantly varied across schools, and the disparities could be reduced by interventions. We discussed our findings, implications for interpreting effect sizes of interventions using disparities as empirical benchmarks, and study limitations. We concluded with suggestions for future research.

Assessing meta-regression methods for examining moderator relationships with dependent effect sizes: A Monte Carlo simulation.

Dependent effect sizes are ubiquitous in meta-analysis. Using Monte Carlo simulation, we compared the performance of 2 methods for meta-regression with dependent effect sizes-robust variance estimation (RVE) and 3-level modeling-with the standard meta-analytic method for independent effect sizes. We further compared bias-reduced linearization and jackknife estimators as small-sample adjustments for RVE and Wald-type and likelihood ratio tests for 3-level models. The bias in the slope estimates, width of the confidence intervals around those estimates, and empirical type I error and statistical power rates of the hypothesis tests from these different methods were compared for mixed-effects meta-regression analysis with one moderator either at the study or at the effect size level. All methods yielded nearly unbiased slope estimates under most scenarios, but as expected, the standard method ignoring dependency provided inflated type I error rates when testing the significance of the moderators. Robust variance estimation methods yielded not only the best results in terms of type I error rate but also the widest confidence intervals and the lowest power rates, especially when using the jackknife adjustments. Three-level models showed a promising performance with a moderate to large number of studies, especially with the likelihood ratio test, and yielded narrower confidence intervals around the slope and higher power rates than those obtained with the RVE approach. All methods performed better when the moderator was at the effect size level, the number of studies was moderate to large, and the between-studies variance was small. Our results can help meta-analysts deal with dependency in their data.

Synthesizing Evidence in Public Policy Contexts: The Challenge of Synthesis When There Are Only a Few Studies.

For a variety of reasons, researchers and evidence-based clearinghouses synthesizing the results of multiple studies often have very few studies that are eligible for any given research question. This situation is less than optimal for meta-analysis as it is usually practiced, that is, by employing inverse variance weights, which allows more informative studies to contribute relatively more to the analysis. This article outlines the choices available for synthesis when there are few studies to synthesize. As background, we review the synthesis practices used in several projects done at the behest of governmental agencies and private foundations. We then discuss the strengths and limitations of different approaches to meta-analysis in a limited information environment. Using examples from the U.S. Department of Education's What Works Clearinghouse as case studies, we conclude with a discussion of Bayesian meta-analysis as a potential solution to the challenges encountered when attempting to draw inferences about the effectiveness of interventions from a small number of studies.

Fitting meta-analytic structural equation models with complex datasets.

A modification of the first stage of the standard procedure for two-stage meta-analytic structural equation modeling for use with large complex datasets is presented. This modification addresses two common problems that arise in such meta-analyses: (a) primary studies that provide multiple measures of the same construct and (b) the correlation coefficients that exhibit substantial heterogeneity, some of which obscures the relationships between the constructs of interest or undermines the comparability of the correlations across the cells. One component of this approach is a three-level random effects model capable of synthesizing a pooled correlation matrix with dependent correlation coefficients. Another component is a meta-regression that can be used to generate covariate-adjusted correlation coefficients that reduce the influence of selected unevenly distributed moderator variables. A non-technical presentation of these techniques is given, along with an illustration of the procedures with a meta-analytic dataset. Copyright © 2016 John Wiley & Sons, Ltd.

Programs to increase high school completion: a community guide systematic health equity review.

CONTEXT: High school completion (HSC) is an established predictor of long-term morbidity and mortality. U.S. rates of HSC are substantially lower among students from low-income families and most racial/ethnic minority populations than students from high-income families and the non-Hispanic white population. This systematic review assesses the effectiveness of programs to increase HSC and the potential of these programs to improve lifelong health among at-risk students. EVIDENCE ACQUISITION: A search located a meta-analysis (search period 1985-2010/2011) on the effects of programs to increase HSC or General Educational Development (GED) diploma receipt; the meta-analysis was concordant with Community Guide definitions and methodologic standards. Programs were assessed separately for the general student population (152 studies) and students who were parents or pregnant (15 studies). A search for studies published between 2010 and August 2012 located ten more recent studies, which were assessed for consistency with the meta-analysis. Analyses were conducted in 2013. EVIDENCE SYNTHESIS: The review focused on the meta-analysis. Program effectiveness was measured as the increased rate of HSC (or GED receipt) by the intervention group compared with controls. All assessed program types were effective in increasing HSC in the general student population: vocational training, alternative schooling, social-emotional skills training, college-oriented programming, mentoring and counseling, supplemental academic services, school and class restructuring, multiservice packages, attendance monitoring and contingencies, community service, and case management. For students who had children or were pregnant, attendance monitoring and multiservice packages were effective. Ten studies published after the search period for the meta-analysis were consistent with its findings. CONCLUSIONS: There is strong evidence that a variety of HSC programs can improve high school or GED completion rates. Because many programs are targeted to high-risk students and communities, they are likely to advance health equity.

A meta-analysis of the effects of dropout prevention programs on school absenteeism.

This study reports findings from a systematic review and meta-analysis of literature examining the effects of school dropout prevention and intervention programs on students' school absenteeism outcomes. The meta-analysis synthesized 74 effect sizes measuring posttest differences in school absenteeism outcomes for youth enrolled in dropout prevention programs relative to a comparison group. Although results from randomized controlled trials indicated significant beneficial program effects, findings from quasi-experimental studies indicated no significant beneficial or detrimental effects. Examination of study characteristics suggested that dropout programs may have beneficial effects on school absenteeism among primarily male samples, and younger samples. Although no single type of intervention program was consistently more effective than others, vocational oriented and supplemental academic training programs showed some promise. However, the inconsistency in results and the possibility of small study bias mean the quality of evidence in this literature is low; at this time there is not enough evidence to conclude that dropout prevention programs have a universal impact on youth's school absenteeism outcomes.

The comparative effectiveness of outpatient treatment for adolescent substance abuse: a meta-analysis.

Meta-analysis was used to synthesize research on the effects of outpatient treatment on substance use outcomes for adolescents with substance use disorders. An extensive literature search located 45 eligible experimental or quasi-experimental studies reporting 73 treatment-comparison group pairs, with many of the comparison groups also receiving some treatment. The first analysis examined 250 effect sizes for the substance use outcomes of adolescents receiving different types of treatment relative to the respective comparison groups. As a category, family therapy programs were found to be more effective than their comparison conditions, whereas no treatment programs were less effective. However, not all treatment types were compared with each other in the available research, making it difficult to assess the comparative effectiveness of the different treatments. To provide a more differentiated picture of the relative improvement in substance use outcomes for different treatments, a second analysis examined 311 pre-post effect sizes measuring changes in substance use for adolescents in the separate treatment and comparison arms of the studies. The adolescents in almost all types of treatment showed reductions in substance use. The greatest improvements were found for family therapy and mixed and group counseling. Longer treatment duration was associated with smaller improvements, but other treatment characteristics and participant characteristics had little relationship to the pre-post changes in substance use. Based on these findings family therapy is the treatment with the strongest evidence of comparative effectiveness, although most types of treatment appear to be beneficial in helping adolescents reduce their substance use.

School-based interventions for aggressive and disruptive behavior: update of a meta-analysis.

BACKGROUND: Research about the effectiveness of school-based psychosocial prevention programs for reducing aggressive and disruptive behavior was synthesized using meta-analysis. This work updated previous work by the authors and further investigated which program and student characteristics were associated with the most positive outcomes. METHODS: Two hundred forty-nine experimental and quasi-experimental studies of school-based programs with outcomes representing aggressive and/or disruptive behavior were obtained. Effect sizes and study characteristics were coded from these studies and analyzed. RESULTS: Positive overall intervention effects were found on aggressive and disruptive behavior and other relevant outcomes. The most common and most effective approaches were universal programs and targeted programs for selected/indicated children. The mean effect sizes for these types of programs represent a decrease in aggressive/disruptive behavior that is likely to be of practical significance to schools. Multicomponent comprehensive programs did not show significant effects and those for special schools or classrooms were marginal. Different treatment modalities (e.g., behavioral, cognitive, social skills) produced largely similar effects. Effects were larger for better-implemented programs and those involving students at higher risk for aggressive behavior. CONCLUSIONS: Schools seeking prevention programs may choose from a range of effective programs with some confidence that whatever they pick will be effective. Without the researcher involvement that characterizes the great majority of programs in this meta-analysis, schools might be well-advised to give priority to those that will be easiest to implement well in their settings.

The effects of school-based intervention programs on aggressive behavior: a meta-analysis.

Research on the effectiveness of school-based programs for preventing or reducing aggressive behavior was synthesized with a meta-analysis. Changes in aggressive behavior between pretest and posttest were analyzed for developmental patterns and characteristics associated with differential effects. Control groups showed little change in aggressive behavior, but there were significant reductions among intervention groups. Most studies were conducted on demonstration programs; the few studies of routine practice programs showed much smaller effects. Among demonstration programs, positive outcomes were associated with a variety of study, subject, and intervention characteristics. Most notably, higher risk youth showed greater reductions in aggressive behavior, poorly implemented programs produced smaller effects, and different types of programs were generally similar in their effectiveness, other things equal.