Evaluation of various estimators for standardized mean difference in meta-analysis.

Meta-analyses of a treatment's effect compared with a control frequently calculate the meta-effect from standardized mean differences (SMDs). SMDs are usually estimated by Cohen's d or Hedges' g. Cohen's d divides the difference between sample means of a continuous response by the pooled standard deviation, but is subject to nonnegligible bias for small sample sizes. Hedges' g removes this bias with a correction factor. The current literature (including meta-analysis books and software packages) is confusingly inconsistent about methods for synthesizing SMDs, potentially making reproducibility a problem. Using conventional methods, the variance estimate of SMD is associated with the point estimate of SMD, so Hedges' g is not guaranteed to be unbiased in meta-analyses. This article comprehensively reviews and evaluates available methods for synthesizing SMDs. Their performance is compared using extensive simulation studies and analyses of actual datasets. We find that because of the intrinsic association between point estimates and standard errors, the usual version of Hedges' g can result in more biased meta-estimation than Cohen's d. We recommend using average-adjusted variance estimators to obtain an unbiased meta-estimate, and the Hartung-Knapp-Sidik-Jonkman method for accurate estimation of its confidence interval.

Estimating outcome-specific effects in meta-analyses of multiple outcomes: A simulation study.

In meta-analysis, primary studies often include multiple, dependent effect sizes. Several methods address this dependency, such as the multivariate approach, three-level models, and the robust variance estimation (RVE) method. As for today, most simulation studies that explore the performance of these methods have focused on the estimation of the overall effect size. However, researchers are sometimes interested in obtaining separate effect size estimates for different types of outcomes. A recent simulation study (Park & Beretvas, 2019) has compared the performance of the three-level approach and the RVE method in estimating outcome-specific effects when several effect sizes are reported for different types of outcomes within studies. The goal of this paper is to extend that study by incorporating additional simulation conditions and by exploring the performance of additional models, such as the multivariate model, a three-level model that specifies different study-effects for each type of outcome, a three-level model that specifies a common study-effect for all outcomes, and separate three-level models for each type of outcome. Additionally, we also tested whether the a posteriori application of the RV correction improves the standard error estimates and the 95% confidence intervals. Results show that the application of separate three-level models for each type of outcome is the only approach that consistently gives adequate standard error estimates. Also, the a posteriori application of the RV correction results in correct 95% confidence intervals in all models, even if they are misspecified, meaning that Type I error rate is adequate when the RV correction is implemented.

Concealed correlations meta-analysis: A new method for synthesizing standardized regression coefficients.

The synthesis of standardized regression coefficients is still a controversial issue in the field of meta-analysis. The difficulty lies in the fact that the standardized regression coefficients belonging to regression models that include different sets of covariates do not represent the same parameter, and thus their direct combination is meaningless. In the present study, a new approach called concealed correlations meta-analysis is proposed that allows for using the common information that standardized regression coefficients from different regression models contain to improve the precision of a combined focal standardized regression coefficient estimate. The performance of this new approach was compared with that of two other approaches: (1) carrying out separate meta-analyses for standardized regression coefficients from studies that used the same regression model, and (2) performing a meta-regression on the focal standardized regression coefficients while including an indicator variable as a moderator indicating the regression model to which each standardized regression coefficient belongs. The comparison was done through a simulation study. The results showed that, as expected, the proposed approach led to more accurate estimates of the combined standardized regression coefficients under both random- and fixed-effect models.

The Incremental Validity of the Trait Emotional Intelligence Questionnaire (TEIQue): A Systematic Review and Meta-Analysis.

A criticism leveled against the conceptualization of emotional intelligence (EI) as a personality trait is that it overlaps considerably with the higher order personality dimensions and, therefore, has weak utility. To investigate this criticism, a systematic review and meta-analysis were conducted to synthesize the literature examining the incremental validity of the 2 adult self-report forms of the Trait Emotional Intelligence Questionnaire (TEIQue). Twenty-four articles reporting 114 incremental validity analyses of the TEIQue were reviewed according to the studies' methodological features. Additionally, data from 18 studies (providing 105 effect sizes) were pooled in a meta-analysis. Results suggest that the TEIQue consistently explains incremental variance in criteria pertaining to different areas of functioning, beyond higher order personality dimensions and other emotion-related variables. The pooled effect size was relatively small, but statistically and practically significant (ΔR(2) = .06, SE = .0116; 95% CI [.03, .08]). The number of covariates controlled for, the form of the TEIQue, and the focus on higher order personality dimensions versus other individual-difference constructs as baseline predictors did not affect the effect size. Analyses conducted at the factor level indicated that the incremental contribution is mainly due to the well-being and self-control factors of trait EI. Methodological issues and directions for future research are discussed.

Measurement of the bystander intervention model for bullying and sexual harassment.

Although peer bystanders can exacerbate or prevent bullying and sexual harassment, research has been hindered by the absence of a validated assessment tool to measure the process and sequential steps of the bystander intervention model. A measure was developed based on the five steps of Latané and Darley's (1970) bystander intervention model applied to bullying and sexual harassment. Confirmatory factor analysis with a sample of 562 secondary school students confirmed the five-factor structure of the measure. Structural equation modeling revealed that all the steps were influenced by the previous step in the model, as the theory proposed. In addition, the bystander intervention measure was positively correlated with empathy, attitudes toward bullying and sexual harassment, and awareness of bullying and sexual harassment facts. This measure can be used for future research and to inform intervention efforts related to the process of bystander intervention for bullying and sexual harassment.

The prevalence of pathological gambling among college students: a meta-analytic synthesis, 2005-2013.

The problem of gambling addiction can be especially noteworthy among college and university students, many of whom have the resources, proximity, free time, and desire to become involved in the myriad options of gambling now available. Although limited attention has been paid specifically to college student gambling in the body of literature, there have been two published meta-analyses estimating the prevalence of probable pathological gambling among college students. This present study aims to be the third, presenting an up-to-date proportion of those students exhibiting gambling pathology, and is the first to include international studies from outside the United States and Canada. The purpose of this study was to use the most up-to-date meta-analytical procedures to synthesize the rates of probable pathological gambling for college and university students worldwide. A thorough literature review and coding procedure resulted in 19 independent data estimates retrieved from 18 studies conducted between 2005 and 2013. To synthesize the studies, a random effects model for meta-analysis was applied. The estimated proportion of probable pathological gamblers among the over 13,000 college students surveyed was computed at 10.23%, considerably higher than either of the two previously published meta-analyses, and more than double the rate reported in the first meta-analysis of this type published in 1999. Implications and recommendations for future practice in dealing with college students and gambling addiction are outlined and described for both administrators and mental health professionals.

Campbell Standards: Modernizing Campbell's Methodologic Expectations for Campbell Collaboration Intervention Reviews (MECCIR).

Introduction: The authors formed a small working group to modernize the Methodological Expectations for Campbell Collaboration Intervention Reviews (MECCIR). We reviewed comments and feedback from editors, peer reviewers of Campbell submissions, and authors; for example, that the Campbell MECCIR was long and some of the items in the reporting and conduct checklists were difficult to cross-reference. We also wanted to make the checklist more relevant for reviews of associations or risk factors and other quantitative non-intervention review types, which we welcome in Campbell. Thus, our aim was to develop a shorter, more holistic guidance and checklist of Campbell Standards, encompassing both conduct and reporting of these standards within the same checklist. Methods: Our updated Campbell Standards will be a living document. To develop this first iteration, we invited Campbell members to join a virtual working group; we sought experience in conducting Campbell systematic reviews and in conducting methods editor reviews for Campbell. We aligned the items from the MECCIR for conduct and reporting, then compared the principles of conduct that apply across review types to Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA)-literature search extension (S) and PRISMA-2020 reporting standards. We discussed each section with the aim of developing a parsimonious checklist with explanatory guidance while avoiding losing important concepts that are relevant to all types of reviews. We held nine meetings to discuss each section in detail between September 2022 and March 2023. We circulated this initial checklist and guidance to all Campbell editors, methods editors, information specialists and co-chairs to seek their feedback. All feedback was discussed by the working group and incorporated to the Standards or, if not incorporated, a formal response was returned about the rationale for why the feedback was not incorporated. Campbell Policy: The guidance includes seven main sections with 35 items multifaceted but distinct concepts that authors must adhere to when conducting Campbell reviews. Authors and reviewers must be mindful that multiple factors need to be assessed for each item. According to the Campbell Standards, the reporting of Campbell reviews must adhere to appropriate PRISMA reporting guidelines(s) such as PRISMA-2020. How to Use: The editorial board recommends authors use the checklist during their work in formulating their protocol, carrying out their review, and reporting it. Authors will be asked to submit a completed checklist with their submission. We plan to develop an online tool to facilitate use of the form by author teams and those reviewing submissions. Providing Feedback: We invite the scientific community to provide their comments using this anonymous google form. Plan for Updating: We will update the Campbell Standards periodically in light of new evidence.

Diversity and representation in ADHD psychosocial treatment research: A comprehensive synthesis with data from over 10,000 participants.

Demographic data from nearly 50 years of treatment research for children and adolescents with attention-deficit/hyperactivity disorder (ADHD) are synthesized. Comprehensive search identified ADHD treatment studies that were between-group designs, included a psychosocial, evidence-based treatment, and were conducted in the United States. One hundred and twenty-six studies that included 10,604 youth were examined. Reporting of demographics varied with 48% of studies (k = 61) reporting ethnicity, 73% (k = 92) reporting race, 80% (k = 101) reporting age (M age = 8.81, SD = 2.82), and 88% (k = 111) reporting gender. Most participants identified as non-Hispanic/Latine (15.99% Hispanic/Latine), White (62.54%), and boys (74.39%; 24.47% girls). Since the 1970s, zero youth in ADHD treatment studies identified as Middle Eastern/North African, 0.1% were American Indian/Alaskan Native or Native Hawaiian Pacific Islander, 1.77% were Asian, 15.10% were Black, and 3.14% were Multiracial. Based on publication year, the proportions of girls, racially minoritized youth, and Hispanic/Latine youth included in ADHD treatment research have increased over time. Girls, non-binary and non-cisgender youth, young children, adolescents, Hispanic/Latine youth, and youth from all racial groups other than White are underrepresented in ADHD treatment research. Research gaps are discussed, and recommendations for comprehensive demographic reporting in child and adolescent psychological research are provided.

Examining the Associations Between Childhood Trauma, Resilience, and Depression: A Multivariate Meta-Analysis.

TOPIC OF REVIEW: Childhood trauma has been associated with increased depression; however, resilience has been found to reduce this association. METHOD OF REVIEW: Present analyses were based on multivariate meta-analytical techniques, an extension of univariate meta-analysis. All computations were performed using the metafor package and the metaRmat package from R. Bivariate associations (r) between trauma, resilience, and depression were utilized as the pooled effect sizes. CRITERIA FOR INCLUSION: Following Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines, articles were coded based on the following inclusion criteria: (a) sample participants had a history of childhood trauma; (b) studies included one of the four instruments of trait resilience; (c) studies included measures of individual outcomes of depression; (d) studies were published in peer-reviewed journals, dissertations, book chapters since 2009, or provided by leading scholars who had yet to publish their data; (e) all manuscripts were written in English; and (f) studies included the Pearson Product Moment Correlation Coefficient (r) for the effect size. CRITERIA FOR REVIEW: We systematically coded for the following items for each study: year of publication, type of report, peer-reviewed, funding, sampling strategy, sample size, gender, mean age, country of study, measure of resilience, measure depression, and measure of childhood trauma. MAJOR FINDINGS: The pooled correlations indicate that trauma, resilience, and depression are significantly associated. There were no significant differences in symptoms of depression for high versus low reports of resilience for individuals with a history of trauma. Year of publication was a significant moderator for the associations between trauma, resilience, and depression. Resilience significantly mediated the association between trauma and depression.

Student self-ratings of social-emotional competencies: Dimensional structure and outcome associations of the WCSD-SECA among Hispanic and non-Hispanic White boys and girls in elementary through high school.

School-based assessments of students' self-reported social-emotional competencies (SECs) are an essential part of social and emotional learning (SEL) initiatives. Few studies, however, have investigated whether such assessments align with the frameworks that inform SEL practices, especially for diverse populations. In the present study we investigated the dimensional structure of the 40-item Washoe County School District Social-Emotional Competency Assessment (WCSD-SECA), which was designed to measure the five domains of SECs defined by the widely used Collaborative for Academic Social and Emotional Learning framework (CASEL 5). Findings showed that a subset of 21 items fit a 3-factor solution that reflected Intrapersonal, Interpersonal, and Emotion-Focused competencies, a structure consistent with previous theorizing of broad SEC constructs. This 3-dimensional structure was partially invariant, with differences especially evident in item thresholds across subpopulations (defined by the intersection of grade level, gender, and race/ethnicity). Accounting for differences in item thresholds increased mean differences among subpopulations in the three domains. Across subpopulations, Intrapersonal scores were positively associated with students' standardized test scores and GPAs, and negatively related to the number of days they were absent from school, in multilevel models that adjusted for school-level clustering and included all three SEC scores and student demographic controls. Interpersonal scores were associated with fewer suspensions. Interpersonal and Emotion-Focused scores demonstrated unexpectedly negative associations with some outcomes in these models. Findings contribute to an emerging body of research that aims to deepen understandings of the content and structure of students' SECs as well as the factors that contribute to growth in these competencies.

Association of Nonobstructive Chronic Bronchitis With All-Cause Mortality: A Systematic Literature Review and Meta-analysis.

BACKGROUND: The effect of nonobstructive chronic bronchitis (CB) on mortality is unclear. RESEARCH QUESTION: Is nonobstructive CB associated with increased all-cause mortality? STUDY DESIGN AND METHODS: We conducted a systematic literature review and meta-analysis to assess the association of nonobstructive CB and all-cause mortality. We searched for articles that included both CB and mortality in the title, abstract, or both in PubMed and EMBASE. We excluded studies in which participants demonstrated obstructive spirometry findings and studies in which CB and mortality were not defined. We used the Newcastle-Ottawa Quality Assessment Scale to assess study quality. We pooled adjusted hazard ratios (HRs) using the random effects model and inverse variance weighting. We conducted stratified analysis by the definition of CB and smoking status. We used Cochran's Q and I2 to assess for heterogeneity. We assessed publication bias by visual inspection of a funnel plot. RESULTS: Of 5,014 titles identified, eight fulfilled the inclusion and exclusion criteria. Overall nonobstructive CB was associated with all-cause mortality (HR, 1.37; 95% CI, 1.26-1.50) with no statistically significant heterogeneity (P = .14; I2 = 29%). Nonobstructive CB was associated with increased mortality in studies that defined CB as any respiratory symptoms (broad definition; HR, 1.28; 95% CI, 1.10-1.48; I2 = 0%) as well as in the rest of the studies (HR, 1.40; 95% CI, 1.26-1.56; I2 = 37%). Nonobstructive CB was associated with increased mortality in ever smokers (HR, 1.49; 95% CI, 1.35-1.64; I2 = 0%), but was not associated with increased mortality in never smokers (HR, 1.22; 95% CI, 0.90-1.66), and moderate heterogeneity was found (P = .10; I2 = 49%). The funnel plot did not indicate evidence of a publication bias because it showed symmetrical distribution of studies. INTERPRETATION: Nonobstructive CB is associated with increased all-cause mortality, and this association seems to be present only in current and former smokers. Further research should investigate whether this high-risk population may benefit from early therapeutic intervention. TRIAL REGISTRY: PROSPERO; No.: CRD42021253596; URL: www.crd.york.ac.uk/prospero.

Interpreting the effectiveness of a summer reading program: The eye of the beholder.

In applying a methods-oriented approach to evaluation, this study interpreted the effectiveness of a summer reading program from three different stakeholder perspectives: practitioners from the school district, the funding agency supporting the program, and the policymakers considering mandating summer school. Archival data were obtained on 2330 students reading below benchmark in Grades 2-5. After propensity score matching participants to peers who did not attend the summer program, the final sample consisted of 630 students. Pre-to-posttest growth models revealed positive effects in Grades 2-4 (standardized slopes of .40-.54), but fifth graders demonstrated negligible improvement (standardized slope of .15). The standardized mean differences of propensity score matched treatment and control group students indicated null effects in all grade levels (d = -.13 to .05). Achieving proficient reading performance also was not attributable to summer school participation. Findings underscore the importance of operationalizing effectiveness in summative evaluation.

Does the group in group psychotherapy matter? A meta-analysis of the intraclass correlation coefficient in group treatment research.

OBJECTIVE: Over the last 3 decades, group treatment researchers have become increasingly knowledgeable of the impact of within-group dependency on analyses of group treatment data and of mutual influence processes that occur within therapy groups. Despite these advancements, there remains a lack of consensus on the magnitude of mutual influence, or group effects, in group treatment research. As such, this study sought to estimate the size of group effects on members' posttreatment outcomes by meta-analyzing the intraclass correlation coefficients (ICCs) in group treatment research. In addition, we tested several moderators of the ICC, including outcome type, outcome reactivity, outcome specificity, group format, treatment length, and group size. METHOD: Using robust variance estimations, we meta-analyzed 169 effect sizes from 37 group treatment studies. RESULTS: Findings indicated an average ICC of 0.06. Group size, group format, treatment length, outcome specificity, and outcome type did not significantly moderate the ICC; however, we did find evidence to suggest that the ICC varies as a function of outcome reactivity, with observer-rated outcome measures resulting in the largest ICC. CONCLUSION: These findings suggest that interdependence in group treatment research is an important concept both theoretically and statistically. (PsycINFO Database Record (c) 2020 APA, all rights reserved).

Psychosocial Interventions for Attention-Deficit/Hyperactivity Disorder: Systematic Review with Evidence and Gap Maps.

OBJECTIVE: To inform the scope of future systematic reviews, meta-analyses, and treatment outcome studies, this review aims to describe the extent of the evidence for psychosocial interventions for children and adolescents with attention-deficit/hyperactivity disorder, with particular attention to specific types of interventions, targets of outcome assessment, and risk of bias. METHOD: A comprehensive search of relevant databases (i.e., Medline, PsychInfo, Education Resources Information Center, and ProQuest Dissertation Database) was conducted. Detailed information related to treatment type, outcome assessment, study design, and risk of bias was extracted by trained coders. Evidence and gap maps were created to summarize evidence within types of treatments and targets of outcome assessment. Indicators of risk of bias were assessed for selected combinations of treatments and outcome assessment. RESULTS: We identified 185 eligible individual studies and 3817 effect sizes. Behavioral parent training and cognitive training (COG) were the most commonly studied stand-alone interventions. Treatment versus control comparisons for stand-alone interventions (s = 70) were less common than for complex interventions involving combinations of psychosocial interventions (s = 100). Combinations of behavioral and child training (e.g., COG, organizational training) interventions were the most frequently studied. CONCLUSION: There is a considerable variability within this literature regarding combinations of treatments across outcome assessment targets. To address gaps in existing evidence, more primary studies assessing direct comparisons of isolated and combined treatment effects of specific types of psychosocial treatments relative to control and other treatments are needed. Future meta-analyses should take into account the complexity and breadth of available evidence.

The effects of varied practice on the oral reading fluency of fourth-grade students.

To improve oral reading fluency rate and promote its generalization to unpracticed texts, this study investigated a Varied Practice approach that involved passages with a high proportion of overlapping words (M = 85% unique word overlap). Fourth graders were randomly assigned either to the Varied Practice treatment (n = 405), where they read three different passages one time each, or the Repeated Reading comparison (n = 422), in which they read the same passage three times each. Both groups read with a partner for about 20 min, 3-4 times per week, over an average 12 weeks (30 total sessions). Results indicated that students in Varied Practice demonstrated significantly better fluency outcomes than students in Repeated Reading, but both groups demonstrated growth near the 90th percentile. Results of a quantile regression revealed that low-to-middle achievers benefited from Varied Practice the most. Overall, the findings suggest fluency approaches rooted in statistical learning hold promise as an alternative to approaches focused on practicing words in redundant contexts.

Quasi-experimental study designs series-paper 9: collecting data from quasi-experimental studies.

OBJECTIVE: To identify variables that must be coded when synthesizing primary studies that use quasi-experimental designs. STUDY DESIGN AND SETTING: All quasi-experimental (QE) designs. RESULTS: When designing a systematic review of QE studies, potential sources of heterogeneity-both theory-based and methodological-must be identified. We outline key components of inclusion criteria for syntheses of quasi-experimental studies. We provide recommendations for coding content-relevant and methodological variables and outlined the distinction between bivariate effect sizes and partial (i.e., adjusted) effect sizes. Designs used and controls used are viewed as of greatest importance. Potential sources of bias and confounding are also addressed. CONCLUSION: Careful consideration must be given to inclusion criteria and the coding of theoretical and methodological variables during the design phase of a synthesis of quasi-experimental studies. The success of the meta-regression analysis relies on the data available to the meta-analyst. Omission of critical moderator variables (i.e., effect modifiers) will undermine the conclusions of a meta-analysis.

Quasi-experimental study designs series-paper 10: synthesizing evidence for effects collected from quasi-experimental studies presents surmountable challenges.

OBJECTIVE: To outline issues of importance to analytic approaches to the synthesis of quasi-experiments (QEs) and to provide a statistical model for use in analysis. STUDY DESIGN AND SETTING: We drew on studies of statistics, epidemiology, and social-science methodology to outline methods for synthesis of QE studies. The design and conduct of QEs, effect sizes from QEs, and moderator variables for the analysis of those effect sizes were discussed. RESULTS: Biases, confounding, design complexities, and comparisons across designs offer serious challenges to syntheses of QEs. Key components of meta-analyses of QEs were identified, including the aspects of QE study design to be coded and analyzed. Of utmost importance are the design and statistical controls implemented in the QEs. Such controls and any potential sources of bias and confounding must be modeled in analyses, along with aspects of the interventions and populations studied. Because of such controls, effect sizes from QEs are more complex than those from randomized experiments. A statistical meta-regression model that incorporates important features of the QEs under review was presented. CONCLUSION: Meta-analyses of QEs provide particular challenges, but thorough coding of intervention characteristics and study methods, along with careful analysis, should allow for sound inferences.

Quasi-experimental study designs series-paper 6: risk of bias assessment.

OBJECTIVES: Rigorous and transparent bias assessment is a core component of high-quality systematic reviews. We assess modifications to existing risk of bias approaches to incorporate rigorous quasi-experimental approaches with selection on unobservables. These are nonrandomized studies using design-based approaches to control for unobservable sources of confounding such as difference studies, instrumental variables, interrupted time series, natural experiments, and regression-discontinuity designs. STUDY DESIGN AND SETTING: We review existing risk of bias tools. Drawing on these tools, we present domains of bias and suggest directions for evaluation questions. RESULTS: The review suggests that existing risk of bias tools provide, to different degrees, incomplete transparent criteria to assess the validity of these designs. The paper then presents an approach to evaluating the internal validity of quasi-experiments with selection on unobservables. CONCLUSION: We conclude that tools for nonrandomized studies of interventions need to be further developed to incorporate evaluation questions for quasi-experiments with selection on unobservables.

How to communicate effect sizes for continuous outcomes: a review of existing options and introducing a new metric.

BACKGROUND AND OBJECTIVE: Communicating the results of studies is an important problem. For binary outcomes, communication is relatively straightforward and supported by several statistical tools (e.g., the risk ratio and the number needed to treat [NNT]). For continuous outcomes, the situation is much worse. We discuss several alternatives and propose a new metric, NNTthreshold, that is analogous to the NNT statistic for binary outcomes and that is useful when a clinically meaningful threshold can be identified. METHOD: We use both statistical theory and a statistical simulation to demonstrate how to compute NNTthreshold and to investigate how it behaves. RESULTS: NNTthreshold appears to function well, though is slightly conservative in our simulation. CONCLUSION: Study authors need to think carefully about how to discuss study results when outcomes are continuously scaled. NNTthreshold should be helpful in situations in which a clinically meaningful threshold can be identified.