PROTOCOL: Assessment of outcome reporting bias in studies included in Campbell systematic reviews.

This is the protocol for a Campbell systematic review. The objectives are as follows: To identify methods used to assess the risk of outcome reporting bias (ORB) in studies included in recent Campbell systematic reviews of intervention effects. The review will answer the following questions: What proportion of recent Campbell reviews included assessment of ORB? How did recent reviews define levels of risk of ORB (what categories, labels, and definitions did they use)? To what extent and how did these reviews use study protocols as sources of data on ORB? To what extent and how did reviews document reasons for judgments about risk of ORB? To what extent and how did reviews assess the inter-rater reliability of ORB ratings? To what extent and how were issues of ORB considered in the review's abstract, plain language summary, and conclusions?

Can Non-Randomised Studies of Interventions Provide Unbiased Effect Estimates? A Systematic Review of Internal Replication Studies.

Non-randomized studies of intervention effects (NRS), also called quasi-experiments, provide useful decision support about development impacts. However, the assumptions underpinning them are usually untestable, their verification resting on empirical replication. The internal replication study aims to do this by comparing results from a causal benchmark study, usually a randomized controlled trial (RCT), with those from an NRS conducted at the same time in the sampled population. We aimed to determine the credibility and generalizability of findings in internal replication studies in development economics, through a systematic review and meta-analysis. We systematically searched for internal replication studies of RCTs conducted on socioeconomic interventions in low- and middle-income countries. We critically appraised the benchmark randomized studies, using an adapted tool. We extracted and statistically synthesized empirical measures of bias. We included 600 estimates of correspondence between NRS and benchmark RCTs. All internal replication studies were found to have at least "some concerns" about bias and some had high risk of bias. We found that study designs with selection on unobservables, in particular regression discontinuity, on average produced absolute standardized bias estimates that were approximately zero, that is, equivalent to the estimates produced by RCTs. But study conduct also mattered. For example, matching using pre-tests and nearest neighbor algorithms corresponded more closely to the benchmarks. The findings from this systematic review confirm that NRS can produce unbiased estimates. Authors of internal replication studies should publish pre-analysis protocols to enhance their credibility.

A Primer on Meta-Analytic Structural Equation Modeling: the Case of Depression.

In this paper, we show how the methods of systematic reviewing and meta-analysis can be used in conjunction with structural equation modeling to summarize the results of studies in a way that will facilitate the theory development and testing needed to advance prevention science. We begin with a high-level overview of the considerations that researchers need to address when using meta-analytic structural equation modeling (MASEM) and then discuss a research project that brings together theoretically important cognitive constructs related to depression to (a) show how these constructs are related, (b) test the direct and indirect effects of dysfunctional attitudes on depression, and (c) test the effects of study-level moderating variables. Our results suggest that the indirect effect of dysfunctional attitudes (via negative automatic thinking) on depression is two and a half times larger than the direct effect of dysfunctional attitudes on depression. Of the three study-level moderators tested, only sample recruitment method (clinical vs general vs mixed) yielded different patterns of results. The primary difference observed was that the dysfunctional attitudes → automatic thoughts path was less strong for clinical samples than it was for general and mixed samples. These results illustrate how MASEM can be used to compare theoretically derived models and predictions resulting in a richer understanding of both the empirical results and the theories underlying them.

Families and Schools Together (FAST) for improving outcomes for children and their families.

BACKGROUND: Parents and carers have a major influence on children's learning and development from birth, through the school years, and into adulthood. Parental contributions to education include providing a secure environment in which to learn, providing intellectual stimulation, transmitting social norms and values, shaping the child's resilience through fostering literacy and problem-solving, and encouraging personal and social aspiration. Increasingly, providers of formalised education are recognising the primary role of parents, carers, and the wider family, as well as peers and the environment, in shaping children's education, health, and life experiences. OBJECTIVES: To assess the effectiveness of the Families and Schools Together (FAST) programme in improving outcomes among children and their families. SEARCH METHODS: Between October 2018 and December 2018, we searched CENTRAL, MEDLINE, Embase, PsycINFO, 11 additional databases, and three trial registers. We handsearched the reference lists of included studies and relevant reports and reviews, contacted the programme developer and independent researchers, and searched relevant websites to identify other eligible studies. SELECTION CRITERIA: We included randomised controlled trials (RCTs) and quasi-RCTs examining the effects of FAST, relative to waiting list, usual or alternative services, or no intervention, on outcomes for children (aged from birth to completion of compulsory education) and their families. DATA COLLECTION AND ANALYSIS: At least two review authors independently evaluated the records retrieved from the search for relevance. One review author (JV) extracted data from eligible studies with a second independent review author (AF, DK, or SL). Review authors consulted with one another to resolve disagreements. We used a fixed-effect model for meta-analysis. We presented results as standardised mean differences (SMDs) because all outcomes were continuously scaled, and we accompanied these with 95% confidence intervals (CIs). We used the GRADE approach to assess the certainty of evidence for each outcome. MAIN RESULTS: We identified 10 completed RCTs, most of which were relatively recent (2007 or later) and were conducted with at least some involvement from the intervention developer or the FAST organisation. Nine of the 10 trials were from the USA; the other was from the UK. Children were young (five to nine years old; mean age approximately six years), and therefore, whilst not so named in the reports, evaluations consisted of what is sometimes referred to as 'Kids FAST' and sometimes 'Elementary Level FAST'). Among the USA-based studies, at least 62% of participants were members of a racial/ethnic minority group (most commonly, African American or Latino). FAST was usually delivered in schools after the school day. Trials lasted about eight weeks and usually examined the effects of FAST relative to no additional intervention. Most studies were funded by agencies in the US federal government. We judged the certainty of evidence in the included studies to be moderate or low for the main review outcomes. Failure to include all families in outcome analyses (attrition) and possible bias in recruitment of families into the trials were the main limitations in the evidence.We included over 9000 children and their families in at least one meta-analysis. The follow results relate to meta-analyses of data at long-term follow-up.Primary outcomesFour studies (approximately 6276 children) assessed child school performance at long-term follow-up. The effect size was very small, and the CI did not include effects that, if real, suggest possibly important positive or negative effects if viewed from an individual perspective (SMD -0.02, 95% CI -0.11 to 0.08). We assessed the certainty of evidence for this outcome as moderate. No studies assessed child adverse events, parental substance use, or parental stress.Secondary outcomesParent reports of child internalising behaviour (SMD -0.03, 95% CI -0.11 to 0.17; 4 RCTs, approximately 908 children; low-certainty evidence) and family relationships (SMD 0.08, 95% CI -0.03 to 0.19; 4 RCTs, approximately 2569 children; moderate-certainty evidence) also yielded CIs that did not include effects that, if real, suggest possibly important positive or negative effects.The CI for parent reports of child externalising behaviour, however, did include effects that, if real, were possibly large enough to be important (SMD -0.19, 95% CI -0.32 to -0.05; 4 RCTs, approximately 754 children; low-certainty evidence). AUTHORS' CONCLUSIONS: Given these results, it is hard to support the assertion that assignment to FAST is associated with important positive outcomes for children and their parents.

Therapeutic alliance, empathy, and genuineness in individual adult psychotherapy: A meta-analytic review.

OBJECTIVE: Alliance, empathy, and genuineness are each integral parts of the therapeutic relationship. No previous meta-analysis has explored the extent to which therapist empathy and genuineness contribute to the therapeutic alliance. METHOD: In this meta-analysis, a multifaceted search strategy yielded 53 studies. Forty studies reported alliance/empathy relationships, eight studies reported alliance/genuineness relationships, and five studies reported both. RESULTS: Random effects meta-analyses revealed that therapeutic alliance was significantly related to perceptions of therapist empathy with a mean r = 0.50 (95% CI = 0.42, 0.57). Therapeutic alliance was also significantly related to perceptions of therapist genuineness with a mean r = 0.59 (95% CI = 0.45, 0.71). Tests of publication bias indicated a low likelihood of publication bias affecting the strength and direction of the results. Potential moderating variables were explored, including rater perspective, measure of therapeutic relationship variables, and client race/ethnicity. CONCLUSIONS: Therapeutic alliance has a moderate relationship with perceptions of therapist empathy and genuineness. Of note, there may be reason to believe that when rated by the same person, these constructs have significant overlap and lack discreteness. Future directions for study of the therapeutic relationship are discussed. Implications for practice are provided.

Meta-analyses of cardiovascular reactivity to rumination: A possible mechanism linking depression and hostility to cardiovascular disease.

Rumination is a way of cognitive coping associated with depression and hostility that prolongs cardiovascular responses to stress. If repeated over time, the associated autonomic dysregulation may be 1 mechanism linking depression and hostility to cardiovascular disease. The current meta-analyses investigate the magnitude of cardiovascular responses (heart rate, diastolic blood pressure, and systolic blood pressure) to induced state sadness and angry rumination which are associated with depression and hostility, respectively. A literature search identified 43 studies (3,348 participants) meeting inclusion criteria. A random effects model was applied to calculate cardiovascular reactivity weighted effect sizes during induced sadness and angry rumination. Large and significant effect sizes were found for all analyses, with the standardized mean statistic, d, for single group designs ranging from .75 to 1.39. Results suggest that angry rumination may have larger cardiovascular effects than sadness rumination, and that rumination likely affects blood pressure more than heart rate. Potential implications of this relationship are discussed in light of limitations of the current study and existing rumination research. (PsycINFO Database Record

Rumination and posttraumatic stress symptoms in trauma-exposed adults: a systematic review and meta-analysis.

BACKGROUND AND OBJECTIVES: Rumination is a correlate of increased posttraumatic stress (PTS) symptoms. This study quantitatively reviewed the literature on rumination and PTS symptoms in trauma-exposed adults, extending prior research by using an inclusive definition of trauma, addressing PTS symptom clusters, and conducting moderator analyses. METHOD: Searches were conducted in PsycINFO, PubMed, PILOTS, EBSCO Psychology and Behavioral Sciences Collection, Google Scholar, and Dissertation Abstracts. Sixty-four unique samples from 59 articles were included. RESULTS: Results showed a moderate, positive relationship between rumination and PTS symptoms (r = .50, p < .001). This was not moderated by time since trauma, gender, prior trauma history, Criterion A congruence of events, type of rumination or PTS symptom measure, or sample setting. However, trauma-focused rumination yielded smaller effect sizes than trait rumination. The association between rumination and intrusive re-experiencing was stronger than that between rumination and avoidance (t (13) = 9.18, p < .001), or rumination and hyperarousal (t (9) = 2.70, p = .022). CONCLUSIONS: Results confirm that rumination is associated with increased PTS symptoms. Future research should identify mechanisms underlying this association and their potential specificity by symptoms cluster, as well as further examine the potential moderating roles of gender and prior trauma history.

Quasi-experimental study designs series-paper 9: collecting data from quasi-experimental studies.

OBJECTIVE: To identify variables that must be coded when synthesizing primary studies that use quasi-experimental designs. STUDY DESIGN AND SETTING: All quasi-experimental (QE) designs. RESULTS: When designing a systematic review of QE studies, potential sources of heterogeneity-both theory-based and methodological-must be identified. We outline key components of inclusion criteria for syntheses of quasi-experimental studies. We provide recommendations for coding content-relevant and methodological variables and outlined the distinction between bivariate effect sizes and partial (i.e., adjusted) effect sizes. Designs used and controls used are viewed as of greatest importance. Potential sources of bias and confounding are also addressed. CONCLUSION: Careful consideration must be given to inclusion criteria and the coding of theoretical and methodological variables during the design phase of a synthesis of quasi-experimental studies. The success of the meta-regression analysis relies on the data available to the meta-analyst. Omission of critical moderator variables (i.e., effect modifiers) will undermine the conclusions of a meta-analysis.

Quasi-experimental study designs series-paper 10: synthesizing evidence for effects collected from quasi-experimental studies presents surmountable challenges.

OBJECTIVE: To outline issues of importance to analytic approaches to the synthesis of quasi-experiments (QEs) and to provide a statistical model for use in analysis. STUDY DESIGN AND SETTING: We drew on studies of statistics, epidemiology, and social-science methodology to outline methods for synthesis of QE studies. The design and conduct of QEs, effect sizes from QEs, and moderator variables for the analysis of those effect sizes were discussed. RESULTS: Biases, confounding, design complexities, and comparisons across designs offer serious challenges to syntheses of QEs. Key components of meta-analyses of QEs were identified, including the aspects of QE study design to be coded and analyzed. Of utmost importance are the design and statistical controls implemented in the QEs. Such controls and any potential sources of bias and confounding must be modeled in analyses, along with aspects of the interventions and populations studied. Because of such controls, effect sizes from QEs are more complex than those from randomized experiments. A statistical meta-regression model that incorporates important features of the QEs under review was presented. CONCLUSION: Meta-analyses of QEs provide particular challenges, but thorough coding of intervention characteristics and study methods, along with careful analysis, should allow for sound inferences.

Synthesizing Evidence in Public Policy Contexts: The Challenge of Synthesis When There Are Only a Few Studies.

For a variety of reasons, researchers and evidence-based clearinghouses synthesizing the results of multiple studies often have very few studies that are eligible for any given research question. This situation is less than optimal for meta-analysis as it is usually practiced, that is, by employing inverse variance weights, which allows more informative studies to contribute relatively more to the analysis. This article outlines the choices available for synthesis when there are few studies to synthesize. As background, we review the synthesis practices used in several projects done at the behest of governmental agencies and private foundations. We then discuss the strengths and limitations of different approaches to meta-analysis in a limited information environment. Using examples from the U.S. Department of Education's What Works Clearinghouse as case studies, we conclude with a discussion of Bayesian meta-analysis as a potential solution to the challenges encountered when attempting to draw inferences about the effectiveness of interventions from a small number of studies.

How to communicate effect sizes for continuous outcomes: a review of existing options and introducing a new metric.

BACKGROUND AND OBJECTIVE: Communicating the results of studies is an important problem. For binary outcomes, communication is relatively straightforward and supported by several statistical tools (e.g., the risk ratio and the number needed to treat [NNT]). For continuous outcomes, the situation is much worse. We discuss several alternatives and propose a new metric, NNTthreshold, that is analogous to the NNT statistic for binary outcomes and that is useful when a clinically meaningful threshold can be identified. METHOD: We use both statistical theory and a statistical simulation to demonstrate how to compute NNTthreshold and to investigate how it behaves. RESULTS: NNTthreshold appears to function well, though is slightly conservative in our simulation. CONCLUSION: Study authors need to think carefully about how to discuss study results when outcomes are continuously scaled. NNTthreshold should be helpful in situations in which a clinically meaningful threshold can be identified.

Psychotherapist effects in meta-analyses: How accurate are treatment effects?

Psychotherapists are known to vary in their effectiveness with their clients, in randomized clinical trials as well as naturally occurring treatment settings. The fact that therapists matter has 2 effects in psychotherapy studies. First, if therapists are not randomly assigned to modalities (which is rare) this may bias the estimation of the treatment effects, as the modalities may have therapists of differing skill. In addition, if the data are analyzed at the client level (which is virtually always the case) then the standard errors for the effect sizes will be biased due to a violation of the assumption of independence. Thus, the conclusions of many meta-analyses may not reflect true estimates of treatment differences. We reexamined 20 treatment effects selected from 17 meta-analyses. We focused on meta-analyses that found statistically significant differences between treatments for a variety of disorders by correcting the treatment effects according to the variability in outcomes known to be associated with psychotherapists. The results demonstrated that after adjusting the results based on most small estimates of therapist effects, ∼80% of the reported treatment effects would still be statistically significant. However, at larger estimates, only 20% of the treatment effects would still be statistically significant after controlling for therapist effects. Although some meta-analyses were consistent in their estimates for treatment differences, the degree of certainty in the results was considerably reduced after considering therapist effects. Practice implications for understanding treatment effects, namely, therapist effects, in meta-analyses and original studies are provided.

Systematic reviews of complex interventions: framing the review question.

The first and most important decision in preparing any systematic review is to clearly frame the question the review team seeks to answer. However, this is not always straightforward, particularly if synthesis teams are interested in the effects of complex interventions. In this article, we discuss how to formulate good systematic review questions of complex interventions. We describe the rationale for developing well-formulated review questions and review the existing guidance on formulating review questions. We discuss that complex interventions can contain a mix of effective and ineffective (or even harmful) actions, which may interact synergistically or dysynergistically or be interdependent, and how these interactions and interdependencies need to be considered when formulating systematic review questions. We discuss complexity specifically in terms of how it relates to the type of question, the scope of the review (i.e., lumping vs. splitting debate), and specification of the intervention. We offer several recommendations to assist review authors in developing a definition for their complex intervention of interest, which is an essential first step in formulating the review question. We end by identifying areas in which future methodological research aimed at improving question formulation, especially as it relates to complex interventions, is needed.

A meta-analytic study: the relationship between acculturation and depression among Asian Americans.

Acculturation is an important and popular cultural research variable among specific ethnic populations that is used to explain the process of assimilating into the host culture. Acculturation has often been used to account for psychosocial changes and health outcomes and has been used to explain health disparities among ethnic groups. Using Asian Americans as an illustrative ethnic group, the authors see that researchers have highlighted the influence of acculturation on health outcomes. Some researchers suggest that this relationship is positive, whereas others postulate that the opposite is true. Because of the highly complex and divergent findings in the literature, this meta-analysis addresses the question of how acculturation (as measured by acculturation scales) is related to depression (a specific mental health outcome) among the Asian American population living in North America. Analyses were based on 38 studies. The meta-analyses reveal that when acculturation is measured as assimilation to the American culture, there is a small but statistically significant negative relationship between acculturation and depression scores. When acculturation is measured as orientation to the Asian culture, the relationship between acculturation and depression scores is also negative, but not statistically significant.

Issues relating to study design and risk of bias when including non-randomized studies in systematic reviews on the effects of interventions.

Non-randomized studies may provide valuable evidence on the effects of interventions. They are the main source of evidence on the intended effects of some types of interventions and often provide the only evidence about the effects of interventions on long-term outcomes, rare events or adverse effects. Therefore, systematic reviews on the effects of interventions may include various types of non-randomized studies. In this second paper in a series, we address how review authors might articulate the particular non-randomized study designs they will include and how they might evaluate, in general terms, the extent to which a particular non-randomized study is at risk of important biases. We offer guidance for describing and classifying different non-randomized designs based on specific features of the studies in place of using non-informative study design labels. We also suggest criteria to consider when deciding whether to include non-randomized studies. We conclude that a taxonomy of study designs based on study design features is needed. Review authors need new tools specifically to assess the risk of bias for some non-randomized designs that involve a different inferential logic compared with parallel group trials. Copyright © 2012 John Wiley & Sons, Ltd.

Issues relating to confounding and meta-analysis when including non-randomized studies in systematic reviews on the effects of interventions.

BACKGROUND: Confounding caused by selection bias is often a key difference between non-randomized studies (NRS) and randomized controlled trials (RCTs) of interventions. KEY METHODOLOGICAL ISSUES: In this third paper of the series, we consider issues relating to the inclusion of NRS in systematic reviews on the effects of interventions. We discuss whether potential biases from confounding in NRS can be accounted for, the limitations of current methods for attempting to do so, the different contexts of NRS and RCTs, the problems these issues create for reviewers, and a research agenda for the future. GUIDANCE: Reviewers who are considering whether or not to include NRS in meta-analyses must weigh a number of factors. Including NRS may allow a review to address outcomes or pragmatic implementations of an intervention not studied in RCTs, but it will also increase the workload for the review team, as well as their required technical repertoire. Furthermore, the results of a synthesis involving NRS will likely be more difficult to interpret, and less certain, relative to the results of a synthesis involving only randomized studies. When both randomized and non-randomized evidence are available, we favor a strategy of including NRS and RCTs in the same systematic review but synthesizing their results separately. CONCLUSION: Including NRS will often make the limitations of the evidence derived from RCTs more apparent, thereby guiding inferences about generalizability, and may help with the design of the next generation of RCTs. Copyright © 2012 John Wiley & Sons, Ltd.

Depression as a risk factor for breast cancer: investigating methodological limitations in the literature.

PURPOSE: A relationship between depression and the development of breast cancer has not been convincingly shown in the research conducted over the past three decades. METHODS: In an effort to better understand the conflicting results, a review was conducted focusing on the methodological problems associated with this literature, including time frame between the assessment of depression and the diagnosis of breast cancer and the use of somatic items in measuring depression. Fifteen breast cancer prospective studies were reviewed. RESULTS: While twelve of the studies found positive associations between depression and breast cancer development, three studies found negative associations. With regard to the predictive associations between depression and breast cancer incidence, the findings revealed that (a) studies using a longer time frame found a stronger association than studies using a shorter time frame and (b) studies utilizing depression measures that did not contain somatic items found a smaller association than studies utilizing depression measures that did contain these items. CONCLUSIONS: Future studies should ensure that sufficient periods of time between the measurement of depression and the assessment of cancer and avoid measuring depression using somatic items.

Cognitive-behavioural interventions for children who have been sexually abused.

BACKGROUND: Despite differences in how it is defined, there is a general consensus amongst clinicians and researchers that the sexual abuse of children and adolescents ('child sexual abuse') is a substantial social problem worldwide. The effects of sexual abuse manifest in a wide range of symptoms, including fear, anxiety, post-traumatic stress disorder and various externalising and internalising behaviour problems, such as inappropriate sexual behaviours. Child sexual abuse is associated with increased risk of psychological problems in adulthood. Cognitive-behavioural approaches are used to help children and their non-offending or 'safe' parent to manage the sequelae of childhood sexual abuse. This review updates the first Cochrane review of cognitive-behavioural approaches interventions for children who have been sexually abused, which was first published in 2006. OBJECTIVES: To assess the efficacy of cognitive-behavioural approaches (CBT) in addressing the immediate and longer-term sequelae of sexual abuse on children and young people up to 18 years of age. SEARCH METHODS: We searched the Cochrane Central Register of Controlled Trials (CENTRAL) (2011 Issue 4); MEDLINE (1950 to November Week 3 2011); EMBASE (1980 to Week 47 2011); CINAHL (1937 to 2 December 2011); PsycINFO (1887 to November Week 5 2011); LILACS (1982 to 2 December 2011) and OpenGrey, previously OpenSIGLE (1980 to 2 December 2011). For this update we also searched ClinicalTrials.gov and the International Clinical Trials Registry Platform (ICTRP). SELECTION CRITERIA: We included randomised or quasi-randomised controlled trials of CBT used with children and adolescents up to age 18 years who had experienced being sexually abused, compared with treatment as usual, with or without placebo control. DATA COLLECTION AND ANALYSIS: At least two review authors independently assessed the eligibility of titles and abstracts identified in the search. Two review authors independently extracted data from included studies and entered these into Review Manager 5 software. We synthesised and presented data in both written and graphical form (forest plots). MAIN RESULTS: We included 10 trials, involving 847 participants. All studies examined CBT programmes provided to children or children and a non-offending parent. Control groups included wait list controls (n = 1) or treatment as usual (n = 9). Treatment as usual was, for the most part, supportive, unstructured psychotherapy. Generally the reporting of studies was poor. Only four studies were judged 'low risk of bias' with regards to sequence generation and only one study was judged 'low risk of bias' in relation to allocation concealment. All studies were judged 'high risk of bias' in relation to the blinding of outcome assessors or personnel; most studies did not report on these, or other issues of bias. Most studies reported results for study completers rather than for those recruited.Depression, post-traumatic stress disorder (PTSD), anxiety and child behaviour problems were the primary outcomes. Data suggest that CBT may have a positive impact on the sequelae of child sexual abuse, but most results were not statistically significant. Strongest evidence for positive effects of CBT appears to be in reducing PTSD and anxiety symptoms, but even in these areas effects tend to be 'moderate' at best. Meta-analysis of data from five studies suggested an average decrease of 1.9 points on the Child Depression Inventory immediately after intervention (95% confidence interval (CI) decrease of 4.0 to increase of 0.4; I(2) = 53%; P value for heterogeneity = 0.08), representing a small to moderate effect size. Data from six studies yielded an average decrease of 0.44 standard deviations on a variety of child post-traumatic stress disorder scales (95% CI 0.16 to 0.73; I(2) = 46%; P value for heterogeneity = 0.10). Combined data from five studies yielded an average decrease of 0.23 standard deviations on various child anxiety scales (95% CI 0.3 to 0.4; I(2) = 0%; P value for heterogeneity = 0.84). No study reported adverse effects. AUTHORS' CONCLUSIONS: The conclusions of this updated review remain the same as those when it was first published. The review confirms the potential of CBT to address the adverse consequences of child sexual abuse, but highlights the limitations of the evidence base and the need for more carefully conducted and better reported trials.