Reporting all results efficiently: A RARE proposal to open up the file drawer.

While the social sciences have made impressive progress in adopting transparent research practices that facilitate verification, replication, and reuse of materials, the problem of publication bias persists. Bias on the part of peer reviewers and journal editors, as well as the use of outdated research practices by authors, continues to skew literature toward statistically significant effects, many of which may be false positives. To mitigate this bias, we propose a framework to enable authors to report all results efficiently (RARE), with an initial focus on experimental and other prospective empirical social science research that utilizes public study registries. This framework depicts an integrated system that leverages the capacities of existing infrastructure in the form of public registries, institutional review boards, journals, and granting agencies, as well as investigators themselves, to efficiently incentivize full reporting and thereby, improve confidence in social science findings. In addition to increasing access to the results of scientific endeavors, a well-coordinated research ecosystem can prevent scholars from wasting time investigating the same questions in ways that have not worked in the past and reduce wasted funds on the part of granting agencies.

Open Science Standards at Journals that Inform Evidence-Based Policy.

Evidence-based policy uses intervention research to inform consequential decisions about resource allocation. Research findings are often published in peer-reviewed journals. Because detrimental research practices associated with closed science are common, journal articles report more false-positives and exaggerated effect sizes than would be desirable. Journal implementation of standards that promote open science-such as the transparency and openness promotion (TOP) guidelines-could reduce detrimental research practices and improve the trustworthiness of research evidence on intervention effectiveness. We evaluated TOP implementation at 339 peer-reviewed journals that have been used to identify evidence-based interventions for policymaking and programmatic decisions. Each of ten open science standards in TOP was not implemented in most journals' policies (instructions to authors), procedures (manuscript submission systems), or practices (published articles). Journals implementing at least one standard typically encouraged, but did not require, an open science practice. We discuss why and how journals could improve implementation of open science standards to safeguard evidence-based policy.

Clearinghouse Standards of Evidence on the Transparency, Openness, and Reproducibility of Intervention Evaluations.

Clearinghouses are influential repositories of information on the effectiveness of social interventions. To identify which interventions are "evidence-based," clearinghouses review intervention evaluations using published standards of evidence that focus primarily on internal validity and causal inferences. Open science practices can improve trust in evidence from evaluations on the effectiveness of social interventions. Including open science practices in clearinghouse standards of evidence is one of many efforts that could increase confidence in designations of interventions as "evidence-based." In this study, we examined the policies, procedures, and practices of 10 federal evidence clearinghouses that review preventive interventions-an important and influential subset of all evidence clearinghouses. We found that seven consider at least one open science practice when evaluating interventions: replication (6 of 10 clearinghouses), public availability of results (6), investigator conflicts of interest (3), design and analysis transparency (3), study registration (2), and protocol sharing (1). We did not identify any policies, procedures, or practices related to analysis plan registration, data sharing, code sharing, material sharing, and citation standards. We provide a framework with specific recommendations to help federal and other evidence clearinghouses implement the Transparency and Openness Promotion (TOP) Guidelines. Our proposed "TOP Guidelines for Clearinghouses" includes reporting whether evaluations used open science practices, incorporating open science practices in their standards for receiving "evidence-based" designations, and verifying that evaluations used open science practices. Doing so could increase the trustworthiness of evidence used for policy making and support improvements throughout the evidence ecosystem.

Modern Meta-Analytic Methods in Prevention Science: Introduction to the Special Issue.

Meta-analyses that statistically synthesize evidence from multiple research studies can play an important role in advancing evidence-informed prevention science. When done in the context of a well-conducted systematic review, meta-analysis is a powerful tool for synthesizing evidence and exploring research questions that are difficult to address in individual studies, such as the association of individual study limitations on intervention effect estimates, replicability of empirical findings, and variation of effect estimates across populations and settings. Alongside the rapid growth in the number of published reviews and meta-analyses, there has been a parallel growth in the development of meta-analytic techniques to handle the increasingly complex types of questions and types of evidence relevant to prevention science. Despite this rapid evolution of meta-analytic techniques and approaches, there is still a lag between the development of new techniques and their uptake by researchers in the field. This paper serves as a brief introduction to this special issue of Prevention Science, entitled "Modern Meta-Analytic Methods in Prevention Science," which highlights recent developments in meta-analytic methods and demonstrates their application to prevention research. This special issue makes an important contribution to the field by ensuring these methodological advances are widely accessible to prevention science researchers, thereby improving their uptake and utilization, and ultimately improving the utility and rigor of research syntheses for informing evidence-based decision making in prevention.

Transparent, Open, and Reproducible Prevention Science.

The field of prevention science aims to understand societal problems, identify effective interventions, and translate scientific evidence into policy and practice. There is growing interest among prevention scientists in the potential for transparency, openness, and reproducibility to facilitate this mission by providing opportunities to align scientific practice with scientific ideals, accelerate scientific discovery, and broaden access to scientific knowledge. The overarching goal of this manuscript is to serve as a primer introducing and providing an overview of open science for prevention researchers. In this paper, we discuss factors motivating interest in transparency and reproducibility, research practices associated with open science, and stakeholders engaged in and impacted by open science reform efforts. In addition, we discuss how and why different types of prevention research could incorporate open science practices, as well as ways that prevention science tools and methods could be leveraged to advance the wider open science movement. To promote further discussion, we conclude with potential reservations and challenges for the field of prevention science to address as it transitions to greater transparency, openness, and reproducibility. Throughout, we identify activities that aim to strengthen the reliability and efficiency of prevention science, facilitate access to its products and outputs, and promote collaborative and inclusive participation in research activities. By embracing principles of transparency, openness, and reproducibility, prevention science can better achieve its mission to advance evidence-based solutions to promote individual and collective well-being.

Expert views on state-level naloxone access laws: a qualitative analysis of an online modified-Delphi process.

BACKGROUND: Expanding availability to naloxone is a core harm reduction strategy in efforts to address the opioid epidemic. In the US, state-level legislation is a prominent mechanism to expand naloxone availability through various venues, such as community pharmacies. This qualitative study aimed to identify and summarize the views of experts on state-level naloxone access laws. METHODS: We conducted a three-round modified-Delphi process using the online ExpertLens platform. Participants included 46 key stakeholders representing various groups (advocates, healthcare providers, human/social service practitioners, policymakers, and researchers) with expertise naloxone access laws. Participants commented on the effectiveness and implementability of 15 state-level naloxone access laws (NALs). We thematically analyzed participant comments to summarize views on NALs overall and specific types of NAL. RESULTS: Participants commented that the effectiveness of NALs in reducing opioid-related mortality depends on their ability to make sustained, significant impacts on population-level naloxone availability. Participants generally believed that increased naloxone availability does not have appreciable negative impacts on the prevalence of opioid misuse, opioid use disorder (OUD), and non-fatal opioid overdoses. Implementation barriers include stigma among the general public, affordability of naloxone, and reliance on an inequitable healthcare system. CONCLUSIONS: Experts believe NALs that significantly increase naloxone access are associated with less overdose mortality without risking substantial unintended public health outcomes. To maximize impacts, high-value NALs should explicitly counter existing healthcare system inequities, address stigmatization of opioid use and naloxone, maintain reasonable prices for purchasing naloxone, and target settings beyond community pharmacies to distribute naloxone.

Clinical interventions for adults with comorbid alcohol use and depressive disorders: A systematic review and network meta-analysis.

BACKGROUND: Uncertainty remains regarding the effectiveness of treatments for patients diagnosed with both an alcohol use disorder (AUD) and depressive disorder. This study aimed to compare the effectiveness of clinical interventions for improving symptoms of adults with co-occurring AUDs and depressive disorders. METHODS AND FINDINGS: We searched CINAHL, ClinicalTrials.gov, Cochrane Central Register of Controlled Trials, Cochrane Database of Systematic Reviews, Excerpta Medica Database, International Clinical Trials Registry Platform (ICTRP), PubMed, PsycINFO, and Web of Science from inception to December 2020. We included randomized controlled trials (RCTs) evaluating clinical interventions for adults with co-occurring AUDs and depressive disorders. Two independent reviewers extracted study-level information and outcome data. We assessed risk of bias using the Cochrane Risk of Bias tool, used frequentist random effects models for network meta-analyses, and rated our confidence in effect estimates using the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) approach. Primary outcomes were remission from depression and alcohol use. Secondary outcomes were depressive symptoms, alcohol use, heavy drinking, health-related quality of life, functional status, and adverse events. We used standardized mean differences (SMDs) for continuous outcomes and odds ratios (ORs) for dichotomous outcomes to estimate intervention effects. Overall, 36 RCTs with 2,729 participants evaluated 14 pharmacological and 4 psychological interventions adjunctive to treatment as usual (TAU). Studies were published from 1971 to 2019, conducted in 13 countries, and had a median sample size of 50 participants (range: 14 to 350 participants). We have very low confidence in all estimates of intervention effects on our primary outcomes (i.e., remission from depression and remission from alcohol use). We have moderate confidence that cognitive behavioral therapies (CBTs) demonstrated greater benefit than no additional treatment (SMD = -0.84; 95% confidence interval [CI], -1.05 to -0.63; p < 0.001) for depressive symptoms and low confidence (SMD = -0.25; 95% CI, -0.47 to -0.04; p = 0.021) for alcohol use. We have low confidence that tricyclic antidepressants (TCAs) demonstrated greater benefit than placebo (SMD = -0.37; 95% CI, -0.72 to -0.02, p = 0.038) for depressive symptoms. Compared with placebo, we have moderate confidence that selective serotonin reuptake inhibitors (SSRIs) demonstrated greater benefit for functional status (SMD = -0.92; 95% CI, -1.36 to -0.47, p < 0.001) and low confidence for alcohol use (SMD = -0.30; 95% CI, -0.59 to -0.02, p = 0.039). However, we have moderate confidence that patients receiving SSRIs also were more likely to experience an adverse event (OR = 2.20; 95% CI, 0.94 to 5.16, p = 0.07). We have very low confidence in all other effect estimates, and we did not have high confidence in any effect estimates. Limitations include the sparsity of evidence on intervention effects over the long term, risks of attrition bias, and heterogeneous definitions of adverse events in the evidence base. CONCLUSIONS: We are very uncertain about the existence (or not) of any non-null effects for our primary outcomes of remission from depression and remission from alcohol use. The available evidence does suggest that CBTs likely reduced, and TCAs may have resulted in a slight reduction of depressive symptoms. SSRIs likely increased functional status, and SSRIs and CBTs may have resulted in a slight reduction of alcohol use. However, patients receiving SSRIs also likely had an increased risk of experiencing an adverse event. In addition, these conclusions only apply to postintervention and are not against active comparators, limiting the understanding of the efficacy of interventions in the long term as well as the comparative effectiveness of active treatments. As we did not have high confidence in any outcomes, additional studies are warranted to provide more conclusive evidence.

The Challenges of Conducting Intrastate Policy Surveillance: A Methods Note on County and City Laws in Indiana.

Policy surveillance is critical in examining the ways law functions as a structural and social determinant of health. To date, little policy surveillance research has focused on examining intrastate variations in the structure and health impact of laws. Intrastate policy surveillance poses unique methodological challenges because of the complex legal architecture within states and inefficient curation of local laws.We discuss our experience with these intrastate policy surveillance challenges in Indiana, a state with 92 counties and several populous cities, a complicated history of home rule, systemically underfunded local governments, and variations in demography, geography, and technology adoption. In our case study, we expended significant time and resources to obtain county and city ordinances through online code libraries, jurisdiction Web sites, and (most notably) visits to offices to scan documents ourselves.A concerted effort is needed to ensure that local laws of all kinds are stored online in organized, searchable, and open access systems. Such an effort is vital to achieve the aspirational goals of policy surveillance at the intrastate level.

Practical Guidance for Involving Stakeholders in Health Research.

Stakeholder engagement is increasingly common in health research, with protocols for engaging multiple stakeholder groups becoming normative in patient-centered outcomes research. Previous work has focused on identifying relevant stakeholder groups with whom to work and on working with stakeholders in evidence implementation. This paper draws on the expertise of a team from four countries-Canada, Australia, the UK, and the USA-to provide researchers with practical guidance for carrying out multi-stakeholder-engaged projects: we present a list of questions to assist in selecting appropriate roles and modes of engagement; we introduce a matrix to help summarize engagement activities; and we provide a list of online resources. This guidance, matrix, and list of resources can assist researchers to consider more systematically which stakeholder groups to involve, in what study roles, and by what modes of engagement. By documenting how stakeholders are paired up with specific roles, the matrix also provides a potential structure for evaluating the impact of stakeholder engagement.

Comparative Effectiveness of Brief Alcohol Interventions for College Students: Results from a Network Meta-Analysis.

Late adolescence is a time of increased drinking, and alcohol plays a predominant role in college social experiences. Colleges seeking to prevent students' hazardous drinking may elect to implement brief alcohol interventions (BAIs). However, numerous manualized BAIs exist, so an important question remains regarding the comparative effectiveness of these different types of BAIs for college students. This study uses network meta-analyses (NMA) to compare seven manualized BAIs for reducing problematic alcohol use among college students. We systematically searched multiple sources for literature, and we screened studies and extracted data in duplicate. For the quantitative synthesis, we employed a random-effects frequentist NMA to determine the effectiveness of different BAIs compared to controls and estimated the relative effectiveness ranking of each BAI. A systematic literature search resulted in 52 included studies: On average, 58% of participants were male, 75% were binge drinkers, and 20% were fraternity/sorority-affiliated students. Consistency models demonstrated that BASICS was consistently effective in reducing students' problematic alcohol use (ES range: g = - 0.23, 95%CI [- 0.36, - 0.16] to g = - 0.36, 95% CI [- 0.55, - 0.18]), but AlcoholEDU (g = - 0.13, 95%CI [- 0.22, - 0.04]), e-CHUG (g = - 0.35, 95%CI [- 0.45, - 0.05]), and THRIVE (g = - 0.47, 95%CI [- 0.60, - 0.33]) were also effective for some outcomes. Intervention rankings indicated that BASICS, THRIVE, and AlcoholEDU hold the most promise for future trials. Several BAIs appear effective for college students. BASICS was the most effective but is resource intensive and may be better suited for higher risk students; THRIVE and e-CHUG are less resource intensive and show promise for universal prevention efforts.

Meta-Analysis of Complex Interventions.

Meta-analysis is a prominent method for estimating the effects of public health interventions, yet these interventions are often complex in ways that pose challenges to using conventional meta-analytic methods. This article discusses meta-analytic techniques that can be used in research syntheses on the effects of complex public health interventions. We first introduce the use of complexity frameworks to conceptualize public health interventions. We then present a menu of meta-analytic procedures for addressing various sources of complexity when answering questions about the effects of public health interventions in research syntheses. We conclude with a review of important practices and key resources for conducting meta-analyses on complex interventions, as well as future directions for research synthesis more generally. Overall, we argue that it is possible to conduct meaningful quantitative syntheses of research on the effects of public health interventions, though these meta-analyses may require the use of advanced techniques to properly consider and attend to issues of complexity.

Research Review: Cognitive bias modification of interpretations in youth and its effect on anxiety: a meta-analysis.

BACKGROUND: Emerging evidence suggests that cognitive bias modification of interpretations (CBM-I) is effective in altering interpretation biases and reducing anxiety in adults. Less is known about the impact of CBM-I in young people, but some recent findings, including a meta-analysis of combined cognitive bias modification of interpretation and attention techniques, have cast doubt on its clinical utility. Given the current debate, this meta-analysis sought to establish the independent effects of CBM-I on interpretations biases and anxiety in youth. METHODS: Studies were identified through a systematic literature search of PsycINFO, Ovid MEDLINE, PsycARTICLES, Web of Science and EMBASE between January 1992 and March 2017. Eligible studies aimed to target interpretation biases; did not combine CBM-I with another intervention; included a control condition; randomly allocated participants to conditions; assessed interpretation bias and/or anxiety as an outcome; included individuals up to age 18; and did not present previously reported data. Reference lists of included articles were checked for further eligible studies, and authors were contacted for unpublished data. RESULTS: We identified 26 studies meeting eligibility criteria that included in the meta-analysis. CBM-I had moderate effects on negative and positive interpretations (g = -0.70 and g = -0.52, respectively) and a small but significant effect on anxiety assessed after training (g = -0.17) and after a stressor (g = -0.34). No significant moderators were identified. CONCLUSIONS: In contrast to previous meta-analytic findings, our results indicate that CBM-I has potential but weak anxiolytic effects in youth. Our findings suggest that it may be premature to disregard the potential value of CBM-I research and further research in this field is warranted.

Acupuncture for the Treatment of Adults with Posttraumatic Stress Disorder: A Systematic Review and Meta-Analysis.

Acupuncture has been suggested as a treatment for posttraumatic stress disorder (PTSD), yet its clinical effects are unclear. This review aims to estimate effects of acupuncture on PTSD symptoms, depressive symptoms, anxiety symptoms, and sleep quality for adults with PTSD. We searched 10 databases in January 2016 to identify eligible randomized controlled trials (RCTs). We performed random effects meta-analyses and examined quality of the body of evidence (QoE) using the GRADE approach to rate confidence in meta-analytic effect estimates. Seven RCTs with 709 participants met inclusion criteria. We identified very low QoE indicating significant differences favoring acupuncture (versus any comparator) at post-intervention on PTSD symptoms (standardized mean difference [SMD] = -0.80, 95% confidence interval [CI] [-1.59, -0.01], 6 RCTs), and low QoE at longer follow-up on PTSD (SMD = -0.46, 95% CI [-0.85, -0.06], 4 RCTs) and depressive symptoms (SMD = -0.56; 95% CI [-0.88, -0.23], 4 RCTs). No significant differences were observed between acupuncture and comparators at post-intervention for depressive symptoms (SMD = -0.58, 95% CI [-1.18, 0.01], 6 RCTs, very low QoE), anxiety symptoms (SMD = -0.82, 95% CI [-2.16, 0.53], 4 RCTs, very low QoE), and sleep quality (SMD = -0.46, 95% CI [-3.95, 3.03], 2 RCTs, low QoE). Safety data (7 RCTs) suggest little risk of serious adverse events, though some participants experienced minor/moderate pain, superficial bleeding, and hematoma at needle insertion sites. To increase confidence in findings, sufficiently powered replication trials are needed that measure all relevant clinical outcomes and dedicate study resources to minimizing participant attrition.

Prevalence of, risk factors for, and consequences of posttraumatic stress disorder and other mental health problems in military populations deployed to Iraq and Afghanistan.

This review summarizes the epidemiology of posttraumatic stress disorder (PTSD) and related mental health problems among persons who served in the armed forces during the Iraq and Afghanistan conflicts, as reflected in the literature published between 2009 and 2014. One-hundred and sixteen research studies are reviewed, most of which are among non-treatment-seeking US service members or treatment-seeking US veterans. Evidence is provided for demographic, military, and deployment-related risk factors for PTSD, though most derive from cross-sectional studies and few control for combat exposure, which is a primary risk factor for mental health problems in this cohort. Evidence is also provided linking PTSD with outcomes in the following domains: physical health, suicide, housing and homelessness, employment and economic well-being, social well-being, and aggression, violence, and criminality. Also included is evidence about the prevalence of mental health service use in this cohort. In many instances, the current suite of studies replicates findings observed in civilian samples, but new findings emerge of relevance to both military and civilian populations, such as the link between PTSD and suicide. Future research should make effort to control for combat exposure and use longitudinal study designs; promising areas for investigation are in non-treatment-seeking samples of US veterans and the role of social support in preventing or mitigating mental health problems in this group.

ROMPER: The RAND/USC OPTIC Method for Policy Expert Ratings.

We developed an expert panel approach for identifying expert views on the effectiveness and implementability of population-level policy interventions. ROMPER-the RAND/USC OPTIC Method for Policy Expert Ratings-involves an online, three-round, modified-Delphi process:•Experts rate and comment on policies according to domains of the Grading of Recommendations Assessment, Development and Evaluation (GRADE) Evidence-to-Decision framework.•To identify consensus on policy effectiveness and implementability, expert ratings are analyzed using the Inter-Percentile Range Adjusted for Symmetry (IPRAS) technique from the RAND/UCLA Appropriateness Method and visualized using a forest plot. To explain consensus, expert comments are analyzed using reflexive thematic analysis and reported following the Standards for Reporting Qualitative Research.•To provide actionable information for decisionmakers, each policy is summarized in a "Policy Profile" adapted from GRADEPro Evidence-to-Decision tables.We validated ROMPER in two studies that successfully recruited the targeted sample size, retained experts through all three rounds, and examined consensus on which policies are (not) effective and implementable. ROMPER protocols, materials, data, and code are openly available on the Open Science Framework with Creative Commons licensing for replication and reuse. ROMPER provides a validated, replicable, open access approach for eliciting expert views on both policy effectiveness and implementability-and for summarizing (lack of) consensus specifically for policymakers.

Using the Delphi Process to Prioritize an Agenda for Care Transition Research for Patients With Substance Use Disorders.

Medical hospitalizations are increasingly recognized as important opportunities to engage individuals with substance use disorders (SUD) and offer treatment. While a growing number of hospitals have instituted interventions to support the provision of SUD care during medical admissions, post-hospitalization transitions of care remain a challenge for patients and clinicians and an understudied area of SUD care. Evidence is lacking on the most effective and feasible models of care to improve post-hospitalization care transitions for people with SUD. In the absence of strong empirical evidence to guide practice and policy, consensus-based research methods such as the Delphi process can play an important role in efficiently prioritizing existing models of care for future study and implementation. We conducted a Delphi study that convened a group of 25 national interdisciplinary experts with direct clinical experience facilitating post-hospitalization care transitions for people with SUD. Our panelists rated 10 existing care transition models according to anticipated effectiveness and facility of implementation based on the GRADE Evidence to Decision framework. Qualitative data on each care model were also gathered through comments and an online moderated discussion board. Our results help establish a hierarchy of SUD care transition models to inform future study and program development.