Exploring enablers and barriers to implementing the Transparency and Openness Promotion Guidelines: a theory-based survey of journal editors.

The Transparency and Openness Promotion (TOP) Guidelines provide a framework to help journals develop open science policies. Theories of behaviour change can guide understanding of why journals do (not) implement open science policies and the development of interventions to improve these policies. In this study, we used the Theoretical Domains Framework to survey 88 journal editors on their capability, opportunity and motivation to implement TOP. Likert-scale questions assessed editor support for TOP, and enablers and barriers to implementing TOP. A qualitative question asked editors to provide reflections on their ratings. Most participating editors supported adopting TOP at their journal (71%) and perceived other editors in their discipline to support adopting TOP (57%). Most editors (93%) agreed their roles include maintaining policies that reflect current best practices. However, most editors (74%) did not see implementing TOP as a high priority compared with other editorial responsibilities. Qualitative responses expressed structural barriers to implementing TOP (e.g. lack of time, resources and authority to implement changes) and varying support for TOP depending on study type, open science standard, and level of implementation. We discuss how these findings could inform the development of theoretically guided interventions to increase open science policies, procedures and practices.

Expert-generated standard practice elements for evidence-based home visiting programs using a Delphi process.

BACKGROUND: States, territories, non-profits, and tribes are eligible to obtain federal funding to implement federally endorsed evidence-based home visiting programs. This represents a massive success in translational science, with $400 million a year allocated to these implementation efforts. This legislation also requires that 3% of this annual funding be allocated to tribal entities implementing home visiting in their communities. However, implementing stakeholders face challenges with selecting which program is best for their desired outcomes and context. Moreover, recent reviews have indicated that when implemented in practice and delivered at scale, many evidence-based home visiting programs fail to replicate the retention rates and effects achieved during clinical trials. To inform program implementers and better identify the active ingredients in home visiting programs that drive significant impacts, we aimed to develop an expert derived consensus taxonomy on the elements used in home visiting practice that are essential to priority outcome domains. METHODS: We convened a panel of 16 experts representing researchers, model representatives, and program implementers using a Delphi approach. We first elicited standard practice elements (SPEs) using open-ended inquiry, then compared these elements to behavior change techniques (BCTs) given their general importance in the field of home visiting; and finally rated their importance to 10 outcome domains. RESULTS: Our process identified 48 SPEs derived from the panel, with 83 additional BCTs added based on the literature. Six SPEs, mostly related to home visitor characteristics and skills, were rated essential across all outcome domains. Fifty-three of the 83 BCTs were rated unnecessary across all outcome domains. CONCLUSIONS: This work represents the first step in a consensus-grounded taxonomy of techniques and strategies necessary for home visiting programs and provides a framework for future hypothesis testing and replication studies.

Transparent, Open, and Reproducible Prevention Science.

The field of prevention science aims to understand societal problems, identify effective interventions, and translate scientific evidence into policy and practice. There is growing interest among prevention scientists in the potential for transparency, openness, and reproducibility to facilitate this mission by providing opportunities to align scientific practice with scientific ideals, accelerate scientific discovery, and broaden access to scientific knowledge. The overarching goal of this manuscript is to serve as a primer introducing and providing an overview of open science for prevention researchers. In this paper, we discuss factors motivating interest in transparency and reproducibility, research practices associated with open science, and stakeholders engaged in and impacted by open science reform efforts. In addition, we discuss how and why different types of prevention research could incorporate open science practices, as well as ways that prevention science tools and methods could be leveraged to advance the wider open science movement. To promote further discussion, we conclude with potential reservations and challenges for the field of prevention science to address as it transitions to greater transparency, openness, and reproducibility. Throughout, we identify activities that aim to strengthen the reliability and efficiency of prevention science, facilitate access to its products and outputs, and promote collaborative and inclusive participation in research activities. By embracing principles of transparency, openness, and reproducibility, prevention science can better achieve its mission to advance evidence-based solutions to promote individual and collective well-being.

The Role of Open Science Practices in Scaling Evidence-Based Prevention Programs.

The goal of creating evidence-based programs is to scale them at sufficient breadth to support population-level improvements in critical outcomes. However, this promise is challenging to fulfill. One of the biggest issues for the field is the reduction in effect sizes seen when a program is taken to scale. This paper discusses an economic perspective that identifies the underlying incentives in the research process that lead to scale up problems and to deliver potential solutions to strengthen outcomes at scale. The principles of open science are well aligned with this goal. One prevention program that has begun to scale across the USA is early childhood home visiting. While there is substantial impact research on home visiting, overall average effect size is .10 and a recent national randomized trial found attenuated effect sizes in programs implemented under real-world conditions. The paper concludes with a case study of the relevance of the economic model and open science in developing and scaling evidence-based home visiting. The case study considers how the traditional approach for testing interventions has influenced home visiting's evolution to date and how open science practices could have supported efforts to maintain impacts while scaling home visiting. It concludes by considering how open science can accelerate the refinement and scaling of home visiting interventions going forward, through accelerated translation of research into policy and practice.

Clearinghouse Standards of Evidence on the Transparency, Openness, and Reproducibility of Intervention Evaluations.

Clearinghouses are influential repositories of information on the effectiveness of social interventions. To identify which interventions are "evidence-based," clearinghouses review intervention evaluations using published standards of evidence that focus primarily on internal validity and causal inferences. Open science practices can improve trust in evidence from evaluations on the effectiveness of social interventions. Including open science practices in clearinghouse standards of evidence is one of many efforts that could increase confidence in designations of interventions as "evidence-based." In this study, we examined the policies, procedures, and practices of 10 federal evidence clearinghouses that review preventive interventions-an important and influential subset of all evidence clearinghouses. We found that seven consider at least one open science practice when evaluating interventions: replication (6 of 10 clearinghouses), public availability of results (6), investigator conflicts of interest (3), design and analysis transparency (3), study registration (2), and protocol sharing (1). We did not identify any policies, procedures, or practices related to analysis plan registration, data sharing, code sharing, material sharing, and citation standards. We provide a framework with specific recommendations to help federal and other evidence clearinghouses implement the Transparency and Openness Promotion (TOP) Guidelines. Our proposed "TOP Guidelines for Clearinghouses" includes reporting whether evaluations used open science practices, incorporating open science practices in their standards for receiving "evidence-based" designations, and verifying that evaluations used open science practices. Doing so could increase the trustworthiness of evidence used for policy making and support improvements throughout the evidence ecosystem.

Evaluating implementation of the Transparency and Openness Promotion (TOP) guidelines: the TRUST process for rating journal policies, procedures, and practices.

BACKGROUND: The Transparency and Openness Promotion (TOP) Guidelines describe modular standards that journals can adopt to promote open science. The TOP Factor is a metric to describe the extent to which journals have adopted the TOP Guidelines in their policies. Systematic methods and rating instruments are needed to calculate the TOP Factor. Moreover, implementation of these open science policies depends on journal procedures and practices, for which TOP provides no standards or rating instruments. METHODS: We describe a process for assessing journal policies, procedures, and practices according to the TOP Guidelines. We developed this process as part of the Transparency of Research Underpinning Social Intervention Tiers (TRUST) Initiative to advance open science in the social intervention research ecosystem. We also provide new instruments for rating journal instructions to authors (policies), manuscript submission systems (procedures), and published articles (practices) according to standards in the TOP Guidelines. In addition, we describe how to determine the TOP Factor score for a journal, calculate reliability of journal ratings, and assess coherence among a journal's policies, procedures, and practices. As a demonstration of this process, we describe a protocol for studying approximately 345 influential journals that have published research used to inform evidence-based policy. DISCUSSION: The TRUST Process includes systematic methods and rating instruments for assessing and facilitating implementation of the TOP Guidelines by journals across disciplines. Our study of journals publishing influential social intervention research will provide a comprehensive account of whether these journals have policies, procedures, and practices that are consistent with standards for open science and thereby facilitate the publication of trustworthy findings to inform evidence-based policy. Through this demonstration, we expect to identify ways to refine the TOP Guidelines and the TOP Factor. Refinements could include: improving templates for adoption in journal instructions to authors, manuscript submission systems, and published articles; revising explanatory guidance intended to enhance the use, understanding, and dissemination of the TOP Guidelines; and clarifying the distinctions among different levels of implementation. Research materials are available on the Open Science Framework: https://osf.io/txyr3/ .

Lawmakers' use of scientific evidence can be improved.

Core to the goal of scientific exploration is the opportunity to guide future decision-making. Yet, elected officials often miss opportunities to use science in their policymaking. This work reports on an experiment with the US Congress-evaluating the effects of a randomized, dual-population (i.e., researchers and congressional offices) outreach model for supporting legislative use of research evidence regarding child and family policy issues. In this experiment, we found that congressional offices randomized to the intervention reported greater value of research for understanding issues than the control group following implementation. More research use was also observed in legislation introduced by the intervention group. Further, we found that researchers randomized to the intervention advanced their own policy knowledge and engagement as well as reported benefits for their research following implementation.

Innovative Research Methods to Advance Precision in Home Visiting for More Efficient and Effective Programs.

Home visiting during early childhood can improve a range of outcomes for children and families. As evidence-based models are implemented across the nation, two questions have emerged. First, can home visiting improve outcomes more efficiently? Second, can overall effects be strengthened for specific subgroups of families? For the past several decades, research focused on testing the average effects of home visiting models on short- to long-term outcomes has found small impacts. These effects are not the same for all families. The field needs new evidence produced in new ways to overcome these challenges. In this article, we provide an overview of the evidence in this field, including what works and for whom. Next, we explain precision approaches to various fields and how this approach could be used in home visiting programs. Research on precision home visiting focuses on the ingredients of home visiting models, collaborating with practitioners to identify the ingredients and testing them on near-term outcomes, and using innovative study designs to learn more quickly what works best for which families. We conclude by proposing four pillars of research that will help achieve precision home visiting services.

The role of psychology in evidence-based policymaking: Mapping opportunities for strategic investment in poverty reduction.

Increasing efforts are being undertaken to understand how to improve the use of research evidence in policy settings. In particular, growing efforts to understand the use of research in legislative contexts. Although high-profile examples of psychology's contributions to public policy exist-particularly around antipoverty legislation-little systematic review has quantified how the field has informed federal policy across time. Recognizing the importance of exploring psychology's use in policymaking, we provide an overview of psychology's presence in federal antipoverty legislation over the last 2 decades by reviewing the over 6,000 antipoverty bills introduced to the U.S. Congress since 1993 for mentions of psychology. Further, to explore how psychology's contributions are related to policymakers' attributions about the causes of poverty, their public statements and voting behavior is considered. Key gaps in our scientific knowledge for informing poverty-related policy are identified. Opportunities to enhance the relevance of psychology in poverty reduction efforts, including the evidence-based policy movement, are described. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Scaling up Evidence-Based Interventions in US Public Systems to Prevent Behavioral Health Problems: Challenges and Opportunities.

A number of programs, policies, and practices have been tested using rigorous scientific methods and shown to prevent behavioral health problems (Catalano et al., Lancet 379:1653-1664, 2012; National Research Council and Institute of Medicine, 2009). Yet these evidence-based interventions (EBIs) are not widely used in public systems, and they have limited reach (Glasgow et al., American Journal of Public Health 102:1274-1281, 2012; National Research Council and Institute of Medicine 2009; Prinz and Sanders, Clinical Psychology Review 27:739-749, 2007). To address this challenge and improve public health and well-being at a population level, the Society for Prevention Research (SPR) formed the Mapping Advances in Prevention Science (MAPS) IV Translation Research Task Force, which considered ways to scale up EBIs in five public systems: behavioral health, child welfare, education, juvenile justice, and public health. After reviewing other efforts to scale up EBIs in public systems, a common set of factors were identified as affecting scale-up in all five systems. The most important factor was the degree to which these systems enacted public policies (i.e., statutes, regulations, and guidance) requiring or recommending EBIs and provided public funds for EBIs. Across systems, other facilitators of scale-up were creating EBIs that are ready for scale-up, public awareness of and support for EBIs, community engagement and capacity to implement EBIs, leadership support for EBIs, a skilled workforce capable of delivering EBIs, and data monitoring and evaluation capacity. It was concluded that the following actions are needed to significantly increase EBI scale-up in public systems: (1) provide more public policies and funding to support the creation, testing, and scaling up of EBIs; (2) develop and evaluate specific frameworks that address systems level barriers impeding EBI scale-up; and (3) promote public support for EBIs, community capacity to implement EBIs at scale, and partnerships between community stakeholders, policy makers, practitioners, and scientists within and across systems.

Principles of Precision Prevention Science for Improving Recruitment and Retention of Participants.

Precision medicine and precision public health focus on identifying and providing the right intervention to the right population at the right time. Expanding on the concept, precision prevention science could allow the field to examine prevention programs to identify ways to make them more efficient and effective at scale, including addressing issues related to engagement and retention of participants. Research to date on engagement and retention has often focused on demographics and risk factors. The current paper proposes using McCurdy and Daro (Family Relations, 50, 113-121, 2001) model that posits a complex mixture of individual, provider, program, and community-level factors synergistically affect enrollment, engagement, and retention. The paper concludes recommending the use of research-practice partnerships and innovative, rapid cycle methods to design and improve prevention programs related to participant engagement and retention at scale.

Behavior Problems in Elementary School among Low-Income Males: The Role of Teacher-Child Relationships.

The present study identified trajectories of teacher-child relationship conflict and closeness from first through sixth grades, and associations between these trajectories and externalizing and internalizing behaviors at age 11 among low-income, urban males (N = 262). There were three main findings. Nagin cluster analyses indicated five trajectories for conflict with all children evidencing increases in conflict, and four trajectories for closeness with all children demonstrating decreases in closeness. Trajectories with higher levels of conflict and lower levels of closeness were associated with higher levels of externalizing and internalizing behavior problems at age 11. Moreover, conflictual teacher-child relationships exacerbated the effects of externalizing and internalizing behavior problems in early childhood; children with conflictual teacher-child relationships had higher levels of behavior problems in middle childhood relative to children with low conflictual teacher-child relationships. Implications of targeting teacher-child relationships as interventions to help prevent behavior problems are discussed.

The Intersection Between Prevention Science and Evidence-Based Policy: How the SPR Evidence Standards Support Human Services Prevention Programs.

The revised Society for Prevention Research (SPR) standards of evidence are an exciting advance in the field of prevention science. We appreciate the committee's vision that the standards represent goals to aspire to rather than a set of benchmarks for where prevention science is currently. The discussion about the standards highlights how much has changed in the field over the last 10 years and as knowledge, theory, and methods continue to advance, the new standards push the field toward increasing rigor and relevance. This commentary discusses how the revised standards support work of translating high-quality evaluations to support evidence-based policy and work supporting evidence-based programs' ability to implement at scale. The commentary ends by raising two areas, generating evidence at scale and transparency of research, as additional areas for consideration in future standards.

Supporting replication and scale-up of evidence-based home visiting programs: assessing the implementation knowledge base.

In recent years, researchers, policymakers, and practitioners have expressed a growing interest in the use of interventions with scientific evidence of effectiveness. Reproducing positive effects shown in research, however, requires more than simply adopting an evidence-based program. There is growing recognition across disciplines of the importance of implementation research to guide adoption, replication, and scale-up of evidence-based interventions. We evaluate the state of the knowledge base supporting replication and scale-up of evidence-based programs by reviewing information on implementation included in the research literature on 22 home visiting programs that have or are building an evidence base. We used the Interactive Systems Framework for Dissemination and Implementation to assess programs.

Overview of the federal home visiting program.

On March 23, 2010, the President signed into law the Affordable Care Act (Public Law 111-148), which included an amendment of Title V of the Social Security Act authorizing the creation of the Maternal, Infant, and Early Childhood Home Visiting (MIECHV) program. Authorized and funded at $1.5 billion for 5 years, the MIECHV represents a large investment in health and development outcomes for at-risk children through evidence-based home visiting programs. The MIECHV presents unprecedented opportunities to integrate early childhood services systems, not only on the federal level but also within states and local communities. The MIECHV is funded in escalating amounts over 5-year period authorized, as follows: $100 million in fiscal year (FY) 2010, $250 million in FY 2011, $350 million in FY 2012, $400 million in FY 2013, and $400 million in FY 2014. Most of the funding is being provided to states and territories to provide home visiting services in their at-risk communities. In addition, the legislation included a 3% set-aside for tribes, tribal organizations, and urban Indian organizations and a 3% set-aside for research and evaluation. This investment has spurred the creation of more comprehensive and coordinated early childhood service systems across the United States. This article provides an overview of the MIECHV program, including descriptions of the various requirements under the Affordable Care Act. These include partnering with states to provide evidence-based home visiting services to at-risk families, working with tribal communities to implement culturally competent home visiting programs, and developing a mechanism to systematically review the evidence of effectiveness for home visiting program models and to conduct a national evaluation of the MIECHV program.

Effectiveness of home visiting in improving child health and reducing child maltreatment.

BACKGROUND AND OBJECTIVE: The Patient Protection and Affordable Care Act established the Maternal, Infant, and Early Childhood Home Visiting Program, which provides $1.5 billion to states over 5 years for home visiting program models serving at-risk pregnant women and children from birth to age 5. The act stipulates that 75% of the funds must be used for programs with evidence of effectiveness based on rigorous evaluation research. Home Visiting Evidence of Effectiveness reviewed the home visiting research literature and provided an assessment of the evidence of effectiveness for program models that serve families with pregnant women and children from birth to age 5. METHODS: Home Visiting Evidence of Effectiveness included a systematic search and screening process, a review of the research quality, and an assessment of program effectiveness. Reviewers rated studies' capacity to provide unbiased estimates of program impacts and determined whether a program met the Department of Health and Human Services' criteria for an evidence-based model. RESULTS: As of July 2012, 32 models were reviewed, of which 12 met the Department of Health and Human Services criteria. Most of these models were shown to have favorable effects on child development. Other common favorable effects included health care usage and reductions in child maltreatment. Less common were favorable effects on birth outcomes. CONCLUSIONS: Home visiting is a promising way to serve families who may be difficult to engage in supportive services. Existing rigorous research indicates that home visiting has the potential for positive results among high-risk families, particularly on health care usage and child development.

Directions in implementation research methods for behavioral and social science.

There is a growing interest, by researchers, policymakers, and practitioners, in evidence-based policy and practice. As a result, more dollars are being invested in program evaluation in order to establish "what works," and in some cases, funding is specifically tied to those programs found to be effective. However, reproducing positive effects found in research requires more than simply adopting an evidence-based program. Implementation research can provide guidance on which components of an intervention matter most for program impacts and how implementation components can best be implemented. However, while the body of rigorous research on effective practices continues to grow, research on implementation lags behind. To address these issues, the Administration for Children and Families and federal partners convened a roundtable meeting entitled, Improving Implementation Research Methods for Behavioral and Social Science, in the fall of 2010. This special section of the Journal of Behavioral Health Services & Research includes papers from the roundtable and highlights the role implementation science can play in shedding light on the difficult task of taking evidence-based practices to scale.

Behavior problems in late childhood: the roles of early maternal attachment and teacher-child relationship trajectories.

The purposes of the current study were: (1) to examine the roles of early maternal attachment relationships and teacher-child relationships during childhood for externalizing and internalizing behaviors in late childhood, and (2) to investigate teacher-child relationships, as well as externalizing and internalizing behaviors in early childhood as possible mechanisms linking early maternal attachment relationships to behavior problems in late childhood. Longitudinal data from the National Institute of Child Health and Human Development Early Child Care Research Network Study of Early Child Care and Youth Development (N = 1140 mothers and children) were used in this investigation. There were three main findings. First, insecure/other maternal attachment relationships in early childhood (i.e., 36 months) were associated with externalizing and internalizing behaviors in late childhood (Grade 5). Second, elevated levels of teacher-child conflict during childhood were associated with externalizing behaviors in late childhood whereas low levels of teacher-child closeness were associated with internalizing behaviors. Third, the effects of insecure/other attachment on externalizing and internalizing behaviors in late childhood were mediated through teacher-child relationships during childhood and early externalizing and internalizing behaviors. Implications for attachment theory are discussed.