"What We Have Here, Is a Failure to [Replicate]": Ways to Solve a Replication Crisis in Implementation Science.

Adapting the classic line from the 1967 film Cool Hand Luke, the title is meant to convey that implementation science (IS), like other fields, has not been embracing replication studies, which is a key component to the open science movement. The purpose of this article is to review what is known about replication of implementation trials and identify the gaps and next steps to continue increasing the transparency, openness, and replicability of implementation research. After presenting an overview of study replication and how it is a key component of open science, the article will examine how replication of implementation studies has (or more accurately has not) been approached in IS. As will be discussed, replication in IS shares some challenges with studies that attempt to replicate interventions, but also presents unique challenges. This article discusses different types of replications (e.g., direct vs. conceptual) and how they can benefit the field of IS. The article then presents a specific example of an implementation strategy called Getting To Outcomes© to describe how to design a replication study and interpret the results. The article ends with multiple options implementation scientists could consider to improve the likelihood and quality of replication studies. The discussion also envisions how implementation science can enable researchers and practitioners to work together in real-world contexts to encourage wide replication of implementation studies and advance the goal of improving public health.

Cost analysis of a randomized trial of Getting to Outcomes implementation support for a teen pregnancy prevention program offered in Boys and Girls Clubs in Alabama and Georgia.

Implementation support can improve outcomes of evidence-based programs (EBP) for adolescents, but with a cost. To assist in determining whether this cost is worthwhile, this study estimated the cost of adding Getting To Outcomes© (GTO) implementation support to a teen pregnancy and sexually transmitted infection prevention EBP called Making Proud Choices (MPC) in 32 Boys and Girls Clubs (BGCs) in Alabama and Georgia. Enhancing Quality Interventions Promoting Healthy Sexuality (EQUIPS) was a 2-year, cluster-randomized controlled trial comparing MPC with MPC + GTO. We used micro-costing to estimate costs and captured MPC and GTO time from activity logs completed by GTO staff. Key resource use and cost components were compared between the randomized groups, years, and states (to capture different community site circumstances) using 2-sample t tests. There were no significant differences between randomized groups in attendees per site, resource use, or costs for either year. However, there were significant differences between states. Adding GTO to MPC increased the societal costs per attendee from $67 to $144 (2015 US dollars) in Georgia and from $106 to $314 in Alabama. The higher Alabama cost was due to longer travel distances and to more BGC staff time spent on GTO in that state. GTO also improved adherence, classroom delivery, and condom-use intentions more in Alabama youth. Thus, Alabama's GTO-related BGC staff time costs may be better estimates of effective GTO. If teen childbearing costs taxpayers approximately $20,000 per teen birth, adding GTO to MPC would be worthwhile to society if it prevented one more teen birth per 140 attendees than MPC alone.Trial registration. ClinicalTrials.gov , NCT01818791. Registered March 26, 2013, https://clinicaltrials.gov/ct2/show/NCT01818791?term=NCT01818791&draw=2&rank=1.

Sustaining an Evidence-Based Program Over Time: Moderators of Sustainability and the Role of the Getting to Outcomes® Implementation Support Intervention.

Problematic rates of alcohol, e-cigarette, and other drug use among US adolescents highlight the need for effective implementation of evidence-based programs (EBPs), yet schools and community organizations have great difficulty implementing and sustaining EBPs. Although a growing number of studies show that implementation support interventions can improve EBP implementation, the literature on how to improve sustainability through implementation support is limited. This randomized controlled trial advances the literature by testing the effects of one such implementation intervention-Getting To Outcomes (GTO)-on sustainability of CHOICE, an after-school EBP for preventing substance use among middle-school students. CHOICE implementation was tracked for 2 years after GTO support ended across 29 Boys and Girls Club sites in the greater Los Angeles area. Predictors of sustainability were identified for a set of key tasks targeted by the GTO approach (e.g., goal setting, evaluation, collectively called "GTO performance") and for CHOICE fidelity using a series of path models. One year after GTO support ended, we found no differences between GTO and control sites on CHOICE fidelity. GTO performance was also similar between groups; however, GTO sites were superior in conducting evaluation. Better GTO performance predicted better CHOICE fidelity. Two years after GTO support ended, GTO sites were significantly more likely to sustain CHOICE implementation when compared with control sites. This study suggests that using an implementation support intervention like GTO can help low-resource settings continue to sustain their EBP implementation to help them get the most out of their investment. ClinicalTrials.gov Identifier: NCT02135991.

Cost Analysis of a Randomized Trial of Getting to Outcomes Implementation Support of CHOICE in Boys and Girls Clubs in Southern California.

Costs of supporting prevention program implementation are not well known. This study estimates the societal costs of implementing CHOICE, a voluntary after-school alcohol and other drug prevention program for adolescents, in Boys and Girls Clubs (BGCs) across Southern California with and without an implementation support system called Getting To Outcomes© (GTO). This article uses micro-costing methods to estimate the cost of the CHOICE program and GTO support. Labor and expense data were obtained from logs kept by the BGC staff and by the GTO technical assistance (TA) staff, and staff time was valued based on Bureau of Labor Statistics estimates. From the societal perspective, the cost of implementing CHOICE at BGCs over the 2-year study period was $27 per attendee when CHOICE was offered by itself (all costs incurred by the BGCs) and $177 per attendee when CHOICE was offered with GTO implementation support ($67 cost to the BGCs; $110 to the entity funding GTO). These results were most sensitive to assumptions as to the number of times CHOICE was offered per year. Adding GTO implementation support to CHOICE increased the cost per attendee by approximately $150. For this additional cost, there was evidence that the CHOICE program was offered with more fidelity and offered more often after the 2-year intervention ended. If the long-term benefits of this better and continued implementation are found to exceed these additional costs, GTO could be an attractive structure to support evidence-based substance misuse prevention programs. Trial Registration. This project is registered at ClinicalTrials.gov with number NCT02135991 (URL: https://clinicaltrials.gov/show/NCT02135991). The trial was registered May 12, 2014.

Influence of an Implementation Support Intervention on Barriers and Facilitators to Delivery of a Substance Use Prevention Program.

Implementation support interventions have helped organizations implement programs with quality and obtain intended outcomes. For example, a recent randomized controlled trial called Preparing to Run Effective Programs (PREP) showed that an implementation support intervention called Getting To Outcomes (GTO) improved implementation of an evidence-based substance use prevention program (CHOICE) run in community-based settings. However, more information is needed on how these interventions affect organizational barriers and facilitators of implementation. This paper aims to identify differences in implementation facilitators and barriers in sites conducting a substance use prevention program with and without GTO. PREP is a cluster-randomized controlled trial testing GTO, a two-year implementation support intervention, in Boys & Girls Clubs. The trial compares 15 Boys & Girls Club sites implementing CHOICE (control group), a five-session evidence-based alcohol and drug prevention program, with 14 Boys & Girls Club sites implementing CHOICE supported by GTO (intervention group). All sites received CHOICE training. Intervention sites also received GTO manuals, training, and onsite technical assistance to help practitioners complete implementation best practices specified by GTO (i.e., GTO steps). During the first year, technical assistance providers helped the intervention group adopt, plan, and deliver CHOICE, and then evaluate and make quality improvements to CHOICE implementation using feedback reports summarizing their data. Following the second year of CHOICE and GTO implementation, all sites participated in semi-structured interviews to identify barriers and facilitators to CHOICE implementation using the Consolidated Framework for Implementation Research (CFIR). This paper assesses the extent to which these facilitators and barriers differed between intervention and control group. Intervention sites had significantly higher average ratings than control sites for two constructs from the CFIR process domain: planning and reflecting and evaluating. At the same time, intervention sites had significantly lower ratings on the culture and available resources constructs. Findings suggest that strong planning, evaluation, and reflection-likely improved with GTO support-can facilitate implementation even in the face of perceptions of a less desirable implementation climate. These findings highlight that implementation support, such as GTO, is likely to help low-resourced community-based organizations improve program delivery through a focus on implementation processes. TRIAL REGISTRATION: This project is registered at ClinicalTrials.gov with number NCT02135991 (URL: https://clinicaltrials.gov/show/NCT02135991). The trial was first registered May 12, 2014.

Evaluation of a Whole-School Change Intervention: Findings from a Two-Year Cluster-Randomized Trial of the Restorative Practices Intervention.

This study fills a gap in research on multi-level school-based approaches to promoting positive youth development and reducing bullying, in particular cyberbullying, among middle school youth. The study evaluates the Restorative Practices Intervention, a novel whole-school intervention designed to build a supportive environment through the use of 11 restorative practices (e.g., communication approaches that aim to build stronger bonds among leadership, staff, and students such as using "I" statements, encouraging students to express their feelings) that had only quasi-experimental evidence prior to this study. Studying multilevel (e.g., individual, peer group, school) approaches like the Restorative Practices Intervention is important because they are hypothesized to address a more complex interaction of risk factors than single level efforts, which are more common. Baseline and two-year post survey data was collected from 2771 students at 13 middle schools evenly split between grades 6 (48 percent) and 7 (52 percent), and primarily ages 11 (38 percent) or 12 (41 percent). Gender was evenly split (51 percent male), and 92 percent of students were white. The intervention did not yield significant changes in the treatment schools. However, student self-reported experience with restorative practices significantly predicted improved school climate and connectedness, peer attachment, and social skills, and reduced cyberbullying victimization. While more work is needed on how interventions can reliably produce restorative experiences, this study suggests that the restorative model can be useful in promoting positive behaviors and addressing bullying.

A Cluster-Randomized Trial of Getting To Outcomes' Impact on Sexual Health Outcomes in Community-Based Settings.

The USA has high teen pregnancy rates compared to other developed nations. Many community-based organizations need assistance conducting evidence-based teen pregnancy prevention programs (EBPs) appropriately. This study evaluated the impact of an implementation support intervention called Getting To Outcomes (GTO) designed to help such organizations. This cluster randomized controlled trial compared 16 Boys and Girls Clubs (BGCs) implementing a teen pregnancy prevention EBP called Making Proud Choices for two years, with 16 BGCs implementing MPC augmented with GTO training, tools, and technical assistance. Participating middle school youth were compared on proximal outcomes (knowledge, attitudes, and intentions about sex and condoms from baseline to post) and sexual behaviors (frequency of sex and condom use, from baseline to 6-month follow-up). In year 1, there were no significant effects of GTO for any proximal outcome. After GTO-stimulated quality improvement in year 2, the GTO group improved significantly more on condom attitudes and use intentions. Frequency of sex and condom use did not differ between the two groups in either year; however, base rates of these behaviors in the sample were very low. Findings suggest that in typical community-based settings, detailed manuals and training common to structured EBPs may be sufficient to yield some improvement in key proximal outcomes, but that more systematic implementation support is needed to achieve greater improvement in these outcomes. Using GTO with many communities, as currently supported by various federal agencies, could yield public health impact via improvements in condom attitudes and use intentions.

Evaluating the impact of getting to outcomes-underage drinking on prevention capacity and alcohol merchant attitudes and selling behaviors.

Underage drinking is a significant problem facing US communities. Several environmental alcohol prevention (EAP) strategies (laws, regulations, responsible beverage service training and practices) successfully address underage drinking. Communities, however, face challenges carrying out these EAP strategies effectively. This small-scale, 3-year, randomized controlled trial assessed whether providing prevention coalitions with Getting To Outcomes-Underage Drinking (GTO-UD), a tool kit and implementation support intervention, helped improve implementation of two common EAP strategies, responsible beverage service training (RBS) and compliance checks. Three coalitions in South Carolina and their RBS and compliance check programs received the 16-month GTO-UD intervention, including the GTO-UD manual, training, and onsite technical assistance, while another three in South Carolina maintained routine operations. The measures, collected at baseline and after the intervention, were a structured interview assessing how well coalitions carried out their work and a survey of merchant attitudes and practices in the six counties served by the participating coalitions. Over time, the quality of some RBS and compliance check activities improved more in GTO-UD coalitions than in the control sites. No changes in merchant practices or attitudes significantly differed between the GTO-UD and control groups, although merchants in the GTO-UD counties did significantly improve on refusing sales to minors while control merchants did not.

Intervening with practitioners to improve the quality of prevention: one-year findings from a randomized trial of assets-getting to outcomes.

There continues to be a gap in prevention outcomes achieved in research trials versus those achieved in "real-world" practice. This article reports interim findings from a randomized controlled trial evaluating Assets-Getting To Outcomes (AGTO), a two-year intervention designed to build prevention practitioners' capacity to implement positive youth development-oriented practices in 12 community coalitions in Maine. A survey of coalition members was used to assess change on individual practitioners' prevention capacity between baseline and one year later. Structured interviews with 32 program directors (16 in the intervention group and 16 in the control group) were used to assess changes in programs' prevention practices during the same time period. Change in prevention capacity over time did not differ significantly between the intervention and control groups. However, in secondary analyses of only those assigned to the AGTO intervention, users showed greater improvement in their self-efficacy to conduct Assets-based programming and increases in the frequency with which they engaged in AGTO behaviors, whereas among non-users, self-efficacy to conduct Assets-based programming declined. Interview ratings showed improvement in several key areas of performance among intervention programs. Improvement was associated with the number of technical assistance hours received. These results suggest that, after one year, AGTO is beginning to improve the capacity of community practitioners who make use of it.

Scaling Up Implementation Support for Violence Prevention and Resilience Promotion in the Air Force.

Closing the gap between research and practice requires that organizations can consistently incorporate new ideas and best practices. The Department of the Air Force (DAF) Integrated Resilience Directorate is leading a coordinated effort to increase the adoption of evidence-based violence prevention and resilience promotion programs across the entire Department. To support this effort, DAF is using Getting To Outcomes (GTO), an evidence-based implementation support that helps organizations plan, implement, and self-evaluate programs. Thus, the DAF is not only scaling up prevention programs but is also embarking on the largest scale-up of GTO to date. The study team trained personnel from every Air Force installation across the world to use GTO for their programs. Quantitative findings from training questionnaires and ratings of implementation plans as well as qualitative results from resilience personnel interviews suggest some lessons learned for scaling up implementation support. This study builds on established implementation science frameworks for scaling up interventions by identifying critical tasks and unique supports needed to scale up evidence-based prevention. Results suggest GTO helped DAF scale-up prevention across the Department, and that establishing leadership buy-in, simplifying evidence-based program selection and adaptation, monitoring implementation and outcomes, and creating dedicated prevention practitioner roles are critical tasks to support scale-up of evidence-based prevention. Unique supports needed to scale-up evidence-based prevention include multitiered learning systems; integrated tools that improve access to data and evolving evidence; prevention personnel with skills in program management, adaptation, and evaluation; timely crosscutting data; continuous learning to support sustainability; and leadership buy-in.

Establishing and evaluating the key functions of an interactive systems framework using an assets-getting to outcomes intervention.

Community practitioners can face difficulty in achieving outcomes demonstrated by prevention science. Building a community practitioner's prevention capacity-the knowledge and skills needed to conduct critical prevention practices-could improve the quality of prevention and its outcomes. The purpose of this article is to: (1) describe how an intervention called Assets-Getting To Outcomes (AGTO) was used to establish the key functions of the ISF and present early lessons learned from that intervention's first 6 months and (2) examine whether there is an empirical relationship between practitioner capacity at the individual level and the performance of prevention at the program level-a relationship predicted by the ISF but untested. The article describes an operationalization of the ISF in the context of a five-year randomized controlled efficacy trial that combines two complementary models designed to build capacity: Getting To Outcomes (GTO) and Developmental Assets. The trial compares programs and individual practitioners from six community-based coalitions using AGTO with programs and practitioners from six similar coalitions that are not. In this article, we primarily focus on what the ISF calls innovation specific capacity and discuss how the combined AGTO innovation structures and uses feedback about its capacity-building activities, which can serve as a model for implementing the ISF. Focus group discussions used to gather lessons learned from the first 6 months of the AGTO intervention suggest that while the ISF may have been conceptualized as three distinct systems, in practice they are less distinct. Findings from the baseline wave of data collection of individual capacity and program performance suggest that practitioner capacity predicts, in part, performance of prevention programs. Empirically linking practitioner capacity and performance of prevention provides empirical support for both the ISF and AGTO.

Employing continuous quality improvement in community-based substance abuse programs.

PURPOSE: This article aims to describe continuous quality improvement (CQI) for substance abuse prevention and treatment programs in a community-based organization setting. DESIGN/METHODOLOGY/APPROACH: CQI (e.g., plan-do-study-act cycles (PDSA)) applied in healthcare and industry was adapted for substance abuse prevention and treatment programs in a community setting. The authors assessed the resources needed, acceptability and CQI feasibility for ten programs by evaluating CQI training workshops with program staff and a series of three qualitative interviews over a nine-month implementation period with program participants. The CQI activities, PDSA cycle progress, effort, enthusiasm, benefits and challenges were examined. FINDINGS: Results indicated that CQI was feasible and acceptable for community-based substance abuse prevention and treatment programs; however, some notable resource challenges remain. Future studies should examine CQI impact on service quality and intended program outcomes. RESEARCH LIMITATIONS/IMPLICATIONS: The study was conducted on a small number of programs. It did not assess CQI impact on service quality and intended program outcomes. Practical implications- This project shows that it is feasible to adapt CQI techniques and processes for community-based programs substance abuse prevention and treatment programs. These techniques may help community-based program managers to improve service quality and achieve program outcomes. ORIGINALITY/VALUE: This is one of the first studies to adapt traditional CQI techniques for community-based settings delivering substance abuse prevention and treatment programs.

The premises is the premise: understanding off- and on-premises alcohol sales outlets to improve environmental alcohol prevention strategies.

Environmental strategies to prevent the misuse of alcohol among youth--e.g., use of public policies to restrict minors' access to alcohol--have been shown to reduce underage drinking. However, implementation of policy changes often requires public and private partnerships. One way to support these partnerships is to better understand the target of many of the environmental strategies, which is the alcohol sales outlet. Knowing more about how off-premises outlets (e.g., liquor and convenience stores) and on-premises outlets (e.g., bars and restaurants) are alike and different could help community-based organizations better tailor, plan, and implement their environmental strategies and strengthen partnerships between the public and commercial sectors. We conducted a survey of managerial or supervisory staff and/or owners of 336 off- and on-premises alcohol outlets in six counties in South Carolina, comparing these two outlet types on their preferences regarding certain alcohol sales practices, beliefs toward underage drinking, alcohol sales practices, and outcomes. Multilevel logistic regression showed that while off- and on-premises outlets did have many similarities, off-premises outlets appear to engage in more practices designed to prevent sales of alcohol to minors than on-premises outlets. The relationship between certain Responsible Beverage Service (RBS) practices and outcomes varied by outlet type. This study furthers the understanding of the differences between off- and on-premises alcohol sales outlets and offers options for increasing and tailoring environmental prevention efforts to specific settings.

Understanding the relationship between perceived school climate and bullying: A mediator analysis.

This study assesses how perceptions of school climate and four mediating factors (school connectedness, peer attachment, assertiveness, and empathy) influence reports of bullying behaviors among 2,834 students in 14 middle schools. Results revealed that students in positive school climates reported experiencing fewer physical, emotional, and cyberbullying behaviors. They also reported greater levels of school connectedness, peer attachment, assertiveness, and empathy, which in turn helped explain the influence of perceived school climate on bullying. In addition, the greater levels of empathy that students reported, the more likely they were to report being bullied. These results highlight the role that perceptions of school climate can play in influencing bullying and underscore the importance of mediating factors as schools work to track and improve school climate.

Mental Health Service Preferences and Utilization Among Women Veterans in Crisis: Perspectives of Veterans Crisis Line Responders.

Women military veterans are at increased risk of suicide compared to non-veterans, but little is known about the mental health service preferences and needs of women veterans in crisis. This study used qualitative, secondary source key informant interviews to ascertain the experiences of women veterans in crisis from 54 responders working at the Veterans Crisis Line. Responders indicated that women veterans reported different experiences with Veterans Administration (VA) and non-VA care, though drivers of satisfaction or dissatisfaction were similar. Availability of specialty care, sensitivity to veterans' issues or Military Sexual Trauma, strong provider relationships, and continuity of care contributed to satisfaction; lengthy appointment wait times, limited service options, and insensitivity to veterans' issues contributed to dissatisfaction. Responders suggested that barriers limiting VA access for women veterans are perceived as similar to non-VA care. Findings suggest that caller experiences with providers drive satisfaction with VA and non-VA mental health services.

Testing implementation support for evidence-based programs in community settings: a replication cluster-randomized trial of Getting To Outcomes®.

BACKGROUND: Community organizations can have difficulty implementing evidence-based prevention programs. More research is needed on implementation support interventions designed to help these organizations implement programs with quality. METHODS: Preparing to Run Effective Programs (PREP) is a randomized controlled trial testing Getting To Outcomes (GTO), a 2-year implementation support intervention. It compares 15 Boys and Girls Club sites implementing CHOICE (control group), a five-session evidence-based alcohol and drug prevention program, with 14 similar sites implementing CHOICE supported by GTO (intervention group). PREP replicates a previous GTO study that had the same design, but featured a teen pregnancy prevention program instead. All sites received typical CHOICE training. Fourteen intervention sites received GTO manuals, training, and onsite technical assistance to help practitioners complete implementation best practices specified by GTO (i.e., GTO steps). During the first year, technical assistance providers helped the intervention group adopt, plan, and deliver CHOICE. Then, this group was trained on evaluation and quality improvement steps of GTO using feedback reports summarizing their own data, which yielded revised plans for subsequent implementation of CHOICE. This paper presents results regarding GTO's impact on CHOICE fidelity (adherence, quality of delivery, dosage) and the proximal outcomes of the youth participants (aged 10-14)-attitudes and intentions regarding cigarettes, alcohol, and marijuana use. Fidelity was assessed at all sites by observer ratings and attendance logs. Proximal outcomes were assessed via survey at baseline, 3, and 6 months. RESULTS: After 1 year, fidelity and proximal outcomes were similar between Intervention and control groups. After 2 years (which included GTO quality improvement activities that took place between years 1 and 2), intervention sites had higher ratings of CHOICE adherence and quality of delivery (dosage remained similar). Proximal outcomes did not differ between groups in either year, although there was universally high endorsement of prosocial responses to those outcomes from the start. CONCLUSIONS: Findings suggest that systematic implementation support provided by GTO can help community organizations achieve better fidelity. Findings replicate the implementation results from a previous GTO study using the same design, but with a different evidence-based program and different fidelity measures. Although proximal outcomes did not change, in large part due to ceiling effects, the implementation findings suggest GTO can support a variety of programs. TRIAL REGISTRATION: This project is registered at ClinicalTrials.gov with number NCT02135991 . The trial was first registered on May 12, 2014.

Treatment process in the therapeutic community: associations with retention and outcomes among adolescent residential clients.

This study examined associations of a new multidimensional measure of therapeutic community (TC) treatment process, the Dimensions of Change Instrument (DCI), with treatment retention (n = 397) and posttreatment outcomes (n = 207) among adolescent residential clients. Adolescents who increased during the first 30 days in treatment on three of the eight DCI factors (Positive Self-Attitude and Commitment to Abstinence; Problem Recognition; Social Network) were more likely to stay in treatment for 90 days or more. Although change in DCI factor scores was not directly predictive of posttreatment outcomes, staying in treatment for 90 days or more increased the likelihood of attending 12-step meetings and having a 12-step sponsor after leaving treatment. Several demographic and pretreatment characteristics also predicted retention, posttreatment outcomes, or both. Much remains to be learned about how the TC treatment process produces positive outcomes for adolescents; however, results from this study highlight components of the treatment process that may be particularly important for practitioners to consider and monitor in their clients.

Suicide Prevention Hotlines in California: Diversity in Services, Structure, and Organization and the Potential Challenges Ahead.

This article evaluates the landscape in which suicide prevention hotlines operate, focusing on California but with nationwide implications, and provides recommendations to ensure that callers at risk of suicide seek and receive the help they need.

Expanding Suicide Crisis Services to Text and Chat.

BACKGROUND: Crisis support services have historically been offered by phone-based suicide prevention hotlines, but are increasingly becoming available through alternative modalities, including Internet chat and text messaging. AIMS: To better understand differences in the use of phone and chat/text services. METHOD: We conducted semistructured interviews with call responders at the Veterans Crisis Line who utilize multimodal methods to respond to veterans in crisis. RESULTS: Responders indicated that veterans may access the chat/text service primarily for reasons that included a desire for anonymity and possible inability to use the phone. Responders were divided on whether callers and chatters presented with different issues or risk of suicide; however, they suggested that veterans frequently use chat/text to make their first contact with mental health services. LIMITATIONS: We spoke with call responders, not the veterans themselves. Additionally, as this is qualitative research, applicability to other settings may be limited. CONCLUSION: While new platforms offer promise, participants also indicated that chat services can supplement phone lines, but not replace them.

Can implementation support help community-based settings better deliver evidence-based sexual health promotion programs? A randomized trial of Getting To Outcomes®.

BACKGROUND: Research is needed to evaluate the impact of implementation support interventions over and above typical efforts by community settings to deploy evidence-based prevention programs. METHODS: Enhancing Quality Interventions Promoting Healthy Sexuality is a randomized controlled trial testing Getting To Outcomes (GTO), a 2-year implementation support intervention. It compares 16 Boys and Girls Club sites implementing Making Proud Choices (MPC, control group), a structured teen pregnancy prevention evidence-based program with 16 similar sites implementing MPC augmented with GTO (intervention group). All sites received training and manuals typical for MPC. GTO has its own manuals, training, and onsite technical assistance (TA) to help practitioners complete key programming practices specified by GTO. During the first year, TA providers helped the intervention group adopt, plan, and deliver MPC. This group then received training on the evaluation and quality improvement steps of GTO, including feedback reports summarizing their data, which were used in a TA-facilitated quality improvement process that yielded revised plans for the second MPC implementation. This paper presents results regarding GTO's impact on performance of the sites (i.e., how well key programming practices were carried out), fidelity of MPC implementation, and the relationship between amount of TA support, performance, and fidelity. Performance was measured using ratings made from a standardized, structured interview conducted with participating staff at all 32 Boys and Girls Clubs sites after the first and second years of MPC implementation. Multiple elements of fidelity (adherence, classroom delivery, dosage) were assessed at all sites by observer ratings and attendance logs. RESULTS: After 2 years, the intervention sites had higher ratings of performance, adherence, and classroom delivery (dosage remained similar). Higher performance predicted greater adherence in both years. CONCLUSIONS: These findings suggest that in typical community-based settings, manuals and training common to structured EBPs may be sufficient to yield low levels of performance and moderate levels of fidelity but that systematic implementation support is needed to achieve high levels of performance and fidelity. TRIAL REGISTRATION: ClinicalTrials.gov, NCT01818791.