Scaling Up Implementation Support for Violence Prevention and Resilience Promotion in the Air Force.

Closing the gap between research and practice requires that organizations can consistently incorporate new ideas and best practices. The Department of the Air Force (DAF) Integrated Resilience Directorate is leading a coordinated effort to increase the adoption of evidence-based violence prevention and resilience promotion programs across the entire Department. To support this effort, DAF is using Getting To Outcomes (GTO), an evidence-based implementation support that helps organizations plan, implement, and self-evaluate programs. Thus, the DAF is not only scaling up prevention programs but is also embarking on the largest scale-up of GTO to date. The study team trained personnel from every Air Force installation across the world to use GTO for their programs. Quantitative findings from training questionnaires and ratings of implementation plans as well as qualitative results from resilience personnel interviews suggest some lessons learned for scaling up implementation support. This study builds on established implementation science frameworks for scaling up interventions by identifying critical tasks and unique supports needed to scale up evidence-based prevention. Results suggest GTO helped DAF scale-up prevention across the Department, and that establishing leadership buy-in, simplifying evidence-based program selection and adaptation, monitoring implementation and outcomes, and creating dedicated prevention practitioner roles are critical tasks to support scale-up of evidence-based prevention. Unique supports needed to scale-up evidence-based prevention include multitiered learning systems; integrated tools that improve access to data and evolving evidence; prevention personnel with skills in program management, adaptation, and evaluation; timely crosscutting data; continuous learning to support sustainability; and leadership buy-in.

"What We Have Here, Is a Failure to [Replicate]": Ways to Solve a Replication Crisis in Implementation Science.

Adapting the classic line from the 1967 film Cool Hand Luke, the title is meant to convey that implementation science (IS), like other fields, has not been embracing replication studies, which is a key component to the open science movement. The purpose of this article is to review what is known about replication of implementation trials and identify the gaps and next steps to continue increasing the transparency, openness, and replicability of implementation research. After presenting an overview of study replication and how it is a key component of open science, the article will examine how replication of implementation studies has (or more accurately has not) been approached in IS. As will be discussed, replication in IS shares some challenges with studies that attempt to replicate interventions, but also presents unique challenges. This article discusses different types of replications (e.g., direct vs. conceptual) and how they can benefit the field of IS. The article then presents a specific example of an implementation strategy called Getting To Outcomes© to describe how to design a replication study and interpret the results. The article ends with multiple options implementation scientists could consider to improve the likelihood and quality of replication studies. The discussion also envisions how implementation science can enable researchers and practitioners to work together in real-world contexts to encourage wide replication of implementation studies and advance the goal of improving public health.

Cost analysis of a randomized trial of Getting to Outcomes implementation support for a teen pregnancy prevention program offered in Boys and Girls Clubs in Alabama and Georgia.

Implementation support can improve outcomes of evidence-based programs (EBP) for adolescents, but with a cost. To assist in determining whether this cost is worthwhile, this study estimated the cost of adding Getting To Outcomes© (GTO) implementation support to a teen pregnancy and sexually transmitted infection prevention EBP called Making Proud Choices (MPC) in 32 Boys and Girls Clubs (BGCs) in Alabama and Georgia. Enhancing Quality Interventions Promoting Healthy Sexuality (EQUIPS) was a 2-year, cluster-randomized controlled trial comparing MPC with MPC + GTO. We used micro-costing to estimate costs and captured MPC and GTO time from activity logs completed by GTO staff. Key resource use and cost components were compared between the randomized groups, years, and states (to capture different community site circumstances) using 2-sample t tests. There were no significant differences between randomized groups in attendees per site, resource use, or costs for either year. However, there were significant differences between states. Adding GTO to MPC increased the societal costs per attendee from $67 to $144 (2015 US dollars) in Georgia and from $106 to $314 in Alabama. The higher Alabama cost was due to longer travel distances and to more BGC staff time spent on GTO in that state. GTO also improved adherence, classroom delivery, and condom-use intentions more in Alabama youth. Thus, Alabama's GTO-related BGC staff time costs may be better estimates of effective GTO. If teen childbearing costs taxpayers approximately $20,000 per teen birth, adding GTO to MPC would be worthwhile to society if it prevented one more teen birth per 140 attendees than MPC alone.Trial registration. ClinicalTrials.gov , NCT01818791. Registered March 26, 2013, https://clinicaltrials.gov/ct2/show/NCT01818791?term=NCT01818791&draw=2&rank=1.

Sustaining an Evidence-Based Program Over Time: Moderators of Sustainability and the Role of the Getting to Outcomes® Implementation Support Intervention.

Problematic rates of alcohol, e-cigarette, and other drug use among US adolescents highlight the need for effective implementation of evidence-based programs (EBPs), yet schools and community organizations have great difficulty implementing and sustaining EBPs. Although a growing number of studies show that implementation support interventions can improve EBP implementation, the literature on how to improve sustainability through implementation support is limited. This randomized controlled trial advances the literature by testing the effects of one such implementation intervention-Getting To Outcomes (GTO)-on sustainability of CHOICE, an after-school EBP for preventing substance use among middle-school students. CHOICE implementation was tracked for 2 years after GTO support ended across 29 Boys and Girls Club sites in the greater Los Angeles area. Predictors of sustainability were identified for a set of key tasks targeted by the GTO approach (e.g., goal setting, evaluation, collectively called "GTO performance") and for CHOICE fidelity using a series of path models. One year after GTO support ended, we found no differences between GTO and control sites on CHOICE fidelity. GTO performance was also similar between groups; however, GTO sites were superior in conducting evaluation. Better GTO performance predicted better CHOICE fidelity. Two years after GTO support ended, GTO sites were significantly more likely to sustain CHOICE implementation when compared with control sites. This study suggests that using an implementation support intervention like GTO can help low-resource settings continue to sustain their EBP implementation to help them get the most out of their investment. ClinicalTrials.gov Identifier: NCT02135991.

Cost Analysis of a Randomized Trial of Getting to Outcomes Implementation Support of CHOICE in Boys and Girls Clubs in Southern California.

Costs of supporting prevention program implementation are not well known. This study estimates the societal costs of implementing CHOICE, a voluntary after-school alcohol and other drug prevention program for adolescents, in Boys and Girls Clubs (BGCs) across Southern California with and without an implementation support system called Getting To Outcomes© (GTO). This article uses micro-costing methods to estimate the cost of the CHOICE program and GTO support. Labor and expense data were obtained from logs kept by the BGC staff and by the GTO technical assistance (TA) staff, and staff time was valued based on Bureau of Labor Statistics estimates. From the societal perspective, the cost of implementing CHOICE at BGCs over the 2-year study period was $27 per attendee when CHOICE was offered by itself (all costs incurred by the BGCs) and $177 per attendee when CHOICE was offered with GTO implementation support ($67 cost to the BGCs; $110 to the entity funding GTO). These results were most sensitive to assumptions as to the number of times CHOICE was offered per year. Adding GTO implementation support to CHOICE increased the cost per attendee by approximately $150. For this additional cost, there was evidence that the CHOICE program was offered with more fidelity and offered more often after the 2-year intervention ended. If the long-term benefits of this better and continued implementation are found to exceed these additional costs, GTO could be an attractive structure to support evidence-based substance misuse prevention programs. Trial Registration. This project is registered at ClinicalTrials.gov with number NCT02135991 (URL: https://clinicaltrials.gov/show/NCT02135991). The trial was registered May 12, 2014.

Influence of an Implementation Support Intervention on Barriers and Facilitators to Delivery of a Substance Use Prevention Program.

Implementation support interventions have helped organizations implement programs with quality and obtain intended outcomes. For example, a recent randomized controlled trial called Preparing to Run Effective Programs (PREP) showed that an implementation support intervention called Getting To Outcomes (GTO) improved implementation of an evidence-based substance use prevention program (CHOICE) run in community-based settings. However, more information is needed on how these interventions affect organizational barriers and facilitators of implementation. This paper aims to identify differences in implementation facilitators and barriers in sites conducting a substance use prevention program with and without GTO. PREP is a cluster-randomized controlled trial testing GTO, a two-year implementation support intervention, in Boys & Girls Clubs. The trial compares 15 Boys & Girls Club sites implementing CHOICE (control group), a five-session evidence-based alcohol and drug prevention program, with 14 Boys & Girls Club sites implementing CHOICE supported by GTO (intervention group). All sites received CHOICE training. Intervention sites also received GTO manuals, training, and onsite technical assistance to help practitioners complete implementation best practices specified by GTO (i.e., GTO steps). During the first year, technical assistance providers helped the intervention group adopt, plan, and deliver CHOICE, and then evaluate and make quality improvements to CHOICE implementation using feedback reports summarizing their data. Following the second year of CHOICE and GTO implementation, all sites participated in semi-structured interviews to identify barriers and facilitators to CHOICE implementation using the Consolidated Framework for Implementation Research (CFIR). This paper assesses the extent to which these facilitators and barriers differed between intervention and control group. Intervention sites had significantly higher average ratings than control sites for two constructs from the CFIR process domain: planning and reflecting and evaluating. At the same time, intervention sites had significantly lower ratings on the culture and available resources constructs. Findings suggest that strong planning, evaluation, and reflection-likely improved with GTO support-can facilitate implementation even in the face of perceptions of a less desirable implementation climate. These findings highlight that implementation support, such as GTO, is likely to help low-resourced community-based organizations improve program delivery through a focus on implementation processes. TRIAL REGISTRATION: This project is registered at ClinicalTrials.gov with number NCT02135991 (URL: https://clinicaltrials.gov/show/NCT02135991). The trial was first registered May 12, 2014.

Evaluation of a Whole-School Change Intervention: Findings from a Two-Year Cluster-Randomized Trial of the Restorative Practices Intervention.

This study fills a gap in research on multi-level school-based approaches to promoting positive youth development and reducing bullying, in particular cyberbullying, among middle school youth. The study evaluates the Restorative Practices Intervention, a novel whole-school intervention designed to build a supportive environment through the use of 11 restorative practices (e.g., communication approaches that aim to build stronger bonds among leadership, staff, and students such as using "I" statements, encouraging students to express their feelings) that had only quasi-experimental evidence prior to this study. Studying multilevel (e.g., individual, peer group, school) approaches like the Restorative Practices Intervention is important because they are hypothesized to address a more complex interaction of risk factors than single level efforts, which are more common. Baseline and two-year post survey data was collected from 2771 students at 13 middle schools evenly split between grades 6 (48 percent) and 7 (52 percent), and primarily ages 11 (38 percent) or 12 (41 percent). Gender was evenly split (51 percent male), and 92 percent of students were white. The intervention did not yield significant changes in the treatment schools. However, student self-reported experience with restorative practices significantly predicted improved school climate and connectedness, peer attachment, and social skills, and reduced cyberbullying victimization. While more work is needed on how interventions can reliably produce restorative experiences, this study suggests that the restorative model can be useful in promoting positive behaviors and addressing bullying.

Understanding the relationship between perceived school climate and bullying: A mediator analysis.

This study assesses how perceptions of school climate and four mediating factors (school connectedness, peer attachment, assertiveness, and empathy) influence reports of bullying behaviors among 2,834 students in 14 middle schools. Results revealed that students in positive school climates reported experiencing fewer physical, emotional, and cyberbullying behaviors. They also reported greater levels of school connectedness, peer attachment, assertiveness, and empathy, which in turn helped explain the influence of perceived school climate on bullying. In addition, the greater levels of empathy that students reported, the more likely they were to report being bullied. These results highlight the role that perceptions of school climate can play in influencing bullying and underscore the importance of mediating factors as schools work to track and improve school climate.

Mental Health Service Preferences and Utilization Among Women Veterans in Crisis: Perspectives of Veterans Crisis Line Responders.

Women military veterans are at increased risk of suicide compared to non-veterans, but little is known about the mental health service preferences and needs of women veterans in crisis. This study used qualitative, secondary source key informant interviews to ascertain the experiences of women veterans in crisis from 54 responders working at the Veterans Crisis Line. Responders indicated that women veterans reported different experiences with Veterans Administration (VA) and non-VA care, though drivers of satisfaction or dissatisfaction were similar. Availability of specialty care, sensitivity to veterans' issues or Military Sexual Trauma, strong provider relationships, and continuity of care contributed to satisfaction; lengthy appointment wait times, limited service options, and insensitivity to veterans' issues contributed to dissatisfaction. Responders suggested that barriers limiting VA access for women veterans are perceived as similar to non-VA care. Findings suggest that caller experiences with providers drive satisfaction with VA and non-VA mental health services.

Testing implementation support for evidence-based programs in community settings: a replication cluster-randomized trial of Getting To Outcomes®.

BACKGROUND: Community organizations can have difficulty implementing evidence-based prevention programs. More research is needed on implementation support interventions designed to help these organizations implement programs with quality. METHODS: Preparing to Run Effective Programs (PREP) is a randomized controlled trial testing Getting To Outcomes (GTO), a 2-year implementation support intervention. It compares 15 Boys and Girls Club sites implementing CHOICE (control group), a five-session evidence-based alcohol and drug prevention program, with 14 similar sites implementing CHOICE supported by GTO (intervention group). PREP replicates a previous GTO study that had the same design, but featured a teen pregnancy prevention program instead. All sites received typical CHOICE training. Fourteen intervention sites received GTO manuals, training, and onsite technical assistance to help practitioners complete implementation best practices specified by GTO (i.e., GTO steps). During the first year, technical assistance providers helped the intervention group adopt, plan, and deliver CHOICE. Then, this group was trained on evaluation and quality improvement steps of GTO using feedback reports summarizing their own data, which yielded revised plans for subsequent implementation of CHOICE. This paper presents results regarding GTO's impact on CHOICE fidelity (adherence, quality of delivery, dosage) and the proximal outcomes of the youth participants (aged 10-14)-attitudes and intentions regarding cigarettes, alcohol, and marijuana use. Fidelity was assessed at all sites by observer ratings and attendance logs. Proximal outcomes were assessed via survey at baseline, 3, and 6 months. RESULTS: After 1 year, fidelity and proximal outcomes were similar between Intervention and control groups. After 2 years (which included GTO quality improvement activities that took place between years 1 and 2), intervention sites had higher ratings of CHOICE adherence and quality of delivery (dosage remained similar). Proximal outcomes did not differ between groups in either year, although there was universally high endorsement of prosocial responses to those outcomes from the start. CONCLUSIONS: Findings suggest that systematic implementation support provided by GTO can help community organizations achieve better fidelity. Findings replicate the implementation results from a previous GTO study using the same design, but with a different evidence-based program and different fidelity measures. Although proximal outcomes did not change, in large part due to ceiling effects, the implementation findings suggest GTO can support a variety of programs. TRIAL REGISTRATION: This project is registered at ClinicalTrials.gov with number NCT02135991 . The trial was first registered on May 12, 2014.

A Cluster-Randomized Trial of Getting To Outcomes' Impact on Sexual Health Outcomes in Community-Based Settings.

The USA has high teen pregnancy rates compared to other developed nations. Many community-based organizations need assistance conducting evidence-based teen pregnancy prevention programs (EBPs) appropriately. This study evaluated the impact of an implementation support intervention called Getting To Outcomes (GTO) designed to help such organizations. This cluster randomized controlled trial compared 16 Boys and Girls Clubs (BGCs) implementing a teen pregnancy prevention EBP called Making Proud Choices for two years, with 16 BGCs implementing MPC augmented with GTO training, tools, and technical assistance. Participating middle school youth were compared on proximal outcomes (knowledge, attitudes, and intentions about sex and condoms from baseline to post) and sexual behaviors (frequency of sex and condom use, from baseline to 6-month follow-up). In year 1, there were no significant effects of GTO for any proximal outcome. After GTO-stimulated quality improvement in year 2, the GTO group improved significantly more on condom attitudes and use intentions. Frequency of sex and condom use did not differ between the two groups in either year; however, base rates of these behaviors in the sample were very low. Findings suggest that in typical community-based settings, detailed manuals and training common to structured EBPs may be sufficient to yield some improvement in key proximal outcomes, but that more systematic implementation support is needed to achieve greater improvement in these outcomes. Using GTO with many communities, as currently supported by various federal agencies, could yield public health impact via improvements in condom attitudes and use intentions.

Suicide Prevention Hotlines in California: Diversity in Services, Structure, and Organization and the Potential Challenges Ahead.

This article evaluates the landscape in which suicide prevention hotlines operate, focusing on California but with nationwide implications, and provides recommendations to ensure that callers at risk of suicide seek and receive the help they need.

Expanding Suicide Crisis Services to Text and Chat.

BACKGROUND: Crisis support services have historically been offered by phone-based suicide prevention hotlines, but are increasingly becoming available through alternative modalities, including Internet chat and text messaging. AIMS: To better understand differences in the use of phone and chat/text services. METHOD: We conducted semistructured interviews with call responders at the Veterans Crisis Line who utilize multimodal methods to respond to veterans in crisis. RESULTS: Responders indicated that veterans may access the chat/text service primarily for reasons that included a desire for anonymity and possible inability to use the phone. Responders were divided on whether callers and chatters presented with different issues or risk of suicide; however, they suggested that veterans frequently use chat/text to make their first contact with mental health services. LIMITATIONS: We spoke with call responders, not the veterans themselves. Additionally, as this is qualitative research, applicability to other settings may be limited. CONCLUSION: While new platforms offer promise, participants also indicated that chat services can supplement phone lines, but not replace them.

Suicide Risk among Women Veterans in Distress: Perspectives of Responders on the Veterans Crisis Line.

BACKGROUND: Women veterans are at increasingly high risk of suicide, but little is known about the concerns and needs of this population. This is, in part, owing to the low base rate of suicide and the inability to conduct retrospective interviews with individuals who died. In this study, we used a qualitative approach to gain insight about the concerns and nature of comments regarding suicidal ideation and intent among women veterans calling the Veterans Crisis Line (VCL). METHODS: Fifty-four VCL call responders were interviewed in the spring of 2015. They were asked about the concerns and level of suicide risk of women veteran callers with whom they have spoken and about the ways in which women callers are similar to or different from men callers. Interviews were transcribed and thematic analyses were conducted to examine patterns or themes emerging from the data. FINDINGS: Military sexual trauma and non-suicidal self-harm were two commonly reported concerns of women veteran callers according to responders. VCL responders also noted differences between men and women veteran callers, including differences in clinical presentation, suicidal means, and protective factors. CONCLUSIONS: Our findings shed light on potential avenues to prevent suicide among women veterans, although we spoke to VCL responders about their impressions, rather than to women veterans themselves. Efforts to 1) prevent and treat the consequences of military sexual trauma, 2) recognize, prevent, and treat non-suicidal self-harm, and 3) restrict access to lethal means most commonly reported among women veteran callers may be helpful to mitigate suicide risk in this vulnerable group of veterans.

Can implementation support help community-based settings better deliver evidence-based sexual health promotion programs? A randomized trial of Getting To Outcomes®.

BACKGROUND: Research is needed to evaluate the impact of implementation support interventions over and above typical efforts by community settings to deploy evidence-based prevention programs. METHODS: Enhancing Quality Interventions Promoting Healthy Sexuality is a randomized controlled trial testing Getting To Outcomes (GTO), a 2-year implementation support intervention. It compares 16 Boys and Girls Club sites implementing Making Proud Choices (MPC, control group), a structured teen pregnancy prevention evidence-based program with 16 similar sites implementing MPC augmented with GTO (intervention group). All sites received training and manuals typical for MPC. GTO has its own manuals, training, and onsite technical assistance (TA) to help practitioners complete key programming practices specified by GTO. During the first year, TA providers helped the intervention group adopt, plan, and deliver MPC. This group then received training on the evaluation and quality improvement steps of GTO, including feedback reports summarizing their data, which were used in a TA-facilitated quality improvement process that yielded revised plans for the second MPC implementation. This paper presents results regarding GTO's impact on performance of the sites (i.e., how well key programming practices were carried out), fidelity of MPC implementation, and the relationship between amount of TA support, performance, and fidelity. Performance was measured using ratings made from a standardized, structured interview conducted with participating staff at all 32 Boys and Girls Clubs sites after the first and second years of MPC implementation. Multiple elements of fidelity (adherence, classroom delivery, dosage) were assessed at all sites by observer ratings and attendance logs. RESULTS: After 2 years, the intervention sites had higher ratings of performance, adherence, and classroom delivery (dosage remained similar). Higher performance predicted greater adherence in both years. CONCLUSIONS: These findings suggest that in typical community-based settings, manuals and training common to structured EBPs may be sufficient to yield low levels of performance and moderate levels of fidelity but that systematic implementation support is needed to achieve high levels of performance and fidelity. TRIAL REGISTRATION: ClinicalTrials.gov, NCT01818791.

A Cluster-Randomized Trial of Restorative Practices: An Illustration to Spur High-Quality Research and Evaluation.

Restorative Practices in schools lack rigorous evaluation studies. As an example of rigorous school-based research, this paper describes the first randomized control trial of restorative practices to date, the Study of Restorative Practices. It is a 5-year, cluster-randomized controlled trial (RCT) of the Restorative Practices Intervention (RPI) in 14 middle schools in Maine to assess whether RPI impacts both positive developmental outcomes and problem behaviors and whether the effects persist during the transition from middle to high school. The two-year RPI intervention began in the 2014-2015 school year. The study's rationale and theoretical concerns are discussed along with methodological concerns including teacher professional development. The theoretical rationale and description of the methods from this study may be useful to others conducting rigorous research and evaluation in this area.

RAND's Silent Monitoring Protocol for Assessing Suicide Crisis Line Call Content and Quality.

This article describes the methods for developing and testing a silent monitoring protocol for California suicide crisis call centers to evaluate call content and quality, document staff strengths and weaknesses, and identify areas for further training.

Evaluation of California's Statewide Mental Health Prevention and Early Intervention Programs: Summary of Key Year 2 Findings.

This article summarizes key findings from the second year of an evaluation of the California Mental Health Services Authority's statewide prevention and early intervention programs.

Evaluating the impact of getting to outcomes-underage drinking on prevention capacity and alcohol merchant attitudes and selling behaviors.

Underage drinking is a significant problem facing US communities. Several environmental alcohol prevention (EAP) strategies (laws, regulations, responsible beverage service training and practices) successfully address underage drinking. Communities, however, face challenges carrying out these EAP strategies effectively. This small-scale, 3-year, randomized controlled trial assessed whether providing prevention coalitions with Getting To Outcomes-Underage Drinking (GTO-UD), a tool kit and implementation support intervention, helped improve implementation of two common EAP strategies, responsible beverage service training (RBS) and compliance checks. Three coalitions in South Carolina and their RBS and compliance check programs received the 16-month GTO-UD intervention, including the GTO-UD manual, training, and onsite technical assistance, while another three in South Carolina maintained routine operations. The measures, collected at baseline and after the intervention, were a structured interview assessing how well coalitions carried out their work and a survey of merchant attitudes and practices in the six counties served by the participating coalitions. Over time, the quality of some RBS and compliance check activities improved more in GTO-UD coalitions than in the control sites. No changes in merchant practices or attitudes significantly differed between the GTO-UD and control groups, although merchants in the GTO-UD counties did significantly improve on refusing sales to minors while control merchants did not.

Recommendations for Sustaining California's Statewide Mental Health Prevention and Early Intervention Programs.

Provides preliminary advice about which of the California Mental Health Services Authority's Prevention and Early Intervention activities seem most valuable to sustain or, in some cases, enhance.