Mechanisms of change in external implementation support: A conceptual model and case examples to guide research and practice.

Background: External implementation support (EIS) is a well-recognized feature of implementation science and practice, often under related terms such as technical assistance and implementation facilitation. Existing models of EIS have gaps related to addressing practice outcomes at both individual and organizational levels, connecting practice activities to intended outcomes, or grounding in well-established theories of behavior and organization change. Moreover, there have been calls to clarify the mechanisms of change through which EIS influences related outcomes. Method: In this article, we theorize about mechanisms of change within EIS. Our theorizing process aligns with the approach advocated by Kislov et al. We aim to consolidate prior EIS literature, combining related constructs from previous empirical and conceptual work while drawing on our extensive EIS experience to develop a higher-order, midrange theory of change. Results: Our theory of change is empirically and practically informed, conceptually situated within an established grand theory of change, and guided by eight practice principles and social cognitive theory. The theory of change proposes 10 core practice components as mechanisms of change within EIS. When used according to underlying theory and principles, they are believed to contribute to favorable practice outcomes at individual, team, organizational, and system levels. The model offers flexibility by recognizing the need for sequential support processes and the demand to practice in dynamic and responsive ways. Case examples are presented to illustrate major themes and patterns of the model in action. Conclusions: The proposed model is intended to support prospective EIS studies by conceptualizing discernable practice components with hypothesized relationships to proximal and distal practice outcomes. The model can be behaviorally operationalized to compliment and extend competency-based approaches to implementation support practitioner (ISP) training and coaching. Over time, the model should be refined based on new empirical findings and contributions from ISPs across the field.

There are few models that help us understand how external support providers work with organizational, system, and community partners to improve their efforts to implement innovative programs and practices. Existing models typically describe characteristics and features of the process but lack grounding in well-established theories of behavior and organizational change. In this paper, we theorize about mechanisms of change within the support process, which we label core practice components, and explain how their use might improve implementation efforts through shorter- and longer-term practice outcomes. We believe that our model holds promise for informing future advancements in both research and practice. Foremost, the core practice components lend themselves to behavioral definitions and thus being observed and reported in action. In research, this will allow the relationships we propose in our model to be tested and refined over time, resulting in an incremental accumulation of knowledge. In practice, a greater understanding of core practice components and their relationships to key practice outcomes offers ways to enhance training and coaching activities for external support providers. The model may also aid support providers to more effectively navigate the support process and plan more timely and effective support strategies.

Scaling up Evidence-Based Interventions in US Public Systems to Prevent Behavioral Health Problems: Challenges and Opportunities.

A number of programs, policies, and practices have been tested using rigorous scientific methods and shown to prevent behavioral health problems (Catalano et al., Lancet 379:1653-1664, 2012; National Research Council and Institute of Medicine, 2009). Yet these evidence-based interventions (EBIs) are not widely used in public systems, and they have limited reach (Glasgow et al., American Journal of Public Health 102:1274-1281, 2012; National Research Council and Institute of Medicine 2009; Prinz and Sanders, Clinical Psychology Review 27:739-749, 2007). To address this challenge and improve public health and well-being at a population level, the Society for Prevention Research (SPR) formed the Mapping Advances in Prevention Science (MAPS) IV Translation Research Task Force, which considered ways to scale up EBIs in five public systems: behavioral health, child welfare, education, juvenile justice, and public health. After reviewing other efforts to scale up EBIs in public systems, a common set of factors were identified as affecting scale-up in all five systems. The most important factor was the degree to which these systems enacted public policies (i.e., statutes, regulations, and guidance) requiring or recommending EBIs and provided public funds for EBIs. Across systems, other facilitators of scale-up were creating EBIs that are ready for scale-up, public awareness of and support for EBIs, community engagement and capacity to implement EBIs, leadership support for EBIs, a skilled workforce capable of delivering EBIs, and data monitoring and evaluation capacity. It was concluded that the following actions are needed to significantly increase EBI scale-up in public systems: (1) provide more public policies and funding to support the creation, testing, and scaling up of EBIs; (2) develop and evaluate specific frameworks that address systems level barriers impeding EBI scale-up; and (3) promote public support for EBIs, community capacity to implement EBIs at scale, and partnerships between community stakeholders, policy makers, practitioners, and scientists within and across systems.

Achieving successful evidence-based practice implementation in juvenile justice: The importance of diagnostic and evaluative capacity.

Evidence-based programs (EBPs) are an increasingly visible aspect of the treatment landscape in juvenile justice. Research demonstrates that such programs yield positive returns on investment and are replacing more expensive, less effective options. However, programs are unlikely to produce expected benefits when they are not well-matched to community needs, not sustained and do not reach sufficient reach and scale. We argue that achieving these benchmarks for successful implementation will require states and county governments to invest in data-driven decision infrastructure in order to respond in a rigorous and flexible way to shifting political and funding climates. We conceptualize this infrastructure as diagnostic capacity and evaluative capacity: Diagnostic capacity is defined as the process of selecting appropriate programing and evaluative capacity is defined as the ability to monitor and evaluate progress. Policy analyses of Washington State, Pennsylvania and Louisiana's program implementation successes are used to illustrate the benefits of diagnostic and evaluate capacity as a critical element of EBP implementation.

Sustaining evidence-based prevention programs: correlates in a large-scale dissemination initiative.

Over the past four decades, significant strides have been made in the science of preventing youth problem behaviors. Subsequently, policymakers and funders have begun to insist on the use of evidence-based programs (EBPs) as a requirement for funding. However, unless practitioners are able to sustain these programs beyond initial seed funding, they are unlikely to achieve their ultimate goal of broad public health impact. Despite its obvious importance, sustainability has received relatively little attention in prevention science until recently. Moreover, there have been few opportunities to study the correlates of sustainability in large-scale, multi-year initiatives involving multiple programs. The present study examined rates of sustainment of a wide range of proven-effective prevention and intervention programs; identified factors related to organizational support and readiness, program and implementer characteristics, and sustainability planning that distinguished sustained programs; and examined variability in these associations across classroom-based, community/mentoring, family-focused prevention, and family treatment program types within the context of a state-wide EBP dissemination initiative in Pennsylvania over 4 years. The majority of EBPs sustained functioning 2 years or more beyond their initial funding. In general, sustained programs reported greater community coalition functioning, communication to key stakeholders, knowledge of the program's logical model, communication with the trainer or program developer, and sustainability planning. In addition to these universal correlates, important program-specific correlates emerged as well. Implications for the technical assistance and support necessary to promote the sustainability of EBPs in nonresearch contexts are also discussed.

Examining adaptations of evidence-based programs in natural contexts.

When evidence-based programs (EBPs) are scaled up in natural, or non-research, settings, adaptations are commonly made. Given the fidelity-versus-adaptation debate, theoretical rationales have been provided for the pros and cons of adaptations. Yet the basis of this debate is theoretical; thus, empirical evidence is needed to understand the types of adaptations made in natural settings. In the present study, we introduce a taxonomy for understanding adaptations. This taxonomy addresses several aspects of adaptations made to programs including the fit (philosophical or logistical), timing (proactive or reactive), and valence, or the degree to which the adaptations align with the program's goals and theory, (positive, negative, or neutral). Self-reported qualitative data from communities delivering one of ten state-funded EBPs were coded based on the taxonomy constructs; additionally, quantitative data were used to examine the types and reasons for making adaptations under natural conditions. Forty-four percent of respondents reported making adaptations. Adaptations to the procedures, dosage, and content were cited most often. Lack of time, limited resources, and difficulty retaining participants were listed as the most common reasons for making adaptations. Most adaptations were made reactively, as a result of issues of logistical fit, and were not aligned with, or deviated from, the program's goals and theory.

The role of a state-level prevention support system in promoting high-quality implementation and sustainability of evidence-based programs.

Although numerous evidence-based programs (EBPs) have been proven effective in research trials and are being widely promoted through federal, state, and philanthropic dollars, few have been "scaled up" in a manner likely to have a measurable impact on today's critical social problems. The Interactive Systems Framework for Dissemination and Implementation (ISF) explicates three systems that are critical in addressing the barriers that prevent these programs from having their intended public health impact. In this article we describe the relevance of these systems in a real-world context with a specific focus on the Prevention Support System (PSS). We expand on the ISF model by presenting funders and policy-makers as active and engaged stakeholders, and demonstrate how a state-level PSS has used empirical evidence to inform general and program-specific capacity-building and support interactions among researchers, funders, and practitioners in Pennsylvania. By embracing this expanded ISF framework as a conceptual model for the wide-scale dissemination and support of EBPs, and recognizing the need for a distinct state-level PSS, Pennsylvania has created an infrastructure to effectively address the primary barriers to moving from lists of EBPs to achieving population-level public health improvement.

A state agency-university partnership for translational research and the dissemination of evidence-based prevention and intervention.

This article describes a decade-long partnership between the Prevention Research Center at Penn State and the Pennsylvania Commission on Crime and Delinquency. This partnership has evolved into a multi-agency initiative supporting the implementation of nearly 200 replications of evidence-based prevention and intervention programs, and a series of studies indicating a significant and sustained impact on youth outcomes and more efficient utilization of system resources. We describe how the collaboration has developed into a sophisticated prevention support infrastructure, discuss the partnership and policy lessons learned throughout this journey, and identify remaining issues in promoting this type of research-policy partnership.

Sustaining evidence-based interventions under real-world conditions: results from a large-scale diffusion project.

This study examined factors associated with the predicted and actual post-funding sustainability of evidence-based interventions implemented as part of the Pennsylvania Commission on Crime and Delinquency's Research-Based Delinquency and Violence Prevention Initiative. Correlates of predicted post-funding sustainability included program staff, overall school support, and school administrator support. Additionally, predicted post-funding sustainability was strongly associated with actual post-funding sustainability. Other correlates of actual post-funding sustainability included financial sustainability planning and aligning the intervention with the goals of the agency/school. Five years post-funding 33% of the interventions were no longer operating, 22% were operating at a reduced level, and 45% were operating at the same level or a higher level than the final year of funding. These findings are discussed in terms of implications for increasing intervention sustainability, as well as implications for future research on intervention sustainability.