Leading the way: competencies of leadership to prevent mis-implementation of public health programs.

Public health agencies are increasingly concerned with ensuring that they are maximizing limited resources by delivering effective programs to enhance population-level health outcomes. Preventing mis-implementation (ending effective activities prematurely or continuing ineffective ones) is necessary to sustain public health efforts and resources needed to improve health and well-being. The purpose of this paper is to identify the important qualities of leadership in preventing mis-implementation of public health programs. In 2019, 45 state health department chronic disease employees were interviewed via phone and audio-recorded, and the conversations were transcribed verbatim. Thematic analysis focused on items related to mis-implementation and the manners in which leadership were involved in continuing ineffective programs. Final themes were based on a Public Health Leadership Competency Framework. The following themes emerged from their interviews regarding the important leadership competencies to prevent mis-implementation: '(1) leadership and communication; (2) collaborative leadership (3) leadership to adapt programs; (4) leadership and organizational learning and development; and (5) political leadership'. This first of its kind study showed the close interrelationship between mis-implementation and leadership. Increased attention to public health leader competencies might help to reduce mis-implementation in public health practice and lead to more effective and efficient use of limited resources.

Patterns and correlates of mis-implementation in state chronic disease public health practice in the United States.

BACKGROUND: Much of the disease burden in the United States is preventable through application of existing knowledge. State-level public health practitioners are in ideal positions to affect programs and policies related to chronic disease, but the extent to which mis-implementation occurring with these programs is largely unknown. Mis-implementation refers to ending effective programs and policies prematurely or continuing ineffective ones. METHODS: A 2018 comprehensive survey assessing the extent of mis-implementation and multi-level influences on mis-implementation was reported by state health departments (SHDs). Questions were developed from previous literature. Surveys were emailed to randomly selected SHD employees across the Unites States. Spearman's correlation and multinomial logistic regression were used to assess factors in mis-implementation. RESULTS: Half (50.7%) of respondents were chronic disease program managers or unit directors. Forty nine percent reported that programs their SHD oversees sometimes, often or always continued ineffective programs. Over 50% also reported that their SHD sometimes or often ended effective programs. The data suggest the strongest correlates and predictors of mis-implementation were at the organizational level. For example, the number of organizational layers impeded decision-making was significant for both continuing ineffective programs (OR=4.70; 95% CI=2.20, 10.04) and ending effective programs (OR=3.23; 95% CI=1.61, 7.40). CONCLUSION: The data suggest that changing certain agency practices may help in minimizing the occurrence of mis-implementation. Further research should focus on adding context to these issues and helping agencies engage in appropriate decision-making. Greater attention to mis-implementation should lead to greater use of effective interventions and more efficient expenditure of resources, ultimately to improve health outcomes.

Perspectives on program mis-implementation among U.S. local public health departments.

BACKGROUND: Public health resources are limited and best used for effective programs. This study explores associations of mis-implementation in public health (ending effective programs or continuing ineffective programs) with organizational supports for evidence-based decision making among U.S. local health departments. METHODS: The national U.S. sample for this cross-sectional study was stratified by local health department jurisdiction population size. One person was invited from each randomly selected local health department: the leader in chronic disease, or the director. Of 600 selected, 579 had valid email addresses; 376 completed the survey (64.9% response). Survey items assessed frequency of and reasons for mis-implementation. Participants indicated agreement with statements on organizational supports for evidence-based decision making (7-point Likert). RESULTS: Thirty percent (30.0%) reported programs often or always ended that should have continued (inappropriate termination); organizational supports for evidence-based decision making were not associated with the frequency of programs ending. The main reason given for inappropriate termination was grant funding ended (86.0%). Fewer (16.4%) reported programs often or always continued that should have ended (inappropriate continuation). Higher perceived organizational supports for evidence-based decision making were associated with less frequent inappropriate continuation (odds ratio = 0.86, 95% confidence interval 0.79, 0.94). All organizational support factors were negatively associated with inappropriate continuation. Top reasons were sustained funding (55.6%) and support from policymakers (34.0%). CONCLUSIONS: Organizational supports for evidence-based decision making may help local health departments avoid continuing programs that should end. Creative mechanisms of support are needed to avoid inappropriate termination. Understanding what influences mis-implementation can help identify supports for de-implementation of ineffective programs so resources can go towards evidence-based programs.

Use and Awareness of The Community Guide in State and Local Health Department Chronic Disease Programs.

INTRODUCTION: The Community Guide (Guide) is a user-friendly, systematic review system that provides information on evidence-based interventions (EBIs) in public health practice. Little is known about what predicts Guide awareness and use in state health departments (SHDs) and local health departments (LHDs). METHODS: We pooled data from 3 surveys (administered in 2016, 2017, and 2018) to employees in chronic disease programs at SHDs and LHDs. Participants (n = 1,039) represented all 50 states. The surveys asked about department practices and individual, organizational, and external factors related to decisions about EBIs. We used χ2 tests of independence for analyses. RESULTS: Eighty-one percent (n = 498) of SHD and 54% (n = 198) of LHD respondents reported their agency uses the Guide. Additionally, 13% of SHD participants reported not being aware of the Guide. Significant relationships were found between reporting using the Guide and academic collaboration, population size, rated importance of forming partnerships, and accreditation. CONCLUSION: Awareness and use of the Guide in LHD and SHD chronic disease programs is widespread. Awareness of the Guide can be vital to implementation practice, because it enhances implementation of EBI practices. However, awareness of the Guide alone is likely not enough for health departments to implement EBIs. Changes at the organizational level, including sharing information about the Guide and providing training on how to best use it, may increase its awareness and use.

The "secret sauce" for a mentored training program: qualitative perspectives of trainees in implementation research for cancer control.

BACKGROUND: Mentored training approaches help build capacity for research through mentoring networks and skill building activities. Capacity for dissemination and implementation (D&I) research in cancer is needed and mentored training programs have been developed. Evaluation of mentored training programs through quantitative approaches often provides us with information on "what" improved for participants. Qualitative approaches provide a deeper understanding of "how" programs work best. METHODS: Qualitative interviews were conducted with 21 fellows of the National Cancer Institute-funded Mentored Training for Dissemination and Implementation in Cancer to gain understanding of their experiences with mentoring received during the program. Fellows were selected from all 55 trained participants based upon their gain in D&I research skills (highest and lowest) and number of collaborative connections in the program network (highest and lowest) reported in previous quantitative surveys. Phone interviews were recorded with permission, transcribed verbatim, and de-identified for analysis. Codes were developed a priori to reflect interview guide concepts followed by further development and iterative coding of three common themes that emerged: 1) program and mentoring structure, 2) importance of mentor attributes, and 3) enhanced capacity: credentials, confidence, credibility and connections. RESULTS: Interviews provided valuable information about program components that worked best and impacts attributed to participation in the program. Fellows reported that regular monthly check-in calls with mentors helped to keep their research moving forward and that group mentoring structures aided in their learning of basic D&I research concepts and their application. Accessible, responsive, and knowledgeable mentors were commonly mentioned by fellows as a key to their success in the program. Fellows mentioned various forms of impact that they attributed to their participation in the program including gaining credibility in the field, a network of peers and experts, and career developments (e.g., collaborative publications and grant funding). CONCLUSIONS: These findings suggest that mentored training works best when mentoring is structured and coupled with applied learning and when respected and dedicated mentors are on board. Increased scientific collaborations and credibility within a recognized network are important trainee experiences that should be considered when designing, implementing, and sustaining mentored training programs.

The Association Between Evidence-Based Decision Making and Accreditation of State Health Departments.

OBJECTIVE: To assess the association between evidence-based decision making, including implementation of evidence-based interventions (EBIs), with accreditation of state health departments through the Public Health Accreditation Board (PHAB). DESIGN: This was a cross-sectional, electronic survey of state health department practitioners. We utilized a survey instrument focused on evidence-based public health, de-implementation, and sustainability of public health programs. Survey questions were organized into 6 domains: (1) demographic information; (2) individual-level skills; (3) decision making on programs ending; (4) decision making on programs continuing; (5) organization/agency capacity; and (6) external influences. PARTICIPANTS: The targeted practitioners were randomly selected from the 3000-person membership of National Association of Chronic Disease Directors and program manager lists from key Centers for Disease Control and Prevention-supported programs in cancer and cancer risk factors. The final target audience for the survey totaled 1329 practitioners, representing all 50 states. MAIN OUTCOME MEASURE(S): The main outcome measures included the strength of association between a state's PHAB accreditation status and variables related to evidence-based public health and use of EBIs that fell within the individual participant skills, organization/agency capacity, and external influences domains. RESULTS: We received 643 valid responses (response rate = 48.4%), representing all 50 states, with 35 states being PHAB accredited. There was a statistically significant association between PHAB accreditation and state health department use of quality improvement processes (P = .002), leadership plans to implement EBIs (P = .009), and leadership reactions to EBI implementation issues (P = .004). Respondents from PHAB-accredited states were significantly more likely than participants from nonaccredited states to report greater engagement with legislators and governors regarding EBIs and 14% less likely to report the inappropriate termination of programs in their work unit (P = .05). CONCLUSIONS: The importance of accreditation relates to both internally focused functions and externally focused activities, especially regarding policy-related impact.

Going beyond the individual: how state-level characteristics relate to HPV vaccine rates in the United States.

BACKGROUND: The human papillomavirus (HPV) vaccine is an underutilized cancer control practice in the United States. Although individual contextual factors are known to impact HPV vaccine coverage rates, the impact of macro-level elements are still unclear. The aim of this analysis was to use HPV vaccination rates to explore the underuse of an evidence-based cancer control intervention and explore broader-level correlates influencing completion rates. METHODS: A comprehensive database was developed using individual-level date from the National Immunization Survey (NIS)-Teen (2016) and state-level data collected from publically available sources to analyze HPV vaccine completion. Multi-level logistic models were fit to identify significant correlates. Level-1 (individual) and level-2 (state) correlates were fitted to a random intercept model. Deviance and AIC assessed model fit and sampling weights were applied. RESULTS: The analysis included 20,495 adolescents from 50 U.S. states and the District of Columbia. Teen age, gender, race/ethnicity, and maternal education were significant individual predictors of HPV completion rates. Significant state-level predictors included sex education policy, religiosity, and HPV vaccine mandate. States with the lowest HPV coverage rates were found to be conservative and highly religious. Little variation in vaccine exemptions and enacted sex and abstinence education polices were observed between states with high and low HPV vaccine coverage suggesting various contextual and situational factors impact HPV vaccine completion rates. CONCLUSIONS: Given that gender, religiosity, political ideology, and education policies are predictors of HPV vaccine completion, the interaction and underlying mechanism of these factors can be used to address the underutilization of the HPV vaccine.

Understanding Misimplementation in U.S. State Health Departments: An Agent-Based Model.

INTRODUCTION: The research goal of this study is to explore why misimplementation occurs in public health agencies and how it can be reduced. Misimplementation is ending effective activities prematurely or continuing ineffective ones, which contributes to wasted resources and suboptimal health outcomes. METHODS: The study team created an agent-based model that represents how information flow, filtered through organizational structure, capacity, culture, and leadership priorities, shapes continuation decisions. This agent-based model used survey data and interviews with state health department personnel across the U.S. between 2014 and 2020; model design and analyses were conducted with substantial input from stakeholders between 2019 and 2021. The model was used experimentally to identify potential approaches for reducing misimplementation. RESULTS: Simulations showed that increasing either organizational evidence-based decision-making capacity or information sharing could reduce misimplementation. Shifting leadership priorities to emphasize effectiveness resulted in the largest reduction, whereas organizational restructuring did not reduce misimplementation. CONCLUSIONS: The model identifies for the first time a specific set of factors and dynamic pathways most likely driving misimplementation and suggests a number of actionable strategies for reducing it. Priorities for training the public health workforce include evidence-based decision making and effective communication. Organizations will also benefit from an intentional shift in leadership decision-making processes. On the basis of this initial, successful application of agent-based model to misimplementation, this work provides a framework for further analyses.

"It's good to feel like you're doing something": a qualitative study examining state health department employees' views on why ineffective programs continue to be implemented in the USA.

BACKGROUND: Mis-implementation, the inappropriate continuation of programs or policies that are not evidence-based or the inappropriate termination of evidence-based programs and policies, can lead to the inefficient use of scarce resources in public health agencies and decrease the ability of these agencies to deliver effective programs and improve population health. Little is known about why mis-implementation occurs, which is needed to understand how to address it. This study sought to understand the state health department practitioners' perspectives about what makes programs ineffective and the reasons why ineffective programs continue. METHODS: Eight state health departments (SHDs) were selected to participate in telephone-administered qualitative interviews about decision-making around ending or continuing programs. States were selected based on geographic representation and on their level of mis-implementation (low and high) categorized from our previous national survey. Forty-four SHD chronic disease staff participated in interviews, which were audio-recorded and transcribed verbatim. Transcripts were consensus coded, and themes were identified and summarized. This paper presents two sets of themes, related to (1) what makes a program ineffective and (2) why ineffective programs continue to be implemented according to SHD staff. RESULTS: Participants considered programs ineffective if they were not evidence-based or if they did not fit well within the population; could not be implemented well due to program restraints or a lack of staff time and resources; did not reach those who could most benefit from the program; or did not show the expected program outcomes through evaluation. Practitioners described several reasons why ineffective programs continued to be implemented, including concerns about damaging the relationships with partner organizations, the presence of program champions, agency capacity, and funding restrictions. CONCLUSIONS: The continued implementation of ineffective programs occurs due to a number of interrelated organizational, relational, human resources, and economic factors. Efforts should focus on preventing mis-implementation since it limits public health agencies' ability to conduct evidence-based public health, implement evidence-based programs effectively, and reduce the high burden of chronic diseases. The use of evidence-based decision-making in public health agencies and supporting adaptation of programs to improve their fit may prevent mis-implementation. Future work should identify effective strategies to reduce mis-implementation, which can optimize public health practice and improve population health.

Program adaptation by health departments.

Introduction: The dissemination of evidence-based interventions (i.e., programs, practices, and policies) is a core function of US state health departments (SHDs). However, interventions are originally designed and tested with a specific population and context. Hence, adapting the intervention to meet the real-world circumstances and population's needs can increase the likelihood of achieving the expected health outcomes for the target population from the implemented intervention. This study identified how SHD employees decide to adapt public health programs and what influences decisions on how to adapt them. Materials and methods: SHD employees (n = 45) were interviewed using a qualitative semi-structured interview guide. Telephone interviews were audio-recorded and transcribed verbatim. The transcripts were consensus-coded and themes were identified using thematic analysis. Several themes aligned with the Model for Adaptation Design and Impact. Results: Data, outcomes, and health department evaluations influenced decisions to adapt a program (pre-adaptation), and reasons to adapt a program included organizational and sociopolitical contextual factors. SHD middle-level managers, program managers and staff, and local agencies were involved in the decisions to adapt the programs. Finally, the goals for adapting a program included enhancing effectiveness/outcomes, reach and satisfaction with the program; funding; and partner engagement. After SHD employees decided to adapt a program, data and evidence guided the changes. Program staff and evaluators were engaged in the adaptation process. Program managers consulted partners to gather ideas on how best to adapt a program based on partners' experiences implementing the program and obtaining community input. Lastly, program managers also received input on adapting content and context from coalition meetings and periodic technical assistance calls. Discussion: The findings related to decisions to adapt public health programs provide practitioners with considerations for adapting them. Findings reaffirm the importance of promoting public health competencies in program evaluation and adaptation, as well as systematically documenting and evaluating the adaptation processes. In addition, the themes could be studied in future research as mechanisms, mediators, and moderators to implementation outcomes.

Approaches for Ending Ineffective Programs: Strategies From State Public Health Practitioners.

Background: Public health agencies are increasingly concerned with ensuring they are maximizing limited resources by delivering evidence-based programs to enhance population-level chronic disease outcomes. Yet, there is little guidance on how to end ineffective programs that continue in communities. The purpose of this analysis is to identify what strategies public health practitioners perceive to be effective in de-implementing, or reducing the use of, ineffective programs. Methods: From March to July 2019, eight states were selected to participate in qualitative interviews from our previous national survey of US state health department (SHD) chronic disease practitioners on program decision making. This analysis examined responses to a question about "…advice for others who want to end an ineffective program." Forty-five SHD employees were interviewed via phone. Interviews were audio-recorded, and the conversations were transcribed verbatim. All transcripts were consensus coded, and themes were identified and summarized. Results: Participants were program managers or section directors who had on average worked 11 years at their agency and 15 years in public health. SHD employees provided several strategies they perceived as effective for de-implementation. The major themes were: (1) collect and rely on evaluation data; (2) consider if any of the programs can be saved; (3) transparently communicate and discuss program adjustments; (4) be tactful and respectful of partner relationships; (5) communicate in a way that is meaningful to your audience. Conclusions: This analysis provides insight into how experienced SHD practitioners recommend ending ineffective programs which may be useful for others working at public health agencies. As de-implementation research is limited in public health settings, this work provides a guiding point for future researchers to systematically assess these strategies and their effects on public health programming.

Building the Next Generation of Researchers: Mentored Training in Dissemination and Implementation Science.

PROBLEM: Dissemination and implementation (D&I) science provides the tools needed to close the gap between known intervention strategies and their effective application. The authors report on the Mentored Training for Dissemination and Implementation Research in Cancer (MT-DIRC) program-a D&I training program for postdoctoral or early-career cancer prevention and control scholars. APPROACH: MT-DIRC was a 2-year training institute in which fellows attended 2 annual Summer Institutes and other conferences and received didactic, group, and individual instruction; individualized mentoring; and other supports (e.g., pilot funding). A quasi-experimental design compared changes in 3 areas: mentoring, skills, and network composition. To evaluate mentoring and D&I skills, data from fellows on their mentors' mentoring competencies, their perspectives on the importance of and satisfaction with mentoring priority areas, and their self-rated skills in D&I competency domains were collected. Network composition data were collected from faculty and fellows for 3 core social network domains: contact, mentoring, and collaboration. Paired t tests (mentoring), linear mixed models (skills), and descriptive analyses (network composition) were performed. OUTCOMES: Mentors were rated as highly competent across all mentoring competencies, and each mentoring priority area showed reductions in gaps between satisfaction and importance between the 6 and 18 months post-first Summer Institute. Fellows' self-rated skills in D&I competencies improved significantly in all domains over time (range: 42.5%-52.9% increase from baseline to 18 months post-first Summer Institute). Mentorship and collaboration networks grew over time, with the highest number of collaboration network ties for scholarly manuscripts (n = 199) in 2018 and for research projects (n = 160) in 2019. NEXT STEPS: Building on study findings and existing literature, mentored training of scholars is an important approach for building D&I skills and networks, and thus to better applying the vast amount of available intervention evidence to benefit cancer control.

Rugged landscapes: complexity and implementation science.

BACKGROUND: Mis-implementation-defined as failure to successfully implement and continue evidence-based programs-is widespread in public health practice. Yet the causes of this phenomenon are poorly understood. METHODS: We develop an agent-based computational model to explore how complexity hinders effective implementation. The model is adapted from the evolutionary biology literature and incorporates three distinct complexities faced in public health practice: dimensionality, ruggedness, and context-specificity. Agents in the model attempt to solve problems using one of three approaches-Plan-Do-Study-Act (PDSA), evidence-based interventions (EBIs), and evidence-based decision-making (EBDM). RESULTS: The model demonstrates that the most effective approach to implementation and quality improvement depends on the underlying nature of the problem. Rugged problems are best approached with a combination of PDSA and EBI. Context-specific problems are best approached with EBDM. CONCLUSIONS: The model's results emphasize the importance of adapting one's approach to the characteristics of the problem at hand. Evidence-based decision-making (EBDM), which combines evidence from multiple independent sources with on-the-ground local knowledge, is a particularly potent strategy for implementation and quality improvement.

Mentored training and its association with dissemination and implementation research output: a quasi-experimental evaluation.

BACKGROUND: There is a continued need to evaluate training programs in dissemination and implementation (D&I) research. Scientific products yielded from trainees are an important and objective measure to understand the capacity growth within the D&I field. This study evaluates our mentored training program in terms of scientific productivity among applicants. METHODS: Post-doctoral and early-career cancer researchers were recruited and applied to the R25 Mentored Training for Dissemination and Implementation Research in Cancer (MT-DIRC) between 2014 and 2017. Using application details and publicly available bibliometric and funding data, we compared selected fellows with unsuccessful applicants (nonfellows). We extracted Scopus citations and US federal grant funding records for all applicants (N = 102). Funding and publication abstracts were de-identified and coded for D&I focus and aggregated to the applicant level for analysis. Logistic regression models were explored separately for the odds of (1) a D&I publication and (2) US federal grant funding post year of application among fellows (N = 55) and nonfellows (N = 47). Additional models were constructed to include independent variables that attenuated the program's association by 5% or more. Only US-based applicants (N = 87) were included in the grant funding analysis. RESULTS: Fellows and nonfellows were similar across several demographic characteristics. Fellows were more than 3 times more likely than nonfellows to have grant funding after MT-DIRC application year (OR 3.2; 95% CI 1.1-11.0) while controlling for time since application year; the association estimate was 3.1 (95% CI 0.98-11.0) after adjusting for both cancer research area and previous grant funding. For publications, fellows were almost 4 times more likely to publish D&I-focused work adjusting for time (OR 3.8; 95% CI 1.7-9.0). This association lessened after adjusting for previous D&I publication and years since undergraduate degree (OR 2.9; 95% CI 1.2-7.5). CONCLUSIONS: We document the association of a mentored training approach with built-in networks of peers to yield productive D&I researchers. Future evaluation efforts could be expanded to include other forms of longer-term productivity such as policy or practice change as additional objective measures. D&I research trainings in the USA and internationally should consider common evaluation measures.

Toward optimal implementation of cancer prevention and control programs in public health: a study protocol on mis-implementation.

BACKGROUND: Much of the cancer burden in the USA is preventable, through application of existing knowledge. State-level funders and public health practitioners are in ideal positions to affect programs and policies related to cancer control. Mis-implementation refers to ending effective programs and policies prematurely or continuing ineffective ones. Greater attention to mis-implementation should lead to use of effective interventions and more efficient expenditure of resources, which in the long term, will lead to more positive cancer outcomes. METHODS: This is a three-phase study that takes a comprehensive approach, leading to the elucidation of tactics for addressing mis-implementation. Phase 1: We assess the extent to which mis-implementation is occurring among state cancer control programs in public health. This initial phase will involve a survey of 800 practitioners representing all states. The programs represented will span the full continuum of cancer control, from primary prevention to survivorship. Phase 2: Using data from phase 1 to identify organizations in which mis-implementation is particularly high or low, the team will conduct eight comparative case studies to get a richer understanding of mis-implementation and to understand contextual differences. These case studies will highlight lessons learned about mis-implementation and identify hypothesized drivers. Phase 3: Agent-based modeling will be used to identify dynamic interactions between individual capacity, organizational capacity, use of evidence, funding, and external factors driving mis-implementation. The team will then translate and disseminate findings from phases 1 to 3 to practitioners and practice-related stakeholders to support the reduction of mis-implementation. DISCUSSION: This study is innovative and significant because it will (1) be the first to refine and further develop reliable and valid measures of mis-implementation of public health programs; (2) bring together a strong, transdisciplinary team with significant expertise in practice-based research; (3) use agent-based modeling to address cancer control implementation; and (4) use a participatory, evidence-based, stakeholder-driven approach that will identify key leverage points for addressing mis-implementation among state public health programs. This research is expected to provide replicable computational simulation models that can identify leverage points and public health system dynamics to reduce mis-implementation in cancer control and may be of interest to other health areas.

Variation in Research Designs Used to Test the Effectiveness of Dissemination and Implementation Strategies: A Review.

BACKGROUND: The need for optimal study designs in dissemination and implementation (D&I) research is increasingly recognized. Despite the wide range of study designs available for D&I research, we lack understanding of the types of designs and methodologies that are routinely used in the field. This review assesses the designs and methodologies in recently proposed D&I studies and provides resources to guide design decisions. METHODS: We reviewed 404 study protocols published in the journal Implementation Science from 2/2006 to 9/2017. Eligible studies tested the efficacy or effectiveness of D&I strategies (i.e., not effectiveness of the underlying clinical or public health intervention); had a comparison by group and/or time; and used ≥1 quantitative measure. Several design elements were extracted: design category (e.g., randomized); design type [e.g., cluster randomized controlled trial (RCT)]; data type (e.g., quantitative); D&I theoretical framework; levels of treatment assignment, intervention, and measurement; and country in which the research was conducted. Each protocol was double-coded, and discrepancies were resolved through discussion. RESULTS: Of the 404 protocols reviewed, 212 (52%) studies tested one or more implementation strategy across 208 manuscripts, therefore meeting inclusion criteria. Of the included studies, 77% utilized randomized designs, primarily cluster RCTs. The use of alternative designs (e.g., stepped wedge) increased over time. Fewer studies were quasi-experimental (17%) or observational (6%). Many study design categories (e.g., controlled pre-post, matched pair cluster design) were represented by only one or two studies. Most articles proposed quantitative and qualitative methods (61%), with the remaining 39% proposing only quantitative. Half of protocols (52%) reported using a theoretical framework to guide the study. The four most frequently reported frameworks were Consolidated Framework for Implementing Research and RE-AIM (n = 16 each), followed by Promoting Action on Research Implementation in Health Services and Theoretical Domains Framework (n = 12 each). CONCLUSION: While several novel designs for D&I research have been proposed (e.g., stepped wedge, adaptive designs), the majority of the studies in our sample employed RCT designs. Alternative study designs are increasing in use but may be underutilized for a variety of reasons, including preference of funders or lack of awareness of these designs. Promisingly, the prevalent use of quantitative and qualitative methods together reflects methodological innovation in newer D&I research.

Training scholars in dissemination and implementation research for cancer prevention and control: a mentored approach.

BACKGROUND: As the field of D&I (dissemination and implementation) science grows to meet the need for more effective and timely applications of research findings in routine practice, the demand for formalized training programs has increased concurrently. The Mentored Training for Dissemination and Implementation Research in Cancer (MT-DIRC) Program aims to build capacity in the cancer control D&I research workforce, especially among early career researchers. This paper outlines the various components of the program and reports results of systematic evaluations to ascertain its effectiveness. METHODS: Essential features of the program include selection of early career fellows or more experienced investigators with a focus relevant to cancer control transitioning to a D&I research focus, a 5-day intensive training institute, ongoing peer and senior mentoring, mentored planning and work on a D&I research proposal or project, limited pilot funding, and training and ongoing improvement activities for mentors. The core faculty and staff members of the MT-DIRC program gathered baseline and ongoing evaluation data regarding D&I skill acquisition and mentoring competency through participant surveys and analyzed it by iterative collective reflection. RESULTS: A majority (79%) of fellows are female, assistant professors (55%); 59% are in allied health disciplines, and 48% focus on cancer prevention research. Forty-three D&I research competencies were assessed; all improved from baseline to 6 and 18 months. These effects were apparent across beginner, intermediate, and advanced initial D&I competency levels and across the competency domains. Mentoring competency was rated very highly by the fellows--higher than rated by the mentors themselves. The importance of different mentoring activities, as rated by the fellows, was generally congruent with their satisfaction with the activities, with the exception of relatively greater satisfaction with the degree of emotional support and relatively lower satisfaction for skill building and opportunity initially. CONCLUSIONS: These first years of MT-DIRC demonstrated the program's ability to attract, engage, and improve fellows' competencies and skills and implement a multicomponent mentoring program that was well received. This account of the program can serve as a basis for potential replication and evolution of this model in training future D&I science researchers.

Dissemination and Implementation Science Training Needs: Insights From Practitioners and Researchers.

INTRODUCTION: Dissemination and implementation research training has great potential to improve the impact and reach of health-related research; however, research training needs from the end user perspective are unknown. This paper identifies and prioritizes dissemination and implementation research training needs. METHODS: A diverse sample of researchers, practitioners, and policymakers was invited to participate in Concept Mapping in 2014-2015. Phase 1 (Brainstorming) gathered participants' responses to the prompt: To improve the impact of research evidence in practice and policy settings, a skill in which researchers need more training is… The resulting statement list was edited and included subsequent phases. Phase 2 (Sorting) asked participants to sort each statement into conceptual piles. In Phase 3 (Rating), participants rated the difficulty and importance of incorporating each statement into a training curriculum. A multidisciplinary team synthesized and interpreted the results in 2015-2016. RESULTS: During Brainstorming, 60 researchers and 60 practitioners/policymakers contributed 274 unique statements. Twenty-nine researchers and 16 practitioners completed sorting and rating. Nine concept clusters were identified: Communicating Research Findings, Improve Practice Partnerships, Make Research More Relevant, Strengthen Communication Skills, Develop Research Methods and Measures, Consider and Enhance Fit, Build Capacity for Research, and Understand Multilevel Context. Though researchers and practitioners had high agreement about importance (r =0.93) and difficulty (r =0.80), ratings differed for several clusters (e.g., Build Capacity for Research). CONCLUSIONS: Including researcher and practitioner perspectives in competency development for dissemination and implementation research identifies skills and capacities needed to conduct and communicate contextualized, meaningful, and relevant research.

Sustainability of evidence-based healthcare: research agenda, methodological advances, and infrastructure support.

BACKGROUND: Little is known about how well or under what conditions health innovations are sustained and their gains maintained once they are put into practice. Implementation science typically focuses on uptake by early adopters of one healthcare innovation at a time. The later-stage challenges of scaling up and sustaining evidence-supported interventions receive too little attention. This project identifies the challenges associated with sustainability research and generates recommendations for accelerating and strengthening this work. METHODS: A multi-method, multi-stage approach, was used: (1) identifying and recruiting experts in sustainability as participants, (2) conducting research on sustainability using concept mapping, (3) action planning during an intensive working conference of sustainability experts to expand the concept mapping quantitative results, and (4) consolidating results into a set of recommendations for research, methodological advances, and infrastructure building to advance understanding of sustainability. Participants comprised researchers, funders, and leaders in health, mental health, and public health with shared interest in the sustainability of evidence-based health care. RESULTS: Prompted to identify important issues for sustainability research, participants generated 91 distinct statements, for which a concept mapping process produced 11 conceptually distinct clusters. During the conference, participants built upon the concept mapping clusters to generate recommendations for sustainability research. The recommendations fell into three domains: (1) pursue high priority research questions as a unified agenda on sustainability; (2) advance methods for sustainability research; (3) advance infrastructure to support sustainability research. CONCLUSIONS: Implementation science needs to pursue later-stage translation research questions required for population impact. Priorities include conceptual consistency and operational clarity for measuring sustainability, developing evidence about the value of sustaining interventions over time, identifying correlates of sustainability along with strategies for sustaining evidence-supported interventions, advancing the theoretical base and research designs for sustainability research, and advancing the workforce capacity, research culture, and funding mechanisms for this important work.

Concocting that Magic Elixir: Successful Grant Application Writing in Dissemination and Implementation Research.

BACKGROUND: This paper reports core competencies for dissemination and implementation (D&I) grant application writing and provides tips for writing a successful proposal. METHODS: Two related phases were used to collect the data: a card sorting process among D&I researchers and an expert review among a smaller set of researchers. Card sorting was completed by 123 respondents. In the second phase, a series of grant application writing tips were developed based on the combined 170 years of grant review experience of the writing team. RESULTS: The card sorting resulted in 12 core competencies for D&I grant application writing that covered the main sections in a grant application to the US National Institutes of Health: (a) specific aims that provide clear rationale, objectives, and an overview of the research plan; (b) significance that frames and justifies the importance of a D&I question; (c) innovation that articulates novel products and new knowledge; and (d) approach that uses a relevant D&I model, addresses measurement and the D&I context, and includes an analysis plan well-tied to the aims and measures. CONCLUSIONS: Writing a successful D&I grant application is a skill that can be learned with experience and attention to the core competencies articulated in this paper.