Understanding Misimplementation in U.S. State Health Departments: An Agent-Based Model.

INTRODUCTION: The research goal of this study is to explore why misimplementation occurs in public health agencies and how it can be reduced. Misimplementation is ending effective activities prematurely or continuing ineffective ones, which contributes to wasted resources and suboptimal health outcomes. METHODS: The study team created an agent-based model that represents how information flow, filtered through organizational structure, capacity, culture, and leadership priorities, shapes continuation decisions. This agent-based model used survey data and interviews with state health department personnel across the U.S. between 2014 and 2020; model design and analyses were conducted with substantial input from stakeholders between 2019 and 2021. The model was used experimentally to identify potential approaches for reducing misimplementation. RESULTS: Simulations showed that increasing either organizational evidence-based decision-making capacity or information sharing could reduce misimplementation. Shifting leadership priorities to emphasize effectiveness resulted in the largest reduction, whereas organizational restructuring did not reduce misimplementation. CONCLUSIONS: The model identifies for the first time a specific set of factors and dynamic pathways most likely driving misimplementation and suggests a number of actionable strategies for reducing it. Priorities for training the public health workforce include evidence-based decision making and effective communication. Organizations will also benefit from an intentional shift in leadership decision-making processes. On the basis of this initial, successful application of agent-based model to misimplementation, this work provides a framework for further analyses.

Practices Among Local Public Health Agencies to Support Evidence-Based Decision Making: A Qualitative Study.

OBJECTIVES: Evidence-based decision making (EBDM) capacity in local public health departments is foundational to meeting both organizational and individual competencies and fulfilling expanded roles. In addition to on-the-job training, organizational supports are needed to prepare staff; yet, less is known in this area. This qualitative study explores supportive management practices instituted as part of a training and technical assistance intervention. DESIGN: This qualitative study used a semistructured interview guide to elicit participants' descriptions and perceptions via key informant interviews. Verbatim transcripts were coded and thematic analyses were conducted. SETTING: Local public health departments in a US Midwestern state participated in the project. PARTICIPANTS: Seventeen middle managers and staff from 4 local health departments participated in remote, audio-recorded interviews. INTERVENTION: Following delivery of a 3½-day in-person training, the study team met with health department leadership teams for department selection of supportive agency policies and procedures to revise or newly create. Periodic remote meetings included collaborative problem-solving, sharing of informational resources, and encouragement. MAIN OUTCOME MEASURES: Included management practices instituted to support EBDM and impact on day-to-day work as described by the interview participants. RESULTS: Leadership and middle management practices deemed most helpful included dedicating staff; creating specific guidelines; setting expectations; and providing trainings, resources, and guidance. Health departments with a preexisting supportive organizational culture and climat e were able to move more quickly and fully to integrate supportive management practices. Workforce development included creation of locally tailored overviews for all staff members and onboarding of new staff. Staff wanted additional hands-on skill-building trainings. Several worked with partners to incorporate evidence-based processes into community health improvement plans. CONCLUSIONS: Ongoing on-the-job experiential learning is needed to integrate EBDM principles into day-to-day public health practice. Management practices established by leadership teams and middle managers can create supportive work environments for EBDM integration.

Program adaptation by health departments.

Introduction: The dissemination of evidence-based interventions (i.e., programs, practices, and policies) is a core function of US state health departments (SHDs). However, interventions are originally designed and tested with a specific population and context. Hence, adapting the intervention to meet the real-world circumstances and population's needs can increase the likelihood of achieving the expected health outcomes for the target population from the implemented intervention. This study identified how SHD employees decide to adapt public health programs and what influences decisions on how to adapt them. Materials and methods: SHD employees (n = 45) were interviewed using a qualitative semi-structured interview guide. Telephone interviews were audio-recorded and transcribed verbatim. The transcripts were consensus-coded and themes were identified using thematic analysis. Several themes aligned with the Model for Adaptation Design and Impact. Results: Data, outcomes, and health department evaluations influenced decisions to adapt a program (pre-adaptation), and reasons to adapt a program included organizational and sociopolitical contextual factors. SHD middle-level managers, program managers and staff, and local agencies were involved in the decisions to adapt the programs. Finally, the goals for adapting a program included enhancing effectiveness/outcomes, reach and satisfaction with the program; funding; and partner engagement. After SHD employees decided to adapt a program, data and evidence guided the changes. Program staff and evaluators were engaged in the adaptation process. Program managers consulted partners to gather ideas on how best to adapt a program based on partners' experiences implementing the program and obtaining community input. Lastly, program managers also received input on adapting content and context from coalition meetings and periodic technical assistance calls. Discussion: The findings related to decisions to adapt public health programs provide practitioners with considerations for adapting them. Findings reaffirm the importance of promoting public health competencies in program evaluation and adaptation, as well as systematically documenting and evaluating the adaptation processes. In addition, the themes could be studied in future research as mechanisms, mediators, and moderators to implementation outcomes.

Facilitators of Multisector Collaboration for Delivering Cancer Control Interventions in Rural Communities: A Descriptive Qualitative Study.

PURPOSE AND OBJECTIVES: Multisector collaboration is a widely promoted strategy to increase equitable availability, access, and use of healthy foods, safe places for physical activity, social supports, and preventive health care services. Yet fewer studies and resources exist for collaboration among governmental and nongovernmental agencies to address public problems in rural areas, despite an excess burden of risk factors for cancer morbidity and mortality. We aimed to learn about cancer prevention activities and collaboration facilitators among rural informal interagency networks. EVALUATION METHODS: In 2020, researchers conducted semistructured interviews with staff from rural public health and social services agencies, community health centers, and extension offices. Agency staff were from 5 service areas across 27 rural counties in Missouri and Illinois with high poverty rates and excess cancer risks and mortality. We conducted a thematic analysis to code interview transcripts and identify key themes. RESULTS: Exchanging information, cohosting annual or one-time events, and promoting other agencies' services and programs were the most commonly described collaborative activities among the 32 participants interviewed. Participants indicated a desire to improve collaborations by writing more grants together to codevelop ongoing prevention programs and further share resources. Participants expressed needs to increase community outreach, improve referral systems, and expand screenings. We identified 5 facilitator themes: commitment to address community needs, mutual willingness to collaborate, long-standing relationships, smaller community structures, and necessity of leveraging limited resources. Challenges included lack of funding and time, long travel distances, competing priorities, difficulty replacing staff in remote communities, and jurisdictional boundaries. Although the COVID-19 pandemic further limited staff availability for collaboration, participants noted benefits of remote collaborative meetings. IMPLICATIONS FOR PUBLIC HEALTH: Rural areas need consistent funding and other resources to support health-improving multisector initiatives. Existing strengths found in the rural underresourced areas can facilitate multisector collaborations for cancer prevention, including long-standing relationships, small community structures, and the need to leverage limited resources.

The power of partnerships: state public health department multisector collaborations in major chronic disease programme areas in the United States.

BACKGROUND: Multisector collaboration between state public health departments (SHDs) and diverse community partners is increasingly recognized as important for promoting positive public health outcomes, addressing social determinants of health, and reducing health inequalities. This study investigates collaborations between SHDs in the United States and different types of organizations addressing chronic disease in and outside of the health sector. METHODS: SHD employees were randomly selected from the National Association of Chronic Disease Directors membership list for participation in an online survey. Participants were asked about their primary chronic disease work unit (cancer, obesity, tobacco, diabetes, cardiovascular disease, and others), as well as their work unit collaborations (exchange of information/cooperation in activities) with organizations in health and non-health sectors. As a measure of the different organizations SHDs collaborated with in health and non-health sectors, a collaboration heterogeneity score for each programme area was calculated. One-way analysis of variance (ANOVA) with Tukey's post hoc tests were used to assess differences in collaborator heterogeneity between programme areas. RESULTS: A total of 574 participants were surveyed. Results indicated that the cancer programme area, along with diabetes and cardiovascular disease, had significantly less collaboration heterogeneity with organizations outside of the health sector compared to the obesity and tobacco programme areas. CONCLUSIONS: While collaborations with health sector organizations are commonly reported, public health departments can increase collaboration with sectors outside of health to more fully address chronic disease prevention.

How to "Start Small and Just Keep Moving Forward": Mixed Methods Results From a Stepped-Wedge Trial to Support Evidence-Based Processes in Local Health Departments.

Background: Local health departments (LHDs) in the United States are charged with preventing disease and promoting health in their respective communities. Understanding and addressing what supports LHD's need to foster a climate and culture supportive of evidence-based decision making (EBDM) processes can enhance delivery of effective practices and services. Methods: We employed a stepped-wedge trial design to test staggered delivery of implementation supports in 12 LHDs (Missouri, USA) to expand capacity for EBDM processes. The intervention was an in-person training in EBDM and continued support by the research team over 24 months (March 2018-February 2020). We used a mixed-methods approach to evaluate: (1) individuals' EBDM skills, (2) organizational supports for EBDM, and (3) administered evidence-based interventions. LHD staff completed a quantitative survey at 4 time points measuring their EBDM skills, organizational supports, and evidence-based interventions. We selected 4 LHDs with high contact and engagement during the intervention period to interview staff (n = 17) about facilitators and barriers to EBDM. We used mixed-effects linear regression to examine quantitative survey outcomes. Interviews were transcribed verbatim and coded through a dual independent process. Results: Overall, 519 LHD staff were eligible and invited to complete quantitative surveys during control periods and 593 during intervention (365 unique individuals). A total of 434 completed during control and 492 during intervention (83.6 and 83.0% response, respectively). In both trial modes, half the participants had at least a master's degree (49.7-51.7%) and most were female (82.1-83.8%). No significant intervention effects were found in EBDM skills or in implementing evidence-based interventions. Two organizational supports scores decreased in intervention vs. control periods: awareness (-0.14, 95% CI -0.26 to -0.01, p < 0.05) and climate cultivation (-0.14, 95% CI -0.27 to -0.02, p < 0.05) but improved over time among all participants. Interviewees noted staff turnover, limited time, resources and momentum as challenges to continue EBDM work. Setting expectations, programmatic reviews, and pre-existing practices were seen as facilitators. Conclusions: Challenges (e.g., turnover, resources) may disrupt LHDs' abilities to fully embed organizational processes which support EBDM. This study and related literature provides understanding on how best to support LHDs in building capacity to use and sustain evidence-based practices.

Mapping the Lay of the Land: Using Interactive Network Analytic Tools for Collaboration in Rural Cancer Prevention and Control.

BACKGROUND: Cancer mortality rates in the United States are higher in rural than urban areas, especially for colorectal cancer. Modifiable cancer risks (e.g., tobacco use, obesity) are more prevalent among U.S. rural than urban residents. Social network analyses are common, yet rural informal collaborative networks for cancer prevention and control and practitioner uses of network findings are less well understood. METHODS: In five service areas in rural Missouri and Illinois, we conducted a network survey of informal multisector networks among agencies that address cancer risk (N = 152 individuals). The survey asked about contact, collaborative activities, and referrals. We calculated descriptive network statistics and disseminated network visualizations with rural agencies through infographics and interactive Network Navigator platforms. We also collected feedback on uses of network findings from agency staff (N = 14). RESULTS: Service areas had more connections (average degree) for exchanging information than for more time-intensive collaborative activities of co-developing and sustaining ongoing services and programs, and co-developing and sharing resources. On average, collaborative activities were not dependent on just a few agencies to bridge gaps to hold networks together. Users found the network images and information useful for identifying gaps, planning which relationships to establish or enhance to strengthen certain collaborative activities and cross-referrals, and showing network strengths to current and potential funders. CONCLUSIONS: Rural informal cancer prevention and control networks in this study are highly connected and largely decentralized. IMPACT: Disseminating network findings help ensure usefulness to rural health and social service practitioners who address cancer risks.

"It's good to feel like you're doing something": a qualitative study examining state health department employees' views on why ineffective programs continue to be implemented in the USA.

BACKGROUND: Mis-implementation, the inappropriate continuation of programs or policies that are not evidence-based or the inappropriate termination of evidence-based programs and policies, can lead to the inefficient use of scarce resources in public health agencies and decrease the ability of these agencies to deliver effective programs and improve population health. Little is known about why mis-implementation occurs, which is needed to understand how to address it. This study sought to understand the state health department practitioners' perspectives about what makes programs ineffective and the reasons why ineffective programs continue. METHODS: Eight state health departments (SHDs) were selected to participate in telephone-administered qualitative interviews about decision-making around ending or continuing programs. States were selected based on geographic representation and on their level of mis-implementation (low and high) categorized from our previous national survey. Forty-four SHD chronic disease staff participated in interviews, which were audio-recorded and transcribed verbatim. Transcripts were consensus coded, and themes were identified and summarized. This paper presents two sets of themes, related to (1) what makes a program ineffective and (2) why ineffective programs continue to be implemented according to SHD staff. RESULTS: Participants considered programs ineffective if they were not evidence-based or if they did not fit well within the population; could not be implemented well due to program restraints or a lack of staff time and resources; did not reach those who could most benefit from the program; or did not show the expected program outcomes through evaluation. Practitioners described several reasons why ineffective programs continued to be implemented, including concerns about damaging the relationships with partner organizations, the presence of program champions, agency capacity, and funding restrictions. CONCLUSIONS: The continued implementation of ineffective programs occurs due to a number of interrelated organizational, relational, human resources, and economic factors. Efforts should focus on preventing mis-implementation since it limits public health agencies' ability to conduct evidence-based public health, implement evidence-based programs effectively, and reduce the high burden of chronic diseases. The use of evidence-based decision-making in public health agencies and supporting adaptation of programs to improve their fit may prevent mis-implementation. Future work should identify effective strategies to reduce mis-implementation, which can optimize public health practice and improve population health.

Quantitative measures used in empirical evaluations of mental health policy implementation: A systematic review.

Background: Mental health is a critical component of wellness. Public policies present an opportunity for large-scale mental health impact, but policy implementation is complex and can vary significantly across contexts, making it crucial to evaluate implementation. The objective of this study was to (1) identify quantitative measurement tools used to evaluate the implementation of public mental health policies; (2) describe implementation determinants and outcomes assessed in the measures; and (3) assess the pragmatic and psychometric quality of identified measures. Method: Guided by the Consolidated Framework for Implementation Research, Policy Implementation Determinants Framework, and Implementation Outcomes Framework, we conducted a systematic review of peer-reviewed journal articles published in 1995-2020. Data extracted included study characteristics, measure development and testing, implementation determinants and outcomes, and measure quality using the Psychometric and Pragmatic Evidence Rating Scale. Results: We identified 34 tools from 25 articles, which were designed for mental health policies or used to evaluate constructs that impact implementation. Many measures lacked information regarding measurement development and testing. The most assessed implementation determinants were readiness for implementation, which encompassed training (n = 20, 57%) and other resources (n = 12, 34%), actor relationships/networks (n = 15, 43%), and organizational culture and climate (n = 11, 31%). Fidelity was the most prevalent implementation outcome (n = 9, 26%), followed by penetration (n = 8, 23%) and acceptability (n = 7, 20%). Apart from internal consistency and sample norms, psychometric properties were frequently unreported. Most measures were accessible and brief, though minimal information was provided regarding interpreting scores, handling missing data, or training needed to administer tools. Conclusions: This work contributes to the nascent field of policy-focused implementation science by providing an overview of existing measurement tools used to evaluate mental health policy implementation and recommendations for measure development and refinement. To advance this field, more valid, reliable, and pragmatic measures are needed to evaluate policy implementation and close the policy-to-practice gap. Plain Language Summary: Mental health is a critical component of wellness, and public policies present an opportunity to improve mental health on a large scale. Policy implementation is complex because it involves action by multiple entities at several levels of society. Policy implementation is also challenging because it can be impacted by many factors, such as political will, stakeholder relationships, and resources available for implementation. Because of these factors, implementation can vary between locations, such as states or countries. It is crucial to evaluate policy implementation, thus we conducted a systematic review to identify and evaluate the quality of measurement tools used in mental health policy implementation studies. Our search and screening procedures resulted in 34 measurement tools. We rated their quality to determine if these tools were practical to use and would yield consistent (i.e., reliable) and accurate (i.e., valid) data. These tools most frequently assessed whether implementing organizations complied with policy mandates and whether organizations had the training and other resources required to implement a policy. Though many were relatively brief and available at little-to-no cost, these findings highlight that more reliable, valid, and practical measurement tools are needed to assess and inform mental health policy implementation. Findings from this review can guide future efforts to select or develop policy implementation measures.

Approaches for Ending Ineffective Programs: Strategies From State Public Health Practitioners.

Background: Public health agencies are increasingly concerned with ensuring they are maximizing limited resources by delivering evidence-based programs to enhance population-level chronic disease outcomes. Yet, there is little guidance on how to end ineffective programs that continue in communities. The purpose of this analysis is to identify what strategies public health practitioners perceive to be effective in de-implementing, or reducing the use of, ineffective programs. Methods: From March to July 2019, eight states were selected to participate in qualitative interviews from our previous national survey of US state health department (SHD) chronic disease practitioners on program decision making. This analysis examined responses to a question about "…advice for others who want to end an ineffective program." Forty-five SHD employees were interviewed via phone. Interviews were audio-recorded, and the conversations were transcribed verbatim. All transcripts were consensus coded, and themes were identified and summarized. Results: Participants were program managers or section directors who had on average worked 11 years at their agency and 15 years in public health. SHD employees provided several strategies they perceived as effective for de-implementation. The major themes were: (1) collect and rely on evaluation data; (2) consider if any of the programs can be saved; (3) transparently communicate and discuss program adjustments; (4) be tactful and respectful of partner relationships; (5) communicate in a way that is meaningful to your audience. Conclusions: This analysis provides insight into how experienced SHD practitioners recommend ending ineffective programs which may be useful for others working at public health agencies. As de-implementation research is limited in public health settings, this work provides a guiding point for future researchers to systematically assess these strategies and their effects on public health programming.

A systematic review of school health policy measurement tools: implementation determinants and outcomes.

BACKGROUND: Governments in some countries or states/provinces mandate school-based policies intended to improve the health and well-being of primary and secondary students and in some cases the health of school staff. Examples include mandating a minimum time spent per week in programmed physical activity, mandating provision of healthy foods and limiting fat content of school meals, and banning tobacco products or use on school campuses. Although school health researchers have studied whether schools, districts, or states/provinces are meeting requirements, it is unclear to what extent implementation processes and determinants are assessed. The purposes of the present systematic review of quantitative measures of school policy implementation were to (1) identify quantitative school health policy measurement tools developed to measure implementation at the school, district, or state/provincial levels; (2) describe the policy implementation outcomes and determinants assessed and identify the trends in measurement; and (3) assess pragmatic and psychometric properties of identified implementation measures to understand their quality and suitability for broader application. METHODS: Peer-reviewed journal articles published 1995-2020 were included if they (1) had multiple-item quantitative measures of school policy implementation and (2) addressed overall wellness, tobacco, physical activity, nutrition, obesity prevention, or mental health/bullying/social-emotional learning. The final sample comprised 86 measurement tools from 67 peer-review articles. We extracted study characteristics, such as psychometric and pragmatic measure properties, from included articles based on three frameworks: (1) Implementation Outcomes Framework, (2) Consolidated Framework for Implementation Research, and (3) Policy Implementation Determinants Framework. RESULTS: Most implementation tools were developed to measure overall wellness policies which combined multiple policy topics (n = 35, 40%) and were in survey form (n = 75, 87%). Fidelity was the most frequently prevalent implementation outcome (n = 70, 81%), followed by adoption (n = 32, 81%). The implementation determinants most assessed were readiness for implementation, including resources (n = 43, 50%), leadership (n = 42, 49%), and policy communication (n = 41, 48%). Overall, measures were low-cost and had easy readability. However, lengthy tools and lack of reported validity/reliability data indicate low transferability. CONCLUSIONS: Implementation science can contribute to more complete and rigorous assessment of school health policy implementation processes, which can improve implementation strategies and ultimately the intended health benefits. Several high-quality measures of implementation determinants and implementation outcomes can be applied to school health policy implementation assessment. Dissemination and implementation science researchers can also benefit from measurement experiences of school health researchers.

Patterns and correlates of mis-implementation in state chronic disease public health practice in the United States.

BACKGROUND: Much of the disease burden in the United States is preventable through application of existing knowledge. State-level public health practitioners are in ideal positions to affect programs and policies related to chronic disease, but the extent to which mis-implementation occurring with these programs is largely unknown. Mis-implementation refers to ending effective programs and policies prematurely or continuing ineffective ones. METHODS: A 2018 comprehensive survey assessing the extent of mis-implementation and multi-level influences on mis-implementation was reported by state health departments (SHDs). Questions were developed from previous literature. Surveys were emailed to randomly selected SHD employees across the Unites States. Spearman's correlation and multinomial logistic regression were used to assess factors in mis-implementation. RESULTS: Half (50.7%) of respondents were chronic disease program managers or unit directors. Forty nine percent reported that programs their SHD oversees sometimes, often or always continued ineffective programs. Over 50% also reported that their SHD sometimes or often ended effective programs. The data suggest the strongest correlates and predictors of mis-implementation were at the organizational level. For example, the number of organizational layers impeded decision-making was significant for both continuing ineffective programs (OR=4.70; 95% CI=2.20, 10.04) and ending effective programs (OR=3.23; 95% CI=1.61, 7.40). CONCLUSION: The data suggest that changing certain agency practices may help in minimizing the occurrence of mis-implementation. Further research should focus on adding context to these issues and helping agencies engage in appropriate decision-making. Greater attention to mis-implementation should lead to greater use of effective interventions and more efficient expenditure of resources, ultimately to improve health outcomes.

Use and Awareness of The Community Guide in State and Local Health Department Chronic Disease Programs.

INTRODUCTION: The Community Guide (Guide) is a user-friendly, systematic review system that provides information on evidence-based interventions (EBIs) in public health practice. Little is known about what predicts Guide awareness and use in state health departments (SHDs) and local health departments (LHDs). METHODS: We pooled data from 3 surveys (administered in 2016, 2017, and 2018) to employees in chronic disease programs at SHDs and LHDs. Participants (n = 1,039) represented all 50 states. The surveys asked about department practices and individual, organizational, and external factors related to decisions about EBIs. We used χ2 tests of independence for analyses. RESULTS: Eighty-one percent (n = 498) of SHD and 54% (n = 198) of LHD respondents reported their agency uses the Guide. Additionally, 13% of SHD participants reported not being aware of the Guide. Significant relationships were found between reporting using the Guide and academic collaboration, population size, rated importance of forming partnerships, and accreditation. CONCLUSION: Awareness and use of the Guide in LHD and SHD chronic disease programs is widespread. Awareness of the Guide can be vital to implementation practice, because it enhances implementation of EBI practices. However, awareness of the Guide alone is likely not enough for health departments to implement EBIs. Changes at the organizational level, including sharing information about the Guide and providing training on how to best use it, may increase its awareness and use.

The Association Between Evidence-Based Decision Making and Accreditation of State Health Departments.

OBJECTIVE: To assess the association between evidence-based decision making, including implementation of evidence-based interventions (EBIs), with accreditation of state health departments through the Public Health Accreditation Board (PHAB). DESIGN: This was a cross-sectional, electronic survey of state health department practitioners. We utilized a survey instrument focused on evidence-based public health, de-implementation, and sustainability of public health programs. Survey questions were organized into 6 domains: (1) demographic information; (2) individual-level skills; (3) decision making on programs ending; (4) decision making on programs continuing; (5) organization/agency capacity; and (6) external influences. PARTICIPANTS: The targeted practitioners were randomly selected from the 3000-person membership of National Association of Chronic Disease Directors and program manager lists from key Centers for Disease Control and Prevention-supported programs in cancer and cancer risk factors. The final target audience for the survey totaled 1329 practitioners, representing all 50 states. MAIN OUTCOME MEASURE(S): The main outcome measures included the strength of association between a state's PHAB accreditation status and variables related to evidence-based public health and use of EBIs that fell within the individual participant skills, organization/agency capacity, and external influences domains. RESULTS: We received 643 valid responses (response rate = 48.4%), representing all 50 states, with 35 states being PHAB accredited. There was a statistically significant association between PHAB accreditation and state health department use of quality improvement processes (P = .002), leadership plans to implement EBIs (P = .009), and leadership reactions to EBI implementation issues (P = .004). Respondents from PHAB-accredited states were significantly more likely than participants from nonaccredited states to report greater engagement with legislators and governors regarding EBIs and 14% less likely to report the inappropriate termination of programs in their work unit (P = .05). CONCLUSIONS: The importance of accreditation relates to both internally focused functions and externally focused activities, especially regarding policy-related impact.

Quantitative measures of health policy implementation determinants and outcomes: a systematic review.

BACKGROUND: Public policy has tremendous impacts on population health. While policy development has been extensively studied, policy implementation research is newer and relies largely on qualitative methods. Quantitative measures are needed to disentangle differential impacts of policy implementation determinants (i.e., barriers and facilitators) and outcomes to ensure intended benefits are realized. Implementation outcomes include acceptability, adoption, appropriateness, compliance/fidelity, feasibility, penetration, sustainability, and costs. This systematic review identified quantitative measures that are used to assess health policy implementation determinants and outcomes and evaluated the quality of these measures. METHODS: Three frameworks guided the review: Implementation Outcomes Framework (Proctor et al.), Consolidated Framework for Implementation Research (Damschroder et al.), and Policy Implementation Determinants Framework (Bullock et al.). Six databases were searched: Medline, CINAHL Plus, PsycInfo, PAIS, ERIC, and Worldwide Political. Searches were limited to English language, peer-reviewed journal articles published January 1995 to April 2019. Search terms addressed four levels: health, public policy, implementation, and measurement. Empirical studies of public policies addressing physical or behavioral health with quantitative self-report or archival measures of policy implementation with at least two items assessing implementation outcomes or determinants were included. Consensus scoring of the Psychometric and Pragmatic Evidence Rating Scale assessed the quality of measures. RESULTS: Database searches yielded 8417 non-duplicate studies, with 870 (10.3%) undergoing full-text screening, yielding 66 studies. From the included studies, 70 unique measures were identified to quantitatively assess implementation outcomes and/or determinants. Acceptability, feasibility, appropriateness, and compliance were the most commonly measured implementation outcomes. Common determinants in the identified measures were organizational culture, implementation climate, and readiness for implementation, each aspects of the internal setting. Pragmatic quality ranged from adequate to good, with most measures freely available, brief, and at high school reading level. Few psychometric properties were reported. CONCLUSIONS: Well-tested quantitative measures of implementation internal settings were under-utilized in policy studies. Further development and testing of external context measures are warranted. This review is intended to stimulate measure development and high-quality assessment of health policy implementation outcomes and determinants to help practitioners and researchers spread evidence-informed policies to improve population health. REGISTRATION: Not registered.

Perspectives on program mis-implementation among U.S. local public health departments.

BACKGROUND: Public health resources are limited and best used for effective programs. This study explores associations of mis-implementation in public health (ending effective programs or continuing ineffective programs) with organizational supports for evidence-based decision making among U.S. local health departments. METHODS: The national U.S. sample for this cross-sectional study was stratified by local health department jurisdiction population size. One person was invited from each randomly selected local health department: the leader in chronic disease, or the director. Of 600 selected, 579 had valid email addresses; 376 completed the survey (64.9% response). Survey items assessed frequency of and reasons for mis-implementation. Participants indicated agreement with statements on organizational supports for evidence-based decision making (7-point Likert). RESULTS: Thirty percent (30.0%) reported programs often or always ended that should have continued (inappropriate termination); organizational supports for evidence-based decision making were not associated with the frequency of programs ending. The main reason given for inappropriate termination was grant funding ended (86.0%). Fewer (16.4%) reported programs often or always continued that should have ended (inappropriate continuation). Higher perceived organizational supports for evidence-based decision making were associated with less frequent inappropriate continuation (odds ratio = 0.86, 95% confidence interval 0.79, 0.94). All organizational support factors were negatively associated with inappropriate continuation. Top reasons were sustained funding (55.6%) and support from policymakers (34.0%). CONCLUSIONS: Organizational supports for evidence-based decision making may help local health departments avoid continuing programs that should end. Creative mechanisms of support are needed to avoid inappropriate termination. Understanding what influences mis-implementation can help identify supports for de-implementation of ineffective programs so resources can go towards evidence-based programs.

Assessing Organizational Supports for Evidence-Based Decision Making in Local Public Health Departments in the United States: Development and Psychometric Properties of a New Measure.

CONTEXT: Fostering evidence-based decision making (EBDM) within local public health departments and among local health department (LHD) practitioners is crucial for the successful translation of research into public health practice to prevent and control chronic disease. OBJECTIVE: The purpose of this study was to identify organizational supports for EBDM within LHDs and determine psychometric properties of a measure of organizational supports for EBDM in LHDs. DESIGN: Cross-sectional, observation study. SETTING: Local public health departments in the United States. PARTICIPANTS: Local health department practitioners (N = 376) across the United States participated in the study. MAIN OUTCOME MEASURES: Local health department practitioners completed a survey containing 27 items about organizational supports for EBDM. Most items were adapted from previously developed surveys, and input from researchers and practitioners guided survey development. Confirmatory factor analysis was used to test and refine the psychometric properties of the measure. RESULTS: The final solution included 6 factors of 22 items: awareness of EBDM (3 items), capacity for EBDM (7 items), resources availability (3 items), evaluation capacity (3 items), EBDM climate cultivation (3 items), and partnerships to support EBDM (3 items). This factor solution achieved acceptable fit (eg, Comparative Fit Index = 0.965). Logistic regression models showed positive relationships between the 6 factors and the number of evidence-based interventions delivered. CONCLUSIONS: This study identified important organizational supports for EBDM within LHDs. Results of this study can be used to understand and enhance organizational processes and structures to support EBDM to improve LHD performance and population health. Strong measures are important for understanding how LHDs support EBDM, evaluating interventions to improve LHD capacity, and to guide programmatic and policy efforts within LHDs.

Organizational Supports for Research Evidence Use in State Public Health Agencies: A Latent Class Analysis.

OBJECTIVE: Use of research evidence in public health decision making can be affected by organizational supports. Study objectives are to identify patterns of organizational supports and explore associations with research evidence use for job tasks among public health practitioners. DESIGN: In this longitudinal study, we used latent class analysis to identify organizational support patterns, followed by mixed logistic regression analysis to quantify associations with research evidence use. SETTING: The setting included 12 state public health department chronic disease prevention units and their external partnering organizations involved in chronic disease prevention. PARTICIPANTS: Chronic disease prevention staff from 12 US state public health departments and partnering organizations completed self-report surveys at 2 time points, in 2014 and 2016 (N = 872). MAIN OUTCOME MEASURES: Latent class analysis was employed to identify subgroups of survey participants with distinct patterns of perceived organizational supports. Two classify-analyze approaches (maximum probability assignment and multiple pseudo-class draws) were used in 2017 to investigate the association between latent class membership and research evidence use. RESULTS: The optimal model identified 4 latent classes, labeled as "unsupportive workplace," "low agency leadership support," "high agency leadership support," and "supportive workplace." With maximum probability assignment, participants in "high agency leadership support" (odds ratio = 2.08; 95% CI, 1.35-3.23) and "supportive workplace" (odds ratio = 1.74; 95% CI, 1.10-2.74) were more likely to use research evidence in job tasks than "unsupportive workplace." The multiple pseudo-class draws produced comparable results with odds ratio = 2.09 (95% CI, 1.31-3.30) for "high agency leadership support" and odds ratio = 1.74 (95% CI, 1.07-2.82) for "supportive workplace." CONCLUSIONS: Findings suggest that leadership support may be a crucial element of organizational supports to encourage research evidence use. Organizational supports such as supervisory expectations, access to evidence, and participatory decision making may need leadership support as well to improve research evidence use in public health job tasks.

Evidence-Based Public Health Provided Through Local Health Departments: Importance of Academic-Practice Partnerships.

OBJECTIVES: To determine the extent to which US local health departments (LHDs) are engaged in evidence-based public health and whether this is influenced by the presence of an academic health department (AHD) partnership. METHODS: We surveyed a cross-sectional stratified random sample of 579 LHDs in 2017. We ascertained the extent of support for evidence-based decision-making and the use of evidence-based interventions in several chronic disease programs and whether the LHD participated in a formal, informal, or no AHD partnership. RESULTS: We received 376 valid responses (response rate 64.9%). There were 192 (51.6%) LHDs with a formal, 80 (21.6%) with an informal, and 99 (26.7%) with no AHD partnership. Participants with formal AHD partnerships reported higher perceived organizational supports for evidence-based decision-making and interventions compared with either informal or no AHD partnerships. The odds of providing 1 or more chronic disease evidence-based intervention were significantly higher in LHDs with formal AHD partnerships compared with LHDs with no AHD partnerships (adjusted odds ratio = 2.3; 95% confidence interval = 1.3, 4.0). CONCLUSIONS: Formal academic-practice partnerships can be important means for advancing evidence-based decision-making and for implementing evidence-based programs and policies.

Local Health Department Accreditation Is Associated With Organizational Supports for Evidence-Based Decision Making.

Introduction: Recent studies show that health department accreditation from the U.S. Public Health Accreditation Board (PHAB) drives performance management and quality improvement. PHAB standards call for agencies to use evidence in decision making. It is unknown whether accreditation is associated with organizational supports for evidence-based decision making (EBDM). Self-report data from a 2017 survey of U.S. local health departments were analyzed to test relationships of accreditation status with organizational supports for EBDM. Methods: A cross-sectional survey was conducted in this observational study. A total of 579 local health departments were invited to complete an online survey; 350 (60.4%) provided complete data for the present study. The dependent variables were six factors of organizational supports for EBDM previously validated through confirmatory factor analyses. Accreditation status (PHAB-accredited, preparing, not preparing) was the independent variable of interest. Logistic regression analyses controlled for governance (presence of a local board of health; state, local, or shared state and local governance) and jurisdiction population size. Results: PHAB-accredited health departments were more likely to report higher capacity for EBDM, resource availability for EBDM, and evaluation capacity than health departments that reported not yet preparing for accreditation. Health departments that reported preparing for PHAB accreditation showed a non-significant pattern of higher perceived supports for EBDM compared to departments not preparing for accreditation. Conclusion: PHAB standards and the accreditation process may help stimulate health department organizational supports for EBDM.