Policy implementation and outcome evaluation: establishing a framework and expanding capacity for advocacy organizations to assess the impact of their work in public policy.

Advocacy organizations can play a crucial role in evaluating whether legislation or regulation has had its intended effect by supporting robust public policy implementation and outcome evaluation. The American Heart Association, working with expert advisors, has developed a framework for effective evaluation that can be used by advocacy organizations, in partnership with researchers, public health agencies, funders, and policy makers to assess the health and equity impact of legislation and regulation over time. Advocacy organizations can use parts of this framework to evaluate the impact of policies relevant to their own advocacy and public policy efforts and inform policy development and guide their organizational resource allocation. Ultimately, working in partnership, advocacy organizations can help bring capacity, commitment and funding to this important implementation and outcome evaluation work that informs impactful public policy for equitable population health and well-being.

Design and Implementation of an Innovative, Rapid Data-Monitoring Strategy for Public Health Emergencies: Pilot of the United States School COVID-19 Mitigation Strategies Project.

During the COVID-19 pandemic, an urgent need existed for near-real-time data collection to better understand how individual beliefs and behaviors, state and local policies, and organizational practices influenced health outcomes. We describe the processes, methods, and lessons learned during the development and pilot testing of an innovative rapid data collection process we developed to inform decision-making during the COVID-19 public health emergency. We used a fully integrated mixed-methods approach to develop a structured process for triangulating quantitative and qualitative data from traditional (cross-sectional surveys, focus groups) and nontraditional (social media listening) sources. Respondents included students, parents, teachers, and key school personnel (eg, nurses, administrators, mental health providers). During the pilot phase (February-June 2021), data from 12 cross-sectional and sector-based surveys (n = 20 302 participants), 28 crowdsourced surveys (n = 26 820 participants), 10 focus groups (n = 64 participants), and 11 social media platforms (n = 432 754 503 responses) were triangulated with other data to support COVID-19 mitigation in schools. We disseminated findings through internal dashboards, triangulation reports, and policy briefs. This pilot demonstrated that triangulating traditional and nontraditional data sources can provide rapid data about barriers and facilitators to mitigation implementation during an evolving public health emergency. Such a rapid feedback and continuous improvement model can be tailored to strengthen response efforts. This approach emphasizes the value of nimble data modernization efforts to respond in real time to public health emergencies.

Timing of State and Territorial COVID-19 Stay-at-Home Orders and Changes in Population Movement - United States, March 1-May 31, 2020.

SARS-CoV-2, the virus that causes coronavirus disease 2019 (COVID-19), is thought to spread from person to person primarily by the respiratory route and mainly through close contact (1). Community mitigation strategies can lower the risk for disease transmission by limiting or preventing person-to-person interactions (2). U.S. states and territories began implementing various community mitigation policies in March 2020. One widely implemented strategy was the issuance of orders requiring persons to stay home, resulting in decreased population movement in some jurisdictions (3). Each state or territory has authority to enact its own laws and policies to protect the public's health, and jurisdictions varied widely in the type and timing of orders issued related to stay-at-home requirements. To identify the broader impact of these stay-at-home orders, using publicly accessible, anonymized location data from mobile devices, CDC and the Georgia Tech Research Institute analyzed changes in population movement relative to stay-at-home orders issued during March 1-May 31, 2020, by all 50 states, the District of Columbia, and five U.S. territories.* During this period, 42 states and territories issued mandatory stay-at-home orders. When counties subject to mandatory state- and territory-issued stay-at-home orders were stratified along rural-urban categories, movement decreased significantly relative to the preorder baseline in all strata. Mandatory stay-at-home orders can help reduce activities associated with the spread of COVID-19, including population movement and close person-to-person contact outside the household.

Public Attitudes, Behaviors, and Beliefs Related to COVID-19, Stay-at-Home Orders, Nonessential Business Closures, and Public Health Guidance - United States, New York City, and Los Angeles, May 5-12, 2020.

SARS-CoV-2, the virus that causes coronavirus disease 2019 (COVID-19), is thought to be transmitted mainly by person-to-person contact (1). Implementation of nationwide public health orders to limit person-to-person interaction and of guidance on personal protective practices can slow transmission (2,3). Such strategies can include stay-at-home orders, business closures, prohibitions against mass gatherings, use of cloth face coverings, and maintenance of a physical distance between persons (2,3). To assess and understand public attitudes, behaviors, and beliefs related to this guidance and COVID-19, representative panel surveys were conducted among adults aged ≥18 years in New York City (NYC) and Los Angeles, and broadly across the United States during May 5-12, 2020. Most respondents in the three cohorts supported stay-at-home orders and nonessential business closures* (United States, 79.5%; New York City, 86.7%; and Los Angeles, 81.5%), reported always or often wearing cloth face coverings in public areas (United States, 74.1%, New York City, 89.6%; and Los Angeles 89.8%), and believed that their state's restrictions were the right balance or not restrictive enough (United States, 84.3%; New York City, 89.7%; and Los Angeles, 79.7%). Periodic assessments of public attitudes, behaviors, and beliefs can guide evidence-based public health decision-making and related prevention messaging about mitigation strategies needed as the COVID-19 pandemic evolves.

An exploratory review of the literature evaluating nonclinical fellowship programs.

Fellowship programs offer career development opportunities, provide experiential training, and can be used to recruit personnel to address specific challenges facing the public health workforce. Given the potential influence fellowships have on the future public health workforce, it is important to understand and articulate the results of such programs and to identify areas of improvement to meet current workforce needs. The purpose of this literature review was to identify common practices used to evaluate nonclinical fellowship programs. After a search of the internet and selected databases, we screened titles and abstracts using predetermined selection criteria. We then conducted a detailed review of selected papers to extract information about program characteristics (program description, sector, and program length) and evaluation characteristics (primary evaluation type, framework for evaluation, data collection methods, and respondent populations) from 33 papers. We found a limited number of published papers on the evaluation of nonclinical fellowship programs, and most focused on outcomes associated with fellows or alumni. The most useful papers for our purposes clearly described the evaluation framework that guided the evaluation.

Social Environmental Correlates of Health Behaviors in a Faith-Based Policy and Environmental Change Intervention.

Diet and physical activity are behavioral risk factors for many chronic diseases, which are among the most common health conditions in the United States. Yet most Americans fall short of meeting established dietary and physical activity guidelines. Faith-based organizations as settings for health promotion interventions can affect members at multiple levels of the social ecological model. The present study investigated whether change in the church social environment was associated with healthier behavior at church and in general at 1-year follow-up. Six churches received mini-grants and technical assistance for 1 year to support policy and environmental changes for healthy eating (HE) and physical activity (PA). Socioenvironmental (social support and social norms) and behavioral (HE and PA at church and in general) outcomes were derived from baseline and 1-year follow-up church member surveys ( n = 258). Three of six churches demonstrated significant improvements in all three socioenvironmental aspects of HE. Two of five churches exhibited significant socioenvironmental improvements for PA at follow-up. Church social environmental changes were related to health behaviors at church and in general ( p < .05). Change in social support for HE, social support for PA, and social norms for PA were each associated with three church-based and general behavioral outcomes. Social norms for healthy eating were related to two general behavior outcomes and social norms for unhealthy eating to one general behavioral outcome. Study findings demonstrate that socioenvironmental characteristics are essential to multilevel interventions and merit consideration in designing policy and environmental change interventions.

Results from an Evaluation of the Georgia Colorectal Cancer Control Program's Community Education and Outreach Events, 2013.

Although public health practitioners commonly use community education and outreach events to promote cancer screening, the effectiveness of this strategy remains unclear. This study evaluated 23 outreach events, conducted as part of the Georgia Colorectal Cancer Control Program. Of the estimated 1778 individuals who attended these events, those ages 50-75 were eligible to participate in a telephone survey 3 months after attending an event. Surveys measured colorectal cancer (CRC) risk status, CRC screening history at the time of the event, seeking or obtaining CRC screening at 3-month follow-up, and participants' knowledge of their CRC screening status. Of the 335 individuals contacted for this evaluation, 185 completed the survey. Eighty participants (43.2 %) were at elevated risk for CRC and 99 participants (53.5 %) were at average risk. Of the 99 average-risk participants, the majority (n = 69) were not due for CRC screening at the time they attended an event because they had previously received screening within the recommended time intervals. Thirty average-risk participants were due for CRC screening, either because they had never been screened before (n = 19) or because they were due for rescreening (n = 11). Approximately half of these 30 participants who were due for screening either sought (n = 6, 20.0 %) or obtained screening (n = 8, 26.7 %) 3 months following the event. Community education and outreach events may play an important role in motivating participants to seek or obtain CRC screening, but unless priority audiences are identified and recruited, events may attract people who are already compliant with CRC screening.

Promoting Policy and Environmental Change in Faith-Based Organizations: Description and Findings From a Mini-Grants Program.

PURPOSE: The Emory Prevention Research Center's Cancer Prevention and Control Research Network mini-grant program funded faith-based organizations to implement policy and environmental change to promote healthy eating and physical activity in rural South Georgia. This study describes the existing health promotion environment and its relationship to church member behavior. DESIGN: Cross-sectional. SETTING: Data were obtained from parishioners of six churches in predominantly rural South Georgia. SUBJECTS: Participants were 319 church members with average age of 48 years, of whom 80% were female and 84% were black/African-American. MEASURES: Questionnaires assessed perceptions of the existing church health promotion environment relative to nutrition and physical activity, eating behavior and intention to use physical activity facilities at church, and eating and physical activity behaviors outside of church. ANALYSIS: Multiple regression and ordinal logistic regression using generalized estimating equations were used to account for clustered data. RESULTS: Results indicate that delivering messages via sermons and church bulletins, having healthy eating programs, and serving healthy foods are associated with participants' self-reported consumption of healthy foods at church (all p values ≤ .001). Serving more healthy food and less unhealthy food was associated with healthier eating in general but not to physical activity in general (p values ≤ .001). CONCLUSION: The church environment may play an important role in supporting healthy eating in this setting and more generally.

Practice to Evidence: Using Evaluability Assessment to Generate Practice-Based Evidence in Rural South Georgia.

Evidence from formal evaluation of real-world practice can address gaps in the public health knowledge base and provide information about feasible, relevant strategies for varied settings. Interest in evaluability assessment (EA) as an approach for generating practice-based evidence has grown. EA has been central to several structured assessment processes that identify and select promising programs and evaluate those most likely to produce useful findings. The Emory Prevention Research Center used EA as part of an initiative to generate practice-based evidence for cancer prevention in southwest Georgia. Our initiative consisted of five steps: (1) environmental scan to identify potential programs, (2) program selection, (3) EA, (4) evaluation, and (5) dissemination. We identified nine programs, four of which completed a formal application, and conducted two EAs. EAs consisted of document review, site visits, and literature reviews. The EA purpose was to assess the program model, data availability, stakeholder interest in evaluation, feasibility of an outcome evaluation, and potential contribution to the literature. We conducted one outcome evaluation and one descriptive qualitative study; both were published in peer-reviewed journals. The outcome evaluation addressed knowledge gaps about strategies to promote colorectal cancer screening. Results led to the program's inclusion in national resources for practitioners seeking evidence-based practices and helped the community organization expand and strengthen the program. As part of a structured assessment process, EA can identify programs most likely to produce useful results for dissemination and is a viable approach for local initiatives to generate practice-based evidence in rural or low-resource settings.

Increasing Community Access to Fresh Fruits and Vegetables: A Case Study of the Farm Fresh Market Pilot Program in Cobb County, Georgia, 2014.

BACKGROUND: Ecological models of health suggest that to effectively prevent chronic disease, community food environments must support healthy eating behaviors. However, disparities in access to healthy foods persist in the United States. COMMUNITY CONTEXT: The Farm Fresh Market (FFM) was a fruit and vegetable market that sold low-cost fresh produce in Cobb County, Georgia in 2014. METHODS: This case study describes the development of the FFM through a community engagement process and presents evaluation results from the project's pilot implementation. Community engagement strategies included forming a community advisory board, conducting a needs assessment, and contracting with a community-based organization to implement the FFM. OUTCOME: In the pilot year, the FFM served an average of 28.7 customers and generated an average of $140.20 in produce sales per market day. Most returning customers lived in the local community and reported a range of socioeconomic backgrounds. Most returning customers strongly agreed that the FFM made it easier (69.0%) and less expensive (79.0%) for them to buy fresh fruits and vegetables, reported that they ate more vegetables (65.0%) and fruit (55.0%) as a result of the FFM, and reported that they were very satisfied with the FFM overall (92.0%). INTERPRETATION: Results from this community case study underscore the importance of engaging communities in the development of community food environment interventions. Results also suggest that the FFM initiative was a feasible and acceptable way to respond to the community-identified public health priority of increasing access to healthy foods.

Promoting Policy and Environmental Change in Faith-Based Organizations: Outcome Evaluation of a Mini-Grants Program.

High rates of heart disease, cancer, and stroke exist in rural South Georgia where the Emory Prevention Research Center's Cancer Prevention and Control Research Network provided mini-grant funding to six churches to implement policy and environmental change to promote healthy eating and physical activity. This study sought to determine whether perceptions of the health promotion environment changed over time and whether perceived environmental change was associated with healthy behavior at church and in general. This study used a single-group pre-post design with 1-year follow-up. Parishioners (N = 258) completed self-administered questionnaires assessing perceptions of the church health promotion environment relative to healthy eating and physical activity, eating behavior and intention to use physical activity facilities at church, and eating and physical activity behaviors generally. Results indicate that perceived improvements in church nutrition environments were most strongly associated with decreases in unhealthy food consumed and stronger intentions to use physical activity resources at church (ps ≤ .05). Perceived changes in the physical activity environment were unrelated to church or general behavior. Findings suggest that church environments may play an important role in supporting healthy eating and physical activity at church; however, whether the influence of the church environment extends to other settings is unknown.

Evaluating Policy, Systems, and Environmental Change Interventions: Lessons Learned From CDC's Prevention Research Centers.

INTRODUCTION: The field of public health is increasingly implementing initiatives intended to make policies, systems, and environments (PSEs) more supportive of healthy behaviors, even though the evidence for many of these strategies is only emerging. Our objective was 3-fold: 1) to describe evaluations of PSE-change programs in which the evaluators followed the steps of the Centers for Disease Control and Prevention's (CDC's) Framework for Program Evaluation in Public Health, 2) to share the resulting lessons learned, and 3) to assist future evaluators of PSE-change programs with their evaluation design decisions. METHODS: Seven Prevention Research Centers (PRCs) applied CDC's framework to evaluate their own PSE-change initiatives. The PRCs followed each step of the framework: 1) engage stakeholders, 2) describe program, 3) focus evaluation design, 4) gather credible evidence, 5) justify conclusions, and 6) ensure use and share lessons learned. RESULTS: Evaluation stakeholders represented a range of sectors, including public health departments, partner organizations, and community members. Public health departments were the primary stakeholders for 4 of the 7 evaluations. Four PRCs used logic models to describe the initiatives being evaluated. Their evaluations typically included both process and outcome questions and used mixed methods. Evaluation findings most commonly focused on contextual factors influencing change (process) and the adoption or implementation of PSE-change strategies (outcome). Evaluators shared lessons learned through various channels to reach local stakeholders and broader public health audiences. CONCLUSION: Framework for Program Evaluation in Public Health is applicable to evaluations of PSE-change initiatives. Using this framework to guide such evaluations builds practice-based evidence for strategies that are increasingly being used to promote healthful behaviors.

Policy, systems, and environmental change in the Mississippi Delta: considerations for evaluation design.

Community-level policy, systems, and environmental (PSE) change strategies may offer an economical and sustainable approach to chronic disease prevention. The rapidly growing number of untested but promising PSE strategies currently underway offers an exciting opportunity to establish practice-based evidence for this approach. This article presents lessons learned from an evaluation of a community-based PSE initiative targeting stroke and cardiovascular disease prevention in the Mississippi Delta. Its purpose is to describe one approach to evaluating this type of PSE initiative, to stimulate discussion about best practices for evaluating PSE strategies, and to inform future evaluation and research efforts to expand practice-based evidence. The evaluation used a descriptive mixed-methods design and focused on the second year of a multisectoral, multiyear initiative. Cross-sectional data were collected in the summer and fall of 2010 using four data collection instruments: a grantee interview guide (n = 32), a health council member survey (n = 256), an organizational survey (n = 60), and a grantee progress report (n = 26). Fifty-eight PSE changes were assessed across five sectors: health, faith, education, worksite, and community/city government. PSE strategies aligned with increased access to physical activity opportunities, healthy food and beverage options, quality health care, and reduced exposure to tobacco. Results showed that grantees were successful in completing a series of steps toward PSE change and that sector-specific initiatives resulted in a range of PSE changes that were completed or in progress. Considerations for designing evaluations of community-based PSE initiatives are discussed.

Applying the chronic care model to an employee benefits program: a qualitative inquiry.

OBJECTIVE: To assess how employee benefits programs may strengthen and/or complement elements of the chronic care model (CCM), a framework used by health systems to improve chronic illness care. METHODS: A qualitative inquiry consisting of semi-structured interviews with employee benefit administrators and partners from a self-insured, self-administered employee health benefits program was conducted at a large family-owned business in southwest Georgia. RESULTS: Results indicate that the employer adapted and used many health system-related elements of the CCM in the design of their benefit program. Data also suggest that the employee benefits program contributed to self-management skills and to informing and activating patients to interact with the health system. CONCLUSION: Findings suggest that employee benefits programs can use aspects of the CCM in their own benefit design, and can structure their benefits to contribute to patient-related elements from the CCM.

Evaluation of a patient navigation program to promote colorectal cancer screening in rural Georgia, USA.

BACKGROUND: Colorectal cancer (CRC) is a leading cause of cancer death in the United States. Early detection through recommended screening has been shown to have favorable treatment outcomes, yet screening rates among the medically underserved and uninsured are low, particularly for rural and minority populations. This study evaluated the effectiveness of a patient navigation program that addresses individual and systemic barriers to CRC screening for patients at rural, federally qualified community health centers. METHODS: This quasi-experimental evaluation compared low-income patients at average risk for CRC (n = 809) from 4 intervention clinics and 9 comparison clinics. We abstracted medical chart data on patient demographics, CRC history and risk factors, and CRC screening referrals and examinations. Outcomes of interest were colonoscopy referral and examination during the study period and being compliant with recommended screening guidelines at the end of the study period. We conducted multilevel logistic analyses to evaluate the program's effectiveness. RESULTS: Patients at intervention clinics were significantly more likely than patients at comparison clinics to undergo colonoscopy screening (35% versus 7%, odds ratio = 7.9, P < .01) and be guideline-compliant on at least one CRC screening test (43% versus 11%, odds ratio = 5.9, P < .001). CONCLUSIONS: Patient navigation, delivered through the Community Cancer Screening Program, can be an effective approach to ensure that lifesaving, preventive health screenings are provided to low-income adults in a rural setting.

Balancing fidelity and adaptation: implementing evidence-based chronic disease prevention programs.

OBJECTIVES: To describe adaptations that community-based organizations (CBOs) made to evidence-based chronic disease prevention intervention programs and to discuss reasons for those adaptations. DESIGN: The process evaluation used project report forms, interviews, and focus groups to obtain information from organizational staff. SETTING: Programs were conducted in community-based organizations (n = 12) in rural southwest Georgia including churches, worksites, community coalitions, a senior center, and a clinical patient setting. PARTICIPANTS: Site coordinators (n = 15), organizational leaders (n = 7), and project committee members (n = 25) involved in program implementation at 12 funded organizations. INTERVENTION: The Emory Cancer Prevention and Control Research Network awarded mini grants to rural CBOs to implement one of 5 evidence-based nutrition or physical activity programs. These sites received funding and technical assistance from Emory and agreed to conduct all required elements of the selected evidence-based program. MAIN OUTCOME MEASURES: Program implementation and context were explored, including completion of core elements, program adaptation, and reasons for adaptation that occurred at sites implementing evidence-based chronic disease prevention programs. RESULTS: Five major types of adaptations were observed: changing educational materials, intended audience, and program delivery; adding new activities; and deleting core elements. Sites had intentional or unintentional reasons for making program adaptations including enhancing engagement in the program, reaching specific audiences, increasing program fit, and reinforcing program messages. Reasons for not completing core elements (program deletions) included various types of "turbulence" or competing demands (eg, leadership/staff transitions and time constraints). CONCLUSIONS: The types of adaptations and reasons described in this evaluation support the idea that adaptation is a natural element of implementing evidence-based interventions. Building this understanding into dissemination strategies may help researchers and funders better reach communities with evidence-based interventions that are a relevant fit, while striving for fidelity.

Research to reality: a process evaluation of a mini-grants program to disseminate evidence-based nutrition programs to rural churches and worksites.

OBJECTIVES: To describe a project that used mini-grants plus technical assistance to disseminate evidence-based programs, to understand how the project worked in different settings, and to generate recommendations for future programming and evaluation. DESIGN: Process evaluation using program records, activity forms completed by grantees, interviews, and focus groups. SETTING: Churches and worksites in rural, southwest Georgia. PARTICIPANTS: Site coordinators (n = 10), organizational leaders (n = 7), and project committee members (n = 25) involved in program implementation at 7 funded organizations. INTERVENTION: The Emory Cancer Prevention and Control Research Network solicited applications from churches and worksites to implement one of 2 evidence-based nutrition programs: Body & Soul for churches and Treatwell 5-a-Day for worksites. Successful applicants (n = 7) received funding and technical assistance from Emory and agreed to conduct all required elements of the evidence-based program. MAIN OUTCOME MEASURES: We assessed adoption, reach, implementation, and maintenance of specific programs and their core elements, as well as contextual influences and the resources required to implement the mini-grants program. RESULTS: Four of the 7 funded organizations conducted all programmatic core elements; all 7 sites conducted at least 6 of 8 core elements, including at least 1 food-related policy or environmental change as a result of the program. Program reach varied widely across sites and core elements. All site coordinators stated that they intend to continue at least some of the activities conducted under the project. Sites reported that contextual factors such as the program's fit with the organization's mission, leadership support, and leadership or staffing transitions influenced program implementation. Over 18 months, Emory staff spent 47.7 hours providing technical assistance to grantees. CONCLUSIONS: A mini-grants and technical assistance model has the potential to be an effective mechanism for disseminating evidence-based programs to community organizations, and further study of this method is warranted.

The role of community context in planning and implementing community-based health promotion projects.

The current study examines how community context affected collaborative planning and implementation in eight sites participating in a healthy cities and communities initiative in California. Data are from 23 focus groups conducted with coalition members, and 76 semi-structured interviews with local coordinators and community leaders. Multiple case study methods were used to identify major themes related to how five contextual domains influenced collaborative planning and implementation. Results showed that history of collaboration can influence resources and interpersonal and organizational connections available for planning and implementation, as well as priorities selected for action. Community politics and history can affect which segments of the community participate in a planning process and what issues are prioritized, as well as the pool of partners willing to aid in implementation. Some community norms and values bring people together and others appear to limit involvement from certain groups. Community demographics and economic conditions may shape outreach strategies for planning and implementation, and may also shape priorities. Geography can play a role in assessment methods, priority selection, partners available to aid in implementation, and participation in activities and events. Results suggest that community context plays a substantive role in shaping how community-based health promotion projects unfold.

Training for and dissemination of the Nutrition Environment Measures Surveys (NEMS).

INTRODUCTION: Researchers believe that nutrition environments contribute to obesity and may explain some health disparities. The Nutrition Environment Measures Surveys (NEMS) are valid and reliable observational measures of the nutrition environment. This article describes the dissemination of the measures, including the development, implementation, and evaluation of training workshops, and a follow-up survey of training participants. METHODS: To disseminate the NEMS measures, we developed a 2-day intensive, participatory workshop. We used an immediate postcourse evaluation and a structured telephone follow-up interview to evaluate the workshops and the dissemination strategy. Topics included use of the NEMS measures, reactions to the workshops, and participants' training others on the measures. RESULTS: During the study period, 173 people participated in 14 workshops. Participants indicated a high level of satisfaction with the training workshops. Almost two-thirds of respondents reported using the measures to train an additional 292 people and to rate more than 3,000 food outlets. The measures have been used in diverse locations across the United States for various purposes. Respondents have reported NEMS results in peer-reviewed journals, master's theses, newspaper articles, and presentations. CONCLUSION: The NEMS measures are the only nutrition environment measures that have been packaged for distribution and widely disseminated. The measures fill a need in the worlds of research and community action, and dissemination was successful in accelerating diffusion and promoting adoption of the measures. The use of an ongoing, continual process to improve workshops and measures contributes to the usefulness of the surveys and accelerates their adoption and continued use.