Generalizing about Public Health Interventions: A Mixed-Methods Approach to External Validity.

Public health researchers and practitioners are calling for greater focus on external validity, the ability to generalize findings of evidence-based interventions (EBIs) beyond the limited number of studies testing effectiveness. For public health, the goal is applicability: to translate, disseminate, and implement EBIs for an impact on population health. This article is a review of methods and how they might be combined to better assess external validity. The methods include (a) better description of EBIs and their contexts; (b) combining of statistical tools and logic to draw inferences about study samples; (c) sharper definition of the theory behind the intervention and core intervention components; and (d) more systematic consultation of practitioners. For population impact, studies should focus on context features that are likely to be both important (based on program theory) and frequently encountered by practitioners. Mixed-method programs of research will allow public health to expand causal generalizations.

Interaction of Theory and Practice to Assess External Validity.

BACKGROUND: Variations in local context bedevil the assessment of external validity: the ability to generalize about effects of treatments. For evaluation, the challenges of assessing external validity are intimately tied to the translation and spread of evidence-based interventions. This makes external validity a question for decision makers, who need to determine whether to endorse, fund, or adopt interventions that were found to be effective and how to ensure high quality once they spread. OBJECTIVE: To present the rationale for using theory to assess external validity and the value of more systematic interaction of theory and practice. METHODS: We review advances in external validity, program theory, practitioner expertise, and local adaptation. Examples are provided for program theory, its adaptation to diverse contexts, and generalizing to contexts that have not yet been studied. The often critical role of practitioner experience is illustrated in these examples. Work is described that the Robert Wood Johnson Foundation is supporting to study treatment variation and context more systematically. RESULTS: Researchers and developers generally see a limited range of contexts in which the intervention is implemented. Individual practitioners see a different and often a wider range of contexts, albeit not a systematic sample. Organized and taken together, however, practitioner experiences can inform external validity by challenging the developers and researchers to consider a wider range of contexts. Researchers have developed a variety of ways to adapt interventions in light of such challenges. CONCLUSIONS: In systematic programs of inquiry, as opposed to individual studies, the problems of context can be better addressed. Evaluators have advocated an interaction of theory and practice for many years, but the process can be made more systematic and useful. Systematic interaction can set priorities for assessment of external validity by examining the prevalence and importance of context features and treatment variations. Practitioner interaction with researchers and developers can assist in sharpening program theory, reducing uncertainty about treatment variations that are consistent or inconsistent with the theory, inductively ruling out the ones that are harmful or irrelevant, and helping set priorities for more rigorous study of context and treatment variation.

Balancing stakeholder needs in the evaluation of healthcare quality improvement.

Quality improvement (QI) efforts affect a broader range of people than we often assume. These are the potential stakeholders for QI and its evaluation, and they have valuable perspectives to offer when they are consulted in planning, conducting and interpreting evaluations. QI practitioners are accustomed to consulting stakeholders to assess unintended consequences or assess patient experiences of care, but in many cases there are additional benefits to a broad inclusion of stakeholders. These benefits are better adherence to ethical standards, to assure that all legitimate interests take part, more useful and relevant evaluation information and better political buy-in to improve impact. Balancing various stakeholder needs for information requires skill for both politics and research management. These challenges have few pat answers, but several preferred practices, which are illustrated with practical examples.

The evaluation of Arkansas Act 1220 of 2003 to reduce childhood obesity: conceptualization, design, and special challenges.

This article describes the evaluation of the Arkansas Act 1220 of 2003, a comprehensive legislative proposal to address the growing epidemic of childhood obesity through changes in the school environment. In addition, the article discusses specific components of the evaluation that may be applicable to other childhood obesity policy evaluation efforts. The conceptual framework for the evaluation, research questions, and evaluation design are described, along with data collection methods and analysis strategies. A mixed methods approach, including both quantitative (surveys, telephone interviews) and qualitative (key informant interviews, records reviews) approaches, was utilized to collect data from a range of informant groups including parents, adolescents, school principals, school district superintendents, and other stakeholders. Challenges encountered with the evaluation are discussed, as are strategies to overcome those challenges. Now in its 9th year, this evaluation has documented substantial changes to school policies and environments but fewer changes to student and family behaviors. The evaluation may inform the methods of other evaluations of childhood obesity prevention policies, as well as inform policymakers about how quickly they might expect implementation of such policies in their own states and localities and anticipate both positive and adverse outcomes.

Lessons from a mixed-methods approach to evaluating Active Living by Design.

BACKGROUND: Beginning in 2003, Active Living by Design (ALbD) established innovative approaches across 25 communities to increase physical activity through community design, public policies, programming, and communication strategies. PURPOSE: The complexity of the ALbD projects called for a mixed-methods evaluation to understand implementation as well as perceived and actual impacts of these efforts. METHODS: Six primary evaluation methods addressed three primary aims: (1) to assess impacts of physical projects and policy changes on community environments; (2) to document intervention strategies implemented, as well as intended and unintended consequences; and (3) to identify strengths and challenges in planning, developing, and implementing interventions. The ALbD evaluation included cross-site comparisons and more in-depth case studies. This article describes the methods used to address the three aims. RESULTS: Analysis of the strengths and challenges associated with the different methods, including partnership capacity surveys, Concept Mapping, an online progress reporting system, key informant interviews, focus groups, and photos and videos. Additional methods, including environmental audits and direct observation, were explored to specifically assess environmental changes. Several important challenges included the lack of baseline data, difficulty in evaluating natural experiments, the need for ongoing policy surveillance, and the need to capture longer-term endpoints. CONCLUSIONS: The mixed-methods evaluation of ALbD advances implementation and evaluation science related to community-based efforts for promoting active living through identification of methods and measures to capture multicomponent and complex interventions as well as translation of a range of approaches to create community change across a variety of populations and settings.

Concept mapping: priority community strategies to create changes to support active living.

BACKGROUND: From 2003 to 2008, a total of 25 cross-sector, multidisciplinary community partnerships funded through the Active Living by Design (ALbD) national program designed, planned, and implemented policy and environmental changes, with complementary programs and promotions. PURPOSE: This paper describes the use of concept mapping methods to gain insights into promising active living intervention strategies based on the collective experience of community representatives implementing ALbD initiatives. METHODS: Using Concept Systems software, community representatives (n=43) anonymously generated actions and changes in their communities to support active living (183 original statements, 79 condensed statements). Next, respondents (n=26, from 23 partnerships) sorted the 79 statements into self-created categories, or active living intervention approaches. Respondents then rated statements based on their perceptions of the most important strategies for creating community changes (n=25, from 22 partnerships) and increasing community rates of physical activity (n=23, from 20 partnerships). Cluster analysis and multidimensional scaling were used to describe data patterns. RESULTS: ALbD community partnerships identified three active living intervention approaches with the greatest perceived importance to create community change and increase population levels of physical activity: changes to the built and natural environment, partnership and collaboration efforts, and land-use and transportation policies. The relative importance of intervention approaches varied according to subgroups of partnerships working with different populations. CONCLUSIONS: Decision makers, practitioners, and community residents can incorporate what has been learned from the 25 community partnerships to prioritize active living policy, physical project, promotional, and programmatic strategies for work in different populations and settings.

Evaluation of the healthy schools program: Part II. The role of technical assistance.

INTRODUCTION: Evidence-based technical assistance may be needed to implement recent federal policy to prevent childhood obesity through the schools. The Healthy Schools Program is the largest school-based obesity prevention program in the United States. The objectives of this study were to evaluate the role of the program's training and technical assistance and to explore other contributing factors in changing school policies, practices, and environments. METHODS: We analyzed interim progress of schools recruited during the 2007-2008 and 2008-2009 school years as of December 2010. Schools reported progress through an online inventory of policies, practices, and school environment. We compared baseline inventories to the most recent follow-up and tracked both training attendance and contact with national experts. To identify the factors associated with progress, we performed regression analysis on school level and demographics, number of months between baseline and follow-up, and technical assistance. RESULTS: The amount of training and technical assistance was significantly associated with school progress, controlling for school level and demographics, number of months between baseline and follow-up, and school status at baseline. Although all types of schools saw progress, schools in the South had the most progress and urban schools had the least progress. CONCLUSION: Evidence-based training and technical assistance were associated with school progress in changing policies, practices, and environment to prevent childhood obesity.

Evaluation of the healthy schools program: Part I. Interim progress.

INTRODUCTION: Federal and state policies identify schools as a setting to prevent childhood obesity, but schools need better health-promoting strategies. The objective of this study was to evaluate interim progress in schools receiving hands-on training from the Healthy Schools Program, the nation's largest school-based program aimed at preventing childhood obesity. The 4-year program targets schools with predominantly low-income, African American, or Hispanic students. METHODS: In 2010 we assessed schools that enrolled in the 2007-2008 and 2008-2009 school years. School representatives completed an inventory of 8 content areas: policy and systems, school meals, competitive foods and beverages, health education, physical education, physical activity outside of physical education, before- and after-school programs, and school employee wellness. Schools' baseline inventory was compared by t test with the most recent inventory available. RESULTS: Schools made significant changes in all content areas, and effect sizes were moderate to large. CONCLUSION: Participating schools improved environmental policies and practices to prevent childhood obesity. The program is a resource to implement recent federal and state policies.

Results of evaluability assessments of local wellness policies in 6 US school districts.

BACKGROUND: A US federal mandate that school districts devise and implement local wellness policies (LWPs) has potential widespread impact on the nutritional content of foods and beverages available in schools and on the amount of physical activity that students engage in; however, evidence concerning the mandate's effectiveness is limited. This study describes the content of LWPs of 6 US school districts and steps taken toward their implementation and evaluation. METHODS: During visits to 6 school districts, we interviewed 88 school and community representatives about the content of their district's LWPs and how the LWPs were being implemented and evaluated. RESULTS: The 6 LWPs were consistent with the federal mandate, although they varied in content and degree of specificity, and none had been fully implemented. All 6 districts were pursuing strategies to ensure that foods and beverages available at school met nutrition standards but did not offer nutrition education to all K-12 students. All 6 districts offered students only limited opportunities for physical activity, and all 6 collected data to monitor process and outcomes of their LWPs. CONCLUSIONS: Partial implementation of LWPs in the districts we visited resulted in significant improvement in the nutritional quality of foods available at district schools, but only slight improvement in students' opportunities for school-based physical activity. We provide recommendations for school districts on implementation and evaluation. Future research is needed to determine the impact of these LWPs on students' health.

Understanding the challenges encountered and adaptations made by community organizations in translation of evidence-based behavior change physical activity interventions: a qualitative study.

PURPOSE: Designing programs for mid-life to older adults whose sedentary behaviors are associated with increased health risks is crucial. The U.S. Task Force on Community Preventive Services strongly recommends individually adapted behavior change programs as one approach to increasing physical activity in communities. The purpose of this study is to report challenges organizations faced when translating two evidence-based programs in real-world settings, adaptations made, and whether or not fidelity was negatively impacted by these adaptations. DESIGN: A grounded theory approach to qualitative research was used. SETTING AND PARTICIPANTS: Nine community organizations across the country participated. Two organizations had more than one site participating, for a total of 12 sites from nine organizations. Within those organizations, 2796 participants were part of the program during the first 2 years. Participants were underactive (i.e., not meeting Centers for Disease Control and Prevention and American College of Sports Medicine recommendations) mid- to older-aged adults. METHODS: Community organizations participated in monthly conference calls, and program information was entered into an electronic database regularly. Data obtained from the calls and database were used for analyses. RESULTS: Challenges and adaptations emerged in three categories: (1) program logistics, (2) program theory, and (3) program philosophy. CONCLUSION: Challenges were present for community organizations; however, with some level of adaptation, the community organizations were able to effectively deliver and maintain fidelity in two evidence-based physical activity programs to a large and diverse group of mid- to older-aged adults.

Measuring food environments: a guide to available instruments.

PURPOSE: Food environments affect people's food-related decisions, consumption, and health outcomes. This paper critiques instruments available for assessing food environments to help researchers, practitioners, and community organizations select those that best meet their needs, resources, and level of expertise. Gaps and recommendations for future instrument development are discussed. METHOD: Instruments were selected by reviewing the literature and contacting researchers and practitioners. Using the social-ecological model, the instruments were classified according to the type of food environments they assess. Each instrument is rated according to criteria for ease of use, detail, resources required, and psychometric testing. RESULTS: Of the 48 instruments identified, only 39% were tested for validity or reliability. Observational tools were the largest category of instruments developed. Fifty-two percent of the instruments provide some level of technical assistance and 37% are considered appropriate for community members. CONCLUSION: Three distinct groups want to assess food environments: researchers, practitioners, and community organizations. These groups have different information needs and different capacities to undertake assessments. At this time there is a tradeoff between simplicity and low cost on the one hand, and detail and accuracy on the other. To choose the most suitable instrument, users should consider the pros and cons of each instrument and base their selection on purpose of their assessment, resources and expertise at hand, and validity, reliability, and ease of use of the instrument.

Evaluability assessment to improve public health policies, programs, and practices.

Evaluability assessment, also commonly known as exploratory evaluation, has assisted the field of public health to improve programs and to develop a pragmatic, practice-based research agenda. Evaluability assessment was originally developed as a low-cost pre-evaluation activity to prepare better for conventional evaluations of programs, practices, and some policies. For public health programs, however, it serves several other important purposes: (a) giving program staff rapid, constructive feedback about program operations; (b) assisting the core public health planning and assurance functions by helping to develop realistic objectives and providing low-cost, rapid feedback on implementation; (c) navigating federal performance measurement requirements; (d) translating research into practice by examining the feasibility, acceptability, and adaptation of evidence-based practices in new settings and populations; and (e) translating practice into research by identifying promising new approaches to achieve public health goals.