A typology of useful evidence: approaches to increase the practical value of intervention research.

BACKGROUND: Too often, studies of evidence-based interventions (EBIs) in preventive, community, and health care are not sufficiently useful to end users (typically practitioners, patients, policymakers, or other researchers). The ways in which intervention studies are conventionally conducted and reported mean that there is often a shortage of information when an EBI is used in practice. The paper aims to invite the research community to consider ways to optimize not only the trustworthiness but also the research's usefulness in intervention studies. This is done by proposing a typology that provides some approaches to useful EBIs for intervention researchers. The approaches originate from different research fields and are summarized to highlight their potential benefits from a usefulness perspective. MAIN MESSAGE: The typology consists of research approaches to increase the usefulness of EBIs by improving the reporting of four features in intervention studies: (1) the interventions themselves, including core components and appropriate adaptations; (2) strategies to support-high-quality implementation of the interventions; (3) generalizations about the evidence in a variety of contexts; and (4) outcomes based on end users' preferences and knowledge. The research approaches fall into three levels: Description, Analysis, and Design. The first level, Description, outlines what types of information about the intervention and its implementation, context, and outcomes can be helpful for end users. Research approaches under analysis offers alternative ways of analyzing data, increasing the precision of information provided to end users. Approaches summarized under design involve more radical changes and far-reaching implications for how research can provide more useful information. These approaches partly flip the order of efficacy and effectiveness, focusing not on whether an intervention works in highly controlled and optimal circumstances, but first and foremost whether an intervention can be implemented and lead to anticipated outcomes in everyday practice. CONCLUSIONS: The research community, as well as the end users of research, are invited to consider ways to optimize research's usefulness as well as its trustworthiness. Many of the research approaches in the typology are not new, and their contributions to quality have been described for generations - but their contributions to useful knowledge need more attention.

Building a National Culture of Health: Background, Action Framework, Measures, and Next Steps.

Because health is a function of more than medical care, solutions to U.S. health problems must encompass more than reforms to health care systems. But those working to improve health, well-being, and equity still too often find themselves traveling on parallel paths that rarely intersect. In 2013, the Robert Wood Johnson Foundation (RWJF) embarked on a pioneering effort to advance a Culture of Health initiative. A Culture of Health places well-being at the center of every aspect of life, with the goal of enabling everyone in our diverse society to lead healthier lives, now and for generations to come. To put this vision into action, RWJF worked with RAND to develop an action framework that identifies how the nation will work toward achieving these outcomes. This article provides background on the development of this action framework. The Culture of Health action framework is designed around four action areas and one outcome area. Action areas are the core areas in which investment and activity are needed: (1) making health a shared value; (2) fostering cross-sector collaboration to improve well-being; (3) creating healthier, more equitable communities; and (4) strengthening integration of health services and systems. Each action area contains a set of drivers indicating where the United States needs to accelerate change and a set of measures illustrating places for progress. Within the primary Culture of Health outcome---improved population health, well-being, and equity---the authors identified three outcome areas: enhanced individual and community well-being, managed chronic disease and reduced toxic stress, and reduced health care costs.

Generalizing about Public Health Interventions: A Mixed-Methods Approach to External Validity.

Public health researchers and practitioners are calling for greater focus on external validity, the ability to generalize findings of evidence-based interventions (EBIs) beyond the limited number of studies testing effectiveness. For public health, the goal is applicability: to translate, disseminate, and implement EBIs for an impact on population health. This article is a review of methods and how they might be combined to better assess external validity. The methods include (a) better description of EBIs and their contexts; (b) combining of statistical tools and logic to draw inferences about study samples; (c) sharper definition of the theory behind the intervention and core intervention components; and (d) more systematic consultation of practitioners. For population impact, studies should focus on context features that are likely to be both important (based on program theory) and frequently encountered by practitioners. Mixed-method programs of research will allow public health to expand causal generalizations.

Interaction of Theory and Practice to Assess External Validity.

BACKGROUND: Variations in local context bedevil the assessment of external validity: the ability to generalize about effects of treatments. For evaluation, the challenges of assessing external validity are intimately tied to the translation and spread of evidence-based interventions. This makes external validity a question for decision makers, who need to determine whether to endorse, fund, or adopt interventions that were found to be effective and how to ensure high quality once they spread. OBJECTIVE: To present the rationale for using theory to assess external validity and the value of more systematic interaction of theory and practice. METHODS: We review advances in external validity, program theory, practitioner expertise, and local adaptation. Examples are provided for program theory, its adaptation to diverse contexts, and generalizing to contexts that have not yet been studied. The often critical role of practitioner experience is illustrated in these examples. Work is described that the Robert Wood Johnson Foundation is supporting to study treatment variation and context more systematically. RESULTS: Researchers and developers generally see a limited range of contexts in which the intervention is implemented. Individual practitioners see a different and often a wider range of contexts, albeit not a systematic sample. Organized and taken together, however, practitioner experiences can inform external validity by challenging the developers and researchers to consider a wider range of contexts. Researchers have developed a variety of ways to adapt interventions in light of such challenges. CONCLUSIONS: In systematic programs of inquiry, as opposed to individual studies, the problems of context can be better addressed. Evaluators have advocated an interaction of theory and practice for many years, but the process can be made more systematic and useful. Systematic interaction can set priorities for assessment of external validity by examining the prevalence and importance of context features and treatment variations. Practitioner interaction with researchers and developers can assist in sharpening program theory, reducing uncertainty about treatment variations that are consistent or inconsistent with the theory, inductively ruling out the ones that are harmful or irrelevant, and helping set priorities for more rigorous study of context and treatment variation.

Cross-Sector Collaborations And Partnerships: Essential Ingredients To Help Shape Health And Well-Being.

Cross-sector collaborations and partnerships are an essential component of the strategy to improve health and well-being in the United States. While their importance is unquestioned, their impact on population health has not yet been fully observed. Cross-sector collaboration also is the second Action Area of the Robert Wood Johnson Foundation's four-part Action Framework to build a Culture of Health in the United States. This Action Area has three constituent parts or drivers: the number, breadth, and quality of successful cross-sector partnerships; the adequacy of investment in these partnerships; and the adoption of policies needed to support them. In this article we analyze outstanding examples of partnership-driven work. We also study the challenges of how partner sectors outside the formal health system, such as organizations working in the education or housing sectors, can effectively lead collaborations. We identify models of leadership that maximize the potential of all participants. We also propose the adoption of models better suited to supporting effective cross-sector collaborations. The analysis builds the evidence base for understanding and sustaining the impact of cross-sector collaboration on population health.

Strengthening Integration Of Health Services And Systems.

New care delivery models that hold providers more accountable for coordinated, high-quality care and the overall health of their patients have appeared in the US health care system, spurred by recent legislation such as the Affordable Care Act. These models support the integration of health care systems, but maximizing health and well-being for all individuals will require a broader conceptualization of health and more explicit connections between diverse partners. Integration of health services and systems constitutes the fourth Action Area in the Robert Wood Johnson Foundation's Culture of Health Action Framework, which is the subject of this article. This Action Area conceives of a strengthened health care system as one in which medical care, public health, and social services interact to produce a more effective, equitable, higher-value whole that maximizes the production of health and well-being for all individuals. Three critical drivers help define and advance this Action Area and identify gaps and needs that must be addressed to move forward. These drivers are access, balance and integration, and consumer experience and quality. This article discusses each driver and summarizes practice gaps that, if addressed, will help move the nation toward a stronger and more integrated health system.

Balancing stakeholder needs in the evaluation of healthcare quality improvement.

Quality improvement (QI) efforts affect a broader range of people than we often assume. These are the potential stakeholders for QI and its evaluation, and they have valuable perspectives to offer when they are consulted in planning, conducting and interpreting evaluations. QI practitioners are accustomed to consulting stakeholders to assess unintended consequences or assess patient experiences of care, but in many cases there are additional benefits to a broad inclusion of stakeholders. These benefits are better adherence to ethical standards, to assure that all legitimate interests take part, more useful and relevant evaluation information and better political buy-in to improve impact. Balancing various stakeholder needs for information requires skill for both politics and research management. These challenges have few pat answers, but several preferred practices, which are illustrated with practical examples.

Demystifying theory and its use in improvement.

The role and value of theory in improvement work in healthcare has been seriously underrecognised. We join others in proposing that more informed use of theory can strengthen improvement programmes and facilitate the evaluation of their effectiveness. Many professionals, including improvement practitioners, are unfortunately mystified-and alienated-by theory, which discourages them from using it in their work. In an effort to demystify theory we make the point in this paper that, far from being discretionary or superfluous, theory ('reason-giving'), both informal and formal, is intimately woven into virtually all human endeavour. We explore the special characteristics of grand, mid-range and programme theory; consider the consequences of misusing theory or failing to use it; review the process of developing and applying programme theory; examine some emerging criteria of 'good' theory; and emphasise the value, as well as the challenge, of combining informal experience-based theory with formal, publicly developed theory. We conclude that although informal theory is always at work in improvement, practitioners are often not aware of it or do not make it explicit. The germane issue for improvement practitioners, therefore, is not whether they use theory but whether they make explicit the particular theory or theories, informal and formal, they actually use.

Rationale for New York City's regulations on nutrition, physical activity, and screen time in early child care centers.

Childhood obesity is associated with health risks in childhood, and it increases the risk of adult obesity, which is associated with many chronic diseases. Therefore, implementing policies that may prevent obesity at young ages is important. In 2007, the New York City Department of Health and Mental Hygiene implemented new regulations for early childhood centers to increase physical activity, limit screen time, and provide healthful beverage offerings (ie, restrict sugar-sweetened beverages for all children, restrict whole milk for those older than 2 years, restrict juice to beverages that are 100% juice and limit serving of juice to only 6 ounces per day, and make water available and accessible at all times). This article explains why these amendments to the Health Code were created, how information about these changes was disseminated, and what training programs were used to help ensure implementation, particularly in high-need neighborhoods.

The evaluation of Arkansas Act 1220 of 2003 to reduce childhood obesity: conceptualization, design, and special challenges.

This article describes the evaluation of the Arkansas Act 1220 of 2003, a comprehensive legislative proposal to address the growing epidemic of childhood obesity through changes in the school environment. In addition, the article discusses specific components of the evaluation that may be applicable to other childhood obesity policy evaluation efforts. The conceptual framework for the evaluation, research questions, and evaluation design are described, along with data collection methods and analysis strategies. A mixed methods approach, including both quantitative (surveys, telephone interviews) and qualitative (key informant interviews, records reviews) approaches, was utilized to collect data from a range of informant groups including parents, adolescents, school principals, school district superintendents, and other stakeholders. Challenges encountered with the evaluation are discussed, as are strategies to overcome those challenges. Now in its 9th year, this evaluation has documented substantial changes to school policies and environments but fewer changes to student and family behaviors. The evaluation may inform the methods of other evaluations of childhood obesity prevention policies, as well as inform policymakers about how quickly they might expect implementation of such policies in their own states and localities and anticipate both positive and adverse outcomes.

Lessons from a mixed-methods approach to evaluating Active Living by Design.

BACKGROUND: Beginning in 2003, Active Living by Design (ALbD) established innovative approaches across 25 communities to increase physical activity through community design, public policies, programming, and communication strategies. PURPOSE: The complexity of the ALbD projects called for a mixed-methods evaluation to understand implementation as well as perceived and actual impacts of these efforts. METHODS: Six primary evaluation methods addressed three primary aims: (1) to assess impacts of physical projects and policy changes on community environments; (2) to document intervention strategies implemented, as well as intended and unintended consequences; and (3) to identify strengths and challenges in planning, developing, and implementing interventions. The ALbD evaluation included cross-site comparisons and more in-depth case studies. This article describes the methods used to address the three aims. RESULTS: Analysis of the strengths and challenges associated with the different methods, including partnership capacity surveys, Concept Mapping, an online progress reporting system, key informant interviews, focus groups, and photos and videos. Additional methods, including environmental audits and direct observation, were explored to specifically assess environmental changes. Several important challenges included the lack of baseline data, difficulty in evaluating natural experiments, the need for ongoing policy surveillance, and the need to capture longer-term endpoints. CONCLUSIONS: The mixed-methods evaluation of ALbD advances implementation and evaluation science related to community-based efforts for promoting active living through identification of methods and measures to capture multicomponent and complex interventions as well as translation of a range of approaches to create community change across a variety of populations and settings.

Concept mapping: priority community strategies to create changes to support active living.

BACKGROUND: From 2003 to 2008, a total of 25 cross-sector, multidisciplinary community partnerships funded through the Active Living by Design (ALbD) national program designed, planned, and implemented policy and environmental changes, with complementary programs and promotions. PURPOSE: This paper describes the use of concept mapping methods to gain insights into promising active living intervention strategies based on the collective experience of community representatives implementing ALbD initiatives. METHODS: Using Concept Systems software, community representatives (n=43) anonymously generated actions and changes in their communities to support active living (183 original statements, 79 condensed statements). Next, respondents (n=26, from 23 partnerships) sorted the 79 statements into self-created categories, or active living intervention approaches. Respondents then rated statements based on their perceptions of the most important strategies for creating community changes (n=25, from 22 partnerships) and increasing community rates of physical activity (n=23, from 20 partnerships). Cluster analysis and multidimensional scaling were used to describe data patterns. RESULTS: ALbD community partnerships identified three active living intervention approaches with the greatest perceived importance to create community change and increase population levels of physical activity: changes to the built and natural environment, partnership and collaboration efforts, and land-use and transportation policies. The relative importance of intervention approaches varied according to subgroups of partnerships working with different populations. CONCLUSIONS: Decision makers, practitioners, and community residents can incorporate what has been learned from the 25 community partnerships to prioritize active living policy, physical project, promotional, and programmatic strategies for work in different populations and settings.

Evaluation of the healthy schools program: Part II. The role of technical assistance.

INTRODUCTION: Evidence-based technical assistance may be needed to implement recent federal policy to prevent childhood obesity through the schools. The Healthy Schools Program is the largest school-based obesity prevention program in the United States. The objectives of this study were to evaluate the role of the program's training and technical assistance and to explore other contributing factors in changing school policies, practices, and environments. METHODS: We analyzed interim progress of schools recruited during the 2007-2008 and 2008-2009 school years as of December 2010. Schools reported progress through an online inventory of policies, practices, and school environment. We compared baseline inventories to the most recent follow-up and tracked both training attendance and contact with national experts. To identify the factors associated with progress, we performed regression analysis on school level and demographics, number of months between baseline and follow-up, and technical assistance. RESULTS: The amount of training and technical assistance was significantly associated with school progress, controlling for school level and demographics, number of months between baseline and follow-up, and school status at baseline. Although all types of schools saw progress, schools in the South had the most progress and urban schools had the least progress. CONCLUSION: Evidence-based training and technical assistance were associated with school progress in changing policies, practices, and environment to prevent childhood obesity.

Evaluation of the healthy schools program: Part I. Interim progress.

INTRODUCTION: Federal and state policies identify schools as a setting to prevent childhood obesity, but schools need better health-promoting strategies. The objective of this study was to evaluate interim progress in schools receiving hands-on training from the Healthy Schools Program, the nation's largest school-based program aimed at preventing childhood obesity. The 4-year program targets schools with predominantly low-income, African American, or Hispanic students. METHODS: In 2010 we assessed schools that enrolled in the 2007-2008 and 2008-2009 school years. School representatives completed an inventory of 8 content areas: policy and systems, school meals, competitive foods and beverages, health education, physical education, physical activity outside of physical education, before- and after-school programs, and school employee wellness. Schools' baseline inventory was compared by t test with the most recent inventory available. RESULTS: Schools made significant changes in all content areas, and effect sizes were moderate to large. CONCLUSION: Participating schools improved environmental policies and practices to prevent childhood obesity. The program is a resource to implement recent federal and state policies.

Strategies proposed by Healthy Kids, Healthy Communities partnerships to prevent childhood obesity.

INTRODUCTION: Healthy Kids, Healthy Communities (HKHC) is an initiative of the Robert Wood Johnson Foundation to prevent obesity among high-risk children by changing local policies, systems, and environments. In 2009, 105 community partnerships applied for funding from HKHC. Later that year, the Centers for Disease Control and Prevention (CDC) released recommended community strategies to prevent obesity by changing environments and policies. The objective of this analysis was to describe the strategies proposed by the 41 HKHC partnerships that received funding and compare them to the CDC recommendations. METHODS: We analyzed the funded proposals to assess the types and prevalence of the strategies proposed and mapped them onto the CDC recommendations. RESULTS: The most prevalent strategies proposed by HKHC-funded partnerships were providing incentives to retailers to locate and serve healthier foods in underserved areas, improving mechanisms for purchasing food from farms, enhancing infrastructure that supports walking and cycling, and improving access to outdoor recreational facilities. CONCLUSION: The strategies proposed by HKHC partnerships were well aligned with the CDC recommendations. The popular strategies proposed by HKHC partnerships were those for which there were existing examples of successful implementation. Our analysis provides an example of how information from communities, obtained through grant-writing efforts, can be used to assess the status of the field, guide future research, and provide direction for future investments.

Results of evaluability assessments of local wellness policies in 6 US school districts.

BACKGROUND: A US federal mandate that school districts devise and implement local wellness policies (LWPs) has potential widespread impact on the nutritional content of foods and beverages available in schools and on the amount of physical activity that students engage in; however, evidence concerning the mandate's effectiveness is limited. This study describes the content of LWPs of 6 US school districts and steps taken toward their implementation and evaluation. METHODS: During visits to 6 school districts, we interviewed 88 school and community representatives about the content of their district's LWPs and how the LWPs were being implemented and evaluated. RESULTS: The 6 LWPs were consistent with the federal mandate, although they varied in content and degree of specificity, and none had been fully implemented. All 6 districts were pursuing strategies to ensure that foods and beverages available at school met nutrition standards but did not offer nutrition education to all K-12 students. All 6 districts offered students only limited opportunities for physical activity, and all 6 collected data to monitor process and outcomes of their LWPs. CONCLUSIONS: Partial implementation of LWPs in the districts we visited resulted in significant improvement in the nutritional quality of foods available at district schools, but only slight improvement in students' opportunities for school-based physical activity. We provide recommendations for school districts on implementation and evaluation. Future research is needed to determine the impact of these LWPs on students' health.