A typology of useful evidence: approaches to increase the practical value of intervention research.

BACKGROUND: Too often, studies of evidence-based interventions (EBIs) in preventive, community, and health care are not sufficiently useful to end users (typically practitioners, patients, policymakers, or other researchers). The ways in which intervention studies are conventionally conducted and reported mean that there is often a shortage of information when an EBI is used in practice. The paper aims to invite the research community to consider ways to optimize not only the trustworthiness but also the research's usefulness in intervention studies. This is done by proposing a typology that provides some approaches to useful EBIs for intervention researchers. The approaches originate from different research fields and are summarized to highlight their potential benefits from a usefulness perspective. MAIN MESSAGE: The typology consists of research approaches to increase the usefulness of EBIs by improving the reporting of four features in intervention studies: (1) the interventions themselves, including core components and appropriate adaptations; (2) strategies to support-high-quality implementation of the interventions; (3) generalizations about the evidence in a variety of contexts; and (4) outcomes based on end users' preferences and knowledge. The research approaches fall into three levels: Description, Analysis, and Design. The first level, Description, outlines what types of information about the intervention and its implementation, context, and outcomes can be helpful for end users. Research approaches under analysis offers alternative ways of analyzing data, increasing the precision of information provided to end users. Approaches summarized under design involve more radical changes and far-reaching implications for how research can provide more useful information. These approaches partly flip the order of efficacy and effectiveness, focusing not on whether an intervention works in highly controlled and optimal circumstances, but first and foremost whether an intervention can be implemented and lead to anticipated outcomes in everyday practice. CONCLUSIONS: The research community, as well as the end users of research, are invited to consider ways to optimize research's usefulness as well as its trustworthiness. Many of the research approaches in the typology are not new, and their contributions to quality have been described for generations - but their contributions to useful knowledge need more attention.

Generalizing about Public Health Interventions: A Mixed-Methods Approach to External Validity.

Public health researchers and practitioners are calling for greater focus on external validity, the ability to generalize findings of evidence-based interventions (EBIs) beyond the limited number of studies testing effectiveness. For public health, the goal is applicability: to translate, disseminate, and implement EBIs for an impact on population health. This article is a review of methods and how they might be combined to better assess external validity. The methods include (a) better description of EBIs and their contexts; (b) combining of statistical tools and logic to draw inferences about study samples; (c) sharper definition of the theory behind the intervention and core intervention components; and (d) more systematic consultation of practitioners. For population impact, studies should focus on context features that are likely to be both important (based on program theory) and frequently encountered by practitioners. Mixed-method programs of research will allow public health to expand causal generalizations.

Rationale for New York City's regulations on nutrition, physical activity, and screen time in early child care centers.

Childhood obesity is associated with health risks in childhood, and it increases the risk of adult obesity, which is associated with many chronic diseases. Therefore, implementing policies that may prevent obesity at young ages is important. In 2007, the New York City Department of Health and Mental Hygiene implemented new regulations for early childhood centers to increase physical activity, limit screen time, and provide healthful beverage offerings (ie, restrict sugar-sweetened beverages for all children, restrict whole milk for those older than 2 years, restrict juice to beverages that are 100% juice and limit serving of juice to only 6 ounces per day, and make water available and accessible at all times). This article explains why these amendments to the Health Code were created, how information about these changes was disseminated, and what training programs were used to help ensure implementation, particularly in high-need neighborhoods.

The evaluation of Arkansas Act 1220 of 2003 to reduce childhood obesity: conceptualization, design, and special challenges.

This article describes the evaluation of the Arkansas Act 1220 of 2003, a comprehensive legislative proposal to address the growing epidemic of childhood obesity through changes in the school environment. In addition, the article discusses specific components of the evaluation that may be applicable to other childhood obesity policy evaluation efforts. The conceptual framework for the evaluation, research questions, and evaluation design are described, along with data collection methods and analysis strategies. A mixed methods approach, including both quantitative (surveys, telephone interviews) and qualitative (key informant interviews, records reviews) approaches, was utilized to collect data from a range of informant groups including parents, adolescents, school principals, school district superintendents, and other stakeholders. Challenges encountered with the evaluation are discussed, as are strategies to overcome those challenges. Now in its 9th year, this evaluation has documented substantial changes to school policies and environments but fewer changes to student and family behaviors. The evaluation may inform the methods of other evaluations of childhood obesity prevention policies, as well as inform policymakers about how quickly they might expect implementation of such policies in their own states and localities and anticipate both positive and adverse outcomes.

Evaluation of the healthy schools program: Part II. The role of technical assistance.

INTRODUCTION: Evidence-based technical assistance may be needed to implement recent federal policy to prevent childhood obesity through the schools. The Healthy Schools Program is the largest school-based obesity prevention program in the United States. The objectives of this study were to evaluate the role of the program's training and technical assistance and to explore other contributing factors in changing school policies, practices, and environments. METHODS: We analyzed interim progress of schools recruited during the 2007-2008 and 2008-2009 school years as of December 2010. Schools reported progress through an online inventory of policies, practices, and school environment. We compared baseline inventories to the most recent follow-up and tracked both training attendance and contact with national experts. To identify the factors associated with progress, we performed regression analysis on school level and demographics, number of months between baseline and follow-up, and technical assistance. RESULTS: The amount of training and technical assistance was significantly associated with school progress, controlling for school level and demographics, number of months between baseline and follow-up, and school status at baseline. Although all types of schools saw progress, schools in the South had the most progress and urban schools had the least progress. CONCLUSION: Evidence-based training and technical assistance were associated with school progress in changing policies, practices, and environment to prevent childhood obesity.

Evaluation of the healthy schools program: Part I. Interim progress.

INTRODUCTION: Federal and state policies identify schools as a setting to prevent childhood obesity, but schools need better health-promoting strategies. The objective of this study was to evaluate interim progress in schools receiving hands-on training from the Healthy Schools Program, the nation's largest school-based program aimed at preventing childhood obesity. The 4-year program targets schools with predominantly low-income, African American, or Hispanic students. METHODS: In 2010 we assessed schools that enrolled in the 2007-2008 and 2008-2009 school years. School representatives completed an inventory of 8 content areas: policy and systems, school meals, competitive foods and beverages, health education, physical education, physical activity outside of physical education, before- and after-school programs, and school employee wellness. Schools' baseline inventory was compared by t test with the most recent inventory available. RESULTS: Schools made significant changes in all content areas, and effect sizes were moderate to large. CONCLUSION: Participating schools improved environmental policies and practices to prevent childhood obesity. The program is a resource to implement recent federal and state policies.

Strategies proposed by Healthy Kids, Healthy Communities partnerships to prevent childhood obesity.

INTRODUCTION: Healthy Kids, Healthy Communities (HKHC) is an initiative of the Robert Wood Johnson Foundation to prevent obesity among high-risk children by changing local policies, systems, and environments. In 2009, 105 community partnerships applied for funding from HKHC. Later that year, the Centers for Disease Control and Prevention (CDC) released recommended community strategies to prevent obesity by changing environments and policies. The objective of this analysis was to describe the strategies proposed by the 41 HKHC partnerships that received funding and compare them to the CDC recommendations. METHODS: We analyzed the funded proposals to assess the types and prevalence of the strategies proposed and mapped them onto the CDC recommendations. RESULTS: The most prevalent strategies proposed by HKHC-funded partnerships were providing incentives to retailers to locate and serve healthier foods in underserved areas, improving mechanisms for purchasing food from farms, enhancing infrastructure that supports walking and cycling, and improving access to outdoor recreational facilities. CONCLUSION: The strategies proposed by HKHC partnerships were well aligned with the CDC recommendations. The popular strategies proposed by HKHC partnerships were those for which there were existing examples of successful implementation. Our analysis provides an example of how information from communities, obtained through grant-writing efforts, can be used to assess the status of the field, guide future research, and provide direction for future investments.

Evaluability assessment to improve public health policies, programs, and practices.

Evaluability assessment, also commonly known as exploratory evaluation, has assisted the field of public health to improve programs and to develop a pragmatic, practice-based research agenda. Evaluability assessment was originally developed as a low-cost pre-evaluation activity to prepare better for conventional evaluations of programs, practices, and some policies. For public health programs, however, it serves several other important purposes: (a) giving program staff rapid, constructive feedback about program operations; (b) assisting the core public health planning and assurance functions by helping to develop realistic objectives and providing low-cost, rapid feedback on implementation; (c) navigating federal performance measurement requirements; (d) translating research into practice by examining the feasibility, acceptability, and adaptation of evidence-based practices in new settings and populations; and (e) translating practice into research by identifying promising new approaches to achieve public health goals.

Measuring food environments: a guide to available instruments.

PURPOSE: Food environments affect people's food-related decisions, consumption, and health outcomes. This paper critiques instruments available for assessing food environments to help researchers, practitioners, and community organizations select those that best meet their needs, resources, and level of expertise. Gaps and recommendations for future instrument development are discussed. METHOD: Instruments were selected by reviewing the literature and contacting researchers and practitioners. Using the social-ecological model, the instruments were classified according to the type of food environments they assess. Each instrument is rated according to criteria for ease of use, detail, resources required, and psychometric testing. RESULTS: Of the 48 instruments identified, only 39% were tested for validity or reliability. Observational tools were the largest category of instruments developed. Fifty-two percent of the instruments provide some level of technical assistance and 37% are considered appropriate for community members. CONCLUSION: Three distinct groups want to assess food environments: researchers, practitioners, and community organizations. These groups have different information needs and different capacities to undertake assessments. At this time there is a tradeoff between simplicity and low cost on the one hand, and detail and accuracy on the other. To choose the most suitable instrument, users should consider the pros and cons of each instrument and base their selection on purpose of their assessment, resources and expertise at hand, and validity, reliability, and ease of use of the instrument.

Understanding the challenges encountered and adaptations made by community organizations in translation of evidence-based behavior change physical activity interventions: a qualitative study.

PURPOSE: Designing programs for mid-life to older adults whose sedentary behaviors are associated with increased health risks is crucial. The U.S. Task Force on Community Preventive Services strongly recommends individually adapted behavior change programs as one approach to increasing physical activity in communities. The purpose of this study is to report challenges organizations faced when translating two evidence-based programs in real-world settings, adaptations made, and whether or not fidelity was negatively impacted by these adaptations. DESIGN: A grounded theory approach to qualitative research was used. SETTING AND PARTICIPANTS: Nine community organizations across the country participated. Two organizations had more than one site participating, for a total of 12 sites from nine organizations. Within those organizations, 2796 participants were part of the program during the first 2 years. Participants were underactive (i.e., not meeting Centers for Disease Control and Prevention and American College of Sports Medicine recommendations) mid- to older-aged adults. METHODS: Community organizations participated in monthly conference calls, and program information was entered into an electronic database regularly. Data obtained from the calls and database were used for analyses. RESULTS: Challenges and adaptations emerged in three categories: (1) program logistics, (2) program theory, and (3) program philosophy. CONCLUSION: Challenges were present for community organizations; however, with some level of adaptation, the community organizations were able to effectively deliver and maintain fidelity in two evidence-based physical activity programs to a large and diverse group of mid- to older-aged adults.

Results from the Active for Life process evaluation: program delivery fidelity and adaptations.

Active for Life((R)) (AFL) was a large (n = 8159) translational initiative to increase physical activity (PA) in midlife and older adults. Translational research calls for a shift in emphasis from just understanding what works (efficacy) to also understanding how it works in more 'real world' settings. This article describes the process evaluation design and findings, discuss how these findings were used to better understand the translational process and provide a set of process evaluation recommendations with community-based translational research. AFL community organizations across the United States implemented one of two evidence-based PA programs (Active Living Every Day-The Cooper Institute; Human Kinetics Inc. or Active Choices-Stanford University). Both programs were based on the transtheoretical model and social cognitive theory. Overall, the process evaluation revealed high-dose delivery and implementation fidelity by quite varied community organizations serving diverse adult populations. Findings reveal most variation occurred for program elements requiring more participant engagement. Additionally, the results show how a collaborative process allowed the organizations to 'fit' the programs to their specific participant base while maintaining fidelity to essential program elements.

Active for life: final results from the translation of two physical activity programs.

BACKGROUND: Most evidence-based programs are never translated into community settings and thus never make a public health impact. DESIGN: Active for Life (AFL) was a 4-year translational initiative using a pre-post, quasi-experimental design. Data were collected from 2003 to 2007. Analyses were conducted in 2005 and 2008. SETTING/PARTICIPANTS: Nine lead organizations at 12 sites participated. Active Choices participants (n=2503) averaged 65.8 years (80% women, 41% non-Hispanic white). Active Living Every Day (ALED) participants (n=3388) averaged 70.6 years (83% women, 64% non-Hispanic white). INTERVENTION: In AFL, Active Choices was a 6-month telephone-based and ALED a 20-week group-based lifestyle behavior change program designed to increase physical activity, and both were grounded in social cognitive theory and the transtheoretical model. The interventions were evaluated in Years 1, 3, and 4. An adapted shortened ALED program was evaluated in Year 4. MAIN OUTCOME MEASURE: Moderate- to vigorous-intensity physical activity, assessed with the CHAMPS self-reported measure. RESULTS: Posttest survey response rates were 61% for Active Choices and 70% for ALED. Significant increases in moderate- to vigorous-intensity physical activity, total physical activity, and satisfaction with body appearance and function, and decreases in BMI were seen for both programs. Depressive symptoms and perceived stress, both low at pretest, also decreased over time in ALED. Results were generally consistent across years and sites. CONCLUSIONS: Active Choices and ALED were successfully translated across a range of real-world settings. Study samples were substantially larger, more ethnically and economically diverse, and more representative of older adult's health conditions than in efficacy studies, yet the magnitude of effect sizes were comparable.

Interaction of Theory and Practice to Assess External Validity.

BACKGROUND: Variations in local context bedevil the assessment of external validity: the ability to generalize about effects of treatments. For evaluation, the challenges of assessing external validity are intimately tied to the translation and spread of evidence-based interventions. This makes external validity a question for decision makers, who need to determine whether to endorse, fund, or adopt interventions that were found to be effective and how to ensure high quality once they spread. OBJECTIVE: To present the rationale for using theory to assess external validity and the value of more systematic interaction of theory and practice. METHODS: We review advances in external validity, program theory, practitioner expertise, and local adaptation. Examples are provided for program theory, its adaptation to diverse contexts, and generalizing to contexts that have not yet been studied. The often critical role of practitioner experience is illustrated in these examples. Work is described that the Robert Wood Johnson Foundation is supporting to study treatment variation and context more systematically. RESULTS: Researchers and developers generally see a limited range of contexts in which the intervention is implemented. Individual practitioners see a different and often a wider range of contexts, albeit not a systematic sample. Organized and taken together, however, practitioner experiences can inform external validity by challenging the developers and researchers to consider a wider range of contexts. Researchers have developed a variety of ways to adapt interventions in light of such challenges. CONCLUSIONS: In systematic programs of inquiry, as opposed to individual studies, the problems of context can be better addressed. Evaluators have advocated an interaction of theory and practice for many years, but the process can be made more systematic and useful. Systematic interaction can set priorities for assessment of external validity by examining the prevalence and importance of context features and treatment variations. Practitioner interaction with researchers and developers can assist in sharpening program theory, reducing uncertainty about treatment variations that are consistent or inconsistent with the theory, inductively ruling out the ones that are harmful or irrelevant, and helping set priorities for more rigorous study of context and treatment variation.

Building a National Culture of Health: Background, Action Framework, Measures, and Next Steps.

Because health is a function of more than medical care, solutions to U.S. health problems must encompass more than reforms to health care systems. But those working to improve health, well-being, and equity still too often find themselves traveling on parallel paths that rarely intersect. In 2013, the Robert Wood Johnson Foundation (RWJF) embarked on a pioneering effort to advance a Culture of Health initiative. A Culture of Health places well-being at the center of every aspect of life, with the goal of enabling everyone in our diverse society to lead healthier lives, now and for generations to come. To put this vision into action, RWJF worked with RAND to develop an action framework that identifies how the nation will work toward achieving these outcomes. This article provides background on the development of this action framework. The Culture of Health action framework is designed around four action areas and one outcome area. Action areas are the core areas in which investment and activity are needed: (1) making health a shared value; (2) fostering cross-sector collaboration to improve well-being; (3) creating healthier, more equitable communities; and (4) strengthening integration of health services and systems. Each action area contains a set of drivers indicating where the United States needs to accelerate change and a set of measures illustrating places for progress. Within the primary Culture of Health outcome---improved population health, well-being, and equity---the authors identified three outcome areas: enhanced individual and community well-being, managed chronic disease and reduced toxic stress, and reduced health care costs.

Results of the first year of active for life: translation of 2 evidence-based physical activity programs for older adults into community settings.

OBJECTIVES: Translating efficacious interventions into practice within community settings is a major public health challenge. We evaluated the effects of 2 evidence-based physical activity interventions on self-reported physical activity and related outcomes in midlife and older adults. METHODS: Four community-based organizations implemented Active Choices, a 6-month, telephone-based program, and 5 implemented Active Living Every Day, a 20-week, group-based program. Both programs emphasize behavioral skills necessary to become more physically active. Participants completed pretest and posttest surveys. RESULTS: Participants (n=838) were aged an average of 68.4 +/-9.4 years, 80.6% were women, and 64.1% were non-Hispanic White. Seventy-two percent returned posttest surveys. Intent-to-treat analyses found statistically significant increases in moderate-to-vigorous physical activity and total physical activity, decreases in depressive symptoms and stress, increases in satisfaction with body appearance and function, and decreases in body mass index. CONCLUSIONS: The first year of Active for Life demonstrated that Active Choices and Active Living Every Day, 2 evidence-based physical activity programs, can be successfully translated into community settings with diverse populations. Further, the magnitudes of change in outcomes were similar to those reported in the efficacy trials.

Balancing stakeholder needs in the evaluation of healthcare quality improvement.

Quality improvement (QI) efforts affect a broader range of people than we often assume. These are the potential stakeholders for QI and its evaluation, and they have valuable perspectives to offer when they are consulted in planning, conducting and interpreting evaluations. QI practitioners are accustomed to consulting stakeholders to assess unintended consequences or assess patient experiences of care, but in many cases there are additional benefits to a broad inclusion of stakeholders. These benefits are better adherence to ethical standards, to assure that all legitimate interests take part, more useful and relevant evaluation information and better political buy-in to improve impact. Balancing various stakeholder needs for information requires skill for both politics and research management. These challenges have few pat answers, but several preferred practices, which are illustrated with practical examples.

Cross-Sector Collaborations And Partnerships: Essential Ingredients To Help Shape Health And Well-Being.

Cross-sector collaborations and partnerships are an essential component of the strategy to improve health and well-being in the United States. While their importance is unquestioned, their impact on population health has not yet been fully observed. Cross-sector collaboration also is the second Action Area of the Robert Wood Johnson Foundation's four-part Action Framework to build a Culture of Health in the United States. This Action Area has three constituent parts or drivers: the number, breadth, and quality of successful cross-sector partnerships; the adequacy of investment in these partnerships; and the adoption of policies needed to support them. In this article we analyze outstanding examples of partnership-driven work. We also study the challenges of how partner sectors outside the formal health system, such as organizations working in the education or housing sectors, can effectively lead collaborations. We identify models of leadership that maximize the potential of all participants. We also propose the adoption of models better suited to supporting effective cross-sector collaborations. The analysis builds the evidence base for understanding and sustaining the impact of cross-sector collaboration on population health.

Strengthening Integration Of Health Services And Systems.

New care delivery models that hold providers more accountable for coordinated, high-quality care and the overall health of their patients have appeared in the US health care system, spurred by recent legislation such as the Affordable Care Act. These models support the integration of health care systems, but maximizing health and well-being for all individuals will require a broader conceptualization of health and more explicit connections between diverse partners. Integration of health services and systems constitutes the fourth Action Area in the Robert Wood Johnson Foundation's Culture of Health Action Framework, which is the subject of this article. This Action Area conceives of a strengthened health care system as one in which medical care, public health, and social services interact to produce a more effective, equitable, higher-value whole that maximizes the production of health and well-being for all individuals. Three critical drivers help define and advance this Action Area and identify gaps and needs that must be addressed to move forward. These drivers are access, balance and integration, and consumer experience and quality. This article discusses each driver and summarizes practice gaps that, if addressed, will help move the nation toward a stronger and more integrated health system.