Evaluation of Diverse Community Asthma Interventions: Balancing Health Outcomes with Developing Community Capacity for Evidence-Based Program Measurement.

The challenge of evaluating community asthma management programs is complicated by balancing the emphasis on health outcomes with the need to build community process capacity for conducting and monitoring evidence-based programs. The evaluation of a Georgia Childhood Asthma Management Program, a Healthcare Georgia Foundation-supported initiative for multiple diverse programs and settings, provides an example of an approach and the results that address this challenge. A "developmental evaluation" approach was applied, using mixed methods of quantitative and qualitative data collection and analysis, to assess the progress of community asthma prevention programs in building community within the context of: where the community is starting, community-level systems changes, and the community's progress toward becoming more outcome measurement oriented and evidence based. Initial evaluation efforts revealed extensive mobilization of community assets to manage childhood asthma. However, there were minimal planned efforts to assess health outcomes and systems changes, and the lack of a logic model-based program design linking evidence-based practices to outcomes. Following developmental technical assistance within evaluation efforts, all programs developed logic models, linking practices to outcomes with data collection processes to assess progress toward achieving the selected outcomes. This developmental approach across diverse projects and communities, along with a quality improvement benchmarking approach to outcomes, created a focus on health status outcome improvement. Specifically, this approach complemented an emphasis on an improved community process capacity to identify, implement, and monitor evidence-based asthma practices that could be used within each community setting.

A merged model of quality improvement and evaluation: maximizing return on investment.

Quality improvement (QI) and evaluation are frequently considered to be alternative approaches for monitoring and assessing program implementation and impact. The emphasis on third-party evaluation, particularly associated with summative evaluation, and the grounding of evaluation in the social and behavioral science contrast with an emphasis on the integration of QI process within programs or organizations and its origins in management science and industrial engineering. Working with a major philanthropic organization in Georgia, we illustrate how a QI model is integrated with evaluation for five asthma prevention and control sites serving poor and underserved communities in rural and urban Georgia. A primary foundation of this merged model of QI and evaluation is a refocusing of the evaluation from an intimidating report card summative evaluation by external evaluators to an internally engaged program focus on developmental evaluation. The benefits of the merged model to both QI and evaluation are discussed. The use of evaluation based logic models can help anchor a QI program in evidence-based practice and provide linkage between process and outputs with the longer term distal outcomes. Merging the QI approach with evaluation has major advantages, particularly related to enhancing the funder's return on investment. We illustrate how a Plan-Do-Study-Act model of QI can (a) be integrated with evaluation based logic models, (b) help refocus emphasis from summative to developmental evaluation, (c) enhance program ownership and engagement in evaluation activities, and (d) increase the role of evaluators in providing technical assistance and support.