CDC Program Evaluation Framework, 2024.

Program evaluation is a critical tool for understanding and improving organizational activities and systems. This report updates the 1999 CDC Framework for Program Evaluation in Public Health (CDC. Framework for program evaluation in public health. MMWR Recomm Rep 1999;48[No. RR-11];1-40) by integrating major advancements in the fields of evaluation and public health, lessons learned from practical applications of the original framework, and current Federal agency policies and practices. A practical, nonprescriptive tool, the updated 2024 framework is designed to summarize and organize essential elements of program evaluation, and can be applied at any level from individual programs to broader systems by novices and experts for planning and implementing an evaluation. Although many of the key aspects from the 1999 framework remain, certain key differences exist. For example, this updated framework also includes six steps that describe the general process of evaluation planning and implementation, but some content and step names have changed (e.g., the first step has been renamed Assess context). The standards for high-quality evaluation remain central to the framework, although they have been updated to the five Federal evaluation standards. The most substantial change from the 1999 framework is the addition of three cross-cutting actions that are core tenets to incorporate within each evaluation step: engage collaboratively, advance equity, and learn from and use insights. The 2024 framework provides a guide for designing and conducting evaluation across many topics within and outside of public health that anyone involved in program evaluation efforts can use alone or in conjunction with other evaluation approaches, tools, or methods to build evidence, understand programs, and refine evidence-based decision-making to improve all program outcomes.

Evaluating the quality of research co-production: Research Quality Plus for Co-Production (RQ + 4 Co-Pro).

BACKGROUND: Co-production is an umbrella term used to describe the process of generating knowledge through partnerships between researchers and those who will use or benefit from research. Multiple advantages of research co-production have been hypothesized, and in some cases documented, in both the academic and practice record. However, there are significant gaps in understanding how to evaluate the quality of co-production. This gap in rigorous evaluation undermines the potential of both co-production and co-producers. METHODS: This research tests the relevance and utility of a novel evaluation framework: Research Quality Plus for Co-Production (RQ + 4 Co-Pro). Following a co-production approach ourselves, our team collaborated to develop study objectives, questions, analysis, and results sharing strategies. We used a dyadic field-test design to execute RQ + 4 Co-Pro evaluations amongst 18 independently recruited subject matter experts. We used standardized reporting templates and qualitative interviews to collect data from field-test participants, and thematic assessment and deliberative dialogue for analysis. Main limitations include that field-test participation included only health research projects and health researchers and this will limit perspective included in the study, and, that our own co-production team does not include all potential perspectives that may add value to this work. RESULTS: The field test surfaced strong support for the relevance and utility of RQ + 4 Co-Pro as an evaluation approach and framework. Research participants shared opportunities for fine-tuning language and criteria within the prototype version, but also, for alternative uses and users of RQ + 4 Co-Pro. All research participants suggested RQ + 4 Co-Pro offered an opportunity for improving how co-production is evaluated and advanced. This facilitated our revision and publication herein of a field-tested RQ + 4 Co-Pro Framework and Assessment Instrument. CONCLUSION: Evaluation is necessary for understanding and improving co-production, and, for ensuring co-production delivers on its promise of better health.. RQ + 4 Co-Pro provides a practical evaluation approach and framework that we invite co-producers and stewards of co-production-including the funders, publishers, and universities who increasingly encourage socially relevant research-to study, adapt, and apply.

Sub-County Life Expectancy: A Tool to Improve Community Health and Advance Health Equity.

Compared with people in other developed countries, Americans live shorter lives, have more disease and disability, and lag on most population health measures. Recent research suggests that this poor comparative performance is primarily driven by profound local place-based disparities. Several initiatives successfully used sub-county life expectancy estimates to identify geographic disparities, generate widespread interest, and catalyze multisector actions. To explore the feasibility of scaling these efforts, the Centers for Disease Control and Prevention and the Council of State and Territorial Epidemiologists initiated a multiphase project - the Sub-County Assessment of Life Expectancy. Phase I participants reviewed the literature, assessed and identified appropriate tools, calculated locally relevant estimates, and developed methodological guidance. Phase I results suggest that most state and local health departments will be able to calculate actionable sub-county life expectancy estimates despite varying resources, expertise, and population sizes, densities, and geographies. To accelerate widespread scaling, we describe several successful case examples, identify user-friendly validated tools, and provide practical tips that resulted from lessons learned.

Mapping the evaluation capacity building landscape: A bibliometric analysis of scholarly communities and themes.

Evaluation capacity building (ECB) continues to attract attention. Over the past two decades, a broad literature has emerged-covering the dimensions, contexts, and practices of ECB. This article presents findings from a bibliometric analysis of ECB articles published in six evaluation journals from 2000 to 2019. The findings shed light on the communities of scholars that contribute to the ECB knowledge base, the connections between these communities, and the themes they cover. Informed by the findings, future directions for ECB scholarship and how bibliometric analysis can supplement more established approaches to literature reviews are discussed.

Evaluating research co-production: protocol for the Research Quality Plus for Co-Production (RQ+ 4 Co-Pro) framework.

BACKGROUND: Research co-production is an umbrella term used to describe research users and researchers working together to generate knowledge. Research co-production is used to create knowledge that is relevant to current challenges and to increase uptake of that knowledge into practice, programs, products, and/or policy. Yet, rigorous theories and methods to assess the quality of co-production are limited. Here we describe a framework for assessing the quality of research co-production-Research Quality Plus for Co-Production (RQ+ 4 Co-Pro)-and outline our field test of this approach. METHODS: Using a co-production approach, we aim to field test the relevance and utility of the RQ+ 4 Co-Pro framework. To do so, we will recruit participants who have led research co-production projects from the international Integrated Knowledge Translation Research Network. We aim to sample 16 to 20 co-production project leads, assign these participants to dyadic groups (8 to 10 dyads), train each participant in the RQ+ 4 Co-Pro framework using deliberative workshops and oversee a simulation assessment exercise using RQ+ 4 Co-Pro within dyadic groups. To study this experience, we use a qualitative design to collect participant demographic information and project demographic information and will use in-depth semi-structured interviews to collect data related to the experience each participant has using the RQ+ 4 Co-Pro framework. DISCUSSION: This study will yield knowledge about a new way to assess research co-production. Specifically, it will address the relevance and utility of using RQ+ 4 Co-Pro, a framework that includes context as an inseparable component of research, identifies dimensions of quality matched to the aims of co-production, and applies a systematic and transferable evaluative method for reaching conclusions. This is a needed area of innovation for research co-production to reach its full potential. The findings may benefit co-producers interested in understanding the quality of their work, but also other stewards of research co-production. Accordingly, we undertake this study as a co-production team representing multiple perspectives from across the research enterprise, such as funders, journal editors, university administrators, and government and health organization leaders.

Evaluative Thinking in Practice: The National Asthma Control Program.

Although evaluative thinking lies at the heart of what we do as evaluators and what we hope to promote in others through our efforts to build evaluation capacity, researchers have given limited attention to measuring this concept. We undertook a research study to better understand how instances of evaluative thinking may present in practice-based settings-specifically within four state asthma control programs funded by the Centers for Disease Control and Prevention's National Asthma Control Program. Through content analyses of documents as well as interviews and a subsequent focus group with four state asthma control programs' evaluators and program managers we identified and defined twenty-two indicators of evaluative thinking. Findings provide insights about what practitioners may wish to look for when they intend to build evaluative thinking and the types of data sources that may be more or less helpful in such efforts.

Evaluation of scanning 2D barcoded vaccines to improve data accuracy of vaccines administered.

BACKGROUND AND OBJECTIVE: Accurately recording vaccine lot number, expiration date, and product identifiers, in patient records is an important step in improving supply chain management and patient safety in the event of a recall. These data are being encoded on two-dimensional (2D) barcodes on most vaccine vials and syringes. Using electronic vaccine administration records, we evaluated the accuracy of lot number and expiration date entered using 2D barcode scanning compared to traditional manual or drop-down list entry methods. METHODS: We analyzed 128,573 electronic records of vaccines administered at 32 facilities. We compared the accuracy of records entered using 2D barcode scanning with those entered using traditional methods using chi-square tests and multilevel logistic regression. RESULTS: When 2D barcodes were scanned, lot number data accuracy was 1.8 percentage points higher (94.3-96.1%, P<0.001) and expiration date data accuracy was 11 percentage points higher (84.8-95.8%, P<0.001) compared with traditional methods. In multivariate analysis, lot number was more likely to be accurate (aOR=1.75; 99% CI, 1.57-1.96) as was expiration date (aOR=2.39; 99% CI, 2.12-2.68). When controlling for scanning and other factors, manufacturer, month vaccine was administered, and vaccine type were associated with variation in accuracy for both lot number and expiration date. CONCLUSION: Two-dimensional barcode scanning shows promise for improving data accuracy of vaccine lot number and expiration date records. Adapting systems to further integrate with 2D barcoding could help increase adoption of 2D barcode scanning technology.