Evaluating the quality of research co-production: Research Quality Plus for Co-Production (RQ + 4 Co-Pro).

BACKGROUND: Co-production is an umbrella term used to describe the process of generating knowledge through partnerships between researchers and those who will use or benefit from research. Multiple advantages of research co-production have been hypothesized, and in some cases documented, in both the academic and practice record. However, there are significant gaps in understanding how to evaluate the quality of co-production. This gap in rigorous evaluation undermines the potential of both co-production and co-producers. METHODS: This research tests the relevance and utility of a novel evaluation framework: Research Quality Plus for Co-Production (RQ + 4 Co-Pro). Following a co-production approach ourselves, our team collaborated to develop study objectives, questions, analysis, and results sharing strategies. We used a dyadic field-test design to execute RQ + 4 Co-Pro evaluations amongst 18 independently recruited subject matter experts. We used standardized reporting templates and qualitative interviews to collect data from field-test participants, and thematic assessment and deliberative dialogue for analysis. Main limitations include that field-test participation included only health research projects and health researchers and this will limit perspective included in the study, and, that our own co-production team does not include all potential perspectives that may add value to this work. RESULTS: The field test surfaced strong support for the relevance and utility of RQ + 4 Co-Pro as an evaluation approach and framework. Research participants shared opportunities for fine-tuning language and criteria within the prototype version, but also, for alternative uses and users of RQ + 4 Co-Pro. All research participants suggested RQ + 4 Co-Pro offered an opportunity for improving how co-production is evaluated and advanced. This facilitated our revision and publication herein of a field-tested RQ + 4 Co-Pro Framework and Assessment Instrument. CONCLUSION: Evaluation is necessary for understanding and improving co-production, and, for ensuring co-production delivers on its promise of better health.. RQ + 4 Co-Pro provides a practical evaluation approach and framework that we invite co-producers and stewards of co-production-including the funders, publishers, and universities who increasingly encourage socially relevant research-to study, adapt, and apply.

Improving WHO's understanding of WHO guideline uptake and use in Member States: a scoping review.

BACKGROUND: WHO publishes public health and clinical guidelines to guide Member States in achieving better health outcomes. Furthermore, WHO's Thirteenth General Programme of Work for 2019-2023 prioritizes strengthening its normative functional role and uptake of normative and standard-setting products, including guidelines at the country level. Therefore, understanding WHO guideline uptake by the Member States, particularly the low- and middle-income countries (LMICs), is of utmost importance for the organization and scholarship. METHODS: We conducted a scoping review using a comprehensive search strategy to include published literature in English between 2007 and 2020. The review was conducted between May and June 2021. We searched five electronic databases including CINAHL, the Cochrane Library, PubMed, Embase and Scopus. We also searched Google Scholar as a supplementary source. The review adhered to the PRISMA-ScR (PRISMA extension for scoping reviews) guidelines for reporting the searches, screening and identification of evaluation studies from the literature. A narrative synthesis of the evidence around key barriers and challenges for WHO guideline uptake in LMICs is thematically presented. RESULTS: The scoping review included 48 studies, and the findings were categorized into four themes: (1) lack of national legislation, regulations and policy coherence, (2) inadequate experience, expertise and training of healthcare providers for guideline uptake, (3) funding limitations for guideline uptake and use, and (4) inadequate healthcare infrastructure for guideline compliance. These challenges were situated in the Member States' health systems. The findings suggest that governance was often weak within the existing health systems amongst most of the LMICs studied, as was the guidance provided by WHO's guidelines on governance requirements. This challenge was further exacerbated by a lack of accountability and transparency mechanisms for uptake and implementation of guidelines. In addition, the WHO guidelines themselves were either unclear and were technically challenging for some health conditions; however, WHO guidelines were primarily used as a reference by Member States when they developed their national guidelines. CONCLUSIONS: The challenges identified reflect the national health systems' (in)ability to allocate, implement and monitor the guidelines. Historically this is beyond the remit of WHO, but Member States could benefit from WHO implementation guidance on requirements and needs for successful uptake and use of WHO guidelines.

Back to the future: Are we trapped in our past?

This paper is a reflection on how evaluators could approach theories of change through a foresight lens. It explores the role of assumptions and especially anticipatory assumptions in how we design our theories of change. It argues for a more open approach, a transdisciplinary approach to the multiple knowledges we bring to bear. It goes on to argue that if we do not build and use our imaginations to think differently about the future than the past, as evaluators we risk being trapped in findings and recommendations that assume continuity in a highly discontinuous world.

Evaluating research co-production: protocol for the Research Quality Plus for Co-Production (RQ+ 4 Co-Pro) framework.

BACKGROUND: Research co-production is an umbrella term used to describe research users and researchers working together to generate knowledge. Research co-production is used to create knowledge that is relevant to current challenges and to increase uptake of that knowledge into practice, programs, products, and/or policy. Yet, rigorous theories and methods to assess the quality of co-production are limited. Here we describe a framework for assessing the quality of research co-production-Research Quality Plus for Co-Production (RQ+ 4 Co-Pro)-and outline our field test of this approach. METHODS: Using a co-production approach, we aim to field test the relevance and utility of the RQ+ 4 Co-Pro framework. To do so, we will recruit participants who have led research co-production projects from the international Integrated Knowledge Translation Research Network. We aim to sample 16 to 20 co-production project leads, assign these participants to dyadic groups (8 to 10 dyads), train each participant in the RQ+ 4 Co-Pro framework using deliberative workshops and oversee a simulation assessment exercise using RQ+ 4 Co-Pro within dyadic groups. To study this experience, we use a qualitative design to collect participant demographic information and project demographic information and will use in-depth semi-structured interviews to collect data related to the experience each participant has using the RQ+ 4 Co-Pro framework. DISCUSSION: This study will yield knowledge about a new way to assess research co-production. Specifically, it will address the relevance and utility of using RQ+ 4 Co-Pro, a framework that includes context as an inseparable component of research, identifies dimensions of quality matched to the aims of co-production, and applies a systematic and transferable evaluative method for reaching conclusions. This is a needed area of innovation for research co-production to reach its full potential. The findings may benefit co-producers interested in understanding the quality of their work, but also other stewards of research co-production. Accordingly, we undertake this study as a co-production team representing multiple perspectives from across the research enterprise, such as funders, journal editors, university administrators, and government and health organization leaders.

Asking questions in the solution space: methodological issues in evaluating equity.

This paper presents an overview of some core methodological issues in improving the evaluation of health equity interventions. It argues that evaluation can play a central role in the solution space if it takes a futures orientation and develops adaptive approaches. It makes the case that purpose must drive method and that clarity in values is central. It suggests a process to rethink health equity interventions, reshape policy based on evaluation at the systems level, and reform evaluation of health equity so that it has the capacity to adapt and to more realistically reflect the dynamic and changing nature of systems.