Generating demand for and use of evaluation evidence in government health ministries: lessons from a pilot programme in Uganda and Zambia.

BACKGROUND: The Demand-Driven Evaluations for Decisions (3DE) programme was piloted in Zambia and Uganda in 2012-2015. It aimed to answer evaluative questions raised by policymakers in Ministries of Health, rapidly and with limited resources. The aim of our evaluation was to assess whether the 3DE model was successful in supporting and increasing evidence-based policymaking, building capacity and changing behaviour of Ministry staff. METHODS: Using mixed methods, we compared the ex-ante theory of change with what had happened in practice, why and with what results (intended and unintended), including a qualitative assessment of 3DE's contribution. Data sources included a structured quality assessment of the five impact evaluations produced, 46 key informant interviews at national and international levels, structured extraction from 170 programme documents, a wider literature review of relevant topics, and a political economy analysis conducted in Zambia. RESULTS: We found that 3DE had a very limited contribution to changing evidence-based policymaking, capacity and behaviour in both countries as a result of having a number of aspirations not all compatible with one another. Co-developing evaluation questions was more time-consuming than anticipated, Ministry evidence needs did not fit neatly into questions suitable for impact evaluations and constricted timeframes for undertaking trials did not necessarily produce the most effective results and value for money. The evaluation recommended a focusing of objectives and a more strategic approach to strengthening evaluative demand and capacity. CONCLUSIONS: Lessons emerge that are likely to apply in other low- and middle-income settings, such as the importance of supporting evaluative thinking and capacity within wider institutions, of understanding the political economy of evidence use and its uptake, and of allowing for some flexibility in terms of programme targets. Fixating on one type of evidence is unhelpful in the context of institutions like ministries of health, which require a wide range of evidence to plan and deliver programmes. In addition, having success tied to indicators, such as number of 'policy decisions made', provides potentially perverse incentives and neglects arguably more important aspects such as incremental programmatic adjustments and improved implementation.

A collaboratively-derived science-policy research agenda.

The need for policy makers to understand science and for scientists to understand policy processes is widely recognised. However, the science-policy relationship is sometimes difficult and occasionally dysfunctional; it is also increasingly visible, because it must deal with contentious issues, or itself becomes a matter of public controversy, or both. We suggest that identifying key unanswered questions on the relationship between science and policy will catalyse and focus research in this field. To identify these questions, a collaborative procedure was employed with 52 participants selected to cover a wide range of experience in both science and policy, including people from government, non-governmental organisations, academia and industry. These participants consulted with colleagues and submitted 239 questions. An initial round of voting was followed by a workshop in which 40 of the most important questions were identified by further discussion and voting. The resulting list includes questions about the effectiveness of science-based decision-making structures; the nature and legitimacy of expertise; the consequences of changes such as increasing transparency; choices among different sources of evidence; the implications of new means of characterising and representing uncertainties; and ways in which policy and political processes affect what counts as authoritative evidence. We expect this exercise to identify important theoretical questions and to help improve the mutual understanding and effectiveness of those working at the interface of science and policy.

Structuring policy problems for plastics, the environment and human health: reflections from the UK.

How can we strengthen the science-policy interface for plastics, the environment and human health? In a complex policy area with multiple stakeholders, it is important to clarify the nature of the particular plastics-related issue before trying to understand how to reconcile the supply and demand for evidence in policy. This article proposes a simple problem typology to assess the fundamental characteristics of a policy issue and thus identify appropriate processes for science-policy interactions. This is illustrated with two case studies from one UK Government Department, showing how policy and science meet over the environmental problems of plastics waste in the marine environment and on land. A problem-structuring methodology helps us understand why some policy issues can be addressed through relatively linear flows of science from experts to policymakers but why others demand a more reflexive approach to brokering the knowledge between science and policy. Suggestions are given at the end of the article for practical actions that can be taken on both sides.

Using research-based knowledge to underpin waste and resources policy.

The concept of a knowledge-based approach to policy making (also referred to as 'evidence-based policy making' or 'science meets policy') has been around for 10 years, and is widely 'signed up to' by Governments. This paper examines the theory and the practice at national government (using the UK as a case study) and European levels, looking at how the knowledge-based approach has been applied in waste and resources management. There is general agreement that a knowledge-(or evidence-) based approach to policy making is a 'good thing'; however putting the concept into practice has proven challenging and is still evolving. The linkages between researchers and policy makers need to be further strengthened, as does the 'demand' for evidence from policy; requiring policy specialists to articulate their requirements for evidence sufficiently well in advance, and to do so routinely, is still an area of weakness. Using a model developed by the UK's Department for Environment, Food and Rural Affairs (Defra), we examine in detail some of the implications of changing the question from 'is this good research on waste and resources issues?' to 'is this good evidence for waste and resources policy?'. The paper also provides recommendations for further work required to achieve this at a European level, and for priority areas where international co-ordination of waste and resources research could add most value to policy.