NGO perspectives on the challenges and opportunities for real-world evaluation: a qualitative study.

BACKGROUND: The move towards robust monitoring and evaluation (M&E) has been increasing in global health, motivated by both an accountability agenda and to increase learning from M&E activities. Many international non-governmental organizations (NGOs) receive funding from one or more large institutional donors. OBJECTIVE: To understand NGOs' perspective on their own role in terms of accountability to both donors and the populations they serve. METHODS: We conducted a series of in-depth interviews with M&E staff in 11 NGOs with projects related to maternal and child health to better understand how M&E is being implemented in these organizations. We then examined the data based on a priori identified themes. RESULTS: We found that despite flexibility from some donors, rigid reporting structures remain a barrier for NGOs to fully communicate the impact of their projects. While NGOs do utilize M&E findings, their use is limited by low staff capacity. The primary audience for the results remains the donor agency, and the primary motivation for M&E remains donor reporting. Reporting remains a burdensome affair, with ongoing limitations around streamlining results for donors. To reduce the burden of reporting for individual projects, the participants in our study suggested placing greater emphasis on process evaluations rather than impact evaluations. Participants also suggested increased data sharing between organizations working in the same regions and making better use of secondary data sources; in both cases to reduce the need for primary data collection. CONCLUSION: We carried out this work to advance the conversation on how NGOs currently manage their M&E - a conversation which should involve NGOs, donors, local health system actors, and the communities with whom they work. More flexibility from donors, increased use of technology, and more transparency on if and how data is being used would help NGOs with their M&E process.

The RADAR coverage tool: developing a toolkit for rigorous household surveys for reproductive, maternal, newborn, and child health & nutrition indicators.

Population-based intervention coverage data are used to inform the design of projects, programs, and policies and to evaluate their impact. In low- and middle-income countries (LMICs), household surveys are the primary source of coverage data. Many coverage surveys are implemented by organizations with limited experience or resources in population-based data collection. We developed a streamlined survey and set of supporting materials to facilitate rigorous survey design and implementation. The RADAR coverage survey tool aimed to 1) rigorously measure priority reproductive, maternal, newborn, child health & nutrition coverage indicators, and allow for equity and gender analyses; 2) use standard, valid questions, to the extent possible; 3) be as light as possible; 4) be flexible to address users' needs; and 5) be compatible with the Lives Saved Tool for analysis of program impact. Early interactions with stakeholders also highlighted survey planning, implementation, and analysis as challenging areas. We therefore developed a suite of resources to support implementers in these areas. The toolkit was piloted by implementers in Tanzania and in Burkina Faso. Although the toolkit was successfully implemented in these settings and facilitated survey planning and implementation, we found that implementers must still have access to sufficient resources, time, and technical expertise in order to use the tool appropriately. This potentially limits the use of the tool to situations where high-quality surveys or evaluations have been prioritized and adequately resourced.

Building coherent monitoring and evaluation plans with the Evaluation Planning Tool for global health.

Practitioners in global health are called to monitor and evaluate their projects. This keeps projects on track, it meets donor and public demand, and it is a key mechanism by which global health organizations hold themselves accountable and improve their community of practice. However, monitoring and evaluation (M&E) is time- and resource-consuming, bringing into question whether the effort expended on M&E is worth it. While there has been a shift towards emphasizing the learning aspect of M&E, non-governmental organizations (NGOs) and other actors still struggle to get value from their efforts. One reason for this is that M&E plans are often not coherent or employed to their full potential. Theories of change, indicator lists, and data collection become a series of disjointed efforts that do not tie together. They become tick-the-box exercises to satisfy donors rather than a logical approach to draw meaningful findings for stakeholders, governments, and local communities. In this paper, we propose a step-by-step approach to utilizing M&E tools to their fullest potential, including: (1) a clearly defined theory of change that captures all program pathways and shows all intermediate objectives needed to achieve impact, (2) indicators which directly reflect the intermediate and ultimate objectives in the theory of change, and (3) a data collection plan which includes appropriate methods to measure indicators and address the questions stakeholders want answered. We make the case for a simpler, more coherent approach to M&E and propose a new tool to help practitioners more easily develop evaluation plans that are rigorous, practical, and worth the effort.

Can We Use Routine Data for Strategic Decision Making? A Time Trend Comparison Between Survey and Routine Data in Mali.

BACKGROUND: Countries with scarce resources need timely and high-quality data on coverage of health interventions to make strategic decisions about where to allocate investments in health. Household survey data are generally regarded as "gold standard," high-quality data. This study assessed the comparability of intervention coverage time trends from routine and survey data at national and subnational levels in Mali. METHODS: We compared 3 coverage indicators: contraceptive prevalence rate, institutional delivery, and 3 doses of diphtheria, pertussis, and tetanus (DPT3) vaccine, using 3 Mali Demographic and Health Surveys (DHS 2001, 2006, and 2012-2013) and routine health system data covering 2001-2012. For routine data, we used local health information system (HIS) annual reports and an HIS database. To compare time trends between the data sources, we calculated the percentage point change and 95% confidence interval from 2001-2006 and 2006-2012. We then computed the absolute and relative differences between the 2 data sources for each indicator over time at national and regional levels and assessed their level of significance. RESULTS: The direction and magnitude of the time trends of contraceptive prevalence rate, institutional delivery, and DPT3 vaccine from 2001 to 2012 were similar at the national level between data sources. At the regional level, there were significant differences in the magnitude and direction of time trends for institutional delivery and the DPT3 vaccine; contraceptive prevalence trends were more consistent. Routine data tended to overestimate DPT3 coverage, and underestimate institutional delivery and contraceptive prevalence relative to survey data. CONCLUSION: Routine data in Mali-particularly at the national level-appear to be appropriate for use to inform program planning and prioritization, but routine time trends should be interpreted with caution at the subnational level. For program evaluations, routine data may not be appropriate to draw accurate inferences about program impact.

The role of gender power relations on women's health outcomes: evidence from a maternal health coverage survey in Simiyu region, Tanzania.

BACKGROUND: Gender is a crucial consideration of human rights that impacts many priority maternal health outcomes. However, gender is often only reported in relation to sex-disaggregated data in health coverage surveys. Few coverage surveys to date have integrated a more expansive set of gender-related questions and indicators, especially in low- to middle-income countries that have high levels of reported gender inequality. Using various gender-sensitive indicators, we investigated the role of gender power relations within households on women's health outcomes in Simiyu region, Tanzania. METHODS: We assessed 34 questions around gender dynamics reported by men and women against 18 women's health outcomes. We created directed acyclic graphs (DAGs) to theorize the relationship between indicators, outcomes, and sociodemographic covariates. We grouped gender variables into four categories using an established gender framework: (1) women's decision-making, (2) household labor-sharing, (3) women's resource access, and (4) norms/beliefs. Gender indicators that were most proximate to the health outcomes in the DAG were tested using multivariate logistic regression, adjusting for sociodemographic factors. RESULTS: The overall percent agreement of gender-related indicators within couples was 68.6%. The lowest couple concordance was a woman's autonomy to decide to see family/friends without permission from her husband/partner (40.1%). A number of relationships between gender-related indicators and health outcomes emerged: questions from the decision-making domain were found to play a large role in women's health outcomes, and condoms and contraceptive outcomes had the most robust relationship with gender indicators. Women who reported being able to make their own health decisions were 1.57 times (95% CI: 1.12, 2.20) more likely to use condoms. Women who reported that they decide how many children they had also reported high contraception use (OR: 1.79, 95% CI: 1.34, 2.39). Seeking care at the health facility was also associated with women's autonomy for making major household purchases (OR: 1.35, 95% CI: 1.13, 1.62). CONCLUSIONS: The association between decision-making and other gender domains with women's health outcomes highlights the need for heightened attention to gender dimensions of intervention coverage in maternal health. Future studies should integrate and analyze gender-sensitive questions within coverage surveys.

The Potential Contribution of the Health System to Reducing Stunting in SUN Countries.

BACKGROUND: The global nutrition community has called for a multisectoral approach to improve nutritional outcomes. While most essential nutrition interventions are delivered through the health system, nutrition-sensitive interventions from other sectors are critical. OBJECTIVE: We modeled the potential impact that Scaling Up Nutrition (SUN) interventions delivered by the health system would have on reaching World Health Assembly (WHA) stunting targets. We also included results for targets 2, 3, and 5. METHODS: Using all available countries enrolled in the SUN movement, we identified nutrition interventions that are delivered by the health system available in the Lives Saved Tool. We then scaled these interventions linearly from 2012 up to nearly universal coverage (90%) in 2025 and estimated the potential impact that this increase would have with regard to the WHA targets. RESULTS: Our results show that only 16 countries out of 56 would reach the 40% reduction in the number of stunted children by 2025, with a combined total reduction of 32% across all countries. Similarly, only 2 countries would achieve the 50% reduction in anemia for women of reproductive age, 41 countries would reach at least 50% exclusive breastfeeding in children under 6 months of age, and 0 countries would reach the 30% reduction in low birth weight. CONCLUSIONS: While the health system has an important role to play in the delivery of health interventions, focusing investments and efforts on the health system alone will not allow countries to reach the WHA targets by 2025. Concerted efforts across multiple sectors are necessary.

Leveraging water, sanitation and hygiene for nutrition in low- and middle-income countries: A conceptual framework.

In low- and middle-income countries (LMICs), access to water, sanitation and hygiene (WASH) is associated with nutritional status including stunting, which affects 144 million children under 5 globally. Despite the consistent epidemiological association between WASH indicators and nutritional status, the provision of WASH interventions alone has not been found to improve child growth in recent randomized control trials. We conducted a literature review to develop a new conceptual framework that highlights what is known about the WASH to nutrition pathways, the limitations of certain interventions and how future WASH could be leveraged to benefit nutritional status in populations. This new conceptual framework will provide policy makers, program implementors and researchers with a visual tool to bring into perspective multiple levels of WASH and how it may effectively influence nutrition while identifying existing gaps in implementation and research.

Integrating nutrition into the education sector in low- and middle-income countries: A framework for a win-win collaboration.

Malnutrition-both undernutrition and overnutrition-is a public health concern worldwide and particularly in low- and middle-income countries (LMICs). The education sector has high potential to improve immediate nutrition outcomes by providing food in schools and to have more long-term impact through education. We developed a conceptual framework to show how the education sector can be leveraged for nutrition. We reviewed the literature to identify existing frameworks outlining how nutrition programs can be delivered by and through the education sector and used these to build a comprehensive framework. We first organized nutrition programs in the education sector into (1) school food, meals, and food environment; (2) nutrition and health education; (3) physical activity and education; (4) school health services; and (5) water, sanitation, and hygiene (WASH) sector. We then discuss how each one can be successfully implemented. We found high potential in improving nutrition standards and quality of school foods, meals and food environment, especially through collaboration with the agriculture sector. There is a need for well-integrated, culturally appropriate nutrition and health education into the existing school curriculum. This must be supported by a skilled workforce-including nutrition and public health professionals and school staff. Parental and community engagement is cornerstone for program sustainability and success. Current monitoring and evaluation of nutrition programming in schools is weak, and effectiveness, including cost-effectiveness, of interventions is not yet adequately quantified. Finally, we note that opportunities for leveraging the education sector in the fight against rising overweight and obesity rates are under-researched and likely underutilized in LMICs.

Making the health system work for the delivery of nutrition interventions.

Addressing malnutrition requires strategies that are comprehensive and multi-sectoral. Within a multi-sectoral approach, the health system is essential to deliver 10 nutrition-specific interventions, which, if scaled up, could substantially reduce under-5 deaths in high-burden countries through improving maternal and child undernutrition. This study identifies the health system components required for the effective delivery of these interventions, highlighting opportunities and challenges for nutrition programmes and policies. We reviewed implementation guidance for each nutrition-specific intervention, mapping the delivery process for each intervention and determining the health system components required for their delivery. We integrated the components into a single health systems framework for nutrition, illustrating the pathways by which health system components influence household-level determinants of nutrition and individual-level health outcomes. Nutrition-specific interventions are typically delivered in one of four ways: (i) when nutrition interventions are intentionally sought out, (ii) when care is sought for other, unrelated interventions, (iii) at a health facility after active community case finding and referral, and (iv) in the community after active community case finding. A health system enables these processes by providing health services and facilitating care seeking for services, which together require a skilled and motivated health workforce, an effective supply chain, demand for services and access to services. The nutrition community should consider the processes by which nutrition-specific interventions are delivered and the health system components required for their success. Programmes should encourage the delivery of nutrition interventions at every client-provider interaction and should actively generate demand for services-in general, and for nutrition services specifically.

When it rains, it pours: detecting seasonal patterns in utilization of maternal healthcare in Mozambique using routine data.

BACKGROUND: Climatic conditions and seasonal trends can affect population health, but typically, we consider the effect of climate on the epidemiology of communicable diseases. However, climate can also have an effect on access to care, particularly in remote rural areas of low- and middle-income countries. In this study, we investigate associations between the rainy season and the utilization of maternal health services in Mozambique. METHODS: We examined patterns in the number of women receiving antenatal care (ANC) and delivering at a health facility for 2012-2019, using data from Mozambique's Health Management Information Systems. We investigated the association between seasonality (rainfall) and maternal health service utilization (ANC and institutional delivery) at national and provincial level. We fit a negative binomial regression model for institutional delivery and used it to estimate the yearly reduction in institutional deliveries due to the rainy season, with other factors held constant. We used the Lives Saved Tool (LiST) to model increases in mortality due to this estimated decrease in institutional delivery associated with the rainy season. RESULTS: In our national analysis, the rate of ANC visits was 1% lower during the rainy season, adjusting for year and province (IRR = 0.99, 95% CI: 0.96-1.03). The rate of institutional deliveries was 6% lower during the rainy season than the dry season, after adjusting for time and province (IRR = 0.94, 95% CI: 0.92-0.96). In provincial analyses, all provinces except for Maputo-Cidade, Maputo-Province, Nampula, and Niassa showed a statistically significantly lower rate of institutional deliveries in the rainy season. None were statistically significantly lower for ANC. We estimate that, due to reductions in institutional delivery attributable only to the rainy season, there were 74 additional maternal deaths and 726 additional deaths of children under the age of 1 month in 2021, that would not have died if the mothers had instead delivered at a facility. CONCLUSION: Fewer women deliver at a health facility during the rainy season in Mozambique than during the dry season. Barriers to receiving care during pregnancy and childbirth must be addressed using a multisectoral approach, considering the impact of geographical inequities.

Challenges and facilitators to evidence-based decision-making for maternal and child health in Mozambique: district, municipal and national case studies.

BACKGROUND: The need for evidence-based decision-making in the health sector is well understood in the global health community. Yet, gaps persist between the availability of evidence and the use of that evidence. Most research on evidence-based decision-making has been carried out in higher-income countries, and most studies look at policy-making rather than decision-making more broadly. We conducted this study to address these gaps and to identify challenges and facilitators to evidence-based decision-making in Maternal, Newborn and Child Health and Nutrition (MNCH&N) at the municipality, district, and national levels in Mozambique. METHODS: We used a case study design to capture the experiences of decision-makers and analysts (n = 24) who participated in evidence-based decision-making processes related to health policies and interventions to improve MNCH&N in diverse decision-making contexts (district, municipality, and national levels) in 2014-2017, in Mozambique. We examined six case studies, at the national level, in Maputo City and in two districts of Sofala Province and two of Zambézia Province, using individual in-depth interviews with key informants and a document review, for three weeks, in July 2018. RESULTS: Our analysis highlighted various challenges for evidence-based decision-making for MNCH&N, at national, district, and municipality levels in Mozambique, including limited demand for evidence, limited capacity to use evidence, and lack of trust in the available evidence. By contrast, access to evidence, and availability of evidence were viewed positively and seen as potential facilitators. Organizational capacity for the demand and use of evidence appears to be the greatest challenge; while individual capacity is also a barrier. CONCLUSION: Evidence-based decision-making requires that actors have access to evidence and are empowered to act on that evidence. This, in turn, requires alignment between those who collect data, those who analyze and interpret data, and those who make and implement decisions. Investments in individual, organizational, and systems capacity to use evidence are needed to foster practices of evidence-based decision-making for improved maternal and child health in Mozambique.

Early estimates of the indirect effects of the COVID-19 pandemic on maternal and child mortality in low-income and middle-income countries: a modelling study.

BACKGROUND: While the COVID-19 pandemic will increase mortality due to the virus, it is also likely to increase mortality indirectly. In this study, we estimate the additional maternal and under-5 child deaths resulting from the potential disruption of health systems and decreased access to food. METHODS: We modelled three scenarios in which the coverage of essential maternal and child health interventions is reduced by 9·8-51·9% and the prevalence of wasting is increased by 10-50%. Although our scenarios are hypothetical, we sought to reflect real-world possibilities, given emerging reports of the supply-side and demand-side effects of the pandemic. We used the Lives Saved Tool to estimate the additional maternal and under-5 child deaths under each scenario, in 118 low-income and middle-income countries. We estimated additional deaths for a single month and extrapolated for 3 months, 6 months, and 12 months. FINDINGS: Our least severe scenario (coverage reductions of 9·8-18·5% and wasting increase of 10%) over 6 months would result in 253 500 additional child deaths and 12 200 additional maternal deaths. Our most severe scenario (coverage reductions of 39·3-51·9% and wasting increase of 50%) over 6 months would result in 1 157 000 additional child deaths and 56 700 additional maternal deaths. These additional deaths would represent an increase of 9·8-44·7% in under-5 child deaths per month, and an 8·3-38·6% increase in maternal deaths per month, across the 118 countries. Across our three scenarios, the reduced coverage of four childbirth interventions (parenteral administration of uterotonics, antibiotics, and anticonvulsants, and clean birth environments) would account for approximately 60% of additional maternal deaths. The increase in wasting prevalence would account for 18-23% of additional child deaths and reduced coverage of antibiotics for pneumonia and neonatal sepsis and of oral rehydration solution for diarrhoea would together account for around 41% of additional child deaths. INTERPRETATION: Our estimates are based on tentative assumptions and represent a wide range of outcomes. Nonetheless, they show that, if routine health care is disrupted and access to food is decreased (as a result of unavoidable shocks, health system collapse, or intentional choices made in responding to the pandemic), the increase in child and maternal deaths will be devastating. We hope these numbers add context as policy makers establish guidelines and allocate resources in the days and months to come. FUNDING: Bill & Melinda Gates Foundation, Global Affairs Canada.

Obstacles to advancing women's health in Mozambique: a qualitative investigation into the perspectives of policy makers.

BACKGROUND: Despite substantial investment in women's health over the past two decades, and enthusiastic government support for MDG 5 and SDG 3, health indicators for women in Mozambique remain among the lowest in the world. Maternal mortality stayed constant from 2003 to 2011, with an MMR of 408; the estimated HIV prevalence for women of 15-24 years is over twice that for men; and only 12.1% of women are estimated to be using modern contraception. This study explores the perspectives of policy makers in the Mozambican health system and affiliates on the challenges that are preventing Mozambique from achieving greater gains in women's health. METHODS: We conducted in-depth interviews with 39 senior- and mid-level policy makers in the Ministry of Health and affiliated institutions (32 women, 7 men). Participants were sampled using a combination of systematic random sampling and snowball sampling. Participants were asked about their experiences formulating and implementing health policies and programs, what is needed to improve women's health in Mozambique, and the barriers and opportunities to achieving such improvement. RESULTS: Participants unanimously argued that women's health is already sufficiently prioritized in national health policies and strategies in Mozambique; the problem, rather, is the implementation and execution of existing women's health policies and programs. Participants raised challenges related to the policy making process itself, including an ever-changing, fragmented decision-making process, lack of long-term perspective, weak evaluation, and misalignment of programs across sectors. The disproportionate influence of donors was also mentioned, with lack of ownership, rapid transitions, and vertical programming limiting the scope for meaningful change. Finally, participants reported a disconnect between policy makers at the national level and realities on the ground, with poor dissemination of strategies, limited district resources, and poor consideration of local cultural contexts. CONCLUSIONS: To achieve meaningful gains in women's health in Mozambique, more focus must be placed on resolving the bottleneck that is the implementation of existing policies. Barriers to implementation exist across multiple health systems components, therefore, solutions to address them must also reach across these multiple components. A holistic approach to strengthening the health system across multiple sectors and at multiple levels is needed. SUPPLEMENTARY INFORMATION: Supplementary information accompanies this paper at (10.1186/s41256-019-0119-x).

Reaching substantive female representation among decision-makers: A qualitative research study of gender-related experiences from the health sector in Mozambique.

BACKGROUND: Achieving significant female representation in government at decision-making levels has been identified as a key step towards achieving gender equality. In 2015, women held 39.6% of parliamentary seats in Mozambique, which is above the benchmark of 30% that has been suggested as the turning point for minority representation to move from token status to having a sizable impact. We undertook a study to identify gender-related barriers and facilitators to improving women-centered policies in the health sector. METHODS: We conducted in-depth interviews with 39 individuals (32 women, 7 men) involved at a senior level in policy making or implementation of woman-centric policies within the Mozambique Ministry of Health and affiliated institutions. We used a semi-structured interview guide that included questions on difficulties and facilitating factors encountered in the policy making process, and the perceived role of gender in this process. We used both deductive and inductive analysis approaches, starting with a set of pre-identified themes and expanding this to include themes that emerged during coding. RESULTS: Our data suggest two main findings: (1) the women who participated in our study generally do not report feeling discrimination in the workplace and (2) senior health sector perceive women to be more personally attuned to women-centric issues than men. Within our specific sample, we found little to suggest that gender discrimination is a problem professionally for female decision-makers in Mozambique. However, these findings should be contextualized using an intersectional lens with recognition of the important difference between descriptive versus substantive female representation, and whether "percentage of women" is truly the best metric for gaging commitment to gender equality at the policy making level. CONCLUSIONS: Mozambique's longstanding significant representation of women may have led to creating an environment that leads to positive experiences for female decision-makers in the government. However, while the current level of female representation should be celebrated, it does not negate the need for continued focus on female representation in decision-making positions.