The Value Added of Incorporating Qualitative Approaches into Malaria Surveillance, Monitoring, and Evaluation.

Progress toward malaria elimination and improvements in the performance of national malaria control programs (NMCPs) have stalled in recent years. The current COVID-19 pandemic further threatens building on previous gains. Surveillance, monitoring, and evaluation (SME) are critical for the continued success of NMCPs because they provide the information necessary for effective program planning and management. Interventions aimed at strengthening NMCPs focus on both the target population and the program provider. Qualitative approaches are often used to understand the target population and barriers to intervention success. Although there is growing emphasis on qualitative approaches in provider-focused SME, metrics of success tend to focus on quantitative measures. The integration of qualitative approaches offers added value because they provide additional data points to facilitate the understanding of barriers that impede sustaining the gains made from provider-focused capacity-building efforts. Qualitative approaches focus on understanding program implementation and interventions, but the systematic integration of qualitative data is limited. Qualitative approaches provide avenues to strengthen SME efforts, can lead to subsequent improvement for NMCPs, and fuel progress toward malaria elimination.

Evaluating malaria programmes in moderate- and low-transmission settings: practical ways to generate robust evidence.

BACKGROUND: Many countries have made substantial progress in scaling-up and sustaining malaria intervention coverage, leading to more focalized and heterogeneous transmission in many settings. Evaluation provides valuable information for programmes to understand if interventions have been implemented as planned and with quality, if the programme had the intended impact on malaria burden, and to guide programmatic decision-making. Low-, moderate-, and heterogeneous-transmission settings present unique evaluation challenges because of dynamic and targeted intervention strategies. This paper provides illustration of evaluation approaches and methodologies for these transmission settings, and suggests how to answer evaluation questions specific to the local context. METHODS: The Roll Back Malaria Monitoring and Evaluation Reference Group formed a task force in October 2017 to lead development of this framework. The task force includes representatives from National Malaria Programmes, funding agencies, and malaria research and implementing partners. The framework builds on existing guidance for process and outcome evaluations and impact evaluations specifically in high transmission settings. RESULTS: The theory of change describes how evaluation questions asked by national malaria programmes in different contexts influence evaluation design. The transmission setting, existing stratification, and data quality and availability are also key considerations. The framework is intended for adaption by countries to their local context, and use for evaluation at sub-national level. Confirmed malaria incidence is recommended as the primary impact indicator due to its sensitivity to detect changes in low-transmission settings. It is expected that process evaluations provide sufficient evidence for programme monitoring and improvement, while impact evaluations are needed following adoption of new mixes of interventions, operational strategies, tools or policies, particularly in contexts of changing malaria epidemiology. Impact evaluations in low-, moderate-, or heterogeneous-transmission settings will likely use plausibility designs, and methods highlighted by the framework include interrupted time series, district-level dose-response analyses, and constructed control methods. Triangulating multiple data sources and analyses is important to strengthen the plausibility argument. CONCLUSIONS: This framework provides a structure to assist national malaria programmes and partners to design evaluations in low-, moderate- or heterogeneous-transmission settings. Emphasizing a continuous cycle along the causal pathway linking process evaluation to impact evaluation and then programmatic decision-making, the framework provides practical guidance in evaluation design, analysis, and interpretation to ensure that the evaluation meets national malaria programme priority questions and guides decision-making at national and sub-national levels.

Evidence of Impact: iCCM as a strategy to save lives of children under five.

BACKGROUND: In 2013, the World Health Organization (WHO) launched the Rapid Access Expansion (RAcE) programme in the Democratic Republic of Congo, Malawi, Mozambique, Niger, and Nigeria to increase coverage of diagnostic, treatment, and referral services for malaria, pneumonia, and diarrhea among children ages 2-59 months. In 2017, a final evaluation of the six RAcE sites was conducted to determine whether the programme goal was reached. A key evaluation objective was to estimate the reduction in childhood mortality and the number of under-five lives saved over the project period in the RAcE project areas. METHODS: The Lives Saved Tool (LiST) was used to estimate reductions in all-cause child mortality due to changes in coverage of treatment for the integrated community case management (iCCM) illnesses - malaria, pneumonia, and diarrhea - while accounting for other changes in maternal and child health interventions in each RAcE project area. Data from RAcE baseline and endline household surveys, Demographic and Health Surveys, and routine health service data were used in each LiST model. The models yielded estimated change in under-five mortality rates, and estimated number of lives saved per year by malaria, pneumonia and diarrhea treatment. We adjusted the results to estimate the number of lives saved by community health worker (CHW)-provided treatment. RESULTS: The LiST model accounts for coverage changes in iCCM intervention coverage and other health trends in each project area to estimate mortality reduction and child lives saved. Under five mortality declined in all six RAcE sites, with an average decline of 10 percent. An estimated 6200 under-five lives were saved by malaria, pneumonia, and diarrhea treatment in the DRC, Malawi, Niger, and Nigeria, of which approximately 4940 (75 percent) were saved by treatment provided by CHWs. This total excludes Mozambique, where there were no estimated under-five lives saved likely due to widespread stockouts of key medications. In all other project areas, lives saved by CHW-provided treatment contributed substantially to the estimated decline in under-five mortality. CONCLUSIONS: Our results suggest that iCCM is a strategy that can save lives and measurably decrease child mortality in settings where access to health facility services is low and adequate resources for iCCM implementation are provided for CHW services.

Effect of community-based intervention on improving access to treatment for sick under-five children in hard-to-reach communities in Niger State, Nigeria.

BACKGROUND: Access to prompt and appropriate treatment is key to survival for children with malaria, pneumonia and diarrhoea. Community-based services are vital to extending care to remote populations. Malaria Consortium supported Niger state Ministry of Health, Nigeria, to introduce and implement an integrated community case management (iCCM) programme for four years in six local government areas (LGAs). The objective was to increase coverage of effective treatment for malaria, pneumonia and diarrhoea among children aged 2-59 months. METHODS: The programme involved training, equipping, ongoing support and supervision of 1320 community volunteers (CORPs) to provide iCCM services to their communities in all six LGAs. Demand creation activities were also conducted; these included community dialogues, household mobilization, sensitization and mass media campaigns targeted at programme communities. To assess the level of changes in care seeking and treatment, baseline and endline household surveys were conducted in 2014 and 2017 respectively. For both surveys, a 30×30 multi-stage cluster sampling method was used, the sampling frame being RAcE programme communities. RESULTS: Care-seeking from an appropriate provider increased overall and for each iCCM illness from 78% to 94% for children presenting with fever (P < 0.01), from 72% to 91% for diarrhoea cases (P < 0.01), and from 76% to 89% for cases of cough with difficult or fast breathing (P < 0.05). For diagnosis and treatment, the coverage of fevers tested for malaria increased from 34% to 77% (P < 0.001) and ACT treatments from 57% to 73% (<0.005); 56% of cases of cough or fast breathing who sought care from a CORP, had their respiratory rate counted and 61% with cough or fast breathing received amoxicillin. At endline caregivers sought care from CORPs in their communities for most cases of childhood illnesses (84%) compared to other providers at hospitals (1%) or health centres (9%).This aligns with caregivers' belief that CORPs are trusted providers (94%) who provide quality services (96%). CONCLUSION: Implementation of iCCM with focused demand creation activities can improve access to quality lifesaving interventions from frontline community providers in Nigeria. This can contribute towards achieving SDGs if iCCM is scaled up to hard-to-reach areas of all states in the country.

iCCM data quality: an approach to assessing iCCM reporting systems and data quality in 5 African countries.

BACKGROUND: Ensuring the quality of health service data is critical for data-driven decision-making. Data quality assessments (DQAs) are used to determine if data are of sufficient quality to support their intended use. However, guidance on how to conduct DQAs specifically for community-based interventions, such as integrated community case management (iCCM) programs, is limited. As part of the World Health Organization's (WHO) Rapid Access Expansion (RAcE) Programme, ICF conducted DQAs in a unique effort to characterize the quality of community health worker-generated data and to use DQA findings to strengthen reporting systems and decision-making. METHODS: We present our experience implementing assessments using standardized DQA tools in the six RAcE project sites in the Democratic Republic of Congo, Malawi, Mozambique, Niger, and Nigeria. We describe the process used to create the RAcE DQA tools, adapt the tools to country contexts, and develop the iCCM DQA Toolkit, which enables countries to carry out regular and rapid DQAs. We provide examples of how we used results to generate recommendations. RESULTS: The DQA tools were customized for each RAcE project to assess the iCCM data reporting system, trace iCCM indicators through this system, and to ensure that DQAs were efficient and generated useful recommendations. This experience led to creation of an iCCM DQA Toolkit comprised of simplified versions of RAcE DQA tools and a guidance document. It includes system assessment questions that elicit actionable responses and a simplified data tracing tool focused on one treatment indicator for each iCCM focus illness: diarrhea, malaria, and pneumonia. The toolkit is intended for use at the national or sub-national level for periodic data quality checks. CONCLUSIONS: The iCCM DQA Toolkit was designed to be easily tailored to different data reporting system structures because iCCM data reporting tools and data flow vary substantially. The toolkit enables countries to identify points in the reporting system where data quality is compromised and areas of the reporting system that require strengthening, so that countries can make informed adjustments that improve data quality, strengthen reporting systems, and inform decision-making.

Data quality assessments stimulate improvements to health management information systems: evidence from five African countries.

BACKGROUND: Health service data are used to inform decisions about planning and implementation, as well as to evaluate performance and outcomes, and the quality of those data are important. Data quality assessments (DQA) afford the opportunity to collect information about health service data. Through its Rapid Access Expansion Programme (RAcE), the World Health Organization (WHO) funded non-governmental organizations (NGO) to support Ministries of Health (MOH) in implementing integrated community case management (iCCM) programs in the Democratic Republic of Congo, Malawi, Mozambique, Niger and Nigeria. WHO contracted ICF to support grantee monitoring and evaluation efforts, part of which was to conduct DQAs to enhance program monitoring and decision making. The contribution of DQAs to data-driven decision making has been documented and the purpose of this paper is to describe how DQAs contributed to health management information system (HMIS) strengthening and the findings of subsequent DQAs in those areas. METHODS: ICF created a mixed-methods DQA for iCCM data, comprising a review of the data collection and management system, a data tracing component and key informant interviews. The DQA was applied twice in each RAcE site, which enables a general comparison of system-level attributes before and after the first DQA application. For this qualitative assessment, we reviewed DQA reports to collate information about DQA recommendations and how they were addressed before a subsequent DQA, along with the findings of the second DQA. RESULTS: Findings from the first DQA in each RAcE site stimulated NGO and MOH efforts to strengthen different aspects of the HMIS in each country, including modifying data collection tools in the Democratic Republic of Congo; training community health workers (CHWs) and supervisors in Malawi; strengthening supervision in Mozambique; improving CHW registers and strengthening staff capacity at all levels to report data in Niger; establishing a data review system in Abia State, Nigeria; and, establishing processes to improve data use and quality in Niger State, Nigeria. CONCLUSION: Data quality assessments stimulated context-specific efforts by NGOs and MOHs to improve iCCM data quality. DQAs can serve as a collaborative and evidence-based activity to influence discussions of data quality and stimulate HMIS strengthening efforts.

Achievements and challenges of implementation in a mature iCCM programme: Malawi case study.

BACKGROUND: Malawi has a mature integrated community case management (iCCM) programme that is led by the Ministry of Health (MOH) but that still relies on donor support. From 2013 until 2017, under the Rapid Access Expansion (RAcE) programme, the World Health Organization supported the MOH to expand and strengthen iCCM services in four districts. This paper examines Malawi's iCCM programme performance and implementation strength in RAcE districts to further strengthen the broader programme. METHODS: Baseline and endline household surveys were conducted in iCCM-eligible areas of RAcE districts. Primary caregivers of recently-sick children under five were interviewed to assess changes in care-seeking and treatment over the project period. Health surveillance assistants (HSAs) were surveyed at endline to assess iCCM implementation strength. RESULTS: Care-seeking from HSAs and treatment of fever improved over the project period. At endline, however, less than half of sick children were brought to an HSA, many caregivers reported a preference for providers other than HSAs, and perceptions of HSAs as trusted providers of high-quality, convenient care had decreased. HSA supervision and mentorship were below MOH targets. Stockouts of malaria medicines were associated with decreased care-seeking from HSAs. Thirty percent of clusters had limited or no access to iCCM (no HSA or an HSA providing iCCM services less than 2 days per week); 50% had moderate access (an HSA providing iCCM services 2 to 4 days per week; and 20% had high access (a resident HSA providing iCCM services 5 or more days per week). Moderate access to iCCM was associated with increased care-seeking from HSAs, increased treatment by HSAs, and more positive perceptions of HSAs compared to areas with limited or no access. Areas with high access to iCCM did not show further improvements above areas with moderate access. CONCLUSIONS: Availability of well-equipped and supported HSAs is critical to the provision of iCCM services. Additional qualitative research is needed to examine challenges and to inform potential solutions. Malawi's mature iCCM programme has a strong foundation but can be improved to strengthen the continuity of care from communities to facilities and to ultimately improve child health outcomes.

Engaging men to promote and support exclusive breastfeeding: a descriptive review of 28 projects in 20 low- and middle-income countries from 2003 to 2013.

BACKGROUND: Lay support has been associated with improved breastfeeding practices, but studies of programs that engage men in breastfeeding support have shown mixed results and most are from high-income countries. The purpose of our research is to review strategies to engage men in exclusive breastfeeding (EBF) promotion or support in 28 project areas across 20 low- and middle-income countries. This information may be used to inform program implementers and policymakers seeking to increase EBF. METHODS: We tested the difference between baseline and final EBF proportions using Pearson's chi-square (a = 0.05) and identified project areas with a significant increase. We categorized male engagement strategies as low- and high-intensity, using information from project reports. We looked for patterns by intensity and geography and described strategies used to engage men in different places. RESULTS: Twenty-eight projects were reviewed; 21 (75%) were in areas where a statistically significant increase in EBF was observed between the beginning and end of the project. A variety of high- and low-intensity male engagement strategies was used in areas with an increase in EBF prevalence and in all geographic regions. High-intensity strategies engaged men directly during home or health visits by forming men's groups and by working with male community leaders or members to promote EBF. Low-intensity strategies included large community meetings that included men, and radio messages, and other behavior change materials directed towards men. CONCLUSION: Male engagement strategies took many forms in these project areas. We did not find consistent associations between the intensities or types of male engagement strategies and increases in EBF proportions. There is a gap in understanding how gender norms might impact male involvement in women's health behaviors. This review does not support the broad application of male engagement to improve EBF practices, and we recommend considering local gender norms when designing programs to support women to EBF.

How do we know? An assessment of integrated community case management data quality in four districts of Malawi.

The World Health Organization contracted annual data quality assessments of Rapid Access Expansion (RAcE) projects to review integrated community case management (iCCM) data quality and the monitoring and evaluation (M&E) system for iCCM, and to suggest ways to improve data quality. The first RAcE data quality assessment was conducted in Malawi in January 2014 and we present findings pertaining to data from the health management information system at the community, facility and other sub-national levels because RAcE grantees rely on that for most of their monitoring data. We randomly selected 10 health facilities (10% of eligible facilities) from the four RAcE project districts, and collected quantitative data with an adapted and comprehensive tool that included an assessment of Malawi's M&E system for iCCM data and a data verification exercise that traced selected indicators through the reporting system. We rated the iCCM M&E system across five function areas based on interviews and observations, and calculated verification ratios for each data reporting level. We also conducted key informant interviews with Health Surveillance Assistants and facility, district and central Ministry of Health staff. Scores show a high-functioning M&E system for iCCM with some deficiencies in data management processes. The system lacks quality controls, including data entry verification, a protocol for addressing errors, and written procedures for data collection, entry, analysis and management. Data availability was generally high except for supervision data. The data verification process identified gaps in completeness and consistency, particularly in Health Surveillance Assistants' record keeping. Staff at all levels would like more training in data management. This data quality assessment illuminates where an otherwise strong M&E system for iCCM fails to ensure some aspects of data quality. Prioritizing data management with documented protocols, additional training and approaches to create efficient supervision practices may improve iCCM data quality.

Neglected value of small population-based surveys: a comparison with demographic and health survey data.

We believe that global health practice and evaluation operate with misleading assumptions about lack of reliability of small population-based health surveys (district level and below), leading managers and decision-makers to under-use this valuable information and programmatic tool and to rely on health information from large national surveys when neither timing nor available data meet their needs. This paper uses a unique opportunity for comparison between a knowledge, practice, and coverage (KPC) household survey and Rwanda Demographic and Health Survey (RDHS) carried out in overlapping timeframes to disprove these enduring suspicions. Our analysis shows that the KPC provides coverage estimates consistent with the RDHS estimates for the same geographic areas. We discuss cases of divergence between estimates. Application of the Lives Saved Tool to the KPC results also yields child mortality estimates comparable with DHS-measured mortality. We draw three main lessons from the study and conclude with recommendations for challenging unfounded assumptions against the value of small household coverage surveys, which can be a key resource in the arsenal of local health programmers.

Community-based intervention packages facilitated by NGOs demonstrate plausible evidence for child mortality impact.

INTRODUCTION: Evidence exists that community-based intervention packages can have substantial child and newborn mortality impact, and may help more countries meet Millennium Development Goal 4 (MDG 4) targets. A non-governmental organization (NGO) project using such programming in Mozambique documented an annual decline in under-five mortality rate (U5MR) of 9.3% in a province in which Demographic and Health Survey (DHS) data showed a 4.2% U5MR decline during the same period. To test the generalizability of this finding, the same analysis was applied to a group of projects funded by the US Agency for International Development. Projects supported implementation of community-based intervention packages aimed at increasing use of health services while improving preventive and home-care practices for children under five. METHODS: All projects collect baseline and endline population coverage data for key child health interventions. Twelve projects fitted the inclusion criteria. U5MR decline was estimated by modelling these coverage changes in the Lives Saved Tool (LiST) and comparing with concurrent measured DHS mortality data. RESULTS: Average coverage changes for all interventions exceeded average concurrent trends. When population coverage changes were modelled in LiST, they were estimated to give a child mortality improvement in the project area that exceeded concurrent secular trend in the subnational DHS region in 11 of 12 cases. The average improvement in modelled U5MR (5.8%) was more than twice the concurrent directly measured average decline (2.5%). CONCLUSIONS: NGO projects implementing community-based intervention packages appear to be effective in reducing child mortality in diverse settings. There is plausible evidence that they raised coverage for a variety of high-impact interventions and improved U5MR by more than twice the concurrent secular trend. All projects used community-based strategies that achieved frequent interpersonal contact for health behaviour change. Further study of the effectiveness and scalability of similar packages should be part of the effort to accelerate progress towards MDG 4.

Comparing estimates of child mortality reduction modelled in LiST with pregnancy history survey data for a community-based NGO project in Mozambique.

BACKGROUND: There is a growing body of evidence that integrated packages of community-based interventions, a form of programming often implemented by NGOs, can have substantial child mortality impact. More countries may be able to meet Millennium Development Goal (MDG) 4 targets by leveraging such programming. Analysis of the mortality effect of this type of programming is hampered by the cost and complexity of direct mortality measurement. The Lives Saved Tool (LiST) produces an estimate of mortality reduction by modelling the mortality effect of changes in population coverage of individual child health interventions. However, few studies to date have compared the LiST estimates of mortality reduction with those produced by direct measurement. METHODS: Using results of a recent review of evidence for community-based child health programming, a search was conducted for NGO child health projects implementing community-based interventions that had independently verified child mortality reduction estimates, as well as population coverage data for modelling in LiST. One child survival project fit inclusion criteria. Subsequent searches of the USAID Development Experience Clearinghouse and Child Survival Grants databases and interviews of staff from NGOs identified no additional projects. Eight coverage indicators, covering all the project's technical interventions were modelled in LiST, along with indicator values for most other non-project interventions in LiST, mainly from DHS data from 1997 and 2003. RESULTS: The project studied was implemented by World Relief from 1999 to 2003 in Gaza Province, Mozambique. An independent evaluation collecting pregnancy history data estimated that under-five mortality declined 37% and infant mortality 48%. Using project-collected coverage data, LiST produced estimates of 39% and 34% decline, respectively. CONCLUSIONS: LiST gives reasonably accurate estimates of infant and child mortality decline in an area where a package of community-based interventions was implemented. This and other validation exercises support use of LiST as an aid for program planning to tailor packages of community-based interventions to the epidemiological context and for project evaluation. Such targeted planning and assessments will be useful to accelerate progress in reaching MDG4 targets.