Strengthening effectiveness evaluations to improve programs for women, children and adolescents.

A full understanding of the pathways from efficacious interventions to population impact requires rigorous effectiveness evaluations conducted under realistic scale-up conditions at country level. In this paper, we introduce a deductive framework that underpins effectiveness evaluations. This framework forms the theoretical and conceptual basis for the 'Real Accountability: Data Analysis for Results' (RADAR) project, intended to address gaps in guidance and tools for the evaluation of projects being implemented at scale to reduce mortality among women and children. These gaps include needs for a framework to guide decisions about evaluations and practical measurement tools, as well as increased capacity in evaluation practice among donors and program planners at global, national and project levels. RADAR aimed to improve the evidence base for program and policy decisions in reproductive, maternal, newborn and child health and nutrition (RMNCH&N). We focus on five linked methodological steps - presented as core evaluation questions - for designing and implementing effectiveness evaluation of large-scale programs that support both the needs of program managers to improve their programs and the needs of donors to meet their accountability responsibilities. RADAR has operationalized each step with a tool to facilitate its application. We also describe cross-cutting methodological issues and broader contextual factors that affect the planning and implementation of such evaluations. We conclude with proposals for how the global RMNCH&N community can support rigorous program evaluations and make better use of the resulting evidence.

Feasibility of engaging "Village Doctors" in the Community-based Integrated Management of Childhood Illness (C-IMCI): experience from rural Bangladesh.

BACKGROUND: Informal health care providers particularly "village doctors" are the first point of care for under-five childhood illnesses in rural Bangladesh. We engaged village doctors as part of the Multi-Country Evaluation (MCE) of Integrated Management of Childhood Illness (IMCI) and assessed their management of sick under-five children before and after a modified IMCI training, supplemented with ongoing monitoring and supportive supervision. METHODS: In 2003-2004, 144 village doctors across 131 IMCI intervention villages in Matlab Bangladesh participated in a two-day IMCI training; 135 of which completed pre- and post-training evaluation tests. In 2007, 38 IMCI-trained village doctors completed an end-of-project knowledge retention test. Village doctor prescription practices for sick under-five children were examined through household surveys, and routine monitoring visits. In-depth interviews were done with mothers seeking care from village doctors. RESULTS: Village doctors' knowledge on the assessment and management of childhood illnesses improved significantly after training; knowledge of danger signs of pneumonia and severe pneumonia increased from 39% to 78% (P < 0.0001) and from 17% to 47% (P < 0.0001) respectively. Knowledge on the correct management of severe pneumonia increased from 62% to 84% (P < 0.0001), and diarrhoea management improved from 65% to 82% (P = 0.0005). Village doctors retained this knowledge over three years except for home management of pneumonia. No significant differences were observed in prescribing practices for diarrhoea and pneumonia management between trained and untrained village doctors. Village doctors were accessible to communities; 76% had cell phones; almost all attended home calls, and did not charge consultation fees. Nearly all (91%) received incentives from pharmaceutical representatives. CONCLUSIONS: Village doctors have the capacity to learn and retain knowledge on the appropriate management of under-five illnesses. Training alone did not improve inappropriate antibiotic prescription practices. Intensive monitoring and efforts to target key actors including pharmaceutical companies, which influence village doctors dispensing practices, and implementation of mechanisms to track and regulate these providers are necessary for future engagement in management of under-five childhood illnesses.

Using community-based reporting of vital events to monitor child mortality: Lessons from rural Ghana.

BACKGROUND: Reducing neonatal and child mortality is a key component of the health-related sustainable development goal (SDG), but most low and middle income countries lack data to monitor child mortality on an annual basis. We tested a mortality monitoring system based on the continuous recording of pregnancies, births and deaths by trained community-based volunteers (CBV). METHODS AND FINDINGS: This project was implemented in 96 clusters located in three districts of the Northern Region of Ghana. Community-based volunteers (CBVs) were selected from these clusters and were trained in recording all pregnancies, births, and deaths among children under 5 in their catchment areas. Data collection lasted from January 2012 through September 2013. All CBVs transmitted tallies of recorded births and deaths to the Ghana Birth and deaths registry each month, except in one of the study districts (approximately 80% reporting). Some events were reported only several months after they had occurred. We assessed the completeness and accuracy of CBV data by comparing them to retrospective full pregnancy histories (FPH) collected during a census of the same clusters conducted in October-December 2013. We conducted all analyses separately by district, as well as for the combined sample of all districts. During the 21-month implementation period, the CBVs reported a total of 2,819 births and 137 under-five deaths. Among the latter, there were 84 infant deaths (55 neonatal deaths and 29 post-neonatal deaths). Comparison of the CBV data with FPH data suggested that CBVs significantly under-estimated child mortality: the estimated under-5 mortality rate according to CBV data was only 2/3 of the rate estimated from FPH data (95% Confidence Interval for the ratio of the two rates = 51.7 to 81.4). The discrepancies between the CBV and FPH estimates of infant and neonatal mortality were more limited, but varied significantly across districts. CONCLUSIONS: In northern Ghana, a community-based data collection systems relying on volunteers did not yield accurate estimates of child mortality rates. Additional implementation research is needed to improve the timeliness, completeness and accuracy of such systems. Enhancing pregnancy monitoring, in particular, may be an essential step to improve the measurement of neonatal mortality.

Improving coverage measurement for reproductive, maternal, neonatal and child health: gaps and opportunities.

BACKGROUND: Regular monitoring of coverage for reproductive, maternal, neonatal, and child health (RMNCH) is central to assessing progress toward health goals. The objectives of this review were to describe the current state of coverage measurement for RMNCH, assess the extent to which current approaches to coverage measurement cover the spectrum of RMNCH interventions, and prioritize interventions for a novel approach to coverage measurement linking household surveys with provider assessments. METHODS: We included 58 interventions along the RMNCH continuum of care for which there is evidence of effectiveness against cause-specific mortality and stillbirth. We reviewed household surveys and provider assessments used in low- and middle-income countries (LMICs) to determine whether these tools generate measures of intervention coverage, readiness, or quality. For facility-based interventions, we assessed the feasibility of linking provider assessments to household surveys to provide estimates of intervention coverage. RESULTS: Fewer than half (24 of 58) of included RMNCH interventions are measured in standard household surveys. The periconceptional, antenatal, and intrapartum periods were poorly represented. All but one of the interventions not measured in household surveys are facility-based, and 13 of these would be highly feasible to measure by linking provider assessments to household surveys. CONCLUSIONS: We found important gaps in coverage measurement for proven RMNCH interventions, particularly around the time of birth. Based on our findings, we propose three sets of actions to improve coverage measurement for RMNCH, focused on validation of coverage measures and development of new measurement approaches feasible for use at scale in LMICs.

A qualitative exploration of care-seeking pathways for sick children in the rural Oromia region of Ethiopia.

BACKGROUND: Ethiopia has experienced rapid improvements in its healthcare infrastructure, such as through the recent scale up of integrated community case management (iCCM) delivered by community-based health extension workers (HEWs) targeting children under the age of five. Despite notable improvements in child outcomes, the use of HEWs delivering iCCM remains very low. The aim of our study was to explain this phenomenon by examining care-seeking practices and treatment for sick children in two rural districts in the Oromia Region of Ethiopia. METHODS: Using qualitative methods, we explored perceptions of child illness, influences on decision-making processes occurring over the course of a child's illness and caregiver perceptions of available community-based sources of child illness care. Sixteen focus group discussions (FGDs) and 40 in-depth interviews (IDIs) were held with mothers of children under age five. For additional perspective, 16 IDIs were conducted fathers and 22 IDIs with health extension workers and community health volunteers. RESULTS: Caregivers often described the act of care-seeking for a sick child as a time of considerable uncertainty. In particular, mothers of sick children described the cultural, social and community-based resources available to minimize this uncertainty as well as constraints and strategies for accessing these resources in order to receive treatment for a sick child. The level of trust and familiarity were the most common dynamics noted as influencing care-seeking strategies; trust in biomedical and government providers was often low. CONCLUSIONS: Overall, our research highlights the multiple and dynamic influences on care-seeking for sick children in rural Ethiopia. An understanding of these influences is critical for the success of existing and future health interventions and continued improvement of child health in Ethiopia.

Assessment of the impact of quality improvement interventions on the quality of sick child care provided by Health Extension Workers in Ethiopia.

BACKGROUND: Ethiopia has scaled up integrated community case management of childhood illness (iCCM), including several interventions to improve the performance of Health Extension Workers (HEWs). We assessed associations between interventions to improve iCCM quality of care and the observed quality of care among HEWs. METHODS: We assessed iCCM implementation strength and quality of care provided by HEWs in Ethiopia. Multivariate logistic regression analyses were performed to assess associations between interventions to improve iCCM quality of care and correct management of iCCM illnesses. FINDINGS: Children who were managed by an HEW who had attended a performance review and clinical mentoring meeting (PRCMM) had 8.3 (95% confidence interval (CI) 2.34-29.51) times the odds of being correctly managed, compared to children managed by an HEW who did not attend a PRCMM. Management by an HEW who received follow-up training also significantly increased the odds of correct management (odds ratio (OR) = 2.09, 95% CI 1.05-4.18). Supervision on iCCM (OR = 0.63, 95% CI 0.23-1.72) did not significantly affect the odds of receiving correct care. CONCLUSIONS: These results suggest PRCMM and follow-up training were effective interventions, while implementation of supportive supervision needs to be reviewed to improve impact.

Diarrhea management in children under five in sub-Saharan Africa: does the source of care matter? A Countdown analysis.

BACKGROUND: Diarrhea remains a high burden disease, responsible for nine percent of deaths in children under five globally. We analyzed diarrhea management practices in young children and their association with the source of care. METHODS: We used Demographic and Health Survey data from 12 countries in sub-Saharan Africa with high burdens of childhood diarrhea. We classified the quality of diarrhea management practices as good, fair, or poor based on mothers' reports for children with diarrhea, using WHO/UNICEF recommendations for appropriate treatment. We described the prevalence of diarrhea management by type and assessed the association between good management and source of care, adjusting for potential confounders. RESULTS: Prevalence of good diarrhea management is low in 11 of the 12 analyzed surveys, varying from 17 % in Cote d'Ivoire to 38 % in Niger. The exception is Sierra Leone, where prevalence of good practice is 67 %. Prevalence of good management was low even among children taken to health facilities [median 52 %, range: 34-64 %]. Diarrhea careseeking from health facilities or community providers was associated with higher odds of good management than care from traditional/informal sources or no care. Careseeking from facilities did not result systematically in a higher likelihood of good diarrhea management than care from community providers. The odds of good diarrhea management were similar for community versus facility providers in six countries, higher in community than facility providers in two countries, and higher in facility than in community providers in four countries. CONCLUSION: Many children's lives can be saved with correct management of childhood diarrhea. Too many children are not receiving adequate care for diarrhea in high-burden sub-Saharan African countries, even among those seen in health facilities. Redoubling efforts to increase careseeking and improve quality of care for childhood diarrhea in both health facilities and at community level is an urgent priority.

Improved measurement for mothers, newborns and children in the era of the Sustainable Development Goals.

BACKGROUND: An urgent priority in maternal, newborn and child health is to accelerate the scale-up of cost-effective essential interventions, especially during labor, the immediate postnatal period and for the treatment of serious infectious diseases and acute malnutrition. Tracking intervention coverage is a key activity to support scale-up and in this paper we examine priorities in coverage measurement, distinguishing between essential interventions that can be measured now and those that require methodological development. METHODS: We conceptualized a typology of indicators related to intervention coverage that distinguishes access to care from receipt of an intervention by the population in need. We then built on documented evidence on coverage measurement to determine the status of indicators for essential interventions and to identify areas for development. RESULTS: Contact indicators from pregnancy to childhood were identified as current indicators for immediate use, but indicators reflecting the quality of care provided during these contacts need development. At each contact point, some essential interventions can be measured now, but the need for development of indicators predominates around interventions at the time of birth and interventions to treat infections. Addressing this need requires improvements in routine facility based data capture, methods for linking provider and community-based data, and improved guidance for effective coverage measurement that reflects the provision of high-quality care. CONCLUSION: Coverage indicators for some essential interventions can be measured accurately through household surveys and be used to track progress in maternal, newborn and child health. Other essential interventions currently rely on contact indicators as proxies for coverage but urgent attention is needed to identify new measurement approaches that directly and reliably measure their effective coverage.

Patterns and determinants of antenatal care utilization: analysis of national survey data in seven countdown countries.

BACKGROUND: Antenatal care (ANC) is critical for improving maternal and newborn health. WHO recommends that pregnant women complete at least four ANC visits. Countdown and other global monitoring efforts track the proportions of women who receive one or more visits by a skilled provider (ANC1+) and four or more visits by any provider (ANC4+). This study investigates patterns of drop-off in use between ANC1+ and ANC4+, and explores inequalities in women's use of ANC services. It also identifies determinants of utilization and describes countries' ANC-related policies, and programs. METHODS: We performed secondary analyses using Demographic Health Survey (DHS) data from seven Countdown countries: Bangladesh, Cambodia, Cameroon, Nepal, Peru, Senegal and Uganda. The descriptive analysis illustrates country variations in the frequency of visits by provider type, content, and by household wealth, women's education and type of residence. We conducted a multivariable analysis using a conceptual framework to identify determinants of ANC utilization. We collected contextual information from countries through a standard questionnaire completed by country-based informants. RESULTS: Each country had a unique pattern of ANC utilization in terms of coverage, inequality and the extent to which predictors affected the frequency of visits. Nevertheless, common patterns arise. Women having four or more visits usually saw a skilled provider at least once, and received more evidence-based content interventions than women reporting fewer than four visits. A considerable proportion of women reporting four or more visits did not report receiving the essential interventions. Large disparities exist in ANC use by household wealth, women's education and residence area; and are wider for a larger number of visits. The multivariable analyses of two models in each country showed that determinants had different effects on the dependent variable in each model. Overall, strong predictors of ANC initiation and having a higher frequency (4+) of visits were woman's education and household wealth. Gestational age at first visit, birth rank and preceding birth interval were generally negatively associated with initiating visits and with having four or more visits. Information on country policies and programs were somewhat informative in understanding the utilization patterns across the countries, although timing of adoption and actual implementation make direct linkages impossible to verify. CONCLUSION: Secondary analyses provided a more detailed picture of ANC utilization patterns in the seven countries. While coverage levels differ by country and sub-groups, all countries can benefit from specific in-country assessments to properly identify the underserved women and the reasons behind low coverage and missed interventions. Overall, emphasis needs to be put on assessing the quality of care offered and identifying women's perception to the care as well as the barriers hindering utilization. Country policies and programs need to be reviewed, evaluated and/or implemented properly to ensure that women receive the recommended number of ANC visits with appropriate content, especially, poor and less educated women residing in rural areas.

Evaluation of a mHealth Data Quality Intervention to Improve Documentation of Pregnancy Outcomes by Health Surveillance Assistants in Malawi: A Cluster Randomized Trial.

BACKGROUND: While community health workers are being recognized as an integral work force with growing responsibilities, increased demands can potentially affect motivation and performance. The ubiquity of mobile phones, even in hard-to-reach communities, has facilitated the pursuit of novel approaches to support community health workers beyond traditional modes of supervision, job aids, in-service training, and material compensation. We tested whether supportive short message services (SMS) could improve reporting of pregnancies and pregnancy outcomes among community health workers (Health Surveillance Assistants, or HSAs) in Malawi. METHODS AND FINDINGS: We designed a set of one-way SMS that were sent to HSAs on a regular basis during a 12-month period. We tested the effectiveness of the cluster-randomized intervention in improving the complete documentation of a pregnancy. We defined complete documentation as a pregnancy for which a specific outcome was recorded. HSAs in the treatment group received motivational and data quality SMS. HSAs in the control group received only motivational SMS. During baseline and intervention periods, we matched reported pregnancies to reported outcomes to determine if reporting of matched pregnancies differed between groups and by period. The trial is registered as ISCTRN24785657. CONCLUSIONS: Study results show that the mHealth intervention improved the documentation of matched pregnancies in both the treatment (OR 1.31, 95% CI: 1.10-1.55, p<0.01) and control (OR 1.46, 95% CI: 1.11-1.91, p = 0.01) groups relative to the baseline period, despite differences in SMS content between groups. The results should be interpreted with caution given that the study was underpowered. We did not find a statistically significant difference in matched pregnancy documentation between groups during the intervention period (OR 0.94, 95% CI: 0.63-1.38, p = 0.74). mHealth applications have the potential to improve the tracking and data quality of pregnancies and pregnancy outcomes, particularly in low-resource settings.

Strengthening Community-Based Vital Events Reporting for Real-Time Monitoring of Under-Five Mortality: Lessons Learned from the Balaka and Salima Districts in Malawi.

BACKGROUND: Malawi ratified a compulsory birth and death registration system in 2009. Until it captures complete coverage of vital events, Malawi relies on other data sources to calculate mortality estimates. We tested a community-based method to estimate annual under-five mortality rates (U5MR) through the Real-Time Monitoring of Under-Five Mortality (RMM) project in Malawi. We implemented RMM in two phases, and conducted an independent evaluation of phase one after 21 months of implementation. We present results of the phase two validation that covers the full project time span, and compare the results to those of the phase one validation. METHODS AND FINDINGS: We assessed the completeness of the counts of births and deaths and the accuracy of disaggregated U5MR from the community-based method against a retrospective full pregnancy history for rolling twelve-month periods after the independent evaluation. We used full pregnancy histories collected through household interviews carried out between November 2013 and January 2014 as the validation data source. Health Surveillance Agents (HSAs) across the 160 catchment areas submitted routine reports on pregnancies, births, and deaths consistently. However, for the 15-month implementation period post-evaluation, average completeness of birth event reporting was 76%, whereas average completeness of death event reporting was 67% relative to that expected from a comparable pregnancy history. HSAs underestimated the U5MR by an average of 21% relative to that estimated from a comparable pregnancy history. CONCLUSIONS: On a medium scale, the community-based RMM method in Malawi produced substantial underestimates of annualized U5MR relative to those obtained from a full pregnancy history, despite the additional incentives and quality-control activities. We were not able to achieve an optimum level of incentive and support to make the system work while ensuring sustainability. Lessons learned from the implementation of RMM can inform programs supporting community-based interventions through HSAs in Malawi.

Malawi and Millennium Development Goal 4: a Countdown to 2015 country case study.

BACKGROUND: Several years in advance of the 2015 endpoint for the Millennium Development Goals (MDGs), Malawi was already thought to be one of the few countries in sub-Saharan Africa likely to meet the MDG 4 target of reducing under-5 mortality by two-thirds between 1990 and 2015. Countdown to 2015 therefore selected the Malawi National Statistical Office to lead an in-depth country case study, aimed mainly at explaining the country's success in improving child survival. METHODS: We estimated child and neonatal mortality for the years 2000-14 using five district-representative household surveys. The study included recalculation of coverage indicators for that period, and used the Lives Saved Tool (LiST) to attribute the child lives saved in the years from 2000 to 2013 to various interventions. We documented the adoption and implementation of policies and programmes affecting the health of women and children, and developed estimates of financing. FINDINGS: The estimated mortality rate in children younger than 5 years declined substantially in the study period, from 247 deaths (90% CI 234-262) per 1000 livebirths in 1990 to 71 deaths (58-83) in 2013, with an annual rate of decline of 5·4%. The most rapid mortality decline occurred in the 1-59 months age group; neonatal mortality declined more slowly (from 50 to 23 deaths per 1000 livebirths), representing an annual rate of decline of 3·3%. Nearly half of the coverage indicators have increased by more than 20 percentage points between 2000 and 2014. Results from the LiST analysis show that about 280,000 children's lives were saved between 2000 and 2013, attributable to interventions including treatment for diarrhoea, pneumonia, and malaria (23%), insecticide-treated bednets (20%), vaccines (17%), reductions in wasting (11%) and stunting (9%), facility birth care (7%), and prevention and treatment of HIV (7%). The amount of funding allocated to the health sector has increased substantially, particularly to child health and HIV and from external sources, but remains below internationally agreed targets. Key policies to address the major causes of child mortality and deliver high-impact interventions at scale throughout Malawi began in the late 1990s and intensified in the latter half of the 2000s and into the 2010s, backed by health-sector-wide policies to improve women's and children's health. INTERPRETATION: This case study confirmed that Malawi had achieved MDG 4 for child survival by 2013. Our findings suggest that this was achieved mainly through the scale-up of interventions that are effective against the major causes of child deaths (malaria, pneumonia, and diarrhoea), programmes to reduce child undernutrition and mother-to-child transmission of HIV, and some improvements in the quality of care provided around birth. The Government of Malawi was among the first in sub-Saharan Africa to adopt evidence-based policies and implement programmes at scale to prevent unnecessary child deaths. Much remains to be done, building on this success and extending it to higher proportions of the population and targeting continued high neonatal mortality rates. FUNDING: Bill & Melinda Gates Foundation, WHO, The World Bank, Government of Australia, Government of Canada, Government of Norway, Government of Sweden, Government of the UK, and UNICEF.

Independent Evaluation of the Rapid Scale-Up Program to Reduce Under-Five Mortality in Burkina Faso.

We conducted a prospective evaluation of the "Rapid Scale-Up" (RSU) program in Burkina Faso, focusing on the integrated community case management (iCCM) component of the program. We used a quasi-experimental design in which nine RSU districts were compared with seven districts without the program. The evaluation included documentation of program implementation, assessments of implementation and quality of care, baseline and endline coverage surveys, and estimation of mortality changes using the Lives Saved Tool. Although the program trained large numbers of community health workers, there were implementation shortcomings related to training, supervision, and drug stockouts. The quality of care provided to sick children was poor, and utilization of community health workers was low. Changes in intervention coverage were comparable in RSU and comparison areas. Estimated under-five mortality declined by 6.2% (from 110 to 103 deaths per 1,000 live births) in the RSU area and 4.2% (from 114 to 109 per 1,000 live births) in the comparison area. The RSU did not result in coverage increases or mortality reductions in Burkina Faso, but we cannot draw conclusions about the effectiveness of the iCCM strategy, given implementation shortcomings. The evaluation results highlight the need for greater attention to implementation of iCCM programs.

Effects of the integrated Community Case Management of Childhood Illness Strategy on Child Mortality in Ethiopia: A Cluster Randomized Trial.

We conducted a cluster randomized trial of the effects of the integrated community case management of childhood illness (iCCM) strategy on careseeking for and coverage of correct treatment of suspected pneumonia, diarrhea, and malaria, and mortality among children aged 2-59 months in 31 districts of the Oromia region of Ethiopia. We conducted baseline and endline coverage and mortality surveys approximately 2 years apart, and assessed program strength after about 1 year of implementation. Results showed strong iCCM implementation, with iCCM-trained workers providing generally good quality of care. However, few sick children were taken to iCCM providers (average 16 per month). Difference in differences analyses revealed that careseeking for childhood illness was low and similar in both study arms at baseline and endline, and increased only marginally in intervention (22.9-25.7%) and comparison (23.3-29.3%) areas over the study period (P = 0.77). Mortality declined at similar rates in both study arms. Ethiopia's iCCM program did not generate levels of demand and utilization sufficient to achieve significant increases in intervention coverage and a resulting acceleration in reductions in child mortality. This evaluation has allowed Ethiopia to strengthen its strategic approaches to increasing population demand and use of iCCM services.

Independent Evaluation of the integrated Community Case Management of Childhood Illness Strategy in Malawi Using a National Evaluation Platform Design.

We evaluated the impact of integrated community case management of childhood illness (iCCM) on careseeking for childhood illness and child mortality in Malawi, using a National Evaluation Platform dose-response design with 27 districts as units of analysis. "Dose" variables included density of iCCM providers, drug availability, and supervision, measured through a cross-sectional cellular telephone survey of all iCCM-trained providers. "Response" variables were changes between 2010 and 2014 in careseeking and mortality in children aged 2-59 months, measured through household surveys. iCCM implementation strength was not associated with changes in careseeking or mortality. There were fewer than one iCCM-ready provider per 1,000 under-five children per district. About 70% of sick children were taken outside the home for care in both 2010 and 2014. Careseeking from iCCM providers increased over time from about 2% to 10%; careseeking from other providers fell by a similar amount. Likely contributors to the failure to find impact include low density of iCCM providers, geographic targeting of iCCM to "hard-to-reach" areas although women did not identify distance from a provider as a barrier to health care, and displacement of facility careseeking by iCCM careseeking. This suggests that targeting iCCM solely based on geographic barriers may need to be reconsidered.