Hierarchical Statistical Models to Represent and Visualize Survey Evidence for Program Evaluation: iCCM in Malawi.

Policy and Program evaluation for maternal, newborn and child health is becoming increasingly complex due to changing contexts. Monitoring and evaluation efforts in this area can take advantage of large nationally representative household surveys such as DHS or MICS that are increasing in size and frequency, however, this analysis presents challenges on several fronts. We propose an approach with hierarchical models for cross-sectional survey data to describe evidence relating to program evaluation, and apply this approach to the recent scale up of iCCM in Malawi. We describe careseeking for children sick with diarrhea, pneumonia, or malaria with empirical Bayes estimates for each district of Malawi at two time points, both for careseeking from any source, and for careseeking only from health surveillance assistants (HSA). We do not find evidence that children in areas with more HSA trained in iCCM are more likely to seek care for pneumonia, diarrhea, or malaria, despite evidence that many indeed are seeking care from HSA. Children in areas with more HSA trained in iCCM are more likely to seek care from a HSA, with 100 additional trained health workers in a district corresponding to a 2% average increase in careseeking from HSA. The hierarchical models presented here provide a flexible set of methods that describe the primary evidence for evaluating iCCM in Malawi and which could be extended to formal causal analyses, and to analysis for other similar evaluations with national survey data.

Effects of the integrated Community Case Management of Childhood Illness Strategy on Child Mortality in Ethiopia: A Cluster Randomized Trial.

We conducted a cluster randomized trial of the effects of the integrated community case management of childhood illness (iCCM) strategy on careseeking for and coverage of correct treatment of suspected pneumonia, diarrhea, and malaria, and mortality among children aged 2-59 months in 31 districts of the Oromia region of Ethiopia. We conducted baseline and endline coverage and mortality surveys approximately 2 years apart, and assessed program strength after about 1 year of implementation. Results showed strong iCCM implementation, with iCCM-trained workers providing generally good quality of care. However, few sick children were taken to iCCM providers (average 16 per month). Difference in differences analyses revealed that careseeking for childhood illness was low and similar in both study arms at baseline and endline, and increased only marginally in intervention (22.9-25.7%) and comparison (23.3-29.3%) areas over the study period (P = 0.77). Mortality declined at similar rates in both study arms. Ethiopia's iCCM program did not generate levels of demand and utilization sufficient to achieve significant increases in intervention coverage and a resulting acceleration in reductions in child mortality. This evaluation has allowed Ethiopia to strengthen its strategic approaches to increasing population demand and use of iCCM services.

Measuring Implementation Strength for Integrated Community Case Management in Malawi: Results from a National Cell Phone Census.

Program managers, investors, and evaluators need real-time information on how program strategies are being scaled up and implemented. Integrated Community Case Management (iCCM) of childhood illnesses is a strategy for increasing access to diagnosis and treatment of malaria, pneumonia, and diarrhea through community-based health workers. We collected real-time data on iCCM implementation strength through cell phone interviews with community-based health workers in Malawi and calculated indicators of implementation strength and utilization at district level using consensus definitions from the Ministry of Health (MOH) and iCCM partners. All of the iCCM implementation strength indicators varied widely within and across districts. Results show that Malawi has made substantial progress in the scale-up of iCCM since the 2008 program launch. However, there are wide differences in iCCM implementation strength by district. Districts that performed well according to the survey measures demonstrate that MOH implementation strength targets are achievable with the right combination of supportive structures. Using the survey results, specific districts can now be targeted with additional support.

Coverage and equitability of interventions to prevent child mortality in rural Jimma and West Hararghe Zones, Oromia Region, Ethiopia.

BACKGROUND: Interventions to prevent childhood illnesses are important components of the Ethiopian Health Extension Program (HEP). Although the HEP was designed to reduce inequities in access to health care, there is little evidence on equitability of preventive interventions in Ethiopia. PURPOSE: This article describes coverage of preventive interventions and how many interventions individual children received We also examined which factors were associated with the number of preventive interventions received, and assessed the extent to which interventions were equitably distributed. METHODS: We conducted a cross-sectional survey in 3,200 randomly selected households in the rural Jimma and West Hararghe Zones of Ethiopia's Oromia Region. We calculated coverage of 10 preventive interventions and a composite of eight interventions (co-coverage) representing the number of interventions received by children. Mul- tiple linear regressions were used to assess associations between co-coverage and explanatory variables. Finally, we assessed the equitability of preventive interventions by comparing coverage among children in the poorest and the least poor wealth quintiles. RESULTS: Coverage was less than 50% for six of the 10 interventions. Children received on average only three of the eight interventions included in the co-coverage calculation. Zone, gender, caretaker age, religion, and household wealth were all significantly associated with co-coverage, controlling for key covariates. Exclusive breastfeeding, vaccine uptake, and vitamin A supplementation were all relatively equitable. On the other hand, coverage of insecticide-treated nets or indoor residual spraying (ITN/IRS) and access to safe water were significantly higher among the least poor children compared to children in the poorest quintile. CONCLUSION: Coverage of key interventions to prevent childhood illnesses is generally low in Jimma and West Hararghe. Although a number of interventions were equitably distributed, there were marked wealth-based inequities for interventions that are possessed at the household level, even among relatively homogeneous rural communities.

Assessment of the monitoring and evaluation system for integrated community case management (ICCM) in Ethiopia: a comparison against global benchmark indicators.

BACKGROUND: Program managers require feasible, timely, reliable, and valid measures of iCCM implementation to identify problems and assess progress. The global iCCM Task Force developed benchmark indicators to guide implementers to develop or improve monitoring and evaluation (M&E) systems. OBJECTIVE: To assesses Ethiopia's iCCM M&E system by determining the availability and feasibility of the iCCM benchmark indicators. METHODS: We conducted a desk review of iCCM policy documents, monitoring tools, survey reports, and other rele- vant documents; and key informant interviews with government and implementing partners involved in iCCM scale-up and M&E. RESULTS: Currently, Ethiopia collects data to inform most (70% [33/47]) iCCM benchmark indicators, and modest extra effort could boost this to 83% (39/47). Eight (17%) are not available given the current system. Most benchmark indicators that track coordination and policy, human resources, service delivery and referral, supervision, and quality assurance are available through the routine monitoring systems or periodic surveys. Indicators for supply chain management are less available due to limited consumption data and a weak link with treatment data. Little information is available on iCCM costs. CONCLUSION: Benchmark indicators can detail the status of iCCM implementation; however, some indicators may not fit country priorities, and others may be difficult to collect. The government of Ethiopia and partners should review and prioritize the benchmark indicators to determine which should be included in the routine M&E system, especially since iCCMdata are being reviewed for addition to the HMIS. Moreover, the Health Extension Worker's reporting burden can be minimized by an integrated reporting approach.

Coverage and Equitability of Interventions to Prevent Child Mortality in Rural Jimma and West Hararghe Zones; Oromia Region; Ethiopia

Background: Interventions to prevent childhood illnesses are important components of the Ethiopian Health Extension Program (HEP). Although the HEP was designed to reduce inequities in access to health care; there is little evidence on equitability of preventive interventions in Ethiopia. Purpose: This article describes coverage of preventive interventions and how many interventions individual children received. We also examined which factors were associated with the number of preventive interventions received; and assessed the extent to which interventions were equitably distributed. Methods: We conducted a cross-sectional survey in 3;200 randomly selected households in the rural Jimma and West Hararghe Zones of Ethiopia's Oromia Region. We calculated coverage of 10 preventive interventions and a composite of eight interventions (co-coverage) representing the number of interventions received by children. Multiple linear regressions were used to assess associations between co-coverage and explanatory variables. Finally; we assessed the equitability of preventive interventions by comparing coverage among children in the poorest and the least poor wealth quintiles. Results: Coverage was less than 50 for six of the 10 interventions. Children received on average only three of the eight interventions included in the co-coverage calculation. Zone; gender; caretaker age; religion; and household wealth were all significantly associated with co-coverage; controlling for key covariates. Exclusive breastfeeding; vaccine uptake; and vitamin A supplementation were all relatively equitable. On the other hand; coverage of insecticide-treated nets or indoor residual spraying (ITN/IRS) and access to safe water were significantly higher among the least poor children compared to children in the poorest quintile. Conclusion: Coverage of key interventions to prevent childhood illnesses is generally low in Jimma and West Hararghe. Although a number of interventions were equitably distributed; there were marked wealth-based inequities for interventions that are possessed at the household level; even among relatively homogeneous rural communities

The Accelerated Child Survival and Development programme in west Africa: a retrospective evaluation.

BACKGROUND: UNICEF implemented the Accelerated Child Survival and Development (ACSD) programme in 11 west African countries between 2001 and 2005 to reduce child mortality by at least 25% by the end of 2006. We undertook a retrospective evaluation of the programme in Benin, Ghana, and Mali. METHODS: We used data from Demographic and Health Surveys and Multiple Indicator Cluster Surveys to compare changes in coverage for 14 ACSD interventions, nutritional status (stunting and wasting), and mortality in children younger than 5 years in the ACSD focus districts with those in the remainder of every country (comparison areas), after excluding major metropolitan areas. FINDINGS: Mortality in children younger than 5 years decreased in ACSD areas by 13% in Benin (absolute decrease 18 deaths per 1000 livebirths, p=0.12), 20% in Ghana (21 per 1000 livebirths, p=0.10), and 24% in Mali (63 per 1000 livebirths, p<0.0001), but these decreases were not greater than those in comparison areas in Benin (25%; absolute decrease 36 deaths per 1000 livebirths, p=0.15) or Mali (31%; 76 per 1000 livebirths, p=0.30; comparison data not available for Ghana). ACSD districts showed significantly greater increases than did comparison areas in coverage for preventive interventions delivered through outreach and campaign strategies in Ghana and Mali, but not Benin. Coverage in ACSD areas for correct treatment of childhood pneumonia, diarrhoea, and malaria did not differ significantly from before to after programme implementation in Benin and Mali, but decreased significantly in Ghana for malaria (from 78% to 53%, p<0.0001) and diarrhoea (from 39% to 28%, p=0.05). We recorded no significant improvements in nutritional status attributable to ACSD in the three countries. INTERPRETATION: The ACSD project did not accelerate child survival in Benin and Mali focus districts relative to comparison areas, probably because coverage for effective treatment interventions for malaria and pneumonia were not accelerated, causes of neonatal deaths and undernutrition were not addressed, and stock shortages of insecticide-treated nets restricted the potential effect of this intervention. Changes in policy and nationwide programme strengthening may have benefited from inputs by UNICEF and other partners, making an acceleration effect in the ACSD focus districts difficult to capture. FUNDING: UNICEF, Canadian International Development Agency, Coordenação de Aperfeiçoamento de Pessoal do Nível Superior (Brazil), and Fulbright Fellowship.