From raw data to a score: comparing quantitative methods that construct multi-level composite implementation strength scores of family planning programs in Malawi.

BACKGROUND: Data that capture implementation strength can be combined in multiple ways across content and health system levels to create a summary measure that can help us to explore and compare program implementation across facility catchment areas. Summary indices can make it easier for national policymakers to understand and address variation in strength of program implementation across jurisdictions. In this paper, we describe the development of an index that we used to describe the district-level strength of implementation of Malawi's national family planning program. METHODS: To develop the index, we used data collected during a 2017 national, health facility and community health worker Implementation Strength Assessment survey in Malawi to test different methods to combine indicators within and then across domains (4 methods-simple additive, weighted additive, principal components analysis, exploratory factor analysis) and combine scores across health facility and community health worker levels (2 methods-simple average and mixed effects model) to create a catchment area-level summary score for each health facility in Malawi. We explored how well each model captures variation and predicts couple-years protection and how feasible it is to conduct each type of analysis and the resulting interpretability. RESULTS: We found little difference in how the four methods combined indicator data at the individual and combined levels of the health system. However, there were major differences when combining scores across health system levels to obtain a score at the health facility catchment area level. The scores resulting from the mixed effects model were able to better discriminate differences between catchment area scores compared to the simple average method. The scores using the mixed effects combination method also demonstrated more of a dose-response relationship with couple-years protection. CONCLUSIONS: The summary measure that was calculated from the mixed effects combination method captured the variation of strength of implementation of Malawi's national family planning program at the health facility catchment area level. However, the best method for creating an index should be based on the pros and cons listed, not least, analyst capacity and ease of interpretability of findings. Ultimately, the resulting summary measure can aid decision-makers in understanding the combined effect of multiple aspects of programs being implemented in their health system and comparing the strengths of programs across geographies.

Independent Evaluation of the integrated Community Case Management of Childhood Illness Strategy in Malawi Using a National Evaluation Platform Design.

We evaluated the impact of integrated community case management of childhood illness (iCCM) on careseeking for childhood illness and child mortality in Malawi, using a National Evaluation Platform dose-response design with 27 districts as units of analysis. "Dose" variables included density of iCCM providers, drug availability, and supervision, measured through a cross-sectional cellular telephone survey of all iCCM-trained providers. "Response" variables were changes between 2010 and 2014 in careseeking and mortality in children aged 2-59 months, measured through household surveys. iCCM implementation strength was not associated with changes in careseeking or mortality. There were fewer than one iCCM-ready provider per 1,000 under-five children per district. About 70% of sick children were taken outside the home for care in both 2010 and 2014. Careseeking from iCCM providers increased over time from about 2% to 10%; careseeking from other providers fell by a similar amount. Likely contributors to the failure to find impact include low density of iCCM providers, geographic targeting of iCCM to "hard-to-reach" areas although women did not identify distance from a provider as a barrier to health care, and displacement of facility careseeking by iCCM careseeking. This suggests that targeting iCCM solely based on geographic barriers may need to be reconsidered.