From raw data to a score: comparing quantitative methods that construct multi-level composite implementation strength scores of family planning programs in Malawi.

BACKGROUND: Data that capture implementation strength can be combined in multiple ways across content and health system levels to create a summary measure that can help us to explore and compare program implementation across facility catchment areas. Summary indices can make it easier for national policymakers to understand and address variation in strength of program implementation across jurisdictions. In this paper, we describe the development of an index that we used to describe the district-level strength of implementation of Malawi's national family planning program. METHODS: To develop the index, we used data collected during a 2017 national, health facility and community health worker Implementation Strength Assessment survey in Malawi to test different methods to combine indicators within and then across domains (4 methods-simple additive, weighted additive, principal components analysis, exploratory factor analysis) and combine scores across health facility and community health worker levels (2 methods-simple average and mixed effects model) to create a catchment area-level summary score for each health facility in Malawi. We explored how well each model captures variation and predicts couple-years protection and how feasible it is to conduct each type of analysis and the resulting interpretability. RESULTS: We found little difference in how the four methods combined indicator data at the individual and combined levels of the health system. However, there were major differences when combining scores across health system levels to obtain a score at the health facility catchment area level. The scores resulting from the mixed effects model were able to better discriminate differences between catchment area scores compared to the simple average method. The scores using the mixed effects combination method also demonstrated more of a dose-response relationship with couple-years protection. CONCLUSIONS: The summary measure that was calculated from the mixed effects combination method captured the variation of strength of implementation of Malawi's national family planning program at the health facility catchment area level. However, the best method for creating an index should be based on the pros and cons listed, not least, analyst capacity and ease of interpretability of findings. Ultimately, the resulting summary measure can aid decision-makers in understanding the combined effect of multiple aspects of programs being implemented in their health system and comparing the strengths of programs across geographies.

Testing the validity and feasibility of using a mobile phone-based method to assess the strength of implementation of family planning programs in Malawi.

BACKGROUND: To effectively deliver on proposed objectives, it is vital that practitioners, policymakers, and other stakeholders are able to clearly understand how strongly their large-scale program is being implemented. This study sought to test the feasibility, cost-effectiveness, and validity of a phone-based method as an innovative and cost-efficient approach to assessing program implementation strength (through an Implementation Strength Assessment - ISA), alternative to the traditional in-person field methods. METHODS: We conducted 701 mobile phone and 356 in-person interviews with facility in-Charges and two types of community health workers who provide family planning services in the Dowa and Ntcheu districts in Malawi. Responses received via the phone interview were validated through in-person review of records and inspections. Sensitivity and specificity were calculated to determine validity. RESULTS: Most indicators at the health facility and community health worker levels were above a 70% threshold for sensitivity. However, there were fewer indicators that met this threshold for specificity. The primary reason for lower specificity was due to poor recordkeeping. Collecting data via mobile phone was found to be feasible and twice as cost-efficient as collecting the same data via in-person inspections. CONCLUSIONS: The rapid increase in mobile phone ownership and network availability in lower income countries could offer an alternative, cost-effective avenue to collect data for a better understanding of program implementation. Through rigorous assessment, this study found that using mobile phones could be a low-cost alternative to collect data on health system delivery of services, especially in places where routine data quality is poor and traditional, in-person methods are costly.

Adjustments in purchasing arrangements to support the COVID-19 health sector response: evidence from eight middle-income countries.

The COVID-19 pandemic has triggered several changes in countries' health purchasing arrangements to accompany the adjustments in service delivery in order to meet the urgent and additional demands for COVID-19-related services. However, evidence on how these adjustments have played out in low- and middle-income countries is scarce. This paper provides a synthesis of a multi-country study of the adjustments in purchasing arrangements for the COVID-19 health sector response in eight middle-income countries (Armenia, Cameroon, Ghana, Kenya, Nigeria, Philippines, Romania and Ukraine). We use secondary data assembled by country teams, as well as applied thematic analysis to examine the adjustments made to funding arrangements, benefits packages, provider payments, contracting, information management systems and governance arrangements as well as related implementation challenges. Our findings show that all countries in the study adjusted their health purchasing arrangements to varying degrees. While the majority of countries expanded their benefit packages and several adjusted payment methods to provide selected COVID-19 services, only half could provide these services free of charge. Many countries also streamlined their processes for contracting and accrediting health providers, thereby reducing administrative hurdles. In conclusion, it was important for the countries to adjust their health purchasing arrangements so that they could adequately respond to the COVID-19 pandemic, but in some countries financing challenges resulted in issues with equity and access. However, it is uncertain whether these adjustments can and will be sustained over time, even where they have potential to contribute to making purchasing more strategic to improve efficiency, quality and equitable access in the long run.

The quantitative story behind the quality improvement storyboards: a synthesis of quality improvement projects conducted by the multi-state learning collaborative.

CONTEXT: The Multi-State Learning Collaborative: Lead States in Public Health Quality Improvement (MLC) brought state and local health departments in 16 states together with public health system and national partners to prepare for national voluntary accreditation and to implement quality improvement (QI) practices. OBJECTIVE: The MLC has collected the single largest repository of qualitative public health QI data to date. A preliminary study was conducted to explore the potential merits of further mining data sets of this size and scope and examining them quantitatively. DESIGN: We addressed the following research question: What characteristics of QI projects/mini-collaboratives make them more or less likely to attain their stated objectives? Qualitative MLC data were modified and coded as quantifiable measures using categorical or Likert scale measures analyzable through quantitative methods. Descriptive and inferential statistics were calculated. RESULTS: Of the 156 mini-collaboratives with complete data, chronic disease was the most commonly selected target area. Among the 4 dependent variables, results varied somewhat by outcome. There was support in 1 or more analytical models for a positive relationship between aim statements that included target objectives, time frames, measurable goals, and well-defined processes. The degree to which the intervention was logically aligned with the aim and the comprehensiveness of the QI project were also positively associated with 1 or more outcomes. The large number of statistical tests conducted may have led to type I errors for some comparisons. CONCLUSIONS: Quantitative analysis and modeling of public health QI activities are feasible and desirable. It may provide critical information leading to incremental improvement in QI performance within public health practice. This work can inform the nascent national accreditation program and the developing QI in Public Health Practice Exchange.

National health insurance contribution to family planning program funding in Indonesia: A fund flow analysis.

Background: Launched in 2014, Indonesia's national health insurance system (JKN) aimed to provide universal health coverage, including contraceptive services, to its population. We aim to evaluate the contribution of JKN to the overall spending for the family planning program in Indonesia. Methods: Data from the Indonesian Demographic Health Survey, Survey on Financial Flows for Family Planning, Indonesia Motion Tracker Matrix, World Population Prospect, and Indonesian ministries' budget accountability reports were entered into the CastCost Contraceptive Projection Tool to define budgetary allocation and spending for the family planning program at the national level in 2019. Results: Indonesia's family planning program in 2019 was financed mostly by the national budget (64.0%) and out-of-pocket payments (34.6%). There were three main ministries responsible for family planning financing: the National Population and Family Planning Board (BKKBN) (35.8%), the Ministry of Finance (26.2%), and the Ministry of Health (2.0%). Overall, JKN contributed less than 0.4% of the funding for family planning services in Indonesia in 2019. The majority of family planning spending was by public facilities (57.3%) as opposed to private facilities (28.6%). Conclusion: JKN's contribution to funding Indonesia's family planning programs in 2019 was low and highlights a huge opportunity to expand these contributions. A coordinated effort should be conducted to identify possible opportunities to realign BKKBN and JKN roles in the family planning programs and lift barriers to accessing family planning services in public and private facilities. This includes a concerted effort to improve integration of private family planning providers into the JKN program.

How Jaminan Kesehatan Nasional (JKN) coverage influences out-of-pocket (OOP) payments by vulnerable populations in Indonesia.

While Indonesia introduced a national health insurance scheme (JKN) in 2014 and coverage has grown to over 80% of the population, Indonesians still spend significant sums out-of-pocket (OOP) for their healthcare-over 30% of current health expenditure (CHE). This study aims to better understand how JKN is influencing OOP payments, especially among the poor and rural, at the range of health facilities. This study uses data from the National Socio-Economic Survey (SUSENAS) in 2018 and 2019, as these surveys started including a question on how much OOP spending a household incurs on health. The results show that households with JKN membership are far less likely than the uninsured to pay OOP for healthcare, and that if they do incur a cost, the magnitude of this cost is much lower among JKN households than uninsured ones. The results also show that JKN households in the two poorest quintiles have a higher probability to not incur any OOP (37% and 35%, respectively) compared to those in the wealthier quintiles 4 (32%) and 5 (30%). Poorer JKN households living in the eastern part of Indonesia-the less urbanized and developed regions-experienced the most cost-savings, though largely due to supply-side constraints. In fact, JKN members save more at public primary health care facilities vs. private ones (who often do not contract with JKN) and also save significantly more (over 50%) than uninsured households at both public and private hospitals. The study demonstrates the positive influence JKN has on OOP payments, especially among the poor and rural, but also highlights how the scheme needs to better engage with the growing private sector and invest in infrastructure in rural areas to help secure financial protection for its entire population.

The aggregate effect of implementation strength of family planning programs on modern contraceptive use at the health systems level in rural Malawi.

To explore the association between the strength of implementation of family planning (FP) programs on the use of modern contraceptives. Specifically, how strongly these programs are being implemented across a health facility's catchment area in Malawi and the odds of a woman in that catchment area is using modern contraceptives. This information can be used to assess whether the combined impact of multiple large-scale FP programs is leading to change in the health outcomes they aim to improve. We used data from the 2017 Implementation Strength Assessment (ISA) that quantified how much of family planning programs at the health facility and community health worker levels were being implemented across every district of Malawi. We used a summary measure developed in a previous study that employs quantitative methods to combine data across FP domains and health system levels. We tested the association of this summary measure for implementation strength with household data from the 2015 Malawi Demographic Health Survey (DHS). We found that areas with stronger implementation of FP programs had higher odds of women using modern contraceptives compared with areas with weaker implementation. The association of ISA with use of modern contraception was different by education, marital status, and geography. After controlling for these factors, we found that the adjusted odds of using a modern contraceptive was three times higher in catchment areas with high implementation strength compared to those with lower strength. Metrics that summarize how strongly FP programs are being implemented were used to show a statistically significantly positive relationship between increasing implementation strength and higher rates of modern contraceptive use. Decisionmakers at the various levels of health authority can use this type of summary measure to better understand the combined impact of their diverse FP programming and inform future programmatic and policy decisions. The findings also reinforce the idea that having a well-supported and supplied cadre of community health workers supplementing FP provision at the health facility can be an important health systems mechanism, especially in rural settings and to target youth populations.

How strong are Malawi's family planning programs for adolescent and adult women? Results of a national assessment of implementation strength conducted by Malawi's National Evaluation Platform.

BACKGROUND: To assess the strength of implementation of family planning programs targeting youth (15-24) in Malawi with a specific focus on youth and the Youth-Friendly Health Services program. METHODS: We conducted 9781 mobile phone interviews with facility in-Charge Nurses and health workers (health facility workers, health surveillance assistants [HSAs] and community-based distribution agent [CBDAs]) who provide family planning (FP) services across the 28 districts. Responses were entered in tablet using Open Data Kit. They were summarized and presented using R, Stata (College Station, TX, USA, StatsReport, JHU, Baltimore, MD, USA) and ArcView GIS (ESRI, Redlands, CA, USA). RESULTS: Availability of key products was a challenge across all health worker types as only 39% of health facilities, 29% of HSAs and 45% of CBDAs had all the FP methods they are supposed to provide on the day of the interview. About 50% of health workers were supervised within past 90 days preceding the study. Despite most facilities saying that they provide youth friendly health services, youth-specific FP guidelines or protocols were not available in 43% of facilities that provide these services and only 33% of facilities had special rooms and 58% have special days for youth. CONCLUSIONS: The commodity supply system needs to ensure that all facilities and workers have a consistent supply of all contraceptive methods. Government and program implementers should ensure availability of all FP guidelines and information, education, communication materials at all service delivery points and facilitate creation of special rooms or days for youth.