Comparing program supervision with an external RADAR evaluation of quality of care in integrated community case management for childhood illnesses in Mali.

BACKGROUND: Many countries have adopted integrated community case management (iCCM) to reduce mortality among children under five years from common childhood illnesses. The 2016-2020 Malian Red Cross iCCM program trained 441 Community Health Workers (CHWs) to treat malaria, pneumonia, diarrhea, and malnutrition for children under five years of age in six districts. Implementation strength and quality of care (QoC) were assessed through the program's supervision function, using the Malian Ministry of Health's system. OBJECTIVE: This paper compares methods and results of program supervision data and an independent evaluation to assess the effectiveness of program implementation and supervision and inform program improvement. It also presents the benefits and limitations of each method. METHOD: An independent QoC evaluation was conducted using tools developed by the Real Accountability: Data Analysis for Results (RADAR) project, hereafter referred to as the RADAR evaluation. RADAR evaluation data collected in July and August 2018 were compared with program supervision data collected mostly between May and December 2018. RESULTS: The RADAR evaluation provided detailed findings on correct assessment, classification, and treatment per illness, medication type, and dosage. Program supervision combined the findings for all illnesses, medication type, and dosage due to limitations in the data collection process. Six indicators were comparable between both methods. Findings were similar for temperature and mid-upper arm circumference measurements but diverged between program supervision and the RADAR evaluation, respectively, on correct classification for all illnesses (87.1% vs. 65.3%), correct treatment for all illnesses (69.5% vs. 39.8%), correct respiratory rate counting (88.5% vs. 54.7%), and administering the first dose by CHW (75.4% vs. 65.0%). Findings from the RADAR evaluation guided improvements in program supervision. CONCLUSIONS: A robust program supervision system can serve as a credible method to assess QoC. However, a rigorous independent QoC evaluation provides a valuable benchmark to gauge the effectiveness of the supervisory process.

Validity of using mobile phone surveys to evaluate community health worker program in Mali.

BACKGROUND: The monitoring and evaluation of public health programs based on traditional face-to-face interviews in hard-to-reach and unstable regions present many challenges. Mobile phone-based methods are considered to be an effective alternative, but the validity of mobile phone-based data for assessing implementation strength has not been sufficiently studied yet. Nested within an evaluation project for an integrated community case management (iCCM) and family planning program in Mali, this study aimed to assess the validity of a mobile phone-based health provider survey to measure the implementation strength of this program. METHODS: From July to August 2018, a cross-sectional survey was conducted among the community health workers (ASCs) from six rural districts working with the iCCM and family planning program. ASCs were first reached to complete the mobile phone-based survey; within a week, ASCs were visited in their communities to complete the in-person survey. Both surveys used identical implementation strength tools to collect data on program activities related to iCCM and family planning. Sensitivity and specificity were calculated for each implementation strength indicator collected from the phone-based survey, with the in-person survey as the gold standard. A threshold of ≥ 80% for sensitivity and specificity was considered adequate for evaluation purposes. RESULTS: Of the 157 ASCs interviewed by mobile phone, 115 (73.2%) were reached in person. Most of the training (2/2 indicators), supervision (2/3), treatment/modern contraceptive supply (9/9), and reporting (3/3) indicators reached the 80% threshold for sensitivity, while only one supervision indicator and one supply indicator reached 80% for specificity. In contrast, most of the stock-out indicators (8/9) reached 80% for specificity, while only two indicators reached the threshold for sensitivity. CONCLUSIONS: The validity of mobile phone-based data was adequate for general training, supervision, and supply indicators for iCCM and family planning. With sufficient mobile phone coverage and reliable mobile network connection, mobile phone-based surveys are useful as an alternative for data collection to assess the implementation strength of general activities in hard-to-reach areas.

Community-based health care is an essential component of a resilient health system: evidence from Ebola outbreak in Liberia.

BACKGROUND: Trained community health workers (CHW) enhance access to essential primary health care services in contexts where the health system lacks capacity to adequately deliver them. In Liberia, the Ebola outbreak further disrupted health system function. The objective of this study is to examine the value of a community-based health system in ensuring continued treatment of child illnesses during the outbreak and the role that CHWs had in Ebola prevention activities. METHODS: A descriptive observational study design used mixed methods to collect data from CHWs (structured survey, n = 60; focus group discussions, n = 16), government health facility workers and project staff. Monthly data on child diarrhea and pneumonia treatment were gathered from CHW case registers and local health facility records. RESULTS: Coverage for community-based treatment of child diarrhea and pneumonia continued throughout the outbreak in project areas. A slight decrease in cases treated during the height of the outbreak, from 50 to 28% of registers with at least one treatment per month, was attributed to directives not to touch others, lack of essential medicines and fear of contracting Ebola. In a climate of distrust, where health workers were reluctant to treat patients, sick people were afraid to self-identify and caregivers were afraid to take children to the clinic, CHWs were a trusted source of advice and Ebola prevention education. These findings reaffirm the value of recruiting and training local workers who are trusted by the community and understand the social and cultural complexities of this relationship. "No touch" integrated community case management (iCCM) guidelines distributed at the height of the outbreak gave CHWs renewed confidence in assessing and treating sick children. CONCLUSIONS: Investments in community-based health service delivery contributed to continued access to lifesaving treatment for child pneumonia and diarrhea during the Ebola outbreak, making communities more resilient when facility-based health services were impacted by the crisis. To maximize the effectiveness of these interventions during a crisis, proactive training of CHWs in infection prevention and "no touch" iCCM guidelines, strengthening drug supply chain management and finding alternative ways to provide supportive supervision when movements are restricted are recommended.