The effectiveness of pay-for-performance contracts with non-governmental organizations in Afghanistan - results of a controlled interrupted time series analysis.

BACKGROUND: In many contexts, including fragile settings like Afghanistan, the coverage of basic health services is low. To address these challenges there has been considerable interest in working with NGOs and examining the effect of financial incentives on service providers. The Government of Afghanistan has used contracting with NGOs for more than 15 years and in 2019 introduced pay-for-performance (P4P) into the contracts. This study examines the impact of P4P on health service delivery in Afghanistan. METHODS: We conducted an interrupted time series (ITS) analysis with a non-randomized comparison group that employed segmented regression models and used independently verified health management information system (HMIS) data from 2015 to 2021. We compared 31 provinces with P4P contracts to 3 provinces where the Ministry of Public Health (MOPH) continued to deliver services without P4P. We used data from annual health facility surveys to assess the quality of care. FINDINGS: Independent verification of the HMIS data found that consistency and accuracy was greater than 90% in the contracted provinces. The introduction of P4P increased the 10 P4P-compensated service delivery outcomes by a median of 22.1 percentage points (range 10.2 to 43.8) for the two-arm analysis and 19.9 percentage points (range: - 8.3 to 56.1) for the one-arm analysis. There was a small decrease in quality of care initially, but it was short-lived. We found few other unintended consequences. INTERPRETATION: P4P contracts with NGOs led to a substantial improvement in service delivery at lower cost despite a very difficult security situation. The promising results from this large-scale experience warrant more extensive application of P4P contracts in other fragile settings or wherever coverage remains low.

The effectiveness of supervision strategies to improve health care provider practices in low- and middle-income countries: secondary analysis of a systematic review.

BACKGROUND: Although supervision is a ubiquitous approach to support health programs and improve health care provider (HCP) performance in low- and middle-income countries (LMICs), quantitative evidence of its effects is unclear. The objectives of this study are to describe the effect of supervision strategies on HCP practices in LMICs and to identify attributes associated with greater effectiveness of routine supervision. METHODS: We performed a secondary analysis of data on HCP practice outcomes (e.g., percentage of patients correctly treated) from a systematic review on improving HCP performance. The review included controlled trials and interrupted time series studies. We described distributions of effect sizes (defined as percentage-point [%-point] changes) for each supervision strategy. To identify attributes associated with supervision effectiveness, we performed random-effects linear regression modeling and examined studies that directly compared different approaches of routine supervision. RESULTS: We analyzed data from 81 studies from 36 countries. For professional HCPs, such as nurses and physicians, primarily working at health facilities, routine supervision (median improvement when compared to controls: 10.7%-points; IQR: 9.9, 27.9) had similar effects on HCP practices as audit with feedback (median improvement: 10.1%-points; IQR: 6.2, 23.7). Two attributes were associated with greater mean effectiveness of routine supervision (p < 0.10): supervisors received supervision (by 8.8-11.5%-points), and supervisors participated in problem-solving with HCPs (by 14.2-20.8%-points). Training for supervisors and use of a checklist during supervision visits were not associated with effectiveness. The effects of supervision frequency (i.e., number of visits per year) and dose (i.e., the number of supervision visits during a study) were unclear. For lay HCPs, the effect of routine supervision was difficult to characterize because few studies existed, and effectiveness in those studies varied considerably. Evidence quality for all findings was low primarily because many studies had a high risk of bias. CONCLUSIONS: Although evidence is limited, to promote more effective supervision, our study supports supervising supervisors and having supervisors engage in problem-solving with HCPs. Supervision's integral role in health systems in LMICs justifies a more deliberate research agenda to identify how to deliver supervision to optimize its effect on HCP practices.

Opening the 'black box' of collaborative improvement: a qualitative evaluation of a pilot intervention to improve quality of malaria surveillance data in public health centres in Uganda.

BACKGROUND: Demand for high-quality surveillance data for malaria, and other diseases, is greater than ever before. In Uganda, the primary source of malaria surveillance data is the Health Management Information System (HMIS). However, HMIS data may be incomplete, inaccurate or delayed. Collaborative improvement (CI) is a quality improvement intervention developed in high-income countries, which has been advocated for low-resource settings. In Kayunga, Uganda, a pilot study of CI was conducted in five public health centres, documenting a positive effect on the quality of HMIS and malaria surveillance data. A qualitative evaluation was conducted concurrently to investigate the mechanisms of effect and unintended consequences of the intervention, aiming to inform future implementation of CI. METHODS: The study intervention targeted health workers, including brief in-service training, plus CI with 'plan-do-study-act' (PDSA) cycles emphasizing self-reflection and group action, periodic learning sessions, and coaching from a CI mentor. Health workers collected data on standard HMIS out-patient registers. The qualitative evaluation (July 2015 to September 2016) included ethnographic observations at each health centre (over 12-14 weeks), in-depth interviews with health workers and stakeholders (n = 20), and focus group discussions with health workers (n = 6). RESULTS: The results suggest that the intervention did facilitate improvement in data quality, but through unexpected mechanisms. The CI intervention was implemented as planned, but the PDSA cycles were driven largely by the CI mentor, not the health workers. In this context, characterized by a rigid hierarchy within the health system of limited culture of self-reflection and inadequate training and supervision, CI became an effective form of high-quality training with frequent supervisory visits. Health workers appeared motivated to improve data collection habits by their loyalty to the CI mentor and the potential for economic benefits, rather than a desire for self-improvement. CONCLUSIONS: CI is a promising method of quality improvement and could have a positive impact on malaria surveillance data. However, successful scale-up of CI in similar settings may require deployment of highly skilled mentors. Further research, focusing on the effectiveness of 'real world' mentors using robust study designs, will be required to determine whether CI can be translated effectively and sustainably to low-resource settings.

Effectiveness of in-service training plus the collaborative improvement strategy on the quality of routine malaria surveillance data: results of a pilot study in Kayunga District, Uganda.

BACKGROUND: Surveillance data are essential for malaria control, but quality is often poor. The aim of the study was to evaluate the effectiveness of the novel combination of training plus an innovative quality improvement method-collaborative improvement (CI)-on the quality of malaria surveillance data in Uganda. METHODS: The intervention (training plus CI, or TCI), including brief in-service training and CI, was delivered in 5 health facilities (HFs) in Kayunga District from November 2015 to August 2016. HF teams monitored data quality, conducted plan-do-study-act cycles to test changes, attended periodic learning sessions, and received CI coaching. An independent evaluation was conducted to assess data completeness, accuracy, and timeliness. Using an interrupted time series design without a separate control group, data were abstracted from 156,707 outpatient department (OPD) records, laboratory registers, and aggregated monthly reports (MR) for 4 time periods: baseline-12 months, TCI scale-up-5 months; CI implementation-9 months; post-intervention-4 months. Monthly OPD register completeness was measured as the proportion of patient records with a malaria diagnosis with: (1) all data fields completed, and (2) all clinically-relevant fields completed. Accuracy was the relative difference between: (1) number of monthly malaria patients reported in OPD register versus MR, and (2) proportion of positive malaria tests reported in the laboratory register versus MR. Data were analysed with segmented linear regression modelling. RESULTS: Data completeness increased substantially following TCI. Compared to baseline, all-field completeness increased by 60.1%-points (95% confidence interval [CI]: 46.9-73.2%) at mid-point, and clinically-relevant completeness increased by 61.6%-points (95% CI: 56.6-66.7%). A relative - 57.4%-point (95% confidence interval: - 105.5, - 9.3%) change, indicating an improvement in accuracy of malaria test positivity reporting, but no effect on data accuracy for monthly malaria patients, were observed. Cost per additional malaria patient, for whom complete clinically-relevant data were recorded in the OPD register, was $3.53 (95% confidence interval: $3.03, $4.15). CONCLUSIONS: TCI improved malaria surveillance completeness considerably, with limited impact on accuracy. Although these results are promising, the intervention's effectiveness should be evaluated in more HFs, with longer follow-up, ideally in a randomized trial, before recommending CI for wide-scale use.

The effect of mobile phone text message reminders on health workers' adherence to case management guidelines for malaria and other diseases in Malawi: lessons from qualitative data from a cluster-randomized trial.

BACKGROUND: Mobile health (mHealth), which uses technology such as mobile phones to improve patient health and health care delivery, is increasingly being tested as an intervention to promote health worker (HW) performance. This study assessed the effect of short messaging services (SMS) reminders in a study setting. Following a trial of text-message reminders to HWs to improve case management of malaria and other childhood diseases in southern Malawi that showed little effect, qualitative data was collected to explore the reasons why the intervention was ineffective and describe lessons learned. METHODS: Qualitative data collection was undertaken to lend insight into quantitative results from a trial in which 105 health facilities were randomized to three arms: (1) twice-daily text-message reminders to HWs, including clinicians and drug dispensers, on case management of malaria; (2) twice-daily text-message reminders to HWs on case management of malaria, pneumonia and diarrhoea; and, (3) a control arm. In-depth interviews were conducted with 50 HWs in the intervention arms across seven districts. HWs were asked about acceptability and feasibility of the text-messaging intervention and its perceived impact on recommended case management. The interviews were recorded, transcribed and translated into English for a thematic and framework analysis. Nvivo 11 software was used for data management and analysis. RESULTS: A total of 50 HWs were interviewed at 22 facilities. HWs expressed high acceptance of text-message reminders and appreciated messages as job aids and practical reference material for their day-to-day work. However, HWs said that health systems barriers, including very high outpatient workload, commodity stock-outs, and lack of supportive supervision and financial incentives demotivated them, limited their ability to act on messages and therefore adherence to case management guidelines. Drug dispensers were more likely than clinicians to report usage of text-message reminders. Despite these challenges, nearly all HWs expressed a desire for a longer duration of the SMS intervention. CONCLUSIONS: Text-message reminders to HWs can provide a platform to improve understanding of treatment guidelines and case management decision-making skills, but might not improve actual adherence to guidelines. More interaction, for example through targeted supervision or two-way technology communication, might be an essential intervention component to help address structural barriers and facilitate improved clinical practice.

Practical recommendations for the evaluation of improvement initiatives.

A lack of clear guidance for funders, evaluators and improvers on what to include in evaluation proposals can lead to evaluation designs that do not answer the questions stakeholders want to know. These evaluation designs may not match the iterative nature of improvement and may be imposed onto an initiative in a way that is impractical from the perspective of improvers and the communities with whom they work. Consequently, the results of evaluations are often controversial, and attribution remains poorly understood. Improvement initiatives are iterative, adaptive and context-specific. Evaluation approaches and designs must align with these features, specifically in their ability to consider complexity, to evolve as the initiative adapts over time and to understand the interaction with local context. Improvement initiatives often identify broadly defined change concepts and provide tools for care teams to tailor these in more detail to local conditions. Correspondingly, recommendations for evaluation are best provided as broad guidance, to be tailored to the specifics of the initiative. In this paper, we provide practical guidance and recommendations that funders and evaluators can use when developing an evaluation plan for improvement initiatives that seeks to: identify the questions stakeholders want to address; develop the initial program theory of the initiative; identify high-priority areas to measure progress over time; describe the context the initiative will be applied within; and identify experimental or observational designs that will address attribution.

Health worker adherence to malaria treatment guidelines at outpatient health facilities in southern Malawi following implementation of universal access to diagnostic testing.

BACKGROUND: Appropriate diagnosis and treatment are essential for reducing malaria mortality. A cross-sectional outpatient health facility (HF) survey was conducted in southern Malawi from January to March 2015 to determine appropriate malaria testing and treatment practices four years after implementation of a policy requiring diagnostic confirmation before treatment. METHODS: Enrolled patients were interviewed, examined and had their health booklet reviewed. Health workers (HWs) were asked about training, supervision and access to the 2013 national malaria treatment guidelines. HFs were assessed for malaria diagnostic and treatment capacity. Weighted descriptive analyses and logistic regression of patient, HW and HF characteristics related to testing and treatment were performed. RESULTS: An evaluation of 105 HFs, and interviews of 150 HWs and 2342 patients was completed. Of 1427 suspect uncomplicated malaria patients seen at HFs with testing available, 1072 (75.7%) were tested, and 547 (53.2%) tested positive. Testing was more likely if patients spontaneously reported fever (odds ratio (OR) 2.6; 95% confidence interval (CI) 1.7-4.0), headache (OR 1.5; 95% CI 1.1-2.1) or vomiting (OR 2.0; 95% CI 1.0-4.0) to HWs and less likely if they reported skin problems (OR 0.4; 95% CI 0.2-0.6). Altogether, 511 (92.7%) confirmed cases and 98 (60.3%) of 178 presumed uncomplicated malaria patients (at HFs without testing) were appropriately treated, while 500 (96.6%) of 525 patients with negative tests did not receive anti-malarials. Only eight (5.7%) suspect severe malaria patients received appropriate pre-referral treatment. Appropriate treatment was more likely for presumed uncomplicated malaria patients (at HFs without testing) with elevated temperature (OR 1.5/1 °C increase; 95% CI 1.1-1.9), who reported fever to HWs (OR 5.7; 95% CI 1.9-17.6), were seen by HWs with additional supervision visits in the previous 6 months (OR 1.2/additional visit; 95% CI 1.0-1.4), or were seen by older HWs (OR 1.1/year of age; 95% CI 1.0-1.1). CONCLUSIONS: Correct testing and treatment practices were reasonably good for uncomplicated malaria when testing was available. Pre-referral treatment for suspect severe malaria was unacceptably rare. Encouraging HWs to elicit and appropriately respond to patient symptoms may improve practices.

Predictors of health worker performance after Integrated Management of Childhood Illness training in Benin: a cohort study.

BACKGROUND: Correct treatment of potentially life-threatening illnesses (PLTIs) in children under 5 years, such as malaria, pneumonia, and diarrhea, can substantially reduce mortality. The Integrated Management of Childhood Illness (IMCI) strategy has been shown to improve treatment of child illnesses, but multiple studies have shown that gaps in health worker performance remain after training. To better understand factors related to health worker performance, we analyzed 9,330 patient consultations in Benin from 2001-2002, after training one of the first cohorts of 32 health workers in IMCI. METHODS: With data abstracted from patient registers specially designed for IMCI-trained health workers, we examined associations between health facility-, health worker-, and patient-level factors and 10 case-management outcomes for PLTIs. RESULTS: Altogether, 63.6% of children received treatment for all their PLTIs in accordance with IMCI guidelines, and 77.8% received life-saving treatment (i.e., clinically effective treatment, even if not exactly in accordance with IMCI guidelines). Performance of individual health workers varied greatly, from 15-88% of patients treated correctly, on average. Multivariate regression analyses identified several factors that might have influenced case-management quality, many outside a manager's direct control. Younger health workers significantly outperformed older ones, and infants received better care than older children. Children with danger signs, those with more complex illnesses, and those with anemia received worse care. Health worker supervision was associated with improved performance for some outcomes. CONCLUSIONS: A variety of factors, some outside the direct control of program managers, can influence health worker practices. An understanding of these influences can help inform the development of strategies to improve performance.

The effect of performance indicator category on estimates of intervention effectiveness.

BACKGROUND: A challenge for systematic reviews on improving health worker performance is that included studies often use different performance indicators, and the validity of comparing interventions with different indicators is unclear. One potential solution is to adjust comparisons by indicator category, with categories based on steps of the case-management process that can be easily recognized (assessment of symptoms, treatment etc.) and that might require different levels of effort to bring about improvements. However, this approach would only be useful if intervention effect sizes varied by indicator category. To explore this approach, studies were analyzed that evaluated the Integrated Management of Childhood Illness (IMCI) strategy. METHODS: Performance indicators were grouped into four categories: patient assessment, diagnosis, treatment and counseling. An effect size of IMCI was calculated for each indicator. Linear regression modeling was used to test for differences among the mean effect sizes of the indicator categories. RESULTS: Six studies were included, with data from 3136 ill child consultations. Mean effect sizes for 63 assessment indicators, 12 diagnosis indicators, 31 treatment indicators and 34 counseling indicators were 50.9 percentage-points (%-points), 44.7, 36.5 and 46.6%-points, respectively. After adjusting for baseline indicator value, compared with the assessment mean effect size, the diagnosis mean was 7.3%-points lower (P = 0.23), the treatment mean was 15.2%-points lower (P = 0.0004) and the counseling mean was 12.9%-points lower (P = 0.0027). CONCLUSION: Adjusting the results of systematic reviews for indicator category and baseline indicator value might be useful for improving the validity of intervention comparisons.

Surveillance as a Core Intervention to Strengthen Malaria Control Programs in Moderate to High Transmission Settings.

New tools are needed for malaria control, and recent improvements in malaria surveillance have opened the possibility of transforming surveillance into a core intervention. Implementing this strategy can be challenging in moderate to high transmission settings. However, there is a wealth of practical experience among national malaria control programs and partners working to improve and use malaria surveillance data to guide programming. Granular and timely data are critical to understanding geographic heterogeneity, appropriately defining and targeting interventions packages, and enabling timely decision-making at the operational level. Resources to be targeted based on surveillance data include vector control, case management commodities, outbreak responses, quality improvement interventions, and human resources, including community health workers, as they contribute to a more refined granularity of the surveillance system. Effectively transforming malaria surveillance into a core intervention will require strong global and national leadership, empowerment of subnational and local leaders, collaboration among development partners, and global coordination. Ensuring that national health systems include community health work can contribute to a successful transformation. It will require a strong supply chain to ensure that all suspected cases can be diagnosed and data reporting tools including appropriate electronic devices to provide timely data. Regular data quality audits, decentralized implementation, supportive supervision, data-informed decision-making processes, and harnessing technology for data analysis and visualization are needed to improve the capacity for data-driven decision-making at all levels. Finally, resources must be available to respond programmatically to these decisions.

The effect of mobile phone text-message reminders on Kenyan health workers' adherence to malaria treatment guidelines: a cluster randomised trial.

BACKGROUND: Health workers' malaria case-management practices often differ from national guidelines. We assessed whether text-message reminders sent to health workers' mobile phones could improve and maintain their adherence to treatment guidelines for outpatient paediatric malaria in Kenya. METHODS: From March 6, 2009, to May 31, 2010, we did a cluster-randomised controlled trial at 107 rural health facilities in 11 districts in coastal and western Kenya. With a computer-generated sequence, health facilities were randomly allocated to either the intervention group, in which all health workers received text messages on their personal mobile phones on malaria case-management for 6 months, or the control group, in which health workers did not receive any text messages. Health workers were not masked to the intervention, although patients were unaware of whether they were in an intervention or control facility. The primary outcome was correct management with artemether-lumefantrine, defined as a dichotomous composite indicator of treatment, dispensing, and counselling tasks concordant with Kenyan national guidelines. The primary analysis was by intention to treat. The trial is registered with Current Controlled Trials, ISRCTN72328636. FINDINGS: 119 health workers received the intervention. Case-management practices were assessed for 2269 children who needed treatment (1157 in the intervention group and 1112 in the control group). Intention-to-treat analysis showed that correct artemether-lumefantrine management improved by 23·7 percentage-points (95% CI 7·6-40·0; p=0·004) immediately after intervention and by 24·5 percentage-points (8·1-41·0; p=0·003) 6 months later. INTERPRETATION: In resource-limited settings, malaria control programmes should consider use of text messaging to improve health workers' case-management practices. FUNDING: The Wellcome Trust.

Trends in health worker performance after implementing the Integrated Management of Childhood Illness strategy in Benin.

OBJECTIVE: Training health workers to use Integrated Management of Childhood Illness (IMCI) guidelines can improve care for ill children in outpatient settings in developing countries. However, even after IMCI training, important performance gaps exist. One potential reason is that the effect of training can rapidly wane. Our aim was to determine if the performance of IMCI-trained health workers deteriorated over 3 years. METHODS: We studied two departments in Benin. First, we performed a record review of 32 IMCI-trained health workers during the first year of IMCI implementation (2001-2002). Second, we analysed data from cross-sectional health facility surveys from 2001 to 2004 that represented the entire study area. Primary outcomes were the proportion of children under 5 years old with potentially life-threatening illnesses who received either recommended or adequate treatment, and among all children, an index of overall guideline adherence. Secondary outcomes reflected the treatment of individual diseases. Outcomes were calculated monthly, and time trends were evaluated with regression modelling. RESULTS: The record review included 9393 consultations, and the surveys included 411 consultations performed by 105 health workers. For both data sources, performance trends were essentially flat for nearly all outcomes. Absolute levels of performance revealed substantial performance gaps. CONCLUSIONS: We found no evidence that performance declined over 3 years after IMCI training. However, important performance gaps found immediately after IMCI training persisted and should be addressed.

How does the effectiveness of strategies to improve healthcare provider practices in low-income and middle-income countries change after implementation? Secondary analysis of a systematic review.

BACKGROUND: A recent systematic review evaluated the effectiveness of strategies to improve healthcare provider (HCP) performance in low-income and middle-income countries. The review identified strategies with varying effects, including in-service training, supervision and group problem-solving. However, whether their effectiveness changed over time remained unclear. In particular, understanding whether effects decay over time is crucial to improve sustainability. METHODS: We conducted a secondary analysis of data from the aforementioned review to explore associations between time and effectiveness. We calculated effect sizes (defined as percentage-point (%-point) changes) for HCP practice outcomes (eg, percentage of patients correctly treated) at each follow-up time point after the strategy was implemented. We estimated the association between time and effectiveness using random-intercept linear regression models with time-specific effect sizes clustered within studies and adjusted for baseline performance. RESULTS: The primary analysis included 37 studies, and a sensitivity analysis included 77 additional studies. For training, every additional month of follow-up was associated with a 0.19 %-point decrease in effectiveness (95% CI: -0.36 to -0.03). For training combined with supervision, every additional month was associated with a 0.40 %-point decrease in effectiveness (95% CI: -0.68 to -0.12). Time trend results for supervision were inconclusive. For group problem-solving alone, time was positively associated with effectiveness, with a 0.50 %-point increase in effect per month (95% CI: 0.37 to 0.64). Group problem-solving combined with training was associated with large improvements, and its effect was not associated with time. CONCLUSIONS: Time trends in the effectiveness of different strategies to improve HCP practices vary among strategies. Programmes relying solely on in-service training might need periodical refresher training or, better still, consider combining training with group problem-solving. Although more high-quality research is needed, these results, which are important for decision-makers as they choose which strategies to use, underscore the utility of studies with multiple post-implementation measurements so sustainability of the impact on HCP practices can be assessed.

Impact of a malaria-control project in Benin that included the integrated management of childhood illness strategy.

OBJECTIVES: To estimate the impact of the Integrated Management of Childhood Illness (IMCI) strategy on early-childhood mortality, we evaluated a malaria-control project in Benin that implemented IMCI and promoted insecticide-treated nets (ITNs). METHODS: We conducted a before-and-after intervention study that included a nonrandomized comparison group. We used the preceding birth technique to measure early-childhood mortality (risk of dying before age 30 months), and we used health facility surveys and household surveys to measure process indicators. RESULTS: Most process indicators improved in the area covered by the intervention. Notably, because ITNs were also promoted in the comparison area children's ITN use increased by about 20 percentage points in both areas. Regarding early-childhood mortality, the trend from baseline (1999-2001) to follow-up (2002-2004) for the intervention area (13.0% decrease; P < .001) was 14.1% (P < .001) lower than was the trend for the comparison area (1.3% increase; P = .46). CONCLUSIONS: Mortality decreased in the intervention area after IMCI and ITN promotion. ITN use increased similarly in both study areas, so the mortality impact of ITNs in the 2 areas might have canceled each other out. Thus, the mortality reduction could have been primarily attributable to IMCI's effect on health care quality and care-seeking.

The effectiveness of training strategies to improve healthcare provider practices in low-income and middle-income countries.

INTRODUCTION: In low/middle-income countries (LMICs), training is often used to improve healthcare provider (HCP) performance. However, important questions remain about how well training works and the best ways to design training strategies. The objective of this study is to characterise the effectiveness of training strategies to improve HCP practices in LMICs and identify attributes associated with training effectiveness. METHODS: We performed a secondary analysis of data from a systematic review on improving HCP performance. The review included controlled trials and interrupted time series, and outcomes measuring HCP practices (eg, percentage of patients correctly treated). Distributions of effect sizes (defined as percentage-point (%-point) changes) were described for each training strategy. To identify effective training attributes, we examined studies that directly compared training approaches and performed random-effects linear regression modelling. RESULTS: We analysed data from 199 studies from 51 countries. For outcomes expressed as percentages, educational outreach visits (median effect size when compared with controls: 9.9 %-points; IQR: 4.3-20.6) tended to be somewhat more effective than in-service training (median: 7.3 %-points; IQR: 3.6-17.4), which seemed more effective than peer-to-peer training (4.0 %-points) and self-study (by 2.0-9.3 %-points). Mean effectiveness was greater (by 6.0-10.4 %-points) for training that incorporated clinical practice and training at HCPs' work site. Attributes with little or no effect were: training with computers, interactive methods or over multiple sessions; training duration; number of educational methods; distance training; trainers with pedagogical training and topic complexity. For lay HCPs, in-service training had no measurable effect. Evidence quality for all findings was low. CONCLUSIONS: Although additional research is needed, by characterising the effectiveness of training strategies and identifying attributes of effective training, decision-makers in LMICs can improve how these strategies are selected and implemented.

A multifaceted intervention to improve health worker adherence to integrated management of childhood illness guidelines in Benin.

OBJECTIVES: We evaluated an intervention to support health workers after training in Integrated Management of Childhood Illness (IMCI), a strategy that can improve outcomes for children in developing countries by encouraging workers' use of evidence-based guidelines for managing the leading causes of child mortality. METHODS: We conducted a randomized trial in Benin. We administered a survey in 1999 to assess health care quality before IMCI training. Health workers then received training plus either study supports (job aids, nonfinancial incentives, and supervision of workers and supervisors) or usual supports. Follow-up surveys conducted in 2001 to 2004 assessed recommended treatment, recommended or adequate treatment, and an index of overall guideline adherence. RESULTS: We analyzed 1244 consultations. Performance improved in both intervention and control groups, with no significant differences between groups. However, training proceeded slowly, and low-quality care from health workers without IMCI training diluted intervention effects. Per-protocol analyses revealed that workers with IMCI training plus study supports provided better care than did those with training plus usual supports (27.3 percentage-point difference for recommended treatment; P < .05), and both groups outperformed untrained workers. CONCLUSIONS: IMCI training was useful but insufficient. Relatively inexpensive supports can lead to additional improvements.

Caution is required when using health facility-based data to evaluate the health impact of malaria control efforts in Africa.

The global health community is interested in the health impact of the billions of dollars invested to fight malaria in Africa. A recent publication used trends in malaria cases and deaths based on health facility records to evaluate the impact of malaria control efforts in Rwanda and Ethiopia. Although the authors demonstrate the use of facility-based data to estimate the impact of malaria control efforts, they also illustrate several pitfalls of such analyses that should be avoided, minimized, or actively acknowledged. A critique of this analysis is presented because many country programmes and donors are interested in evaluating programmatic impact with facility-based data. Key concerns related to: 1) clarifying the objective of the analysis; 2) data validity; 3) data representativeness; 4) the exploration of trends in factors that could influence malaria rates and thus confound the relationship between intervention scale-up and the observed changes in malaria outcomes; 5) the analytic approaches, including small numbers of patient outcomes, selective reporting of results, and choice of statistical and modeling methods; and 6) internal inconsistency on the strength and interpretation of the data. In conclusion, evaluations of malaria burden reduction using facility-based data could be very helpful, but those data should be collected, analysed, and interpreted with care, transparency, and a full recognition of their limitations.

Quality of malaria case management at outpatient health facilities in Angola.

BACKGROUND: Angola's malaria case-management policy recommends treatment with artemether-lumefantrine (AL). In 2006, AL implementation began in Huambo Province, which involved training health workers (HWs), supervision, delivering AL to health facilities, and improving malaria testing with microscopy and rapid diagnostic tests (RDTs). Implementation was complicated by a policy that was sometimes ambiguous. METHODS: Fourteen months after implementation began, a cross-sectional survey was conducted in 33 outpatient facilities in Huambo Province to assess their readiness to manage malaria and the quality of malaria case-management for patients of all ages. Consultations were observed, patients were interviewed and re-examined, and HWs were interviewed. RESULTS: Ninety-three HWs and 177 consultations were evaluated, although many sampled consultations were missed. All facilities had AL in-stock and at least one HW trained to use AL and RDTs. However, anti-malarial stock-outs in the previous three months were common, clinical supervision was infrequent, and HWs had important knowledge gaps. Except for fever history, clinical assessments were often incomplete. Although testing was recommended for all patients with suspected malaria, only 30.7% of such patients were tested. Correct testing was significantly associated with caseloads < 25 patients/day (odds ratio: 18.4; p < 0.0001) and elevated patient temperature (odds ratio: 2.5 per 1 degrees C increase; p = 0.007). Testing was more common among AL-trained HWs, but the association was borderline significant (p = 0.072). When the malaria test was negative, HWs often diagnosed patients with malaria (57.8%) and prescribed anti-malarials (60.0%). Sixty-six percent of malaria-related diagnoses were correct, 20.1% were minor errors, and 13.9% were major (potentially life-threatening) errors. Only 49.0% of malaria treatments were correct, 5.4% were minor errors, and 45.6% were major errors. HWs almost always dosed AL correctly and gave accurate dosing instructions to patients; however, other aspects of counseling needed improvement. CONCLUSION: By late-2007, substantial progress had been made to implement the malaria case-management policy in a setting with weak infrastructure. However, policy ambiguities, under-use of malaria testing, and distrust of negative test results led to many incorrect malaria diagnoses and treatments. In 2009, Angola published a policy that clarified many issues. As problems identified in this survey are not unique to Angola, better strategies for improving HW performance are urgently needed.

Improving pneumonia case-management in Benin: a randomized trial of a multi-faceted intervention to support health worker adherence to Integrated Management of Childhood Illness guidelines.

BACKGROUND: Pneumonia is a leading cause of death among children under five years of age. The Integrated Management of Childhood Illness strategy can improve the quality of care for pneumonia and other common illnesses in developing countries, but adherence to these guidelines could be improved. We evaluated an intervention in Benin to support health worker adherence to the guidelines after training, focusing on pneumonia case management. METHODS: We conducted a randomized trial. After a health facility survey in 1999 to assess health care quality before Integrated Management of Childhood Illness training, health workers received training plus either study supports (job aids, non-financial incentives and supervision of workers and supervisors) or "usual" supports. Follow-up surveys were conducted in 2001, 2002 and 2004. Outcomes were indicators of health care quality for Integrated Management-defined pneumonia. Further analyses included a graphical pathway analysis and multivariable logistic regression modelling to identify factors influencing case-management quality. RESULTS: We observed 301 consultations of children with non-severe pneumonia that were performed by 128 health workers in 88 public and private health facilities. Although outcomes improved in both intervention and control groups, we found no statistically significant difference between groups. However, training proceeded slowly, and low-quality care from untrained health workers diluted intervention effects. Per-protocol analyses suggested that health workers with training plus study supports performed better than those with training plus usual supports (20.4 and 19.2 percentage-point improvements for recommended treatment [p=0.08] and "recommended or adequate" treatment [p=0.01], respectively). Both groups tended to perform better than untrained health workers. Analyses of treatment errors revealed that incomplete assessment and difficulties processing clinical findings led to missed pneumonia diagnoses, and missed diagnoses led to inadequate treatment. Increased supervision frequency was associated with better care (odds ratio for recommended treatment=2.1 [95% confidence interval: 1.13.9] per additional supervisory visit). CONCLUSION: Integrated Management of Childhood Illness training was useful, but insufficient, to achieve high-quality pneumonia case management. Our study supports led to additional improvements, although large gaps in performance still remained. A simple graphical pathway analysis can identify specific, common errors that health workers make in the case-management process; this information could be used to target quality improvement activities, such as supervision (ClinicalTrials.gov number NCT00510679).

The effectiveness of the quality improvement collaborative strategy in low- and middle-income countries: A systematic review and meta-analysis.

BACKGROUND: Quality improvement collaboratives (QICs) have been used to improve health care for decades. Evidence on QIC effectiveness has been reported, but systematic reviews to date have little information from low- and middle-income countries (LMICs). OBJECTIVE: To assess the effectiveness of QICs in LMICs. METHODS: We conducted a systematic review following Cochrane methods, the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) approach for quality of evidence grading, and the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) statement for reporting. We searched published and unpublished studies between 1969 and March 2019 from LMICs. We included papers that compared usual practice with QICs alone or combined with other interventions. Pairs of reviewers independently selected and assessed the risk of bias and extracted data of included studies. To estimate strategy effectiveness from a single study comparison, we used the median effect size (MES) in the comparison for outcomes in the same outcome group. The primary analysis evaluated each strategy group with a weighted median and interquartile range (IQR) of MES values. In secondary analyses, standard random-effects meta-analysis was used to estimate the weighted mean MES and 95% confidence interval (CI) of the mean MES of each strategy group. This review is registered with PROSPERO (International Prospective Register of Systematic Reviews): CRD42017078108. RESULTS: Twenty-nine studies were included; most (21/29, 72.4%) were interrupted time series studies. Evidence quality was generally low to very low. Among studies involving health facility-based health care providers (HCPs), for "QIC only", effectiveness varied widely across outcome groups and tended to have little effect for patient health outcomes (median MES less than 2 percentage points for percentage and continuous outcomes). For "QIC plus training", effectiveness might be very high for patient health outcomes (for continuous outcomes, median MES 111.6 percentage points, range: 96.0 to 127.1) and HCP practice outcomes (median MES 52.4 to 63.4 percentage points for continuous and percentage outcomes, respectively). The only study of lay HCPs, which used "QIC plus training", showed no effect on patient care-seeking behaviors (MES -0.9 percentage points), moderate effects on non-care-seeking patient behaviors (MES 18.7 percentage points), and very large effects on HCP practice outcomes (MES 50.4 percentage points). CONCLUSIONS: The effectiveness of QICs varied considerably in LMICs. QICs combined with other invention components, such as training, tended to be more effective than QICs alone. The low evidence quality and large effect sizes for QIC plus training justify additional high-quality studies assessing this approach in LMICs.