Practical recommendations for the evaluation of improvement initiatives.

A lack of clear guidance for funders, evaluators and improvers on what to include in evaluation proposals can lead to evaluation designs that do not answer the questions stakeholders want to know. These evaluation designs may not match the iterative nature of improvement and may be imposed onto an initiative in a way that is impractical from the perspective of improvers and the communities with whom they work. Consequently, the results of evaluations are often controversial, and attribution remains poorly understood. Improvement initiatives are iterative, adaptive and context-specific. Evaluation approaches and designs must align with these features, specifically in their ability to consider complexity, to evolve as the initiative adapts over time and to understand the interaction with local context. Improvement initiatives often identify broadly defined change concepts and provide tools for care teams to tailor these in more detail to local conditions. Correspondingly, recommendations for evaluation are best provided as broad guidance, to be tailored to the specifics of the initiative. In this paper, we provide practical guidance and recommendations that funders and evaluators can use when developing an evaluation plan for improvement initiatives that seeks to: identify the questions stakeholders want to address; develop the initial program theory of the initiative; identify high-priority areas to measure progress over time; describe the context the initiative will be applied within; and identify experimental or observational designs that will address attribution.

Effect of computer-assisted interviewing on self-reported sexual behavior data in a microbicide clinical trial.

In a microbicide safety and effectiveness trial (HPTN 035) in Malawi, 585 women completed the same questionnaire through a face-to-face interview (FTFI) and an audio computer-assisted self-interview (ACASI). Concordance between FTFI and ACASI responses ranged from 72.0 % for frequency of sex in the past week to 95.2 % for anal intercourse (AI) in the past 3 months. Reported gel and condom use at last sex act were marginally lower with ACASI than FTFI (73.5 % vs. 77.2 %, p = 0.11 and 60.9 % vs. 65.5 %, p = 0.05, respectively). More women reported AI with ACASI than FTFI (5.0 % vs. 0.2 %, p < 0.001). Analyses of consistency of responses within ACASI revealed that 15.0 % of participants in the condom-only arm and 28.7 % in the gel arm provided at least one discrepant answer regarding total sex acts and sex acts where condom and gel were used (19.2 % reported one inconsistent answer, 8.1 % reported two inconsistent answers, and 1.4 % reported three inconsistent answers). While ACASI may provide more accurate assessments of sensitive behaviors in HIV prevention trials, it also results in a high level of internally inconsistent responses.

Evaluation of the rapid, multi-country, parallel process, multi-tasking approach to startup of short-term technical assistance to improve service delivery in newborn and child health in the context of USAID's Zika response in four Eastern and Southern Caribbean countries.

Background: In 2018, the USAID Applying Science to Strengthen and Improve Systems (ASSIST) Project started a new partnership with four Eastern and Southern Caribbean countries impacted by the Zika virus: Antigua and Barbuda, Dominica, St. Kitts and Nevis, and St. Vincent and the Grenadines.  The goal of the project was to provide short-term technical assistance (STTA) to strengthen the health systems' capacity to detect newborns and young children potentially affected by Zika and to address their health needs.  To meet these objectives, ASSIST developed an innovative approach based on its existing model for service delivery improvement. This novel approach is known as Rapid, Multi-country, Parallel Process, Multi-tasking Approach for a Project Startup (RMPP-MAPS).  An evaluation was conducted to document the STTA startup activities, to identify enabling and constraining factors, and to capture lessons learned. Methods: An external consultant conducted remote in-depth interviews with individuals involved in the startup using semi-structured interview guides and retrieved data from the review of project documents. Results: Using RMPP-MAPS, the ASSIST Project successfully implemented the startup for complex STTA in four countries within less than four months, spanning mid-May to early September 2018. Project milestones included achieving buy-in from stakeholders, co-developing the technical scope and materials, and rapidly executing critical operational functions.  Dedicated project teams, country leaderships, and local champions were essential to overcoming the main challenges, which included a condensed timeframe, lack of in-country offices, and country-level factors such as a shortage of health care workers and a weak health infrastructure.  Conclusions: The RMPP-MAPS is a feasible and resource-efficient mechanism of interest to implementers, donors, and low and middle-income countries facing temporal and financial limitations to rapidly addressing public health priorities.

Reducing Mother-to-Child Transmission of HIV Using Quality Improvement Approaches.

Over half of mother-to-child HIV transmission (MTCT) occurs postdelivery. Keeping mother-infant pairs in care remains challenging. Health workers in 3 countries used quality improvement (QI) approaches to improve data systems, mother-infant retention, and facility-based care delivery. The number and proportion of infants with known HIV status at time of discharge from early infant diagnosis programs increased in Tanzania and Uganda. We analyzed data using statistical process control charts. Mother-to-child HIV transmission did not decrease in 15 Kenyan sites, decreased from 12.7% to 3.8% in 28 Tanzanian sites, and decreased from 17.2% to 1.5% in 10 Ugandan sites with baseline data. This improvement is likely due to the combination of option B+, service delivery improvements, and retention through QI approaches. Reaching the global MTCT elimination target and maximizing infant survival will require health systems to support mother-infant pairs to remain in care and support health workers to deliver care. Quality improvement approaches can support these changes.

Assessment of tuberculosis treatment completion in an ethnically diverse population using two data sources. Implications for treatment interventions.

Many adolescents who are prescribed therapy for tuberculosis (TB) infection fail to complete it. This article presents the results of a demographic and epidemiological assessment of TB treatment completion in adolescent populations using (a) surveillance data from the Los Angeles Health County Department and (b) a prospective/retrospective medical chart review from targeted clinics. Patients who did not complete the 6-month recommended medical treatment for latent tuberculosis therapy (LTBI) averaged 13 weeks in care. Younger age (OR = 1.15; 95% CI 1.03-1.29), birth in the United States (OR = 1.92; 95% CI 1.14-3.22, compared to Mexico), and Asian ethnicity were associated with completion of care. In multiple logistic regression analysis, age (OR = .88; 95% CI .78-.98) and Latino ethnicity (OR = .53; 95% CI .29-.95) remained significant predictors of completion of treatment. These findings indicate the need for age-specific educational reinforcement and cultural differentials in completing care for LTBI.

Microbicide acceptability research: recent findings and evolution across phases of product development.

PURPOSE OF REVIEW: Research on the acceptability of microbicides has the potential to inform the microbicide development process by shedding light on desirable product characteristics, issues around product use and potential barriers. The purpose of this review is both to synthesize recent findings that are related to microbicide acceptability, highlight areas of deficiencies, and to point to the new directions in which research is needed. RECENT FINDINGS: Recent studies have assessed acceptability using candidate microbicides in clinical trials, surrogate products, and descriptions of hypothetical products. While most studies have focused on physical characteristics of products, some small studies have investigated participants' short-term experiences with products during sexual intercourse. Overall, as currently measured and in their present formulations, vaginal microbicides have been found to be acceptable to adolescent girls, women, heterosexual men, whereas rectal microbicides were acceptable to men who have sex with men. Few studies have examined acceptability among high-risk HIV-uninfected women. The relationship between acceptability and adherence in trials, and ultimately in real-world settings, remains unknown. SUMMARY: Data are needed on acceptability of microbicides from larger populations, high-risk HIV-uninfected women, as well as long-term acceptability and contextual factors surrounding acceptability. The association between acceptability and long-term adherence in clinical trials or ultimately in the open market has yet to be demonstrated.

Costs and cost-effectiveness of adolescent compliance with treatment for latent tuberculosis infection: results from a randomized trial.

PURPOSE: Assess the costs and cost-effectiveness of an incentive-based tuberculosis (TB) program designed to promote adolescents' compliance with treatment for latent TB infection (LTBI). METHODS: Randomized controlled trial. Adolescents between the ages of 11 and 19 years who were referred to one of two participating clinics after being screened for TB and receiving a positive diagnosis indicating LTBI (n = 794) were assigned to one of four groups: usual care, peer counseling, contingency contracting, and combined peer counseling/contingency contracting. Primary outcome variables were completion of isoniazid preventive therapy (IPT), total treatment costs, and lifetime TB-related costs per quality-adjusted life year (QALY) in each of the four study groups (three treatment, one control). Cost effectiveness was evaluated using a five-stage Markov model and a Monte Carlo simulation with 10,000 trials. RESULTS: Average costs were 199 dollars for usual care (UC), 277 dollars for peer counseling (PC), 326 dollars for contingency contracting (CC), and 341 dollars for PC + CC combined. The differences among these groups were all significant at the p = .001 level. Only the PC + CC group improved the rate of IPT completion (83.8%) relative to usual care (75.9%) (p = .051), with an overall incremental CE ratio of 209 dollars per QALY relative to usual care. CONCLUSION: Incentives combined with peer counseling are a cost-effective strategy for helping adolescents to complete care when combined with peer counseling.

A model HIV/AIDS risk reduction programme in the Philippines: a comprehensive community-based approach through participatory action research.

A 3-year, longitudinal, quasi-experimental study using participatory action research (PAR) was conducted to determine the feasibility and efficiency of an expanded sexually transmitted infection (STI) HIV/AIDS prevention program among diverse high-risk male heterosexual populations in the southern Philippines. A total of 3389 participants ( approximately 200 males from each of 18 study groups) were recruited, and 221 were trained as peer counselors to develop educational materials and reinforce safe sexual practices among their peers. Condom usage (36.10% to 38.70% to 46.31%), attitudes towards condoms (21.67% to 24.55% to 25.15%) and knowledge about HIV/STI transmission (41.87% to 42.19% to 33.31%) increased significantly from baseline to post-test and 6-month follow up, respectively (p < 0.01). Furthermore, the reported STI incidence decreased significantly (7.4% to 4.6% to 2.4%, respectively). Changes differed significantly between the intervention and control group at post-test and follow up (p < 0.01). These findings illustrate the appropriateness of using PAR methodology in promoting and sustaining positive behavior change.