Health impact and cost-effectiveness of expanding routine immunization coverage in India through Intensified Mission Indradhanush.

Many children do not receive a full schedule of childhood vaccines, yet there is limited evidence on the cost-effectiveness of strategies for improving vaccination coverage. Evidence is even scarcer on the cost-effectiveness of strategies for reaching "zero-dose children," who have not received any routine vaccines. We evaluated the cost-effectiveness of periodic intensification of routine immunization (PIRI), a widely applied strategy for increasing vaccination coverage. We focused on Intensified Mission Indradhanush (IMI), a large-scale PIRI intervention implemented in India in 2017-2018. In 40 sampled districts, we measured the incremental economic cost of IMI using primary data, and used controlled interrupted time-series regression to estimate incremental vaccination doses delivered. We estimated deaths and disability-adjusted life years (DALYs) averted using the Lives Saved Tool and reported cost-effectiveness from immunization program and societal perspectives. We found that, in sampled districts, IMI had an estimated incremental cost of 2021US$13.7 (95% uncertainty interval: 10.6 to 17.4) million from an immunization program perspective and increased vaccine delivery by an estimated 2.2 (-0.5 to 4.8) million doses over a 12-month period, averting an estimated 1,413 (-350 to 3,129) deaths. The incremental cost from a program perspective was $6.21 per dose ($2.80 to dominated), $82.99 per zero-dose child reached ($39.85 to dominated), $327.63 ($147.65 to dominated) per DALY averted, $360.72 ($162.56 to dominated) per life-year saved, and $9,701.35 ($4,372.01 to dominated) per under-five death averted. At a cost-effectiveness threshold of 1x per-capita GDP per DALY averted, IMI was estimated to be cost-effective with 90% probability. This evidence suggests IMI was both impactful and cost-effective for improving vaccination coverage, though there is a high degree of uncertainty in the results. As vaccination programs expand coverage, unit costs may increase due to the higher costs of reaching currently unvaccinated children.

Health system inequities in Lao People's Democratic Republic: Evidence from a nationally representative phone survey.

BACKGROUND: Despite substantial economic growth in Lao People's Democratic Republic (PDR) over the past 20 years, high levels of income inequality and poverty persist and have likely been exacerbated by the COVID-19 pandemic. In this article, we use novel survey data to assess the extent to which socioeconomic status is associated with access to quality care in Lao PDR. METHODS: We utilised data from the Lao People's Voice Survey (PVS), which was designed to measure health system performance from the perspective of the population. The survey was conducted between May and August 2022. Primary outcomes of interest were having a usual source of healthcare, using a government health centre (rather than a hospital or private clinic) as the usual source for care, receiving preventive health services, experiencing unmet healthcare needs, the quality of recent healthcare visit, and confidence in accessing and affording healthcare when needed. Poverty was measured using household asset ownership. We used logit models to assess the associations between poverty and health system performance measures, and additionally assessed differences between these associations in urban vs. rural areas by interacting urban residence with poverty. RESULTS: Poverty was negatively associated with having a regular provider for care (adjusted odds ratio (aOR) 0.45, 95% CI 0.26-0.78), receiving preventive health services (aOR 0.54, 95% CI 0.37-0.80), and confidence in the ability to receive care (aOR 0.50, 95% CI 0.34-0.72) and afford care (aOR 0.50, 95% CI 0.34-0.73) when needed. Poverty was positively associated with using government managed health centres as a usual source or for care (aOR 2.16, 95% CI 1.35-3.48). Poverty was not significantly associated with user experience or perceived quality of care in the last visit to the health facility. No differences in the associations between poverty and access to quality care were found between rural and urban settings. CONCLUSIONS: The results presented in this article suggest socioeconomic disparities in health care access in Lao PDR despite major national efforts to provide universal access to care. Universal health care policies may not be reaching the poor and additional targeted efforts may be needed to meet their healthcare needs.

Health system utilization and perceived quality among adults in Lao PDR: evidence from a nationally representative phone survey.

BACKGROUND: The World Health Organization (WHO) defines quality health services as being effective, safe, people-centered, timely, equitable, integrated and efficient. It is critical to understand people's perspectives and care experiences to measure progress against these goals. However, many low- and middle-income governments do not routinely collect such information. In this study, we aim to measure health systems performance from the perspective of the adult population of users and non-users in Lao PDR. METHODS: Using the People's Voice Survey (PVS), a novel phone-based survey designed to integrate people's voices into primary care performance measurement, we conducted a cross-sectional survey of the general adult (18+) population in Lao PDR in 2022. We analyzed health care utilization patterns, user-reported quality of care, and coverage of key preventive health services. Data from the most recent MICS survey was used to create sampling weights generating nationally representative estimates. RESULTS: A total of 2007 adults completed interviews in approximately 3.5 months. About two thirds (65%) of respondents reported visiting a health facility in the past year and, of these, the majority (61%) visited a hospital as opposed to a health center or clinic. Among those that recently visited health facilities, 28% rated their experience as "poor" or "fair". 16% had unmet need for care and 12% reported discrimination during treatment in the past year. 12% of women over 50 years old reported receiving a mammogram and 59% of adults reported receiving blood pressure screening in the previous year. CONCLUSIONS: The study presents data from the first nationally representative survey in Lao PDR to measure health system performance. The results indicate that, despite progress towards universal coverage of health insurance in Lao PDR, significant gaps remain, particularly with respect to bypassing of primary care facilities, significant unmet need for care, experiences of discrimination, and overall low perceptions of quality of care.

The path to universal health coverage in five African and Asian countries: examining the association between insurance status and health-care use.

Despite major efforts to achieve universal health coverage (UHC), progress has lagged in many African and Asian countries. A key strategy pursued by many countries is the use of health insurance to increase access and affordability. However, evidence on insurance coverage and on the association between insurance and UHC is mixed. We analysed nationally representative cross-sectional data collected between 2022 and 2023 in Ethiopia, Kenya, South Africa, India, and Laos. We described public and private insurance coverage by sociodemographic factors and used logistic regression to examine the associations between insurance status and seven health-care use outcomes. Health insurance coverage ranged from 25% in India to 100% in Laos. The share of private insurance ranged from 1% in Ethiopia to 13% in South Africa. Relative to the population with private insurance, the uninsured population had reduced odds of health-care use (adjusted odds ratio 0·68, 95% CI 0·50-0·94), cardiovascular examinations (0·63, 0·47-0·85), eye and dental examinations (0·54, 0·42-0·70), and ability to get or afford care (0·64, 0·48-0·86); private insurance was not associated with unmet need, mental health care, and cancer screening. Relative to private insurance, public insurance was associated with reduced odds of health-care use (0·60, 0·43-0·82), mental health care (0·50, 0·31-0·80), cardiovascular examinations (0·62, 0·46-0·84), and eye and dental examinations (0·50, 0·38-0·65). Results were highly heterogeneous across countries. Public health insurance appears to be only weakly associated with access to health services in the countries studied. Further research is needed to improve understanding of these associations and to identify the most effective financing strategies to achieve UHC.

Measuring people's views on health system performance: Design and development of the People's Voice Survey.

Todd Lewis and co-authors discuss development and use of the People's Voice Survey for health system assessment.

Impacts of performance-based financing on health system performance: evidence from the Democratic Republic of Congo.

BACKGROUND: Health systems' weakness remains one of the primary obstacles towards achieving universal access to quality healthcare in low-income settings. Performance-based financing (PBF) programs have been increasingly used to increase access to quality care in LMICs. However, evidence on the impacts of these programs remains fragmented and inconclusive. We analyze the health system impacts of the PBF program in the Democratic Republic of the Congo (DRC), one of the largest such programs introduced in LMICs to date. METHODS: We used a health systems perspective to analyze the benefits of PBF relative to unconditional financing of health facilities. Fifty-eight health zones in six provinces were randomly assigned to either a control group (28 zones) in which facilities received unconditional transfers or to a PBF program (30 zones) that started at the end of 2016. Follow-up data collection took place in 2021-2022 and included health facility assessments, health worker interviews, direct observations of consultations and deliveries, patient exit interviews, and household surveys. Using multivariate regression models, we estimated the impact of the program on 55 outcomes in seven health system domains: structural quality, technical process quality, non-technical process quality, service fees, facility management, providers' satisfaction, and service coverage. We used random-effects meta-analysis to generate pooled average estimates within each domain. RESULTS: The PBF program improved the structural quality of health facilities by 4 percentage points (ppts) (95% CI 0.01-0.08), technical process quality by 5 ppts (0.03-0.07), and non-technical process by 2 ppts (0-0.04). PBF also increased coverage of priority health services by 3 ppts (0.02-0.04). Improvements were also observed for facility management (9 ppts, 0.04-0.15), service fee policies, and users' satisfaction with service affordability (14 ppts, 0.07-0.20). Service fees and health workers' satisfaction were not affected by the program. CONCLUSIONS: The results suggest that well-designed PBF programs can lead to improvements in most health systems domains relative to comparable unconditional financing. However, the large persisting gaps suggest that additional changes, such as allocating more resources to the health system and reforming the human resources for health management, will be necessary in DRC to achieve the ambitious global universal health coverage and mortality goals. TRIAL REGISTRATION: American Economics Association Trial registry AEARCTR-0002880.

Health care provider decision-making and the quality of maternity care: An analysis of postpartum care in Kenyan hospitals.

Evidence suggests that health care providers' non-adherence to clinical guidelines is widespread and contributes to poor patient outcomes across low- and middle-income countries. Through observations of maternity care in Kenya, we found limited adherence to guideline-recommended active monitoring of patients for signs of postpartum hemorrhage, the leading cause of maternal mortality, despite providers' having the necessary training and equipment. Using survey vignettes conducted with 144 maternity providers, we documented evidence consistent with subjective risk and perceived uncertainty driving providers' decisions to actively monitor patients. Motivated by these findings, we introduced a simple model of providers' decision-making about whether to monitor a patient, which may depend on their perceptions of risk, diagnostic uncertainty, and the value of new information. The model highlights key trade-offs between gathering diagnostic information through active monitoring versus waiting for signs and symptoms of hemorrhage to manifest. Our work provides a template for understanding provider decision-making and could inform interventions to encourage more proactive obstetric care.

Quality of care for postpartum hemorrhage: A direct observation study in referral hospitals in Kenya.

Postpartum hemorrhage (PPH) is the leading cause of maternal mortality in Kenya. The aim of this study was to measure quality and timeliness of care for PPH in a sample of deliveries in referral hospitals in Kenya. We conducted direct observations of 907 vaginal deliveries in three Kenyan hospitals from October 2018 through February 2019, observing the care women received from admission for labor and delivery through hospital discharge. We identified cases of "suspected PPH", defined as cases in which providers indicated suspicion of and/or took an action to manage abnormal bleeding. We measured adherence to World Health Organization and Kenyan guidelines for PPH risk assessment, prevention, identification, and management and the timeliness of care in each domain. The rate of suspected PPH among the observed vaginal deliveries was 9% (95% Confidence Interval: 7% - 11%). Health care providers followed all guidelines for PPH risk assessment in 7% (5% - 10%) of observed deliveries and all guidelines for PPH prevention in 4% (3% - 6%) of observed deliveries. Lowest adherence was observed for taking vital signs and for timely administration of a prophylactic uterotonic. Providers did not follow guidelines for postpartum monitoring in any of the observed deliveries. When suspected PPH occurred, providers performed all recommended actions in 23% (6% - 40%) of cases. Many of the critical actions for suspected PPH were performed in a timely manner, but, in some cases, substantial delays were observed. In conclusion, we found significant gaps in the quality of risk assessment, prevention, identification, and management of PPH after vaginal deliveries in referral hospitals in Kenya. Efforts to reduce maternal morbidity and mortality from PPH should emphasize improvements in the quality of care, with a particular focus on postpartum monitoring and timely emergency response.

Immediate postpartum care in low- and middle-income countries: A gap in healthcare quality research and practice.

The immediate postpartum period carries significant risks for complications such as postpartum hemorrhage and sepsis. Postpartum monitoring, including taking vital signs and monitoring blood loss, is important for the early identification and management of complications, but many women in low- and middle-income countries receive minimal attention in the period following childbirth to facility discharge. The World Health Organization recently released new guidelines on postnatal care, which include recommendations for immediate postpartum monitoring. In light of the new guidelines, this presented an opportune moment to address the gaps in postpartum monitoring in low- and middle-income countries. In this commentary, we bring attention to the importance of immediate postpartum monitoring. We identified opportunities for strengthening this often overlooked aspect of maternity care through improvements in quality measurement and data availability, research into barriers against high-quality care, and innovations in service delivery design.

Examining decentralization and managerial decision making for child immunization program performance in India.

Despite widespread adoption of decentralization reforms, the impact of decentralization on health system attributes, such as access to health services, responsiveness to population health needs, and effectiveness in affecting health outcomes, remains unclear. This study examines how decision space, institutional capacities, and accountability mechanisms of the Intensified Mission Indradhanush (IMI) in India relate to measurable performance of the immunization program. Data on decision space and its related dimensions of institutional capacity and accountability were collected by conducting structured interviews with managers based in 24 districts, 61 blocks, and 279 subcenters. Two measures by which to assess performance were selected: (1) proportion reduction in the DTP3 coverage gap (i.e., effectiveness), and (2) total IMI doses delivered per incremental USD spent on program implementation (i.e., efficiency). Descriptive statistics on decision space, institutional capacity, and accountability for IMI managers were generated. Structural equation models (SEM) were specified to detect any potential associations between decision space dimensions and performance measures. The majority of districts and blocks indicated low levels of decision space. Institutional capacity and accountability were similar across areas. Increases in decision space were associated with less progress towards closing the immunization coverage gap in the IMI context. Initiatives to support health workers and managers based on their specific contextual challenges could further improve outcomes of the program. Similar to previous studies, results revealed strong associations between each of the three decentralization dimensions. Health systems should consider the impact that management structures have on the efficiency and effectiveness of health services delivery. Future research could provide greater evidence for directionality of direct and indirect effects, interaction effects, and/or mediators of relationships.

Health care providers' knowledge of clinical protocols for postpartum hemorrhage care in Kenya: a cross-sectional study.

BACKGROUND: Postpartum hemorrhage (PPH) remains the leading cause of maternal death worldwide despite its often-preventable nature. Understanding health care providers' knowledge of clinical protocols is imperative for improving quality of care and reducing mortality. This is especially pertinent in referral and teaching hospitals that train nursing and medical students and interns in addition to managing emergency and referral cases. METHODS: This study aimed to (1) measure health care providers' knowledge of clinical protocols for risk assessment, prevention, and management of PPH in 3 referral hospitals in Kenya and (2) examine factors associated with providers' knowledge. We developed a knowledge assessment tool based on past studies and clinical guidelines from the World Health Organization and the Kenyan Ministry of Health. We conducted in-person surveys with health care providers in three high-volume maternity facilities in Nairobi and western Kenya from October 2018-February 2019. We measured gaps in knowledge using a summative index and examined factors associated with knowledge (such as age, gender, qualification, experience, in-service training attendance, and a self-reported measure of peer-closeness) using linear regression. RESULTS: We interviewed 172 providers including consultants, medical officers, clinical officers, nurse-midwives, and students. Overall, knowledge was lowest for prevention-related protocols (an average of 0.71 out of 1.00; 95% CI 0.69-0.73) and highest for assessment-related protocols (0.81; 95% CI 0.79-0.83). Average knowledge scores did not differ significantly between qualified providers and students. Finally, we found that being a qualified nurse, having a specialization, being female, having a bachelor's degree and self-reported closer relationships with colleagues were statistically significantly associated with higher knowledge scores. CONCLUSION: We found gaps in knowledge of PPH care clinical protocols in Kenya. There is a clear need for innovations in clinical training to ensure that providers in teaching referral hospitals are prepared to prevent, assess, and manage PPH. It is possible that training interventions focused on learning by doing and teamwork may be beneficial.

Health care seeking in modern urban LMIC settings: evidence from Lusaka, Zambia.

BACKGROUND: In an effort to improve population health, many low- and middle-income countries (LMICs) have expanded access to public primary care facilities and removed user fees for services in these facilities. However, a growing literature suggests that many patients bypass nearby primary care facilities to seek care at more distant or higher-level facilities. Patients in urban areas, a growing segment of the population in LMICs, generally have more options for where to seek care than patients in rural areas. However, evidence on care-seeking trajectories and bypassing patterns in urban areas remains relatively scarce. METHODS: We obtained a complete list of public health facilities and interviewed randomly selected informal sector households across 31 urban areas in Lusaka District, Zambia. All households and facilities listed were geocoded, and care-seeking trajectories mapped across the entire urban area. We analyzed three types of bypassing: i) not using health centers or health posts for primary care; ii) seeking care outside of the residential neighborhood; iii) directly seeking care at teaching hospitals. RESULTS: A total of 620 households were interviewed, linked to 88 health facilities. Among 571 adults who had recently sought non-emergency care, 65% sought care at a hospital. Among 141 children who recently sought care for diarrhea, cough, fever, or fast breathing, 34% sought care at a hospital. 71% of adults bypassed primary care facilities, 26% bypassed health centers and hospitals close to them for more distant facilities, and 8% directly sought care at a teaching hospital. Bypassing was also observed for 59% of children, who were more likely to seek care outside of the formal care sector, with 21% of children treated at drug shops or pharmacies. CONCLUSIONS: The results presented here strongly highlight the complexity of urban health systems. Most adult patients in Lusaka do not use public primary health facilities for non-emergency care, and heavily rely on pharmacies and drug shops for treatment of children. Major efforts will likely be needed if the government wants to instate health centers as the principal primary care access point in this setting.

Measuring user experience of care among caregivers of sick children: Validation and descriptive analysis in a sample of 75 health facilities in rural Tanzania.

OBJECTIVE: User experience is an important aspect of quality of care that is highly valued by patients. However, there are currently no validated tools for measuring user experience among caregivers of sick children in low- and middle-income countries. We aimed to develop and validate a measure of user experience in this population in primary healthcare facilities in rural Tanzania, where major quality improvement efforts to date have not included a large focus on user experience. We then aimed to describe variation in user experience between and within facilities. METHODS: Informed by theory and formative qualitative research, we developed questions to measure user experience across three domains: prompt care, respect, and communication. We then conducted interviewer-administered surveys with caregivers of sick children. Using survey data, we conducted psychometric analyses to inform the development of a composite measure of user experience. Finally, we used multilevel models to describe variation in user experience and examine associations with facility, patient, and caregiver characteristics. RESULTS: Surveys were completed by 1085 caregivers across 75 facilities. In exploratory factor analysis, user experience items did not group according to theoretical domains. We therefore assessed items individually and designed a single 8-item additive measure of user experience. Using this composite measure, and adjusting for differences in case mix across facilities, we found that 69% of variation in user experience was within facilities and 31% was between facilities. Smaller facility size and more caregiver education were positively associated with user experience. CONCLUSIONS: We found that user experience varied significantly across health facilities in rural Tanzania, highlighting opportunities for improvement. Measurement tools are needed to inform efforts to improve user experience and monitor changes over time. The scale developed in this study could serve as a starting point for the measurement of user experience among caregivers in similar settings.

Problems with evidence assessment in COVID-19 health policy impact evaluation: a systematic review of study design and evidence strength.

INTRODUCTION: Assessing the impact of COVID-19 policy is critical for informing future policies. However, there are concerns about the overall strength of COVID-19 impact evaluation studies given the circumstances for evaluation and concerns about the publication environment. METHODS: We included studies that were primarily designed to estimate the quantitative impact of one or more implemented COVID-19 policies on direct SARS-CoV-2 and COVID-19 outcomes. After searching PubMed for peer-reviewed articles published on 26 November 2020 or earlier and screening, all studies were reviewed by three reviewers first independently and then to consensus. The review tool was based on previously developed and released review guidance for COVID-19 policy impact evaluation. RESULTS: After 102 articles were identified as potentially meeting inclusion criteria, we identified 36 published articles that evaluated the quantitative impact of COVID-19 policies on direct COVID-19 outcomes. Nine studies were set aside because the study design was considered inappropriate for COVID-19 policy impact evaluation (n=8 pre/post; n=1 cross-sectional), and 27 articles were given a full consensus assessment. 20/27 met criteria for graphical display of data, 5/27 for functional form, 19/27 for timing between policy implementation and impact, and only 3/27 for concurrent changes to the outcomes. Only 4/27 were rated as overall appropriate. Including the 9 studies set aside, reviewers found that only four of the 36 identified published and peer-reviewed health policy impact evaluation studies passed a set of key design checks for identifying the causal impact of policies on COVID-19 outcomes. DISCUSSION: The reviewed literature directly evaluating the impact of COVID-19 policies largely failed to meet key design criteria for inference of sufficient rigour to be actionable by policy-makers. More reliable evidence review is needed to both identify and produce policy-actionable evidence, alongside the recognition that actionable evidence is often unlikely to be feasible.

Levels of depression, anxiety, and psychological distress among Ugandan adults during the first wave of the COVID-19 pandemic: cross-sectional evidence from a mobile phone-based population survey.

Objectives: Policy measures to slow the spread of coronavirus disease 2019 (COVID-19), such as curfews and business closures, may have negative effects on mental health. Populations in low- and middle-income countries (LMICs) may be particularly affected due to high rates of poverty and less comprehensive welfare systems, but the evidence is scarce. We evaluated predictors of depression, anxiety, and psychological distress in Uganda, which implemented one of the world's most stringent lockdowns. Methods: We conducted a mobile phone-based cross-sectional survey from December 2020 through April 2021 among individuals aged 18 years or over in Uganda. We measured depression, anxiety, and psychological distress using the Patient Health Questionnaire (PHQ)-2, the Generalized Anxiety Disorder (GAD)-2, and the PHQ-4. We applied linear regression to assess associations between experiences of COVID-19 (including fear of infection, social isolation, income loss, difficulty accessing medical care, school closings, and interactions with police) and PHQ-4 score, adjusted for sociodemographic characteristics. Results: 29.2% of 4066 total participants reported scores indicating moderate psychological distress, and 12.1% reported scores indicating severe distress. Distress was most common among individuals who were female, had lower levels of education, and lived in households with children. Related to COVID-19, PHQ-4 score was significantly associated with difficulty accessing medical care, worries about COVID-19, worries about interactions with police over lockdown measures, and days spent at home. Conclusions: There is an urgent need to address the significant burden of psychological distress associated with COVID-19 and policy responses in LMICs. Pandemic mitigation strategies must consider mental health consequences.

Impact Evaluation of Coronavirus Disease 2019 Policy: A Guide to Common Design Issues.

Policy responses to coronavirus disease 2019 (COVID-19), particularly those related to nonpharmaceutical interventions, are unprecedented in scale and scope. However, evaluations of policy impacts require a complex combination of circumstance, study design, data, statistics, and analysis. Beyond the issues that are faced for any policy, evaluation of COVID-19 policies is complicated by additional challenges related to infectious disease dynamics and a multiplicity of interventions. The methods needed for policy-level impact evaluation are not often used or taught in epidemiology, and they differ in important ways that may not be obvious. Methodological complications of policy evaluations can make it difficult for decision-makers and researchers to synthesize and evaluate the strength of the evidence in COVID-19 health policy papers. Here we 1) introduce the basic suite of policy-impact evaluation designs for observational data, including cross-sectional analyses, pre-/post- analyses, interrupted time-series analysis, and difference-in-differences analysis; 2) demonstrate key ways in which the requirements and assumptions underlying these designs are often violated in the context of COVID-19; and 3) provide decision-makers and reviewers with a conceptual and graphical guide to identifying these key violations. Our overall goal is to help epidemiologists, policy-makers, journal editors, journalists, researchers, and other research consumers understand and weigh the strengths and limitations of evidence.

Impact of campaign-style delivery of routine vaccines: a quasi-experimental evaluation using routine health services data in India.

The world is not on track to achieve the goals for immunization coverage and equity described by the World Health Organization's Global Vaccine Action Plan. Many countries struggle to increase coverage of routine vaccination, and there is little evidence about how to do so effectively. In India in 2016, only 62% of children had received a full course of basic vaccines. In response, in 2017-18 the government implemented Intensified Mission Indradhanush (IMI), a nationwide effort to improve coverage and equity using a campaign-style strategy. Campaign-style approaches to routine vaccine delivery like IMI, sometimes called 'periodic intensification of routine immunization' (PIRI), are widely used, but there is little robust evidence on their effectiveness. We conducted a quasi-experimental evaluation of IMI using routine data on vaccine doses delivered, comparing districts participating and not participating in IMI. Our sample included all districts that could be merged with India's 2016 Demographic and Health Surveys data and had available data for the full study period. We used controlled interrupted time-series analysis to estimate the impact of IMI during the 4-month implementation period and in subsequent months. This method assumes that, if IMI had not occurred, vaccination trends would have changed in the same way in the participating and not participating districts. We found that, during implementation, IMI increased delivery of 13 infant vaccines, with a median effect of 10.6% (95% confidence interval 5.1% to 16.5%). We did not find evidence of a sustained effect during the 8 months after implementation ended. Over the 12 months from the beginning of implementation, we estimated reductions in the number of under-immunized children that were large but not statistically significant, ranging from 3.9% (-6.9% to 13.7%) to 35.7% (-7.5% to 77.4%) for different vaccines. The largest effects were for the first doses of vaccines against diphtheria-tetanus-pertussis and polio: IMI reached approximately one-third of children who would otherwise not have received these vaccines. This suggests that PIRI can be successful in increasing routine immunization coverage, particularly for early infant vaccines, but other approaches may be needed for sustained coverage improvements.

Problems with Evidence Assessment in COVID-19 Health Policy Impact Evaluation: A systematic review of study design and evidence strength.

INTRODUCTION: Assessing the impact of COVID-19 policy is critical for informing future policies. However, there are concerns about the overall strength of COVID-19 impact evaluation studies given the circumstances for evaluation and concerns about the publication environment. This study systematically reviewed the strength of evidence in the published COVID-19 policy impact evaluation literature. METHODS: We included studies that were primarily designed to estimate the quantitative impact of one or more implemented COVID-19 policies on direct SARS-CoV-2 and COVID-19 outcomes. After searching PubMed for peer-reviewed articles published on November 26, 2020 or earlier and screening, all studies were reviewed by three reviewers first independently and then to consensus. The review tool was based on previously developed and released review guidance for COVID-19 policy impact evaluation, assessing what impact evaluation method was used, graphical display of outcomes data, functional form for the outcomes, timing between policy and impact, concurrent changes to the outcomes, and an overall rating. RESULTS: After 102 articles were identified as potentially meeting inclusion criteria, we identified 36 published articles that evaluated the quantitative impact of COVID-19 policies on direct COVID-19 outcomes. The majority (n=23/36) of studies in our sample examined the impact of stay-at-home requirements. Nine studies were set aside because the study design was considered inappropriate for COVID-19 policy impact evaluation (n=8 pre/post; n=1 cross-section), and 27 articles were given a full consensus assessment. 20/27 met criteria for graphical display of data, 5/27 for functional form, 19/27 for timing between policy implementation and impact, and only 3/27 for concurrent changes to the outcomes. Only 1/27 studies passed all of the above checks, and 4/27 were rated as overall appropriate. Including the 9 studies set aside, reviewers found that only four of the 36 identified published and peer-reviewed health policy impact evaluation studies passed a set of key design checks for identifying the causal impact of policies on COVID-19 outcomes. DISCUSSION: The reviewed literature directly evaluating the impact of COVID-19 policies largely failed to meet key design criteria for inference of sufficient rigor to be actionable by policymakers. This was largely driven by the circumstances under which policies were passed making it difficult to attribute changes in COVID-19 outcomes to particular policies. More reliable evidence review is needed to both identify and produce policy-actionable evidence, alongside the recognition that actionable evidence is often unlikely to be feasible.

Immunization costs, from evidence to policy: Findings from a nationally representative costing study and policy translation effort in Tanzania.

INTRODUCTION: Information on the costs of routine immunization programs is needed for budgeting, planning, and domestic resource mobilization. This information is particularly important for countries such as Tanzania that are preparing to transition out of support from Gavi, the Vaccine Alliance. This study aimed to estimate the total and unit costs for of child immunization in Tanzania from July 2016 to June 2017 and make this evidence available to key stakeholders. METHODS: We used an ingredients-based approach to collect routine immunization cost data from the facility, district, regional, and national levels. We collected data on the cost of vaccines as well as non-vaccine delivery costs. We estimated total and unit costs from a provider perspective for each level and overall, and examined how costs varied by delivery strategy, geographic area, and facility-level service delivery volume. An evidence-to-policy plan identified key opportunities and stakeholders to target to facilitate the use of results. RESULTS: The total annual economic cost of the immunization program, inclusive of vaccines, was estimated to be US$138 million (95% CI: 133, 144), or $4.32 ($3.72, $4.98) per dose. The delivery costs made up $45 million (38, 52), or $1.38 (1.06, 1.70) per dose. The costs of facility-based delivery were similar in urban and rural areas, but the costs of outreach delivery were higher in rural areas than in urban areas. The facility-level delivery cost per dose decreased with the facility service delivery volume. DISCUSSION: We estimated the costs of the routine immunization program in Tanzania, where no immunization costing study had been conducted for five years. These estimates can inform the program's budgeting and planning as Tanzania prepares to transition out of Gavi support. Next steps for evidence-to-policy translation have been identified, including technical support requirements for policy advocacy and planning.

Producing Standardized Country-Level Immunization Delivery Unit Cost Estimates.

BACKGROUND: To plan for the financial sustainability of immunization programs and make informed decisions to improve immunization coverage and equity, decision-makers need to know how much these programs cost beyond the cost of the vaccine. Non-vaccine delivery cost estimates can significantly influence the cost-effectiveness estimates used to allocate resources at the country level. However, many low- and middle-income countries (LMICs) do not have immunization delivery unit cost estimates available, or have estimates that are uncertain, unreliable, or old. We undertook a Bayesian evidence synthesis to generate country-level estimates of immunization delivery unit costs for LMICs. METHODS: From a database of empirical immunization costing studies, we extracted estimates of the delivery cost per dose for routine childhood immunization services, excluding vaccine costs. A Bayesian meta-regression model was used to regress delivery cost per dose estimates, stratified by cost category, against a set of predictor variables including country-level [gross domestic product per capita, reported diphtheria-tetanus-pertussis third dose coverage (DTP3), population, and number of doses in the routine vaccination schedule] and study-level (study year, single antigen or programmatic cost per dose, and financial or economic cost) predictors. The fitted prediction model was used to generate standardized estimates of the routine immunization delivery cost per dose for each LMIC for 2009-2018. Alternative regression models were specified in sensitivity analyses. RESULTS: We estimated the prediction model using the results from 29 individual studies, covering 24 countries. The predicted economic cost per dose for routine delivery of childhood vaccines (2018 US dollars), not including the price of the vaccine, was $1.87 (95% uncertainty interval $0.64-4.38) across all LMICs. By individual cost category, the programmatic economic cost per dose for routine delivery of childhood vaccines was $0.74 ($0.26-1.70) for labor, $0.26 ($0.08-0.67) for supply chain, $0.22 ($0.06-0.57) for capital, and $0.65 ($0.20-1.66) for other service delivery costs. CONCLUSIONS: Accurate immunization delivery costs are necessary for assessing the cost-effectiveness and strategic planning needs of immunization programs. The cost estimates from this analysis provide a broad indication of immunization delivery costs that may be useful when accurate local data are unavailable.