Problems with evidence assessment in COVID-19 health policy impact evaluation: a systematic review of study design and evidence strength.

INTRODUCTION: Assessing the impact of COVID-19 policy is critical for informing future policies. However, there are concerns about the overall strength of COVID-19 impact evaluation studies given the circumstances for evaluation and concerns about the publication environment. METHODS: We included studies that were primarily designed to estimate the quantitative impact of one or more implemented COVID-19 policies on direct SARS-CoV-2 and COVID-19 outcomes. After searching PubMed for peer-reviewed articles published on 26 November 2020 or earlier and screening, all studies were reviewed by three reviewers first independently and then to consensus. The review tool was based on previously developed and released review guidance for COVID-19 policy impact evaluation. RESULTS: After 102 articles were identified as potentially meeting inclusion criteria, we identified 36 published articles that evaluated the quantitative impact of COVID-19 policies on direct COVID-19 outcomes. Nine studies were set aside because the study design was considered inappropriate for COVID-19 policy impact evaluation (n=8 pre/post; n=1 cross-sectional), and 27 articles were given a full consensus assessment. 20/27 met criteria for graphical display of data, 5/27 for functional form, 19/27 for timing between policy implementation and impact, and only 3/27 for concurrent changes to the outcomes. Only 4/27 were rated as overall appropriate. Including the 9 studies set aside, reviewers found that only four of the 36 identified published and peer-reviewed health policy impact evaluation studies passed a set of key design checks for identifying the causal impact of policies on COVID-19 outcomes. DISCUSSION: The reviewed literature directly evaluating the impact of COVID-19 policies largely failed to meet key design criteria for inference of sufficient rigour to be actionable by policy-makers. More reliable evidence review is needed to both identify and produce policy-actionable evidence, alongside the recognition that actionable evidence is often unlikely to be feasible.

Problems with Evidence Assessment in COVID-19 Health Policy Impact Evaluation: A systematic review of study design and evidence strength.

INTRODUCTION: Assessing the impact of COVID-19 policy is critical for informing future policies. However, there are concerns about the overall strength of COVID-19 impact evaluation studies given the circumstances for evaluation and concerns about the publication environment. This study systematically reviewed the strength of evidence in the published COVID-19 policy impact evaluation literature. METHODS: We included studies that were primarily designed to estimate the quantitative impact of one or more implemented COVID-19 policies on direct SARS-CoV-2 and COVID-19 outcomes. After searching PubMed for peer-reviewed articles published on November 26, 2020 or earlier and screening, all studies were reviewed by three reviewers first independently and then to consensus. The review tool was based on previously developed and released review guidance for COVID-19 policy impact evaluation, assessing what impact evaluation method was used, graphical display of outcomes data, functional form for the outcomes, timing between policy and impact, concurrent changes to the outcomes, and an overall rating. RESULTS: After 102 articles were identified as potentially meeting inclusion criteria, we identified 36 published articles that evaluated the quantitative impact of COVID-19 policies on direct COVID-19 outcomes. The majority (n=23/36) of studies in our sample examined the impact of stay-at-home requirements. Nine studies were set aside because the study design was considered inappropriate for COVID-19 policy impact evaluation (n=8 pre/post; n=1 cross-section), and 27 articles were given a full consensus assessment. 20/27 met criteria for graphical display of data, 5/27 for functional form, 19/27 for timing between policy implementation and impact, and only 3/27 for concurrent changes to the outcomes. Only 1/27 studies passed all of the above checks, and 4/27 were rated as overall appropriate. Including the 9 studies set aside, reviewers found that only four of the 36 identified published and peer-reviewed health policy impact evaluation studies passed a set of key design checks for identifying the causal impact of policies on COVID-19 outcomes. DISCUSSION: The reviewed literature directly evaluating the impact of COVID-19 policies largely failed to meet key design criteria for inference of sufficient rigor to be actionable by policymakers. This was largely driven by the circumstances under which policies were passed making it difficult to attribute changes in COVID-19 outcomes to particular policies. More reliable evidence review is needed to both identify and produce policy-actionable evidence, alongside the recognition that actionable evidence is often unlikely to be feasible.

Exploring gender and partner communication: Theory of planned behavior predictors for condom use among urban youth in Zambia.

OBJECTIVE: To understand how knowledge and perceptions of condoms and partner communication influence use of condoms in a high HIV prevalence setting and gender-specific differences. METHODS: A cross-sectional study was conducted in Zambia from 2015 to 2016. The survey included questions on demographics, sexual behavior, contraceptive perceptions, and behaviors. We constructed multivariate regression models using the Theory of Planned Behavior to determine associations between knowledge, perceptions, and perceived control with intended, communicated, and reported use of condoms by gender. RESULTS: The participants were 2388 sexually active urban residents aged 18-24 years. In the sample, 1646 (69%) were female, 841 (35%) married, and 1894 (61%) unemployed. Partner communication was the predictor most associated with use of condoms. Among women, partner communication was associated with over three times higher odds of condom use (odds ratio [OR] 3.51, 95% confidence interval [CI] 2.65-4.65) but being married reduced the odds of condom use by 76% (OR 0.24, 95% CI 0.17-0.33). For men, a network of friends that was supportive of the use of contraception was associated with increased odds of 55% for use of condoms (OR 1.55, 95% CI 1.10-2.18). CONCLUSION: Public health programs aimed at increasing safer sexual behavior and use of condoms must consider improving gender equity and partner communication, as knowledge of contraceptives and positive perceptions are not enough to ensure their use.

The child survival impact of the Ghana Essential Health Interventions Program: A health systems strengthening plausibility trial in Northern Ghana.

BACKGROUND: The Ghana Health Service in collaboration with partner institutions implemented a five-year primary health systems strengthening program known as the Ghana Essential Health Intervention Program (GEHIP). GEHIP was a plausibility trial implemented in an impoverished region of northern Ghana around the World Health Organizations (WHO) six pillars combined with community engagement, leadership development and grassroots political support, the program organized a program of training and action focused on strategies for saving newborn lives and community-engaged emergency referral services. This paper analyzes the effect of the GEHIP program on child survival. METHODS: Birth history data assembled from baseline and endline surveys are used to assess the hazard of child mortality in GEHIP treatment and comparison areas prior to and after the start of treatment. Difference-in-differences (DiD) methods are used to compare mortality change over time among children exposed to GEHIP relative to children in the comparison area over the same time period. Models test the hypothesis that a package of systems strengthening activities improved childhood survival. Models adjusted for the potentially confounding effects of baseline differentials, secular mortality trends, household characteristics such as relative wealth and parental educational attainment, and geographic accessibility of clinical care. RESULTS: The GEHIP combination of health systems strengthening activities reduced neonatal mortality by approximately one half (HR = 0.52, 95% CI = 0.28,0.98, p = 0.045). There was a null incremental effect of GEHIP on mortality of post-neonate infants (from 1 to 12 months old) (HR = 0.72; 95% CI = 0.30,1.79; p = 0.480) and post-infants (from 1 year to 5 years old) -(HR = 1.02; 95% CI = 0.55-1.90; p = 0.940). Age-specific analyses show that impact was concentrated among neonates. However, effect ratios for post-infancy were inefficiently assessed owing to extensive survival history censoring for the later months of childhood. Children were observed only rarely for periods over 40 months of age. CONCLUSION: GEHIP results show that a comprehensive approach to newborn care is feasible, if care is augmented by community-based nurses. It supports the assertion that if appropriate mechanisms are put in place to enable the various pillars of the health system as espoused by WHO in rural impoverished settings where childhood mortality is high, it could lead to accelerated reductions in mortality thereby increasing survival of children. Policy implications of the pronounced neonatal effect of GEHIP merit national review for possible scale-up.