Problems with evidence assessment in COVID-19 health policy impact evaluation: a systematic review of study design and evidence strength.

INTRODUCTION: Assessing the impact of COVID-19 policy is critical for informing future policies. However, there are concerns about the overall strength of COVID-19 impact evaluation studies given the circumstances for evaluation and concerns about the publication environment. METHODS: We included studies that were primarily designed to estimate the quantitative impact of one or more implemented COVID-19 policies on direct SARS-CoV-2 and COVID-19 outcomes. After searching PubMed for peer-reviewed articles published on 26 November 2020 or earlier and screening, all studies were reviewed by three reviewers first independently and then to consensus. The review tool was based on previously developed and released review guidance for COVID-19 policy impact evaluation. RESULTS: After 102 articles were identified as potentially meeting inclusion criteria, we identified 36 published articles that evaluated the quantitative impact of COVID-19 policies on direct COVID-19 outcomes. Nine studies were set aside because the study design was considered inappropriate for COVID-19 policy impact evaluation (n=8 pre/post; n=1 cross-sectional), and 27 articles were given a full consensus assessment. 20/27 met criteria for graphical display of data, 5/27 for functional form, 19/27 for timing between policy implementation and impact, and only 3/27 for concurrent changes to the outcomes. Only 4/27 were rated as overall appropriate. Including the 9 studies set aside, reviewers found that only four of the 36 identified published and peer-reviewed health policy impact evaluation studies passed a set of key design checks for identifying the causal impact of policies on COVID-19 outcomes. DISCUSSION: The reviewed literature directly evaluating the impact of COVID-19 policies largely failed to meet key design criteria for inference of sufficient rigour to be actionable by policy-makers. More reliable evidence review is needed to both identify and produce policy-actionable evidence, alongside the recognition that actionable evidence is often unlikely to be feasible.

Problems with Evidence Assessment in COVID-19 Health Policy Impact Evaluation: A systematic review of study design and evidence strength.

INTRODUCTION: Assessing the impact of COVID-19 policy is critical for informing future policies. However, there are concerns about the overall strength of COVID-19 impact evaluation studies given the circumstances for evaluation and concerns about the publication environment. This study systematically reviewed the strength of evidence in the published COVID-19 policy impact evaluation literature. METHODS: We included studies that were primarily designed to estimate the quantitative impact of one or more implemented COVID-19 policies on direct SARS-CoV-2 and COVID-19 outcomes. After searching PubMed for peer-reviewed articles published on November 26, 2020 or earlier and screening, all studies were reviewed by three reviewers first independently and then to consensus. The review tool was based on previously developed and released review guidance for COVID-19 policy impact evaluation, assessing what impact evaluation method was used, graphical display of outcomes data, functional form for the outcomes, timing between policy and impact, concurrent changes to the outcomes, and an overall rating. RESULTS: After 102 articles were identified as potentially meeting inclusion criteria, we identified 36 published articles that evaluated the quantitative impact of COVID-19 policies on direct COVID-19 outcomes. The majority (n=23/36) of studies in our sample examined the impact of stay-at-home requirements. Nine studies were set aside because the study design was considered inappropriate for COVID-19 policy impact evaluation (n=8 pre/post; n=1 cross-section), and 27 articles were given a full consensus assessment. 20/27 met criteria for graphical display of data, 5/27 for functional form, 19/27 for timing between policy implementation and impact, and only 3/27 for concurrent changes to the outcomes. Only 1/27 studies passed all of the above checks, and 4/27 were rated as overall appropriate. Including the 9 studies set aside, reviewers found that only four of the 36 identified published and peer-reviewed health policy impact evaluation studies passed a set of key design checks for identifying the causal impact of policies on COVID-19 outcomes. DISCUSSION: The reviewed literature directly evaluating the impact of COVID-19 policies largely failed to meet key design criteria for inference of sufficient rigor to be actionable by policymakers. This was largely driven by the circumstances under which policies were passed making it difficult to attribute changes in COVID-19 outcomes to particular policies. More reliable evidence review is needed to both identify and produce policy-actionable evidence, alongside the recognition that actionable evidence is often unlikely to be feasible.

Ventilator Triage Policies During the COVID-19 Pandemic at U.S. Hospitals Associated With Members of the Association of Bioethics Program Directors.

BACKGROUND: The coronavirus disease 2019 pandemic has or threatens to overwhelm health care systems. Many institutions are developing ventilator triage policies. OBJECTIVE: To characterize the development of ventilator triage policies and compare policy content. DESIGN: Survey and mixed-methods content analysis. SETTING: North American hospitals associated with members of the Association of Bioethics Program Directors. PARTICIPANTS: Program directors. MEASUREMENTS: Characteristics of institutions and policies, including triage criteria and triage committee membership. RESULTS: Sixty-seven program directors responded (response rate, 91.8%); 36 (53.7%) hospitals did not yet have a policy, and 7 (10.4%) hospitals' policies could not be shared. The 29 institutions providing policies were relatively evenly distributed among the 4 U.S. geographic regions (range, 5 to 9 policies per region). Among the 26 unique policies analyzed, 3 (11.3%) were produced by state health departments. The most frequently cited triage criteria were benefit (25 policies [96.2%]), need (14 [53.8%]), age (13 [50.0%]), conservation of resources (10 [38.5%]), and lottery (9 [34.6%]). Twenty-one (80.8%) policies use scoring systems, and 20 of these (95.2%) use a version of the Sequential Organ Failure Assessment score. Among the policies that specify the triage team's composition (23 [88.5%]), all require or recommend a physician member, 20 (87.0%) a nurse, 16 (69.6%) an ethicist, 8 (34.8%) a chaplain, and 8 (34.8%) a respiratory therapist. Thirteen (50.0% of all policies) require or recommend that those making triage decisions not be involved in direct patient care, but only 2 (7.7%) require that their decisions be blinded to ethically irrelevant considerations. LIMITATION: The results may not be generalizable to institutions without academic bioethics programs. CONCLUSION: Over one half of respondents did not have ventilator triage policies. Policies have substantial heterogeneity, and many omit guidance on fair implementation. PRIMARY FUNDING SOURCE: None.

Stoic beliefs and health: development and preliminary validation of the Pathak-Wieten Stoicism Ideology Scale.

INTRODUCTION: We developed and validated a new parsimonious scale to measure stoic beliefs. Key domains of stoicism are imperviousness to strong emotions, indifference to death, taciturnity and self-sufficiency. In the context of illness and disease, a personal ideology of stoicism may create an internal resistance to objective needs, which can lead to negative consequences. Stoicism has been linked to help-seeking delays, inadequate pain treatment, caregiver strain and suicide after economic stress. METHODS: During 2013-2014, 390 adults aged 18+ years completed a brief anonymous paper questionnaire containing the preliminary 24-item Pathak-Wieten Stoicism Ideology Scale (PW-SIS). Confirmatory factor analysis (CFA) was used to test an a priori multidomain theoretical model. Content validity and response distributions were examined. Sociodemographic predictors of strong endorsement of stoicism were explored with logistic regression. RESULTS: The final PW-SIS contains four conceptual domains and 12 items. CFA showed very good model fit: root mean square error of approximation (RMSEA)=0.05 (95% CI 0.04 to 0.07), goodness-of-fit index=0.96 and Tucker-Lewis Index=0.93. Cronbach's alpha was 0.78 and ranged from 0.64 to 0.71 for the subscales. Content validity analysis showed a statistically significant trend, with respondents who reported trying to be a stoic 'all of the time' having the highest PW-SIS scores. Men were over two times as likely as women to fall into the top quartile of responses (OR=2.30, 95% CI 1.44 to 3.68, P<0.001). ORs showing stronger endorsement of stoicism by Hispanics, Blacks and biracial persons were not statistically significant. DISCUSSION: The PW-SIS is a valid and theoretically coherent scale which is brief and practical for integration into a wide range of health behaviour and outcomes research studies.