Simple graphical rules for assessing selection bias in general-population and selected-sample treatment effects.

When analyzing a selected sample from a general population, selection bias can arise relative to the causal average treatment effect (ATE) for the general population, and also relative to the ATE for the selected sample itself. We provide simple graphical rules that indicate: (1) if a selected-sample analysis will be unbiased for each ATE; (2) whether adjusting for certain covariates could eliminate selection bias. The rules can easily be checked in a standard single-world intervention graph. When the treatment could affect selection, a third estimand of potential scientific interest is the "net treatment difference", namely the net change in outcomes that would occur for the selected sample if all members of the general population were treated versus not treated, including any effects of the treatment on which individuals are in the selected sample . We provide graphical rules for this estimand as well. We decompose bias in a selected-sample analysis relative to the general-population ATE into: (1) "internal bias" relative to the net treatment difference; (2) "net-external bias", a discrepancy between the net treatment difference and the general-population ATE. Each bias can be assessed unambiguously via a distinct graphical rule, providing new conceptual insight into the mechanisms by which certain causal structures produce selection bias.

Toward evidence-based communication on overweight body mass index and mortality.

BACKGROUND: Reducing overweight and obesity has been a longstanding focus of public health messaging and physician-patient interactions. Clinical guidelines by major public health organizations describe both overweight and obesity as risk factors for mortality and other health conditions. Accordingly, a majority of primary care physicians believe that overweight BMI (even without obesity) strongly increases mortality risk. MAIN POINTS: The current evidence base suggests that although both obese BMI and underweight BMI are consistently associated with increased all-cause mortality, overweight BMI (without obesity) is not meaningfully associated with increased mortality. In fact, a number of studies suggest modest protective, rather than detrimental, associations of overweight BMI with all-cause mortality. Given this current evidence base, clinical guidelines and physician perceptions substantially overstate all-cause mortality risks associated with the range of BMIs classified as "overweight" but not "obese." Discrepancies between evidence and communication regarding mortality raise the question of whether similar discrepancies exist for other health outcomes. CONCLUSIONS: Health communication that inaccurately conveys current evidence may do more harm than good; this applies to communication from health authorities to health practitioners as well as to communication from health practitioners to individual patients. We give three recommendations to better align health communication with the current evidence. First, recommendations to the public and health practitioners should distinguish overweight from obese BMI and at this time should not describe overweight BMI as a risk factor for all-cause mortality. Second, primary care physicians' widespread misconceptions about overweight BMI should be rectified. Third, the evidence basis for other potential risks or benefits of overweight BMI should be rigorously examined and incorporated appropriately into health communication.

The M-Value: A Simple Sensitivity Analysis for Bias Due to Missing Data in Treatment Effect Estimates.

Complete-case analyses can be biased if missing data are not missing completely at random. We propose simple sensitivity analyses that apply to complete-case estimates of treatment effects; these analyses use only simple summary data and obviate specifying the precise mechanism of missingness and making distributional assumptions. Bias arises when treatment effects differ between retained and nonretained participants or, among retained participants, the estimate is biased because conditioning on retention has induced a noncausal path between the treatment and outcome. We thus bound the overall treatment effect on the difference scale by specifying: 1) the unobserved treatment effect among nonretained participants; and 2) the strengths of association that unobserved variables have with the exposure and with the outcome among retained participants ("induced confounding associations"). Working with the former sensitivity parameter subsumes certain existing methods of worst-case imputation while also accommodating less-conservative assumptions (e.g., that the treatment is not detrimental on average even among nonretained participants). As an analog to the E-value for confounding, we propose the M-value, which represents, for a specified treatment effect among nonretained participants, the strength of induced confounding associations required to reduce the treatment effect to the null or to any other value. These methods could help characterize the robustness of complete-case analyses to potential bias due to missing data.

Toward Open and Reproducible Epidemiology.

Starting in the 2010s, researchers in the experimental social sciences rapidly began to adopt increasingly open and reproducible scientific practices. These practices include publicly sharing deidentified data when possible, sharing analytical code, and preregistering study protocols. Empirical evidence from the social sciences suggests such practices are feasible, can improve analytical reproducibility, and can reduce selective reporting. In academic epidemiology, adoption of open-science practices has been slower than in the social sciences (with some notable exceptions, such as registering clinical trials). Epidemiologic studies are often large, complex, conceived after data have already been collected, and difficult to replicate directly by collecting new data. These characteristics make it especially important to ensure their integrity and analytical reproducibility. Open-science practices can also pay immediate dividends to researchers' own work by clarifying scientific reasoning and encouraging well-documented, organized workflows. We consider how established epidemiologists and early-career researchers alike can help midwife a culture of open science in epidemiology through their research practices, mentorship, and editorial activities.

Exogenous female sex steroid hormones and new-onset asthma in women: a matched case-control study.

BACKGROUND: Evidence on the role of exogenous female sex steroid hormones in asthma development in women remains conflicting. We sought to quantify the potential causal role of hormonal contraceptives and menopausal hormone therapy (MHT) in the development of asthma in women. METHODS: We conducted a matched case-control study based on the West Sweden Asthma Study, nested in a representative cohort of 15,003 women aged 16-75 years, with 8-year follow-up (2008-2016). Data were analyzed using Frequentist and Bayesian conditional logistic regression models. RESULTS: We included 114 cases and 717 controls. In Frequentist analysis, the odds ratio (OR) for new-onset asthma with ever use of hormonal contraceptives was 2.13 (95% confidence interval [CI] 1.03-4.38). Subgroup analyses showed that the OR increased consistently with older baseline age. The OR for new-onset asthma with ever MHT use among menopausal women was 1.17 (95% CI 0.49-2.82). In Bayesian analysis, the ORs for ever use of hormonal contraceptives and MHT were, respectively, 1.11 (95% posterior interval [PI] 0.79-1.55) and 1.18 (95% PI 0.92-1.52). The respective probability of each OR being larger than 1 was 72.3% and 90.6%. CONCLUSIONS: Although use of hormonal contraceptives was associated with an increased risk of asthma, this may be explained by selection of women by baseline asthma status, given the upward trend in the effect estimate with older age. This indicates that use of hormonal contraceptives may in fact decrease asthma risk in women. Use of MHT may increase asthma risk in menopausal women.

Effect Measure Modification by Covariates in Mediation: Extending Regression-based Causal Mediation Analysis.

Existing methods for regression-based mediation analysis assume that the exposure-mediator effect, exposure-outcome effect, and mediator-outcome effect are constant across levels of the baseline characteristics of patients. However, investigators often have insight into how these underlying effects may be modified by baseline characteristics and are interested in how the resulting mediation effects, such as the natural direct effect (NDE), the natural indirect effect. (NIE), and the proportion mediated, are modified by these baseline characteristics. Motivated by an empirical example of anti-interleukin-1 therapy's benefit on incident anemia reduction and its mediation by an early change in an inflammatory biomarker, we extended the closed-form regression-based causal mediation analysis with effect measure modification (EMM). Using a simulated numerical example, we demonstrated that naive analysis without considering EMM can give biased estimates of NDE and NIE and visually illustrated how baseline characteristics affect the presence and magnitude of EMM of NDE and NIE. We then applied the extended method to the empirical example informed by pathophysiologic insights into potential EMM by age, diabetes, and baseline inflammation. We found that the proportion modified through the early post-treatment inflammatory biomarker was greater for younger, nondiabetic patients with lower baseline level of inflammation, suggesting differential usefulness of the early post-treatment inflammatory biomarker in monitoring patients depending on baseline characteristics. To facilitate the adoption of EMM considerations in causal mediation analysis by the wider clinical and epidemiologic research communities, we developed a free- and open-source R package, regmedint.

Is There Evidence of P-Hacking in Imaging Research?

BACKGROUND: P-hacking, the tendency to run selective analyses until they become significant, is prevalent in many scientific disciplines. PURPOSE: This study aims to assess if p-hacking exists in imaging research. METHODS: Protocol, data, and code available here https://osf.io/xz9ku/?view_only=a9f7c2d841684cb7a3616f567db273fa. We searched imaging journals Ovid MEDLINE from 1972 to 2021. Text mining using Python script was used to collect metadata: journal, publication year, title, abstract, and P-values from abstracts. One P-value was randomly sampled per abstract. We assessed for evidence of p-hacking using a p-curve, by evaluating for a concentration of P-values just below .05. We conducted a one-tailed binomial test (α = .05 level of significance) to assess whether there were more P-values falling in the upper range (e.g., .045 < P < .05) than in the lower range (e.g., .04 < P < .045). To assess variation in results introduced by our random sampling of a single P-value per abstract, we repeated the random sampling process 1000 times and pooled results across the samples. Analysis was done (divided into 10-year periods) to determine if p-hacking practices evolved over time. RESULTS: Our search of 136 journals identified 967,981 abstracts. Text mining identified 293,687 P-values, and a total of 4105 randomly sampled P-values were included in the p-hacking analysis. The number of journals and abstracts that were included in the analysis as a fraction and percentage of the total number was, respectively, 108/136 (80%) and 4105/967,981 (.4%). P-values did not concentrate just under .05; in fact, there were more P-values falling in the lower range (e.g., .04 < P < .045) than falling just below .05 (e.g., .045 < P < .05), indicating lack of evidence for p-hacking. Time trend analysis did not identify p-hacking in any of the five 10-year periods. CONCLUSION: We did not identify evidence of p-hacking in abstracts published in over 100 imaging journals since 1972. These analyses cannot detect all forms of p-hacking, and other forms of bias may exist in imaging research such as publication bias and selective outcome reporting.

Methods to Address Confounding and Other Biases in Meta-Analyses: Review and Recommendations.

Meta-analyses contribute critically to cumulative science, but they can produce misleading conclusions if their constituent primary studies are biased, for example by unmeasured confounding in nonrandomized studies. We provide practical guidance on how meta-analysts can address confounding and other biases that affect studies' internal validity, focusing primarily on sensitivity analyses that help quantify how biased the meta-analysis estimates might be. We review a number of sensitivity analysis methods to do so, especially recent developments that are straightforward to implement and interpret and that use somewhat less stringent statistical assumptions than do earlier methods. We give recommendations for how these newer methods could be applied in practice and illustrate using a previously published meta-analysis. Sensitivity analyses can provide informative quantitative summaries of evidence strength, and we suggest reporting them routinely in meta-analyses of potentially biased studies. This recommendation in no way diminishes the importance of defining study eligibility criteria that reduce bias and of characterizing studies' risks of bias qualitatively.

Menopausal hormone therapy and women's health: An umbrella review.

BACKGROUND: There remains uncertainty about the impact of menopausal hormone therapy (MHT) on women's health. A systematic, comprehensive assessment of the effects on multiple outcomes is lacking. We conducted an umbrella review to comprehensively summarize evidence on the benefits and harms of MHT across diverse health outcomes. METHODS AND FINDINGS: We searched MEDLINE, EMBASE, and 10 other databases from inception to November 26, 2017, updated on December 17, 2020, to identify systematic reviews or meta-analyses of randomized controlled trials (RCTs) and observational studies investigating effects of MHT, including estrogen-alone therapy (ET) and estrogen plus progestin therapy (EPT), in perimenopausal or postmenopausal women in all countries and settings. All health outcomes in previous systematic reviews were included, including menopausal symptoms, surrogate endpoints, biomarkers, various morbidity outcomes, and mortality. Two investigators independently extracted data and assessed methodological quality of systematic reviews using the updated 16-item AMSTAR 2 instrument. Random-effects robust variance estimation was used to combine effect estimates, and 95% prediction intervals (PIs) were calculated whenever possible. We used the term MHT to encompass ET and EPT, and results are presented for MHT for each outcome, unless otherwise indicated. Sixty systematic reviews were included, involving 102 meta-analyses of RCTs and 38 of observational studies, with 102 unique outcomes. The overall quality of included systematic reviews was moderate to poor. In meta-analyses of RCTs, MHT was beneficial for vasomotor symptoms (frequency: 9 trials, 1,104 women, risk ratio [RR] 0.43, 95% CI 0.33 to 0.57, p < 0.001; severity: 7 trials, 503 women, RR 0.29, 95% CI 0.17 to 0.50, p = 0.002) and all fracture (30 trials, 43,188 women, RR 0.72, 95% CI 0.62 to 0.84, p = 0.002, 95% PI 0.58 to 0.87), as well as vaginal atrophy (intravaginal ET), sexual function, vertebral and nonvertebral fracture, diabetes mellitus, cardiovascular mortality (ET), and colorectal cancer (EPT), but harmful for stroke (17 trials, 37,272 women, RR 1.17, 95% CI 1.05 to 1.29, p = 0.027) and venous thromboembolism (23 trials, 42,292 women, RR 1.60, 95% CI 0.99 to 2.58, p = 0.052, 95% PI 1.03 to 2.99), as well as cardiovascular disease incidence and recurrence, cerebrovascular disease, nonfatal stroke, deep vein thrombosis, gallbladder disease requiring surgery, and lung cancer mortality (EPT). In meta-analyses of observational studies, MHT was associated with decreased risks of cataract, glioma, and esophageal, gastric, and colorectal cancer, but increased risks of pulmonary embolism, cholelithiasis, asthma, meningioma, and thyroid, breast, and ovarian cancer. ET and EPT had opposite effects for endometrial cancer, endometrial hyperplasia, and Alzheimer disease. The major limitations include the inability to address the varying effects of MHT by type, dose, formulation, duration of use, route of administration, and age of initiation and to take into account the quality of individual studies included in the systematic reviews. The study protocol is publicly available on PROSPERO (CRD42017083412). CONCLUSIONS: MHT has a complex balance of benefits and harms on multiple health outcomes. Some effects differ qualitatively between ET and EPT. The quality of available evidence is only moderate to poor.

Multiple-bias Sensitivity Analysis Using Bounds.

Confounding, selection bias, and measurement error are well-known sources of bias in epidemiologic research. Methods for assessing these biases have their own limitations. Many quantitative sensitivity analysis approaches consider each type of bias individually, although more complex approaches are harder to implement or require numerous assumptions. By failing to consider multiple biases at once, researchers can underestimate-or overestimate-their joint impact. We show that it is possible to bound the total composite bias owing to these three sources and to use that bound to assess the sensitivity of a risk ratio to any combination of these biases. We derive bounds for the total composite bias under a variety of scenarios, providing researchers with tools to assess their total potential impact. We apply this technique to a study where unmeasured confounding and selection bias are both concerns and to another study in which possible differential exposure misclassification and confounding are concerns. The approach we describe, though conservative, is easier to implement and makes simpler assumptions than quantitative bias analysis. We provide R functions to aid implementation.

Interventions to reduce meat consumption by appealing to animal welfare: Meta-analysis and evidence-based recommendations.

Reducing meat consumption may improve human health, curb environmental damage, and limit the large-scale suffering of animals raised in factory farms. Most attention to reducing consumption has focused on restructuring environments where foods are chosen or on making health or environmental appeals. However, psychological theory suggests that interventions appealing to animal welfare concerns might operate on distinct, potent pathways. We conducted a systematic review and meta-analysis evaluating the effectiveness of these interventions. We searched eight academic databases and extensively searched grey literature. We meta-analyzed 100 studies assessing interventions designed to reduce meat consumption or purchase by mentioning or portraying farm animals, that measured behavioral or self-reported outcomes related to meat consumption, purchase, or related intentions, and that had a control condition. The interventions consistently reduced meat consumption, purchase, or related intentions at least in the short term with meaningfully large effects (meta-analytic mean risk ratio [RR] = 1.22; 95% CI: [1.13, 1.33]). We estimated that a large majority of population effect sizes (71%; 95% CI: [59%, 80%]) were stronger than RR = 1.1 and that few were in the unintended direction. Via meta-regression, we identified some specific characteristics of studies and interventions that were associated with effect size. Risk-of-bias assessments identified both methodological strengths and limitations of this literature; however, results did not differ meaningfully in sensitivity analyses retaining only studies at the lowest risk of bias. Evidence of publication bias was not apparent. In conclusion, animal welfare interventions preliminarily appear effective in these typically short-term studies of primarily self-reported outcomes. Future research should use direct behavioral outcomes that minimize the potential for social desirability bias and are measured over long-term follow-up.

Robust Metrics and Sensitivity Analyses for Meta-analyses of Heterogeneous Effects.

We recently suggested new statistical metrics for routine reporting in random-effects meta-analyses to convey evidence strength for scientifically meaningful effects under effect heterogeneity. First, given a chosen threshold of meaningful effect size, we suggested reporting the estimated proportion of true population effect sizes above this threshold. Second, we suggested reporting the proportion of effect sizes below a second, possibly symmetric, threshold in the opposite direction from the estimated mean. Our previous methods applied when the true population effects are approximately normal, when the number of studies is relatively large, and when the proportion is between approximately 0.15 and 0.85. Here, we additionally describe robust methods for point estimation and inference that perform well under considerably more general conditions, as we validate in an extensive simulation study. The methods are implemented in the R package MetaUtility (function prop_stronger). We describe application of the robust methods to conducting sensitivity analyses for unmeasured confounding in meta-analyses.

Is Optimism Associated With Healthier Cardiovascular-Related Behavior? Meta-Analyses of 3 Health Behaviors.

Optimistic people have reduced risk for cardiovascular disease and cardiovascular-related mortality compared with their less optimistic peers. One explanation for this is that optimistic people may be more likely to engage in healthy behavior like exercising frequently, eating fruits and vegetables, and avoiding cigarette smoking. However, researchers have not formally determined the extent or direction of optimism's association with health behaviors. Moreover, it is unclear whether optimism temporally precedes health behaviors or whether the relationship is because of shared common causes. We conducted random effects meta-analyses examining optimism's association with 3 health behaviors relevant for the prevention of cardiovascular disease. PubMed and PsycINFO databases were searched for studies published through November 2017 reporting on optimism's relationship with physical activity, diet, and cigarette smoking. We identified 34 effect sizes for physical activity (n=90 845), 15 effect sizes for diet (n=47 931), and 15 effect sizes for cigarette smoking (n=15 052). Findings suggested that more optimistic individuals tended to engage in healthier behaviors compared with less optimistic individuals, but effect sizes were modest (ractivity=0.07, P<0.0001; rdiet=0.12, P<0.0001; and rsmoking=0.07, P=0.001). Most evidence was cross-sectional (≥53% of effect sizes) and did not consider sociodemographic characteristics (<53% of effect sizes) or psychological distress (<27% of effect sizes) as potential confounders. Optimism is associated with healthier behaviors that protect against cardiovascular disease, although most evidence was relatively low quality. Additional longitudinal and experimental research is required to determine whether optimism causally contributes to healthy behaviors and whether optimism could be an effective target for preventing cardiovascular disease.

Assessing the impact of unmeasured confounders for credible and reliable real-world evidence.

PURPOSE: We review statistical methods for assessing the possible impact of bias due to unmeasured confounding in real world data analysis and provide detailed recommendations for choosing among the methods. METHODS: By updating an earlier systematic review, we summarize modern statistical best practices for evaluating and correcting for potential bias due to unmeasured confounding in estimating causal treatment effect from non-interventional studies. RESULTS: We suggest a hierarchical structure for assessing unmeasured confounding. First, for initial sensitivity analyses, we strongly recommend applying a recently developed method, the E-value, that is straightforward to apply and does not require prior knowledge or assumptions about the unmeasured confounder(s). When some such knowledge is available, the E-value could be supplemented by the rule-out or array method at this step. If these initial analyses suggest results may not be robust to unmeasured confounding, subsequent analyses could be conducted using more specialized statistical methods, which we categorize based on whether they require access to external data on the suspected unmeasured confounder(s), internal data, or no data. Other factors for choosing the subsequent sensitivity analysis methods are also introduced and discussed, including the types of unmeasured confounders and whether the subsequent sensitivity analysis is intended to provide a corrected causal treatment effect. CONCLUSION: Various analytical methods have been proposed to address unmeasured confounding, but little research has discussed a structured approach to select appropriate methods in practice. In providing practical suggestions for choosing appropriate initial and, potentially, more specialized subsequent sensitivity analyses, we hope to facilitate the widespread reporting of such sensitivity analyses in non-interventional studies. The suggested approach also has the potential to inform pre-specification of sensitivity analyses before executing the analysis, and therefore increase the transparency and limit selective study reporting.

New metrics for meta-analyses of heterogeneous effects.

We provide two simple metrics that could be reported routinely in random-effects meta-analyses to convey evidence strength for scientifically meaningful effects under effect heterogeneity (ie, a nonzero estimated variance of the true effect distribution). First, given a chosen threshold of meaningful effect size, meta-analyses could report the estimated proportion of true effect sizes above this threshold. Second, meta-analyses could estimate the proportion of effect sizes below a second, possibly symmetric, threshold in the opposite direction from the estimated mean. These metrics could help identify if (1) there are few effects of scientifically meaningful size despite a "statistically significant" pooled point estimate, (2) there are some large effects despite an apparently null point estimate, or (3) strong effects in the direction opposite the pooled estimate also regularly occur (and thus, potential effect modifiers should be examined). These metrics should be presented with confidence intervals, which can be obtained analytically or, under weaker assumptions, using bias-corrected and accelerated bootstrapping. Additionally, these metrics inform relative comparison of evidence strength across related meta-analyses. We illustrate with applied examples and provide an R function to compute the metrics and confidence intervals.

Missing data strategies for time-varying confounders in comparative effectiveness studies of non-missing time-varying exposures and right-censored outcomes.

The treatment of missing data in comparative effectiveness studies with right-censored outcomes and time-varying covariates is challenging because of the multilevel structure of the data. In particular, the performance of an accessible method like multiple imputation (MI) under an imputation model that ignores the multilevel structure is unknown and has not been compared to complete-case (CC) and single imputation methods that are most commonly applied in this context. Through an extensive simulation study, we compared statistical properties among CC analysis, last value carried forward, mean imputation, the use of missing indicators, and MI-based approaches with and without auxiliary variables under an extended Cox model when the interest lies in characterizing relationships between non-missing time-varying exposures and right-censored outcomes. MI demonstrated favorable properties under a moderate missing-at-random condition (absolute bias <0.1) and outperformed CC and single imputation methods, even when the MI method did not account for correlated observations in the imputation model. The performance of MI decreased with increasing complexity such as when the missing data mechanism involved the exposure of interest, but was still preferred over other methods considered and performed well in the presence of strong auxiliary variables. We recommend considering MI that ignores the multilevel structure in the imputation model when data are missing in a time-varying confounder, incorporating variables associated with missingness in the MI models as well as conducting sensitivity analyses across plausible assumptions.

Open-source software for mouse-tracking in Qualtrics to measure category competition.

Mouse-tracking is a sophisticated tool for measuring rapid, dynamic cognitive processes in real time, particularly in experiments investigating competition between perceptual or cognitive categories. We provide user-friendly, open-source software ( https://osf.io/st2ef/ ) for designing and analyzing such experiments online using the Qualtrics survey platform. The software consists of a Qualtrics template with embedded JavaScript and CSS along with R code to clean, parse, and analyze the data. No special programming skills are required to use this software. As we discuss, this software could be readily modified for use with other online survey platforms that allow the addition of custom JavaScript. We empirically validate the provided software by benchmarking its performance on previously tested stimuli (android robot faces) in a category-competition experiment with realistic crowdsourced data collection.