Efficiently exploring the causal role of contextual moderators in behavioral science.

Behavioral science interventions have the potential to address longstanding policy problems, but their effects are typically heterogeneous across contexts (e.g., teachers, schools, and geographic regions). This contextual heterogeneity is poorly understood, however, which reduces the field's impact and its understanding of mechanisms. Here, we present an efficient way to interrogate heterogeneity and address these gaps in knowledge. This method a) presents scenarios that vividly represent different moderating contexts, b) measures a short-term behavioral outcome (e.g., an academic choice) that is known to relate to typical intervention outcomes (e.g., academic achievement), and c) assesses the causal effect of the moderating context on the link between the psychological variable typically targeted by interventions and this short-term outcome. We illustrated the utility of this approach across four experiments (total n = 3,235) that directly tested contextual moderators of the links between growth mindset, which is the belief that ability can be developed, and students' academic choices. The present results showed that teachers' growth mindset-supportive messages and the structural opportunities they provide moderated the link between students' mindsets and their choices (studies 1 to 3). This pattern was replicated in a nationally representative sample of adolescents and did not vary across demographic subgroups (study 2), nor was this pattern the result of several possible confounds (studies 3 to 4). Discussion centers on how this method of interrogating contextual heterogeneity can be applied to other behavioral science interventions and broaden their impact in other policy domains.

Vaccination, immunity, and the changing impact of COVID-19 on infant health.

In utero exposure to COVID-19 infection may lead to large intergenerational health effects. The impact of infection exposure has likely evolved since the onset of the pandemic as new variants emerge, immunity from prior infection increases, vaccines become available, and vaccine hesitancy persists, such that when infection is experienced is as important as whether it is experienced. We examine the changing impact of COVID-19 infection on preterm birth and the moderating role of vaccination. We offer the first plausibly causal estimate of the impact of maternal COVID-19 infection by using population data with no selectivity, universal information on maternal COVID-19 infection, and linked sibling data. We then assess change in this impact from 2020 to 2023 and evaluate the protective role of COVID-19 vaccination on infant health. We find a substantial adverse effect of prenatal COVID-19 infection on the probability of preterm birth. The impact was large during the first 2 y of the pandemic but had fully disappeared by 2022. The harmful impact of COVID-19 infection disappeared almost a year earlier in zip codes with high vaccination rates, suggesting that vaccines might have prevented thousands of preterm births. The findings highlight the need to monitor the changing consequences of emerging infectious diseases over time and the importance of mitigation strategies to reduce the burden of infection on vulnerable populations.

Can the potential benefit of individualizing treatment be assessed using trial summary statistics alone?

Individualizing treatment assignment can improve outcomes for diseases with patient-to-patient variability in comparative treatment effects. When a clinical trial demonstrates that some patients improve on treatment while others do not, it is tempting to assume that treatment effect heterogeneity exists. However, if outcome variability is mainly driven by factors other than variability in the treatment effect, investigating the extent to which covariate data can predict differential treatment response is a potential waste of resources. Motivated by recent meta-analyses assessing the potential of individualizing treatment for major depressive disorder using only summary statistics, we provide a method that uses summary statistics widely available in published clinical trial results to bound the benefit of optimally assigning treatment to each patient. We also offer alternate bounds for settings in which trial results are stratified by another covariate. Our upper bounds can be especially informative when they are small, as there is then little benefit to collecting additional covariate data. We demonstrate our approach using summary statistics from a depression treatment trial. Our methods are implemented in the rct2otrbounds R package.

Quantile regressions as a tool to evaluate how an exposure shifts and reshapes the outcome distribution: A primer for epidemiologists.

Quantifying how an exposure affects the entire outcome distribution is often important, e.g., for outcomes such as blood pressure which have non-linear effects on long-term morbidity and mortality. Quantile regressions offer a powerful way of estimating an exposure's relationship with the outcome distribution but remain underused in epidemiology. We introduce quantile regressions with a focus on distinguishing estimators for quantiles of the conditional and unconditional outcome distributions. We also present an empirical example in which we fit mean and quantile regressions to investigate educational attainment's association with later-life systolic blood pressure (SBP). We use data on 8,875 US-born respondents aged 50+ years from the US Health and Retirement Study. More education was negatively associated with mean SBP. Conditional and unconditional quantile regressions both suggested a negative association between education and SBP at all levels of SBP, but the absolute magnitudes of these associations were higher at higher SBP quantiles relative to lower quantiles. In addition to showing that educational attainment shifted the SBP distribution left-wards, quantile regression results revealed that education may have reshaped the SBP distribution through larger protective associations in the right tail, thus benefiting those at highest risk of cardiovascular diseases.

A generalizability score for aggregate causal effect.

Scientists frequently generalize population level causal quantities such as average treatment effect from a source population to a target population. When the causal effects are heterogeneous, differences in subject characteristics between the source and target populations may make such a generalization difficult and unreliable. Reweighting or regression can be used to adjust for such differences when generalizing. However, these methods typically suffer from large variance if there is limited covariate distribution overlap between the two populations. We propose a generalizability score to address this issue. The score can be used as a yardstick to select target subpopulations for generalization. A simplified version of the score avoids using any outcome information and thus can prevent deliberate biases associated with inadvertent access to such information. Both simulation studies and real data analysis demonstrate convincing results for such selection.

Comparison of methods that combine multiple randomized trials to estimate heterogeneous treatment effects.

Individualized treatment decisions can improve health outcomes, but using data to make these decisions in a reliable, precise, and generalizable way is challenging with a single dataset. Leveraging multiple randomized controlled trials allows for the combination of datasets with unconfounded treatment assignment to better estimate heterogeneous treatment effects. This article discusses several nonparametric approaches for estimating heterogeneous treatment effects using data from multiple trials. We extend single-study methods to a scenario with multiple trials and explore their performance through a simulation study, with data generation scenarios that have differing levels of cross-trial heterogeneity. The simulations demonstrate that methods that directly allow for heterogeneity of the treatment effect across trials perform better than methods that do not, and that the choice of single-study method matters based on the functional form of the treatment effect. Finally, we discuss which methods perform well in each setting and then apply them to four randomized controlled trials to examine effect heterogeneity of treatments for major depressive disorder.

Nonparametric estimation of the random effects distribution for the risk or rate ratio in rare events meta-analysis with the arm-based and contrast-based approaches.

Rare events are events which occur with low frequencies. These often arise in clinical trials or cohort studies where the data are arranged in binary contingency tables. In this article, we investigate the estimation of effect heterogeneity for the risk-ratio parameter in meta-analysis of rare events studies through two likelihood-based nonparametric mixture approaches: an arm-based and a contrast-based model. Maximum likelihood estimation is achieved using the EM algorithm. Special attention is given to the choice of initial values. Inspired by the classification likelihood, a strategy is implemented which repeatably uses random allocation of the studies to the mixture components as choice of initial values. The likelihoods under the contrast-based and arm-based approaches are compared and differences are highlighted. We use simulations to assess the performance of these two methods. Under the design of sampling studies with nested treatment groups, the results show that the nonparametric mixture model based on the contrast-based approach is more appropriate in terms of model selection criteria such as AIC and BIC. Under the arm-based design the results from the arm-based model performs well although in some cases it is also outperformed by the contrast-based model. Comparisons of the estimators are provided in terms of bias and mean squared error. Also included in the comparison is the mixed Poisson regression model as well as the classical DerSimonian-Laird model (using the Mantel-Haenszel estimator for the common effect). Using simulation, estimating effect heterogeneity in the case of the contrast-based method appears to behave better than the compared methods although differences become negligible for large within-study sample sizes. We illustrate the methodologies using several meta-analytic data sets in medicine.

Finding the best subgroup with differential treatment effect with multiple outcomes.

Precision medicine aims to identify specific patient subgroups that may benefit the most from a particular treatment than the whole population. Existing definitions for the best subgroup in subgroup analysis are based on a single outcome and do not consider multiple outcomes; specifically, outcomes of different types. In this article, we introduce a definition for the best subgroup under a multiple-outcome setting with continuous, binary, and censored time-to-event outcomes. Our definition provides a trade-off between the subgroup size and the conditional average treatment effects (CATE) in the subgroup with respect to each of the outcomes while taking the relative contribution of the outcomes into account. We conduct simulations to illustrate the proposed definition. By examining the outcomes of urinary tract infection and renal scarring in the RIVUR clinical trial, we identify a subgroup of children that would benefit the most from long-term antimicrobial prophylaxis.

genRCT: a statistical analysis framework for generalizing RCT findings to real-world population.

When evaluating the real-world treatment effect, the analysis based on randomized clinical trials (RCTs) often introduces generalizability bias due to the difference in risk factors between the trial participants and the real-world patient population. This problem of lack of generalizability associated with the RCT-only analysis can be addressed by leveraging observational studies with large sample sizes that are representative of the real-world population. A set of novel statistical methods, termed "genRCT", for improving the generalizability of the trial has been developed using calibration weighting, which enforces the covariates balance between the RCT and observational study. This paper aims to review statistical methods for generalizing the RCT findings by harnessing information from large observational studies that represent real-world patients. Specifically, we discuss the choices of data sources and variables to meet key theoretical assumptions and principles. We introduce and compare estimation methods for continuous, binary, and survival endpoints. We showcase the use of the R package genRCT through a case study that estimates the average treatment effect of adjuvant chemotherapy for the stage 1B non-small cell lung patients represented by a large cancer registry.

A new method for clustered survival data: Estimation of treatment effect heterogeneity and variable selection.

We recently developed a new method random-intercept accelerated failure time model with Bayesian additive regression trees (riAFT-BART) to draw causal inferences about population treatment effect on patient survival from clustered and censored survival data while accounting for the multilevel data structure. The practical utility of this method goes beyond the estimation of population average treatment effect. In this work, we exposit how riAFT-BART can be used to solve two important statistical questions with clustered survival data: estimating the treatment effect heterogeneity and variable selection. Leveraging the likelihood-based machine learning, we describe a way in which we can draw posterior samples of the individual survival treatment effect from riAFT-BART model runs, and use the drawn posterior samples to perform an exploratory treatment effect heterogeneity analysis to identify subpopulations who may experience differential treatment effects than population average effects. There is sparse literature on methods for variable selection among clustered and censored survival data, particularly ones using flexible modeling techniques. We propose a permutation-based approach using the predictor's variable inclusion proportion supplied by the riAFT-BART model for variable selection. To address the missing data issue frequently encountered in health databases, we propose a strategy to combine bootstrap imputation and riAFT-BART for variable selection among incomplete clustered survival data. We conduct an expansive simulation study to examine the practical operating characteristics of our proposed methods, and provide empirical evidence that our proposed methods perform better than several existing methods across a wide range of data scenarios. Finally, we demonstrate the methods via a case study of predictors for in-hospital mortality among severe COVID-19 patients and estimating the heterogeneous treatment effects of three COVID-specific medications. The methods developed in this work are readily available in the R ${\textsf {R}}$ package riAFTBART $\textsf {riAFTBART}$ .

What drives the relationship between digitalization and energy demand? Exploring heterogeneity in German manufacturing firms.

The growing use of information and communication technologies (ICT) has the potential to increase productivity and improve energy efficiency. However, digital technologies also consume energy, resulting in a complex relationship between digitalization and energy demand and an uncertain net effect. To steer digital transformation towards sustainability, it is crucial to understand the conditions under which digital technologies increase or decrease firm-level energy consumption. This study examines the drivers of this relationship, focusing on German manufacturing firms and leveraging comprehensive administrative panel data from 2009 to 2017, analyzed using the Generalized Random Forest algorithm. Our results reveal that the relationship between digitalization and energy use at the firm level is heterogeneous. However, we find that digitalization more frequently increases energy use, mainly driven by a rise in electricity consumption. This increase is lower in energy-intensive industries and higher in markets with low competition. Smaller firms in structurally weak regions show higher energy consumption growth than larger firms in economically stronger regions. Our study contributes to the literature by using a non-parametric method to identify specific firm-level and external characteristics that influence the impact of digital technologies on energy demand, highlighting the need for carefully designed digitalization policies to achieve climate goals.

Uncovering Heterogeneous Associations Between Disaster-Related Trauma and Subsequent Functional Limitations: A Machine-Learning Approach.

This study examined heterogeneity in the association between disaster-related home loss and functional limitations of older adults, and identified characteristics of vulnerable subpopulations. Data were from a prospective cohort study of Japanese older survivors of the 2011 Japan Earthquake. Complete home loss was objectively assessed. Outcomes in 2013 (n = 3,350) and 2016 (n = 2,664) included certified physical disability levels, self-reported activities of daily living, and instrumental activities of daily living. We estimated population average associations between home loss and functional limitations via targeted maximum likelihood estimation with SuperLearning and its heterogeneity via the generalized random forest algorithm. We adjusted for 55 characteristics of survivors from the baseline survey conducted 7 months before the disaster. While home loss was consistently associated with increased functional limitations on average, there was evidence of effect heterogeneity for all outcomes. Comparing the most and least vulnerable groups, the most vulnerable group tended to be older, not married, living alone, and not working, with preexisting health problems before the disaster. Individuals who were less educated but had higher income also appeared vulnerable for some outcomes. Our inductive approach for effect heterogeneity using machine learning algorithm uncovered large and complex heterogeneity in postdisaster functional limitations among Japanese older survivors.

Methods for Integrating Trials and Non-experimental Data to Examine Treatment Effect Heterogeneity.

Estimating treatment effects conditional on observed covariates can improve the ability to tailor treatments to particular individuals. Doing so effectively requires dealing with potential confounding, and also enough data to adequately estimate effect moderation. A recent influx of work has looked into estimating treatment effect heterogeneity using data from multiple randomized controlled trials and/or observational datasets. With many new methods available for assessing treatment effect heterogeneity using multiple studies, it is important to understand which methods are best used in which setting, how the methods compare to one another, and what needs to be done to continue progress in this field. This paper reviews these methods broken down by data setting: aggregate-level data, federated learning, and individual participant-level data. We define the conditional average treatment effect and discuss differences between parametric and nonparametric estimators, and we list key assumptions, both those that are required within a single study and those that are necessary for data combination. After describing existing approaches, we compare and contrast them and reveal open areas for future research. This review demonstrates that there are many possible approaches for estimating treatment effect heterogeneity through the combination of datasets, but that there is substantial work to be done to compare these methods through case studies and simulations, extend them to different settings, and refine them to account for various challenges present in real data.

Improved inference for doubly robust estimators of heterogeneous treatment effects.

We propose a doubly robust approach to characterizing treatment effect heterogeneity in observational studies. We develop a frequentist inferential procedure that utilizes posterior distributions for both the propensity score and outcome regression models to provide valid inference on the conditional average treatment effect even when high-dimensional or nonparametric models are used. We show that our approach leads to conservative inference in finite samples or under model misspecification and provides a consistent variance estimator when both models are correctly specified. In simulations, we illustrate the utility of these results in difficult settings such as high-dimensional covariate spaces or highly flexible models for the propensity score and outcome regression. Lastly, we analyze environmental exposure data from NHANES to identify how the effects of these exposures vary by subject-level characteristics.

Efficient targeted learning of heterogeneous treatment effects for multiple subgroups.

In biomedical science, analyzing treatment effect heterogeneity plays an essential role in assisting personalized medicine. The main goals of analyzing treatment effect heterogeneity include estimating treatment effects in clinically relevant subgroups and predicting whether a patient subpopulation might benefit from a particular treatment. Conventional approaches often evaluate the subgroup treatment effects via parametric modeling and can thus be susceptible to model mis-specifications. In this paper, we take a model-free semiparametric perspective and aim to efficiently evaluate the heterogeneous treatment effects of multiple subgroups simultaneously under the one-step targeted maximum-likelihood estimation (TMLE) framework. When the number of subgroups is large, we further expand this path of research by looking at a variation of the one-step TMLE that is robust to the presence of small estimated propensity scores in finite samples. From our simulations, our method demonstrates substantial finite sample improvements compared to conventional methods. In a case study, our method unveils the potential treatment effect heterogeneity of rs12916-T allele (a proxy for statin usage) in decreasing Alzheimer's disease risk.

Testing weak nulls in matched observational studies.

We develop sensitivity analyses for the sample average treatment effect in matched observational studies while allowing unit-level treatment effects to vary. The methods may be applied to studies using any optimal without-replacement matching algorithm. In contrast to randomized experiments and to paired observational studies, we show for general matched designs that over a large class of test statistics, any procedure bounding the worst-case expectation while allowing for arbitrary effect heterogeneity must be unnecessarily conservative if treatment effects are actually constant across individuals. We present a sensitivity analysis which bounds the worst-case expectation while allowing for effect heterogeneity, and illustrate why it is generally conservative if effects are constant. An alternative procedure is presented that is asymptotically sharp if treatment effects are constant, and that is valid for testing the sample average effect under additional restrictions which may be deemed benign by practitioners. Simulations demonstrate that this alternative procedure results in a valid sensitivity analysis for the weak null hypothesis under a host of reasonable data-generating processes. The procedures allow practitioners to assess robustness of estimated sample average treatment effects to hidden bias while allowing for effect heterogeneity in matched observational studies.

Subgroup analysis using Bernoulli-gated hierarchical mixtures of experts models.

When it is suspected that the treatment effect may only be strong for certain subpopulations, identifying the baseline covariate profiles of subgroups who benefit from such a treatment is of key importance. In this paper, we propose an approach for subgroup analysis by firstly introducing Bernoulli-gated hierarchical mixtures of experts (BHME), a binary-tree structured model to explore heterogeneity of the underlying distribution. We show identifiability of the BHME model and develop an EM-based maximum likelihood method for optimization. The algorithm automatically determines a partition structure with optimal prediction but possibly suboptimal in identifying treatment effect heterogeneity. We then suggest a testing-based postscreening step to further capture effect heterogeneity. Simulation results show that our approach outperforms competing methods on discovery of differential treatment effects and other related metrics. We finally apply the proposed approach to a real dataset from the Tennessee's Student/Teacher Achievement Ratio project.

Estimating individualized treatment effects from randomized controlled trials: a simulation study to compare risk-based approaches.

BACKGROUND: Baseline outcome risk can be an important determinant of absolute treatment benefit and has been used in guidelines for "personalizing" medical decisions. We compared easily applicable risk-based methods for optimal prediction of individualized treatment effects. METHODS: We simulated RCT data using diverse assumptions for the average treatment effect, a baseline prognostic index of risk, the shape of its interaction with treatment (none, linear, quadratic or non-monotonic), and the magnitude of treatment-related harms (none or constant independent of the prognostic index). We predicted absolute benefit using: models with a constant relative treatment effect; stratification in quarters of the prognostic index; models including a linear interaction of treatment with the prognostic index; models including an interaction of treatment with a restricted cubic spline transformation of the prognostic index; an adaptive approach using Akaike's Information Criterion. We evaluated predictive performance using root mean squared error and measures of discrimination and calibration for benefit. RESULTS: The linear-interaction model displayed optimal or close-to-optimal performance across many simulation scenarios with moderate sample size (N = 4,250; ~ 785 events). The restricted cubic splines model was optimal for strong non-linear deviations from a constant treatment effect, particularly when sample size was larger (N = 17,000). The adaptive approach also required larger sample sizes. These findings were illustrated in the GUSTO-I trial. CONCLUSIONS: An interaction between baseline risk and treatment assignment should be considered to improve treatment effect predictions.

Principled estimation and evaluation of treatment effect heterogeneity: A case study application to dabigatran for patients with atrial fibrillation.

OBJECTIVE: To apply the latest guidance for estimating and evaluating heterogeneous treatment effects (HTEs) in an end-to-end case study of the Long-term Anticoagulation Therapy (RE-LY) trial, and summarize the main takeaways from applying state-of-the-art metalearners and novel evaluation metrics in-depth to inform their applications to personalized care in biomedical research. METHODS: Based on the characteristics of the RE-LY data, we selected four metalearners (S-learner with Lasso, X-learner with Lasso, R-learner with random survival forest and Lasso, and causal survival forest) to estimate the HTEs of dabigatran. For the outcomes of (1) stroke or systemic embolism and (2) major bleeding, we compared dabigatran 150 mg, dabigatran 110 mg, and warfarin. We assessed the overestimation of treatment heterogeneity by the metalearners via a global null analysis and their discrimination and calibration ability using two novel metrics: rank-weighted average treatment effects (RATE) and estimated calibration error for treatment heterogeneity. Finally, we visualized the relationships between estimated treatment effects and baseline covariates using partial dependence plots. RESULTS: The RATE metric suggested that either the applied metalearners had poor performance of estimating HTEs or there was no treatment heterogeneity for either the stroke/SE or major bleeding outcome of any treatment comparison. Partial dependence plots revealed that several covariates had consistent relationships with the treatment effects estimated by multiple metalearners. The applied metalearners showed differential performance across outcomes and treatment comparisons, and the X- and R-learners yielded smaller calibration errors than the others. CONCLUSIONS: HTE estimation is difficult, and a principled estimation and evaluation process is necessary to provide reliable evidence and prevent false discoveries. We have demonstrated how to choose appropriate metalearners based on specific data properties, applied them using the off-the-shelf implementation tool survlearners, and evaluated their performance using recently defined formal metrics. We suggest that clinical implications should be drawn based on the common trends across the applied metalearners.

Variation in the infant health effects of the women, infants, and children program by predicted risk using novel machine learning methods.

The Special Supplemental Nutrition Program for Women, Infants, and Children (WIC) has an extensive literature documenting positive effects on infant health outcomes, specifically preterm birth, low birthweight, small size for gestational age, and infant mortality. However, existing studies focus on average effects for these relatively infrequent outcomes, thus providing no evidence for how WIC affects those at greatest risk of negative infant health outcomes. Our study focuses on documenting how WIC's infant health effects vary by level of risk. In doing so, we leverage a uniquely rich database describing maternal and infant outcomes and risk factors. Additionally, we use high dimensional data to generate predictions of risk and combine these predictions with the novel double machine learning method to stratify the effects of WIC by predicted risk. Our estimates of WIC's average treatment effects align with those in the existing literature. More importantly, we document significant variation in the effects of WIC on infant health by predicted risk level. Our results show that WIC is most beneficial among those at greatest risk of poor outcomes.