Systematically missing data in causally interpretable meta-analysis.

Causally interpretable meta-analysis combines information from a collection of randomized controlled trials to estimate treatment effects in a target population in which experimentation may not be possible but from which covariate information can be obtained. In such analyses, a key practical challenge is the presence of systematically missing data when some trials have collected data on one or more baseline covariates, but other trials have not, such that the covariate information is missing for all participants in the latter. In this article, we provide identification results for potential (counterfactual) outcome means and average treatment effects in the target population when covariate data are systematically missing from some of the trials in the meta-analysis. We propose three estimators for the average treatment effect in the target population, examine their asymptotic properties, and show that they have good finite-sample performance in simulation studies. We use the estimators to analyze data from two large lung cancer screening trials and target population data from the National Health and Nutrition Examination Survey (NHANES). To accommodate the complex survey design of the NHANES, we modify the methods to incorporate survey sampling weights and allow for clustering.

Tree-based subgroup discovery using electronic health record data: heterogeneity of treatment effects for DTG-containing therapies.

The rich longitudinal individual level data available from electronic health records (EHRs) can be used to examine treatment effect heterogeneity. However, estimating treatment effects using EHR data poses several challenges, including time-varying confounding, repeated and temporally non-aligned measurements of covariates, treatment assignments and outcomes, and loss-to-follow-up due to dropout. Here, we develop the subgroup discovery for longitudinal data algorithm, a tree-based algorithm for discovering subgroups with heterogeneous treatment effects using longitudinal data by combining the generalized interaction tree algorithm, a general data-driven method for subgroup discovery, with longitudinal targeted maximum likelihood estimation. We apply the algorithm to EHR data to discover subgroups of people living with human immunodeficiency virus who are at higher risk of weight gain when receiving dolutegravir (DTG)-containing antiretroviral therapies (ARTs) versus when receiving non-DTG-containing ARTs.

Early Mortality After the First Dose of COVID-19 Vaccination: A Target Trial Emulation.

BACKGROUND: Vaccine hesitancy persists alongside concerns about the safety of coronavirus disease 2019 (COVID-19) vaccines. We aimed to examine the effect of COVID-19 vaccination on risk of death among US veterans. METHODS: We conducted a target trial emulation to estimate and compare risk of death up to 60 days under two COVID-19 vaccination strategies: vaccination within 7 days of enrollment versus no vaccination through follow-up. The study cohort included individuals aged ≥18 years enrolled in the Veterans Health Administration system and eligible to receive a COVID-19 vaccination according to guideline recommendations from 1 March 2021 through 1 July 2021. The outcomes of interest included deaths from any cause and excluding a COVID-19 diagnosis. Observations were cloned to both treatment strategies, censored, and weighted to estimate per-protocol effects. RESULTS: We included 3 158 507 veterans. Under the vaccination strategy, 364 993 received vaccine within 7 days. At 60 days, there were 156 deaths per 100 000 veterans under the vaccination strategy versus 185 deaths under the no vaccination strategy, corresponding to an absolute risk difference of -25.9 (95% confidence limit [CL], -59.5 to 2.7) and relative risk of 0.86 (95% CL, .7 to 1.0). When those with a COVID-19 infection in the first 60 days were censored, the absolute risk difference was -20.6 (95% CL, -53.4 to 16.0) with a relative risk of 0.88 (95% CL, .7 to 1.1). CONCLUSIONS: Vaccination against COVID-19 was associated with a lower but not statistically significantly different risk of death in the first 60 days. These results agree with prior scientific knowledge suggesting vaccination is safe with the potential for substantial health benefits.

Invited Commentary: Combining Information to Answer Epidemiologic Questions About a Target Population.

Epidemiologists are attempting to address research questions of increasing complexity by developing novel methods for combining information from diverse sources. Cole et al. (Am J Epidemiol. 2023;192(3)467-474) provide 2 examples of the process of combining information to draw inferences about a population proportion. In this commentary, we consider combining information to learn about a target population as an epidemiologic activity and distinguish it from more conventional meta-analyses. We examine possible rationales for combining information and discuss broad methodological considerations, with an emphasis on study design, assumptions, and sources of uncertainty.

Evidence Synthesis for Complex Interventions Using Meta-Regression Models.

A goal of evidence synthesis for trials of complex interventions is to inform the design or implementation of novel versions of complex interventions by predicting expected outcomes with each intervention version. Conventional aggregate data meta-analyses of studies comparing complex interventions have limited ability to provide such information. We argue that evidence synthesis for trials of complex interventions should forgo aspirations of estimating causal effects and instead model the response surface of study results to 1) summarize the available evidence and 2) predict the average outcomes of future studies or in new settings. We illustrate this modeling approach using data from a systematic review of diabetes quality improvement (QI) interventions involving at least 1 of 12 QI strategy components. We specify a series of meta-regression models to assess the association of specific components with the posttreatment outcome mean and compare the results to conventional meta-analysis approaches. Compared with conventional approaches, modeling the response surface of study results can better reflect the associations between intervention components and study characteristics with the posttreatment outcome mean. Modeling study results using a response surface approach offers a useful and feasible goal for evidence synthesis of complex interventions that rely on aggregate data.

Learning about treatment effects in a new target population under transportability assumptions for relative effect measures.

Investigators often believe that relative effect measures conditional on covariates, such as risk ratios and mean ratios, are "transportable" across populations. Here, we examine the identification of causal effects in a target population using an assumption that conditional relative effect measures are transportable from a trial to the target population. We show that transportability for relative effect measures is largely incompatible with transportability for difference effect measures, unless the treatment has no effect on average or one is willing to make even stronger transportability assumptions that imply the transportability of both relative and difference effect measures. We then describe how marginal (population-averaged) causal estimands in a target population can be identified under the assumption of transportability of relative effect measures, when we are interested in the effectiveness of a new experimental treatment in a target population where the only treatment in use is the control treatment evaluated in the trial. We extend these results to consider cases where the control treatment evaluated in the trial is only one of the treatments in use in the target population, under an additional partial exchangeability assumption in the target population (i.e., an assumption of no unmeasured confounding in the target population with respect to potential outcomes under the control treatment in the trial). We also develop identification results that allow for the covariates needed for transportability of relative effect measures to be only a small subset of the covariates needed to control confounding in the target population. Last, we propose estimators that can be easily implemented in standard statistical software and illustrate their use using data from a comprehensive cohort study of stable ischemic heart disease.

Causal Inference About the Effects of Interventions From Observational Studies in Medical Journals.

Importance: Many medical journals, including JAMA, restrict the use of causal language to the reporting of randomized clinical trials. Although well-conducted randomized clinical trials remain the preferred approach for answering causal questions, methods for observational studies have advanced such that causal interpretations of the results of well-conducted observational studies may be possible when strong assumptions hold. Furthermore, observational studies may be the only practical source of information for answering some questions about the causal effects of medical or policy interventions, can support the study of interventions in populations and settings that reflect practice, and can help identify interventions for further experimental investigation. Identifying opportunities for the appropriate use of causal language when describing observational studies is important for communication in medical journals. Observations: A structured approach to whether and how causal language may be used when describing observational studies would enhance the communication of research goals, support the assessment of assumptions and design and analytic choices, and allow for more clear and accurate interpretation of results. Building on the extensive literature on causal inference across diverse disciplines, we suggest a framework for observational studies that aim to provide evidence about the causal effects of interventions based on 6 core questions: what is the causal question; what quantity would, if known, answer the causal question; what is the study design; what causal assumptions are being made; how can the observed data be used to answer the causal question in principle and in practice; and is a causal interpretation of the analyses tenable? Conclusions and Relevance: Adoption of the proposed framework to identify when causal interpretation is appropriate in observational studies promises to facilitate better communication between authors, reviewers, editors, and readers. Practical implementation will require cooperation between editors, authors, and reviewers to operationalize the framework and evaluate its effect on the reporting of empirical research.

Transporting a Prediction Model for Use in a New Target Population.

We considered methods for transporting a prediction model for use in a new target population, both when outcome and covariate data for model development are available from a source population that has a different covariate distribution compared with the target population and when covariate data (but not outcome data) are available from the target population. We discuss how to tailor the prediction model to account for differences in the data distribution between the source population and the target population. We also discuss how to assess the model's performance (e.g., by estimating the mean squared prediction error) in the target population. We provide identifiability results for measures of model performance in the target population for a potentially misspecified prediction model under a sampling design where the source and the target population samples are obtained separately. We introduce the concept of prediction error modifiers that can be used to reason about tailoring measures of model performance to the target population. We illustrate the methods in simulated data and apply them to transport a prediction model for lung cancer diagnosis from the National Lung Screening Trial to the nationally representative target population of trial-eligible individuals in the National Health and Nutrition Examination Survey.

Evaluating Model Specification When Using the Parametric G-Formula in the Presence of Censoring.

The noniterative conditional expectation (NICE) parametric g-formula can be used to estimate the causal effect of sustained treatment strategies. In addition to identifiability conditions, the validity of the NICE parametric g-formula generally requires the correct specification of models for time-varying outcomes, treatments, and confounders at each follow-up time point. An informal approach for evaluating model specification is to compare the observed distributions of the outcome, treatments, and confounders with their parametric g-formula estimates under the "natural course." In the presence of loss to follow-up, however, the observed and natural-course risks can differ even if the identifiability conditions of the parametric g-formula hold and there is no model misspecification. Here, we describe 2 approaches for evaluating model specification when using the parametric g-formula in the presence of censoring: 1) comparing factual risks estimated by the g-formula with nonparametric Kaplan-Meier estimates and 2) comparing natural-course risks estimated by inverse probability weighting with those estimated by the g-formula. We also describe how to correctly compute natural-course estimates of time-varying covariate means when using a computationally efficient g-formula algorithm. We evaluate the proposed methods via simulation and implement them to estimate the effects of dietary interventions in 2 cohort studies.

Using Trial and Observational Data to Assess Effectiveness: Trial Emulation, Transportability, Benchmarking, and Joint Analysis.

Comparisons between randomized trial analyses and observational analyses that attempt to address similar research questions have generated many controversies in epidemiology and the social sciences. There has been little consensus on when such comparisons are reasonable, what their implications are for the validity of observational analyses, or whether trial and observational analyses can be integrated to address effectiveness questions. Here, we consider methods for using observational analyses to complement trial analyses when assessing treatment effectiveness. First, we review the framework for designing observational analyses that emulate target trials and present an evidence map of its recent applications. We then review approaches for estimating the average treatment effect in the target population underlying the emulation: using observational analyses of the emulation data alone; and using transportability analyses to extend inferences from a trial to the target population. We explain how comparing treatment effect estimates from the emulation against those from the trial can provide evidence on whether observational analyses can be trusted to deliver valid estimates of effectiveness - a process we refer to as benchmarking - and, in some cases, allow the joint analysis of the trial and observational data. We illustrate different approaches using a simplified example of a pragmatic trial and its emulation in registry data. We conclude that synthesizing trial and observational data - in transportability, benchmarking, or joint analyses - can leverage their complementary strengths to enhance learning about comparative effectiveness, through a process combining quantitative methods and epidemiological judgements.

Generalizability of the Necrotizing Enterocolitis Surgery Trial to the Target Population of Eligible Infants.

OBJECTIVE: The objective of this study was to evaluate whether infants randomized in the Eunice Kennedy Shriver National Institute of Child Health and Human Development Neonatal Research Network Necrotizing Enterocolitis Surgery Trial differed from eligible infants and whether differences affected the generalizability of trial results. STUDY DESIGN: Secondary analysis of infants enrolled in Necrotizing Enterocolitis Surgery Trial (born 2010-2017, with follow-up through 2019) at 20 US academic medical centers and an observational data set of eligible infants through 2013. Infants born ≤1000 g and diagnosed with necrotizing enterocolitis or spontaneous intestinal perforation requiring surgical intervention at ≤8 weeks were eligible. The target population included trial-eligible infants (randomized and nonrandomized) born during the first half of the study with available detailed preoperative data. Using model-based weighting methods, we estimated the effect of initial laparotomy vs peritoneal drain had the target population been randomized. RESULTS: The trial included 308 randomized infants. The target population included 382 (156 randomized and 226 eligible, non-randomized) infants. Compared with the target population, fewer randomized infants had necrotizing enterocolitis (31% vs 47%) or died before discharge (27% vs 41%). However, incidence of the primary composite outcome, death or neurodevelopmental impairment, was similar (69% vs 72%). Effect estimates for initial laparotomy vs drain weighted to the target population were largely unchanged from the original trial after accounting for preoperative diagnosis of necrotizing enterocolitis (adjusted relative risk [95% CI]: 0.85 [0.71-1.03] in target population vs 0.81 [0.64-1.04] in trial) or spontaneous intestinal perforation (1.02 [0.79-1.30] vs 1.11 [0.95-1.31]). CONCLUSION: Despite differences between randomized and eligible infants, estimated treatment effects in the trial and target population were similar, supporting the generalizability of trial results. TRIAL REGISTRATION: ClinicalTrials.gov ID: NCT01029353.

Estimating the area under the ROC curve when transporting a prediction model to a target population.

We propose methods for estimating the area under the receiver operating characteristic (ROC) curve (AUC) of a prediction model in a target population that differs from the source population that provided the data used for original model development. If covariates that are associated with model performance, as measured by the AUC, have a different distribution in the source and target populations, then AUC estimators that only use data from the source population will not reflect model performance in the target population. Here, we provide identification results for the AUC in the target population when outcome and covariate data are available from the sample of the source population, but only covariate data are available from the sample of the target population. In this setting, we propose three estimators for the AUC in the target population and show that they are consistent and asymptotically normal. We evaluate the finite-sample performance of the estimators using simulations and use them to estimate the AUC in a nationally representative target population from the National Health and Nutrition Examination Survey for a lung cancer risk prediction model developed using source population data from the National Lung Screening Trial.

Efficient and robust methods for causally interpretable meta-analysis: Transporting inferences from multiple randomized trials to a target population.

We present methods for causally interpretable meta-analyses that combine information from multiple randomized trials to draw causal inferences for a target population of substantive interest. We consider identifiability conditions, derive implications of the conditions for the law of the observed data, and obtain identification results for transporting causal inferences from a collection of independent randomized trials to a new target population in which experimental data may not be available. We propose an estimator for the potential outcome mean in the target population under each treatment studied in the trials. The estimator uses covariate, treatment, and outcome data from the collection of trials, but only covariate data from the target population sample. We show that it is doubly robust in the sense that it is consistent and asymptotically normal when at least one of the models it relies on is correctly specified. We study the finite sample properties of the estimator in simulation studies and demonstrate its implementation using data from a multicenter randomized trial.

Sensitivity analysis using bias functions for studies extending inferences from a randomized trial to a target population.

Extending (i.e., generalizing or transporting) causal inferences from a randomized trial to a target population requires assumptions that randomized and nonrandomized individuals are exchangeable conditional on baseline covariates. These assumptions are made on the basis of background knowledge, which is often uncertain or controversial, and need to be subjected to sensitivity analysis. We present simple methods for sensitivity analyses that directly parameterize violations of the assumptions using bias functions and do not require detailed background knowledge about specific unknown or unmeasured determinants of the outcome or modifiers of the treatment effect. We show how the methods can be applied to non-nested trial designs, where the trial data are combined with a separately obtained sample of nonrandomized individuals, as well as to nested trial designs, where the trial is embedded within a cohort sampled from the target population.

Can Observational Analyses of Routinely Collected Data Emulate Randomized Trials? Design and Feasibility of the Observational Patient Evidence for Regulatory Approval Science and Understanding Disease Project.

OBJECTIVES: The Observational Patient Evidence for Regulatory Approval Science and Understanding Disease (OPERAND) project examines whether real-world data (RWD) can be used to inform regulatory decision making. METHODS: OPERAND evaluates whether observational analyses using RWD to emulate index trials can produce effect estimates similar to those of the trials and examines the impact of relaxing the eligibility criteria of the observational analyses to obtain samples that more closely match the real-world populations receiving the treatments. In OPERAND, 2 research teams independently attempt to emulate the ROCKET Atrial Fibrillation and LEAD-2 trials using OptumLabs data. This article describes the design of the project, summarizes the approaches of the 2 research teams, and presents feasibility results for 2 emulations using new-user designs. RESULTS: There were differences in the teams' conceptualizations of the emulation, design decisions for cohort identification, and resulting RWD cohorts. These differences occurred even though both teams were guided by the same index trials and had access to the same source of RWD. CONCLUSIONS: Reasonable alternative design and analysis approaches may be taken to answer the same research question, even when attempting to emulate the same index trial. Researcher decision making is an understudied and potentially important source of variability across RWD analyses.

Regression-based estimation of heterogeneous treatment effects when extending inferences from a randomized trial to a target population.

Most work on extending (generalizing or transporting) inferences from a randomized trial to a target population has focused on estimating average treatment effects (i.e., averaged over the target population's covariate distribution). Yet, in the presence of strong effect modification by baseline covariates, the average treatment effect in the target population may be less relevant for guiding treatment decisions. Instead, the conditional average treatment effect (CATE) as a function of key effect modifiers may be a more useful estimand. Recent work on estimating target population CATEs using baseline covariate, treatment, and outcome data from the trial and covariate data from the target population only allows for the examination of heterogeneity over distinct subgroups. We describe flexible pseudo-outcome regression modeling methods for estimating target population CATEs conditional on discrete or continuous baseline covariates when the trial is embedded in a sample from the target population (i.e., in nested trial designs). We construct pointwise confidence intervals for the CATE at a specific value of the effect modifiers and uniform confidence bands for the CATE function. Last, we illustrate the methods using data from the Coronary Artery Surgery Study (CASS) to estimate CATEs given history of myocardial infarction and baseline ejection fraction value in the target population of all trial-eligible patients with stable ischemic heart disease.

Assessing the representativeness of cluster randomized trials: Evidence from two large pragmatic trials in United States nursing homes.

BACKGROUND/AIMS: When the randomized clusters in a cluster randomized trial are selected based on characteristics that influence treatment effectiveness, results from the trial may not be directly applicable to the target population. We used data from two large nursing home-based pragmatic cluster randomized trials to compare nursing home and resident characteristics in randomized facilities to eligible non-randomized and ineligible facilities. METHODS: We linked data from the high-dose influenza vaccine trial and the Music & Memory Pragmatic TRIal for Nursing Home Residents with ALzheimer's Disease (METRICaL) to nursing home assessments and Medicare fee-for-service claims. The target population for the high-dose trial comprised Medicare-certified nursing homes; the target population for the METRICaL trial comprised nursing homes in one of four US-based nursing home chains. We used standardized mean differences to compare facility and individual characteristics across the three groups and logistic regression to model the probability of nursing home trial participation. RESULTS: In the high-dose trial, 4476 (29%) of the 15,502 nursing homes in the target population were eligible for the trial, of which 818 (18%) were randomized. Of the 1,361,122 residents, 91,179 (6.7%) were residents of randomized facilities, 463,703 (34.0%) of eligible non-randomized facilities, and 806,205 (59.3%) of ineligible facilities. In the METRICaL trial, 160 (59%) of the 270 nursing homes in the target population were eligible for the trial, of which 80 (50%) were randomized. Of the 20,262 residents, 973 (34.4%) were residents of randomized facilities, 7431 (36.7%) of eligible non-randomized facilities, and 5858 (28.9%) of ineligible facilities. In the high-dose trial, randomized facilities differed from eligible non-randomized and ineligible facilities by the number of beds (132.5 vs 145.9 and 91.9, respectively), for-profit status (91.8% vs 66.8% and 68.8%), belonging to a nursing home chain (85.8% vs 49.9% and 54.7%), and presence of a special care unit (19.8% vs 25.9% and 14.4%). In the METRICaL trial randomized facilities differed from eligible non-randomized and ineligible facilities by the number of beds (103.7 vs 110.5 and 67.0), resource-poor status (4.6% vs 10.0% and 18.8%), and presence of a special care unit (26.3% vs 33.8% and 10.9%). In both trials, the characteristics of residents in randomized facilities were similar across the three groups. CONCLUSION: In both trials, facility-level characteristics of randomized nursing homes differed considerably from those of eligible non-randomized and ineligible facilities, while there was little difference in resident-level characteristics across the three groups. Investigators should assess the characteristics of clusters that participate in cluster randomized trials, not just the individuals within the clusters, when examining the applicability of trial results beyond participating clusters.

Medicare eligibility and healthcare access, affordability, and financial strain for low- and higher-income adults in the United States: A regression discontinuity analysis.

BACKGROUND: US policymakers are debating whether to expand the Medicare program by lowering the age of eligibility. The goal of this study was to determine the association of Medicare eligibility and enrollment with healthcare access, affordability, and financial strain from medical bills in a contemporary population of low- and higher-income adults in the US. METHODS AND FINDINGS: We used cross-sectional data from the National Health Interview Survey (2019) to examine the association of Medicare eligibility and enrollment with outcomes by income status using a local randomization-based regression discontinuity approach. After weighting to account for survey sampling, the low-income group consisted of 1,660,188 adults age 64 years and 1,488,875 adults age 66 years, with similar baseline characteristics, including distribution of sex (59.2% versus 59.7% female) and education (10.8% versus 12.5% with bachelor's degree or higher). The higher-income group consisted of 2,110,995 adults age 64 years and 2,167,676 adults age 66 years, with similar distribution of baseline characteristics, including sex (40.0% versus 49.4% female) and education (41.0% versus 41.6%). The share of adults age 64 versus 66 years enrolled in Medicare differed within low-income (27.6% versus 87.8%, p < 0.001) and higher-income groups (8.0% versus 85.9%, p < 0.001). Medicare eligibility at 65 years was associated with a decreases in the percentage of low-income adults who delayed (14.7% to 6.2%; -8.5% [95% CI, -14.7%, -2.4%], P = 0.007) or avoided medical care (15.5% to 5.9%; -9.6% [-15.9%, -3.2%], P = 0.003) due to costs, and a larger decrease in the percentage who were worried about (66.5% to 51.1%; -15.4% [-25.4%, -5.4%], P = 0.003) or had problems (33.9% to 20.6%; -13.3% [-23.0%, -3.6%], P = 0.007) paying medical bills. In contrast, there were no significant associations between Medicare eligibility and measures of cost-related barriers to medication use. For higher-income adults, there was a large decrease in worrying about paying medical bills (40.5% to 27.5%; -13.0% [-21.4%, -4.5%], P = 0.003), a more modest decrease in avoiding medical care due to cost (3.5% to 0.6%; -2.9% [-5.3%, -0.5%], P = 0.02), and no significant association between eligibility and other measures of healthcare access and affordability. All estimates were stronger when examining the association of Medicare enrollment with outcomes for low and higher-income adults. Additional analyses that adjusted for clinical comorbidities and employment status were largely consistent with the main findings, as were analyses stratified by levels of educational attainment. Study limitations include the assumption adults age 64 and 66 would have similar outcomes if both groups were eligible for Medicare or if eligibility were withheld from both. CONCLUSIONS: Medicare eligibility and enrollment at age 65 years were associated with improvements in healthcare access, affordability, and financial strain in low-income adults and, to a lesser extent, in higher-income adults. Our findings provide evidence that lowering the age of eligibility for Medicare may improve health inequities in the US.

Using Numerical Methods to Design Simulations: Revisiting the Balancing Intercept.

In this paper, we consider methods for generating draws of a binary random variable whose expectation conditional on covariates follows a logistic regression model with known covariate coefficients. We examine approximations for finding a "balancing intercept," that is, a value for the intercept of the logistic model that leads to a desired marginal expectation for the binary random variable. We show that a recently proposed analytical approximation can produce inaccurate results, especially when targeting more extreme marginal expectations or when the linear predictor of the regression model has high variance. We then formulate the balancing intercept as a solution to an integral equation, implement a numerical approximation for solving the equation based on Monte Carlo methods, and show that the approximation works well in practice. Our approach to the basic problem of the balancing intercept provides an example of a broadly applicable strategy for formulating and solving problems that arise in the design of simulation studies used to evaluate or teach epidemiologic methods.

Estimating subgroup effects in generalizability and transportability analyses.

Methods for extending - generalizing or transporting - inferences from a randomized trial to a target population involve conditioning on a large set of covariates that is sufficient for rendering the randomized and non-randomized groups exchangeable. Yet, decision-makers are often interested in examining treatment effects in subgroups of the target population defined in terms of only a few discrete covariates. Here, we propose methods for estimating subgroup-specific potential outcome means and average treatment effects in generalizability and transportability analyses, using outcome model-based (g-formula), weighting, and augmented weighting estimators. We consider estimating subgroup-specific average treatment effects in the target population and its non-randomized subset, and provide methods that are appropriate both for nested and non-nested trial designs. As an illustration, we apply the methods to data from the Coronary Artery Surgery Study to compare the effect of surgery plus medical therapy versus medical therapy alone for chronic coronary artery disease in subgroups defined by history of myocardial infarction.