Targeted Learning with an Undersmoothed Lasso Propensity Score Model for Large-Scale Covariate Adjustment in Healthcare Database Studies.

Lasso regression is widely used for large-scale propensity score (PS) estimation in healthcare database studies. In these settings, previous work has shown that undersmoothing (overfitting) Lasso PS models can improve confounding control, but it can also cause problems of non-overlap in covariate distributions. It remains unclear how to select the degree of undersmoothing when fitting large-scale Lasso PS models to improve confounding control while avoiding issues that can result from reduced covariate overlap. Here, we used simulations to evaluate the performance of using collaborative-controlled targeted learning to data-adaptively select the degree of undersmoothing when fitting large-scale PS models within both singly and doubly robust frameworks to reduce bias in causal estimators. Simulations showed that collaborative learning can data-adaptively select the degree of undersmoothing to reduce bias in estimated treatment effects. Results further showed that when fitting undersmoothed Lasso PS-models, the use of cross-fitting was important for avoiding non-overlap in covariate distributions and reducing bias in causal estimates.

Considerations for master protocols using external controls.

There has been an increasing use of master protocols in oncology clinical trials because of its efficiency to accelerate cancer drug development and flexibility to accommodate multiple substudies. Depending on the study objective and design, a master protocol trial can be a basket trial, an umbrella trial, a platform trial, or any other form of trials in which multiple investigational products and/or subpopulations are studied under a single protocol. Master protocols can use external data and evidence (e.g. external controls) for treatment effect estimation, which can further improve efficiency of master protocol trials. This paper provides an overview of different types of external controls and their unique features when used in master protocols. Some key considerations in master protocols with external controls are discussed including construction of estimands, assessment of fit-for-use real-world data, and considerations for different types of master protocols. Similarities and differences between regular randomized controlled trials and master protocols when using external controls are discussed. A targeted learning-based causal roadmap is presented which constitutes three key steps: (1) define a target statistical estimand that aligns with the causal estimand for the study objective, (2) use an efficient estimator to estimate the target statistical estimand and its uncertainty, and (3) evaluate the impact of causal assumptions on the study conclusion by performing sensitivity analyses. Two illustrative examples for master protocols using external controls are discussed for their merits and possible improvement in causal effect estimation.

Nonparametric targeted Bayesian estimation of class proportions in unlabeled data.

We introduce a novel Bayesian estimator for the class proportion in an unlabeled dataset, based on the targeted learning framework. The procedure requires the specification of a prior (and outputs a posterior) only for the target of inference, and yields a tightly concentrated posterior. When the scientific question can be characterized by a low-dimensional parameter functional, this focus on target prior and posterior distributions perfectly aligns with Bayesian subjectivism. We prove a Bernstein-von Mises-type result for our proposed Bayesian procedure, which guarantees that the posterior distribution converges to the distribution of an efficient, asymptotically linear estimator. In particular, the posterior is Gaussian, doubly robust, and efficient in the limit, under the only assumption that certain nuisance parameters are estimated at slower-than-parametric rates. We perform numerical studies illustrating the frequentist properties of the method. We also illustrate their use in a motivating application to estimate the proportion of embolic strokes of undetermined source arising from occult cardiac sources or large-artery atherosclerotic lesions. Though we focus on the motivating example of the proportion of cases in an unlabeled dataset, the procedure is general and can be adapted to estimate any pathwise differentiable parameter in a non-parametric model.

Efficiently transporting causal direct and indirect effects to new populations under intermediate confounding and with multiple mediators.

The same intervention can produce different effects in different sites. Existing transport mediation estimators can estimate the extent to which such differences can be explained by differences in compositional factors and the mechanisms by which mediating or intermediate variables are produced; however, they are limited to consider a single, binary mediator. We propose novel nonparametric estimators of transported interventional (in)direct effects that consider multiple, high-dimensional mediators and a single, binary intermediate variable. They are multiply robust, efficient, asymptotically normal, and can incorporate data-adaptive estimation of nuisance parameters. They can be applied to understand differences in treatment effects across sites and/or to predict treatment effects in a target site based on outcome data in source sites.

Evaluating and improving real-world evidence with Targeted Learning.

BACKGROUND: The Targeted Learning roadmap provides a systematic guide for generating and evaluating real-world evidence (RWE). From a regulatory perspective, RWE arises from diverse sources such as randomized controlled trials that make use of real-world data, observational studies, and other study designs. This paper illustrates a principled approach to assessing the validity and interpretability of RWE. METHODS: We applied the roadmap to a published observational study of the dose-response association between ritodrine hydrochloride and pulmonary edema among women pregnant with twins in Japan. The goal was to identify barriers to causal effect estimation beyond unmeasured confounding reported by the study's authors, and to explore potential options for overcoming the barriers that robustify results. RESULTS: Following the roadmap raised issues that led us to formulate alternative causal questions that produced more reliable, interpretable RWE. The process revealed a lack of information in the available data to identify a causal dose-response curve. However, under explicit assumptions the effect of treatment with any amount of ritodrine versus none, albeit a less ambitious parameter, can be estimated from data. CONCLUSIONS: Before RWE can be used in support of clinical and regulatory decision-making, its quality and reliability must be systematically evaluated. The TL roadmap prescribes how to carry out a thorough, transparent, and realistic assessment of RWE. We recommend this approach be a routine part of any decision-making process.

Evaluating the robustness of targeted maximum likelihood estimators via realistic simulations in nutrition intervention trials.

Several recently developed methods have the potential to harness machine learning in the pursuit of target quantities inspired by causal inference, including inverse weighting, doubly robust estimating equations and substitution estimators like targeted maximum likelihood estimation. There are even more recent augmentations of these procedures that can increase robustness, by adding a layer of cross-validation (cross-validated targeted maximum likelihood estimation and double machine learning, as applied to substitution and estimating equation approaches, respectively). While these methods have been evaluated individually on simulated and experimental data sets, a comprehensive analysis of their performance across real data based simulations have yet to be conducted. In this work, we benchmark multiple widely used methods for estimation of the average treatment effect using ten different nutrition intervention studies data. A nonparametric regression method, undersmoothed highly adaptive lasso, is used to generate the simulated distribution which preserves important features from the observed data and reproduces a set of true target parameters. For each simulated data, we apply the methods above to estimate the average treatment effects as well as their standard errors and resulting confidence intervals. Based on the analytic results, a general recommendation is put forth for use of the cross-validated variants of both substitution and estimating equation estimators. We conclude that the additional layer of cross-validation helps in avoiding unintentional over-fitting of nuisance parameter functionals and leads to more robust inferences.

Nonparametric variable importance assessment using machine learning techniques.

In a regression setting, it is often of interest to quantify the importance of various features in predicting the response. Commonly, the variable importance measure used is determined by the regression technique employed. For this reason, practitioners often only resort to one of a few regression techniques for which a variable importance measure is naturally defined. Unfortunately, these regression techniques are often suboptimal for predicting the response. Additionally, because the variable importance measures native to different regression techniques generally have a different interpretation, comparisons across techniques can be difficult. In this work, we study a variable importance measure that can be used with any regression technique, and whose interpretation is agnostic to the technique used. This measure is a property of the true data-generating mechanism. Specifically, we discuss a generalization of the analysis of variance variable importance measure and discuss how it facilitates the use of machine learning techniques to flexibly estimate the variable importance of a single feature or group of features. The importance of each feature or group of features in the data can then be described individually, using this measure. We describe how to construct an efficient estimator of this measure as well as a valid confidence interval. Through simulations, we show that our proposal has good practical operating characteristics, and we illustrate its use with data from a study of risk factors for cardiovascular disease in South Africa.

The Impact of Delayed Switch to Second-Line Antiretroviral Therapy on Mortality, Depending on Definition of Failure Time and CD4 Count at Failure.

Little is known about the functional relationship of delaying second-line treatment initiation for human immunodeficiency virus-positive patients and mortality, given a patient's immune status. We included 7,255 patients starting antiretroviral therapy during 2004-2017, from 9 South African cohorts, with virological failure and complete baseline data. We estimated the impact of switch time on the hazard of death using inverse probability of treatment weighting of marginal structural models. The nonlinear relationship between month of switch and the 5-year survival probability, stratified by CD4 count at failure, was estimated with targeted maximum likelihood estimation. We adjusted for measured time-varying confounding by CD4 count, viral load, and visit frequency. Five-year mortality was estimated to be 10.5% (95% CI: 2.2, 18.8) for immediate switch and to be 26.6% (95% CI: 20.9, 32.3) for no switch (51.1% if CD4 count was <100 cells/mm3). The hazard of death was estimated to be 0.37 (95% CI: 0.30, 0.46) times lower if everyone had been switched immediately compared with never. The shorter the delay in switching, the lower the hazard of death-delaying 30-59 days reduced the hazard by 0.53 (95% CI: 0.43, 0.65) times and 60-119 days by 0.58 (95% CI: 0.49, 0.69) times, compared with no switch. Early treatment switch is particularly important for patients with low CD4 counts at failure.

The obesity paradox in critically ill patients: a causal learning approach to a casual finding.

BACKGROUND: While obesity confers an increased risk of death in the general population, numerous studies have reported an association between obesity and improved survival among critically ill patients. This contrary finding has been referred to as the obesity paradox. In this retrospective study, two causal inference approaches were used to address whether the survival of non-obese critically ill patients would have been improved if they had been obese. METHODS: The study cohort comprised 6557 adult critically ill patients hospitalized at the Intensive Care Unit of the Ghent University Hospital between 2015 and 2017. Obesity was defined as a body mass index of ≥ 30 kg/m2. Two causal inference approaches were used to estimate the average effect of obesity in the non-obese (AON): a traditional approach that used regression adjustment for confounding and that assumed missingness completely at random and a robust approach that used machine learning within the targeted maximum likelihood estimation framework along with multiple imputation of missing values under the assumption of missingness at random. 1754 (26.8%) patients were discarded in the traditional approach because of at least one missing value for obesity status or confounders. RESULTS: Obesity was present in 18.9% of patients. The in-hospital mortality was 14.6% in non-obese patients and 13.5% in obese patients. The raw marginal risk difference for in-hospital mortality between obese and non-obese patients was - 1.06% (95% confidence interval (CI) - 3.23 to 1.11%, P = 0.337). The traditional approach resulted in an AON of - 2.48% (95% CI - 4.80 to - 0.15%, P = 0.037), whereas the robust approach yielded an AON of - 0.59% (95% CI - 2.77 to 1.60%, P = 0.599). CONCLUSIONS: A causal inference approach that is robust to residual confounding bias due to model misspecification and selection bias due to missing (at random) data mitigates the obesity paradox observed in critically ill patients, whereas a traditional approach results in even more paradoxical findings. The robust approach does not provide evidence that the survival of non-obese critically ill patients would have been improved if they had been obese.

Targeted learning in real-world comparative effectiveness research with time-varying interventions.

In comparative effectiveness research (CER), often the aim is to contrast survival outcomes between exposure groups defined by time-varying interventions. With observational data, standard regression analyses (e.g., Cox modeling) cannot account for time-dependent confounders on causal pathways between exposures and outcome nor for time-dependent selection bias that may arise from informative right censoring. Inverse probability weighting (IPW) estimation to fit marginal structural models (MSMs) has commonly been applied to properly adjust for these expected sources of bias in real-world observational studies. We describe the application and performance of an alternate estimation approach in such a study. The approach is based on the recently proposed targeted learning methodology and consists in targeted minimum loss-based estimation (TMLE) with super learning (SL) within a nonparametric MSM. The evaluation is based on the analysis of electronic health record data with both IPW estimation and TMLE to contrast cumulative risks under four more or less aggressive strategies for treatment intensification in adults with type 2 diabetes already on 2+ oral agents or basal insulin. Results from randomized experiments provide a surrogate gold standard to validate confounding and selection bias adjustment. Bootstrapping is used to validate analytic estimation of standard errors. This application does the following: (1) establishes the feasibility of TMLE in real-world CER based on large healthcare databases; (2) provides evidence of proper confounding and selection bias adjustment with TMLE and SL; and (3) motivates their application for improving estimation efficiency. Claims are reinforced with a simulation study that also illustrates the double-robustness property of TMLE.

A general framework for inference on algorithm-agnostic variable importance.

In many applications, it is of interest to assess the relative contribution of features (or subsets of features) toward the goal of predicting a response - in other words, to gauge the variable importance of features. Most recent work on variable importance assessment has focused on describing the importance of features within the confines of a given prediction algorithm. However, such assessment does not necessarily characterize the prediction potential of features, and may provide a misleading reflection of the intrinsic value of these features. To address this limitation, we propose a general framework for nonparametric inference on interpretable algorithm-agnostic variable importance. We define variable importance as a population-level contrast between the oracle predictiveness of all available features versus all features except those under consideration. We propose a nonparametric efficient estimation procedure that allows the construction of valid confidence intervals, even when machine learning techniques are used. We also outline a valid strategy for testing the null importance hypothesis. Through simulations, we show that our proposal has good operating characteristics, and we illustrate its use with data from a study of an antibody against HIV-1 infection.

Resident Readiness for Senior Level Decision Making: Identifying the Domains for Formative Assessment and Feedback.

OBJECTIVE: To establish expert consensus regarding the domains and topics for senior surgery residents (PGY-4) to make critical decisions and assume senior-level responsibilities, and to develop the formative American College of Surgeons Senior Resident Readiness Assessment (ACS SRRA) Program. DESIGN: The American College of Surgeons (ACS) education leadership team conducted a focus group with surgical experts to identify the content for an assessment tool to evaluate senior residents' readiness for their increased levels of responsibility. After the focus group, national experts were recruited to develop consensus on the topics through three rounds of surveys using Delphi methodology. The Delphi participants rated topics using Likert-type scales and their comments were incorporated into subsequent rounds. Consensus was defined as ≥ 80% agreement with internal-consistency reliability (Cronbach's alpha) ≥ 0.8. In a stepwise fashion, topics that did not achieve consensus for inclusion were removed from subsequent survey rounds. SETTING: The surveys were administered via an online questionnaire. PARTICIPANTS: Twelve program directors and assistant program directors made up the focus group. The 39 Delphi participants represented seven different surgical subspecialties and were from diverse practice settings. The median length of experience in general surgery resident education was 20 years (IQR 14.3-30.0) with 64% of the experts being either current or past general surgery residency program directors. RESULTS: The response rate was 100% and Cronbach's alpha was ≥ 0.9 for each round. The Delphi participants contributed a large number of comments. Of the 201 topics that were evaluated initially, 120 topics in 25 core clinical areas were included to create the final domains of ACS SRRA. CONCLUSIONS: National consensus on the domain of the ACS SRRA has been achieved via the modified Delphi method among expert surgeon educators. ACS SRRA will identify clinical topics and areas in which each senior resident needs improvement and provide data to residents and residency programs to develop individualized learning plans. This would help in preparing the senior residents to assume their responsibilities and support their readiness for future fellowship training or surgical practice.

Estimation of causal effect in integrating randomized clinical trial and observational data - An example application to cardiovascular outcome trial.

Safety evaluation of drug development is a comprehensive process across the product lifecycle. While a randomized clinical trial (RCT) can provide high-quality data to assess the efficacy and safety of a new intervention, the pre-marketing trials are limited in statistical power to detect causal elevation of rare but potentially serious adverse events. On the other hand, real-world data (RWD) sources play a critical role in further understanding the safety profile of the new intervention. Bringing together the breadth and strength of RWD and RCT data, we can maximize the utility of RWD and answer broader questions. In this manuscript, we propose a three-step statistical framework to corroborate findings from both RCT and RWD for evaluating important safety concerns identified in the pre-marketing setting. By the proposed approach, we first match the observational study to RCT, then the causal estimation is validated via the matched observational study with the target RCT by targeted maximum likelihood estimation (TMLE) method, and lastly the evidence from RCT and RWD can be combined in an integrative analysis. A potential application to cardiovascular outcome trials for type 2 diabetes mellitus is illustrated. Finally, simulation results suggest that the heterogeneity of patient population from RCT and RWD can lead to varying degrees of treatment effect estimation and the proposed approach may be able to mitigate such difference in the integrative analysis.

Examining Obedience Training as a Physical Activity Intervention for Dog Owners: Findings from the Stealth Pet Obedience Training (SPOT) Pilot Study.

Dog training may strengthen the dog-owner bond, a consistent predictor of dog walking behavior. The Stealth Pet Obedience Training (SPOT) study piloted dog training as a stealth physical activity (PA) intervention. In this study, 41 dog owners who reported dog walking ≤3 days/week were randomized to a six-week basic obedience training class or waitlist control. Participants wore accelerometers and logged dog walking at baseline, 6- and 12-weeks. Changes in PA and dog walking were compared between arms with targeted maximum likelihood estimation. At baseline, participants (39 ± 12 years; females = 85%) walked their dog 1.9 days/week and took 5838 steps/day, on average. At week 6, intervention participants walked their dog 0.7 more days/week and took 480 more steps/day, on average, than at baseline, while control participants walked their dog, on average, 0.6 fewer days/week and took 300 fewer steps/day (difference between arms: 1.3 dog walking days/week; 95% CI = 0.2, 2.5; 780 steps/day, 95% CI = -746, 2307). Changes from baseline were similar at week 12 (difference between arms: 1.7 dog walking days/week; 95% CI = 0.6, 2.9; 1084 steps/day, 95% CI = -203, 2370). Given high rates of dog ownership and low rates of dog walking in the United States, this novel PA promotion strategy warrants further investigation.

On adaptive propensity score truncation in causal inference.

The positivity assumption, or the experimental treatment assignment (ETA) assumption, is important for identifiability in causal inference. Even if the positivity assumption holds, practical violations of this assumption may jeopardize the finite sample performance of the causal estimator. One of the consequences of practical violations of the positivity assumption is extreme values in the estimated propensity score (PS). A common practice to address this issue is truncating the PS estimate when constructing PS-based estimators. In this study, we propose a novel adaptive truncation method, Positivity-C-TMLE, based on the collaborative targeted maximum likelihood estimation (C-TMLE) methodology. We demonstrate the outstanding performance of our novel approach in a variety of simulations by comparing it with other commonly studied estimators. Results show that by adaptively truncating the estimated PS with a more targeted objective function, the Positivity-C-TMLE estimator achieves the best performance for both point estimation and confidence interval coverage among all estimators considered.

Evaluating the Impact of a HIV Low-Risk Express Care Task-Shifting Program: A Case Study of the Targeted Learning Roadmap.

In conducting studies on an exposure of interest, a systematic roadmap should be applied for translating causal questions into statistical analyses and interpreting the results. In this paper we describe an application of one such roadmap applied to estimating the joint effect of both time to availability of a nurse-based triage system (low risk express care (LREC)) and individual enrollment in the program among HIV patients in East Africa. Our study population is comprised of 16,513 subjects found eligible for this task-shifting program within 15 clinics in Kenya between 2006 and 2009, with each clinic starting the LREC program between 2007 and 2008. After discretizing follow-up into 90-day time intervals, we targeted the population mean counterfactual outcome (i. e. counterfactual probability of either dying or being lost to follow up) at up to 450 days after initial LREC eligibility under three fixed treatment interventions. These were (i) under no program availability during the entire follow-up, (ii) under immediate program availability at initial eligibility, but non-enrollment during the entire follow-up, and (iii) under immediate program availability and enrollment at initial eligibility. We further estimated the controlled direct effect of immediate program availability compared to no program availability, under a hypothetical intervention to prevent individual enrollment in the program. Targeted minimum loss-based estimation was used to estimate the mean outcome, while Super Learning was implemented to estimate the required nuisance parameters. Analyses were conducted with the ltmle R package; analysis code is available at an online repository as an R package. Results showed that at 450 days, the probability of in-care survival for subjects with immediate availability and enrollment was 0.93 (95% CI: 0.91, 0.95) and 0.87 (95% CI: 0.86, 0.87) for subjects with immediate availability never enrolling. For subjects without LREC availability, it was 0.91 (95% CI: 0.90, 0.92). Immediate program availability without individual enrollment, compared to no program availability, was estimated to slightly albeit significantly decrease survival by 4% (95% CI 0.03,0.06, p<0.01). Immediately availability and enrollment resulted in a 7 % higher in-care survival compared to immediate availability with non-enrollment after 450 days (95% CI-0.08,-0.05, p<0.01). The results are consistent with a fairly small impact of both availability and enrollment in the LREC program on incare survival.

Discussion of Identification, Estimation and Approximation of Risk under Interventions that Depend on the Natural Value of Treatment Using Observational Data, by Jessica Young, Miguel Hernán, and James Robins.

Young, Hernán, and Robins consider the mean outcome under a dynamic intervention that may rely on the natural value of treatment. They first identify this value with a statistical target parameter, and then show that this statistical target parameter can also be identified with a causal parameter which gives the mean outcome under a stochastic intervention. The authors then describe estimation strategies for these quantities. Here we augment the authors' insightful discussion by sharing our experiences in situations where two causal questions lead to the same statistical estimand, or the newer problem that arises in the study of data adaptive parameters, where two statistical estimands can lead to the same estimation problem. Given a statistical estimation problem, we encourage others to always use a robust estimation framework where the data generating distribution truly belongs to the statistical model. We close with a discussion of a framework which has these properties.