Invited Commentary: Conducting and Emulating Trials to Study Effects of Social Interventions.

All else being equal, if we had 1 causal effect we wished to estimate, we would conduct a randomized trial with a protocol that mapped onto that causal question, or we would attempt to emulate that target trial with observational data. However, studying the social determinants of health often means there are not just 1 but several causal contrasts of simultaneous interest and importance, and each of these related but distinct causal questions may have varying degrees of feasibility in conducting trials. With this in mind, we discuss challenges and opportunities that arise when conducting and emulating such trials. We describe designing trials with the simultaneous goals of estimating the intention-to-treat effect, the per-protocol effect, effects of alternative protocols or joint interventions, effects within subgroups, and effects under interference, and we describe ways to make the most of all feasible randomized trials and emulated trials using observational data. Our comments are grounded in the study results of Courtin et al. (Am J Epidemiol. 2022;191(8):1444-1452).

Adjusting for adherence in randomized trials when adherence is measured as a continuous variable: An application to the Lipid Research Clinics Coronary Primary Prevention Trial.

BACKGROUND: Clinicians and patients may be more interested in per-protocol effect estimates than intention-to-treat effect estimates from randomized trials. However, per-protocol effect estimates may be biased due to insufficient adjustment for prognostic factors that predict adherence. Adjustment for this bias is possible when appropriate methods, such as inverse probability weighting, are used. But, when adherence is measured as a continuous variable, constructing these weights can be challenging. METHODS: In the placebo arm of the Lipid Research Clinics Coronary Primary Prevention Trial, we estimated the 7-year cumulative incidence of coronary heart disease under 100% adherence and 0% adherence to placebo. We used dose-response discrete-hazards models with inverse probability weighting to adjust for pre- and post-randomization covariates. We considered several continuous distributions for constructing the inverse probability weights. RESULTS: The risk difference estimate for 100% adherence compared with 0% adherence ranged from -7.7 to -6.1 percentage points without adjustment for baseline and post-baseline covariates, and ranged from -1.8 to 2.2 percentage points with adjustment using inverse probability weights, depending on the dose-response model and inverse probability weight distribution used. CONCLUSIONS: Methods which appropriately adjust for time-varying post-randomization variables can explain away much of the bias in the "effect" of adherence to placebo. When considering continuous adherence, investigators should consider several models as estimates may be sensitive to the model chosen.

Effect Estimates in Randomized Trials and Observational Studies: Comparing Apples With Apples.

Effect estimates from randomized trials and observational studies might not be directly comparable because of differences in study design, other than randomization, and in data analysis. We propose a 3-step procedure to facilitate meaningful comparisons of effect estimates from randomized trials and observational studies: 1) harmonization of the study protocols (eligibility criteria, treatment strategies, outcome, start and end of follow-up, causal contrast) so that the studies target the same causal effect, 2) harmonization of the data analysis to estimate the causal effect, and 3) sensitivity analyses to investigate the impact of discrepancies that could not be accounted for in the harmonization process. To illustrate our approach, we compared estimates of the effect of immediate with deferred initiation of antiretroviral therapy in individuals positive for the human immunodeficiency virus from the Strategic Timing of Antiretroviral Therapy (START) randomized trial and the observational HIV-CAUSAL Collaboration.

Diagnosing Covariate Balance Across Levels of Right-Censoring Before and After Application of Inverse-Probability-of-Censoring Weights.

Covariate balance is a central concept in the potential outcomes literature. With selected populations or missing data, balance across treatment groups can be insufficient for estimating marginal treatment effects. Recently, a framework for using covariate balance to describe measured confounding and selection bias for time-varying and other multivariate exposures in the presence of right-censoring has been proposed. Here, we revisit this framework to consider balance across levels of right-censoring over time in more depth. Specifically, we develop measures of covariate balance that can describe what is known as "dependent censoring" in the literature, along with its associated selection bias, under multiple mechanisms for right censoring. Such measures are interesting because they substantively describe the evolution of dependent censoring mechanisms. Furthermore, we provide weighted versions that can depict how well such dependent censoring has been eliminated when inverse-probability-of-censoring weights are applied. These results provide a conceptually grounded way to inspect covariate balance across levels of right-censoring as a validity check. As a motivating example, we applied these measures to a study of hypothetical "static" and "dynamic" treatment protocols in a sequential multiple-assignment randomized trial of antipsychotics with high dropout rates.

Generalizing the per-protocol treatment effect: The case of ACTG A5095.

BACKGROUND: Intention-to-treat comparisons of randomized trials provide asymptotically consistent estimators of the effect of treatment assignment, without regard to compliance. However, decision makers often wish to know the effect of a per-protocol comparison. Moreover, decision makers may also wish to know the effect of treatment assignment or treatment protocol in a user-specified target population other than the sample in which the trial was fielded. Here, we aimed to generalize results from the ACTG A5095 trial to the US recently HIV-diagnosed target population. METHODS: We first replicated the published conventional intention-to-treat estimate (2-year risk difference and hazard ratio) comparing a four-drug antiretroviral regimen to a three-drug regimen in the A5095 trial. We then estimated the intention-to-treat effect that accounted for informative dropout and the per-protocol effect that additionally accounted for protocol deviations by constructing inverse probability weights. Furthermore, we employed inverse odds of sampling weights to generalize both intention-to-treat and per-protocol effects to a target population comprising US individuals with HIV diagnosed during 2008-2014. RESULTS: Of 761 subjects in the analysis, 82 dropouts (36 in the three-drug arm and 46 in the four-drug arm) and 59 protocol deviations (25 in the three-drug arm and 34 in the four-drug arm) occurred during the first 2 years of follow-up. A total of 169 subjects incurred virologic failure or death. The 2-year risks were similar both in the trial and in the US HIV-diagnosed target population for estimates from the conventional intention-to-treat, dropout-weighted intention-to-treat, and per-protocol analyses. In the US target population, the 2-year conventional intention-to-treat risk difference (unit: %) for virologic failure or death comparing the four-drug arm to the three-drug arm was -0.4 (95% confidence interval: -6.2, 5.1), while the hazard ratio was 0.97 (95% confidence interval: 0.70, 1.34); the 2-year risk difference was -0.9 (95% confidence interval: -6.9, 5.3) for the dropout-weighted intention-to-treat comparison (hazard ratio = 0.95, 95% confidence interval: 0.68, 1.32) and -0.7 (95% confidence interval: -6.7, 5.5) for the per-protocol comparison (hazard ratio = 0.96, 95% confidence interval: 0.69, 1.34). CONCLUSION: No benefit of four-drug antiretroviral regimen over three-drug regimen was found from the conventional intention-to-treat, dropout-weighted intention-to-treat or per-protocol estimates in the trial sample or target population.

Adherence adjustment in the Coronary Drug Project: A call for better per-protocol effect estimates in randomized trials.

BACKGROUND: In many randomized controlled trials, patients and doctors are more interested in the per-protocol effect than in the intention-to-treat effect. However, valid estimation of the per-protocol effect generally requires adjustment for prognostic factors associated with adherence. These adherence adjustments have been strongly questioned in the clinical trials community, especially after 1980 when the Coronary Drug Project team found that adherers to placebo had lower 5-year mortality than non-adherers to placebo. METHODS: We replicated the original Coronary Drug Project findings from 1980 and re-analyzed the Coronary Drug Project data using technical and conceptual developments that have become established since 1980. Specifically, we used logistic models for binary outcomes, decoupled the definition of adherence from loss to follow-up, and adjusted for pre-randomization covariates via standardization and for post-randomization covariates via inverse probability weighting. RESULTS: The original Coronary Drug Project analysis reported a difference in 5-year mortality between adherers and non-adherers in the placebo arm of 9.4 percentage points. Using modern approaches, we found that this difference was reduced to 2.5 (95% confidence interval: -2.1 to 7.0). CONCLUSION: Valid estimation of per-protocol effects may be possible in randomized clinical trials when analysts use appropriate methods to adjust for post-randomization variables.

Estimating the per-protocol effect of lithium on suicidality in a randomized trial of individuals with depression or bipolar disorder.

BACKGROUND: The CSP590 randomized trial was designed to estimate the effect of lithium on suicidality. After a third of the intended number of participants were enrolled, the hazard ratio of suicidality was 1.10 (95% CI: 0.77, 1.55). Based on this, the trial was stopped for futility. However, only 17% of patients adhered to the specified protocol. AIMS: The objective was to estimate the per-protocol effect of lithium on suicidality, that is, the effect of adhering to the treatment strategies as specified in the protocol. METHODS: We stopped individuals' follow-up if/when they showed evidence of nonadherence. We then conducted the analysis in the restricted sample, adjusting for prognostic factors that predict adherence via inverse probability weighting. The primary outcome was the 12-month risk of suicidality (including death from suicide, suicide attempt, interrupted attempt, hospitalization specifically to prevent suicide). RESULTS: The estimated 12-month risk of suicidality was 18.8% for lithium, and 24.3% for placebo. The risk ratio was 0.78 (95% CI: 0.43, 1.37) and the risk difference -5.5 percentage points (95% CI: -17.5, 5.5). Results were consistent across sensitivity analyses. CONCLUSIONS: With one-third of the targeted sample size, lithium effects (compared with placebo) ranging between a 17.5% reduction and a 5.5% increase in the risk of suicidality were highly compatible with the data. Thus, a protective effect of lithium on suicidality among patients with bipolar disorder or major depressive disorder cannot be ruled out. Trials should incorporate adequate per-protocol analyses into the decision-making processes for stopping trials for futility.

Adherence-adjustment in placebo-controlled randomized trials: An application to the candesartan in heart failure randomized trial.

BACKGROUND: The per-protocol effect provides important information in randomized trials with incomplete adherence. Yet, because valid estimation typically requires adjustment for prognostic factors that predict adherence, per-protocol effect estimates are often met with skepticism. In placebo-controlled trials, however, the validity of adjustment can be indirectly verified by demonstrating no association between adherence and the outcome among the placebo arm. Here, we describe a two-stage procedure in which we first adjust for time-varying adherence in the placebo arm and then use a similar procedure to estimate the per-protocol effect. METHODS: We use the Candesartan in Heart Failure: Assessment of Reduction in Mortality and Morbidity (CHARM) randomized trial. First, we compare adherers versus non-adherers in the placebo arm, adjusting for pre- and post-randomization variables. Second, we use models validated in the placebo arm to estimate the per-protocol effect of adherence to candesartan versus placebo in the full trial. FINDINGS: We successfully estimated no association between adherence and mortality in the placebo arm; hazard ratio: 0.91 (95% CI: 0.51, 2.52). We then estimated the per-protocol effect under two sets of protocol-defined stopping criteria after adjustment for post-randomization confounders. The mortality hazard ratio estimates ranged from 0.91 to 0.93 for the per-protocol effect estimates, similar to the intention-to-treat effect estimates. INTERPRETATION: Adherence adjustment in the CHARM trial is feasible when appropriate assumptions about missing data and confounding are made. These assumptions cannot be verified but can be supported through the use of placebo-arm adherence assessment.

Interval-cohort designs and bias in the estimation of per-protocol effects: a simulation study.

BACKGROUND: Randomized trials are considered the gold standard for making inferences about the causal effects of treatments. However, when protocol deviations occur, the baseline randomization of the trial is no longer sufficient to ensure unbiased estimation of the per-protocol effect: post-randomization, time-varying confounders must be sufficiently measured and adjusted for in the analysis. Given the historical emphasis on intention-to-treat effects in randomized trials, measurement of post-randomization confounders is typically infrequent. This may induce bias in estimates of the per-protocol effect, even using methods such as inverse probability weighting, which appropriately account for time-varying confounders affected by past treatment. METHODS/DESIGN: In order to concretely illustrate the potential magnitude of bias due to infrequent measurement of time-varying covariates, we simulated data from a very large trial with a survival outcome and time-varying confounding affected by past treatment. We generated the data such that the true underlying per-protocol effect is null and under varying degrees of confounding (strong, moderate, weak). In the simulated data, we estimated per-protocol survival curves and associated contrasts using inverse probability weighting under monthly measurement of the time-varying covariates (which constituted complete measurement in our simulation), yearly measurement, as well as 3- and 6-month intervals. RESULTS: Using inverse probability weighting, we were able to recover the true null under the complete measurement scenario no matter the strength of confounding. Under yearly measurement intervals, the estimate of the per-protocol effect diverged from the null; inverse probability weighted estimates of the per-protocol 5-year risk ratio based on yearly measurement were 1.19, 1.12, and 1.03 under strong, moderate, and weak confounding, respectively. Bias decreased with measurement interval length. Under all scenarios, inverse probability weighted estimators were considerably less biased than a naive estimator that ignored time-varying confounding completely. CONCLUSIONS: Bias that arises from interval measurement designs highlights the need for planning in the design of randomized trials for collection of time-varying covariate data. This may come from more frequent in-person measurement or external sources (e.g., electronic medical record data). Such planning will provide improved estimates of the per-protocol effect through the use of methods that appropriately adjust for time-varying confounders.

Improved adherence adjustment in the Coronary Drug Project.

BACKGROUND: The survival difference between adherers and non-adherers to placebo in the Coronary Drug Project has been used to support the thesis that adherence adjustment in randomized trials is not generally possible and, therefore, that only intention-to-treat analyses should be trusted. We previously demonstrated that adherence adjustment can be validly conducted in the Coronary Drug Project using a simplistic approach. Here, we re-analyze the data using an approach that takes full advantage of recent methodological developments. METHODS: We used inverse-probability weighted hazards models to estimate the 5-year survival and mortality risk when individuals in the placebo arm of the Coronary Drug Project adhere to at least 80% of the drug continuously or never during the 5-year follow-up period. RESULTS: Adjustment for post-randomization covariates resulted in 5-year mortality risk difference estimates ranging from - 0.7 (95% confidence intervals (CI), - 12.2, 10.7) to 4.5 (95% CI, - 6.3, 15.3) percentage points. CONCLUSIONS: Our analysis confirms that appropriate adjustment for post-randomization predictors of adherence largely removes the association between adherence to placebo and mortality originally described in this trial. TRIAL REGISTRATION: ClinicalTrials.gov, Identifier: NCT00000482 . Registered retrospectively on 27 October 1999.

The per-protocol effect of immediate versus deferred antiretroviral therapy initiation.

OBJECTIVE: The Strategic Timing of AntiRetroviral Treatment (START) trial found a lower risk of a composite clinical outcome in HIV-positive individuals assigned to immediate initiation of antiretroviral therapy (ART) compared with those assigned to deferred initiation. However, 30% of those assigned to deferred initiation started ART earlier than the protocol specified. To supplement the published intention-to-treat (ITT) effect estimates, here we estimate the per-protocol effect of immediate versus deferred ART initiation in START. DESIGN: The START trial randomized 4685 HIV-positive participants with CD4 cell counts more than 500 cells/µl to start ART immediately after randomization (immediate initiation group) or to wait until the CD4 cell count dropped below 350 cells/µl or an AIDS diagnosis (deferred initiation group). METHODS: We used the parametric g-formula to estimate and compare the cumulative 5-year risk of the composite clinical outcome in the immediate initiation group, and deferred initiation groups had all the trial participants adhered to the protocol. RESULTS: We estimated that the 5-year risk of the composite outcome would have been 3.2% under immediate ART initiation and 7.0% under deferred initiation. The difference of 3.8% (95% confidence interval 1.5, 6.5) was larger than the ITT effect estimate of 3.1%, corresponding to a difference in effect estimates of 0.72% (-0.35, 2.35). CONCLUSION: The ITT effect estimate may underestimate the benefit of immediate ART initiation by 23%. This estimate can be used by patients and policy-makers who need to understand the full extent of the benefit of changes in ART initiation policies.