A causal roadmap for generating high-quality real-world evidence.

Increasing emphasis on the use of real-world evidence (RWE) to support clinical policy and regulatory decision-making has led to a proliferation of guidance, advice, and frameworks from regulatory agencies, academia, professional societies, and industry. A broad spectrum of studies use real-world data (RWD) to produce RWE, ranging from randomized trials with outcomes assessed using RWD to fully observational studies. Yet, many proposals for generating RWE lack sufficient detail, and many analyses of RWD suffer from implausible assumptions, other methodological flaws, or inappropriate interpretations. The Causal Roadmap is an explicit, itemized, iterative process that guides investigators to prespecify study design and analysis plans; it addresses a wide range of guidance within a single framework. By supporting the transparent evaluation of causal assumptions and facilitating objective comparisons of design and analysis choices based on prespecified criteria, the Roadmap can help investigators to evaluate the quality of evidence that a given study is likely to produce, specify a study to generate high-quality RWE, and communicate effectively with regulatory agencies and other stakeholders. This paper aims to disseminate and extend the Causal Roadmap framework for use by clinical and translational researchers; three companion papers demonstrate applications of the Causal Roadmap for specific use cases.

Sensitivity analysis for causality in observational studies for regulatory science.

Objective: The United States Congress passed the 21st Century Cures Act mandating the development of Food and Drug Administration guidance on regulatory use of real-world evidence. The Forum on the Integration of Observational and Randomized Data conducted a meeting with various stakeholder groups to build consensus around best practices for the use of real-world data (RWD) to support regulatory science. Our companion paper describes in detail the context and discussion of the meeting, which includes a recommendation to use a causal roadmap for study designs using RWD. This article discusses one step of the roadmap: the specification of a sensitivity analysis for testing robustness to violations of causal model assumptions. Methods: We present an example of a sensitivity analysis from a RWD study on the effectiveness of Nifurtimox in treating Chagas disease, and an overview of various methods, emphasizing practical considerations on their use for regulatory purposes. Results: Sensitivity analyses must be accompanied by careful design of other aspects of the causal roadmap. Their prespecification is crucial to avoid wrong conclusions due to researcher degrees of freedom. Sensitivity analysis methods require auxiliary information to produce meaningful conclusions; it is important that they have at least two properties: the validity of the conclusions does not rely on unverifiable assumptions, and the auxiliary information required by the method is learnable from the corpus of current scientific knowledge. Conclusions: Prespecified and assumption-lean sensitivity analyses are a crucial tool that can strengthen the validity and trustworthiness of effectiveness conclusions for regulatory science.

What matters most? Different stakeholder perspectives on estimands for an invented case study in COPD.

In drug development, we ask ourselves which population, endpoint and treatment comparison should be investigated. In this context, we also debate what matters most to the different stakeholders that are involved in clinical drug development, for example, patients, physicians, regulators and payers. With the publication of draft ICH E9 addendum on estimands in 2017, we now have a common framework and language to discuss such questions in an informed and transparent way. This has led to the estimand discussion being a key element in study development, including design, analysis and interpretation of a treatment effect. At an invited session at the 2018 PSI annual conference, PSI hosted a role-play debate where the aim of the session was to mimic a regulatory and payer scientific advice discussion for a COPD drug. Including role-play views from an industry sponsor, a patient, a regulator and a payer. This paper presents the invented COPD case-study design and considerations relating to appropriate estimands are discussed by each of the stakeholders from their differing viewpoints with the additional inclusion of a technical (academic) perspective. The rationale for each perspective on approaches for handling intercurrent events is presented, with a key emphasis on the application of while-on-treatment and treatment policy estimands in this context. It is increasingly recognised that the treatment effect estimated by the treatment policy approach may not always be of primary clinical interest and may not appropriately communicate to patients the efficacy they can expect if they take the treatment as directed.

Challenges in Assessing the Impact of the COVID-19 Pandemic on the Integrity and Interpretability of Clinical Trials.

Abstract-The COVID-19 pandemic has a global impact on the conduct of clinical trials of medical products. This article discusses implications of the COVID-19 pandemic on clinical research methodology aspects and provides points to consider to assess and mitigate the risk of seriously compromising the integrity and interpretability of clinical trials. The information in this article will support discussions that need to occur cross-functionally on an ongoing basis to "integrate all available knowledge from the ethical, the medical, and the methodological perspective into decision making." This article aims at facilitating: (i) risk assessments of the impact of the pandemic on trial integrity and interpretability; (ii) identification of the relevant data and information related to the impact of the pandemic on the trial that needs to be collected; (iii) short-term decision making impacting ongoing trial operations; (iv) ongoing monitoring of the trial conduct until completion, including the possible involvement of data monitoring committees, and adequately documenting all measures taken to secure trial integrity throughout and after the pandemic, and (v) proper analysis and interpretation of the eventual interim or final trial data.

Assessing Treatment Effects That Capture Disease Burden in Serious Chronic Diseases.

Serious and chronic health conditions such as cardiovascular diseases (CVDs) are posing challenges to the health system. Recently clinical trials in these fields have focused on composite endpoints that take into account both disease-related mortality and major disease-related morbidity events. It is the time to the first component of the composite endpoint experienced by a patient that is the traditional study endpoint and treatment aims are to delay the time to the first event and to reduce its frequency. As the name implies, the time-to-first composite event analysis approach focuses only on the first composite event and ignores subsequent events. For a chronic disease, this can lead to a substantial loss of potentially important information. For instance, in chronic heart failure (HF) studies, the traditional composite endpoint of HF-related hospitalizations and CVD death will ignore CVD deaths that are preceded by HF-related hospitalizations. This paper explores the limitations of the traditional time-to-first event approach and discusses the potential value of incorporating all events. The authors argue that endpoints capturing recurrent event information can lead to interpretable measures of treatment effect that better reflect disease burden than traditional time-to-first event endpoints by using the available information beyond the first event. This paper aims to raise awareness of the value and potential pitfalls of alternative treatment effect measures to facilitate meaningful cross-functional conversations among trialists and other stakeholders such as regulators, payers, and treating physicians who all are striving to the same goal-to deliver the most effective treatments to patients.

Estimands and the Patient Journey: Addressing the Right Question in Oncology Clinical Trials.

The diversity of patient journeys can raise fundamental questions regarding the evaluation of drug effects in clinical trials to inform clinical practice. When defining the treatment effect of interest in a trial, the researcher needs to account for events occurring after treatment initiation, such as the start of a new therapy, before observing the end point. We review the newly introduced estimand framework to structure discussions on the relationship between patient journeys and the treatment effect of interest in oncology trials. In 2017, the International Council for Harmonization of Technical Requirements for Pharmaceuticals for Human Use released a draft addendum to its E9 guideline. The addendum introduces the concept of an estimand to precisely describe the treatment effect of interest. This estimand framework provides a structured approach to discuss how to account for intercurrent events that occur after random assignment and may affect the assessment or interpretation of the treatment effect. The framework is expected to improve coherence between trial objectives, design, analysis, and interpretation, as illustrated by examples in oncology disease settings. The estimand framework was applied to design a trial for a chimeric antigen receptor T-cell therapy. The treatment effect of interest was carefully defined considering the range of patient journeys expected for this particular indication and treatment. The trial design was developed accordingly to assess that treatment effect. All parties involved in the design of clinical trials need to consider possible patient journeys to define appropriate treatment effects and corresponding trial designs and analysis strategies. The estimand framework provides a common language to address the complexity introduced by varied patient journeys.

Estimands: A More Strategic Approach to Study Design and Analysis.

Do we always precisely define the treatment effects that our clinical trial will estimate? Our tenet is that this is not always done, or is done inadequately. This lack of clarity can result in a misalignment among trial objectives, trial design, and statistical methods. We will discuss these challenges and present an improved framework using estimands that is proposed in a draft International Council for Harmonization (ICH) E9 addendum.

Considerations for Evaluating Treatment Effects From Randomized Clinical Trials.

This article focuses on the choice of treatment effect measures in randomized clinical trials (RCTs). Traditionally, an intention-to-treat (ITT) analysis is conducted with an implicit understanding that a treatment-policy effect is of greatest interest. In this article we contend that this approach may not always provide accurate information about clinically meaningful treatment effects, and we present an argument that for any RCT it is desirable to require an explicit definition of what treatment effect is of primary interest, known as the "estimand." We will discuss the limitations of the traditional ITT effect measures as well as the state-of-the art thinking with regard to estimands. Furthermore, we will offer alternate choices that acknowledge that treatments have multiple effects.

Estimands in clinical trials - broadening the perspective.

Defining the scientific questions of interest in a clinical trial is crucial to align its planning, design, conduct, analysis, and interpretation. However, practical experience shows that oftentimes specific choices in the statistical analysis blur the scientific question either in part or even completely, resulting in misalignment between trial objectives, conduct, analysis, and confusion in interpretation. The need for more clarity was highlighted by the Steering Committee of the International Council for Harmonization (ICH) in 2014, which endorsed a Concept Paper with the goal of developing a new regulatory guidance, suggested to be an addendum to ICH guideline E9. Triggered by these developments, we elaborate in this paper what the relevant questions in drug development are and how they fit with the current practice of intention-to-treat analyses. To this end, we consider the perspectives of patients, physicians, regulators, and payers. We argue that despite the different backgrounds and motivations of the various stakeholders, they all have similar interests in what the clinical trial estimands should be. Broadly, these can be classified into estimands addressing (a) lack of adherence to treatment due to different reasons and (b) efficacy and safety profiles when patients, in fact, are able to adhere to the treatment for its intended duration. We conclude that disentangling adherence to treatment and the efficacy and safety of treatment in patients that adhere leads to a transparent and clinical meaningful assessment of treatment risks and benefits. We touch upon statistical considerations and offer a discussion of additional implications. Copyright © 2016 John Wiley & Sons, Ltd.

Sensitivity analyses for partially observed recurrent event data.

Recurrent events involve the occurrences of the same type of event repeatedly over time and are commonly encountered in longitudinal studies. Examples include seizures in epileptic studies or occurrence of cancer tumors. In such studies, interest lies in the number of events that occur over a fixed period of time. One considerable challenge in analyzing such data arises when a large proportion of patients discontinues before the end of the study, for example, because of adverse events, leading to partially observed data. In this situation, data are often modeled using a negative binomial distribution with time-in-study as offset. Such an analysis assumes that data are missing at random (MAR). As we cannot test the adequacy of MAR, sensitivity analyses that assess the robustness of conclusions across a range of different assumptions need to be performed. Sophisticated sensitivity analyses for continuous data are being frequently performed. However, this is less the case for recurrent event or count data. We will present a flexible approach to perform clinically interpretable sensitivity analyses for recurrent event data. Our approach fits into the framework of reference-based imputations, where information from reference arms can be borrowed to impute post-discontinuation data. Different assumptions about the future behavior of dropouts dependent on reasons for dropout and received treatment can be made. The imputation model is based on a flexible model that allows for time-varying baseline intensities. We assess the performance in a simulation study and provide an illustration with a clinical trial in patients who suffer from bladder cancer.

Choice of estimand and analysis methods in diabetes trials with rescue medication.

The analysis of clinical trials aiming to show symptomatic benefits is often complicated by the ethical requirement for rescue medication when the disease state of patients worsens. In type 2 diabetes trials, patients receive glucose-lowering rescue medications continuously for the remaining trial duration, if one of several markers of glycemic control exceeds pre-specified thresholds. This may mask differences in glycemic values between treatment groups, because it will occur more frequently in less effective treatment groups. Traditionally, the last pre-rescue medication value was carried forward and analyzed as the end-of-trial value. The deficits of such simplistic single imputation approaches are increasingly recognized by regulatory authorities and trialists. We discuss alternative approaches and evaluate them through a simulation study. When the estimand of interest is the effect attributable to the treatments initially assigned at randomization, then our recommendation for estimation and hypothesis testing is to treat data after meeting rescue criteria as deterministically 'missing' at random, because initiation of rescue medication is determined by observed in-trial values. An appropriate imputation of values after meeting rescue criteria is then possible either directly through multiple imputation or implicitly with a repeated measures model. Crucially, one needs to jointly impute or model all markers of glycemic control that can lead to the initiation of rescue medication. An alternative for hypothesis testing only are rank tests with outcomes from patients 'requiring rescue medication' ranked worst, and non-rescued patients ranked according to final visit values. However, an appropriate ranking of not observed values may be controversial.

Higher Risk of Hypoglycemia with Glimepiride Versus Vildagliptin in Patients with Type 2 Diabetes is not Driven by High Doses of Glimepiride: Divergent Patient Susceptibilities?

INTRODUCTION: In a previously published study, vildagliptin showed a reduced risk of hypoglycemia versus glimepiride as add-on therapy to metformin at similar efficacy. Glimepiride was titrated from a starting dose of 2 mg/day to a maximum dose of 6 mg/day. It is usually assumed that the increased hypoglycemia with glimepiride was driven by the 6 mg/day dose; it was therefore of interest to assess whether the risk of hypoglycemia is also different between vildagliptin and a low (2 mg/day) dose of glimepiride. METHODS: Data (n = 3,059) were from the aforementioned randomized, double-blind study. Comparisons between vildagliptin (50 mg twice daily) and glimepiride (subgroups of patients on 2 mg/day, 6 mg/day, and 'other', and overall glimepiride group) were done by modeling hypoglycemia risk as a function of time and last-measured glycated hemoglobin (HbA1c) using discrete event time modeling, with treatment, age, gender as additional covariates. RESULTS: The hypoglycemia risk was significantly lower in patients receiving vildagliptin versus patients remaining on glimepiride 2 mg/day throughout the study, with similar results unadjusted or adjusted for last HbA1c [adjusted hazard ratio (HR) = 0.06 (95% CI 0.03, 0.11)]. The risk of hypoglycemia was very low with vildagliptin over the full HbA1c range, while the risk with glimepiride 2 mg/day increased with lower HbA1c. The increase for lower levels of HbA1c was more pronounced in the glimepiride 2 mg/day than 6 mg/day subgroup, with the 6 mg/day subgroup showing the lowest hypoglycemia risk among the glimepiride groups [adjusted HR vildagliptin vs. 6 mg/day glimepiride = 0.21 (95% CI 0.11, 0.40)]. CONCLUSION: The data show a substantially lower risk of confirmed hypoglycemia with vildagliptin compared to low-dose (2 mg/day) glimepiride. The analysis indicates that the previously reported results are not driven by high doses of glimepiride and points to interesting differences among patients regarding the susceptibility to hypoglycemia with sulfonylureas.

Estimating time-varying effects for overdispersed recurrent events data with treatment switching.

In the analysis of multivariate event times, frailty models assuming time-independent regression coefficients are often considered, mainly due to their mathematical convenience. In practice, regression coefficients are often time dependent and the temporal effects are of clinical interest. Motivated by a phase III clinical trial in multiple sclerosis, we develop a semiparametric frailty modelling approach to estimate time-varying effects for overdispersed recurrent events data with treatment switching. The proposed model incorporates the treatment switching time in the time-varying coefficients. Theoretical properties of the proposed model are established and an efficient expectation-maximization algorithm is derived to obtain the maximum likelihood estimates. Simulation studies evaluate the numerical performance of the proposed model under various temporal treatment effect curves. The ideas in this paper can also be used for time-varying coefficient frailty models without treatment switching as well as for alternative models when the proportional hazard assumption is violated. A multiple sclerosis dataset is analysed to illustrate our methodology.

Modelling the rate of change in a longitudinal study with missing data, adjusting for contact attempts.

The Collaborative Ankle Support Trial (CAST) is a longitudinal trial of treatments for severe ankle sprains in which interest lies in the rate of improvement, the effectiveness of reminders and potentially informative missingness. A model is proposed for continuous longitudinal data with non-ignorable or informative missingness, taking into account the nature of attempts made to contact initial non-responders. The model combines a non-linear mixed model for the outcome model with logistic regression models for the reminder processes. A sensitivity analysis is used to contrast this model with the traditional selection model, where we adjust for missingness by modelling the missingness process. The conclusions that recovery is slower, and less satisfactory with age and more rapid with below knee cast than with a tubular bandage do not alter materially across all models investigated. The results also suggest that phone calls are most effective in retrieving questionnaires.

The impact of dropouts on the analysis of dose-finding studies with recurrent event data.

This work is motivated by dose-finding studies, where the number of events per subject within a specified study period form the primary outcome. The aim of the considered studies is to identify the target dose for which the new drug can be shown to be as effective as a competitor medication. Given a pain-related outcome, we expect a considerable number of patients to drop out before the end of the study period. The impact of missingness on the analysis and models for the missingness process must be carefully considered.The recurrent events are modeled as over-dispersed Poisson process data, with dose as the regressor. Additional covariates may be included. Constant and time-varying rate functions are examined. Based on these models, the impact of missingness on the precision of the target dose estimation is evaluated. Diverse models for the missingness process are considered, including dependence on covariates and number of events. The performances of five different analysis methods are assessed via simulations: a complete case analysis; two analyses using different single imputation techniques; a direct-likelihood analysis and an analysis using pattern-mixture models.The target dose estimation is robust if the same missingness process holds for the target dose group and the active control group. Furthermore, we demonstrate that this robustness is lost as soon as the missingness mechanisms for the active control and the target dose differ. Of the methods explored, the direct-likelihood approach performs best, even when a missing not at random mechanism holds.