A platform for comparing subgroup identification methodologies.

Since the advent of the phrase "subgroup identification," there has been an explosion of methodologies that seek to identify meaningful subgroups of patients with exceptional response in order to further the realization of personalized medicine. However, to perform fair comparison and understand what methods work best under different clinical trials situations, a common platform is needed for comparative effectiveness of these various approaches. In this paper, we describe a comprehensive project that created an extensive platform for evaluating subgroup identification methods as well as a publicly posted challenge that was used to elicit new approaches. We proposed a common data-generating model for creating virtual clinical trial datasets that contain subgroups of exceptional responders encompassing the many dimensions of the problem or null scenarios in which there are no such subgroups. Furthermore, we created a common scoring system for evaluating performance of purported methods for identifying subgroups. This makes it possible to benchmark methodologies in order to understand what methods work best under different clinical trial situations. The findings from this project produced considerable insights and allow us to make recommendations for how the statistical community can better compare and contrast old and new subgroup identification methodologies.

CITIES: Clinical trials with intercurrent events simulator.

Although clinical trials are often designed with randomization and well-controlled protocols, complications will inevitably arise in the presence of intercurrent events (ICEs) such as treatment discontinuation. These can lead to missing outcome data and possibly confounding causal inference when the missingness is a function of a latent stratification of patients defined by intermediate outcomes. The pharmaceutical industry has been focused on developing new methods that can yield pertinent causal inferences in trials with ICEs. However, it is difficult to compare the properties of different methods developed in this endeavor as real-life clinical trial data cannot be easily shared to provide benchmark data sets. Furthermore, different methods consider distinct assumptions for the underlying data-generating mechanisms, and simulation studies often are customized to specific situations or methods. We develop a novel, general simulation model and corresponding Shiny application in R for clinical trials with ICEs, aptly named the Clinical Trials with Intercurrent Events Simulator (CITIES). It is formulated under the Rubin Causal Model where the considered treatment effects account for ICEs in clinical trials with repeated measures. CITIES facilitates the effective generation of data that resemble real-life clinical trials with respect to their reported summary statistics, without requiring the use of the original trial data. We illustrate the utility of CITIES via two case studies involving real-life clinical trials that demonstrate how CITIES provides a comprehensive tool for practitioners in the pharmaceutical industry to compare methods for the analysis of clinical trials with ICEs on identical, benchmark settings that resemble real-life trials.

Estimating the treatment effect for adherers using multiple imputation.

Randomized controlled trials are considered the gold standard to evaluate the treatment effect (estimand) for efficacy and safety. According to the recent International Council on Harmonization (ICH)-E9 addendum (R1), intercurrent events (ICEs) need to be considered when defining an estimand, and principal stratum is one of the five strategies to handle ICEs. Qu et al. (2020, Statistics in Biopharmaceutical Research 12:1-18) proposed estimators for the adherer average causal effect (AdACE) for estimating the treatment difference for those who adhere to one or both treatments based on the causal-inference framework, and demonstrated the consistency of those estimators; however, this method requires complex custom programming related to high-dimensional numeric integrations. In this article, we implemented the AdACE estimators using multiple imputation (MI) and constructed confidence intervals (CIs) through bootstrapping. A simulation study showed that the MI-based estimators provided consistent estimators with the nominal coverage probabilities of CIs for the treatment difference for the adherent populations of interest. As an illustrative example, the new method was applied to data from a real clinical trial comparing two types of basal insulin for patients with type 1 diabetes.

Assessing and communicating heterogeneity of treatment effects for patient subpopulations: The hardest problem there is.

Heterogeneity is an enormously complex problem because there are so many dimensions and variables that can be considered when assessing which ones may influence an efficacy or safety outcome for an individual patient. This is difficult in randomized controlled trials and even more so in observational settings. An alternative approach is presented in which the individual patient becomes the "subgroup," and similar patients are identified in the clinical trial database or electronic medical record that can be used to predict how that individual patient may respond to treatment.

Implementation of tripartite estimands using adherence causal estimators under the causal inference framework.

Intercurrent events (ICEs) and missing values are inevitable in clinical trials of any size and duration, making it difficult to assess the treatment effect for all patients in randomized clinical trials. Defining the appropriate estimand that is relevant to the clinical research question is the first step in analyzing data. The tripartite estimands, which evaluate the treatment differences in the proportion of patients with ICEs due to adverse events, the proportion of patients with ICEs due to lack of efficacy, and the primary efficacy outcome for those who can adhere to study treatment under the causal inference framework, are of interest to many stakeholders in understanding the totality of treatment effects. In this manuscript, we discuss the details of how to estimate tripartite estimands based on a causal inference framework and how to interpret tripartite estimates through a phase 3 clinical study evaluating a basal insulin treatment for patients with type 1 diabetes.

Assessing and communicating heterogeneity of treatment effects for patient subpopulations: Panel discussion on considerations in design and analysis.

Clinical trials are primarily conducted to understand the average effects treatments have on patients. However, patients are heterogeneous in the severity of the condition and in ways that affect what treatment effect they can expect. It is therefore important to understand and characterize how treatment effects vary. The design and analysis of clinical studies play critical roles in evaluating and characterizing heterogeneous treatment effects. This panel discussed considerations in design and analysis under the recognition that there are heterogeneous treatment effects across subgroups of patients. Panel members discussed many questions including: What is a good estimate of the treatment effect in me, a 65-year-old, bald, Caucasian-American, male patient? What magnitude of heterogeneity of treatment effects (HTE) is sufficiently large to merit attention? What role can prior evidence about HTE play in confirmatory trial design and analysis? Is there anything described in the 21st Century Cures Act that would benefit from greater attention to HTE? An example of a Bayesian approach addressing multiplicity when testing for treatment effects in subgroups will be provided. We can do more or better at understanding heterogeneous treatment effects and providing the best information on heterogeneous treatment effects.

What matters most? Different stakeholder perspectives on estimands for an invented case study in COPD.

In drug development, we ask ourselves which population, endpoint and treatment comparison should be investigated. In this context, we also debate what matters most to the different stakeholders that are involved in clinical drug development, for example, patients, physicians, regulators and payers. With the publication of draft ICH E9 addendum on estimands in 2017, we now have a common framework and language to discuss such questions in an informed and transparent way. This has led to the estimand discussion being a key element in study development, including design, analysis and interpretation of a treatment effect. At an invited session at the 2018 PSI annual conference, PSI hosted a role-play debate where the aim of the session was to mimic a regulatory and payer scientific advice discussion for a COPD drug. Including role-play views from an industry sponsor, a patient, a regulator and a payer. This paper presents the invented COPD case-study design and considerations relating to appropriate estimands are discussed by each of the stakeholders from their differing viewpoints with the additional inclusion of a technical (academic) perspective. The rationale for each perspective on approaches for handling intercurrent events is presented, with a key emphasis on the application of while-on-treatment and treatment policy estimands in this context. It is increasingly recognised that the treatment effect estimated by the treatment policy approach may not always be of primary clinical interest and may not appropriately communicate to patients the efficacy they can expect if they take the treatment as directed.

Efficiency of two sample tests via the restricted mean survival time for analyzing event time observations.

In comparing two treatments with the event time observations, the hazard ratio (HR) estimate is routinely used to quantify the treatment difference. However, this model dependent estimate may be difficult to interpret clinically especially when the proportional hazards (PH) assumption is violated. An alternative estimation procedure for treatment efficacy based on the restricted means survival time or t-year mean survival time (t-MST) has been discussed extensively in the statistical and clinical literature. On the other hand, a statistical test via the HR or its asymptotically equivalent counterpart, the logrank test, is asymptotically distribution-free. In this article, we assess the relative efficiency of the hazard ratio and t-MST tests with respect to the statistical power under various PH and non-PH models theoretically and empirically. When the PH assumption is valid, the t-MST test performs almost as well as the HR test. For non-PH models, the t-MST test can substantially outperform its HR counterpart. On the other hand, the HR test can be powerful when the true difference of two survival functions is quite large at end but not the beginning of the study. Unfortunately, for this case, the HR estimate may not have a simple clinical interpretation for the treatment effect due to the violation of the PH assumption.

Estimands in clinical trials - broadening the perspective.

Defining the scientific questions of interest in a clinical trial is crucial to align its planning, design, conduct, analysis, and interpretation. However, practical experience shows that oftentimes specific choices in the statistical analysis blur the scientific question either in part or even completely, resulting in misalignment between trial objectives, conduct, analysis, and confusion in interpretation. The need for more clarity was highlighted by the Steering Committee of the International Council for Harmonization (ICH) in 2014, which endorsed a Concept Paper with the goal of developing a new regulatory guidance, suggested to be an addendum to ICH guideline E9. Triggered by these developments, we elaborate in this paper what the relevant questions in drug development are and how they fit with the current practice of intention-to-treat analyses. To this end, we consider the perspectives of patients, physicians, regulators, and payers. We argue that despite the different backgrounds and motivations of the various stakeholders, they all have similar interests in what the clinical trial estimands should be. Broadly, these can be classified into estimands addressing (a) lack of adherence to treatment due to different reasons and (b) efficacy and safety profiles when patients, in fact, are able to adhere to the treatment for its intended duration. We conclude that disentangling adherence to treatment and the efficacy and safety of treatment in patients that adhere leads to a transparent and clinical meaningful assessment of treatment risks and benefits. We touch upon statistical considerations and offer a discussion of additional implications. Copyright © 2016 John Wiley & Sons, Ltd.

Making what's advanced today routine tomorrow.

Statistical principles and ongoing proliferation of novel statistical methodologies have dramatically improved the clinical drug development process. This journey over the last seven decades reshaped the pharmaceutical industry and regulatory agencies, highlighted the importance of statistical thinking in drug development and decision-making, and, most importantly, improved the lives of countless patients around the world. Some significant highlights in the history of this journey are recounted here as well as some exciting opportunities of what the future may hold for the science and profession of statistics.

Identifying potential adverse events dose-response relationships via Bayesian indirect and mixed treatment comparison models.

Patients and prescribers need to be fully informed regarding the safety profile of approved medications. This includes knowledge and information regarding whether an adverse event of interest exhibits a potential dose-response relationship. In order to thoroughly evaluate whether an adverse event rate increases with increasing dose level, evidence from multiple clinical trials needs to be combined and analyzed. The various clinical trials that need to be combined often include different dose levels. If one evaluates the dose-response relationship by including only the trials with all of the common dose levels, this will lead to the exclusion of potentially several clinical trials as well as dose levels, and thus the loss of important information. Other methods, such as crudely pooling patients on the same dose level across different studies, are subject to bias due to the breakdown of randomization. It is preferable to include all studies and relevant dose levels, even if all studies do not contain the same dose levels. Bayesian methodology has been shown to be useful in the context of indirect and mixed treatment comparison methods, to combine information from different therapies in different studies in order to make treatment effect inferences. This type of approach is foundational to the models presented here, but instead of modeling different dose arms in different studies, we extend the methodology to allow for assessment of the dose-response relationship across multiple clinical trials. In this paper, we propose three Bayesian indirect/mixed treatment comparison models to assess adverse event dose-response relationships. These three models are designed to handle binary responses and time to event responses. We apply the methods to real data sets and demonstrate that our proposed methods are useful in discovering potential dose-response relationships.

Accurate Collection of Reasons for Treatment Discontinuation to Better Define Estimands in Clinical Trials.

BACKGROUND: Reasons for treatment discontinuation are important not only to understand the benefit and risk profile of experimental treatments, but also to help choose appropriate strategies to handle intercurrent events in defining estimands. The current case report form (CRF) commonly in use mixes the underlying reasons for treatment discontinuation and who makes the decision for treatment discontinuation, often resulting in an inaccurate collection of reasons for treatment discontinuation. METHODS AND RESULTS: We systematically reviewed and analyzed treatment discontinuation data from nine phase 2 and phase 3 studies for insulin peglispro. A total of 857 participants with treatment discontinuation were included in the analysis. Our review suggested that, due to the vague multiple-choice options for treatment discontinuation present in the CRF, different reasons were sometimes recorded for the same underlying reason for treatment discontinuation. Based on our review and analysis, we suggest an intermediate solution and a more systematic way to improve the current CRF for treatment discontinuations. CONCLUSION: This research provides insight and directions on how to optimize the CRF for recording treatment discontinuation. Further work needs to be done to build the learning into Clinical Data Interchange Standards Consortium standards. CLINICAL TRIALS: Clinicaltrials.gov numbers: NCT01027871 (Phase 2 for type 2 diabetes), NCT01049412 (Phase 2 for type 1 diabetes), NCT01481779 (IMAGINE 1 Study), NCT01435616 (IMAGINE 2 Study), NCT01454284 (IMAGINE 3 Study), NCT01468987 (IMAGINE 4 Study), NCT01582451 (IMAGINE 5 Study), NCT01790438 (IMAGINE 6 Study), NCT01792284 (IMAGINE 7 Study).

Application of Bayesian approaches in drug development: starting a virtuous cycle.

The pharmaceutical industry and its global regulators have routinely used frequentist statistical methods, such as null hypothesis significance testing and p values, for evaluation and approval of new treatments. The clinical drug development process, however, with its accumulation of data over time, can be well suited for the use of Bayesian statistical approaches that explicitly incorporate existing data into clinical trial design, analysis and decision-making. Such approaches, if used appropriately, have the potential to substantially reduce the time and cost of bringing innovative medicines to patients, as well as to reduce the exposure of patients in clinical trials to ineffective or unsafe treatment regimens. Nevertheless, despite advances in Bayesian methodology, the availability of the necessary computational power and growing amounts of relevant existing data that could be used, Bayesian methods remain underused in the clinical development and regulatory review of new therapies. Here, we highlight the value of Bayesian methods in drug development, discuss barriers to their application and recommend approaches to address them. Our aim is to engage stakeholders in the process of considering when the use of existing data is appropriate and how Bayesian methods can be implemented more routinely as an effective tool for doing so.

A portfolio-based approach to optimize proof-of-concept clinical trials.

Improving proof-of-concept (PoC) studies is a primary lever for improving drug development. Since drug development is often done by institutions that work on multiple drugs simultaneously, the present work focused on optimum choices for rates of false positive (α) and false negative (ß) results across a portfolio of PoC studies. Simple examples and a newly derived equation provided conceptual understanding of basic principles regarding optimum choices of α and ß in PoC trials. In examples that incorporated realistic development costs and constraints, the levels of α and ß that maximized the number of approved drugs and portfolio value varied by scenario. Optimum choices were sensitive to the probability the drug was effective and to the proportion of total investment cost prior to establishing PoC. Results of the present investigation agree with previous research in that it is important to assess optimum levels of α and ß. However, the present work also highlighted the need to consider cost structure using realistic input parameters relevant to the question of interest.

Subgroup identification from randomized clinical trial data.

We consider the problem of identifying a subgroup of patients who may have an enhanced treatment effect in a randomized clinical trial, and it is desirable that the subgroup be defined by a limited number of covariates. For this problem, the development of a standard, pre-determined strategy may help to avoid the well-known dangers of subgroup analysis. We present a method developed to find subgroups of enhanced treatment effect. This method, referred to as 'Virtual Twins', involves predicting response probabilities for treatment and control 'twins' for each subject. The difference in these probabilities is then used as the outcome in a classification or regression tree, which can potentially include any set of the covariates. We define a measure Q(Â) to be the difference between the treatment effect in estimated subgroup  and the marginal treatment effect. We present several methods developed to obtain an estimate of Q(Â), including estimation of Q(Â) using estimated probabilities in the original data, using estimated probabilities in newly simulated data, two cross-validation-based approaches, and a bootstrap-based bias-corrected approach. Results of a simulation study indicate that the Virtual Twins method noticeably outperforms logistic regression with forward selection when a true subgroup of enhanced treatment effect exists. Generally, large sample sizes or strong enhanced treatment effects are needed for subgroup estimation. As an illustration, we apply the proposed methods to data from a randomized clinical trial.

Identification of early changes in specific symptoms that predict longer-term response to atypical antipsychotics in the treatment of patients with schizophrenia.

BACKGROUND: To identify a simple decision tree using early symptom change to predict response to atypical antipsychotic therapy in patients with (Diagnostic and Statistical Manual, Fourth Edition, Text Revised) chronic schizophrenia. METHODS: Data were pooled from moderately to severely ill patients (n = 1494) from 6 randomized, double-blind trials (N = 2543). Response was defined as a ≥ 30% reduction in Positive and Negative Syndrome Scale (PANSS) Total score by Week 8 of treatment. Analyzed predictors were change in individual PANSS items at Weeks 1 and 2. A decision tree was constructed using classification and regression tree (CART) analysis to identify predictors that most effectively differentiated responders from non-responders. RESULTS: A 2-branch, 6-item decision tree was created, producing 3 distinct groups. First branch criterion was a 2-point score decrease in at least 2 of 5 PANSS positive items (Week 2). Second branch criterion was a 2-point score decrease in the PANSS excitement item (Week 2). "Likely responders" met the first branch criteria; "likely non-responders" did not meet first or second criterion; "not predictable" patients did not meet the first but did meet the second criterion. Using this approach, response to treatment could be predicted in most patients (92%) with high positive predictive value (79%) and high negative predictive value (75%). Predictive findings were confirmed through analysis of data from 2 independent trials. CONCLUSIONS: Using a data-driven approach, we identified decision rules using early change in the scores of selected PANSS items to accurately predict longer-term treatment response or non-response to atypical antipsychotic therapy. This could lead to development of a simple quantitative evaluation tool to help guide early treatment decisions. TRIAL REGISTRATION: This is a retrospective, non-intervention study in which pooled results from 6 previously published reports were analyzed; thus, clinical trial registration is not required.

Détente: A Practical Understanding of P values and Bayesian Posterior Probabilities.

Null hypothesis significance testing (NHST) with its benchmark P value < 0.05 has long been a stalwart of scientific reporting and such statistically significant findings have been used to imply scientifically or clinically significant findings. Challenges to this approach have arisen over the past 6 decades, but they have largely been unheeded. There is a growing movement for using Bayesian statistical inference to quantify the probability that a scientific finding is credible. There have been differences of opinion between the frequentist (i.e., NHST) and Bayesian schools of inference, and warnings about the use or misuse of P values have come from both schools of thought spanning many decades. Controversies in this arena have been heightened by the American Statistical Association statement on P values and the further denouncement of the term "statistical significance" by others. My experience has been that many scientists, including many statisticians, do not have a sound conceptual grasp of the fundamental differences in these approaches, thereby creating even greater confusion and acrimony. If we let A represent the observed data, and B represent the hypothesis of interest, then the fundamental distinction between these two approaches can be described as the frequentist approach using the conditional probability pr(A | B) (i.e., the P value), and the Bayesian approach using pr(B | A) (the posterior probability). This paper will further explain the fundamental differences in NHST and Bayesian approaches and demonstrate how they can co-exist harmoniously to guide clinical trial design and inference.

The mean does not mean as much anymore: finding sub-groups for tailored therapeutics.

BACKGROUND: The genomics revolution is still in its infancy, and there is much to learn about how to transform biological knowledge into useful medicines to further public health. At the bedside, we are asking how and why individual patients respond to different drug treatments in different ways. In addition to genetic mechanisms, there are many clinical markers (e.g. medical history, disease severity) as well as social/environmental factors (e.g. smoking habits) that can be used to identify who may or may not respond to treatment. PURPOSE: This issue has some considerable statistical complexity, and different approaches to the analysis of clinical trials may yield more interesting insights into the problem. Novel applications of statistical methods will be discussed, and examples will be used to demonstrate sub-group identification. METHODS: In order to evaluate many potential predictors of response, we use recursive partitioning methods to identify predictor variables and their cut-off values to define sub-groups of patients with differential treatment response. Validation of this variable/model selection approach was done using independent data from other clinical trials. RESULTS: In one example, a classification tree was developed using baseline measures to define important sub-groups of patients that responded much better than the overall mean response in the study. In a second example, a classification tree was built based on measures of response early in treatment to predict longer-term responders and nonresponders. Limitation Classification algorithms can be prone to over-fitting, and validation of results is an important consideration. Obviously, analyses are limited by the available predictor variables. CONCLUSIONS: Using classification trees proved to be very useful in evaluating large numbers of potential predictors to find sub-groups of patients with exceptional response. The method is easy to use, and clinicians can easily interpret and implement results. This approach can be helpful in tailoring treatments to individual patients.

Considerations for Evaluating Treatment Effects From Randomized Clinical Trials.

This article focuses on the choice of treatment effect measures in randomized clinical trials (RCTs). Traditionally, an intention-to-treat (ITT) analysis is conducted with an implicit understanding that a treatment-policy effect is of greatest interest. In this article we contend that this approach may not always provide accurate information about clinically meaningful treatment effects, and we present an argument that for any RCT it is desirable to require an explicit definition of what treatment effect is of primary interest, known as the "estimand." We will discuss the limitations of the traditional ITT effect measures as well as the state-of-the art thinking with regard to estimands. Furthermore, we will offer alternate choices that acknowledge that treatments have multiple effects.