Modern approaches for evaluating treatment effect heterogeneity from clinical trials and observational data.

In this paper, we review recent advances in statistical methods for the evaluation of the heterogeneity of treatment effects (HTE), including subgroup identification and estimation of individualized treatment regimens, from randomized clinical trials and observational studies. We identify several types of approaches using the features introduced in Lipkovich et al (Stat Med 2017;36: 136-196) that distinguish the recommended principled methods from basic methods for HTE evaluation that typically rely on rules of thumb and general guidelines (the methods are often referred to as common practices). We discuss the advantages and disadvantages of various principled methods as well as common measures for evaluating their performance. We use simulated data and a case study based on a historical clinical trial to illustrate several new approaches to HTE evaluation.

Overview of modern approaches for identifying and evaluating heterogeneous treatment effects from clinical data.

There has been much interest in the evaluation of heterogeneous treatment effects (HTE) and multiple statistical methods have emerged under the heading of personalized/precision medicine combining ideas from hypothesis testing, causal inference, and machine learning over the past 10-15 years. We discuss new ideas and approaches for evaluating HTE in randomized clinical trials and observational studies using the features introduced earlier by Lipkovich, Dmitrienko, and D'Agostino that distinguish principled methods from simplistic approaches to data-driven subgroup identification and estimating individual treatment effects and use a case study to illustrate these approaches. We identified and provided a high-level overview of several classes of modern statistical approaches for personalized/precision medicine, elucidated the underlying principles and challenges, and compared findings for a case study across different methods. Different approaches to evaluating HTEs may produce (and actually produced) highly disparate results when applied to a specific data set. Evaluating HTE with machine learning methods presents special challenges since most of machine learning algorithms are optimized for prediction rather than for estimating causal effects. An additional challenge is in that the output of machine learning methods is typically a "black box" that needs to be transformed into interpretable personalized solutions in order to gain acceptance and usability.

A two-stage adaptive clinical trial design with data-driven subgroup identification at interim analysis.

In this paper, we consider randomized controlled clinical trials comparing two treatments in efficacy assessment using a time to event outcome. We assume a relatively small number of candidate biomarkers available in the beginning of the trial, which may help define an efficacy subgroup which shows differential treatment effect. The efficacy subgroup is to be defined by one or two biomarkers and cut-offs that are unknown to the investigator and must be learned from the data. We propose a two-stage adaptive design with a pre-planned interim analysis and a final analysis. At the interim, several subgroup-finding algorithms are evaluated to search for a subgroup with enhanced survival for treated versus placebo. Conditional powers computed based on the subgroup and the overall population are used to make decision at the interim to terminate the study for futility, continue the study as planned, or conduct sample size recalculation for the subgroup or the overall population. At the final analysis, combination tests together with closed testing procedures are used to determine efficacy in the subgroup or the overall population. We conducted simulation studies to compare our proposed procedures with several subgroup-identification methods in terms of a novel utility function and several other measures. This research demonstrated the benefit of incorporating data-driven subgroup selection into adaptive clinical trial designs.

Penalty-based approaches to evaluating multiplicity adjustments in clinical trials: Advanced multiplicity problems.

Given the importance of addressing multiplicity issues in confirmatory clinical trials, several recent publications focused on the general goal of identifying most appropriate methods for multiplicity adjustment in each individual setting. This goal can be accomplished using the Clinical Scenario Evaluation approach. This approach encourages trial sponsors to perform comprehensive assessments of applicable analysis strategies such as multiplicity adjustments under all plausible sets of statistical assumptions using relevant evaluation criteria. This two-part paper applies a novel class of criteria, known as criteria based on multiplicity penalties, to the problem of evaluating the performance of several candidate multiplicity adjustments. The ultimate goal of this evaluation is to identify efficient and robust adjustments for each individual trial and optimally select parameters of these adjustments. Part II focuses on advanced settings with several sources of multiplicity, for example, clinical trials with several endpoints evaluated at two or more doses of an experimental treatment. A case study is given to illustrate a penalty-based approach to evaluating candidate multiple testing procedures in advanced multiplicity problems.

Penalty-based approaches to evaluating multiplicity adjustments in clinical trials: Traditional multiplicity problems.

Given the importance of addressing multiplicity issues in confirmatory clinical trials, several recent publications focused on the general goal of identifying most appropriate methods for multiplicity adjustment in each individual setting. This goal can be accomplished using the Clinical Scenario Evaluation approach. This approach encourages trial sponsors to perform comprehensive assessments of applicable analysis strategies such as multiplicity adjustments under all plausible sets of statistical assumptions using relevant evaluation criteria. This two-part paper applies a novel class of criteria, known as criteria based on multiplicity penalties, to the problem of evaluating the performance of several candidate multiplicity adjustments. The ultimate goal of this evaluation is to identify efficient and robust adjustments for each individual trial and optimally select parameters of these adjustments. Part I deals with traditional problems with a single source of multiplicity. Two case studies based on recently conducted Phase III trials are used to illustrate penalty-based approaches to evaluating candidate multiple testing methods and constructing optimization algorithms.

Mixture-based gatekeeping procedures in adaptive clinical trials.

Clinical trials with data-driven decision rules often pursue multiple clinical objectives such as the evaluation of several endpoints or several doses of an experimental treatment. These complex analysis strategies give rise to "multivariate" multiplicity problems with several components or sources of multiplicity. A general framework for defining gatekeeping procedures in clinical trials with adaptive multistage designs is proposed in this paper. The mixture method is applied to build a gatekeeping procedure at each stage and inferences at each decision point (interim or final analysis) are performed using the combination function approach. An advantage of utilizing the mixture method is that it enables powerful gatekeeping procedures applicable to a broad class of settings with complex logical relationships among the hypotheses of interest. Further, the combination function approach supports flexible data-driven decisions such as a decision to increase the sample size or remove a treatment arm. The paper concludes with a clinical trial example that illustrates the methodology by applying it to develop an adaptive two-stage design with a mixture-based gatekeeping procedure.

Multiplicity issues in exploratory subgroup analysis.

The general topic of subgroup identification has attracted much attention in the clinical trial literature due to its important role in the development of tailored therapies and personalized medicine. Subgroup search methods are commonly used in late-phase clinical trials to identify subsets of the trial population with certain desirable characteristics. Post-hoc or exploratory subgroup exploration has been criticized for being extremely unreliable. Principled approaches to exploratory subgroup analysis based on recent advances in machine learning and data mining have been developed to address this criticism. These approaches emphasize fundamental statistical principles, including the importance of performing multiplicity adjustments to account for selection bias inherent in subgroup search. This article provides a detailed review of multiplicity issues arising in exploratory subgroup analysis. Multiplicity corrections in the context of principled subgroup search will be illustrated using the family of SIDES (subgroup identification based on differential effect search) methods. A case study based on a Phase III oncology trial will be presented to discuss the details of subgroup search algorithms with resampling-based multiplicity adjustment procedures.

An enhanced mixture method for constructing gatekeeping procedures in clinical trials.

It is increasingly common to encounter complex multiplicity problems with several multiplicity components in confirmatory Phase III clinical trials. These components are often based on several endpoints (primary and secondary endpoints) and several dose-control comparisons. When constructing a multiplicity adjustment in these settings, it is important to control the Type I error rate over all multiplicity components. An important class of multiple testing procedures, known as gatekeeping procedures, was derived using the mixture method that enables clinical trial sponsors to set up efficient multiplicity adjustments that account for clinically relevant logical relationships among the hypotheses of interest. An enhanced version of this mixture method is introduced in this paper to construct more powerful gatekeeping procedures for a specific type of logical relationships that rely on transitive serial restrictions. Restrictions of this kind are very common in Phase III clinical trials and the proposed method is applicable to a broad class of multiplicity problems. Several examples are provided to illustrate the new method and results of simulation trials are presented to compare the performance of gatekeeping procedures derived using this method and other available methods.

Tutorial in biostatistics: data-driven subgroup identification and analysis in clinical trials.

It is well known that both the direction and magnitude of the treatment effect in clinical trials are often affected by baseline patient characteristics (generally referred to as biomarkers). Characterization of treatment effect heterogeneity plays a central role in the field of personalized medicine and facilitates the development of tailored therapies. This tutorial focuses on a general class of problems arising in data-driven subgroup analysis, namely, identification of biomarkers with strong predictive properties and patient subgroups with desirable characteristics such as improved benefit and/or safety. Limitations of ad-hoc approaches to biomarker exploration and subgroup identification in clinical trials are discussed, and the ad-hoc approaches are contrasted with principled approaches to exploratory subgroup analysis based on recent advances in machine learning and data mining. A general framework for evaluating predictive biomarkers and identification of associated subgroups is introduced. The tutorial provides a review of a broad class of statistical methods used in subgroup discovery, including global outcome modeling methods, global treatment effect modeling methods, optimal treatment regimes, and local modeling methods. Commonly used subgroup identification methods are illustrated using two case studies based on clinical trials with binary and survival endpoints. Copyright © 2016 John Wiley & Sons, Ltd.

Multiplicity considerations in subgroup analysis.

This paper deals with the general topic of subgroup analysis in late-stage clinical trials with emphasis on multiplicity considerations. The discussion begins with multiplicity issues arising in the context of exploratory subgroup analysis, including principled approaches to subgroup search that are applied as part of subgroup exploration exercises as well as in adaptive biomarker-driven designs. Key considerations in confirmatory subgroup analysis based on one or more pre-specified patient populations are reviewed, including a survey of multiplicity adjustment methods recommended in multi-population phase III clinical trials. Guidelines for interpretation of significant findings in several patient populations are introduced to facilitate the decision-making process and achieve consistent labeling across development programs. Copyright © 2017 John Wiley & Sons, Ltd.

Mixture-based gatekeeping procedures for multiplicity problems with multiple sequences of hypotheses.

Complex multiplicity problems arise in drug development programs with several sets of clinical objectives. This article considers a common setting with two sources of multiplicity induced by the analysis of multiple dose levels based on ordered endpoints. This results in multiplicity problems with multiple sequences of null hypotheses of no effect. Type I error rate inflation in problems of this type is typically addressed by using gatekeeping procedures that account for the hierarchical structure of the trial objectives. A general method for building gatekeeping procedures, known as the mixture method, tends to be conservative in problems with several sequences of hypotheses. This article defines a modified mixture method and shows that this method provides a power advantage over the standard mixture method. In addition, it is demonstrated that in special cases the modified mixture method allows for a stepwise testing algorithm, which facilities the implementation of gatekeeping procedures and general decision making. The new methodology is illustrated using two clinical trial examples.

General guidance on exploratory and confirmatory subgroup analysis in late-stage clinical trials.

This article focuses on a broad class of statistical and clinical considerations related to the assessment of treatment effects across patient subgroups in late-stage clinical trials. This article begins with a comprehensive review of clinical trial literature and regulatory guidelines to help define scientifically sound approaches to evaluating subgroup effects in clinical trials. All commonly used types of subgroup analysis are considered in the article, including different variations of prospectively defined and post-hoc subgroup investigations. In the context of confirmatory subgroup analysis, key design and analysis options are presented, which includes conventional and innovative trial designs that support multi-population tailoring approaches. A detailed summary of exploratory subgroup analysis (with the purpose of either consistency assessment or subgroup identification) is also provided. The article promotes a more disciplined approach to post-hoc subgroup identification and formulates key principles that support reliable evaluation of subgroup effects in this setting.

Tradeoff-based optimization criteria in clinical trials with multiple objectives and adaptive designs.

The article discusses clinical trial optimization problems in the context of mid- to late-stage drug development. Using the Clinical Scenario Evaluation approach, main objectives of clinical trial optimization are formulated, including selection of clinically relevant optimization criteria, identification of sets of optimal and nearly optimal values of the parameters of interest, and sensitivity assessments. The paper focuses on a class of optimization criteria arising in clinical trials with several competing goals, termed tradeoff-based optimization criteria, and discusses key considerations in constructing and applying tradeoff-based criteria. The clinical trial optimization framework considered in the paper is illustrated using two case studies based on a clinical trial with multiple objectives and a two-stage clinical trial which utilizes adaptive decision rules.

Methods for identification and confirmation of targeted subgroups in clinical trials: A systematic review.

Important objectives in the development of stratified medicines include the identification and confirmation of subgroups of patients with a beneficial treatment effect and a positive benefit-risk balance. We report the results of a literature review on methodological approaches to the design and analysis of clinical trials investigating a potential heterogeneity of treatment effects across subgroups. The identified approaches are classified based on certain characteristics of the proposed trial designs and analysis methods. We distinguish between exploratory and confirmatory subgroup analysis, frequentist, Bayesian and decision-theoretic approaches and, last, fixed-sample, group-sequential, and adaptive designs and illustrate the available trial designs and analysis strategies with published case studies.

Multiplicity in confirmatory clinical trials: a case study with discussion from a JSM panel.

An invited panel session was conducted in the 2012 Joint Statistical Meetings, San Diego, California, USA, to stimulate the discussion on multiplicity issues in confirmatory clinical trials for drug development. A total of 11 expert panel members were invited and 9 participated. Prior to the session, a case study was previously provided to the panel members to facilitate the discussion, focusing on the key components of the study design and multiplicity. The Phase 3 development program for this new experimental treatment was based on a single randomized controlled trial alone. Each panelist was asked to clarify if he or she responded as if he or she were a pharmaceutical drug sponsor, an academic panelist or a health regulatory scientist.

Strategies for identifying predictive biomarkers and subgroups with enhanced treatment effect in clinical trials using SIDES.

Several approaches to identification of predictive biomarkers and subgroups of patients with enhanced treatment effect have been proposed in the literature. The SIDES method introduced in Lipkovich et al. (2011) adopts a recursive partitioning algorithm for screening treatment-by-biomarker interactions. This article introduces an improved biomarker discovery/subgroup search method (SIDEScreen). The SIDEScreen method relies on a two-stage procedure that first selects a small number of biomarkers with the highest predictive ability based on an appropriate variable importance score and then identifies subgroups with enhanced treatment effect based on the selected biomarkers. The two-stage approach helps increase the signal-to-noise ratio by screening out noninformative biomarkers. We evaluate operating characteristics of the standard SIDES method and two SIDEScreen procedures based on fixed and adaptive screens. Our main finding is that the adaptive SIDEScreen method is a more flexible biomarker discovery tool than SIDES and it better handles multiplicity in complex subgroup search problems. The methods presented in the article are illustrated using a clinical trial example.

Bayesian assessment of the influence and interaction conditions in multipopulation tailoring clinical trials.

Multipopulation tailoring trials provide a trial design option that supports the realization of tailored therapeutics or personalized medicine. Several recent publications have focused on statistical and clinical considerations that arise in these trials that are designed to study the overall treatment effect in a population of interest as well as one or more prospectively defined subpopulations. Millen et al. (2012) introduced the influence and interaction conditions as part of a general framework to facilitate decision making in multipopulation trials. This article provides Bayesian methods for assessing the influence and interaction conditions. The methods introduced are illustrated using case studies based on clinical trials with biomarker-driven designs.

Assessment of type I error rate associated with dose-group switching in a longitudinal Alzheimer trial.

In clinical trials, there always is the possibility to use data-driven adaptation at the end of a study. There prevails, however, concern on whether the type I error rate of the trial could be inflated with such design, thus, necessitating multiplicity adjustment. In this project, a simulation experiment was set up to assess type I error rate inflation associated with switching dose group as a function of dropout rate at the end of the study, where the primary analysis is in terms of a longitudinal outcome. This simulation is inspired by a clinical trial in Alzheimer's disease. The type I error rate was assessed under a number of scenarios, in terms of differing correlations between efficacy and tolerance, different missingness mechanisms, and different probabilities of switching. A collection of parameter values was used to assess sensitivity of the analysis. Results from ignorable likelihood analysis show that the type I error rate with and without switching was approximately the posited error rate for the various scenarios. Under last observation carried forward (LOCF), the type I error rate blew up both with and without switching. The type I error inflation is clearly connected to the criterion used for switching. While in general switching, in a way related to the primary endpoint, may impact the type I error, this was not the case for most scenarios in the longitudinal Alzheimer trial setting under consideration, where patients are expected to worsen over time.

Superchain procedures in clinical trials with multiple objectives.

This paper introduces a new class of multiple testing procedures for addressing multiplicity problems arising in clinical trials with multiple objectives grouped into families. The families may correspond to equally important sets of objectives (co-primary endpoints) or ordered sets of objectives (primary and secondary endpoints). The procedures, termed superchain procedures, serve as an extension of several classes of other multiple testing procedures, including chain procedures and parallel gatekeeping procedures. Superchain procedures exhibit several desirable features, including flexible decision rules that can be tailored to a broad class of win criteria in confirmatory clinical trials. Additionally, superchain procedures enable the trial's sponsor to efficiently incorporate available distributional information to improve overall power. Finally, superchain procedures rely on straightforward testing algorithms, which facilitates their development. We illustrate the new methodology by using clinical trial examples with two and three families of objectives.