Overview of modern approaches for identifying and evaluating heterogeneous treatment effects from clinical data.

There has been much interest in the evaluation of heterogeneous treatment effects (HTE) and multiple statistical methods have emerged under the heading of personalized/precision medicine combining ideas from hypothesis testing, causal inference, and machine learning over the past 10-15 years. We discuss new ideas and approaches for evaluating HTE in randomized clinical trials and observational studies using the features introduced earlier by Lipkovich, Dmitrienko, and D'Agostino that distinguish principled methods from simplistic approaches to data-driven subgroup identification and estimating individual treatment effects and use a case study to illustrate these approaches. We identified and provided a high-level overview of several classes of modern statistical approaches for personalized/precision medicine, elucidated the underlying principles and challenges, and compared findings for a case study across different methods. Different approaches to evaluating HTEs may produce (and actually produced) highly disparate results when applied to a specific data set. Evaluating HTE with machine learning methods presents special challenges since most of machine learning algorithms are optimized for prediction rather than for estimating causal effects. An additional challenge is in that the output of machine learning methods is typically a "black box" that needs to be transformed into interpretable personalized solutions in order to gain acceptance and usability.

Practical recommendations on double score matching for estimating causal effects.

Unlike in randomized clinical trials (RCTs), confounding control is critical for estimating the causal effects from observational studies due to the lack of treatment randomization. Under the unconfoundedness assumption, matching methods are popular because they can be used to emulate an RCT that is hidden in the observational study. To ensure the key assumption hold, the effort is often made to collect a large number of possible confounders, rendering dimension reduction imperative in matching. Three matching schemes based on the propensity score (PSM), prognostic score (PGM), and double score (DSM, ie, the collection of the first two scores) have been proposed in the literature. However, a comprehensive comparison is lacking among the three matching schemes and has not made inroads into the best practices including variable selection, choice of caliper, and replacement. In this article, we explore the statistical and numerical properties of PSM, PGM, and DSM via extensive simulations. Our study supports that DSM performs favorably with, if not better than, the two single score matching in terms of bias and variance. In particular, DSM is doubly robust in the sense that the matching estimator is consistent requiring either the propensity score model or the prognostic score model is correctly specified. Variable selection on the propensity score model and matching with replacement is suggested for DSM, and we illustrate the recommendations with comprehensive simulation studies. An R package is available at https://github.com/Yunshu7/dsmatch.

Using principal stratification in analysis of clinical trials.

The ICH E9(R1) addendum (2019) proposed principal stratification (PS) as one of five strategies for dealing with intercurrent events. Therefore, understanding the strengths, limitations, and assumptions of PS is important for the broad community of clinical trialists. Many approaches have been developed under the general framework of PS in different areas of research, including experimental and observational studies. These diverse applications have utilized a diverse set of tools and assumptions. Thus, need exists to present these approaches in a unifying manner. The goal of this tutorial is threefold. First, we provide a coherent and unifying description of PS. Second, we emphasize that estimation of effects within PS relies on strong assumptions and we thoroughly examine the consequences of these assumptions to understand in which situations certain assumptions are reasonable. Finally, we provide an overview of a variety of key methods for PS analysis and use a real clinical trial example to illustrate them. Examples of code for implementation of some of these approaches are given in Supplemental Materials.

Evaluating bias control strategies in observational studies using frequentist model averaging.

Estimating a treatment effect from observational data requires modeling treatment and outcome subject to uncertainty/misspecification. A previous research has shown that it is not possible to find a uniformly best strategy. In this article we propose a novel Frequentist Model Averaging (FMA) framework encompassing any estimation strategy and accounting for model uncertainty by computing a cross-validated estimate of Mean Squared Prediction Error (MSPE). We present a simulation study with data mimicking an observational database. Model averaging over 15+ strategies was compared with individual strategies as well as the best strategy selected by minimum MSPE. FMA showed robust performance (Bias, Mean Squared Error (MSE), and Confidence Interval (CI) coverage). Other strategies, such as linear regression, did well in simple scenarios but were inferior to the FMA in a scenario with complex confounding.

A two-stage adaptive clinical trial design with data-driven subgroup identification at interim analysis.

In this paper, we consider randomized controlled clinical trials comparing two treatments in efficacy assessment using a time to event outcome. We assume a relatively small number of candidate biomarkers available in the beginning of the trial, which may help define an efficacy subgroup which shows differential treatment effect. The efficacy subgroup is to be defined by one or two biomarkers and cut-offs that are unknown to the investigator and must be learned from the data. We propose a two-stage adaptive design with a pre-planned interim analysis and a final analysis. At the interim, several subgroup-finding algorithms are evaluated to search for a subgroup with enhanced survival for treated versus placebo. Conditional powers computed based on the subgroup and the overall population are used to make decision at the interim to terminate the study for futility, continue the study as planned, or conduct sample size recalculation for the subgroup or the overall population. At the final analysis, combination tests together with closed testing procedures are used to determine efficacy in the subgroup or the overall population. We conducted simulation studies to compare our proposed procedures with several subgroup-identification methods in terms of a novel utility function and several other measures. This research demonstrated the benefit of incorporating data-driven subgroup selection into adaptive clinical trial designs.

Evaluating the relationship between clinical and demographic characteristics of insulin-using people with diabetes and their health outcomes: a cluster analysis application.

BACKGROUND: The aim of this study was to determine how clusters or subgroups of insulin-treated people with diabetes, based upon healthcare resource utilization, select social demographic and clinical characteristics, and diabetes management parameters, are related to health outcomes including acute care visits and hospital admissions. METHODS: This was a non-experimental, retrospective cluster analysis. We utilized Aetna administrative claims data to identify insulin-using people with diabetes with service dates from 01 January 2015 to 30 June 2018. The study included adults over the age of 18 years who had a diagnosis of type 1 (T1DM) or type 2 diabetes mellitus (T2DM) on insulin therapy and had Aetna medical and pharmacy coverage for at least 18 months (6 months prior and 12 months after their index date, defined as either their first insulin prescription fill date or their earliest date allowing for 6 months' prior coverage). We used K-means clustering methods to identify relevant subgroups of people with diabetes based on 13 primary outcome variables. RESULTS: A total of 100,650 insulin-using people with diabetes were identified in the Aetna administrative claims database and met study criteria, including 11,826 (11.7%) with T1DM and 88,824 (88.3%) with T2DM. Of these 79,053 (78.5%) people were existing insulin users. Seven distinct clusters were identified with different characteristics and potential risks of diabetes complications. Overall, clusters were significantly associated with differences in healthcare utilization (emergency room visits, inpatient admissions, and total inpatient days) after multivariable adjustment. CONCLUSIONS: This analysis of healthcare claims data using clustering methodologies identified meaningful subgroups of patients with diabetes using insulin. The subgroups differed in comorbidity burden, healthcare utilization, and demographic factors which could be used to identify higher risk patients and/or guide the management and treatment of diabetes.

Application of structured statistical analyses to identify a biomarker predictive of enhanced tralokinumab efficacy in phase III clinical trials for severe, uncontrolled asthma.

BACKGROUND: Tralokinumab is an anti-interleukin (IL)-13 monoclonal antibody investigated for the treatment of severe, uncontrolled asthma in two Phase III clinical trials, STRATOS 1 and 2. The STRATOS 1 biomarker analysis plan was developed to identify biomarker(s) indicative of IL-13 activation likely to predict tralokinumab efficacy and define a population in which there was an enhanced treatment effect; this defined population was then tested in STRATOS 2. METHODS: The biomarkers considered were blood eosinophil counts, fractional exhaled nitric oxide (FeNO), serum dipeptidyl peptidase-4, serum periostin and total serum immunoglobulin E. Tralokinumab efficacy was measured as the reduction in annualised asthma exacerbation rate (AAER) compared with placebo (primary endpoint measure of STRATOS 1 and 2). The biomarker analysis plan included negative binomial and generalised additive models, and the Subgroup Identification based on Differential Effect Search (SIDES) algorithm, supported by robustness and sensitivity checks. Effects on the key secondary endpoints of STRATOS 1 and 2, which included changes from baseline in standard measures of asthma outcomes, were also investigated. Prior to the STRATOS 1 read-out, numerous simulations of the methodology were performed with hypothetical data. RESULTS: FeNO and periostin were identified as the only biomarkers potentially predictive of treatment effect, with cut-offs chosen by the SIDES algorithm of > 32.3 ppb and > 27.4 ng/ml, respectively. The FeNO > 32.3 ppb subgroup was associated with greater AAER reductions and improvements in key secondary endpoints compared with the periostin > 27.4 ng/ml subgroup. Upon further evaluation of AAER reductions at different FeNO cut-offs, ≥37 ppb was chosen as the best cut-off for predicting tralokinumab efficacy. DISCUSSION: A rigorous statistical approach incorporating multiple methods was used to investigate the predictive properties of five potential biomarkers and to identify a participant subgroup that demonstrated an enhanced tralokinumab treatment effect. Using STRATOS 1 data, our analyses identified FeNO at a cut-off of ≥37 ppb as the best assessed biomarker for predicting enhanced treatment effect to be tested in STRATOS 2. Our findings were inconclusive, which reflects the complexity of subgroup identification in the severe asthma population. TRIAL REGISTRATION: STRATOS 1 and 2 are registered on ClinicalTrials.gov ( NCT02161757 registered on June 12, 2014, and NCT02194699 registered on July 18, 2014).

Subgroup analysis and interpretation for phase 3 confirmatory trials: White paper of the EFSPI/PSI working group on subgroup analysis.

Subgroup by treatment interaction assessments are routinely performed when analysing clinical trials and are particularly important for phase 3 trials where the results may affect regulatory labelling. Interpretation of such interactions is particularly difficult, as on one hand the subgroup finding can be due to chance, but equally such analyses are known to have a low chance of detecting differential treatment effects across subgroup levels, so may overlook important differences in therapeutic efficacy. EMA have therefore issued draft guidance on the use of subgroup analyses in this setting. Although this guidance provided clear proposals on the importance of pre-specification of likely subgroup effects and how to use this when interpreting trial results, it is less clear which analysis methods would be reasonable, and how to interpret apparent subgroup effects in terms of whether further evaluation or action is necessary. A PSI/EFSPI Working Group has therefore been investigating a focused set of analysis approaches to assess treatment effect heterogeneity across subgroups in confirmatory clinical trials that take account of the number of subgroups explored and also investigating the ability of each method to detect such subgroup heterogeneity. This evaluation has shown that the plotting of standardised effects, bias-adjusted bootstrapping method and SIDES method all perform more favourably than traditional approaches such as investigating all subgroup-by-treatment interactions individually or applying a global test of interaction. Therefore, these approaches should be considered to aid interpretation and provide context for observed results from subgroup analyses conducted for phase 3 clinical trials.

Identifying optimal dosage regimes under safety constraints: An application to long term opioid treatment of chronic pain.

There is growing interest and investment in precision medicine as a means to provide the best possible health care. A treatment regime formalizes precision medicine as a sequence of decision rules, one per clinical intervention period, that specify if, when and how current treatment should be adjusted in response to a patient's evolving health status. It is standard to define a regime as optimal if, when applied to a population of interest, it maximizes the mean of some desirable clinical outcome, such as efficacy. However, in many clinical settings, a high-quality treatment regime must balance multiple competing outcomes; eg, when a high dose is associated with substantial symptom reduction but a greater risk of an adverse event. We consider the problem of estimating the most efficacious treatment regime subject to constraints on the risk of adverse events. We combine nonparametric Q-learning with policy-search to estimate a high-quality yet parsimonious treatment regime. This estimator applies to both observational and randomized data, as well as settings with variable, outcome-dependent follow-up, mixed treatment types, and multiple time points. This work is motivated by and framed in the context of dosing for chronic pain; however, the proposed framework can be applied generally to estimate a treatment regime which maximizes the mean of one primary outcome subject to constraints on one or more secondary outcomes. We illustrate the proposed method using data pooled from 5 open-label flexible dosing clinical trials for chronic pain.

Multiplicity issues in exploratory subgroup analysis.

The general topic of subgroup identification has attracted much attention in the clinical trial literature due to its important role in the development of tailored therapies and personalized medicine. Subgroup search methods are commonly used in late-phase clinical trials to identify subsets of the trial population with certain desirable characteristics. Post-hoc or exploratory subgroup exploration has been criticized for being extremely unreliable. Principled approaches to exploratory subgroup analysis based on recent advances in machine learning and data mining have been developed to address this criticism. These approaches emphasize fundamental statistical principles, including the importance of performing multiplicity adjustments to account for selection bias inherent in subgroup search. This article provides a detailed review of multiplicity issues arising in exploratory subgroup analysis. Multiplicity corrections in the context of principled subgroup search will be illustrated using the family of SIDES (subgroup identification based on differential effect search) methods. A case study based on a Phase III oncology trial will be presented to discuss the details of subgroup search algorithms with resampling-based multiplicity adjustment procedures.

Points to consider for analyzing efficacy outcomes in long-term extension clinical trials.

This article focuses on 2 objectives in the analysis of efficacy in long-term extension studies of chronic diseases: (1) defining and discussing estimands of interest in such studies and (2) evaluating the performance of several multiple imputation methods that may be useful in estimating some of these estimands. Specifically, 4 estimands are defined and their clinical utility and inferential ramifications discussed. The performance of several multiple imputation methods and approaches were evaluated using simulated data. Results suggested that when interest is in a binary outcome derived from an underlying continuous measurement, it is preferable to impute the underlying continuous value that is subsequently dichotomized rather than to directly impute the binary outcome. Results also demonstrated that multivariate Gaussian models with Markov chain Monte Carlo imputation and sequential regression have minimal bias and the anticipated confidence interval coverage, even in settings with ordinal data where departures from normality are a concern. These approaches are further illustrated using a long-term extension study in psoriasis.

Multiplicity considerations in subgroup analysis.

This paper deals with the general topic of subgroup analysis in late-stage clinical trials with emphasis on multiplicity considerations. The discussion begins with multiplicity issues arising in the context of exploratory subgroup analysis, including principled approaches to subgroup search that are applied as part of subgroup exploration exercises as well as in adaptive biomarker-driven designs. Key considerations in confirmatory subgroup analysis based on one or more pre-specified patient populations are reviewed, including a survey of multiplicity adjustment methods recommended in multi-population phase III clinical trials. Guidelines for interpretation of significant findings in several patient populations are introduced to facilitate the decision-making process and achieve consistent labeling across development programs. Copyright © 2017 John Wiley & Sons, Ltd.

Tutorial in biostatistics: data-driven subgroup identification and analysis in clinical trials.

It is well known that both the direction and magnitude of the treatment effect in clinical trials are often affected by baseline patient characteristics (generally referred to as biomarkers). Characterization of treatment effect heterogeneity plays a central role in the field of personalized medicine and facilitates the development of tailored therapies. This tutorial focuses on a general class of problems arising in data-driven subgroup analysis, namely, identification of biomarkers with strong predictive properties and patient subgroups with desirable characteristics such as improved benefit and/or safety. Limitations of ad-hoc approaches to biomarker exploration and subgroup identification in clinical trials are discussed, and the ad-hoc approaches are contrasted with principled approaches to exploratory subgroup analysis based on recent advances in machine learning and data mining. A general framework for evaluating predictive biomarkers and identification of associated subgroups is introduced. The tutorial provides a review of a broad class of statistical methods used in subgroup discovery, including global outcome modeling methods, global treatment effect modeling methods, optimal treatment regimes, and local modeling methods. Commonly used subgroup identification methods are illustrated using two case studies based on clinical trials with binary and survival endpoints. Copyright © 2016 John Wiley & Sons, Ltd.

Bridging data across studies using frequentist and Bayesian estimation.

In drug development programs, an experimental treatment is evaluated across different populations and/or disease types using multiple studies conducted in countries around the world. In order to show the efficacy and safety in a specific population, a bridging study may be required. There are therapeutic areas for which enrolling patients to a trial is very challenging. Therefore, it is of interest to utilize the available historical information from previous studies. However, treatment effect may vary across different subpopulations/disease types; therefore, directly utilizing outcomes from historical studies may result in a biased estimation of treatment effect under investigation in the target trial. In this article, we propose novel approaches using both frequentist and Bayesian frameworks that allow borrowing information from historical studies while accounting for relevant patient's covariates via a propensity-based weighting. We evaluate the operating characteristics of the proposed methods in a simulation study and demonstrate that under certain conditions these methods may lead to improved estimation of a treatment effect.

Evaluating different strategies for estimating treatment effects in observational studies.

Since the introduction of the propensity score (PS), methods for estimating treatment effects with observational data have received growing attention in the literature. Recent research has added substantially to the number of available statistical approaches for controlling confounding in such analyses. However, researchers need guidance to decide on the optimal analytic strategy for any given scenario. To address this gap, we conducted simulations evaluating both well-established methods (regression, PS weighting, stratification, and matching) and more recently proposed approaches (tree-based methods, local control, entropy balancing, genetic matching, prognostic scoring). The simulation scenarios included tree-based and smooth regression models as true data-generation mechanisms. We evaluated an extensive number of analysis strategies combining different treatment choices and outcome models. Key findings include 1) the lack of a single best strategy across all potential scenarios; 2) the importance of appropriately addressing interactions in the treatment choice model and/or outcome model; and 3) a tree-structured treatment choice model and a polynomial outcome model with second-order interactions performed well. One limitation to this initial assessment is the lack of heterogeneous simulation scenarios allowing treatment effects to vary by patient.

General guidance on exploratory and confirmatory subgroup analysis in late-stage clinical trials.

This article focuses on a broad class of statistical and clinical considerations related to the assessment of treatment effects across patient subgroups in late-stage clinical trials. This article begins with a comprehensive review of clinical trial literature and regulatory guidelines to help define scientifically sound approaches to evaluating subgroup effects in clinical trials. All commonly used types of subgroup analysis are considered in the article, including different variations of prospectively defined and post-hoc subgroup investigations. In the context of confirmatory subgroup analysis, key design and analysis options are presented, which includes conventional and innovative trial designs that support multi-population tailoring approaches. A detailed summary of exploratory subgroup analysis (with the purpose of either consistency assessment or subgroup identification) is also provided. The article promotes a more disciplined approach to post-hoc subgroup identification and formulates key principles that support reliable evaluation of subgroup effects in this setting.

Sensitivity to censored-at-random assumption in the analysis of time-to-event endpoints.

Over the past years, significant progress has been made in developing statistically rigorous methods to implement clinically interpretable sensitivity analyses for assumptions about the missingness mechanism in clinical trials for continuous and (to a lesser extent) for binary or categorical endpoints. Studies with time-to-event outcomes have received much less attention. However, such studies can be similarly challenged with respect to the robustness and integrity of primary analysis conclusions when a substantial number of subjects withdraw from treatment prematurely prior to experiencing an event of interest. We discuss how the methods that are widely used for primary analyses of time-to-event outcomes could be extended in a clinically meaningful and interpretable way to stress-test the assumption of ignorable censoring. We focus on a 'tipping point' approach, the objective of which is to postulate sensitivity parameters with a clear clinical interpretation and to identify a setting of these parameters unfavorable enough towards the experimental treatment to nullify a conclusion that was favorable to that treatment. Robustness of primary analysis results can then be assessed based on clinical plausibility of the scenario represented by the tipping point. We study several approaches for conducting such analyses based on multiple imputation using parametric, semi-parametric, and non-parametric imputation models and evaluate their operating characteristics via simulation. We argue that these methods are valuable tools for sensitivity analyses of time-to-event data and conclude that the method based on piecewise exponential imputation model of survival has some advantages over other methods studied here. Copyright © 2016 John Wiley & Sons, Ltd.

To adjust or not to adjust for baseline when analyzing repeated binary responses? The case of complete data when treatment comparison at study end is of interest.

The benefits of adjusting for baseline covariates are not as straightforward with repeated binary responses as with continuous response variables. Therefore, in this study, we compared different methods for analyzing repeated binary data through simulations when the outcome at the study endpoint is of interest. Methods compared included chi-square, Fisher's exact test, covariate adjusted/unadjusted logistic regression (Adj.logit/Unadj.logit), covariate adjusted/unadjusted generalized estimating equations (Adj.GEE/Unadj.GEE), covariate adjusted/unadjusted generalized linear mixed model (Adj.GLMM/Unadj.GLMM). All these methods preserved the type I error close to the nominal level. Covariate adjusted methods improved power compared with the unadjusted methods because of the increased treatment effect estimates, especially when the correlation between the baseline and outcome was strong, even though there was an apparent increase in standard errors. Results of the Chi-squared test were identical to those for the unadjusted logistic regression. Fisher's exact test was the most conservative test regarding the type I error rate and also with the lowest power. Without missing data, there was no gain in using a repeated measures approach over a simple logistic regression at the final time point. Analysis of results from five phase III diabetes trials of the same compound was consistent with the simulation findings. Therefore, covariate adjusted analysis is recommended for repeated binary data when the study endpoint is of interest.

Strategies for identifying predictive biomarkers and subgroups with enhanced treatment effect in clinical trials using SIDES.

Several approaches to identification of predictive biomarkers and subgroups of patients with enhanced treatment effect have been proposed in the literature. The SIDES method introduced in Lipkovich et al. (2011) adopts a recursive partitioning algorithm for screening treatment-by-biomarker interactions. This article introduces an improved biomarker discovery/subgroup search method (SIDEScreen). The SIDEScreen method relies on a two-stage procedure that first selects a small number of biomarkers with the highest predictive ability based on an appropriate variable importance score and then identifies subgroups with enhanced treatment effect based on the selected biomarkers. The two-stage approach helps increase the signal-to-noise ratio by screening out noninformative biomarkers. We evaluate operating characteristics of the standard SIDES method and two SIDEScreen procedures based on fixed and adaptive screens. Our main finding is that the adaptive SIDEScreen method is a more flexible biomarker discovery tool than SIDES and it better handles multiplicity in complex subgroup search problems. The methods presented in the article are illustrated using a clinical trial example.