High-dimensional Iterative Causal Forest (hdiCF) for Subgroup Identification Using Health Care Claims Data.

We recently developed a machine-learning subgrouping algorithm, iterative causal forest (iCF), to identify subgroups with heterogeneous treatment effects (HTEs) using predefined covariates. However, such predefined covariates may miss or poorly define important features leading to inaccurate subgrouping. To address such limitations, we developed a new semi-automatic subgrouping algorithm, hdiCF, which adapts methodology from high-dimensional propensity score for feature recognition in claims data. The hdiCF algorithm has 3 steps: 1) high-dimensional feature identification by International Classification of Diseases, Current Procedural Terminology, and Anatomical Therapeutic Chemical codes (in/outpatient diagnoses, procedures, prescriptions) and creation of ordinal variables by frequency of occurrence; 2) propensity score trimming and high-dimensional feature preparation; 3) iCF implementation to identify subgroups. We applied hdiCF in a 20% random sample of fee-for-service Medicare beneficiaries who initiated sodium-glucose cotransporter-2 inhibitors (SGLT2i) or glucagon-like peptide-1 receptor agonists to identify subgroups with HTEs for incidence of hospitalized heart failure. HdiCF findings were consistent with studies suggesting SGLT2i to be more beneficial for patients with pre-existing heart failure or chronic kidney disease. HdiCF is not dependent on prior hypotheses about HTEs and identifies subgroups with markers for potential HTEs in real-world evidence studies where active-comparator, new-user study designs limit the potential for unmeasured confounding.

Modern approaches for evaluating treatment effect heterogeneity from clinical trials and observational data.

In this paper, we review recent advances in statistical methods for the evaluation of the heterogeneity of treatment effects (HTE), including subgroup identification and estimation of individualized treatment regimens, from randomized clinical trials and observational studies. We identify several types of approaches using the features introduced in Lipkovich et al (Stat Med 2017;36: 136-196) that distinguish the recommended principled methods from basic methods for HTE evaluation that typically rely on rules of thumb and general guidelines (the methods are often referred to as common practices). We discuss the advantages and disadvantages of various principled methods as well as common measures for evaluating their performance. We use simulated data and a case study based on a historical clinical trial to illustrate several new approaches to HTE evaluation.

A promising subgroup identification method based on a genetic algorithm for censored survival data.

Modern precision medicine requires drug development to account for patients' heterogeneity, as only a subgroup of the patient population is likely to benefit from the targeted therapy. In this paper, we propose a novel method for subgroup identification based on a genetic algorithm. The proposed method can detect promising subgroups defined by predictive biomarkers in which the treatment effects are much higher than the population average. The main idea is to search for the subgroup with the greatest predictive ability in the entire subgroup space via a genetic algorithm. We design a real-valued representation of subgroups that evolves according to a genetic algorithm and derive an objective function that properly evaluates the predictive ability of the subgroups. Compared with model- or tree-based subgroup identification methods, the distinctive search strategy of this new approach offers an improved capability to explore subgroups defined by multiple predictive biomarkers. By embedding a resampling scheme, the multiplicity and complexity issues inherent in subgroup identification methods can be addressed flexibly. We evaluate the performance of the proposed method in comparison with two other methods using simulation studies and a real-world example. The results show that the proposed method exhibits good properties in terms of multiplicity and complexity control, and the subgroups identified are much more accurate. Although we focus on the implementation of censored survival data, this method could easily be extended for the realization of continuous and categorical endpoints.

Predicting subgroup treatment effects for a new study: Motivations, results and learnings from running a data challenge in a pharmaceutical corporation.

We present the motivation, experience, and learnings from a data challenge conducted at a large pharmaceutical corporation on the topic of subgroup identification. The data challenge aimed at exploring approaches to subgroup identification for future clinical trials. To mimic a realistic setting, participants had access to 4 Phase III clinical trials to derive a subgroup and predict its treatment effect on a future study not accessible to challenge participants. A total of 30 teams registered for the challenge with around 100 participants, primarily from Biostatistics organization. We outline the motivation for running the challenge, the challenge rules, and logistics. Finally, we present the results of the challenge, the participant feedback as well as the learnings. We also present our view on the implications of the results on exploratory analyses related to treatment effect heterogeneity.

A platform for comparing subgroup identification methodologies.

Since the advent of the phrase "subgroup identification," there has been an explosion of methodologies that seek to identify meaningful subgroups of patients with exceptional response in order to further the realization of personalized medicine. However, to perform fair comparison and understand what methods work best under different clinical trials situations, a common platform is needed for comparative effectiveness of these various approaches. In this paper, we describe a comprehensive project that created an extensive platform for evaluating subgroup identification methods as well as a publicly posted challenge that was used to elicit new approaches. We proposed a common data-generating model for creating virtual clinical trial datasets that contain subgroups of exceptional responders encompassing the many dimensions of the problem or null scenarios in which there are no such subgroups. Furthermore, we created a common scoring system for evaluating performance of purported methods for identifying subgroups. This makes it possible to benchmark methodologies in order to understand what methods work best under different clinical trial situations. The findings from this project produced considerable insights and allow us to make recommendations for how the statistical community can better compare and contrast old and new subgroup identification methodologies.

Iterative Causal Forest: A Novel Algorithm for Subgroup Identification.

Precisely and efficiently identifying subgroups with heterogeneous treatment effects (HTEs) in real-world evidence studies remains a challenge. Based on the causal forest (CF) method, we developed an iterative CF (iCF) algorithm to identify HTEs in subgroups defined by important variables. Our method iteratively grows different depths of the CF with important effect modifiers, performs plurality votes to obtain decision trees (subgroup decisions) for a family of CFs with different depths, then finds the cross-validated subgroup decision that best predicts the treatment effect as a final subgroup decision. We simulated 12 different scenarios and showed that the iCF outperformed other machine learning methods for interaction/subgroup identification in the majority of scenarios assessed. Using a 20% random sample of fee-for-service Medicare beneficiaries initiating sodium-glucose cotransporter-2 inhibitors (SGLT2i) or glucagon-like peptide-1 receptor agonists (GLP1RA), we implemented the iCF to identify subgroups with HTEs for hospitalized heart failure. Consistent with previous studies suggesting patients with heart failure benefit more from SGLT2i, iCF successfully identified such a subpopulation with HTEs and additive interactions. The iCF is a promising method for identifying subgroups with HTEs in real-world data where the potential for unmeasured confounding can be limited by study design.

Depth importance in precision medicine (DIPM): a tree- and forest-based method for right-censored survival outcomes.

Many clinical trials have been conducted to compare right-censored survival outcomes between interventions. Such comparisons are typically made on the basis of the entire group receiving one intervention versus the others. In order to identify subgroups for which the preferential treatment may differ from the overall group, we propose the depth importance in precision medicine (DIPM) method for such data within the precision medicine framework. The approach first modifies the split criteria of the traditional classification tree to fit the precision medicine setting. Then, a random forest of trees is constructed at each node. The forest is used to calculate depth variable importance scores for each candidate split variable. The variable with the highest score is identified as the best variable to split the node. The importance score is a flexible and simply constructed measure that makes use of the observation that more important variables tend to be selected closer to the root nodes of trees. The DIPM method is primarily designed for the analysis of clinical data with two treatment groups. We also present the extension to the case of more than two treatment groups. We use simulation studies to demonstrate the accuracy of our method and provide the results of applications to two real-world data sets. In the case of one data set, the DIPM method outperforms an existing method, and a primary motivation of this article is the ability of the DIPM method to address the shortcomings of this existing method. Altogether, the DIPM method yields promising results that demonstrate its capacity to guide personalized treatment decisions in cases with right-censored survival outcomes.

Estimation and visualization of heterogeneous treatment effects for multiple outcomes.

We consider two-arm comparison in clinical trials. The objective is to identify a population with characteristics that make the treatment effective. Such a population is called a subgroup. This identification can be made by estimating the treatment effect and identifying the interactions between treatments and covariates. For a single outcome, there are several ways available to identify the subgroups. There are also multiple outcomes, but they are difficult to interpret and cannot be applied to outcomes other than continuous values. In this paper, we thus propose a new method that allows for a straightforward interpretation of subgroups and deals with both continuous and binary outcomes. The proposed method introduces latent variables and adds Lasso sparsity constraints to the estimated loadings to facilitate the interpretation of the relationship between outcomes and covariates. The interpretation of the subgroups is made by visualizing treatment effects and latent variables. Since we are performing sparse estimation, we can interpret the covariates related to the treatment effects and subgroups. Finally, simulation and real data examples demonstrate the effectiveness of the proposed method.

IBIS: identify biomarker-based subgroups with a Bayesian enrichment design for targeted combination therapy.

BACKGROUND: Combination therapies directed at multiple targets have potentially improved treatment effects for cancer patients. Compared to monotherapy, targeted combination therapy leads to an increasing number of subgroups and complicated biomarker-based efficacy profiles, making it more difficult for efficacy evaluation in clinical trials. Therefore, it is necessary to develop innovative clinical trial designs to explore the efficacy of targeted combination therapy in different subgroups and identify patients who are more likely to benefit from the investigational combination therapy. METHODS: We propose a statistical tool called 'IBIS' to Identify BIomarker-based Subgroups and apply it to the enrichment design framework. The IBIS contains three main elements: subgroup division, efficacy evaluation and subgroup identification. We first enumerate all possible subgroup divisions based on biomarker levels. Then, Jensen-Shannon divergence is used to distinguish high-efficacy and low-efficacy subgroups, and Bayesian hierarchical model (BHM) is employed to borrow information within these two subsets for efficacy evaluation. Regarding subgroup identification, a hypothesis testing framework based on Bayes factors is constructed. This framework also plays a key role in go/no-go decisions and enriching specific population. Simulation studies are conducted to evaluate the proposed method. RESULTS: The accuracy and precision of IBIS could reach a desired level in terms of estimation performance. In regard to subgroup identification and population enrichment, the proposed IBIS has superior and robust characteristics compared with traditional methods. An example of how to obtain design parameters for an adaptive enrichment design under the IBIS framework is also provided. CONCLUSIONS: IBIS has the potential to be a useful tool for biomarker-based subgroup identification and population enrichment in clinical trials of targeted combination therapy.

Overview of modern approaches for identifying and evaluating heterogeneous treatment effects from clinical data.

There has been much interest in the evaluation of heterogeneous treatment effects (HTE) and multiple statistical methods have emerged under the heading of personalized/precision medicine combining ideas from hypothesis testing, causal inference, and machine learning over the past 10-15 years. We discuss new ideas and approaches for evaluating HTE in randomized clinical trials and observational studies using the features introduced earlier by Lipkovich, Dmitrienko, and D'Agostino that distinguish principled methods from simplistic approaches to data-driven subgroup identification and estimating individual treatment effects and use a case study to illustrate these approaches. We identified and provided a high-level overview of several classes of modern statistical approaches for personalized/precision medicine, elucidated the underlying principles and challenges, and compared findings for a case study across different methods. Different approaches to evaluating HTEs may produce (and actually produced) highly disparate results when applied to a specific data set. Evaluating HTE with machine learning methods presents special challenges since most of machine learning algorithms are optimized for prediction rather than for estimating causal effects. An additional challenge is in that the output of machine learning methods is typically a "black box" that needs to be transformed into interpretable personalized solutions in order to gain acceptance and usability.

Considerations for identifying the "right" subgroup in adaptive enrichment trials.

Adaptive Enrichment Trials aim to make efficient use of data in a pivotal trial of a new targeted therapy to both (a) more precisely identify who benefits from that therapy and (b) improve the likelihood of successfully concluding that the drug is effective, while controlling the probability of false positives. There are a number of frameworks for conducting such a trial and decisions that must be made regarding how to identify that target subgroup. Among those decisions, one must choose how aggressively to restrict enrollment criteria based on the accumulating evidence in the trial. In this article, we empirically evaluate the impact of aggressive versus conservative enrollment restrictions on the power of the trial to detect an effect of treatment. We identify that, in some cases, a more aggressive strategy can substantially improve power. This additionally raises an important question regarding label indication: To what degree do we need a formal test of the hypothesis of no treatment effect in the exact population implied by the label indication? We discuss this question and evaluate how our answer for adaptive enrichment trials may relate to the answer implied by current practice for broad eligibility trials.

Covariate-adjusted value-guided subgroup identification via boosting.

It is widely recognized that treatment effects could differ across subgroups of patients. Subgroup analysis, which assesses such heterogeneity, provides valuable information in developing personalized therapies. There has been extensive research developing novel statistical methods for subgroup identification. The recent contribution is a value-guided subgroup identification method that directly maximizes treatment benefit at the subgroup level for survival outcome, rather than relying on individual treatment effect estimation. In this paper, we first completed this framework by illustrating its application to continuous and binary outcomes. More importantly, we extended the original framework to account for the prognostic effects and named this new method Covariate-Adjusted Value-guided subgroup identification via boosting (CAVboost). The original method directly used the outcome to formulate the value function for subgroup identification. Since the outcome can further be decomposed as prognostic effects and treatment effects, specifying the prognostic effects as the covariates of a model for the outcome can single out the treatment effects and improve the power to detect them across subgroups. Our proposed CAVboost was based on this key idea. It used a covariate-adjusted treatment effect estimator, instead of the outcome itself, to formulate the value function for subgroup identification. CAVboost estimates the treatment effect by using covariates to account for the prognostic effects, which mimics the idea of using covariates in an ANCOVA estimator. We showed that CAVboost could effectively improve the subgroup identification capability for both continuous and binary outcomes.

Identification of Homogeneous Subgroups from Resting-State fMRI Data.

The identification of homogeneous subgroups of patients with psychiatric disorders can play an important role in achieving personalized medicine and is essential to provide insights for understanding neuropsychological mechanisms of various mental disorders. The functional connectivity profiles obtained from functional magnetic resonance imaging (fMRI) data have been shown to be unique to each individual, similar to fingerprints; however, their use in characterizing psychiatric disorders in a clinically useful way is still being studied. In this work, we propose a framework that makes use of functional activity maps for subgroup identification using the Gershgorin disc theorem. The proposed pipeline is designed to analyze a large-scale multi-subject fMRI dataset with a fully data-driven method, a new constrained independent component analysis algorithm based on entropy bound minimization (c-EBM), followed by an eigenspectrum analysis approach. A set of resting-state network (RSN) templates is generated from an independent dataset and used as constraints for c-EBM. The constraints present a foundation for subgroup identification by establishing a connection across the subjects and aligning subject-wise separate ICA analyses. The proposed pipeline was applied to a dataset comprising 464 psychiatric patients and discovered meaningful subgroups. Subjects within the identified subgroups share similar activation patterns in certain brain areas. The identified subgroups show significant group differences in multiple meaningful brain areas including dorsolateral prefrontal cortex and anterior cingulate cortex. Three sets of cognitive test scores were used to verify the identified subgroups, and most of them showed significant differences across subgroups, which provides further confirmation of the identified subgroups. In summary, this work represents an important step forward in using neuroimaging data to characterize mental disorders.

Machine Learning Models for the Identification of Prognostic and Predictive Cancer Biomarkers: A Systematic Review.

The identification of biomarkers plays a crucial role in personalized medicine, both in the clinical and research settings. However, the contrast between predictive and prognostic biomarkers can be challenging due to the overlap between the two. A prognostic biomarker predicts the future outcome of cancer, regardless of treatment, and a predictive biomarker predicts the effectiveness of a therapeutic intervention. Misclassifying a prognostic biomarker as predictive (or vice versa) can have serious financial and personal consequences for patients. To address this issue, various statistical and machine learning approaches have been developed. The aim of this study is to present an in-depth analysis of recent advancements, trends, challenges, and future prospects in biomarker identification. A systematic search was conducted using PubMed to identify relevant studies published between 2017 and 2023. The selected studies were analyzed to better understand the concept of biomarker identification, evaluate machine learning methods, assess the level of research activity, and highlight the application of these methods in cancer research and treatment. Furthermore, existing obstacles and concerns are discussed to identify prospective research areas. We believe that this review will serve as a valuable resource for researchers, providing insights into the methods and approaches used in biomarker discovery and identifying future research opportunities.

Robust structured heterogeneity analysis approach for high-dimensional data.

Revealing relationships between genes and disease phenotypes is a critical problem in biomedical studies. This problem has been challenged by the heterogeneity of diseases. Patients of a perceived same disease may form multiple subgroups, and different subgroups have distinct sets of important genes. It is hence imperative to discover the latent subgroups and reveal the subgroup-specific important genes. Some heterogeneity analysis methods have been proposed in the recent literature. Despite considerable successes, most of the existing studies are still limited as they cannot accommodate data contamination and ignore the interconnections among genes. Aiming at these shortages, we develop a robust structured heterogeneity analysis approach to identify subgroups, select important genes as well as estimate their effects on the phenotype of interest. Possible data contamination is accommodated by employing the Huber loss function. A sparse overlapping group lasso penalty is imposed to conduct regularization estimation and gene identification, while taking into account the possibly overlapping cluster structure of genes. This approach takes an iterative strategy in the similar spirit of K-means clustering. Simulations demonstrate that the proposed approach outperforms alternatives in revealing the heterogeneity and selecting important genes for each subgroup. The analysis of Cancer Cell Line Encyclopedia data leads to biologically meaningful findings with improved prediction and grouping stability.

CAPITAL: Optimal subgroup identification via constrained policy tree search.

Personalized medicine, a paradigm of medicine tailored to a patient's characteristics, is an increasingly attractive field in health care. An important goal of personalized medicine is to identify a subgroup of patients, based on baseline covariates, that benefits more from the targeted treatment than other comparative treatments. Most of the current subgroup identification methods only focus on obtaining a subgroup with an enhanced treatment effect without paying attention to subgroup size. Yet, a clinically meaningful subgroup learning approach should identify the maximum number of patients who can benefit from the better treatment. In this article, we present an optimal subgroup selection rule (SSR) that maximizes the number of selected patients, and in the meantime, achieves the pre-specified clinically meaningful mean outcome, such as the average treatment effect. We derive two equivalent theoretical forms of the optimal SSR based on the contrast function that describes the treatment-covariates interaction in the outcome. We further propose a constrained policy tree search algorithm (CAPITAL) to find the optimal SSR within the interpretable decision tree class. The proposed method is flexible to handle multiple constraints that penalize the inclusion of patients with negative treatment effects, and to address time to event data using the restricted mean survival time as the clinically interesting mean outcome. Extensive simulations, comparison studies, and real data applications are conducted to demonstrate the validity and utility of our method.

Multi-threshold proportional hazards model and subgroup identification.

We propose a novel two-stage procedure for change point detection and parameter estimation in a multi-threshold proportional hazards model. In the first stage, we estimate the number of thresholds by formulating the threshold detection problem as a variable selection problem and applying the penalized partial likelihood approach. In the second stage, the change point locations are refined by a grid search and the standard inference for segment regression can then follow. The proposed model and estimation procedure could lend support to subgroup identification and personalized treatment recommendation in medical research. We establish the consistency of the threshold estimators and regression coefficient estimators under technical conditions. The finite sample performance of the method is demonstrated via simulation studies and two cancer data examples.

A permutation procedure to detect heterogeneous treatment effects in randomized clinical trials while controlling the type I error rate.

BACKGROUND/AIMS: Secondary analyses of randomized clinical trials often seek to identify subgroups with differential treatment effects. These discoveries can help guide individual treatment decisions based on patient characteristics and identify populations for which additional treatments are needed. Traditional analyses require researchers to pre-specify potential subgroups to reduce the risk of reporting spurious results. There is a need for methods that can detect such subgroups without a priori specification while allowing researchers to control the probability of falsely detecting heterogeneous subgroups when treatment effects are uniform across the study population. METHODS: We propose a permutation procedure for tuning parameter selection that allows for type I error control when testing for heterogeneous treatment effects framed within the Virtual Twins procedure for subgroup identification. We verify that the type I error rate can be controlled at the nominal rate and investigate the power for detecting heterogeneous effects when present through extensive simulation studies. We apply our method to a secondary analysis of data from a randomized trial of very low nicotine content cigarettes. RESULTS: In the absence of type I error control, the observed type I error rate for Virtual Twins was between 99% and 100%. In contrast, models tuned via the proposed permutation were able to control the type I error rate and detect heterogeneous effects when present. An application of our approach to a recently completed trial of very low nicotine content cigarettes identified several variables with potentially heterogeneous treatment effects. CONCLUSIONS: The proposed permutation procedure allows researchers to engage in secondary analyses of clinical trials for treatment effect heterogeneity while maintaining the type I error rate without pre-specifying subgroups.

A two-stage adaptive clinical trial design with data-driven subgroup identification at interim analysis.

In this paper, we consider randomized controlled clinical trials comparing two treatments in efficacy assessment using a time to event outcome. We assume a relatively small number of candidate biomarkers available in the beginning of the trial, which may help define an efficacy subgroup which shows differential treatment effect. The efficacy subgroup is to be defined by one or two biomarkers and cut-offs that are unknown to the investigator and must be learned from the data. We propose a two-stage adaptive design with a pre-planned interim analysis and a final analysis. At the interim, several subgroup-finding algorithms are evaluated to search for a subgroup with enhanced survival for treated versus placebo. Conditional powers computed based on the subgroup and the overall population are used to make decision at the interim to terminate the study for futility, continue the study as planned, or conduct sample size recalculation for the subgroup or the overall population. At the final analysis, combination tests together with closed testing procedures are used to determine efficacy in the subgroup or the overall population. We conducted simulation studies to compare our proposed procedures with several subgroup-identification methods in terms of a novel utility function and several other measures. This research demonstrated the benefit of incorporating data-driven subgroup selection into adaptive clinical trial designs.

Comparing algorithms for characterizing treatment effect heterogeneity in randomized trials.

The identification and estimation of heterogeneous treatment effects in biomedical clinical trials are challenging, because trials are typically planned to assess the treatment effect in the overall trial population. Nevertheless, the identification of how the treatment effect may vary across subgroups is of major importance for drug development. In this work, we review some existing simulation work and perform a simulation study to evaluate recent methods for identifying and estimating the heterogeneous treatments effects using various metrics and scenarios relevant for drug development. Our focus is not only on a comparison of the methods in general, but on how well these methods perform in simulation scenarios that reflect real clinical trials. We provide the R package benchtm that can be used to simulate synthetic biomarker distributions based on real clinical trial data and to create interpretable scenarios to benchmark methods for identification and estimation of treatment effect heterogeneity.