Incidence of type 2 diabetes, cardiovascular disease and chronic kidney disease in patients with multiple sclerosis initiating disease-modifying therapies: Retrospective cohort study using a frequentist model averaging statistical framework.

Researchers are increasingly using insights derived from large-scale, electronic healthcare data to inform drug development and provide human validation of novel treatment pathways and aid in drug repurposing/repositioning. The objective of this study was to determine whether treatment of patients with multiple sclerosis with dimethyl fumarate, an activator of the nuclear factor erythroid 2-related factor 2 (Nrf2) pathway, results in a change in incidence of type 2 diabetes and its complications. This retrospective cohort study used administrative claims data to derive four cohorts of adults with multiple sclerosis initiating dimethyl fumarate, teriflunomide, glatiramer acetate or fingolimod between January 2013 and December 2018. A causal inference frequentist model averaging framework based on machine learning was used to compare the time to first occurrence of a composite endpoint of type 2 diabetes, cardiovascular disease or chronic kidney disease, as well as each individual outcome, across the four treatment cohorts. There was a statistically significantly lower risk of incidence for dimethyl fumarate versus teriflunomide for the composite endpoint (restricted hazard ratio [95% confidence interval] 0.70 [0.55, 0.90]) and type 2 diabetes (0.65 [0.49, 0.98]), myocardial infarction (0.59 [0.35, 0.97]) and chronic kidney disease (0.52 [0.28, 0.86]). No differences for other individual outcomes or for dimethyl fumarate versus the other two cohorts were observed. This study effectively demonstrated the use of an innovative statistical methodology to test a clinical hypothesis using real-world data to perform early target validation for drug discovery. Although there was a trend among patients treated with dimethyl fumarate towards a decreased incidence of type 2 diabetes, cardiovascular disease and chronic kidney disease relative to other disease-modifying therapies-which was statistically significant for the comparison with teriflunomide-this study did not definitively support the hypothesis that Nrf2 activation provided additional metabolic disease benefit in patients with multiple sclerosis.

EXIST: EXamining rIsk of excesS adiposiTy-Machine learning to predict obesity-related complications.

Background: Obesity is associated with an increased risk of multiple conditions, ranging from heart disease to cancer. However, there are few predictive models for these outcomes that have been developed specifically for people with overweight/obesity. Objective: To develop predictive models for obesity-related complications in patients with overweight and obesity. Methods: Electronic health record data of adults with body mass index 25-80 kg/m2 treated in primary care practices between 2000 and 2019 were utilized to develop and evaluate predictive models for nine long-term clinical outcomes using a) Lasso-Cox models and b) a machine-learning method random survival forests (RSF). Models were trained on a training dataset and evaluated on a test dataset over 100 replicates. Parsimonious models of <10 variables were also developed using Lasso-Cox. Results: Over a median follow-up of 5.6 years, study outcome incidence in the cohort of 433,272 patients ranged from 1.8% for knee replacement to 11.7% for atherosclerotic cardiovascular disease. Harrell C-index averaged over replicates ranged from 0.702 for liver outcomes to 0.896 for death for RSF, and from 0.694 for liver outcomes to 0.891 for death for Lasso-Cox. The Harrell C-index for parsimonious models ranged from 0.675 for liver outcomes to 0.850 for knee replacement. Conclusions: Predictive modeling can identify patients at high risk of obesity-related complications. Interpretable Cox models achieve results close to those of machine learning methods and could be helpful for population health management and clinical treatment decisions.

Participant Engagement and Adherence to Providing Smartwatch and Patient-Reported Outcome Data: Digital Tracking of Rheumatoid Arthritis Longitudinally (DIGITAL) Real-World Study.

BACKGROUND: Digital health studies using electronic patient-reported outcomes (ePROs) and wearables bring new challenges, including the need for participants to consistently provide trial data. OBJECTIVE: This study aims to characterize the engagement, protocol adherence, and data completeness among participants with rheumatoid arthritis enrolled in the Digital Tracking of Arthritis Longitudinally (DIGITAL) study. METHODS: Participants were invited to participate in this app-based study, which included a 14-day run-in and an 84-day main study. In the run-in period, data were collected via the ArthritisPower mobile app to increase app familiarity and identify the individuals who were motivated to participate. Successful completers of the run-in period were mailed a wearable smartwatch, and automated and manual prompts were sent to participants, reminding them to complete app input or regularly wear and synchronize devices, respectively, during the main study. Study coordinators monitored participant data and contacted participants via email, SMS text messaging, and phone to resolve adherence issues per a priori rules, in which consecutive spans of missing data triggered participant contact. Adherence to data collection during the main study period was defined as providing requested data for >70% of 84 days (daily ePRO, ≥80% daily smartwatch data) or at least 9 of 12 weeks (weekly ePRO). RESULTS: Of the 470 participants expressing initial interest, 278 (59.1%) completed the run-in period and qualified for the main study. Over the 12-week main study period, 87.4% (243/278) of participants met the definition of adherence to protocol-specified data collection for weekly ePRO, and 57.2% (159/278) did so for daily ePRO. For smartwatch data, 81.7% (227/278) of the participants adhered to the protocol-specified data collection. In total, 52.9% (147/278) of the participants met composite adherence. CONCLUSIONS: Compared with other digital health rheumatoid arthritis studies, a short run-in period appears useful for identifying participants likely to engage in a study that collects data via a mobile app and wearables and gives participants time to acclimate to study requirements. Automated or manual prompts (ie, "It's time to sync your smartwatch") may be necessary to optimize adherence. Adherence varies by data collection type (eg, ePRO vs smartwatch data). INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): RR2-10.2196/14665.

Overview of modern approaches for identifying and evaluating heterogeneous treatment effects from clinical data.

There has been much interest in the evaluation of heterogeneous treatment effects (HTE) and multiple statistical methods have emerged under the heading of personalized/precision medicine combining ideas from hypothesis testing, causal inference, and machine learning over the past 10-15 years. We discuss new ideas and approaches for evaluating HTE in randomized clinical trials and observational studies using the features introduced earlier by Lipkovich, Dmitrienko, and D'Agostino that distinguish principled methods from simplistic approaches to data-driven subgroup identification and estimating individual treatment effects and use a case study to illustrate these approaches. We identified and provided a high-level overview of several classes of modern statistical approaches for personalized/precision medicine, elucidated the underlying principles and challenges, and compared findings for a case study across different methods. Different approaches to evaluating HTEs may produce (and actually produced) highly disparate results when applied to a specific data set. Evaluating HTE with machine learning methods presents special challenges since most of machine learning algorithms are optimized for prediction rather than for estimating causal effects. An additional challenge is in that the output of machine learning methods is typically a "black box" that needs to be transformed into interpretable personalized solutions in order to gain acceptance and usability.

Activation, physical activity, and outcomes among individuals with T2D.

OBJECTIVES: To explore the associations among activation, physical activity, hemoglobin A1c (HbA1c), and healthy days in older adults with type 2 diabetes (T2D) who participated in wellness programs. STUDY DESIGN: Observational, longitudinal cohort study utilizing survey, claims, and wellness program data. METHODS: From January to May 2018, individuals enrolled in a commercial or Medicare Advantage and prescription drug plan with T2D (aged 55-89 years) and SilverSneakers or step count data were eligible. Three waves of surveys were mailed (n = 5000) to collect information on activation (Consumer Health Activation Index; Influence, Motivation, and Patient Activation for Diabetes) and health-related quality of life (Healthy Days). Generalized linear models and predictive models evaluated the associations of unhealthy days and HbA1c with physical activity and activation factors. Additional models tested the relationship between physical activity and future acute care visits, accounting for potential confounders via inverse probability of treatment weighting. RESULTS: Respondents to all 3 waves (n = 1147) had higher comorbidity indices but lower HbA1c than individuals with T2D without physical activity data (P < .0001). Individuals with moderate and high activation levels had 67.4% to 74.0% and 71.6% to 85.6% fewer unhealthy days, respectively, than those with lower activation (P < .01). Individuals with high (> 8000/day) step counts at baseline were predicted to have 2.04 fewer unhealthy days/month at follow-up (P < .05) and 0.19% (P < .02) lower HbA1c units, respectively, compared with those with less than 4000 steps per day. High SilverSneakers activity (> 2 activities per week) reduced subsequent acute care visits by 49%. CONCLUSIONS: Increasing patient activation levels encourages physical activity, which can help improve glycemic control and health-related quality of life, especially among older adults.

Using principal stratification in analysis of clinical trials.

The ICH E9(R1) addendum (2019) proposed principal stratification (PS) as one of five strategies for dealing with intercurrent events. Therefore, understanding the strengths, limitations, and assumptions of PS is important for the broad community of clinical trialists. Many approaches have been developed under the general framework of PS in different areas of research, including experimental and observational studies. These diverse applications have utilized a diverse set of tools and assumptions. Thus, need exists to present these approaches in a unifying manner. The goal of this tutorial is threefold. First, we provide a coherent and unifying description of PS. Second, we emphasize that estimation of effects within PS relies on strong assumptions and we thoroughly examine the consequences of these assumptions to understand in which situations certain assumptions are reasonable. Finally, we provide an overview of a variety of key methods for PS analysis and use a real clinical trial example to illustrate them. Examples of code for implementation of some of these approaches are given in Supplemental Materials.

A two-stage adaptive clinical trial design with data-driven subgroup identification at interim analysis.

In this paper, we consider randomized controlled clinical trials comparing two treatments in efficacy assessment using a time to event outcome. We assume a relatively small number of candidate biomarkers available in the beginning of the trial, which may help define an efficacy subgroup which shows differential treatment effect. The efficacy subgroup is to be defined by one or two biomarkers and cut-offs that are unknown to the investigator and must be learned from the data. We propose a two-stage adaptive design with a pre-planned interim analysis and a final analysis. At the interim, several subgroup-finding algorithms are evaluated to search for a subgroup with enhanced survival for treated versus placebo. Conditional powers computed based on the subgroup and the overall population are used to make decision at the interim to terminate the study for futility, continue the study as planned, or conduct sample size recalculation for the subgroup or the overall population. At the final analysis, combination tests together with closed testing procedures are used to determine efficacy in the subgroup or the overall population. We conducted simulation studies to compare our proposed procedures with several subgroup-identification methods in terms of a novel utility function and several other measures. This research demonstrated the benefit of incorporating data-driven subgroup selection into adaptive clinical trial designs.

Evaluating bias control strategies in observational studies using frequentist model averaging.

Estimating a treatment effect from observational data requires modeling treatment and outcome subject to uncertainty/misspecification. A previous research has shown that it is not possible to find a uniformly best strategy. In this article we propose a novel Frequentist Model Averaging (FMA) framework encompassing any estimation strategy and accounting for model uncertainty by computing a cross-validated estimate of Mean Squared Prediction Error (MSPE). We present a simulation study with data mimicking an observational database. Model averaging over 15+ strategies was compared with individual strategies as well as the best strategy selected by minimum MSPE. FMA showed robust performance (Bias, Mean Squared Error (MSE), and Confidence Interval (CI) coverage). Other strategies, such as linear regression, did well in simple scenarios but were inferior to the FMA in a scenario with complex confounding.

Practical recommendations on double score matching for estimating causal effects.

Unlike in randomized clinical trials (RCTs), confounding control is critical for estimating the causal effects from observational studies due to the lack of treatment randomization. Under the unconfoundedness assumption, matching methods are popular because they can be used to emulate an RCT that is hidden in the observational study. To ensure the key assumption hold, the effort is often made to collect a large number of possible confounders, rendering dimension reduction imperative in matching. Three matching schemes based on the propensity score (PSM), prognostic score (PGM), and double score (DSM, ie, the collection of the first two scores) have been proposed in the literature. However, a comprehensive comparison is lacking among the three matching schemes and has not made inroads into the best practices including variable selection, choice of caliper, and replacement. In this article, we explore the statistical and numerical properties of PSM, PGM, and DSM via extensive simulations. Our study supports that DSM performs favorably with, if not better than, the two single score matching in terms of bias and variance. In particular, DSM is doubly robust in the sense that the matching estimator is consistent requiring either the propensity score model or the prognostic score model is correctly specified. Variable selection on the propensity score model and matching with replacement is suggested for DSM, and we illustrate the recommendations with comprehensive simulation studies. An R package is available at https://github.com/Yunshu7/dsmatch.

Data-Driven Subgroup Identification in Confirmatory Clinical Trials.

Data-driven subgroup analysis plays an important role in clinical trials. This paper focuses on practical considerations in post-hoc subgroup investigations in the context of confirmatory clinical trials. The analysis is aimed at assessing the heterogeneity of treatment effects across the trial population and identifying patient subgroups with enhanced treatment benefit. The subgroups are defined using baseline patient characteristics, including demographic and clinical factors. Much progress has been made in the development of reliable statistical methods for subgroup investigation, including methods based on global models and recursive partitioning. The paper provides a review of principled approaches to data-driven subgroup identification and illustrates subgroup analysis strategies using a family of recursive partitioning methods known as the SIDES (subgroup identification based on differential effect search) methods. These methods are applied to a Phase III trial in patients with metastatic colorectal cancer. The paper discusses key considerations in subgroup exploration, including the role of covariate adjustment, subgroup analysis at early decision points and interpretation of subgroup search results in trials with a positive overall effect.

Evaluating the relationship between clinical and demographic characteristics of insulin-using people with diabetes and their health outcomes: a cluster analysis application.

BACKGROUND: The aim of this study was to determine how clusters or subgroups of insulin-treated people with diabetes, based upon healthcare resource utilization, select social demographic and clinical characteristics, and diabetes management parameters, are related to health outcomes including acute care visits and hospital admissions. METHODS: This was a non-experimental, retrospective cluster analysis. We utilized Aetna administrative claims data to identify insulin-using people with diabetes with service dates from 01 January 2015 to 30 June 2018. The study included adults over the age of 18 years who had a diagnosis of type 1 (T1DM) or type 2 diabetes mellitus (T2DM) on insulin therapy and had Aetna medical and pharmacy coverage for at least 18 months (6 months prior and 12 months after their index date, defined as either their first insulin prescription fill date or their earliest date allowing for 6 months' prior coverage). We used K-means clustering methods to identify relevant subgroups of people with diabetes based on 13 primary outcome variables. RESULTS: A total of 100,650 insulin-using people with diabetes were identified in the Aetna administrative claims database and met study criteria, including 11,826 (11.7%) with T1DM and 88,824 (88.3%) with T2DM. Of these 79,053 (78.5%) people were existing insulin users. Seven distinct clusters were identified with different characteristics and potential risks of diabetes complications. Overall, clusters were significantly associated with differences in healthcare utilization (emergency room visits, inpatient admissions, and total inpatient days) after multivariable adjustment. CONCLUSIONS: This analysis of healthcare claims data using clustering methodologies identified meaningful subgroups of patients with diabetes using insulin. The subgroups differed in comorbidity burden, healthcare utilization, and demographic factors which could be used to identify higher risk patients and/or guide the management and treatment of diabetes.

Implementation of ICH E9 (R1): A Few Points Learned During the COVID-19 Pandemic.

The current COVID-19 pandemic poses numerous challenges for ongoing clinical trials and provides a stress-testing environment for the existing principles and practice of estimands in clinical trials. The pandemic may increase the rate of intercurrent events (ICEs) and missing values, spurring a great deal of discussion on amending protocols and statistical analysis plans to address these issues. In this article, we revisit recent research on estimands and handling of missing values, especially the ICH E9 (R1) Addendum on Estimands and Sensitivity Analysis in Clinical Trials. Based on an in-depth discussion of the strategies for handling ICEs using a causal inference framework, we suggest some improvements in applying the estimand and estimation framework in ICH E9 (R1). Specifically, we discuss a mix of strategies allowing us to handle ICEs differentially based on reasons for ICEs. We also suggest ICEs should be handled primarily by hypothetical strategies and provide examples of different hypothetical strategies for different types of ICEs as well as a road map for estimation and sensitivity analyses. We conclude that the proposed framework helps streamline translating clinical objectives into targets of statistical inference and automatically resolves many issues with defining estimands and choosing estimation procedures arising from events such as the pandemic.

Predicting optimal treatment regimens for patients with HR+/HER2- breast cancer using machine learning based on electronic health records.

Aim: To predict optimal treatments maximizing overall survival (OS) and time to treatment discontinuation (TTD) for patients with metastatic breast cancer (MBC) using machine learning methods on electronic health records. Patients/methods: Adult females with HR+/HER2- MBC on first- or second-line systemic therapy were eligible. Random survival forest (RSF) models were used to predict optimal regimen classes for individual patients and each line of therapy based on baseline characteristics. Results: RSF models suggested greater use of CDK4 & 6 inhibitor-based therapies may maximize OS and TTD. RSF-predicted optimal treatments demonstrated longer OS and TTD compared with nonoptimal treatments across line of therapy (hazard ratios = 0.44∼0.79). Conclusion: RSF may help inform optimal treatment choices and improve outcomes for patients with HR+/HER2- MBC.

Choosing Estimands in Clinical Trials: Putting the ICH E9(R1) Into Practice.

The National Research Council (NRC) Expert Panel Report on Prevention and Treatment of Missing Data in Clinical Trials highlighted the need for clearly defining objectives and estimands. That report sparked considerable discussion and literature on estimands and how to choose them. Importantly, consideration moved beyond missing data to include all postrandomization events that have implications for estimating quantities of interest (intercurrent events, aka ICEs). The ICH E9(R1) draft addendum builds on that research to outline key principles in choosing estimands for clinical trials, primarily with focus on confirmatory trials. This paper provides additional insights, perspectives, details, and examples to help put ICH E9(R1) into practice. Specific areas of focus include how the perspectives of different stakeholders influence the choice of estimands; the role of randomization and the intention-to-treat principle; defining the causal effects of a clearly defined treatment regimen, along with the implications this has for trial design and the generalizability of conclusions; detailed discussion of strategies for handling ICEs along with their implications and assumptions; estimands for safety objectives, time-to-event endpoints, early-phase and one-arm trials, and quality of life endpoints; and realistic examples of the thought process involved in defining estimands in specific clinical contexts.

Defining Efficacy Estimands in Clinical Trials: Examples Illustrating ICH E9(R1) Guidelines.

This paper provides examples of defining estimands in real-world scenarios following ICH E9(R1) guidelines. Detailed discussions on choosing the estimands and estimators can be found in our companion papers. Three scenarios of increasing complexity are illustrated. The first example is a proof-of-concept trial in major depressive disorder where the estimand is chosen to support the sponsor decision on whether to continue development. The second and third examples are confirmatory trials in severe asthma and rheumatoid arthritis respectively. We discuss the intercurrent events expected during each trial and how they can be handled so as to be consistent with the study objectives. The estimands discussed in these examples are not the only acceptable choices for their respective scenarios. The intent is to illustrate the key concepts rather than focus on specific choices. Emphasis is placed on following a study development process where estimands link the study objectives with data collection and analysis in a coherent manner, thereby avoiding disconnect between objectives, estimands, and analyses.

The Impact of Major Events on Ongoing Noninferiority Trials, With Application to COVID-19.

Abstract-The COVID-19 pandemic has impacted ongoing clinical trials. We consider particular impacts on noninferiority clinical trials, which aim to show that an investigational treatment is not markedly worse than an existing active control with known benefit. Because interpretation of noninferiority trials requires cross-trial validation involving untestable assumptions, it is vital that they be run to very high standards. The COVID-19 pandemic has introduced an unexpected impact on clinical trials, with subjects possibly missing treatment or assessments due to unforeseen intercurrent events. The resulting data must be carefully considered to ensure proper statistical inference. Missing data can often, but not always, be considered missing completely at random (MCAR). We discuss ways to ensure validity of the analyses through study conduct and data analysis, with focus on the hypothetical strategy for constructing estimands. We assess various analytic strategies of analyzing longitudinal binary data with dropouts where outcomes may be MCAR or missing at random (MAR). Simulations show that certain multiple imputation strategies control the Type I error rate and provide additional power over analysis of observed data when data are MCAR or MAR, with weaker assumptions about the missing data mechanism.

Impact of informative censoring on the treatment effect estimate of disability worsening in multiple sclerosis clinical trials.

OBJECTIVE: To examine the impact of missing data when evaluating the confirmed disability worsening (CDW) endpoint in multiple sclerosis clinical trials and explore analytical methods for handling censored participants (those with missing confirmation data). METHODS: CDW risk factors were assessed among participants with an initial disability worsening (≥ 1.0-point increase in Expanded Disability Status Scale [EDSS] score from a baseline score of ≥ 1.0; ≥ 1.5-point increase from a baseline of 0) using data from the DECIDE trial of daclizumab beta. A post-hoc simulation study was performed to evaluate three strategies for imputing confirmation status in censored participants: assume all were confirmed; assume none were confirmed (standard analytical approach); or use an observed rate multiple imputation (ORMI) approach based on treatment group and similar participant risk factors. Simulation study results were used to evaluate pre-specified analyses in DECIDE. RESULTS: In DECIDE, larger change from baseline to initial disability worsening in EDSS score (p = 0.0003), higher baseline EDSS score (p = 0.0013), age (p = 0.004), and preceding relapse (p < 0.0001) were associated with 12-week CDW. In the simulation study, relative to the full dataset (no missing data), the strategy of assuming no censored participants were confirmed underestimated the treatment effect, and the strategy of assuming all censored participants were confirmed overestimated the treatment effect (hazard ratio 0.749 and 0.713 vs 0.733). ORMI correctly estimated treatment effect and increased study power by ~5-10% compared with the standard analytical approach. CONCLUSION: The ORMI approach based on CDW risk factors minimizes bias and is expected to provide the most accurate treatment effect estimate for the CDW endpoint.

Application of structured statistical analyses to identify a biomarker predictive of enhanced tralokinumab efficacy in phase III clinical trials for severe, uncontrolled asthma.

BACKGROUND: Tralokinumab is an anti-interleukin (IL)-13 monoclonal antibody investigated for the treatment of severe, uncontrolled asthma in two Phase III clinical trials, STRATOS 1 and 2. The STRATOS 1 biomarker analysis plan was developed to identify biomarker(s) indicative of IL-13 activation likely to predict tralokinumab efficacy and define a population in which there was an enhanced treatment effect; this defined population was then tested in STRATOS 2. METHODS: The biomarkers considered were blood eosinophil counts, fractional exhaled nitric oxide (FeNO), serum dipeptidyl peptidase-4, serum periostin and total serum immunoglobulin E. Tralokinumab efficacy was measured as the reduction in annualised asthma exacerbation rate (AAER) compared with placebo (primary endpoint measure of STRATOS 1 and 2). The biomarker analysis plan included negative binomial and generalised additive models, and the Subgroup Identification based on Differential Effect Search (SIDES) algorithm, supported by robustness and sensitivity checks. Effects on the key secondary endpoints of STRATOS 1 and 2, which included changes from baseline in standard measures of asthma outcomes, were also investigated. Prior to the STRATOS 1 read-out, numerous simulations of the methodology were performed with hypothetical data. RESULTS: FeNO and periostin were identified as the only biomarkers potentially predictive of treatment effect, with cut-offs chosen by the SIDES algorithm of > 32.3 ppb and > 27.4 ng/ml, respectively. The FeNO > 32.3 ppb subgroup was associated with greater AAER reductions and improvements in key secondary endpoints compared with the periostin > 27.4 ng/ml subgroup. Upon further evaluation of AAER reductions at different FeNO cut-offs, ≥37 ppb was chosen as the best cut-off for predicting tralokinumab efficacy. DISCUSSION: A rigorous statistical approach incorporating multiple methods was used to investigate the predictive properties of five potential biomarkers and to identify a participant subgroup that demonstrated an enhanced tralokinumab treatment effect. Using STRATOS 1 data, our analyses identified FeNO at a cut-off of ≥37 ppb as the best assessed biomarker for predicting enhanced treatment effect to be tested in STRATOS 2. Our findings were inconclusive, which reflects the complexity of subgroup identification in the severe asthma population. TRIAL REGISTRATION: STRATOS 1 and 2 are registered on ClinicalTrials.gov ( NCT02161757 registered on June 12, 2014, and NCT02194699 registered on July 18, 2014).

Subgroup analysis and interpretation for phase 3 confirmatory trials: White paper of the EFSPI/PSI working group on subgroup analysis.

Subgroup by treatment interaction assessments are routinely performed when analysing clinical trials and are particularly important for phase 3 trials where the results may affect regulatory labelling. Interpretation of such interactions is particularly difficult, as on one hand the subgroup finding can be due to chance, but equally such analyses are known to have a low chance of detecting differential treatment effects across subgroup levels, so may overlook important differences in therapeutic efficacy. EMA have therefore issued draft guidance on the use of subgroup analyses in this setting. Although this guidance provided clear proposals on the importance of pre-specification of likely subgroup effects and how to use this when interpreting trial results, it is less clear which analysis methods would be reasonable, and how to interpret apparent subgroup effects in terms of whether further evaluation or action is necessary. A PSI/EFSPI Working Group has therefore been investigating a focused set of analysis approaches to assess treatment effect heterogeneity across subgroups in confirmatory clinical trials that take account of the number of subgroups explored and also investigating the ability of each method to detect such subgroup heterogeneity. This evaluation has shown that the plotting of standardised effects, bias-adjusted bootstrapping method and SIDES method all perform more favourably than traditional approaches such as investigating all subgroup-by-treatment interactions individually or applying a global test of interaction. Therefore, these approaches should be considered to aid interpretation and provide context for observed results from subgroup analyses conducted for phase 3 clinical trials.