Predicting subgroup treatment effects for a new study: Motivations, results and learnings from running a data challenge in a pharmaceutical corporation.

We present the motivation, experience, and learnings from a data challenge conducted at a large pharmaceutical corporation on the topic of subgroup identification. The data challenge aimed at exploring approaches to subgroup identification for future clinical trials. To mimic a realistic setting, participants had access to 4 Phase III clinical trials to derive a subgroup and predict its treatment effect on a future study not accessible to challenge participants. A total of 30 teams registered for the challenge with around 100 participants, primarily from Biostatistics organization. We outline the motivation for running the challenge, the challenge rules, and logistics. Finally, we present the results of the challenge, the participant feedback as well as the learnings. We also present our view on the implications of the results on exploratory analyses related to treatment effect heterogeneity.

Interim decision making in seamless trial designs: An application in an adaptive dose-finding study in a rare kidney disease.

Adaptive seamless trial designs, combining the learning and confirming cycles of drug development in a single trial, have gained popularity in recent years. Adaptations may include dose selection, sample size re-estimation and enrichment of the study population. Despite methodological advances and recognition of the potential efficiency gains such designs offer, their implementation, including how to enable efficient decision making on the adaptations in interim analyzes, remains a key challenge in their adoption. This manuscript uses a case study of an adaptive seamless proof-of-concept (Phase 2a)/dose-finding (Phase 2b) to showcase potential adaptive features that can be implemented in trial designs at earlier development stages and the role of simulations in assessing the design operating characteristics and specifying the decision rules for the adaptations. It further outlines the elements needed to support successful interim analysis decision making on the adaptations while safeguarding study integrity, including the role of different stakeholders, interactive simulation-based tools to facilitate decision making and operational aspects requiring preplanning. The benefits of the adaptive Phase 2a/2b design chosen compared to following the traditional two separate studies (2a and 2b) paradigm are discussed. With careful planning and appreciation of their complexity and components needed for their implementation, seamless adaptive designs have the potential to yield significant savings both in terms of time and resources.

Bayesian inference for a principal stratum estimand on recurrent events truncated by death.

Recurrent events are often important endpoints in randomized clinical trials. For example, the number of recurrent disease-related hospitalizations may be considered as a clinically meaningful endpoint in cardiovascular studies. In some settings, the recurrent event process may be terminated by an event such as death, which makes it more challenging to define and estimate a causal treatment effect on recurrent event endpoints. In this paper, we focus on the principal stratum estimand, where the treatment effect of interest on recurrent events is defined among subjects who would be alive regardless of the assigned treatment. For the estimation of the principal stratum effect in randomized clinical trials, we propose a Bayesian approach based on a joint model of the recurrent event and death processes with a frailty term accounting for within-subject correlation. We also present Bayesian posterior predictive check procedures for assessing the model fit. The proposed approaches are demonstrated in the randomized Phase III chronic heart failure trial PARAGON-HF (NCT01920711).

Sensitivity analyses for the principal ignorability assumption using multiple imputation.

In the context of clinical trials, there is interest in the treatment effect for subpopulations of patients defined by intercurrent events, namely disease-related events occurring after treatment initiation that affect either the interpretation or the existence of endpoints. With the principal stratum strategy, the ICH E9(R1) guideline introduces a formal framework in drug development for defining treatment effects in such subpopulations. Statistical estimation of the treatment effect can be performed based on the principal ignorability assumption using multiple imputation approaches. Principal ignorability is a conditional independence assumption that cannot be directly verified; therefore, it is crucial to evaluate the robustness of results to deviations from this assumption. As a sensitivity analysis, we propose a joint model that multiply imputes the principal stratum membership and the outcome variable while allowing different levels of violation of the principal ignorability assumption. We illustrate with a simulation study that the joint imputation model-based approaches are superior to naive subpopulation analyses. Motivated by an oncology clinical trial, we implement the sensitivity analysis on a time-to-event outcome to assess the treatment effect in the subpopulation of patients who discontinued due to adverse events using a synthetic dataset. Finally, we explore the potential usage and provide interpretation of such analyses in clinical settings, as well as possible extension of such models in more general cases.

Comparing algorithms for characterizing treatment effect heterogeneity in randomized trials.

The identification and estimation of heterogeneous treatment effects in biomedical clinical trials are challenging, because trials are typically planned to assess the treatment effect in the overall trial population. Nevertheless, the identification of how the treatment effect may vary across subgroups is of major importance for drug development. In this work, we review some existing simulation work and perform a simulation study to evaluate recent methods for identifying and estimating the heterogeneous treatments effects using various metrics and scenarios relevant for drug development. Our focus is not only on a comparison of the methods in general, but on how well these methods perform in simulation scenarios that reflect real clinical trials. We provide the R package benchtm that can be used to simulate synthetic biomarker distributions based on real clinical trial data and to create interpretable scenarios to benchmark methods for identification and estimation of treatment effect heterogeneity.

Eliciting judgements about dependent quantities of interest: The SHeffield ELicitation Framework extension and copula methods illustrated using an asthma case study.

Pharmaceutical companies regularly need to make decisions about drug development programs based on the limited knowledge from early stage clinical trials. In this situation, eliciting the judgements of experts is an attractive approach for synthesising evidence on the unknown quantities of interest. When calculating the probability of success for a drug development program, multiple quantities of interest-such as the effect of a drug on different endpoints-should not be treated as unrelated. We discuss two approaches for establishing a multivariate distribution for several related quantities within the SHeffield ELicitation Framework (SHELF). The first approach elicits experts' judgements about a quantity of interest conditional on knowledge about another one. For the second approach, we first elicit marginal distributions for each quantity of interest. Then, for each pair of quantities, we elicit the concordance probability that both lie on the same side of their respective elicited medians. This allows us to specify a copula to obtain the joint distribution of the quantities of interest. We show how these approaches were used in an elicitation workshop that was performed to assess the probability of success of the registrational program of an asthma drug. The judgements of the experts, which were obtained prior to completion of the pivotal studies, were well aligned with the final trial results.

Identifying treatment effect heterogeneity in dose-finding trials using Bayesian hierarchical models: Identifying treatment effect heterogeneity in dose-finding trials.

An important task in drug development is to identify patients, which respond better or worse to an experimental treatment. Identifying predictive covariates, which influence the treatment effect and can be used to define subgroups of patients, is a key aspect of this task. Analyses of treatment effect heterogeneity are however known to be challenging, since the number of possible covariates or subgroups is often large, while samples sizes in earlier phases of drug development are often small. In addition, distinguishing predictive covariates from prognostic covariates, which influence the response independent of the given treatment, can often be difficult. While many approaches for these types of problems have been proposed, most of them focus on the two-arm clinical trial setting, where patients are given either the treatment or a control. In this article we consider parallel groups dose-finding trials, in which patients are administered different doses of the same treatment. To investigate treatment effect heterogeneity in this setting we propose a Bayesian hierarchical dose-response model with covariate effects on dose-response parameters. We make use of shrinkage priors to prevent overfitting, which can easily occur, when the number of considered covariates is large and sample sizes are small. We compare several such priors in simulations and also investigate dependent modeling of prognostic and predictive effects to better distinguish these two types of effects. We illustrate the use of our proposed approach using a Phase II dose-finding trial and show how it can be used to identify predictive covariates and subgroups of patients with increased treatment effects.

A New Comprehensive Approach to Assess the Probability of Success of Development Programs Before Pivotal Trials.

The point at which clinical development programs transition from early phase to pivotal trials is a critical milestone. Substantial uncertainty about the outcome of pivotal trials may remain even after seeing positive early phase data, and companies may need to make difficult prioritization decisions for their portfolio. The probability of success (PoS) of a program, a single number expressed as a percentage reflecting the multitude of risks that may influence the final program outcome, is a key decision-making tool. Despite its importance, companies often rely on crude industry benchmarks that may be "adjusted" by experts based on undocumented criteria and which are typically misaligned with the definition of success used to drive commercial forecasts, leading to overly optimistic expected net present value calculations. We developed a new framework to assess the PoS of a program before pivotal trials begin. Our definition of success encompasses the successful outcome of pivotal trials, regulatory approval and meeting the requirements for market access as outlined in the target product profile. The proposed approach is organized in four steps and uses an innovative Bayesian approach to synthesize all relevant evidence. The new PoS framework is systematic and transparent. It will help organizations to make more informed decisions. In this paper, we outline the rationale and elaborate on the structure of the proposed framework, provide examples, and discuss the benefits and challenges associated with its adoption.

Improving the assessment of the probability of success in late stage drug development.

There are several steps to confirming the safety and efficacy of a new medicine. A sequence of trials, each with its own objectives, is usually required. Quantitative risk metrics can be useful for informing decisions about whether a medicine should transition from one stage of development to the next. To obtain an estimate of the probability of regulatory approval, pharmaceutical companies may start with industry-wide success rates and then apply to these subjective adjustments to reflect program-specific information. However, this approach lacks transparency and fails to make full use of data from previous clinical trials. We describe a quantitative Bayesian approach for calculating the probability of success (PoS) at the end of phase II which incorporates internal clinical data from one or more phase IIb studies, industry-wide success rates, and expert opinion or external data if needed. Using an example, we illustrate how PoS can be calculated accounting for differences between the phase II data and future phase III trials, and discuss how the methods can be extended to accommodate accelerated drug development pathways.

A framework to guide dose & regimen strategy for clinical drug development.

Optimizing new drug therapies remains a challenge for clinical development, despite the use of ever more sophisticated quantitative methodologies. Although conceptually simple, the idea of finding the right treatment at the right dose for the right patient to ensure an appropriate balance of risks and benefits is challenging and requires a multidisciplinary approach. In this paper, we present a framework developed as a tool for organizing knowledge and facilitating collaboration in development teams.

Connecting Instrumental Variable methods for causal inference to the Estimand Framework.

Causal inference methods are gaining increasing prominence in pharmaceutical drug development in light of the recently published addendum on estimands and sensitivity analysis in clinical trials to the E9 guideline of the International Council for Harmonisation. The E9 addendum emphasises the need to account for post-randomization or 'intercurrent' events that can potentially influence the interpretation of a treatment effect estimate at a trial's conclusion. Instrumental Variables (IV) methods have been used extensively in economics, epidemiology, and academic clinical studies for 'causal inference,' but less so in the pharmaceutical industry setting until now. In this tutorial article we review the basic tools for causal inference, including graphical diagrams and potential outcomes, as well as several conceptual frameworks that an IV analysis can sit within. We discuss in detail how to map these approaches to the Treatment Policy, Principal Stratum and Hypothetical 'estimand strategies' introduced in the E9 addendum, and provide details of their implementation using standard regression models. Specific attention is given to discussing the assumptions each estimation strategy relies on in order to be consistent, the extent to which they can be empirically tested and sensitivity analyses in which specific assumptions can be relaxed. We finish by applying the methods described to simulated data closely matching two recent pharmaceutical trials to further motivate and clarify the ideas.

Principal stratum strategy: Potential role in drug development.

A randomized trial allows estimation of the causal effect of an intervention compared to a control in the overall population and in subpopulations defined by baseline characteristics. Often, however, clinical questions also arise regarding the treatment effect in subpopulations of patients, which would experience clinical or disease related events post-randomization. Events that occur after treatment initiation and potentially affect the interpretation or the existence of the measurements are called intercurrent events in the ICH E9(R1) guideline. If the intercurrent event is a consequence of treatment, randomization alone is no longer sufficient to meaningfully estimate the treatment effect. Analyses comparing the subgroups of patients without the intercurrent events for intervention and control will not estimate a causal effect. This is well known, but post-hoc analyses of this kind are commonly performed in drug development. An alternative approach is the principal stratum strategy, which classifies subjects according to their potential occurrence of an intercurrent event on both study arms. We illustrate with examples that questions formulated through principal strata occur naturally in drug development and argue that approaching these questions with the ICH E9(R1) estimand framework has the potential to lead to more transparent assumptions as well as more adequate analyses and conclusions. In addition, we provide an overview of assumptions required for estimation of effects in principal strata. Most of these assumptions are unverifiable and should hence be based on solid scientific understanding. Sensitivity analyses are needed to assess robustness of conclusions.

Exposure-response modelling approaches for determining optimal dosing rules in children.

Within paediatric populations, there may be distinct age groups characterised by different exposure-response relationships. Several regulatory guidance documents have suggested general age groupings. However, it is not clear whether these categorisations will be suitable for all new medicines and in all disease areas. We consider two model-based approaches to quantify how exposure-response model parameters vary over a continuum of ages: Bayesian penalised B-splines and model-based recursive partitioning. We propose an approach for deriving an optimal dosing rule given an estimate of how exposure-response model parameters vary with age. Methods are initially developed for a linear exposure-response model. We perform a simulation study to systematically evaluate how well the various approaches estimate linear exposure-response model parameters and the accuracy of recommended dosing rules. Simulation scenarios are motivated by an application to epilepsy drug development. Results suggest that both bootstrapped model-based recursive partitioning and Bayesian penalised B-splines can estimate underlying changes in linear exposure-response model parameters as well as (and in many scenarios, better than) a comparator linear model adjusting for a categorical age covariate with levels following International Conference on Harmonisation E11 groupings. Furthermore, the Bayesian penalised B-splines approach consistently estimates the intercept and slope more accurately than the bootstrapped model-based recursive partitioning. Finally, approaches are extended to estimate Emax exposure-response models and are illustrated with an example motivated by an in vitro study of cyclosporine.

A multiple comparison procedure for dose-finding trials with subpopulations.

Identifying subgroups of patients with an enhanced response to a new treatment has become an area of increased interest in the last few years. When there is knowledge about possible subpopulations with an enhanced treatment effect before the start of a trial it might be beneficial to set up a testing strategy, which tests for a significant treatment effect not only in the full population, but also in these prespecified subpopulations. In this paper, we present a parametric multiple testing approach for tests in multiple populations for dose-finding trials. Our approach is based on the MCP-Mod methodology, which uses multiple comparison procedures (MCPs) to test for a dose-response signal, while considering multiple possible candidate dose-response shapes. Our proposed methods allow for heteroscedastic error variances between populations and control the family-wise error rate over tests in multiple populations and for multiple candidate models. We show in simulations that the proposed multipopulation testing approaches can increase the power to detect a significant dose-response signal over the standard single-population MCP-Mod, when the specified subpopulation has an enhanced treatment effect.

Testing multiple dose combinations in clinical trials.

Drug combination trials are often motivated by the fact that individual drugs target the same disease but via different routes. A combination of such drugs may then have an overall better effect than the individual treatments which has to be verified by clinical trials. Several statistical methods have been explored that discuss the problem of comparing a fixed-dose combination therapy to each of its components. But an extension of these approaches to multiple dose combinations can be difficult and is not yet fully investigated. In this paper, we propose two approaches by which one can provide confirmatory assurance with familywise error rate control, that the combination of two drugs at differing doses is more effective than either component doses alone. These approaches involve multiple comparisons in multilevel factorial designs where the type 1 error can be controlled first, by bootstrapping tests, and second, by considering the least favorable null configurations for a family of union intersection tests. The main advantage of the new approaches is that their implementation is simple. The implementation of these new approaches is illustrated with a real data example from a blood pressure reduction trial. Extensive simulations are also conducted to evaluate the new approaches and benchmark them with existing ones. We also present an illustration of the relationship between the different approaches. We observed that the bootstrap provided some power advantages over the other approaches with the disadvantage that there may be some error rate inflation for small sample sizes.

Subgroup identification in dose-finding trials via model-based recursive partitioning.

An important task in early-phase drug development is to identify patients, which respond better or worse to an experimental treatment. While a variety of different subgroup identification methods have been developed for the situation of randomized clinical trials that study an experimental treatment and control, much less work has been done in the situation when patients are randomized to different dose groups. In this article, we propose new strategies to perform subgroup analyses in dose-finding trials and discuss the challenges, which arise in this new setting. We consider model-based recursive partitioning, which has recently been applied to subgroup identification in 2-arm trials, as a promising method to tackle these challenges and assess its viability using a real trial example and simulations. Our results show that model-based recursive partitioning can be used to identify subgroups of patients with different dose-response curves and improves estimation of treatment effects and minimum effective doses compared to models ignoring possible subgroups, when heterogeneity among patients is present.

A Longitudinal Item Response Theory Model to Characterize Cognition Over Time in Elderly Subjects.

For drug development in neurodegenerative diseases such as Alzheimer's disease, it is important to understand which cognitive domains carry the most information on the earliest signs of cognitive decline, and which subject characteristics are associated with a faster decline. A longitudinal Item Response Theory (IRT) model was developed for the Basel Study on the Elderly, in which the Consortium to Establish a Registry for Alzheimer's Disease - Neuropsychological Assessment Battery (with additions) and the California Verbal Learning Test were measured on 1,750 elderly subjects for up to 13.9 years. The model jointly captured the multifaceted nature of cognition and its longitudinal trajectory. The word list learning and delayed recall tasks carried the most information. Greater age at baseline, fewer years of education, and positive APOEɛ4 carrier status were associated with a faster cognitive decline. Longitudinal IRT modeling is a powerful approach for progressive diseases with multifaceted endpoints.

Phase II dose-response trials: A simulation study to compare analysis method performance under design considerations.

Phase II trials are intended to provide information about the dose-response relationship and to support the choice of doses for a pivotal phase III trial. Recently, new analysis methods have been proposed to address these objectives, and guidance is needed to select the most appropriate analysis method in specific situations. We set up a simulation study to evaluate multiple performance measures of one traditional and three more recent dose-finding approaches under four design options and illustrate the investigated analysis methods with an example from clinical practice. Our results reveal no general recommendation for a particular analysis method across all design options and performance measures. However, we also demonstrate that the new analysis methods are worth the effort compared to the traditional ANOVA-based approach.

Likelihood ratio tests for a dose-response effect using multiple nonlinear regression models.

We consider the problem of testing for a dose-related effect based on a candidate set of (typically nonlinear) dose-response models using likelihood-ratio tests. For the considered models this reduces to assessing whether the slope parameter in these nonlinear regression models is zero or not. A technical problem is that the null distribution (when the slope is zero) depends on non-identifiable parameters, so that standard asymptotic results on the distribution of the likelihood-ratio test no longer apply. Asymptotic solutions for this problem have been extensively discussed in the literature. The resulting approximations however are not of simple form and require simulation to calculate the asymptotic distribution. In addition, their appropriateness might be doubtful for the case of a small sample size. Direct simulation to approximate the null distribution is numerically unstable due to the non identifiability of some parameters. In this article, we derive a numerical algorithm to approximate the exact distribution of the likelihood-ratio test under multiple models for normally distributed data. The algorithm uses methods from differential geometry and can be used to evaluate the distribution under the null hypothesis, but also allows for power and sample size calculations. We compare the proposed testing approach to the MCP-Mod methodology and alternative methods for testing for a dose-related trend in a dose-finding example data set and simulations.