Going beyond probability of success: Opportunities for statisticians to influence quantitative decision-making at the portfolio level.

The pharmaceutical industry is plagued with long, costly development and high risk. Therefore, a company's effective management and optimisation of a portfolio of projects is critical for success. Project metrics such as the probability of success enable modelling of a company's pipeline accounting for the high uncertainty inherent within the industry. Making portfolio decisions inherently involves managing risk, and statisticians are ideally positioned to champion not only the derivation of metrics for individual projects, but also advocate decision-making at a broader portfolio level. This article aims to examine the existing different portfolio decision-making approaches and to suggest opportunities for statisticians to add value in terms of introducing probabilistic thinking, quantitative decision-making, and increasingly advanced methodologies.

Estimation of treatment effects in early-phase randomized clinical trials involving external control data.

There are good reasons to perform a randomized controlled trial (RCT) even in early phases of clinical development. However, the low sample sizes in those settings lead to high variability of the treatment effect estimate. The variability could be reduced by adding external control data if available. For the common setting of suitable subject-level control group data only available from one external (clinical trial or real-world) data source, we evaluate different analysis options for estimating the treatment effect via hazard ratios. The impact of the external control data is usually guided by the level of similarity with the current RCT data. Such level of similarity can be determined via outcome and/or baseline covariate data comparisons. We provide an overview over existing methods, propose a novel option for a combined assessment of outcome and baseline data, and compare a selected set of approaches in a simulation study under varying assumptions regarding observable and unobservable confounder distributions using a time-to-event model. Our various simulation scenarios also reflect the differences between external clinical trial and real-world data. Data combinations via simple outcome-based borrowing or simple propensity score weighting with baseline covariate data are not recommended. Analysis options which conflate outcome and baseline covariate data perform best in our simulation study.

An adaptive design for early clinical development including interim decision for single-arm trial with external controls or randomized trial.

In early clinical development, randomized controlled trials (RCT) or single-arm trials with external controls (SATwEC) are design options, which allow adjustment for confounding: RCT via design, SATwEC via analysis using propensity score methods. SATwEC requires less investment than RCT. However, if the confounder space substantially differs between the experimental and external control group, the SATwEC might lead to inappropriate decisions for further development. We develop an adaptive two-stage design (ATD) for early clinical development that reduces the risk of unreliable decision-making at the end of a SATwEC. In Stage I, subjects are solely assigned to the experimental group. If at the interim the propensity score distributions of internal and external data are comparable based on the preference score, the subjects in stage II will again be solely assigned to the experimental arm; if not, a randomized stage II will be conducted. In a simulation study guided by a motivating example, data is generated using a time-to-event model with observable and unobservable confounders. The confounder space is varied to investigate the impact on false go/stop probabilities as well as a loss function, which reflects the quality of treatment effect estimates and decision-making. The proposed ATD provides a compromise between optimizing quality (as expressed by false go/stop probabilities and the loss function) and investment (defined by sample size and trial duration).

Robust group sequential designs for trials with survival endpoints and delayed response.

Randomized clinical trials in oncology typically utilize time-to-event endpoints such as progression-free survival or overall survival as their primary efficacy endpoints, and the most commonly used statistical test to analyze these endpoints is the log-rank test. The power of the log-rank test depends on the behavior of the hazard ratio of the treatment arm to the control arm. Under the assumption of proportional hazards, the log-rank test is asymptotically fully efficient. However, this proportionality assumption does not hold true if there is a delayed treatment effect. Cancer immunology has evolved over time and several cancer vaccines are available in the market for treating existing cancers. This includes sipuleucel-T for metastatic hormone-refractory prostate cancer, nivolumab for metastatic melanoma, and pembrolizumab for advanced nonsmall-cell lung cancer. As cancer vaccines require some time to elicit an immune response, a delayed treatment effect is observed, resulting in a violation of the proportional hazards assumption. Thus, the traditional log-rank test may not be optimal for testing immuno-oncology drugs in randomized clinical trials. Moreover, the new immuno-oncology compounds have been shown to be very effective in prolonging overall survival. Therefore, it is desirable to implement a group sequential design with the possibility of early stopping for overwhelming efficacy. In this paper, we investigate the max-combo test, which utilizes the maximum of two weighted log-rank statistics, as a robust alternative to the log-rank test. The new test is implemented for two-stage designs with possible early stopping at the interim analysis time point. Two classes of weights are investigated for the max-combo test: the Fleming and Harrington (1981) Gρ,γ$G^{\rho , \gamma }$ weights and the Magirr and Burman (2019) modest (τ∗)$ (\tau ^{*})$  weights.

Progression-free survival in oncological clinical studies: Assessment time bias and methods for its correction.

Progression-free survival (PFS) is a frequently used endpoint in oncological clinical studies. In case of PFS, potential events are progression and death. Progressions are usually observed delayed as they can be diagnosed not before the next study visit. For this reason potential bias of treatment effect estimates for progression-free survival is a concern. In randomized trials and for relative treatment effects measures like hazard ratios, bias-correcting methods are not necessarily required or have been proposed before. However, less is known on cross-trial comparisons of absolute outcome measures like median survival times. This paper proposes a new method for correcting the assessment time bias of progression-free survival estimates to allow a fair cross-trial comparison of median PFS. Using median PFS for example, the presented method approximates the unknown posterior distribution by a Bayesian approach based on simulations. It is shown that the proposed method leads to a substantial reduction of bias as compared to estimates derived from maximum likelihood or Kaplan-Meier estimates. Bias could be reduced by more than 90% over a broad range of considered situations differing in assessment times and underlying distributions. By coverage probabilities of at least 94% based on the credibility interval of the posterior distribution the resulting parameters hold common confidence levels. In summary, the proposed approach is shown to be useful for a cross-trial comparison of median PFS.

Delayed treatment effects, treatment switching and heterogeneous patient populations: How to design and analyze RCTs in oncology.

In the analysis of survival times, the logrank test and the Cox model have been established as key tools, which do not require specific distributional assumptions. Under the assumption of proportional hazards, they are efficient and their results can be interpreted unambiguously. However, delayed treatment effects, disease progression, treatment switchers or the presence of subgroups with differential treatment effects may challenge the assumption of proportional hazards. In practice, weighted logrank tests emphasizing either early, intermediate or late event times via an appropriate weighting function may be used to accommodate for an expected pattern of non-proportionality. We model these sources of non-proportional hazards via a mixture of survival functions with piecewise constant hazard. The model is then applied to study the power of unweighted and weighted log-rank tests, as well as maximum tests allowing different time dependent weights. Simulation results suggest a robust performance of maximum tests across different scenarios, with little loss in power compared to the most powerful among the considered weighting schemes and huge power gain compared to unfavorable weights. The actual sources of non-proportional hazards are not obvious from resulting populationwise survival functions, highlighting the importance of detailed simulations in the planning phase of a trial when assuming non-proportional hazards.We provide the required tools in a software package, allowing to model data generating processes under complex non-proportional hazard scenarios, to simulate data from these models and to perform the weighted logrank tests.

Optimal designs for phase II/III drug development programs including methods for discounting of phase II results.

BACKGROUND: Go/no-go decisions after phase II and sample size chosen for phase III are usually based on phase II results (e.g., the treatment effect estimate of phase II). Due to the decision rule (only promising phase II results lead to phase III), treatment effect estimates from phase II that initiate a phase III trial commonly overestimate the true treatment effect. Underpowered phase III trials are the consequence. Optimistic findings may then not be reproduced, leading to the failure of potentially expensive drug development programs. For some disease areas these failure rates are described to be quite high: 62.5%. METHODS: We integrate the ideas of multiplicative and additive adjustment of treatment effect estimates after go decisions in a utility-based framework for optimizing drug development programs. The design of a phase II/III program, i.e., the "right amount of adjustment", the allocation of the resources to phase II and III in terms of sample size, and the rule applied to decide whether to stop or to proceed with phase III influences its success considerably. Given specific drug development program characteristics (e.g., fixed and variable per patient costs for phase II and III, probable gain in case of market launch), optimal designs with respect to the maximal expected utility can be identified by the proposed Bayesian-frequentist approach. The method will be illustrated by application to practical examples characteristic for oncological studies. RESULTS: In general, our results show that the program set-ups with adjusted treatment effect estimate used for phase III planning are superior to the "naïve" program set-ups with respect to the maximal expected utility. Therefore, we recommend considering an adjusted phase II treatment effect estimate for the phase III sample size calculation. However, there is no one-fits-all design. CONCLUSION: Individual drug development planning for a specific program is necessary to find the optimal design. The optimal choice of the design parameters for a specific drug development program at hand can be found by our user friendly R Shiny application and package (both assessable open-source via [1]).

Optimal decision-making in oncology development programs based on probability of success for phase III utilizing phase II/III data on response and overall survival.

In clinical development, there is a trade-off between investment and level of confidence in the potential of the drug before going into phase III. Reduced investment requires the use of short-term endpoints. On new compounds, only limited information about the relationship between treatment effects of short- and long-term endpoints is usually available. Therefore, decision-making solely based on short-term endpoints does not seem desirable. Our goal is to plan an efficient development program, which uses short- and long-term endpoints data for decision-making. We found that with limited prior information and restrictions on maximum sample size, decision-making after phase II cannot be substantially improved. We follow the concept of a "phase 2+" design where after a go-to-phase-III-decision, further follow-up data from phase II are employed to make interim decisions on phase III. The program will be stopped early when additional phase II and/or available phase III data lead to a low probability of success (PoS). We utilize information from a multi-categorical short-term endpoint (response status) and a long-term endpoint (overall survival (OS)) to determine the PoS in phase III with OS as the primary endpoint. Optimal combinations of decision boundaries and time points are demonstrated in a simulation study. Our results show that the proposed second look using additional follow-up data from phase II/III improves PoS estimates compared to the first look, especially when prior data about the control arm is available. The proposed planning strategy allows a customized compromise between the quality of decision-making and program duration.

Adjustment for exploratory cut-off selection in randomized clinical trials with survival endpoint.

Defining the target population based on predictive biomarkers plays an important role during clinical development. After establishing a relationship between a biomarker candidate and response to treatment in exploratory phases, a subsequent confirmatory trial ideally involves only subjects with high potential of benefiting from the new compound. In order to identify those subjects in case of a continuous biomarker, a cut-off is needed. Usually, a cut-off is chosen that resulted in a subgroup with a large observed treatment effect in an exploratory trial. However, such a data-driven selection may lead to overoptimistic expectations for the subsequent confirmatory trial. Treatment effect estimates, probability of success, and posterior probabilities are useful measures for deciding whether or not to conduct a confirmatory trial enrolling the biomarker-defined population. These measures need to be adjusted for selection bias. We extend previously introduced Approximate Bayesian Computation techniques for adjustment of subgroup selection bias to a time-to-event setting with cut-off selection. Challenges in this setting are that treatment effects become time-dependent and that subsets are defined by the biomarker distribution. Simulation studies show that the proposed method provides adjusted statistical measures which are superior to naïve Maximum Likelihood estimators as well as simple shrinkage estimators.

Optimal planning of phase II/III programs for clinical trials with multiple endpoints.

Owing to increased costs and competition pressure, drug development becomes more and more challenging. Therefore, there is a strong need for improving efficiency of clinical research by developing and applying methods for quantitative decision making. In this context, the integrated planning for phase II/III programs plays an important role as numerous quantities can be varied that are crucial for cost, benefit, and program success. Recently, a utility-based framework has been proposed for an optimal planning of phase II/III programs that puts the choice of decision boundaries and phase II sample sizes on a quantitative basis. However, this method is restricted to studies with a single time-to-event endpoint. We generalize this procedure to the setting of clinical trials with multiple endpoints and (asymptotically) normally distributed test statistics. Optimal phase II sample sizes and go/no-go decision rules are provided for both the "all-or-none" and "at-least-one" win criteria. Application of the proposed method is illustrated by drug development programs in the fields of Alzheimer disease and oncology.

Simulation-based adjustment after exploratory biomarker subgroup selection in phase II.

As part of the evaluation of phase II trials, it is common practice to perform exploratory subgroup analyses with the aim of identifying patient populations with a beneficial treatment effect. When investigating targeted therapies, these subgroups are typically defined by biomarkers. Promising results may lead to the decision to select the respective subgroup as the target population for a subsequent phase III trial. However, a selection based on a large observed treatment effect may potentially induce an upwards-bias leading to over-optimistic expectations on the success probability of the phase III trial. We describe how Approximate Bayesian Computation techniques can be used to derive a simulation-based bias adjustment method in this situation. Recommendations for the implementation of the approach are given. Simulation studies show that the proposed method reduces bias substantially compared with the maximum likelihood estimator. The procedure is illustrated with data from an oncology trial. Copyright © 2017 John Wiley & Sons, Ltd.

Probability of success for phase III after exploratory biomarker analysis in phase II.

The probability of success or average power describes the potential of a future trial by weighting the power with a probability distribution of the treatment effect. The treatment effect estimate from a previous trial can be used to define such a distribution. During the development of targeted therapies, it is common practice to look for predictive biomarkers. The consequence is that the trial population for phase III is often selected on the basis of the most extreme result from phase II biomarker subgroup analyses. In such a case, there is a tendency to overestimate the treatment effect. We investigate whether the overestimation of the treatment effect estimate from phase II is transformed into a positive bias for the probability of success for phase III. We simulate a phase II/III development program for targeted therapies. This simulation allows to investigate selection probabilities and allows to compare the estimated with the true probability of success. We consider the estimated probability of success with and without subgroup selection. Depending on the true treatment effects, there is a negative bias without selection because of the weighting by the phase II distribution. In comparison, selection increases the estimated probability of success. Thus, selection does not lead to a bias in probability of success if underestimation due to the phase II distribution and overestimation due to selection cancel each other out. We recommend to perform similar simulations in practice to get the necessary information about the risk and chances associated with such subgroup selection designs.

Utility-based optimization of phase II/III programs.

Phase II and phase III trials play a crucial role in drug development programs. They are costly and time consuming and, because of high failure rates in late development stages, at the same time risky investments. Commonly, sample size calculation of phase III is based on the treatment effect observed in phase II. Therefore, planning of phases II and III can be linked. The performance of the phase II/III program crucially depends on the allocation of the resources to phases II and III by appropriate choice of the sample size and the rule applied to decide whether to stop the program after phase II or to proceed. We present methods for a program-wise phase II/III planning that aim at determining optimal phase II sample sizes and go/no-go decisions in a time-to-event setting. Optimization is based on a utility function that takes into account (fixed and variable) costs of the drug development program and potential gains after successful launch. The proposed methods are illustrated by application to a variety of scenarios typically met in oncology drug development.

Improving Probabilities of Correct Interim Decision in Population Enrichment Designs.

Choice of target population is an essential part at the design stage of clinical trials. Data from earlier clinical development might suggest that the treatment is more effective in a subpopulation, but there might not be enough evidence to restrict the target population upfront. Adaptive designs allow modification of the target population based on interim data. Decision for modification should be based on objective decision rules. The presented decision rules maximize the weighted probability of correct interim decisions based on prior assumptions. Evaluation of decision rules in the planning phase can improve probabilities of correct interim decision and power.

Sample size planning for phase II trials based on success probabilities for phase III.

In recent years, high failure rates in phase III trials were observed. One of the main reasons is overoptimistic assumptions for the planning of phase III resulting from limited phase II information and/or unawareness of realistic success probabilities. We present an approach for planning a phase II trial in a time-to-event setting that considers the whole phase II/III clinical development programme. We derive stopping boundaries after phase II that minimise the number of events under side conditions for the conditional probabilities of correct go/no-go decision after phase II as well as the conditional success probabilities for phase III. In addition, we give general recommendations for the choice of phase II sample size. Our simulations show that unconditional probabilities of go/no-go decision as well as the unconditional success probabilities for phase III are influenced by the number of events observed in phase II. However, choosing more than 150 events in phase II seems not necessary as the impact on these probabilities then becomes quite small. We recommend considering aspects like the number of compounds in phase II and the resources available when determining the sample size. The lower the number of compounds and the lower the resources are for phase III, the higher the investment for phase II should be.

Capecitabine and cisplatin with or without cetuximab for patients with previously untreated advanced gastric cancer (EXPAND): a randomised, open-label phase 3 trial.

BACKGROUND: Patients with advanced gastric cancer have a poor prognosis and few efficacious treatment options. We aimed to assess the addition of cetuximab to capecitabine-cisplatin chemotherapy in patients with advanced gastric or gastro-oesophageal junction cancer. METHODS: In our open-label, randomised phase 3 trial (EXPAND), we enrolled adults aged 18 years or older with histologically confirmed locally advanced unresectable (M0) or metastatic (M1) adenocarcinoma of the stomach or gastro-oesophageal junction. We enrolled patients at 164 sites (teaching hospitals and clinics) in 25 countries, and randomly assigned eligible participants (1:1) to receive first-line chemotherapy with or without cetuximab. Randomisation was done with a permuted block randomisation procedure (variable block size), stratified by disease stage (M0 vs M1), previous oesophagectomy or gastrectomy (yes vs no), and previous (neo)adjuvant (radio)chemotherapy (yes vs no). Treatment consisted of 3-week cycles of twice-daily capecitabine 1000 mg/m(2) (on days 1-14) and intravenous cisplatin 80 mg/m(2) (on day 1), with or without weekly cetuximab (400 mg/m(2) initial infusion on day 1 followed by 250 mg/m(2) per week thereafter). The primary endpoint was progression-free survival (PFS), assessed by a masked independent review committee in the intention-to-treat population. We assessed safety in all patients who received at least one dose of study drug. This study is registered at EudraCT, number 2007-004219-75. FINDINGS: Between June 30, 2008, and Dec 15, 2010, we enrolled 904 patients. Median PFS for 455 patients allocated capecitabine-cisplatin plus cetuximab was 4.4 months (95% CI 4.2-5.5) compared with 5.6 months (5.1-5.7) for 449 patients who were allocated to receive capecitabine-cisplatin alone (hazard ratio 1.09, 95% CI 0.92-1.29; p=0.32). 369 (83%) of 446 patients in the chemotherapy plus cetuximab group and 337 (77%) of 436 patients in the chemotherapy group had grade 3-4 adverse events, including grade 3-4 diarrhoea, hypokalaemia, hypomagnesaemia, rash, and hand-foot syndrome. Grade 3-4 neutropenia was more common in controls than in patients who received cetuximab. Incidence of grade 3-4 skin reactions and acne-like rash was substantially higher in the cetuximab-containing regimen than in the control regimen. 239 (54%) of 446 in the cetuximab group and 194 (44%) of 436 in the control group had any grade of serious adverse event. INTERPRETATION: Addition of cetuximab to capecitabine-cisplatin provided no additional benefit to chemotherapy alone in the first-line treatment of advanced gastric cancer in our trial. FUNDING: Merck KGaA.

Simultaneous inference procedures for the comparison of multiple characteristics of two survival functions.

Survival time is the primary endpoint of many randomized controlled trials, and a treatment effect is typically quantified by the hazard ratio under the assumption of proportional hazards. Awareness is increasing that in many settings this assumption is a priori violated, for example, due to delayed onset of drug effect. In these cases, interpretation of the hazard ratio estimate is ambiguous and statistical inference for alternative parameters to quantify a treatment effect is warranted. We consider differences or ratios of milestone survival probabilities or quantiles, differences in restricted mean survival times, and an average hazard ratio to be of interest. Typically, more than one such parameter needs to be reported to assess possible treatment benefits, and in confirmatory trials, the according inferential procedures need to be adjusted for multiplicity. A simple Bonferroni adjustment may be too conservative because the different parameters of interest typically show considerable correlation. Hence simultaneous inference procedures that take into account the correlation are warranted. By using the counting process representation of the mentioned parameters, we show that their estimates are asymptotically multivariate normal and we provide an estimate for their covariance matrix. We propose according to the parametric multiple testing procedures and simultaneous confidence intervals. Also, the logrank test may be included in the framework. Finite sample type I error rate and power are studied by simulation. The methods are illustrated with an example from oncology. A software implementation is provided in the R package nph.

Sample size planning for survival prediction with focus on high-dimensional data.

Sample size planning should reflect the primary objective of a trial. If the primary objective is prediction, the sample size determination should focus on prediction accuracy instead of power. We present formulas for the determination of training set sample size for survival prediction. Sample size is chosen to control the difference between optimal and expected prediction error. Prediction is carried out by Cox proportional hazards models. The general approach considers censoring as well as low-dimensional and high-dimensional explanatory variables. For dimension reduction in the high-dimensional setting, a variable selection step is inserted. If not all informative variables are included in the final model, the effect estimates are biased towards zero. The bias affects the prediction error, and its magnitude is influenced by the sample size. For variable selection, we consider two approaches: least absolute shrinkage and selection operator (LASCO) and univariable selection. For univariable selection, we can calculate input parameters for the sample size formula. For the LASCO, supportive simulations are necessary to appropriately choose the input parameters. We investigate the performance of the proposed formulas with the use of simulations. Simulation results support the validity of the sample size formulas. An application of a real data example illustrates the practical implementation of the method.

Frozen section diagnosis in donor liver biopsies: observer variation of semiquantitative and quantitative steatosis assessment.

Donor livers are not generally accepted for liver transplantation if intraoperative frozen section histology on wedge biopsies provides evidence for more severe steatosis. In this reliability study, assessment of steatosis in donor liver biopsies by different approaches (frozen sections vs. paraffin sections; macrovesicular steatosis vs. microvesicular steatosis), different observers, and different evaluation methods (conventional microscopy vs. point grid analysis on digital microphotographs) was compared. One hundred twenty consecutive donor liver biopsies were investigated. Intraoperative diagnosis was made on hematoxylin and eosin (H&E)-stained frozen sections. The residual portion of each biopsy was analyzed later on H&E-, diastase-resistant PAS-, and Elastica van Gieson-stained paraffin sections. Microvesicular steatosis and macrovesicular steatosis were classified semiquantitatively into 5 % steps. Additionally, point grid counting was applied on ten digital microphotographs per slide. The values for steatosis revealed a wide range of data between 0 and 70 or 85 % (mean values, 12.0-18.3 %), considering all types of specimens. The results of the two observers were highly correlated for macrovesicular steatosis (r ≥ 0.925) and for microvesicular steatosis (r ≥ 0.880). The values for macrovesicular and microvesicular steatosis, however, showed poor correlation (r ≤ 0.581). The rate of agreement between the two observers ranged between 84.2 and 95.8 % (κ, 0.763-0.937), depending on the threshold setting. For point grid analysis, significantly lower mean values and ranges for both types of steatosis compared to conventional histopathology were found (p < 0.001). Comparing the results of point grid analysis with those of conventional histopathology, a relatively loose correlation was found (r, 0.581-0.779). Intraoperative histology remains a reliable and highly relevant method for the assessment of steatosis in liver donor grafts. It represents one important component in the decision-finding whether or not a donor liver should be accepted and should possibly be combined with results of preoperative computed tomography imaging. Considering our data, macrovesicular and microvesicular steatosis should be analyzed separately due to the limited correlation between them.

Adaptive designs with correlated test statistics.

In clinical trials, the collected observations such as clustered data or repeated measurements are often correlated. As a consequence, test statistics in a multistage design are correlated. Adaptive designs were originally developed for independent test statistics. We present a general framework for two-stage adaptive designs with correlated test statistics. We show that the significance level for the Bauer-Köhne design is inflated for positively correlated test statistics from a bivariate normal distribution. The decision boundary for the second stage can be modified so that type one error is controlled. This general concept is expandable to other adaptive designs. In order to use these designs, the correlation between test statistics has to be estimated. For a known covariance matrix, we show how correlation can be determined within the framework of linear mixed models. A sample size reassessment rule is proposed and evaluated for an unknown covariance matrix by simulation. As Wald test statistics in linear mixed models have independent increments, we use this property to create valid test procedures. We compare these procedures with the proposed design in our simulations.