Statistical approaches for the integration of external controls in a cystic fibrosis clinical trial: a simulation and an application.

Development of new therapeutics for a rare disease such as cystic fibrosis (CF) is hindered by challenges in accruing enough patients for clinical trials. Using external controls from well-matched historical trials can reduce prospective trial sizes, and this approach has supported regulatory approval of new interventions for other rare diseases. We consider three statistical methods that incorporate external controls into a hypothetical clinical trial of a new treatment to reduce pulmonary exacerbations in CF patients: 1) inverse probability weighting, 2) Bayesian modeling with propensity score-based power priors, and 3) hierarchical Bayesian modeling with commensurate priors. We compare the methods via simulation study and in a real clinical trial data setting. Simulations showed that bias in the treatment effect was <4% using any of the methods, with type 1 error (or in the Bayesian cases, posterior probability of the null hypothesis) usually <5%. Inverse probability weighting was sensitive to similarity in prevalence of the covariates between historical and prospective trial populations. The commensurate prior method performed best with real clinical trial data. Using external controls to reduce trial size in future clinical trials holds promise and can advance the therapeutic pipeline for rare diseases.

Applying propensity methods to the United States transplant registry for external real-world evidence control arms for 5-year survival in the BENEFIT study.

To address the challenges of assessing the impact of a reasonably likely surrogate endpoint on long-term graft survival in prospective kidney transplant clinical trials, the Transplant Therapeutics Consortium established a real-world evidence workgroup evaluating the scientific value of using transplant registry data as an external control to supplement the internal control group. The United Network for Organ Sharing retrospectively simulated the use of several distinct contemporaneous external control groups, applied multiple cause inference methods, and compared treatment effects to those observed in the BENEFIT study. Applying BENEFIT study enrollment criteria produced a smaller historical cyclosporine control arm (n = 153) and a larger, alternative (tacrolimus) historical control arm (n = 1069). Following covariate-balanced propensity scoring, Kaplan-Meier 5-year all-cause graft survivals were 81.3% and 81.7% in the Organ Procurement and Transplantation Network (OPTN) tacrolimus and cyclosporine external control arms, similar to 80.3% observed in the BENEFIT cyclosporine treatment arm. Five-year graft survival in the belatacept-less intensive arm was significantly higher than the OPTN controls using propensity scoring for comparing cyclosporine and tacrolimus. Propensity weighting using OPTN controls closely mirrored the BENEFIT study's long-term control (cyclosporine) arm's survival rate and the less intensive arm's treatment effect (significantly higher survival vs control). This study supports the feasibility and validity of using supplemental external registry controls for long-term survival in kidney transplant clinical trials.

Summix: A method for detecting and adjusting for population structure in genetic summary data.

Publicly available genetic summary data have high utility in research and the clinic, including prioritizing putative causal variants, polygenic scoring, and leveraging common controls. However, summarizing individual-level data can mask population structure, resulting in confounding, reduced power, and incorrect prioritization of putative causal variants. This limits the utility of publicly available data, especially for understudied or admixed populations where additional research and resources are most needed. Although several methods exist to estimate ancestry in individual-level data, methods to estimate ancestry proportions in summary data are lacking. Here, we present Summix, a method to efficiently deconvolute ancestry and provide ancestry-adjusted allele frequencies (AFs) from summary data. Using continental reference ancestry, African (AFR), non-Finnish European (EUR), East Asian (EAS), Indigenous American (IAM), South Asian (SAS), we obtain accurate and precise estimates (within 0.1%) for all simulation scenarios. We apply Summix to gnomAD v.2.1 exome and genome groups and subgroups, finding heterogeneous continental ancestry for several groups, including African/African American (∼84% AFR, ∼14% EUR) and American/Latinx (∼4% AFR, ∼5% EAS, ∼43% EUR, ∼46% IAM). Compared to the unadjusted gnomAD AFs, Summix's ancestry-adjusted AFs more closely match respective African and Latinx reference samples. Even on modern, dense panels of summary statistics, Summix yields results in seconds, allowing for estimation of confidence intervals via block bootstrap. Given an accompanying R package, Summix increases the utility and equity of public genetic resources, empowering novel research opportunities.

Analytical Methods for Comparing Uncontrolled Trials With External Controls From Real-World Data: A Systematic Literature Review and Comparison With European Regulatory and Health Technology Assessment Practice.

OBJECTIVES: This study aimed to provide an overview of analytical methods in scientific literature for comparing uncontrolled medicine trials with external controls from individual patient data real-world data (IPD-RWD) and to compare these methods with recommendations made in guidelines from European regulatory and health technology assessment (HTA) organizations and with their evaluations described in assessment reports. METHODS: A systematic literature review (until March 1, 2023) in PubMed and Connected Papers was performed to identify analytical methods for comparing uncontrolled trials with external controls from IPD-RWD. These methods were compared descriptively with methods recommended in method guidelines and encountered in assessment reports of the European Medicines Agency (2015-2020) and 4 European HTA organizations (2015-2023). RESULTS: Thirty-four identified scientific articles described analytical methods for comparing uncontrolled trial data with IPD-RWD-based external controls. The various methods covered controlling for confounding and/or dependent censoring, correction for missing data, and analytical comparative modeling methods. Seven guidelines also focused on research design, RWD quality, and transparency aspects, and 4 of those recommended analytical methods for comparisons with IPD-RWD. The methods discussed in regulatory (n = 15) and HTA (n = 35) assessment reports were often based on aggregate data and lacked transparency owing to the few details provided. CONCLUSIONS: Literature and guidelines suggest a methodological approach to comparing uncontrolled trials with external controls from IPD-RWD similar to target trial emulation, using state-of-the-art methods. External controls supporting regulatory and HTA decision making were rarely in line with this approach. Twelve recommendations are proposed to improve the quality and acceptability of these methods.

Evaluation of the Fill-it-up-design to use historical control data in randomized clinical trials with two arm parallel group design.

PURPOSE: In the context of clinical research, there is an increasing need for new study designs that help to incorporate already available data. With the help of historical controls, the existing information can be utilized to support the new study design, but of course, inclusion also carries the risk of bias in the study results. METHODS: To combine historical and randomized controls we investigate the Fill-it-up-design, which in the first step checks the comparability of the historical and randomized controls performing an equivalence pre-test. If equivalence is confirmed, the historical control data will be included in the new RCT. If equivalence cannot be confirmed, the historical controls will not be considered at all and the randomization of the original study will be extended. We are investigating the performance of this study design in terms of type I error rate and power. RESULTS: We demonstrate how many patients need to be recruited in each of the two steps in the Fill-it-up-design and show that the family wise error rate of the design is kept at 5 % . The maximum sample size of the Fill-it-up-design is larger than that of the single-stage design without historical controls and increases as the heterogeneity between the historical controls and the concurrent controls increases. CONCLUSION: The two-stage Fill-it-up-design represents a frequentist method for including historical control data for various study designs. As the maximum sample size of the design is larger, a robust prior belief is essential for its use. The design should therefore be seen as a way out in exceptional situations where a hybrid design is considered necessary.

Use of quantitative bias analysis to evaluate single-arm trials with real-world data external controls.

PURPOSE: Use of real-world data (RWD) for external controls added to single-arm trials (SAT) is increasingly prevalent in regulatory submissions. Due to inherent differences in the data-generating mechanisms, biases can arise. This paper aims to illustrate how to use quantitative bias analysis (QBA). METHODS: Advanced non-small cell lung cancer (NSCLC) serves as an example, where many small subsets of patients with molecular tumor subtypes exist. First, some sources of bias that may occur in oncology when comparing RWD to SAT are described. Second, using a hypothetical immunotherapy agent, a dataset is simulated based on expert input for survival analysis of advanced NSCLC. Finally, we illustrate the impact of three biases: missing confounder, misclassification of exposure, and outcome evaluation. RESULTS: For each simulated scenario, bias was induced by removing or adding data; hazard ratios (HRs) were estimated applying conventional analyses. Estimating the bias-adjusted treatment effect and uncertainty required carefully selecting the bias model and bias factors. Although the magnitude of each biased and bias-adjusted HR appeared moderate in all three hypothetical scenarios, the direction of bias was variable. CONCLUSION: These findings suggest that QBA can provide an intuitive framework for bias analysis, providing a key means of challenging assumptions about the evidence. However, the accuracy of bias analysis is itself dependent on correct specification of the bias model and bias factors. Ultimately, study design should reduce bias, but QBA allows us to evaluate the impact of unavoidable bias to assess the quality of the evidence.

Propensity score stratified MAP prior and posterior inference for incorporating information across multiple potentially heterogeneous data sources.

Incorporation of external information is becoming increasingly common when designing clinical trials. Availability of multiple sources of information has inspired the development of methodologies that account for potential heterogeneity not only between the prospective trial and the pooled external data sources but also between the different external data sources themselves. Our approach proposes an intuitive way of handling such a scenario for the continuous outcomes setting by using propensity score-based stratification and then utilizing robust meta-analytic predictive priors for each stratum to incorporate the prior data to distinguish among different external data sources in each stratum. Through extensive simulations, our approach proves to be more efficient and less biased than the currently available methods. A real case study using clinical trials that study schizophrenia from multiple different sources is also included.

Estimating treatment effect in randomized trial after control to treatment crossover using external controls.

In clinical trials, it is common to design a study that permits the administration of an experimental treatment to participants in the placebo or standard of care group post primary endpoint. This is often seen in the open-label extension phase of a phase III, pivotal study of the new medicine, where the focus is on assessing long-term safety and efficacy. With the availability of external controls, proper estimation and inference of long-term treatment effect during the open-label extension phase in the absence of placebo-controlled patients are now feasible. Within the framework of causal inference, we propose several difference-in-differences (DID) type methods and a synthetic control method (SCM) for the combination of randomized controlled trials and external controls. Our realistic simulation studies demonstrate the desirable performance of the proposed estimators in a variety of practical scenarios. In particular, DID methods outperform SCM and are the recommended methods of choice. An empirical application of the methods is demonstrated through a phase III clinical trial in rare disease.

Utility of propensity score-based Bayesian borrowing of external adult data in pediatric trials: A pragmatic evaluation through a case study in acute lymphoblastic leukemia (ALL).

A fully powered randomized controlled cancer trial can be challenging to conduct in children because of difficulties in enrollment of pediatric patients due to low disease incidence. One way to improve the feasibility of trials in pediatric patients, when clinically appropriate, is through borrowing information from comparable external adult trials in the same disease. Bayesian analysis of a pediatric trial provides a way of seamlessly augmenting pediatric trial efficacy data with data from external adult trials. However, not all external adult trial subjects may be equally clinically relevant with respect to the baseline disease severity, prognostic factors, co-morbidities, and prior therapy observed in the pediatric trial of interest. The propensity score matching method provides a way of matching the external adult subjects to the pediatric trial subjects on a set of clinically determined baseline covariates, such as baseline disease severity, prognostic factors and prior therapy. The matching then allows Bayesian information borrowing from only the most clinically relevant external adult subjects. Through a case study in pediatric acute lymphoblastic leukemia (ALL), we examine the utility of propensity score matched mixture and power priors in bringing appropriate external adult efficacy information into pediatric trial efficacy assessment, and present considerations for scaling fixed borrowing from external adult data.

A meta-analytic framework to adjust for bias in external control studies.

While randomized controlled trials (RCTs) are the gold standard for estimating treatment effects in medical research, there is increasing use of and interest in using real-world data for drug development. One such use case is the construction of external control arms for evaluation of efficacy in single-arm trials, particularly in cases where randomization is either infeasible or unethical. However, it is well known that treated patients in non-randomized studies may not be comparable to control patients-on either measured or unmeasured variables-and that the underlying population differences between the two groups may result in biased treatment effect estimates as well as increased variability in estimation. To address these challenges for analyses of time-to-event outcomes, we developed a meta-analytic framework that uses historical reference studies to adjust a log hazard ratio estimate in a new external control study for its additional bias and variability. The set of historical studies is formed by constructing external control arms for historical RCTs, and a meta-analysis compares the trial controls to the external control arms. Importantly, a prospective external control study can be performed independently of the meta-analysis using standard causal inference techniques for observational data. We illustrate our approach with a simulation study and an empirical example based on reference studies for advanced non-small cell lung cancer. In our empirical analysis, external control patients had lower survival than trial controls (hazard ratio: 0.907), but our methodology is able to correct for this bias. An implementation of our approach is available in the R package ecmeta.

Novel score test to increase power in association test by integrating external controls.

Recent advances in genotyping and sequencing technologies have enabled genetic association studies to leverage high-quality genotyped data to identify variants accounting for a substantial portion of disease risk. The usage of external controls, whose genomes have already been genotyped and are publicly available, could be a cost-effective approach to increase the power of association testing. There has been recent effort to integrate external controls while adjusting for possible batch effects, such as the integrating External Controls into Association Test (iECAT). The original iECAT test, however, cannot adjust for covariates such as age, gender, and so forth. Hence, based on the insight of iECAT, we propose a novel score-based test that allows for covariate adjustment and constructs a shrinkage score statistic that is a weighted sum of the score statistics using exclusively internal samples and uses both internal and external control samples. We assess the existence of batch effect at a variant by comparing control samples of internal and external sources. We show by simulation studies that our method has increased power over the original iECAT while controlling for type I error rates. We present the application of our method to the association studies of age-related macular degeneration (AMD) utilizing data from the International AMD Genomics Consortium and Michigan Genomics Initiative. Through the incorporation of the score test approach, we extend the use of iECAT to adjust for covariates and improve power, further honing the statistical methods needed to identify disease-causing variants within the human genome.

External control arms in oncology: current use and future directions.

Although randomized control trials allow for a comparison of treatment arms with minimal concern for confounding by known and unknown factors, a randomized study is not feasible in certain disease settings. When a randomized design is not possible, incorporating external control data into the study design can be an effective way to expand the interpretability of the results of an experimental arm by introducing the ability to carry out a formal or an informal comparative analysis. This paper provides an introduction to the concepts of external controls in oncology trials, followed by a review of relevant and current research on this topic. The paper also focuses on general considerations for designing a trial that may incorporate external control data, followed by case studies of the marketing applications submitted to the Food and Drug Administration that included external control data.

Real-world evidence to support regulatory decision-making for medicines: Considerations for external control arms.

Randomized clinical trials (RCTs) are the gold standard in producing clinical evidence of efficacy and safety of medical interventions. More recently, a new paradigm is emerging-specifically within the context of preauthorization regulatory decision-making-for some novel uses of real-world evidence (RWE) from a variety of real-world data (RWD) sources to answer certain clinical questions. Traditionally reserved for rare diseases and other special circumstances, external controls (eg, historical controls) are recognized as a possible type of control arm for single-arm trials. However, creating and analyzing an external control arm using RWD can be challenging since design and analytics may not fully control for all systematic differences (biases). Nonetheless, certain biases can be attenuated using appropriate design and analytical approaches. The main objective of this paper is to improve the scientific rigor in the generation of external control arms using RWD. Here we (a) discuss the rationale and regulatory circumstances appropriate for external control arms, (b) define different types of external control arms, and (c) describe study design elements and approaches to mitigate certain biases in external control arms. This manuscript received endorsement from the International Society for Pharmacoepidemiology (ISPE).

Improving power for rare-variant tests by integrating external controls.

Due to the drop in sequencing cost, the number of sequenced genomes is increasing rapidly. To improve power of rare-variant tests, these sequenced samples could be used as external control samples in addition to control samples from the study itself. However, when using external controls, possible batch effects due to the use of different sequencing platforms or genotype calling pipelines can dramatically increase type I error rates. To address this, we propose novel summary statistics based single and gene- or region-based rare-variant tests that allow the integration of external controls while controlling for type I error. Our approach is based on the insight that batch effects on a given variant can be assessed by comparing odds ratio estimates using internal controls only vs. using combined control samples of internal and external controls. From simulation experiments and the analysis of data from age-related macular degeneration and type 2 diabetes studies, we demonstrate that our method can substantially improve power while controlling for type I error rate.

Reassessing the efficacy of bevacizumab in newly diagnosed glioblastoma: A systematic review and external pseudodata-based analysis.

Background: First-line use of bevacizumab for glioblastoma (GBM) was evaluated in 2 phase 3 randomized controlled trials (RCT), demonstrating an impact on progression-free survival but not overall survival (OS). However, the crossover events of these trials raised concerns regarding the reliability of this latter analysis. In this study, we conducted an external control-based reassessment of the bevacizumab efficacy in newly diagnosed GBM (ndGBM) against the standard Stupp protocol. Methods: A systematic review of the literature was conducted to identify the phase 3 RCTs in ndGBM incorporating the Stupp protocol as an arm. For the selected studies, we extracted individual patient survival pseudodata of the Stupp protocol arm by digitizing the Kaplan-Meier plots. A comprehensive pipeline was established to select suitable control studies as external benchmarks. Results: Among the 13 identified studies identified in our systematic review, 4 studies resulted as comparable with the AVAglio trial and 2 with the RTOG 0825. Pooled individual patient pseudodata analysis showed no differences in terms of OS when bevacizumab was added to the Stupp protocol. Conclusions: The external-controlled-based reassessment of the bevacizumab treatment in ndGBM confirmed its lack of efficacy in extending OS. Our study includes a summary table of individual patient survival pseudodata from all phase 3 RCTs in ndGBM employing the Stupp protocol and provides a pipeline that offers comprehensive guidance for conducting external control-based assessments in ndGBM.

Considerations on Design and Analysis of External Control in Pediatric Oncology.

Pediatric cancer consists of a diverse group of rare diseases. Due to limited patient populations, standard randomized and controlled trials are often infeasible. As a result, single-arm trials are common in pediatric oncology and the use of external controls is often desirable or necessary to help generate actionable evidence and contextualize trial results. In this paper, we illustrate unique features in pediatric oncology clinical trials and describe their impact on the use of external controls. Various types of relevant external control data sources are described in terms of their utility and drawbacks. Statistical methodologies and design implications with external control are discussed. Two recent case studies using external controls to support pediatric oncology drug development are described in detail.

A diagnostic phase III/IV seamless design to investigate the diagnostic accuracy and clinical effectiveness using the example of HEDOS and HEDOS II.

The development process of medical devices can be streamlined by combining different study phases. Here, for a diagnostic medical device, we present the combination of confirmation of diagnostic accuracy (phase III) and evaluation of clinical effectiveness regarding patient-relevant endpoints (phase IV) using a seamless design. This approach is used in the Thyroid HEmorrhage DetectOr Study (HEDOS & HEDOS II) investigating a post-operative hemorrhage detector named ISAR-M THYRO® in patients after thyroid surgery. Data from the phase III trial are reused as external controls in the control group of the phase IV trial. An unblinded interim analysis is planned between the two study stages which includes a recalculation of the sample size for the phase IV part after completion of the first stage of the seamless design. The study concept presented here is the first seamless design proposed in the field of diagnostic studies. Hence, the aim of this work is to emphasize the statistical methodology as well as feasibility of the proposed design in relation to the planning and implementation of the seamless design. Seamless designs can accelerate the overall trial duration and increase its efficiency in terms of sample size and recruitment. However, careful planning addressing numerous methodological and procedural challenges is necessary for successful implementation as well as agreement with regulatory bodies.

Clinical Trials for Wolfram Syndrome Neurodegeneration: Novel Design, Endpoints, and Analysis Models.

Objective: Wolfram syndrome, an ultra-rare condition, currently lacks effective treatment options. The rarity of this disease presents significant challenges in conducting clinical trials, particularly in achieving sufficient statistical power (e.g., 80%). The objective of this study is to propose a novel clinical trial design based on real-world data to reduce the sample size required for conducting clinical trials for Wolfram syndrome. Methods: We propose a novel clinical trial design with three key features aimed at reducing sample size and improve efficiency: (i) Pooling historical/external controls from a longitudinal observational study conducted by the Washington University Wolfram Research Clinic. (ii) Utilizing run-in data to estimate model parameters. (iii) Simultaneously tracking treatment effects in two endpoints using a multivariate proportional linear mixed effects model. Results: Comprehensive simulations were conducted based on real-world data obtained through the Wolfram syndrome longitudinal observational study. Our simulations demonstrate that this proposed design can substantially reduce sample size requirements. Specifically, with a bivariate endpoint and the inclusion of run-in data, a sample size of approximately 30 per group can achieve over 80% power, assuming the placebo progression rate remains consistent during both the run-in and randomized periods. In cases where the placebo progression rate varies, the sample size increases to approximately 50 per group. Conclusions: For rare diseases like Wolfram syndrome, leveraging existing resources such as historical/external controls and run-in data, along with evaluating comprehensive treatment effects using bivariate/multivariate endpoints, can significantly expedite the development of new drugs.

Justifying the source of external comparators in single-arm oncology health technology submissions: a review of NICE and PBAC assessments.

Background: The drive to expedite patient access for diseases with high unmet treatment needs has come with an increasing use of single-arm trials (SATs), especially in oncology. However, the lack of control arms in such trials creates challenges to assess and demonstrate comparative efficacy. External control (EC) arms can be used to bridge this gap, with various types of sources available to obtain relevant data. Objective: To examine the source of ECs in single-arm oncology health technology assessment (HTA) submissions to the National Institute for Health and Care Excellence (NICE) and the Pharmaceutical Benefits Advisory Committee (PBAC) and how this selection was justified by manufacturers and assessed by the respective HTA body. Methods: Single-arm oncology HTA submission reports published by NICE (England) and PBAC (Australia) from January 2011 to August 2021 were reviewed, with data qualitatively synthesized to identify themes. Results: Forty-eight oncology submissions using EC arms between 2011 and 2021 were identified, with most submissions encompassing blood and bone marrow cancers (52%). In HTA submissions to NICE and PBAC, the EC arm was typically constructed from a combination of data sources, with the company's justification in data source selection infrequently provided (PBAC [2 out of 19]; NICE [6 out of 29]), although this lack of justification was not heavily criticized by either HTA body. Conclusion: Although HTA bodies such as NICE and PBAC encourage that EC source justification should be provided in submissions, this review found that this is not typically implemented in practice. Guidance is needed to establish best practices as to how EC selection should be documented in HTA submissions.

On the use of non-concurrent controls in platform trials: a scoping review.

BACKGROUND: Platform trials gained popularity during the last few years as they increase flexibility compared to multi-arm trials by allowing new experimental arms entering when the trial already started. Using a shared control group in platform trials increases the trial efficiency compared to separate trials. Because of the later entry of some of the experimental treatment arms, the shared control group includes concurrent and non-concurrent control data. For a given experimental arm, non-concurrent controls refer to patients allocated to the control arm before the arm enters the trial, while concurrent controls refer to control patients that are randomised concurrently to the experimental arm. Using non-concurrent controls can result in bias in the estimate in case of time trends if the appropriate methodology is not used and the assumptions are not met. METHODS: We conducted two reviews on the use of non-concurrent controls in platform trials: one on statistical methodology and one on regulatory guidance. We broadened our searches to the use of external and historical control data. We conducted our review on the statistical methodology in 43 articles identified through a systematic search in PubMed and performed a review on regulatory guidance on the use of non-concurrent controls in 37 guidelines published on the EMA and FDA websites. RESULTS: Only 7/43 of the methodological articles and 4/37 guidelines focused on platform trials. With respect to the statistical methodology, in 28/43 articles, a Bayesian approach was used to incorporate external/non-concurrent controls while 7/43 used a frequentist approach and 8/43 considered both. The majority of the articles considered a method that downweights the non-concurrent control in favour of concurrent control data (34/43), using for instance meta-analytic or propensity score approaches, and 11/43 considered a modelling-based approach, using regression models to incorporate non-concurrent control data. In regulatory guidelines, the use of non-concurrent control data was considered critical but was deemed acceptable for rare diseases in 12/37 guidelines or was accepted in specific indications (12/37). Non-comparability (30/37) and bias (16/37) were raised most often as the general concerns with non-concurrent controls. Indication specific guidelines were found to be most instructive. CONCLUSIONS: Statistical methods for incorporating non-concurrent controls are available in the literature, either by means of methods originally proposed for the incorporation of external controls or non-concurrent controls in platform trials. Methods mainly differ with respect to how the concurrent and non-concurrent data are combined and temporary changes handled. Regulatory guidance for non-concurrent controls in platform trials are currently still limited.