Propensity score-incorporated adaptive design approaches when incorporating real-world data.

The propensity score-integrated composite likelihood (PSCL) method is one method that can be utilized to design and analyze an application when real-world data (RWD) are leveraged to augment a prospectively designed clinical study. In the PSCL, strata are formed based on propensity scores (PS) such that similar subjects in terms of the baseline covariates from both the current study and RWD sources are placed in the same stratum, and then composite likelihood method is applied to down-weight the information from the RWD. While PSCL was originally proposed for a fixed design, it can be extended to be applied under an adaptive design framework with the purpose to either potentially claim an early success or to re-estimate the sample size. In this paper, a general strategy is proposed due to the feature of PSCL. For the possibility of claiming early success, Fisher's combination test is utilized. When the purpose is to re-estimate the sample size, the proposed procedure is based on the test proposed by Cui, Hung, and Wang. The implementation of these two procedures is demonstrated via an example.

A propensity score-integrated approach for leveraging external data in a randomized controlled trial with time-to-event endpoints.

In a randomized controlled trial with time-to-event endpoint, some commonly used statistical tests to test for various aspects of survival differences, such as survival probability at a fixed time point, survival function up to a specific time point, and restricted mean survival time, may not be directly applicable when external data are leveraged to augment an arm (or both arms) of an RCT. In this paper, we propose a propensity score-integrated approach to extend such tests when external data are leveraged. Simulation studies are conducted to evaluate the operating characteristics of three propensity score-integrated statistical tests, and an illustrative example is given to demonstrate how these proposed procedures can be implemented.

New approaches for testing non-inferiority for three-arm trials with Poisson distributed outcomes.

With the availability of limited resources, innovation for improved statistical method for the design and analysis of randomized controlled trials (RCTs) is of paramount importance for newer and better treatment discovery for any therapeutic area. Although clinical efficacy is almost always the primary evaluating criteria to measure any beneficial effect of a treatment, there are several important other factors (e.g., side effects, cost burden, less debilitating, less intensive, etc.), which can permit some less efficacious treatment options favorable to a subgroup of patients. This leads to non-inferiority (NI) testing. The objective of NI trial is to show that an experimental treatment is not worse than an active reference treatment by more than a pre-specified margin. Traditional NI trials do not include a placebo arm for ethical reason; however, this necessitates stringent and often unverifiable assumptions. On the other hand, three-arm NI trials consisting of placebo, reference, and experimental treatment, can simultaneously test the superiority of the reference over placebo and NI of experimental treatment over the reference. In this article, we proposed both novel Frequentist and Bayesian procedures for testing NI in the three-arm trial with Poisson distributed count outcome. RCTs with count data as the primary outcome are quite common in various disease areas such as lesion count in cancer trials, relapses in multiple sclerosis, dermatology, neurology, cardiovascular research, adverse event count, etc. We first propose an improved Frequentist approach, which is then followed by it's Bayesian version. Bayesian methods have natural advantage in any active-control trials, including NI trial when substantial historical information is available for placebo and established reference treatment. In addition, we discuss sample size calculation and draw an interesting connection between the two paradigms.

Adaptive designs for best treatment identification with top-two Thompson sampling and acceleration.

We consider outcome adaptive phase II or phase II/III trials to identify the best treatment for further development. Different from many other multi-arm multi-stage designs, we borrow approaches for the best arm identification in multi-armed bandit (MAB) approaches developed for machine learning and adapt them for clinical trial purposes. The best arm identification in MAB focuses on the error rate of identification at the end of the trial, but we are also interested in the cumulative benefit of trial patients, for example, the frequency of patients treated with the best treatment. In particular, we consider Top-Two Thompson Sampling (TTTS) and propose an acceleration approach for better performance in drug development scenarios in which the sample size is much smaller than that considered in machine learning applications. We also propose a variant of TTTS (TTTS2) which is simpler, easier for implementation, and has comparable performance in small sample settings. An extensive simulation study was conducted to evaluate the performance of the proposed approach in multiple typical scenarios in drug development.

Principled leveraging of external data in the evaluation of diagnostic devices via the propensity score-integrated composite likelihood approach.

In the area of diagnostics, it is common practice to leverage external data to augment a traditional study of diagnostic accuracy consisting of prospectively enrolled subjects to potentially reduce the time and/or cost needed for the performance evaluation of an investigational diagnostic device. However, the statistical methods currently being used for such leveraging may not clearly separate study design and outcome data analysis, and they may not adequately address possible bias due to differences in clinically relevant characteristics between the subjects constituting the traditional study and those constituting the external data. This paper is intended to draw attention in the field of diagnostics to the recently developed propensity score-integrated composite likelihood approach, which originally focused on therapeutic medical products. This approach applies the outcome-free principle to separate study design and outcome data analysis and can mitigate bias due to imbalance in covariates, thereby increasing the interpretability of study results. While this approach was conceived as a statistical tool for the design and analysis of clinical studies for therapeutic medical products, here, we will show how it can also be applied to the evaluation of sensitivity and specificity of an investigational diagnostic device leveraging external data. We consider two common scenarios for the design of a traditional diagnostic device study consisting of prospectively enrolled subjects, which is to be augmented by external data. The reader will be taken through the process of implementing this approach step-by-step following the outcome-free principle that preserves study integrity.

Conditional borrowing external data to establish a hybrid control arm in randomized clinical trials.

Utilizing external data from the real world, including data from historical clinical trials, has received increasing interest in drug development. The use of external data to support drug evaluation in clinical trials has mainly been through using various matching methods for baseline characteristics to form external control arms in single-arm trials or to augment control arms of randomized controlled trials in hybrid approaches. However, matching the baseline characteristics between the trial and the external subjects can only guarantee comparability on the level of baseline characteristics. Differences in outcomes between the two data sources may still exist due to contemporaneous and operational characteristics. Similarity between the outcomes in the trial control and the external subjects with similar baseline characteristics can be critical in leveraging the external subjects in the clinical trials. In this paper, a resampling method for augmenting control arms in randomized controlled trials is proposed under the conditional borrowing framework. The new method establishes empirical distributions for the hazard ratio in outcomes between the external and trial control subjects. The borrowing decision is then derived from this empirical distribution using a measure of similarity. Once the borrowing decision is established, the borrowing weights for the external subjects, based on the similarity measure, are incorporated in the weighted partial likelihood to evaluate the treatment effect. The operating characteristics of the hybrid control arm, under both the conditional borrowing and unconditional borrowing frameworks, are evaluated. Simulation is conducted to evaluate Type I error, bias, and power. An illustrative example using simulated data is also presented.

A novel approach to augment single-arm clinical studies with real-world data.

In this paper, we develop a methodology for leveraging real-world data into single-arm clinical trial studies. In recent years, the idea of augmenting randomized clinical trials data with real-world data has emerged as a particularly attractive technique for health organizations and drug developers to accelerate the drug development process. Major regulatory authorities such as the Food and Drug Administration and European Medicines Agency have recognized the potential of utilizing real-world data and are advancing toward making regulatory decisions based on real-world evidence. Several statistical methods have been developed in recent years for borrowing data from real-world sources such as electronic health records, product and disease registries, as well as claims and billing data. We propose a novel approach to augment single-arm clinical trials with the real-world data derived from single or multiple data sources. Furthermore, we illustrate the proposed method in the presence of missing data and conduct simulation studies to evaluate its performance in diverse settings.

Propensity score-integrated approach to survival analysis: leveraging external evidence in single-arm studies.

External data, referred to as data external to the traditional clinical study being planned, include but are not limited to real-world data (RWD) and data collected from clinical studies being conducted in the past or in other countries. The external data are sometimes leveraged to augment a single-arm, prospectively designed study when appropriate. In such an application, recently developed propensity score-integrated approaches including PSPP and PSCL can be used for study design and data analysis when the clinical outcomes are binary or continuous. In this paper, the propensity score-integrated Kaplan-Meier (PSKM) method is proposed for a similar situation but the outcome of interest is time-to-event. The propensity score methodology is used to select external subjects that are similar to those in the current study in terms of baseline covariates and to stratify the selected subjects from both data sources into more homogeneous strata. The stratum-specific PSKM estimators are obtained based on all subjects in the stratum with the external data being down-weighted, and then these estimators are combined to obtain an overall PSKM estimator. A simulation study is conducted to assess the performance of the PSKM method, and an illustrative example is presented to demonstrate how to implement the proposed method.

Leverage multiple real-world data sources in single-arm medical device clinical studies.

The interest in utilizing real-world data (RWD) has been considerably increasing in medical product development and evaluation. With proper usage and analysis of high-quality real-world data, real-world evidence (RWE) can be generated to inform regulatory and healthcare decision-making. This paper proposes a study design and data analysis approach for a prospective, single-arm clinical study that is supplemented with patients from multiple real-world data sources containing patient-level covariate and outcome data. After the amount of information to be borrowed from each real-world data source is determined, the propensity score-integrated composite likelihood method is applied to obtain an estimate of the parameter of interest based on data from the prospective clinical study and this real-world data source. This method is applied to each real-world data source. The final estimate of the parameter of interest is then obtained by taking a weighted average of all these estimates. The performance of the proposed approach is evaluated via a simulation study. A hypothetical example is presented to illustrate how to implement the proposed approach.

Propensity score-integrated power prior approach for augmenting the control arm of a randomized controlled trial by incorporating multiple external data sources.

In this paper, a propensity score-integrated power prior approach is developed to augment the control arm of a two-arm randomized controlled trial (RCT) with subjects from multiple external data sources such as real-world data (RWD) and historical clinical studies containing subject-level outcomes and covariates. The propensity scores for the subjects in the external data sources versus the subjects in the RCT are first estimated, and then subjects are placed in different strata based on their estimated propensity scores. Within each propensity score stratum, a power prior is formulated with the information contributed by the external data sources, and Bayesian inference on the treatment effect is obtained. The proposed approach is implemented under the two-stage study design framework utilizing the outcome-free principle to ensure the integrity of a study. An illustrative example is provided to demonstrate the implementation of the proposed approach.

Ensuring exchangeability in data-based priors for a Bayesian analysis of clinical trials.

In many orphan diseases and pediatric indications, the randomized controlled trials may be infeasible because of their size, duration, and cost. Leveraging information on the control through a prior can potentially reduce sample size. However, unless an objective prior is used to impose complete ignorance for the parameter being estimated, it results in biased estimates and inflated type-I error. Hence, it is essential to assess both the confirmatory and supplementary knowledge available during the construction of the prior to avoid "cherry-picking" advantageous information. For this purpose, propensity score methods are employed to minimize selection bias by weighting supplemental control subjects according to their similarity in terms of pretreatment characteristics to the subjects in the current trial. The latter can be operationalized through a proposed measure of overlap in propensity-score distributions. In this paper, we consider single experimental arm in the current trial and the control arm is completely borrowed from the supplemental data. The simulation experiments show that the proposed method reduces prior and data conflict and improves the precision of the of the average treatment effect.

Estimands in observational studies: Some considerations beyond ICH E9 (R1).

The document ICH E9 (R1) has brought much attention to the concept of estimand in the clinical trials community. ICH stands for International Conference for Harmonization. In this article, we draw attention to one facet of estimand that is not discussed in that document but is crucial in the context of observational studies, namely weighting for covariate balance. How weighting schemes are connected to estimand, or more specifically to one of its five attributes identified in ICH E9 (R1), the attribute of population, is illustrated using the Rubin Causal Model. Three estimands are examined from both theoretical and practical perspectives. Factors that may be considered in choosing among these estimands are discussed.

Leveraging real-world evidence for determining performance goals for medical device studies.

Performance goals are numerical target values pertaining to effectiveness or safety endpoints in single-arm medical device clinical studies. Typically, performance goals are determined at the planning stage of the investigational study under consideration based on summarized outcome information from existing relevant clinical trials. In recent years, there is a growing interest in leveraging real-world evidence in medical product development. In this article, we introduce a new method for proposing performance goals by leveraging real-world evidence. The method applies entropy balancing to address possible patient dissimilarities between the study's target patient population and existing real-world patients, and can take into account operation differences between clinical studies and real-world clinical practice. An illustrative example is provided to demonstrate how to implement the proposed method for performance goal determination while leveraging real-world evidence.

Bayesian approaches to benefit-risk assessment for diagnostic tests.

Benefit-risk assessment plays an important role in the evaluation of medical devices. Unlike the therapeutic devices, the diagnostic tests usually affect patient life indirectly since subsequent therapeutic treatment interventions (such as proper treatment in time, further examination or test, no action, etc.) will depend on correct diagnosis and monitoring of the disease status. A benefit-risk score using statistical models by integrating the information from benefit (true positive and true negative) and risk (false positive and false negative) for diagnostic tests with binary outcomes (i.e., positive and negative) will help evaluation of the utility and the uncertainty of a particular diagnostic device. In this paper, we develop two types of Bayesian models with conjugate priors for constructing the benefit-risk (BR) measures with corresponding credible intervals, one based on a Multinomial model with Dirichlet prior, and the other based on independent Binomial models with independent Beta priors. We then propose a Bayesian power prior model to incorporate the historical data or the real-world data (RWD). Both the fixed and random power prior parameters are considered for Bayesian borrowing. We evaluate the performance of the methods by simulations and illustrate their implementation using a real example.

Dynamic borrowing from a single prior data source using the conditional power prior.

The conditional power prior is a popular method to borrow information from a single prior data source. The amount of borrowing is controlled by the power parameter which is fixed before running the new study. However, fixing this parameter before running a new study is often difficult and may be unwise because if the outcomes in the current study are much different from the prior data outcomes, the power parameter cannot be changed to reflect a more appropriate degree of borrowing. On the other hand, treating the power parameter as a random variable to be updated via Bayes theorem may relinquish control over how much to borrow in cases where regulatory oversight recommends a conservative approach.Previous authors have determined the power parameter at the end of the current study based on "stochastic" similarity in the outcomes between the current study and the prior data. In this paper, we introduce some modifications to those methods. First, we determine the power parameter based on similarity between a percentage of the current study outcome data available at an interim look and the prior outcome data. This may limit potential for operational bias resulting from the determination of the power parameter after the current study is complete. Next, we introduce a new measure of similarity between the current (interim) and prior data that limits similarity by a pre-specified clinical margin. The proposed clinical similarity region may be readily understood by clinicians who need to assess when such borrowing is clinically appropriate. Through simulations, we show that our approach has low bias and good power, while reducing type I error rate in areas outside of the "similarity region". An example of a hypothetical medical device study illustrates its potential use in practice.

Evaluating medical device adverse event signals using a likelihood ratio test method.

Signal detection methods have been used extensively in post-market surveillance to identify elevated risks of adverse events. However, these statistical methods have not been widely used in detecting AE signals for medical devices. In this paper, we focused on the use of a likelihood ratio test (LRT)-based method in identifying adverse event (AE) signals associated with left ventricular assist devices (LVADs) using Medical Device Reporting (MDR) data. Among 110,927 adverse event entries identified in MDR data for LVADs, the LRT method detected 18 AE signals which included seven bleeding-related AEs such as hemolysis, thrombosis, hematuria, thrombus, blood loss, and hemorrhage. The LRT method was also applied to longitudinal data from 2007 to 2019 where a monotone alpha-spending function was used to ensure the control of type I error at each look and overall for trend analysis. Furthermore, the LRT method was compared to proportional reporting ratios (PRRs), Bayesian confidence propagation neural network (BCPNN), and simplified Bayes methods and found to be the most conservative method when examining the total number of detected signals, given its ability to control type-I error and the false discovery rate.

Application of a likelihood ratio test based method for safety signal detection to left ventricular assist devices.

Effective post-market safety surveillance of medical devices is critical for public health. However, many current statistical methods for safety signal detection do not control for type I error when assessing multiple device and adverse event (AE) combinations. This can result in increased false signals, underscoring the need for more robust statistical methods. Moreover, the duration of medical device use can be an important factor to consider in safety surveillance. In this study, we adapted a likelihood ratio test (LRT) based method, which was initially developed and applied to drugs, to identify safety signals for left ventricular assist devices (LVAD). Among patients with chronic, advanced left ventricular failure, we analyzed AE data for HeartWare and HeartMate II patients during a two-year period and further incorporated person-years (henceforth exposure-time). The novel modified LRT and conventional Z-test with p-values adjusted by the Benjamini-Hochberg (BH) procedure were used to explore safety signals by comparing HeartWare and HeartMate II patients in the presence of multiple adverse events. Both methods identified greater incidence of stroke among HeartWare as compared to HeartMate II patients without exposure-time (p = .025 for LRT and p = .027 for Z-test with BH) and with exposure-time (p = .002 for LRT and p = .005 for Z-test with BH). By using improved statistical methods for safety signal detection, potential safety issues can be identified and addressed in a more timely manner to enhance public safety.

Evaluation of diagnostic tests for low prevalence diseases: a statistical approach for leveraging real-world data to accelerate the study.

The evaluation of diagnostic tests usually involves statistical inference for its sensitivity. As sensitivity is defined as the probability that the test result will be positive when the target condition is present, the key study design consideration of sample size is the determination of the number of subjects with the target condition such that the estimation has adequate precision, or the hypothesis testing has adequate power. Traditionally, one may rely on prospective screening of subjects to obtain the required sample size, which means that if the prevalence of the disease is very low, a large number of subjects would need to be screened, increasing the study duration and cost. In this paper, we consider the possibility of substantially reducing the length and cost of a clinical study by leveraging subjects from a real-world data (RWD) source, focusing specifically on the diagnostic test for the cancer of interest. Using the propensity score methodology, we developed a procedure which ensures that the real-world subjects being leveraged are similar to their prospectively enrolled counterparts, thereby making the leveraging more justified. The procedure allows the down-weighting of the real-world subjects, which can be achieved by either using a Frequentist's method based on the composite likelihood or a Bayesian method based on the power prior. The proposed approach can be applied to the evaluation of any diagnostic test and it is not limited to the current clinical study regarding a cancer diagnostic test. Notably, this paper is in close alignment with a recently released draft framework by the Medical Device Innovation Consortium (MDIC) on real-world clinical evidence and in vitro diagnostics, being a showcase of appropriately leveraging real-world data in diagnostic test evaluation for diseases with low prevalence to support regulatory decision-making.

A Bayesian approach to benefit-risk assessment in clinical studies with longitudinal data.

Chuang-Stein et al. proposed a method for benefit-risk assessment by formulating a five-category multinomial random variable with the first four categories as a combination of benefit and risk, and the fifth category to include subjects who withdraw from study. In this article, we subdivide the single withdrawal category into four sub-categories to consider withdrawal for different reasons. To analyze eight-category data, we propose a two-level multivariate-Dirichlet Model to identify benefit-risk measures at the population level. For individual benefit-risk, we use a log-odds ratio model with Dirichlet process prior. Two methods are applied to a hypothetical clinical trial data for illustration.

Propensity score-integrated composite likelihood approach for augmenting the control arm of a randomized controlled trial by incorporating real-world data.

In this paper, a propensity score-integrated composite likelihood (PSCL) approach is developed for cases in which the control arm of a two-arm randomized controlled trial (RCT) (treated vs control) is augmented with patients from real-world data (RWD) containing both clinical outcomes and covariates at the patient-level. RWD patients who were treated with the same therapy as the control arm of the RCT are considered for the augmentation. The PSCL approach first estimates the propensity score for every patient as the probability of the patient being in the RCT rather than the RWD, and then stratifies all patients into strata based on the estimated propensity scores. Within each propensity score stratum, a composite likelihood function is specified and utilized to down-weight the information contributed by the RWD source. Estimates of the stratum-specific parameters are obtained by maximizing the composite likelihood function. These stratum-specific estimates are then combined to obtain an overall population-level estimate of the parameter of interest. The performance of the proposed approach is evaluated via a simulation study. A hypothetical two-arm RCT and a hypothetical RWD source are used to illustrate the implementation of the proposed approach.