Point estimation for adaptive trial designs I: A methodological review.

Recent FDA guidance on adaptive clinical trial designs defines bias as "a systematic tendency for the estimate of treatment effect to deviate from its true value," and states that it is desirable to obtain and report estimates of treatment effects that reduce or remove this bias. The conventional end-of-trial point estimates of the treatment effects are prone to bias in many adaptive designs, because they do not take into account the potential and realized trial adaptations. While much of the methodological developments on adaptive designs have tended to focus on control of type I error rates and power considerations, in contrast the question of biased estimation has received relatively less attention. This article is the first in a two-part series that studies the issue of potential bias in point estimation for adaptive trials. Part I provides a comprehensive review of the methods to remove or reduce the potential bias in point estimation of treatment effects for adaptive designs, while part II illustrates how to implement these in practice and proposes a set of guidelines for trial statisticians. The methods reviewed in this article can be broadly classified into unbiased and bias-reduced estimation, and we also provide a classification of estimators by the type of adaptive design. We compare the proposed methods, highlight available software and code, and discuss potential methodological gaps in the literature.

Point estimation for adaptive trial designs II: Practical considerations and guidance.

In adaptive clinical trials, the conventional end-of-trial point estimate of a treatment effect is prone to bias, that is, a systematic tendency to deviate from its true value. As stated in recent FDA guidance on adaptive designs, it is desirable to report estimates of treatment effects that reduce or remove this bias. However, it may be unclear which of the available estimators are preferable, and their use remains rare in practice. This article is the second in a two-part series that studies the issue of bias in point estimation for adaptive trials. Part I provided a methodological review of approaches to remove or reduce the potential bias in point estimation for adaptive designs. In part II, we discuss how bias can affect standard estimators and assess the negative impact this can have. We review current practice for reporting point estimates and illustrate the computation of different estimators using a real adaptive trial example (including code), which we use as a basis for a simulation study. We show that while on average the values of these estimators can be similar, for a particular trial realization they can give noticeably different values for the estimated treatment effect. Finally, we propose guidelines for researchers around the choice of estimators and the reporting of estimates following an adaptive design. The issue of bias should be considered throughout the whole lifecycle of an adaptive design, with the estimation strategy prespecified in the statistical analysis plan. When available, unbiased or bias-reduced estimates are to be preferred.

Practical guidance for planning resources required to support publicly-funded adaptive clinical trials.

Adaptive designs are a class of methods for improving efficiency and patient benefit of clinical trials. Although their use has increased in recent years, research suggests they are not used in many situations where they have potential to bring benefit. One barrier to their more widespread use is a lack of understanding about how the choice to use an adaptive design, rather than a traditional design, affects resources (staff and non-staff) required to set-up, conduct and report a trial. The Costing Adaptive Trials project investigated this issue using quantitative and qualitative research amongst UK Clinical Trials Units. Here, we present guidance that is informed by our research, on considering the appropriate resourcing of adaptive trials. We outline a five-step process to estimate the resources required and provide an accompanying costing tool. The process involves understanding the tasks required to undertake a trial, and how the adaptive design affects them. We identify barriers in the publicly funded landscape and provide recommendations to trial funders that would address them. Although our guidance and recommendations are most relevant to UK non-commercial trials, many aspects are relevant more widely.

A practical guide to sample size calculations: Installation of the app SampSize.

In 2016 we published three articles in Pharmaceutical Statistics that gave a practical guide to sample size calculations. In each of the articles there were instructions on how to obtain the App SampSize. This short communication updates these instructions and highlights the updates and added functionality to the App.

Costs and staffing resource requirements for adaptive clinical trials: quantitative and qualitative results from the Costing Adaptive Trials project.

BACKGROUND: Adaptive designs offer great promise in improving the efficiency and patient-benefit of clinical trials. An important barrier to further increased use is a lack of understanding about which additional resources are required to conduct a high-quality adaptive clinical trial, compared to a traditional fixed design. The Costing Adaptive Trials (CAT) project investigated which additional resources may be required to support adaptive trials. METHODS: We conducted a mock costing exercise amongst seven Clinical Trials Units (CTUs) in the UK. Five scenarios were developed, derived from funded clinical trials, where a non-adaptive version and an adaptive version were described. Each scenario represented a different type of adaptive design. CTU staff were asked to provide the costs and staff time they estimated would be needed to support the trial, categorised into specified areas (e.g. statistics, data management, trial management). This was calculated separately for the non-adaptive and adaptive version of the trial, allowing paired comparisons. Interviews with 10 CTU staff who had completed the costing exercise were conducted by qualitative researchers to explore reasons for similarities and differences. RESULTS: Estimated resources associated with conducting an adaptive trial were always (moderately) higher than for the non-adaptive equivalent. The median increase was between 2 and 4% for all scenarios, except for sample size re-estimation which was 26.5% (as the adaptive design could lead to a lengthened study period). The highest increase was for statistical staff, with lower increases for data management and trial management staff. The percentage increase in resources varied across different CTUs. The interviews identified possible explanations for differences, including (1) experience in adaptive trials, (2) the complexity of the non-adaptive and adaptive design, and (3) the extent of non-trial specific core infrastructure funding the CTU had. CONCLUSIONS: This work sheds light on additional resources required to adequately support a high-quality adaptive trial. The percentage increase in costs for supporting an adaptive trial was generally modest and should not be a barrier to adaptive designs being cost-effective to use in practice. Informed by the results of this research, guidance for investigators and funders will be developed on appropriately resourcing adaptive trials.

Mapping From Visual Acuity to EQ-5D, EQ-5D With Vision Bolt-On, and VFQ-UI in Patients With Macular Edema in the LEAVO Trial.

OBJECTIVES: Mappings to convert clinical measures to preference-based measures of health such as the EQ-5D-3L are sometimes required in cost-utility analyses. We developed mappings to convert best-corrected visual acuity (BCVA) to the EQ-5D-3L, the EQ-5D-3L with a vision bolt-on (EQ-5D V), and the Visual Functioning Questionnaire-Utility Index (VFQ-UI) in patients with macular edema caused by central retinal vein occlusion. METHODS: We used data from Lucentis, Eylea, Avastin in vein occlusion (LEAVO), which is a phase-3 randomized controlled trial comparing ranibizumab, aflibercept, and bevacizumab in 463 patients with observations at 6 time points. We estimated adjusted limited dependent variable mixture models consisting of 1 to 4 distributions (components) using BCVA in each eye, age, and sex to predict utility within the components and BCVA as a determinant of component membership. We compared model fit using mean error, mean absolute error, root mean square error, Akaike information criteria, Bayesian information criteria, and visual inspection of mean predicted and observed utilities and cumulative distribution functions. RESULTS: Mean utility scores were 0.82 for the EQ-5D-3L, 0.79 for the EQ-5D V, and 0.88 for the VFQ-UI. The best-fitting models for the EQ-5D and EQ-5D V had 2 components (with means of approximately 0.44 and 0.85), and the best-fitting model for VFQ-UI had 3 components (with means of approximately 0.95, 0.74, and 0.90). CONCLUSIONS: Models with multiple components better predict utility than those with single components. This article provides a valuable addition to the literature, in which previous mappings in visual acuity have been limited to linear regressions, resulting in unfounded assumptions about the distribution of the dependent variable.

A Review of Clinical Trials With an Adaptive Design and Health Economic Analysis.

OBJECTIVE: An adaptive design uses data collected as a clinical trial progresses to inform modifications to the trial. Hence, adaptive designs and health economics aim to facilitate efficient and accurate decision making. Nevertheless, it is unclear whether the methods are considered together in the design, analysis, and reporting of trials. This review aims to establish how health economic outcomes are used in the design, analysis, and reporting of adaptive designs. METHODS: Registered and published trials up to August 2016 with an adaptive design and health economic analysis were identified. The use of health economics in the design, analysis, and reporting was assessed. Summary statistics are presented and recommendations formed based on the research team's experiences and a practical interpretation of the results. RESULTS: Thirty-seven trials with an adaptive design and health economic analysis were identified. It was not clear whether the health economic analysis accounted for the adaptive design in 17/37 trials where this was thought necessary, nor whether health economic outcomes were used at the interim analysis for 18/19 of trials with results. The reporting of health economic results was suboptimal for the (17/19) trials with published results. CONCLUSIONS: Appropriate consideration is rarely given to the health economic analysis of adaptive designs. Opportunities to use health economic outcomes in the design and analysis of adaptive trials are being missed. Further work is needed to establish whether adaptive designs and health economic analyses can be used together to increase the efficiency of health technology assessments without compromising accuracy.

Adaptive designs in clinical trials: why use them, and how to run and report them.

Adaptive designs can make clinical trials more flexible by utilising results accumulating in the trial to modify the trial's course in accordance with pre-specified rules. Trials with an adaptive design are often more efficient, informative and ethical than trials with a traditional fixed design since they often make better use of resources such as time and money, and might require fewer participants. Adaptive designs can be applied across all phases of clinical research, from early-phase dose escalation to confirmatory trials. The pace of the uptake of adaptive designs in clinical research, however, has remained well behind that of the statistical literature introducing new methods and highlighting their potential advantages. We speculate that one factor contributing to this is that the full range of adaptations available to trial designs, as well as their goals, advantages and limitations, remains unfamiliar to many parts of the clinical community. Additionally, the term adaptive design has been misleadingly used as an all-encompassing label to refer to certain methods that could be deemed controversial or that have been inadequately implemented.We believe that even if the planning and analysis of a trial is undertaken by an expert statistician, it is essential that the investigators understand the implications of using an adaptive design, for example, what the practical challenges are, what can (and cannot) be inferred from the results of such a trial, and how to report and communicate the results. This tutorial paper provides guidance on key aspects of adaptive designs that are relevant to clinical triallists. We explain the basic rationale behind adaptive designs, clarify ambiguous terminology and summarise the utility and pitfalls of adaptive designs. We discuss practical aspects around funding, ethical approval, treatment supply and communication with stakeholders and trial participants. Our focus, however, is on the interpretation and reporting of results from adaptive design trials, which we consider vital for anyone involved in medical research. We emphasise the general principles of transparency and reproducibility and suggest how best to put them into practice.

Appropriate statistical methods for analysing partially nested randomised controlled trials with continuous outcomes: a simulation study.

BACKGROUND: In individually randomised trials we might expect interventions delivered in groups or by care providers to result in clustering of outcomes for participants treated in the same group or by the same care provider. In partially nested randomised controlled trials (pnRCTs) this clustering only occurs in one trial arm, commonly the intervention arm. It is important to measure and account for between-cluster variability in trial design and analysis. We compare analysis approaches for pnRCTs with continuous outcomes, investigating the impact on statistical inference of cluster sizes, coding of the non-clustered arm, intracluster correlation coefficient (ICCs), and differential variance between intervention and control arm, and provide recommendations for analysis. METHODS: We performed a simulation study assessing the performance of six analysis approaches for a two-arm pnRCT with a continuous outcome. These include: linear regression model; fully clustered mixed-effects model with singleton clusters in control arm; fully clustered mixed-effects model with one large cluster in control arm; fully clustered mixed-effects model with pseudo clusters in control arm; partially nested homoscedastic mixed effects model, and partially nested heteroscedastic mixed effects model. We varied the cluster size, number of clusters, ICC, and individual variance between the two trial arms. RESULTS: All models provided unbiased intervention effect estimates. In the partially nested mixed-effects models, methods for classifying the non-clustered control arm had negligible impact. Failure to account for even small ICCs resulted in inflated Type I error rates and over-coverage of confidence intervals. Fully clustered mixed effects models provided poor control of the Type I error rates and biased ICC estimates. The heteroscedastic partially nested mixed-effects model maintained relatively good control of Type I error rates, unbiased ICC estimation, and did not noticeably reduce power even with homoscedastic individual variances across arms. CONCLUSIONS: In general, we recommend the use of a heteroscedastic partially nested mixed-effects model, which models the clustering in only one arm, for continuous outcomes similar to those generated under the scenarios of our simulations study. However, with few clusters (3-6), small cluster sizes (5-10), and small ICC (≤0.05) this model underestimates Type I error rates and there is no optimal model.

Can emergency medicine research benefit from adaptive design clinical trials?

BACKGROUND: Adaptive design clinical trials use preplanned interim analyses to determine whether studies should be stopped or modified before recruitment is complete. Emergency medicine trials are well suited to these designs as many have a short time to primary outcome relative to the length of recruitment. We hypothesised that the majority of published emergency medicine trials have the potential to use a simple adaptive trial design. METHODS: We reviewed clinical trials published in three emergency medicine journals between January 2003 and December 2013. We determined the proportion that used an adaptive design as well as the proportion that could have used a simple adaptive design based on the time to primary outcome and length of recruitment. RESULTS: Only 19 of 188 trials included in the review were considered to have used an adaptive trial design. A total of 154/165 trials that were fixed in design had the potential to use an adaptive design. CONCLUSIONS: Currently, there seems to be limited uptake in the use of adaptive trial designs in emergency medicine despite their potential benefits to save time and resources. Failing to take advantage of adaptive designs could be costly to patients and research. It is recommended that where practical and logistical considerations allow, adaptive designs should be used for all emergency medicine clinical trials.

Recommendations for the analysis of individually randomised controlled trials with clustering in one arm - a case of continuous outcomes.

BACKGROUND: In an individually randomised controlled trial where the treatment is delivered by a health professional it seems likely that the effectiveness of the treatment, independent of any treatment effect, could depend on the skill, training or even enthusiasm of the health professional delivering it. This may then lead to a potential clustering of the outcomes for patients treated by the same health professional, but similar clustering may not occur in the control arm. Using four case studies, we aim to provide practical guidance and recommendations for the analysis of trials with some element of clustering in one arm. METHODS: Five approaches to the analysis of outcomes from an individually randomised controlled trial with clustering in one arm are identified in the literature. Some of these methods are applied to four case studies of completed randomised controlled trials with clustering in one arm with sample sizes ranging from 56 to 539. Results are obtained using the statistical packages R and Stata and summarised using a forest plot. RESULTS: The intra-cluster correlation coefficient (ICC) for each of the case studies was small (<0.05) indicating little dependence on the outcomes related to cluster allocations. All models fitted produced similar results, including the simplest approach of ignoring clustering for the case studies considered. CONCLUSIONS: A partially clustered approach, modelling the clustering in just one arm, most accurately represents the trial design and provides valid results. Modelling homogeneous variances between the clustered and unclustered arm is adequate in scenarios similar to the case studies considered. We recommend treating each participant in the unclustered arm as a single cluster. This approach is simple to implement in R and Stata and is recommended for the analysis of trials with clustering in one arm only. However, the case studies considered had small ICC values, limiting the generalisability of these results.

Practical guide to sample size calculations: an introduction.

A sample size justification is a vital step when designing any trial. However, estimating the number of participants required to give a meaningful result is not always straightforward. A number of components are required to facilitate a suitable sample size calculation. In this paper, the general steps are summarised for conducting sample size calculations with practical advice and guidance on how to utilise the app SampSize.

Practical guide to sample size calculations: non-inferiority and equivalence trials.

A sample size justification is a vital part of any trial design. However, estimating the number of participants required to give a meaningful result is not always straightforward. A number of components are required to facilitate a suitable sample size calculation. In this paper, the steps for conducting sample size calculations for non-inferiority and equivalence trials are summarised. Practical advice and examples are provided that illustrate how to carry out the calculations by hand and using the app SampSize.

Practical guide to sample size calculations: superiority trials.

A sample size justification is a vital part of any investigation. However, estimating the number of participants required to give meaningful results is not always straightforward. A number of components are required to facilitate a suitable sample size calculation. In this paper, the steps for conducting sample size calculations for superiority trials are summarised. Practical advice and examples are provided illustrating how to carry out the calculations by hand and using the app SampSize.

The disagreeable behaviour of the kappa statistic.

It is often of interest to measure the agreement between a number of raters when an outcome is nominal or ordinal. The kappa statistic is used as a measure of agreement. The statistic is highly sensitive to the distribution of the marginal totals and can produce unreliable results. Other statistics such as the proportion of concordance, maximum attainable kappa and prevalence and bias adjusted kappa should be considered to indicate how well the kappa statistic represents agreement in the data. Each kappa should be considered and interpreted based on the context of the data being analysed.

Online randomised trials with children: A scoping review.

BACKGROUND: Paediatric trials must contend with many challenges that adult trials face but often bring additional obstacles. Decentralised trials, where some or all trial methods occur away from a centralised location, are a promising strategy to help meet these challenges. This scoping review aims to (a) identify what methods and tools have been used to create and conduct entirely online-decentralised trials with children and (b) determine the gaps in the knowledge in this field. This review will describe the methods used in these trials to identify their facilitators and the gaps in the knowledge. METHODS: The methods were informed by guidance from the Joanna Briggs Institute and the PRISMA extension for scoping reviews. We systematically searched MEDLINE, CENTRAL, CINAHL, and Embase databases, trial registries, pre-print servers, and the internet. We included randomised and quasi-randomised trials conducted entirely online with participants under 18 published in English. A risk of bias assessment was completed for all included studies. RESULTS: Twenty-one trials met our inclusion criteria. The average age of participants was 14.6 years. Social media was the most common method of online recruitment. Most trials employed an external host website to store and protect their data. Duration of trials ranged from single-session interventions up to ten weeks. Fourteen trials compensated participants. Eight trials involved children in their trial design process; none reported compensation for this. Most trials had a low risk of bias in "random sequence generation", "selective reporting", and "other". Most trials had a high risk of bias in "blinding participants and personnel", "blinding of outcome assessment", and "incomplete outcome data". "Allocation concealment" was unclear in most studies. CONCLUSIONS: There was a lack of transparent reporting of the recruitment, randomisation, and retention methods used in many of the trials included in this review. Patient and public involvement (PPI) was not common, and the compensation of PPI partners was not reported in any study. Consent methods and protection against fraudulent entries to trials were creative and thoroughly discussed by some trials and not addressed by others. More work and thorough reporting of how these trials are conducted is needed to increase their reproducibility and quality. ETHICS AND DISSEMINATION: Ethical approval was not necessary since all data sources used are publicly available.

Cost-Effectiveness of Robot-Assisted Radical Cystectomy vs Open Radical Cystectomy for Patients With Bladder Cancer.

Importance: The value to payers of robot-assisted radical cystectomy with intracorporeal urinary diversion (iRARC) when compared with open radical cystectomy (ORC) for patients with bladder cancer is unclear. Objectives: To compare the cost-effectiveness of iRARC with that of ORC. Design, Setting, and Participants: This economic evaluation used individual patient data from a randomized clinical trial at 9 surgical centers in the United Kingdom. Patients with nonmetastatic bladder cancer were recruited from March 20, 2017, to January 29, 2020. The analysis used a health service perspective and a 90-day time horizon, with supplementary analyses exploring patient benefits up to 1 year. Deterministic and probabilistic sensitivity analyses were undertaken. Data were analyzed from January 13, 2022, to March 10, 2023. Interventions: Patients were randomized to receive either iRARC (n = 169) or ORC (n = 169). Main Outcomes and Measures: Costs of surgery were calculated using surgery timings and equipment costs, with other hospital data based on counts of activity. Quality-adjusted life-years were calculated from European Quality of Life 5-Dimension 5-Level instrument responses. Prespecified subgroup analyses were undertaken based on patient characteristics and type of diversion. Results: A total of 305 patients with available outcome data were included in the analysis, with a mean (SD) age of 68.3 (8.1) years, and of whom 241 (79.0%) were men. Robot-assisted radical cystectomy was associated with statistically significant reductions in admissions to intensive therapy (6.35% [95% CI, 0.42%-12.28%]), and readmissions to hospital (14.56% [95% CI, 5.00%-24.11%]), but increases in theater time (31.35 [95% CI, 13.67-49.02] minutes). The additional cost of iRARC per patient was £1124 (95% CI, -£576 to £2824 [US $1622 (95% CI, -$831 to $4075)]) with an associated gain in quality-adjusted life-years of 0.01124 (95% CI, 0.00391-0.01857). The incremental cost-effectiveness ratio was £100 008 (US $144 312) per quality-adjusted life-year gained. Robot-assisted radical cystectomy had a much higher probability of being cost-effective for subgroups defined by age, tumor stage, and performance status. Conclusions and Relevance: In this economic evaluation of surgery for patients with bladder cancer, iRARC reduced short-term morbidity and some associated costs. While the resulting cost-effectiveness ratio was in excess of thresholds used by many publicly funded health systems, patient subgroups were identified for which iRARC had a high probability of being cost-effective. Trial Registration: ClinicalTrials.gov Identifier: NCT03049410.

Expected Value of Sample Information to Guide the Design of Group Sequential Clinical Trials.

INTRODUCTION: Adaptive designs allow changes to an ongoing trial based on prespecified early examinations of accrued data. Opportunities are potentially being missed to incorporate health economic considerations into the design of these studies. METHODS: We describe how to estimate the expected value of sample information for group sequential design adaptive trials. We operationalize this approach in a hypothetical case study using data from a pilot trial. We report the expected value of sample information and expected net benefit of sampling results for 5 design options for the future full-scale trial including the fixed-sample-size design and the group sequential design using either the Pocock stopping rule or the O'Brien-Fleming stopping rule with 2 or 5 analyses. We considered 2 scenarios relating to 1) using the cost-effectiveness model with a traditional approach to the health economic analysis and 2) adjusting the cost-effectiveness analysis to incorporate the bias-adjusted maximum likelihood estimates of trial outcomes to account for the bias that can be generated in adaptive trials. RESULTS: The case study demonstrated that the methods developed could be successfully applied in practice. The results showed that the O'Brien-Fleming stopping rule with 2 analyses was the most efficient design with the highest expected net benefit of sampling in the case study. CONCLUSIONS: Cost-effectiveness considerations are unavoidable in budget-constrained, publicly funded health care systems, and adaptive designs can provide an alternative to costly fixed-sample-size designs. We recommend that when planning a clinical trial, expected value of sample information methods be used to compare possible adaptive and nonadaptive trial designs, with appropriate adjustment, to help justify the choice of design characteristics and ensure the cost-effective use of research funding. HIGHLIGHTS: Opportunities are potentially being missed to incorporate health economic considerations into the design of adaptive clinical trials.Existing expected value of sample information analysis methods can be extended to compare possible group sequential and nonadaptive trial designs when planning a clinical trial.We recommend that adjusted analyses be presented to control for the potential impact of the adaptive designs and to maintain the accuracy of the calculations.This approach can help to justify the choice of design characteristics and ensure the cost-effective use of limited research funding.

Comprehensive review of statistical methods for analysing patient-reported outcomes (PROs) used as primary outcomes in randomised controlled trials (RCTs) published by the UK's Health Technology Assessment (HTA) journal (1997-2020).

OBJECTIVES: To identify how frequently patient-reported outcomes (PROs) are used as primary and/or secondary outcomes in randomised controlled trials (RCTs) and to summarise what statistical methods are used for the analysis of PROs. DESIGN: Comprehensive review. SETTING: RCTs funded and published by the United Kingdom's (UK) National Institute for Health Research (NIHR) Health Technology Assessment (HTA) Programme. DATA SOURCES AND ELIGIBILITY: HTA reports of RCTs published between January 1997 and December 2020 were reviewed. DATA EXTRACTION: Information relating to PRO use and analysis methods was extracted. PRIMARY AND SECONDARY OUTCOME MEASURES: The frequency of using PROs as primary and/or secondary outcomes; statistical methods that were used for the analysis of PROs as primary outcomes. RESULTS: In this review, 37.6% (114/303) of trials used PROs as primary outcomes, and 82.8% (251/303) of trials used PROs as secondary outcomes from 303 NIHR HTA reports of RCTs. In the 114 RCTs where the PRO was the primary outcome, the most used PRO was the Short-Form 36 (8/114); the most popular methods for multivariable analysis were linear mixed model (45/114), linear regression (29/114) and analysis of covariance (13/114); logistic regression was applied for binary and ordinal outcomes in 14/114 trials; and the repeated measures analysis was used in 39/114 trials. CONCLUSION: The majority of trials used PROs as primary and/or secondary outcomes. Conventional methods such as linear regression are widely used, despite the potential violation of their assumptions. In recent years, there is an increasing trend of using complex models (eg, with mixed effects). Statistical methods developed to address these violations when analysing PROs, such as beta-binomial regression, are not routinely used in practice. Future research will focus on evaluating available statistical methods for the analysis of PROs.