Predicting Long-Term Childhood Survival of Newborns with Congenital Heart Defects: A Population-Based, Prospective Cohort Study (EPICARD).

Backgroud: Congenital heart defects (CHDs) are the most frequent group of major congenital anomalies, accounting for almost 1% of all births. They comprise a very heterogeneous group of birth defects in terms of their severity, clinical management, epidemiology, and embryologic origins. Taking this heterogeneity into account is an important imperative to provide reliable prognostic information to patients and their caregivers, as well as to compare results between centers or to assess alternative diagnostic and treatment strategies. The Anatomic and Clinical Classification of CHD (ACC-CHD) aims to facilitate both the CHD coding process and data analysis in clinical and epidemiological studies. The objectives of the study were to (1) Describe the long-term childhood survival of newborns with CHD, and (2) Develop and validate predictive models of infant mortality based on the ACC-CHD. Methods: This study wasbased on data from a population-based, prospective cohort study: Epidemiological Study of Children with Congenital Heart Defects (EPICARD). The final study population comprised 1881 newborns with CHDs after excluding cases that were associated with chromosomal and other anomalies. Statistical analysis included non-parametric survival analysis and flexible parametric survival models. The predictive performance of models was assessed by Harrell's C index and the Royston-Sauerbrei RD2, with internal validation by bootstrap. Results: The overall 8-year survival rate for newborns with isolated CHDs was 0.96 [0.93-0.95]. There was a substantial difference between the survival rate of the categories of ACC-CHD. The highest and lowest 8-year survival rates were 0.995 [0.989-0.997] and 0.34 [0.21-0.50] for "interatrial communication abnormalities and ventricular septal defects" and "functionally univentricular heart", respectively. Model discrimination, as measured by Harrell's C, was 87% and 89% for the model with ACC-CHD alone and the full model, which included other known predictors of infant mortality, respectively. The predictive performance, as measured by RD2, was 45% and 50% for the ACC-CHD alone and the full model. These measures were essentially the same after internal validation by bootstrap. Conclusions: The ACC-CHD classification provided the basis of a highly discriminant survival model with good predictive ability for the 8-year survival of newborns with CHDs. Prediction of individual outcomes remains an important clinical and statistical challenge.

When should factorial designs be used for late-phase randomised controlled trials?

BACKGROUND: A 2×2 factorial design evaluates two interventions (A versus control and B versus control) by randomising to control, A-only, B-only or both A and B together. Extended factorial designs are also possible (e.g. 3×3 or 2×2×2). Factorial designs often require fewer resources and participants than alternative randomised controlled trials, but they are not widely used. We identified several issues that investigators considering this design need to address, before they use it in a late-phase setting. METHODS: We surveyed journal articles published in 2000-2022 relating to designing factorial randomised controlled trials. We identified issues to consider based on these and our personal experiences. RESULTS: We identified clinical, practical, statistical and external issues that make factorial randomised controlled trials more desirable. Clinical issues are (1) interventions can be easily co-administered; (2) risk of safety issues from co-administration above individual risks of the separate interventions is low; (3) safety or efficacy data are wanted on the combination intervention; (4) potential for interaction (e.g. effect of A differing when B administered) is low; (5) it is important to compare interventions with other interventions balanced, rather than allowing randomised interventions to affect the choice of other interventions; (6) eligibility criteria for different interventions are similar. Practical issues are (7) recruitment is not harmed by testing many interventions; (8) each intervention and associated toxicities is unlikely to reduce either adherence to the other intervention or overall follow-up; (9) blinding is easy to implement or not required. Statistical issues are (10) a suitable scale of analysis can be identified; (11) adjustment for multiplicity is not required; (12) early stopping for efficacy or lack of benefit can be done effectively. External issues are (13) adequate funding is available and (14) the trial is not intended for licensing purposes. An overarching issue (15) is that factorial design should give a lower sample size requirement than alternative designs. Across designs with varying non-adherence, retention, intervention effects and interaction effects, 2×2 factorial designs require lower sample size than a three-arm alternative when one intervention effect is reduced by no more than 24%-48% in the presence of the other intervention compared with in the absence of the other intervention. CONCLUSIONS: Factorial designs are not widely used and should be considered more often using our issues to consider. Low potential for at most small to modest interaction is key, for example, where the interventions have different mechanisms of action or target different aspects of the disease being studied.

Medicines and Healthcare products Regulatory Agency's "Consultation on proposals for legislative changes for clinical trials": a response from the Trials Methodology Research Partnership Adaptive Designs Working Group, with a focus on data sharing.

In the UK, the Medicines and Healthcare products Regulatory Agency consulted on proposals "to improve and strengthen the UK clinical trials legislation to help us make the UK the best place to research and develop safe and innovative medicines". The purpose of the consultation was to help finalise the proposals and contribute to the drafting of secondary legislation. We discussed these proposals as members of the Trials Methodology Research Partnership Adaptive Designs Working Group, which is jointly funded by the Medical Research Council and the National Institute for Health and Care Research. Two topics arose frequently in the discussion: the emphasis on legislation, and the absence of questions on data sharing. It is our opinion that the proposals rely heavily on legislation to change practice. However, clinical trials are heterogeneous, and as a result some trials will struggle to comply with all of the proposed legislation. Furthermore, adaptive design clinical trials are even more heterogeneous than their non-adaptive counterparts, and face more challenges. Consequently, it is possible that increased legislation could have a greater negative impact on adaptive designs than non-adaptive designs. Overall, we are sceptical that the introduction of legislation will achieve the desired outcomes, with some exceptions. Meanwhile the topic of data sharing - making anonymised individual-level clinical trial data available to other investigators for further use - is entirely absent from the proposals and the consultation in general. However, as an aspect of the wider concept of open science and reproducible research, data sharing is an increasingly important aspect of clinical trials. The benefits of data sharing include faster innovation, improved surveillance of drug safety and effectiveness and decreasing participant exposure to unnecessary risk. There are already a number of UK-focused documents that discuss and encourage data sharing, for example, the Concordat on Open Research Data and the Medical Research Council's Data Sharing Policy. We strongly suggest that data sharing should be the norm rather than the exception, and hope that the forthcoming proposals on clinical trials invite discussion on this important topic.

Point estimation for adaptive trial designs II: Practical considerations and guidance.

In adaptive clinical trials, the conventional end-of-trial point estimate of a treatment effect is prone to bias, that is, a systematic tendency to deviate from its true value. As stated in recent FDA guidance on adaptive designs, it is desirable to report estimates of treatment effects that reduce or remove this bias. However, it may be unclear which of the available estimators are preferable, and their use remains rare in practice. This article is the second in a two-part series that studies the issue of bias in point estimation for adaptive trials. Part I provided a methodological review of approaches to remove or reduce the potential bias in point estimation for adaptive designs. In part II, we discuss how bias can affect standard estimators and assess the negative impact this can have. We review current practice for reporting point estimates and illustrate the computation of different estimators using a real adaptive trial example (including code), which we use as a basis for a simulation study. We show that while on average the values of these estimators can be similar, for a particular trial realization they can give noticeably different values for the estimated treatment effect. Finally, we propose guidelines for researchers around the choice of estimators and the reporting of estimates following an adaptive design. The issue of bias should be considered throughout the whole lifecycle of an adaptive design, with the estimation strategy prespecified in the statistical analysis plan. When available, unbiased or bias-reduced estimates are to be preferred.

Treatment selection in multi-arm multi-stage designs: With application to a postpartum haemorrhage trial.

BACKGROUND: Multi-arm multi-stage trials are an efficient, adaptive approach for testing many treatments simultaneously within one protocol. In settings where numbers of patients available to be entered into trials and resources might be limited, such as primary postpartum haemorrhage, it may be necessary to select a pre-specified subset of arms at interim stages even if they are all showing some promise against the control arm. This will put a limit on the maximum number of patients required and reduce the associated costs. Motivated by the World Health Organization Refractory HaEmorrhage Devices trial in postpartum haemorrhage, we explored the properties of such a selection design in a randomised phase III setting and compared it with other alternatives. The objectives are: (1) to investigate how the timing of treatment selection affects the operating characteristics; (2) to explore the use of an information-rich (continuous) intermediate outcome to select the best-performing arm, out of four treatment arms, compared with using the primary (binary) outcome for selection at the interim stage; and (3) to identify factors that can affect the efficiency of the design. METHODS: We conducted simulations based on the refractory haemorrhage devices multi-arm multi-stage selection trial to investigate the impact of the timing of treatment selection and applying an adaptive allocation ratio on the probability of correct selection, overall power and familywise type I error rate. Simulations were also conducted to explore how other design parameters will affect both the maximum sample size and trial timelines. RESULTS: The results indicate that the overall power of the trial is bounded by the probability of 'correct' selection at the selection stage. The results showed that good operating characteristics are achieved if the treatment selection is conducted at around 17% of information time. Our results also showed that although randomising more patients to research arms before selection will increase the probability of selecting correctly, this will not increase the overall efficiency of the (selection) design compared with the fixed allocation ratio of 1:1 to all arms throughout. CONCLUSIONS: Multi-arm multi-stage selection designs are efficient and flexible with desirable operating characteristics. We give guidance on many aspects of these designs including selecting the intermediate outcome measure, the timing of treatment selection, and choosing the operating characteristics.

Point estimation for adaptive trial designs I: A methodological review.

Recent FDA guidance on adaptive clinical trial designs defines bias as "a systematic tendency for the estimate of treatment effect to deviate from its true value," and states that it is desirable to obtain and report estimates of treatment effects that reduce or remove this bias. The conventional end-of-trial point estimates of the treatment effects are prone to bias in many adaptive designs, because they do not take into account the potential and realized trial adaptations. While much of the methodological developments on adaptive designs have tended to focus on control of type I error rates and power considerations, in contrast the question of biased estimation has received relatively less attention. This article is the first in a two-part series that studies the issue of potential bias in point estimation for adaptive trials. Part I provides a comprehensive review of the methods to remove or reduce the potential bias in point estimation of treatment effects for adaptive designs, while part II illustrates how to implement these in practice and proposes a set of guidelines for trial statisticians. The methods reviewed in this article can be broadly classified into unbiased and bias-reduced estimation, and we also provide a classification of estimators by the type of adaptive design. We compare the proposed methods, highlight available software and code, and discuss potential methodological gaps in the literature.

Combining factorial and multi-arm multi-stage platform designs to evaluate multiple interventions efficiently.

BACKGROUND: Factorial designs and multi-arm multi-stage (MAMS) platform designs have many advantages, but the practical advantages and disadvantages of combining the two designs have not been explored. METHODS: We propose practical methods for a combined design within the platform trial paradigm where some interventions are not expected to interact and could be given together. RESULTS: We describe the combined design and suggest diagrams that can be used to represent it. Many properties are common both to standard factorial designs, including the need to consider interactions between interventions and the impact of intervention efficacy on power of other comparisons, and to standard multi-arm multi-stage designs, including the need to pre-specify procedures for starting and stopping intervention comparisons. We also identify some specific features of the factorial-MAMS design: timing of interim and final analyses should be determined by calendar time or total observed events; some non-factorial modifications may be useful; eligibility criteria should be broad enough to include any patient eligible for any part of the randomisation; stratified randomisation may conveniently be performed sequentially; and analysis requires special care to use only concurrent controls. CONCLUSION: A combined factorial-MAMS design can combine the efficiencies of factorial trials and multi-arm multi-stage platform trials. It allows us to address multiple research questions under one protocol and to test multiple new treatment options, which is particularly important when facing a new emergent infection such as COVID-19.

RAMPART: A model for a regulatory-ready academic-led phase III trial in the adjuvant renal cell carcinoma setting.

The development of therapeutics in oncology is a highly active research area for the pharmaceutical and biotechnology industries, but also has a strong academic base. Many new agents have been developed in recent years, most with specific biological targets. This has mandated the need to look at different ways to streamline the evaluation of new agents. One solution has been the development of adaptive trial designs that allow the evaluation of multiple agents, concentrating on the most promising agents while screening out those which are unlikely to benefit patients. Another way forward has been the growth of partnerships between academia and industry with the shared goal of designing and conducting high quality clinical trials which answer important clinical questions as efficiently as possible. The RAMPART trial (NCT03288532) brings together both of these processes in an attempt to improve outcomes for patients with locally advanced renal cell carcinoma (RCC), where no globally acceptable adjuvant strategy after nephrectomy currently exist. RAMPART is led by the MRC CTU at University College London (UCL), in collaboration with other international academic groups and industry. We aim to facilitate the use of data from RAMPART, (dependent on outcomes), for a future regulatory submission that will extend the license of the agents being investigated. We share our experience in order to lay the foundations for an effective trial design and conduct framework and to guide others who may be considering similar collaborations. Trial Registration: ISRCTN #: ISRCTN53348826, NCT #: NCT03288532, EUDRACT #: 2017-002329-39. CTA #: 20363/0380/001-0001. MREC #: 17/LO/1875. ClinicalTrials.gov Identifier: NCT03288532 RAMPART grant number: MC_UU_12023/25. . RAMPART Protocol version 5.0.

RAMPART: A phase III multi-arm multi-stage trial of adjuvant checkpoint inhibitors in patients with resected primary renal cell carcinoma (RCC) at high or intermediate risk of relapse.

BACKGROUND: 20-60% of patients with initially locally advanced Renal Cell Carcinoma (RCC) develop metastatic disease despite optimal surgical excision. Adjuvant strategies have been tested in RCC including cytokines, radiotherapy, hormones and oral tyrosine-kinase inhibitors (TKIs), with limited success. The predominant global standard-of-care after nephrectomy remains active monitoring. Immune checkpoint inhibitors (ICIs) are effective in the treatment of metastatic RCC; RAMPART will investigate these agents in the adjuvant setting. METHODS/DESIGN: RAMPART is an international, UK-led trial investigating the addition of ICIs after nephrectomy in patients with resected locally advanced RCC. RAMPART is a multi-arm multi-stage (MAMS) platform trial, upon which additional research questions may be addressed over time. The target population is patients with histologically proven resected locally advanced RCC (clear cell and non-clear cell histological subtypes), with no residual macroscopic disease, who are at high or intermediate risk of relapse (Leibovich score 3-11). Patients with fully resected synchronous ipsilateral adrenal metastases are included. Participants are randomly assigned (3,2:2) to Arm A - active monitoring (no placebo) for one year, Arm B - durvalumab (PD-L1 inhibitor) 4-weekly for one year; or Arm C - combination therapy with durvalumab 4-weekly for one year plus two doses of tremelimumab (CTLA-4 inhibitor) at day 1 of the first two 4-weekly cycles. The co-primary outcomes are disease-free-survival (DFS) and overall survival (OS). Secondary outcomes include safety, metastasis-free survival, RCC specific survival, quality of life, and patient and clinician preferences. Tumour tissue, plasma and urine are collected for molecular analysis (TransRAMPART). TRIAL REGISTRATION: ISRCTN #: ISRCTN53348826, NCT #: NCT03288532, EUDRACT #: 2017-002329-39, CTA #: 20363/0380/001-0001, MREC #: 17/LO/1875, ClinicalTrials.gov Identifier: NCT03288532, RAMPART grant number: MC_UU_12023/25, TransRAMPART grant number: A28690 Cancer Research UK, RAMPART Protocol version 5.0.

Adding new experimental arms to randomised clinical trials: Impact on error rates.

BACKGROUND: Experimental treatments pass through various stages of development. If a treatment passes through early-phase experiments, the investigators may want to assess it in a late-phase randomised controlled trial. An efficient way to do this is adding it as a new research arm to an ongoing trial while the existing research arms continue, a so-called multi-arm platform trial. The familywise type I error rate is often a key quantity of interest in any multi-arm platform trial. We set out to clarify how it should be calculated when new arms are added to a trial some time after it has started. METHODS: We show how the familywise type I error rate, any-pair and all-pairs powers can be calculated when a new arm is added to a platform trial. We extend the Dunnett probability and derive analytical formulae for the correlation between the test statistics of the existing pairwise comparison and that of the newly added arm. We also verify our analytical derivation via simulations. RESULTS: Our results indicate that the familywise type I error rate depends on the shared control arm information (i.e. individuals in continuous and binary outcomes and primary outcome events in time-to-event outcomes) from the common control arm patients and the allocation ratio. The familywise type I error rate is driven more by the number of pairwise comparisons and the corresponding (pairwise) type I error rates than by the timing of the addition of the new arms. The familywise type I error rate can be estimated using Sidák's correction if the correlation between the test statistics of pairwise comparisons is less than 0.30. CONCLUSIONS: The findings we present in this article can be used to design trials with pre-planned deferred arms or to add new pairwise comparisons within an ongoing platform trial where control of the pairwise error rate or familywise type I error rate (for a subset of pairwise comparisons) is required.

Proposals on Kaplan-Meier plots in medical research and a survey of stakeholder views: KMunicate.

OBJECTIVES: To examine reactions to the proposed improvements to standard Kaplan-Meier plots, the standard way to present time-to-event data, and to understand which (if any) facilitated better depiction of (1) the state of patients over time, and (2) uncertainty over time in the estimates of survival. DESIGN: A survey of stakeholders' opinions on the proposals. SETTING: A web-based survey, open to international participation, for those with an interest in visualisation of time-to-event data. PARTICIPANTS: 1174 people participated in the survey over a 6-week period. Participation was global (although primarily Europe and North America) and represented a wide range of researchers (primarily statisticians and clinicians). MAIN OUTCOME MEASURES: Two outcome measures were of principal importance: (1) participants' opinions of each proposal compared with a 'standard' Kaplan-Meier plot; and (2) participants' overall ranking of the proposals (including the standard). RESULTS: Most proposals were more popular than the standard Kaplan-Meier plot. The most popular proposals in the two categories, respectively, were an extended table beneath the plot depicting the numbers at risk, censored and having experienced an event at periodic timepoints, and CIs around each Kaplan-Meier curve. CONCLUSIONS: This study produced a high response number, reflecting the importance of graphics for time-to-event data. Those producing and publishing Kaplan-Meier plots-both authors and journals-should, as a starting point, consider using the combination of the two favoured proposals.

Combined test versus logrank/Cox test in 50 randomised trials.

BACKGROUND: The logrank test and the Cox proportional hazards model are routinely applied in the design and analysis of randomised controlled trials (RCTs) with time-to-event outcomes. Usually, sample size and power calculations assume proportional hazards (PH) of the treatment effect, i.e. the hazard ratio is constant over the entire follow-up period. If the PH assumption fails, the power of the logrank/Cox test may be reduced, sometimes severely. It is, therefore, important to understand how serious this can become in real trials, and for a proven, alternative test to be available to increase the robustness of the primary test. METHODS: We performed a systematic search to identify relevant articles in four leading medical journals that publish results of phase 3 clinical trials. Altogether, 50 articles satisfied our inclusion criteria. We digitised published Kaplan-Meier curves and created approximations to the original times to event or censoring at the individual patient level. Using the reconstructed data, we tested for non-PH in all 50 trials. We compared the results from the logrank/Cox test with those from the combined test recently proposed by Royston and Parmar. RESULTS: The PH assumption was checked and reported only in 28% of the studies. Evidence of non-PH at the 0.10 level was detected in 31% of comparisons. The Cox test of the treatment effect was significant at the 0.05 level in 49% of comparisons, and the combined test in 55%. In four of five trials with discordant results, the interpretation would have changed had the combined test been used. The degree of non-PH and the dominance of the p value for the combined test were strongly associated. Graphical investigation suggested that non-PH was mostly due to a treatment effect manifesting in an early follow-up and disappearing later. CONCLUSIONS: The evidence for non-PH is checked (and, hence, identified) in only a small minority of RCTs, but non-PH may be present in a substantial fraction of such trials. In our reanalysis of the reconstructed data from 50 trials, the combined test outperformed the Cox test overall. The combined test is a promising approach to making trial design and analysis more robust.

Assessing the impact of efficacy stopping rules on the error rates under the multi-arm multi-stage framework.

BACKGROUND: The multi-arm multi-stage framework uses intermediate outcomes to assess lack-of-benefit of research arms at interim stages in randomised trials with time-to-event outcomes. However, the design lacks formal methods to evaluate early evidence of overwhelming efficacy on the definitive outcome measure. We explore the operating characteristics of this extension to the multi-arm multi-stage design and how to control the pairwise and familywise type I error rate. Using real examples and the updated nstage program, we demonstrate how such a design can be developed in practice. METHODS: We used the Dunnett approach for assessing treatment arms when conducting comprehensive simulation studies to evaluate the familywise error rate, with and without interim efficacy looks on the definitive outcome measure, at the same time as the planned lack-of-benefit interim analyses on the intermediate outcome measure. We studied the effect of the timing of interim analyses, allocation ratio, lack-of-benefit boundaries, efficacy rule, number of stages and research arms on the operating characteristics of the design when efficacy stopping boundaries are incorporated. Methods for controlling the familywise error rate with efficacy looks were also addressed. RESULTS: Incorporating Haybittle-Peto stopping boundaries on the definitive outcome at the interim analyses will not inflate the familywise error rate in a multi-arm design with two stages. However, this rule is conservative; in general, more liberal stopping boundaries can be used with minimal impact on the familywise error rate. Efficacy bounds in trials with three or more stages using an intermediate outcome may inflate the familywise error rate, but we show how to maintain strong control. CONCLUSION: The multi-arm multi-stage design allows stopping for both lack-of-benefit on the intermediate outcome and efficacy on the definitive outcome at the interim stages. We provide guidelines on how to control the familywise error rate when efficacy boundaries are implemented in practice.

Adaptive designs in clinical trials: why use them, and how to run and report them.

Adaptive designs can make clinical trials more flexible by utilising results accumulating in the trial to modify the trial's course in accordance with pre-specified rules. Trials with an adaptive design are often more efficient, informative and ethical than trials with a traditional fixed design since they often make better use of resources such as time and money, and might require fewer participants. Adaptive designs can be applied across all phases of clinical research, from early-phase dose escalation to confirmatory trials. The pace of the uptake of adaptive designs in clinical research, however, has remained well behind that of the statistical literature introducing new methods and highlighting their potential advantages. We speculate that one factor contributing to this is that the full range of adaptations available to trial designs, as well as their goals, advantages and limitations, remains unfamiliar to many parts of the clinical community. Additionally, the term adaptive design has been misleadingly used as an all-encompassing label to refer to certain methods that could be deemed controversial or that have been inadequately implemented.We believe that even if the planning and analysis of a trial is undertaken by an expert statistician, it is essential that the investigators understand the implications of using an adaptive design, for example, what the practical challenges are, what can (and cannot) be inferred from the results of such a trial, and how to report and communicate the results. This tutorial paper provides guidance on key aspects of adaptive designs that are relevant to clinical triallists. We explain the basic rationale behind adaptive designs, clarify ambiguous terminology and summarise the utility and pitfalls of adaptive designs. We discuss practical aspects around funding, ethical approval, treatment supply and communication with stakeholders and trial participants. Our focus, however, is on the interpretation and reporting of results from adaptive design trials, which we consider vital for anyone involved in medical research. We emphasise the general principles of transparency and reproducibility and suggest how best to put them into practice.

Testing many treatments within a single protocol over 10 years at MRC Clinical Trials Unit at UCL: Multi-arm, multi-stage platform, umbrella and basket protocols.

There is real need to change how we do some of our clinical trials, as currently the testing and development process is too slow, too costly and too failure-prone often we find that a new treatment is no better than the current standard. Much of the focus on the development and testing pathway has been in improving the design of phase I and II trials. In this article, we present examples of new methods for improving the design of phase III trials (and the necessary lead up to them) as they are the most time-consuming and expensive part of the pathway. Key to all these methods is the aim to test many treatments and/or pose many therapeutic questions within one protocol.

Review and evaluation of performance measures for survival prediction models in external validation settings.

BACKGROUND: When developing a prediction model for survival data it is essential to validate its performance in external validation settings using appropriate performance measures. Although a number of such measures have been proposed, there is only limited guidance regarding their use in the context of model validation. This paper reviewed and evaluated a wide range of performance measures to provide some guidelines for their use in practice. METHODS: An extensive simulation study based on two clinical datasets was conducted to investigate the performance of the measures in external validation settings. Measures were selected from categories that assess the overall performance, discrimination and calibration of a survival prediction model. Some of these have been modified to allow their use with validation data, and a case study is provided to describe how these measures can be estimated in practice. The measures were evaluated with respect to their robustness to censoring and ease of interpretation. All measures are implemented, or are straightforward to implement, in statistical software. RESULTS: Most of the performance measures were reasonably robust to moderate levels of censoring. One exception was Harrell's concordance measure which tended to increase as censoring increased. CONCLUSIONS: We recommend that Uno's concordance measure is used to quantify concordance when there are moderate levels of censoring. Alternatively, Gönen and Heller's measure could be considered, especially if censoring is very high, but we suggest that the prediction model is re-calibrated first. We also recommend that Royston's D is routinely reported to assess discrimination since it has an appealing interpretation. The calibration slope is useful for both internal and external validation settings and recommended to report routinely. Our recommendation would be to use any of the predictive accuracy measures and provide the corresponding predictive accuracy curves. In addition, we recommend to investigate the characteristics of the validation data such as the level of censoring and the distribution of the prognostic index derived in the validation setting before choosing the performance measures.

Type I error rates of multi-arm multi-stage clinical trials: strong control and impact of intermediate outcomes.

BACKGROUND: The multi-arm multi-stage (MAMS) design described by Royston et al. [Stat Med. 2003;22(14):2239-56 and Trials. 2011;12:81] can accelerate treatment evaluation by comparing multiple treatments with a control in a single trial and stopping recruitment to arms not showing sufficient promise during the course of the study. To increase efficiency further, interim assessments can be based on an intermediate outcome (I) that is observed earlier than the definitive outcome (D) of the study. Two measures of type I error rate are often of interest in a MAMS trial. Pairwise type I error rate (PWER) is the probability of recommending an ineffective treatment at the end of the study regardless of other experimental arms in the trial. Familywise type I error rate (FWER) is the probability of recommending at least one ineffective treatment and is often of greater interest in a study with more than one experimental arm. METHODS: We demonstrate how to calculate the PWER and FWER when the I and D outcomes in a MAMS design differ. We explore how each measure varies with respect to the underlying treatment effect on I and show how to control the type I error rate under any scenario. We conclude by applying the methods to estimate the maximum type I error rate of an ongoing MAMS study and show how the design might have looked had it controlled the FWER under any scenario. RESULTS: The PWER and FWER converge to their maximum values as the effectiveness of the experimental arms on I increases. We show that both measures can be controlled under any scenario by setting the pairwise significance level in the final stage of the study to the target level. In an example, controlling the FWER is shown to increase considerably the size of the trial although it remains substantially more efficient than evaluating each new treatment in separate trials. CONCLUSIONS: The proposed methods allow the PWER and FWER to be controlled in various MAMS designs, potentially increasing the uptake of the MAMS design in practice. The methods are also applicable in cases where the I and D outcomes are identical.

Quantifying the uptake of user-written commands over time.

A major factor in the uptake of new statistical methods is the availability of user-friendly software implementations. One attractive feature of Stata is that users can write their own commands and release them to other users via Statistical Software Components at Boston College. Authors of statistical programs do not always get adequate credit, because programs are rarely cited properly. There is no obvious measure of a program's impact, but researchers are under increasing pressure to demonstrate the impact of their work to funders. In addition to encouraging proper citation of software, the number of downloads of a user-written package can be regarded as a measure of impact over time. In this article, we explain how such information can be accessed for any month from July 2007 and summarized using the new ssccount command.