Student and the Lanarkshire milk experiment.

A detailed examination of the 1930 Lanarkshire Milk Experiment (LME) by the famous statistician William Sealy Gossett ("Student"), which appeared in Biometrika in 1931, is re-examined from a more modern perspective. The LME had a complicated design whereby 67 schools in Lanarkshire were allocated to receive either raw or pasteurised milk but pupils within the schools were allocated to either receive milk or to act as controls. Student's criticisms are considered in detail and examined in terms of subsequent developments on the design and analysis of experiments, in particular as regards appropriate estimation of standard errors of treatment estimates when an incomplete blocks structure has been used. An analogy with a more modern trial in osteoarthritis is made. Suggestions are made as to how analysis might proceed if the original data were available. Some lessons for observational studies in epidemiology are drawn and it is speculated that hidden clustering structures might be an explanation as to why results may vary from observational study to observational study by more than conventionally calculated standard errors might suggest.

On the relevance of prognostic information for clinical trials: A theoretical quantification.

The question of how individual patient data from cohort studies or historical clinical trials can be leveraged for designing more powerful, or smaller yet equally powerful, clinical trials becomes increasingly important in the era of digitalization. Today, the traditional statistical analyses approaches may seem questionable to practitioners in light of ubiquitous historical prognostic information. Several methodological developments aim at incorporating historical information in the design and analysis of future clinical trials, most importantly Bayesian information borrowing, propensity score methods, stratification, and covariate adjustment. Adjusting the analysis with respect to a prognostic score, which was obtained from some model applied to historical data, received renewed interest from a machine learning perspective, and we study the potential of this approach for randomized clinical trials. In an idealized situation of a normal outcome in a two-arm trial with 1:1 allocation, we derive a simple sample size reduction formula as a function of two criteria characterizing the prognostic score: (1) the coefficient of determination R2 on historical data and (2) the correlation ρ between the estimated and the true unknown prognostic scores. While maintaining the same power, the original total sample size n planned for the unadjusted analysis reduces to ( 1 - R 2 ρ 2 ) × n $(1 - R^2 \rho ^2) \times n$ in an adjusted analysis. Robustness in less ideal situations was assessed empirically. We conclude that there is potential for substantially more powerful or smaller trials, but only when prognostic scores can be accurately estimated.

Viewpoint: Do not resurrect the two-stage procedure.

In 1989, Peter Freeman published a paper that challenged a commonly accepted approach for analyzing cross-over trials, the so-called two-stage procedure. Freeman himself recommended using the Bayesian approach of Andy Grieve. The flaws Freeman exposed were serious and led many statisticians to conclude that the procedure was unacceptable. Unfortunately, more than 30 years later, one still encounters its use. This note explains, using a simple simulation, why the two-stage procedure is, indeed, as Freeman showed unacceptable and should not be used.

The design and analysis of vaccine trials for COVID-19 for the purpose of estimating efficacy.

After a preliminary explanation as to how I came to know Andy Grieve and some remarks about his career and mine and how they have intersected, I consider the design and analysis of trials of vaccines for COVID-19 for the purpose of estimating efficacy. Five large trials, run by the sponsors Pfizer/BioNTech, AstraZeneca/Oxford University, Moderna, Novavax and J&J Janssen are considered briefly. Frequentist approaches to analysis were used for four of the trials but Pfizer/BioNTech nominated a Bayesian approach. The design and analysis of this trial is considered in some detail, in particular as regards the choice of prior distribution. I conclude by drawing some general lessons.

Implementing Historical Controls in Oncology Trials.

Drug development in oncology has broadened from mainly considering randomized clinical trials to also including single-arm trials tailored for very specific subtypes of cancer. They often use historical controls, and this article discusses benefits and risks of this paradigm and provide various regulatory and statistical considerations. While leveraging the information brought by historical controls could potentially shorten development time and reduce the number of patients enrolled, a careful selection of the past studies, a prespecified statistical analysis accounting for the heterogeneity between studies, and early engagement with regulators will be key to success. Although both the European Medicines Agency and the U.S. Food and Drug Administration have already approved medicines based on nonrandomized experiments, the evidentiary package can be perceived as less comprehensive than randomized experiments. Use of historical controls, therefore, is better suited for cases of high unmet clinical need, where the disease course is well characterized and the primary endpoint is objective. IMPLICATIONS FOR PRACTICE: Incorporating historical data in single-arm oncology trials has the potential to accelerate drug development and to reduce the number of patients enrolled, compared with standard randomized controlled clinical trials. Given the lack of blinding and randomization, such an approach is better suited for cases of high unmet clinical need and/or difficult experimental situations, in which the trajectory of the disease is well characterized and the endpoint can be measured objectively. Careful pre-specification and selection of the historical data, matching of the patient characteristics with the concurrent trial data, and innovative statistical methodologies accounting for between-study variation will be needed. Early engagement with regulators (e.g., via Scientific Advice) is highly recommended.

The statistical properties of RCTs and a proposal for shrinkage.

We abstract the concept of a randomized controlled trial as a triple (ß,b,s) , where ß is the primary efficacy parameter, b the estimate, and s the standard error ( s>0 ). If the parameter ß is either a difference of means, a log odds ratio or a log hazard ratio, then it is reasonable to assume that b is unbiased and normally distributed. This then allows us to estimate the joint distribution of the z-value z=b/s and the signal-to-noise ratio SNR=ß/s from a sample of pairs (bi,si) . We have collected 23 551 such pairs from the Cochrane database. We note that there are many statistical quantities that depend on (ß,b,s) only through the pair (z,SNR) . We start by determining the estimated distribution of the achieved power. In particular, we estimate the median achieved power to be only 13%. We also consider the exaggeration ratio which is the factor by which the magnitude of ß is overestimated. We find that if the estimate is just significant at the 5% level, we would expect it to overestimate the true effect by a factor of 1.7. This exaggeration is sometimes referred to as the winner's curse and it is undoubtedly to a considerable extent responsible for disappointing replication results. For this reason, we believe it is important to shrink the unbiased estimator, and we propose a method for doing so. We show that our shrinkage estimator successfully addresses the exaggeration. As an example, we re-analyze the ANDROMEDA-SHOCK trial.

Random main effects of treatment: A case study with a network meta-analysis.

If the number of treatments in a network meta-analysis is large, it may be possible and useful to model the main effect of treatment as random, that is to say as random realizations from a normal distribution of possible treatment effects. This then constitutes a third sort of random effect that may be considered in connection with such analyses. The first and most common models treatment-by-trial interaction as being random and the second, rather rarer, models the main effects of trial as being random and thus permits the recovery of intertrial information. Taking the example of a network meta-analysis of 44 similar treatments in 10 trials, we illustrate how a hierarchical approach to modeling a random main effect of treatment can be used to produce shrunk (toward the overall mean) estimates of effects for individual treatments. As a related problem, we also consider the issue of using a random-effect model for the within-trial variances from trial to trial. We provide a number of possible graphical representations of the results and discuss the advantages and disadvantages of such an approach.

Statistical issues in first-in-human studies on BIA 10-2474: Neglected comparison of protocol against practice.

By setting the regulatory-approved protocol for a suite of first-in-human studies on BIA 10-2474 against the subsequent French investigations, we highlight 6 key design and statistical issues, which reinforce recommendations by a Royal Statistical Society Working Party, which were made in the aftermath of cytokine release storm in 6 healthy volunteers in the United Kingdom in 2006. The 6 issues are dose determination, availability of pharmacokinetic results, dosing interval, stopping rules, appraisal by safety committee, and clear algorithm required if combining approvals for single and multiple ascending dose studies.

Guidelines for the Content of Statistical Analysis Plans in Clinical Trials.

Importance: While guidance on statistical principles for clinical trials exists, there is an absence of guidance covering the required content of statistical analysis plans (SAPs) to support transparency and reproducibility. Objective: To develop recommendations for a minimum set of items that should be addressed in SAPs for clinical trials, developed with input from statisticians, previous guideline authors, journal editors, regulators, and funders. Design: Funders and regulators (n = 39) of randomized trials were contacted and the literature was searched to identify existing guidance; a survey of current practice was conducted across the network of UK Clinical Research Collaboration-registered trial units (n = 46, 1 unit had 2 responders) and a Delphi survey (n = 73 invited participants) was conducted to establish consensus on SAPs. The Delphi survey was sent to statisticians in trial units who completed the survey of current practice (n = 46), CONSORT (Consolidated Standards of Reporting Trials) and SPIRIT (Standard Protocol Items: Recommendations for Interventional Trials) guideline authors (n = 16), pharmaceutical industry statisticians (n = 3), journal editors (n = 9), and regulators (n = 2) (3 participants were included in 2 groups each), culminating in a consensus meeting attended by experts (N = 12) with representatives from each group. The guidance subsequently underwent critical review by statisticians from the surveyed trial units and members of the expert panel of the consensus meeting (N = 51), followed by piloting of the guidance document in the SAPs of 5 trials. Findings: No existing guidance was identified. The registered trials unit survey (46 responses) highlighted diversity in current practice and confirmed support for developing guidance. The Delphi survey (54 of 73, 74% participants completing both rounds) reached consensus on 42% (n = 46) of 110 items. The expert panel (N = 12) agreed that 63 items should be included in the guidance, with an additional 17 items identified as important but may be referenced elsewhere. Following critical review and piloting, some overlapping items were combined, leaving 55 items. Conclusions and Relevance: Recommendations are provided for a minimum set of items that should be addressed and included in SAPs for clinical trials. Trial registration, protocols, and statistical analysis plans are critically important in ensuring appropriate reporting of clinical trials.

Contribution to the discussion of '"A critical evaluation of the current p-value controversy"'.

I pick up a very few points of minor disagreement with Stefan Wellek's comprehensive review of P-values in this journal. I conclude that P-values have a limited function in statistical inference but can nevertheless have their uses.

Mastering variation: variance components and personalised medicine.

Various sources of variation in observed response in clinical trials and clinical practice are considered, and ways in which the corresponding components of variation might be estimated are discussed. Although the issues have been generally well-covered in the statistical literature, they seem to be poorly understood in the medical literature and even the statistical literature occasionally shows some confusion. To increase understanding and communication, some simple graphical approaches to illustrating issues are proposed. It is also suggested that reducing variation in medical practice might make as big a contribution to improving health outcome as personalising its delivery according to the patient. It is concluded that the common belief that there is a strong personal element in response to treatment is not based on sound statistical evidence.

Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations.

Misinterpretation and abuse of statistical tests, confidence intervals, and statistical power have been decried for decades, yet remain rampant. A key problem is that there are no interpretations of these concepts that are at once simple, intuitive, correct, and foolproof. Instead, correct use and interpretation of these statistics requires an attention to detail which seems to tax the patience of working scientists. This high cognitive demand has led to an epidemic of shortcut definitions and interpretations that are simply wrong, sometimes disastrously so-and yet these misinterpretations dominate much of the scientific literature. In light of this problem, we provide definitions and a discussion of basic statistics that are more general and critical than typically found in traditional introductory expositions. Our goal is to provide a resource for instructors, researchers, and consumers of statistics whose knowledge of statistical theory and technique may be limited but who wish to avoid and spot misinterpretations. We emphasize how violation of often unstated analysis protocols (such as selecting analyses for presentation based on the P values they produce) can lead to small P values even if the declared test hypothesis is correct, and can lead to large P values even if that hypothesis is incorrect. We then provide an explanatory list of 25 misinterpretations of P values, confidence intervals, and power. We conclude with guidelines for improving statistical interpretation and reporting.

R&D productivity rides again?

A recent analysis of R&D productivity suggests that there are grounds for 'cautious optimism' that the industry 'turned the corner' in 2008 and is 'on the comeback trail'. We believe that this analysis is flawed and most probably wrong. We present an alternative analysis of these same data to suggest that the industry is not yet 'out of the woods' and suggest that many of the systemic issues affecting pharmaceutical R&D productivity are still being resolved.

A note regarding meta-analysis of sequential trials with stopping for efficacy.

It is shown that fixed-effect meta-analyses of naïve treatment estimates from sequentially run trials with the possibility of stopping for efficacy based on a single interim look are unbiassed (or at the very least consistent, depending on the point of view) provided that the trials are weighted by information provided. A simple proof of this is given. An argument is given suggesting that this also applies in the case of multiple looks. The implications for this are discussed.