Bayesian sample size determination for diagnostic accuracy studies.

The development of a new diagnostic test ideally follows a sequence of stages which, among other aims, evaluate technical performance. This includes an analytical validity study, a diagnostic accuracy study, and an interventional clinical utility study. In this article, we propose a novel Bayesian approach to sample size determination for the diagnostic accuracy study, which takes advantage of information available from the analytical validity stage. We utilize assurance to calculate the required sample size based on the target width of a posterior probability interval and can choose to use or disregard the data from the analytical validity study when subsequently inferring measures of test accuracy. Sensitivity analyses are performed to assess the robustness of the proposed sample size to the choice of prior, and prior-data conflict is evaluated by comparing the data to the prior predictive distributions. We illustrate the proposed approach using a motivating real-life application involving a diagnostic test for ventilator associated pneumonia. Finally, we compare the properties of the approach against commonly used alternatives. The results show that, when suitable prior information is available, the assurance-based approach can reduce the required sample size when compared to alternative approaches.

Systematic review of studies investigating ventilator associated pneumonia diagnostics in intensive care.

BACKGROUND: Ventilator-associated pneumonia (VAP) is an important diagnosis in critical care. VAP research is complicated by the lack of agreed diagnostic criteria and reference standard test criteria. Our aim was to review which reference standard tests are used to evaluate novel index tests for suspected VAP. METHODS: We conducted a comprehensive search using electronic databases and hand reference checks. The Cochrane Library, MEDLINE, CINHAL, EMBASE, and web of science were searched from 2008 until November 2018. All terms related to VAP diagnostics in the intensive treatment unit were used to conduct the search. We adopted a checklist from the critical appraisal skills programme checklist for diagnostic studies to assess the quality of the included studies. RESULTS: We identified 2441 records, of which 178 were selected for full-text review. Following methodological examination and quality assessment, 44 studies were included in narrative data synthesis. Thirty-two (72.7%) studies utilised a sole microbiological reference standard; the remaining 12 studies utilised a composite reference standard, nine of which included a mandatory microbiological criterion. Histopathological criteria were optional in four studies but mandatory in none. CONCLUSIONS: Nearly all reference standards for VAP used in diagnostic test research required some microbiological confirmation of infection, with BAL culture being the most common reference standard used.

Between a ROC and a hard place: Teaching prevalence plots to understand real world biomarker performance in the clinic.

The Receiver Operating Characteristic (ROC) curve and the Area Under the Curve (AUC) of the ROC curve are widely used in discovery to compare the performance of diagnostic and prognostic assays. The ROC curve has the advantage that it is independent of disease prevalence. However, in this note, we remind scientists and clinicians that the performance of an assay upon translation to the clinic is critically dependent upon that very same prevalence. Without an understanding of prevalence in the test population, even robust bioassays with excellent ROC characteristics may perform poorly in the clinic. While the exact prevalence in the target population is not always known, simple plots of candidate assay performance as a function of prevalence rate give a better understanding of the likely real-world performance and a greater understanding of the likely impact of variation in that prevalence on translation to the clinic.

Teaching examples for the design of experiments: geographical sensitivity and the self-fulfilling prophecy.

Many scientists believe that small experiments, guided by scientific intuition, are simpler and more efficient than design of experiments. This belief is strong and persists even in the face of data demonstrating that it is clearly wrong. In this paper, we present two powerful teaching examples illustrating the dangers of small experiments guided by scientific intuition. We describe two, simple, two-dimensional spaces. These two spaces give rise to, and at the same time appear to generate supporting data for, scientific intuitions that are deeply flawed or wholly incorrect. We find these spaces useful in unfreezing scientific thinking and challenging the misplaced confidence in scientific intuition.

R&D productivity rides again?

A recent analysis of R&D productivity suggests that there are grounds for 'cautious optimism' that the industry 'turned the corner' in 2008 and is 'on the comeback trail'. We believe that this analysis is flawed and most probably wrong. We present an alternative analysis of these same data to suggest that the industry is not yet 'out of the woods' and suggest that many of the systemic issues affecting pharmaceutical R&D productivity are still being resolved.

Sample size determination for point-of-care COVID-19 diagnostic tests: a Bayesian approach.

BACKGROUND: In a pandemic setting, it is critical to evaluate and deploy accurate diagnostic tests rapidly. This relies heavily on the sample size chosen to assess the test accuracy (e.g. sensitivity and specificity) during the diagnostic accuracy study. Too small a sample size will lead to imprecise estimates of the accuracy measures, whereas too large a sample size may delay the development process unnecessarily. This study considers use of a Bayesian method to guide sample size determination for diagnostic accuracy studies, with application to COVID-19 rapid viral detection tests. Specifically, we investigate whether utilising existing information (e.g. from preceding laboratory studies) within a Bayesian framework can reduce the required sample size, whilst maintaining test accuracy to the desired precision. METHODS: The method presented is based on the Bayesian concept of assurance which, in this context, represents the unconditional probability that a diagnostic accuracy study yields sensitivity and/or specificity intervals with the desired precision. We conduct a simulation study to evaluate the performance of this approach in a variety of COVID-19 settings, and compare it to commonly used power-based methods. An accompanying interactive web application is available, which can be used by researchers to perform the sample size calculations. RESULTS: Results show that the Bayesian assurance method can reduce the required sample size for COVID-19 diagnostic accuracy studies, compared to standard methods, by making better use of laboratory data, without loss of performance. Increasing the size of the laboratory study can further reduce the required sample size in the diagnostic accuracy study. CONCLUSIONS: The method considered in this paper is an important advancement for increasing the efficiency of the evidence development pathway. It has highlighted that the trade-off between lab study sample size and diagnostic accuracy study sample size should be carefully considered, since establishing an adequate lab sample size can bring longer-term gains. Although emphasis is on its use in the COVID-19 pandemic setting, where we envisage it will have the most impact, it can be usefully applied in other clinical areas.

Schrödinger's pipeline and the outsourcing of pharmaceutical innovation.

In the wake of the Global Financial Crisis (2007-2008) cheaper, softer money flooded the worldwide markets. Faced with historically low capital costs, the pharmaceutical industry chose to pay down debt through share buybacks rather than invest in research and development (R&D). Instead, the industry explored new R&D models for open innovation, such as open-sourcing, crowd-sourcing, public-private partnerships, innovation centres, Science Parks, and the wholesale outsourcing of pharmaceutical R&D. However, economic Greater Fool Theory suggests that outsourcing R&D was never likely to increase innovation. Ten years on, the period of cheaper and softer money is coming to an end. So how are things looking? And what happens next?

How to Ease the Pain of Taking a Diagnostic Point of Care Test to the Market: A Framework for Evidence Development.

Bringing a diagnostic point of care test (POCT) to a healthcare market can be a painful experience as it requires the manufacturer to meet considerable technical, financial, managerial, and regulatory challenges. In this opinion article we propose a framework for developing the evidence needed to support product development, marketing, and adoption. We discuss each step in the evidence development pathway from the invention phase to the implementation of a new POCT in the healthcare system. We highlight the importance of articulating the value propositions and documenting the care pathway. We provide guidance on how to conduct care pathway analysis as little has been published on this. We summarize the clinical, economic and qualitative studies to be considered for developing evidence, and provide useful links to relevant software, on-line applications, websites, and give practical advice. We also provide advice on patient and public involvement and engagement (PPIE), and on product management. Our aim is to help device manufacturers to understand the concepts and terminology used in evaluation of in vitro diagnostics (IVDs) so that they can communicate effectively with evaluation methodologists, statisticians, and health economists. Manufacturers of medical tests and devices can use the proposed framework to plan their evidence development strategy in alignment with device development, applications for regulatory approval, and publication.

COVID-19-associated hyperinflammation and escalation of patient care: a retrospective longitudinal cohort study.

BACKGROUND: A subset of patients with severe COVID-19 develop a hyperinflammatory syndrome, which might contribute to morbidity and mortality. This study explores a specific phenotype of COVID-19-associated hyperinflammation (COV-HI), and its associations with escalation of respiratory support and survival. METHODS: In this retrospective cohort study, we enrolled consecutive inpatients (aged ≥18 years) admitted to University College London Hospitals and Newcastle upon Tyne Hospitals in the UK with PCR-confirmed COVID-19 during the first wave of community-acquired infection. Demographic data, laboratory tests, and clinical status were recorded from the day of admission until death or discharge, with a minimum follow-up time of 28 days. We defined COV-HI as a C-reactive protein concentration greater than 150 mg/L or doubling within 24 h from greater than 50 mg/L, or a ferritin concentration greater than 1500 µg/L. Respiratory support was categorised as oxygen only, non-invasive ventilation, and intubation. Initial and repeated measures of hyperinflammation were evaluated in relation to the next-day risk of death or need for escalation of respiratory support (as a combined endpoint), using a multi-level logistic regression model. FINDINGS: We included 269 patients admitted to one of the study hospitals between March 1 and March 31, 2020, among whom 178 (66%) were eligible for escalation of respiratory support and 91 (34%) patients were not eligible. Of the whole cohort, 90 (33%) patients met the COV-HI criteria at admission. Despite having a younger median age and lower median Charlson Comorbidity Index scores, a higher proportion of patients with COV-HI on admission died during follow-up (36 [40%] of 90 patients) compared with the patients without COV-HI on admission (46 [26%] of 179). Among the 178 patients who were eligible for full respiratory support, 65 (37%) met the definition for COV-HI at admission, and 67 (74%) of the 90 patients whose respiratory care was escalated met the criteria by the day of escalation. Meeting the COV-HI criteria was significantly associated with the risk of next-day escalation of respiratory support or death (hazard ratio 2·24 [95% CI 1·62-2·87]) after adjustment for age, sex, and comorbidity. INTERPRETATION: Associations between elevated inflammatory markers, escalation of respiratory support, and survival in people with COVID-19 indicate the existence of a high-risk inflammatory phenotype. COV-HI might be useful to stratify patient groups in trial design. FUNDING: None.

Symptom-based stratification of patients with primary Sjögren's syndrome: multi-dimensional characterisation of international observational cohorts and reanalyses of randomised clinical trials.

BACKGROUND: Heterogeneity is a major obstacle to developing effective treatments for patients with primary Sjögren's syndrome. We aimed to develop a robust method for stratification, exploiting heterogeneity in patient-reported symptoms, and to relate these differences to pathobiology and therapeutic response. METHODS: We did hierarchical cluster analysis using five common symptoms associated with primary Sjögren's syndrome (pain, fatigue, dryness, anxiety, and depression), followed by multinomial logistic regression to identify subgroups in the UK Primary Sjögren's Syndrome Registry (UKPSSR). We assessed clinical and biological differences between these subgroups, including transcriptional differences in peripheral blood. Patients from two independent validation cohorts in Norway and France were used to confirm patient stratification. Data from two phase 3 clinical trials were similarly stratified to assess the differences between subgroups in treatment response to hydroxychloroquine and rituximab. FINDINGS: In the UKPSSR cohort (n=608), we identified four subgroups: Low symptom burden (LSB), high symptom burden (HSB), dryness dominant with fatigue (DDF), and pain dominant with fatigue (PDF). Significant differences in peripheral blood lymphocyte counts, anti-SSA and anti-SSB antibody positivity, as well as serum IgG, κ-free light chain, ß2-microglobulin, and CXCL13 concentrations were observed between these subgroups, along with differentially expressed transcriptomic modules in peripheral blood. Similar findings were observed in the independent validation cohorts (n=396). Reanalysis of trial data stratifying patients into these subgroups suggested a treatment effect with hydroxychloroquine in the HSB subgroup and with rituximab in the DDF subgroup compared with placebo. INTERPRETATION: Stratification on the basis of patient-reported symptoms of patients with primary Sjögren's syndrome revealed distinct pathobiological endotypes with distinct responses to immunomodulatory treatments. Our data have important implications for clinical management, trial design, and therapeutic development. Similar stratification approaches might be useful for patients with other chronic immune-mediated diseases. FUNDING: UK Medical Research Council, British Sjogren's Syndrome Association, French Ministry of Health, Arthritis Research UK, Foundation for Research in Rheumatology. VIDEO ABSTRACT.

Why is it hard to terminate failing projects in pharmaceutical R&D?

'Quick-kill' strategies in pharmaceutical research and development aim to reduce late-stage attrition by bringing project termination decisions forward, to an earlier point in the process. How can the barriers to implementing such strategies be overcome?

Lost in space: design of experiments and scientific exploration in a Hogarth Universe.

A Hogarth, or 'wicked', universe is an irregular environment generating data to support erroneous beliefs. Here, we argue that development scientists often work in such a universe. We demonstrate that exploring these multidimensional spaces using small experiments guided by scientific intuition alone, gives rise to an illusion of validity and a misplaced confidence in that scientific intuition. By contrast, design of experiments (DOE) permits the efficient mapping of such complex, multidimensional spaces. We describe simulation tools that enable research scientists to explore these spaces in relative safety.

The development speed paradox: can increasing development speed reduce R&D productivity?

In the 1990s the pharmaceutical industry sought to increase R&D productivity by shifting development tasks into parallel to reduce development cycle times and increase development speed. This paper presents a simple model demonstrating that, when attrition rates are high as in pharmaceutical development, such development speed initiatives can increase the expected time for the first successful molecule to complete development. Increasing the development speed of successful molecules could actually reduce R&D productivity - the development speed paradox.

Torching the Haystack: modelling fast-fail strategies in drug development.

By quickly clearing the development pipeline of failing or marginal products, fast-fail strategies release resources to focus on more promising molecules. The Quick-Kill model of drug development demonstrates that fast-fail strategies will: (1) reduce the expected time to market; (2) reduce expected R&D costs; and (3) increase R&D productivity. This paper outlines the model and demonstrates the impact of fast-fail strategies. The model is illustrated with costs and risks data from pharmaceutical and biopharmaceutical companies.