Replication of null results: Absence of evidence or evidence of absence?

In several large-scale replication projects, statistically non-significant results in both the original and the replication study have been interpreted as a 'replication success.' Here, we discuss the logical problems with this approach: Non-significance in both studies does not ensure that the studies provide evidence for the absence of an effect and 'replication success' can virtually always be achieved if the sample sizes are small enough. In addition, the relevant error rates are not controlled. We show how methods, such as equivalence testing and Bayes factors, can be used to adequately quantify the evidence for the absence of an effect and how they can be applied in the replication setting. Using data from the Reproducibility Project: Cancer Biology, the Experimental Philosophy Replicability Project, and the Reproducibility Project: Psychology we illustrate that many original and replication studies with 'null results' are in fact inconclusive. We conclude that it is important to also replicate studies with statistically non-significant results, but that they should be designed, analyzed, and interpreted appropriately.

Beyond the two-trials rule.

The two-trials rule for drug approval requires "at least two adequate and well-controlled studies, each convincing on its own, to establish effectiveness." This is usually implemented by requiring two significant pivotal trials and is the standard regulatory requirement to provide evidence for a new drug's efficacy. However, there is need to develop suitable alternatives to this rule for a number of reasons, including the possible availability of data from more than two trials. I consider the case of up to three studies and stress the importance to control the partial Type-I error rate, where only some studies have a true null effect, while maintaining the overall Type-I error rate of the two-trials rule, where all studies have a null effect. Some less-known P $$ P $$ -value combination methods are useful to achieve this: Pearson's method, Edgington's method and the recently proposed harmonic mean χ 2 $$ {\chi}^2 $$ -test. I study their properties and discuss how they can be extended to a sequential assessment of success while still ensuring overall Type-I error control. I compare the different methods in terms of partial Type-I error rate, project power and the expected number of studies required. Edgington's method is eventually recommended as it is easy to implement and communicate, has only moderate partial Type-I error rate inflation but substantially increased project power.

Power priors for replication studies.

The ongoing replication crisis in science has increased interest in the methodology of replication studies. We propose a novel Bayesian analysis approach using power priors: The likelihood of the original study's data is raised to the power of α, and then used as the prior distribution in the analysis of the replication data. Posterior distribution and Bayes factor hypothesis tests related to the power parameter α quantify the degree of compatibility between the original and replication study. Inferences for other parameters, such as effect sizes, dynamically borrow information from the original study. The degree of borrowing depends on the conflict between the two studies. The practical value of the approach is illustrated on data from three replication studies, and the connection to hierarchical modeling approaches explored. We generalize the known connection between normal power priors and normal hierarchical models for fixed parameters and show that normal power prior inferences with a beta prior on the power parameter α align with normal hierarchical model inferences using a generalized beta prior on the relative heterogeneity variance I2. The connection illustrates that power prior modeling is unnatural from the perspective of hierarchical modeling since it corresponds to specifying priors on a relative rather than an absolute heterogeneity scale.

Design differences and variation in results between randomised trials and non-randomised emulations: meta-analysis of RCT-DUPLICATE data.

Objective: To explore how design emulation and population differences relate to variation in results between randomised controlled trials (RCT) and non-randomised real world evidence (RWE) studies, based on the RCT-DUPLICATE initiative (Randomised, Controlled Trials Duplicated Using Prospective Longitudinal Insurance Claims: Applying Techniques of Epidemiology). Design: Meta-analysis of RCT-DUPLICATE data. Data sources: Trials included in RCT-DUPLICATE, a demonstration project that emulated 32 randomised controlled trials using three real world data sources: Optum Clinformatics Data Mart, 2004-19; IBM MarketScan, 2003-17; and subsets of Medicare parts A, B, and D, 2009-17. Eligibility criteria for selecting studies: Trials where the primary analysis resulted in a hazard ratio; 29 RCT-RWE study pairs from RCT-DUPLICATE. Results: Differences and variation in effect sizes between the results from randomised controlled trials and real world evidence studies were investigated. Most of the heterogeneity in effect estimates between the RCT-RWE study pairs in this sample could be explained by three emulation differences in the meta-regression model: treatment started in hospital (which does not appear in health insurance claims data), discontinuation of some baseline treatments at randomisation (which would have been an unusual care decision in clinical practice), and delayed onset of drug effects (which would be under-reported in real world clinical practice because of the relatively short persistence of the treatment). Adding the three emulation differences to the meta-regression reduced heterogeneity from 1.9 to almost 1 (absence of heterogeneity). Conclusions: This analysis suggests that a substantial proportion of the observed variation between results from randomised controlled trials and real world evidence studies can be attributed to differences in design emulation.

A phase Ib/II randomized, open-label drug repurposing trial of glutamate signaling inhibitors in combination with chemoradiotherapy in patients with newly diagnosed glioblastoma: the GLUGLIO trial protocol.

BACKGROUND: Glioblastoma is the most common and most aggressive malignant primary brain tumor in adults. Glioblastoma cells synthesize and secrete large quantities of the excitatory neurotransmitter glutamate, driving epilepsy, neuronal death, tumor growth and invasion. Moreover, neuronal networks interconnect with glioblastoma cell networks through glutamatergic neuroglial synapses, activation of which induces oncogenic calcium oscillations that are propagated via gap junctions between tumor cells. The primary objective of this study is to explore the efficacy of brain-penetrating anti-glutamatergic drugs to standard chemoradiotherapy in patients with glioblastoma. METHODS/DESIGN: GLUGLIO is a 1:1 randomized phase Ib/II, parallel-group, open-label, multicenter trial of gabapentin, sulfasalazine, memantine and chemoradiotherapy (Arm A) versus chemoradiotherapy alone (Arm B) in patients with newly diagnosed glioblastoma. Planned accrual is 120 patients. The primary endpoint is progression-free survival at 6 months. Secondary endpoints include overall and seizure-free survival, quality of life of patients and caregivers, symptom burden and cognitive functioning. Glutamate levels will be assessed longitudinally by magnetic resonance spectroscopy. Other outcomes of interest include imaging response rate, neuronal hyperexcitability determined by longitudinal electroencephalography, Karnofsky performance status as a global measure of overall performance, anticonvulsant drug use and steroid use. Tumor tissue and blood will be collected for translational research. Subgroup survival analyses by baseline parameters include segregation by age, extent of resection, Karnofsky performance status, O6-methylguanine DNA methyltransferase (MGMT) promotor methylation status, steroid intake, presence or absence of seizures, tumor volume and glutamate levels determined by MR spectroscopy. The trial is currently recruiting in seven centers in Switzerland. TRIAL REGISTRATION: NCT05664464. Registered 23 December 2022.

Simulating and reporting frequentist operating characteristics of clinical trials that borrow external information: Towards a fair comparison in case of one-arm and hybrid control two-arm trials.

Borrowing information from historical or external data to inform inference in a current trial is an expanding field in the era of precision medicine, where trials are often performed in small patient cohorts for practical or ethical reasons. Even though methods proposed for borrowing from external data are mainly based on Bayesian approaches that incorporate external information into the prior for the current analysis, frequentist operating characteristics of the analysis strategy are often of interest. In particular, type I error rate and power at a prespecified point alternative are the focus. We propose a procedure to investigate and report the frequentist operating characteristics in this context. The approach evaluates type I error rate of the test with borrowing from external data and calibrates the test without borrowing to this type I error rate. On this basis, a fair comparison of power between the test with and without borrowing is achieved. We show that no power gains are possible in one-sided one-arm and two-arm hybrid control trials with normal endpoint, a finding proven in general before. We prove that in one-arm fixed-borrowing situations, unconditional power (i.e., when external data is random) is reduced. The Empirical Bayes power prior approach that dynamically borrows information according to the similarity of current and external data avoids the exorbitant type I error inflation occurring with fixed borrowing. In the hybrid control two-arm trial we observe power reductions as compared to the test calibrated to borrowing that increase when considering unconditional power.

Bayesian approaches to designing replication studies.

Replication studies are essential for assessing the credibility of claims from original studies. A critical aspect of designing replication studies is determining their sample size; a too-small sample size may lead to inconclusive studies whereas a too-large sample size may waste resources that could be allocated better in other studies. Here, we show how Bayesian approaches can be used for tackling this problem. The Bayesian framework allows researchers to combine the original data and external knowledge in a design prior distribution for the underlying parameters. Based on a design prior, predictions about the replication data can be made, and the replication sample size can be chosen to ensure a sufficiently high probability of replication success. Replication success may be defined by Bayesian or non-Bayesian criteria and different criteria may also be combined to meet distinct stakeholders and enable conclusive inferences based on multiple analysis approaches. We investigate sample size determination in the normal-normal hierarchical model where analytical results are available and traditional sample size determination is a special case where the uncertainty on parameter values is not accounted for. We use data from a multisite replication project of social-behavioral experiments to illustrate how Bayesian approaches can help design informative and cost-effective replication studies. Our methods can be used through the R package BayesRepDesign. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

DESIGN DIFFERENCES EXPLAIN VARIATION IN RESULTS BETWEEN RANDOMIZED TRIALS AND THEIR NON-RANDOMIZED EMULATIONS.

Objectives: While randomized controlled trials (RCTs) are considered a standard for evidence on the efficacy of medical treatments, non-randomized real-world evidence (RWE) studies using data from health insurance claims or electronic health records can provide important complementary evidence. The use of RWE to inform decision-making has been questioned because of concerns regarding confounding in non-randomized studies and the use of secondary data. RCT-DUPLICATE was a demonstration project that emulated the design of 32 RCTs with non-randomized RWE studies. We sought to explore how emulation differences relate to variation in results between the RCT-RWE study pairs. Methods: We include all RCT-RWE study pairs from RCT-DUPLICATE where the measure of effect was a hazard ratio and use exploratory meta-regression methods to explain differences and variation in the effect sizes between the results from the RCT and the RWE study. The considered explanatory variables are related to design and population differences. Results: Most of the observed variation in effect estimates between RCT-RWE study pairs in this sample could be explained by three emulation differences in the meta-regression model: (i) in-hospital start of treatment (not observed in claims data), (ii) discontinuation of certain baseline therapies at randomization (not part of clinical practice), (iii) delayed onset of drug effects (missed by short medication persistence in clinical practice). Conclusions: This analysis suggests that a substantial proportion of the observed variation between results from RCTs and RWE studies can be attributed to design emulation differences. (238 words).

Combining evidence from clinical trials in conditional or accelerated approval.

Conditional (European Medicines Agency) or accelerated (U.S. Food and Drug Administration) approval of drugs allows earlier access to promising new treatments that address unmet medical needs. Certain post-marketing requirements must typically be met in order to obtain full approval, such as conducting a new post-market clinical trial. We study the applicability of the recently developed harmonic mean χ 2 -test to this conditional or accelerated approval framework. The proposed approach can be used both to support the design of the post-market trial and the analysis of the combined evidence provided by both trials. Other methods considered are the two-trials rule, Fisher's criterion and Stouffer's method. In contrast to some of the traditional methods, the harmonic mean χ 2 -test always requires a post-market clinical trial. If the p -value from the pre-market clinical trial is ≪ 0.025 , a smaller sample size for the post-market clinical trial is needed than with the two-trials rule. For illustration, we apply the harmonic mean χ 2 -test to a drug which received conditional (and later full) market licensing by the EMA. A simulation study is conducted to study the operating characteristics of the harmonic mean χ 2 -test and two-trials rule in more detail. We finally investigate the applicability of these two methods to compute the power at interim of an ongoing post-market trial. These results are expected to aid in the design and assessment of the required post-market studies in terms of the level of evidence required for full approval.

Prevention of non-ventilator-associated hospital-acquired pneumonia in Switzerland: a type 2 hybrid effectiveness-implementation trial.

BACKGROUND: Non-ventilator-associated hospital-acquired pneumonia (nvHAP) is a frequent, but under-researched infection. We aimed to simultaneously test an nvHAP prevention intervention and a multifaceted implementation strategy. METHODS: In this single-centre, type 2 hybrid effectiveness-implementation study, all patients of nine surgical and medical departments at the University Hospital Zurich, Switzerland, were included and surveyed over three study periods: baseline (14-33 months, depending on department), implementation (2 months), and intervention (3-22 months, depending on department). The five-measure nvHAP prevention bundle consisted of oral care, dysphagia screening and management, mobilisation, discontinuation of non-indicated proton-pump inhibitors, and respiratory therapy. The implementation strategy comprised department-level implementation teams who conducted and locally adapted the core strategies of education, training, and changing infrastructure. Intervention effectiveness on the primary outcome measure of nvHAP incidence rate was quantified using a generalised estimating equation method in a Poisson regression model, with hospital departments as clusters. Implementation success scores and determinants were derived longitudinally through semistructured interviews with health-care workers. This trial is registered with ClinicalTrials.gov (NCT03361085). FINDINGS: Between Jan 1, 2017, and Feb 29, 2020, 451 nvHAP cases occurred during 361 947 patient-days. nvHAP incidence rate was 1·42 (95% CI 1·27-1·58) per 1000 patient-days in the baseline period and 0·90 (95% CI 0·73-1·10) cases per 1000 patient-days in the intervention period. The intervention-to-baseline nvHAP incidence rate ratio, adjusted for department and seasonality, was 0·69 (95% CI 0·52-0·91; p=0·0084). Implementation success scores correlated with lower nvHAP rate ratios (Pearson correlation -0·71, p=0·034). Determinants of implementation success were positive core business alignment, high perceived nvHAP risk, architectural characteristics promoting physical proximity of health-care staff, and favourable key individual traits. INTERPRETATION: The prevention bundle led to a reduction of nvHAP. Knowledge of the determinants of implementation success might help in upscaling nvHAP prevention. FUNDING: Swiss Federal Office of Public Health.

Renal dysfunction and outcome in left ventricular non-compaction.

BACKGROUND: While renal function has been observed to inversely correlate with clinical outcome in other cardiomyopathies, its prognostic significance in patients with left ventricular non-compaction cardiomyopathy (LVNC) has not been investigated. The aim of this study was to determine the prognostic value of renal function in LVNC patients. METHODS: Patients with isolated LVNC as diagnosed by echocardiography and/or magnetic resonance imaging in 4 Swiss centers were retrospectively analyzed for this study. Values for creatinine, urea, and estimated glomerular filtration rate (eGFR) as assessed by the CKD-EPI 2009 formula were collected and analyzed by a Cox regression model for the occurrence of a composite endpoint (death or heart transplantation). RESULTS: During the median observation period of 7.4 years 23 patients reached the endpoint. The ageand gender-corrected hazard ratios (HR) for death or heart transplantation were: 1.9 (95% confidence interval [CI] 1.4-2.6) for each increase over baseline creatinine level of 30 µmol/L (p < 0.001), 1.6 (95% CI 1.2-2.2) for each increase over baseline urea level of 5 mmol/L (p = 0.004), and 3.6 (95% CI 1.9-6.9) for each decrease below baseline eGFR level of 30 mL/min (p ≤ 0.001). The HR (log2) for every doubling of creatinine was 7.7 (95% CI 3-19.8; p < 0.001), for every doubling of urea 2.5 (95% CI 1.5-4.3; p < 0.001), and for every bisection of eGFR 5.3 (95% CI 2.4-11.6; p < 0.001). CONCLUSIONS: This study provides evidence that in patients with LVNC impairment in renal function is associated with an increased risk of death and heart transplantation suggesting that kidney function assessment should be standard in risk assessment of LVNC patients.

The number of beds occupied is an independent risk factor for discharge of trauma patients.

Reducing the burden of limited capacity on medical practitioners and public health systems requires a time-dependent characterization of hospitalization rates, such that inferences can be drawn about the underlying causes for hospitalization and patient discharge. The aim of this study was to analyze non-medical risk factors that lead to the discharge of trauma patients. This retrospective cohort study includes trauma patients who were treated in Switzerland between 2011 and 2018. The national Swiss database for quality assurance in surgery (AQC) was reviewed for trauma diagnoses according to the ICD-10 code. Non-medical risk factors include seasonal changes, daily changes, holidays, and number of beds occupied by trauma patients across Switzerland. Individual patient information was aggregated into counts per day of total patients, as well as counts per day of levels of each categorical variable of interest. The ARIMA-modeling was utilized to model the number of discharges per day as a function of auto aggressive function of all previously mentioned risk factors. This study includes 226,708 patients, 118,059 male (age 48.18, standard deviation (SD) 22.34 years) and 108,649 female (age 62.57, SD 22.89 years) trauma patients. The mean length of stay was 7.16 (SD 14.84) days and most patients were discharged home (n = 168,582, 74.8%). A weekly and yearly seasonality trend can be observed in admission trends. The mean number of occupied trauma beds ranges from 3700 to 4000 per day. The number of occupied beds increases on weekdays and decreases on holidays. The number of occupied beds is a positive, independent risk factor for discharge in trauma patients; as the number of occupied beds increases at any given time, so does the risk for discharge. The number of beds occupied represents an independent non-medical risk factor for discharge. Capacity determines triage of hospitalized patients and therefore might increase the risk of premature discharge.

Patient-Reported Outcomes in Free-Flap Breast Reconstructive Surgery over Time (PRO-BREST).

Introduction: Patient-reported outcomes (PROMs) are increasingly relevant to assess surgical quality and guide decisions in breast reconstruction (BR). Satisfaction with outcomes may change as time progresses. We assessed satisfaction in patients who underwent free-flap BR in the last 12 years. Methods: All patients who underwent free-flap BR from 2006 to 2018 were invited to complete the validated BREAST-Q for reconstruction. The BREAST-Q comprises 6 domains covering various aspects of satisfaction. Unadjusted linear regression assessed the relationship between different domains of the BREAST-Q and time since BR. Two-sample t tests assessed differences in satisfaction between patients who underwent BR ≥5 years versus <5 years prior. Results: Forty-three women with primary or secondary free-flap BR between 2006 and 2018 were included in the study. Most patients (n = 33, 76.7%) underwent DIEP flap BR. Overall satisfaction with breasts and with outcomes improved as time since BR increased (p = 0.031 and p = 0.017, respectively). Overall satisfaction with outcomes scored higher in patients with BR ≥5 years prior (≥5 years vs. <5 years: breast score 88.6 (SD 12.5) versus 66.9 (SD 21.8); p = 0.005). Satisfaction with breasts and psychosocial well-being also scored higher in these patients. There was no difference in results between primary and secondary BR. Patients who underwent additional surgery (refinements) reported higher satisfaction with outcomes and abdominal well-being. Conclusions: PROMs concerning satisfaction with breast and with outcomes following BR improve as time since treatment progresses. This study demonstrates that time since diagnosis may be an important factor in satisfaction. It underlines the importance of long-term PROMs related to BR, to help provide patients and health care professionals in decision-making and in managing expectations related to BR.

Post-mortem magnetic resonance imaging with computed tomography-guided biopsy for foetuses and infants: a prospective, multicentre, cross-sectional study.

BACKGROUND: Post-mortem imaging has been suggested as an alternative to conventional autopsy in the prenatal and postnatal periods. Noninvasive autopsies do not provide tissue for histological examination, which may limit their clinical value, especially when infection-related morbidity and mortality are suspected. METHODS: We performed a prospective, multicentre, cross-sectional study to compare the diagnostic performance of post-mortem magnetic resonance imaging with computed tomography-guided biopsy (Virtopsy®) with that of conventional autopsy in foetuses and infants. Cases referred for conventional autopsy were eligible for enrolment. After post-mortem imaging using a computed tomography scanner and a magnetic resonance imaging unit, computed tomography-guided tissue sampling was performed. Virtopsy results were compared with conventional autopsy in determining the likely final cause of death and major pathologies. The primary outcome was the proportion of cases for which the same cause of death was determined by both methods. Secondary outcomes included the proportion of false positive and false negative major pathological lesions detected by virtopsy and the proportion of computed tomography-guided biopsies that were adequate for histological examination. RESULTS: Overall, 101 cases (84 fetuses, 17 infants) were included. Virtopsy and autopsy identified the same cause of death in 91 cases (90.1%, 95% CI 82.7 to 94.5). The sensitivity and specificity of virtopsy for determining the cause of death were 96.6% (95% CI 90.6 to 98.8) and 41.7% (95% CI 19.3 to 68.0), respectively. In 32 cases (31.7%, 95% CI 23.4 to 41.3), major pathological findings remained undetected by virtopsy, and in 45 cases (44.6%, 95% CI 35.2 to 54.3), abnormalities were diagnosed by virtopsy but not confirmed by autopsy. Computed tomography-guided tissue sampling was adequate for pathological comments in 506 of 956 biopsies (52.7%) and added important diagnostic value in five of 30 cases (16.1%) with an unclear cause of death before autopsy compared with postmortem imaging alone. In 19 of 20 infective deaths (95%), biopsies revealed infection-related tissue changes. Infection was confirmed by placental examination in all fetal cases. CONCLUSIONS: Virtopsy demonstrated a high concordance with conventional autopsy for the detection of cause of death but was less accurate for the evaluation of major pathologies. Computed tomography-guided biopsy had limited additional diagnostic value. TRIAL REGISTRATION: ClinicalTrials.gov (NCT01888380).

Comment on "Bayesian additional evidence for decision making under small sample uncertainty".

We examine the concept of Bayesian Additional Evidence (BAE) recently proposed by Sondhi et al. We derive simple closed-form expressions for BAE and compare its properties with other methods for assessing findings in the light of new evidence. We find that while BAE is easy to apply, it lacks both a compelling rationale and clarity of use needed for reliable decision-making.