The effect of high-dose erythropoietin perinatally on retinal function in school-aged children born extremely or very preterm.

PURPOSE: To investigate the long-term effects of high-dose recombinant human erythropoietin (rhEPO) administered during the perinatal period on retinal and visual function in children born extremely or very preterm. DESIGN: Randomized, double-blind clinical trial follow-up plus cohort study. METHODS: Setting: Department of Ophthalmology, University Hospital Zurich, Zurich, Switzerland. STUDY POPULATION: extremely or very preterm-born children aged 7-15 years, previously randomized to receive either high-dose rhEPO or placebo in the perinatal period. INCLUSION CRITERIA: participation in an ongoing neuropediatric study (EpoKids), written informed consent (IC). EXCLUSION CRITERIA: previous ocular trauma or surgery; retinal or developmental disease unrelated to prematurity. Healthy control (HC) children of comparable age were recruited. INCLUSION CRITERIA: term birth, IC. EXCLUSION CRITERIA: any ocular/visual abnormality, high refractive error. Intervention status (rhEPO/placebo) was unknown to examiners and subjects at examination, with examiners unblinded only after completion of all analyses. OBSERVATION PROCEDURES: Electroretinography (ERG) was performed with the RETeval device (LKC Technologies, Inc., Gaithersburg MD). Ophthalmological and orthoptic examinations excluded comorbidity in the prematurely born cohort and ocular diseases in the HC group. MAIN OUTCOME MEASURES: Scotopic and photopic ERG response amplitudes and peak times (6 amplitudes; 6 peak times). Secondary outcomes were habitual visual acuity and color discrimination performance (for descriptive summary only). RESULTS: No differences in ERG parameters between EPO (n=52; 104 eyes) and placebo (n=35; 70 eyes) subgroups were observed (all corrected p>0.05). Two cone system-mediated peak times were slightly slower in the placebo than HC (n=52; 104 eyes) subgroup (coefficient/95% confidence interval (CI) = 0.53/0.21 to 0.85 and 0.36/0.13 to 0.60; p = 0.012 and 0.022); a predominantly rod system-mediated peak time was slightly faster in the EPO than the HC subgroup (coefficient/95% CI = -4.33/-6.88 to -1.78; p = 0.011). Secondary outcomes were comparable across subgroups. CONCLUSIONS: Administration of high-dose rhEPO to infants born extremely or very preterm during the perinatal period has no measurable effects on retinal function in childhood compared to placebo. Premature birth may cause small, likely clinically insignificant effects on retinal function in childhood, which may be partially mitigated by administration of rhEPO during the perinatal period.

The Effect of Perinatal High-Dose Erythropoietin on Retinal Structural and Vascular Characteristics in Children Born Preterm.

PURPOSE: To study the long-term effects of perinatal high-dose recombinant human erythropoietin (rhEPO) on macular structural and vascular development in preterm children. DESIGN: Randomized, double-blind clinical trial follow-up plus cohort study. METHODS: Setting: Department of Ophthalmology, University Hospital Zurich, Zurich, Switzerland. STUDY POPULATION: extremely or very preterm born children aged 7-15 years from an ongoing neuropediatric study (EpoKids). These had been previously randomized to receive either high-dose rhEPO or placebo perinatally. INCLUSION CRITERIA: participation in the EpoKids Study, written informed consent (IC). EXCLUSION CRITERIA: previous ocular trauma or surgery; retinal or developmental disease unrelated to prematurity. Term-born children of comparable age were enrolled as a healthy control (HC) group. INCLUSION CRITERIA: term birth, IC. EXCLUSION CRITERIA: any ocular or visual abnormality, high refractive error. Examiners were blinded regarding intervention status until completion of all analyses. (Participants/guardians remain blinded). OBSERVATION PROCEDURES: Spectral-domain OCT scans (Heidelberg Spectralis system) and OCTA imaging (Zeiss PlexElite 9000) were obtained. Ophthalmological and orthoptic examinations excluded ocular comorbidities. MAIN OUTCOME MEASURES: OCT (central retinal thickness, CRT; total macular volume, TMV), superficial plexus OCTA (foveal avascular zone, FAZ; vessel density, VD; vessel length density, VLD) parameters and foveal hypoplasia grade according to published criteria. RESULTS: Macular vessel density parameters (VD and VLD) were significantly lower (p =0.015, CI-95: 0.01 to 0.06 and p=0.015, CI-95: 0.74 to 3.64) in the EPO group (n= 52) when compared to placebo (n=35). No other significant differences were observed between the EPO and placebo group. When comparing the intervention subgroups to HC we found six significant differences in OCT and OCTA parameters (FAZ, VD, VLD and CRT comparing HC and EPO group; FAZ and CRT when comparing HC and placebo group). CONCLUSIONS: Early high-dose rhEPO in infants born extremely or very preterm affects macular vessel density parameters compared to placebo. Premature birth (regardless of intervention status) affects retinal structure and vascular development. Our findings on macular vascular development do not contraindicate the administration of early high-dose EPO in preterm infants. For further understanding of the role of EPO on macular development and its clinical significance, future studies are needed.

Replication of null results: Absence of evidence or evidence of absence?

In several large-scale replication projects, statistically non-significant results in both the original and the replication study have been interpreted as a 'replication success.' Here, we discuss the logical problems with this approach: Non-significance in both studies does not ensure that the studies provide evidence for the absence of an effect and 'replication success' can virtually always be achieved if the sample sizes are small enough. In addition, the relevant error rates are not controlled. We show how methods, such as equivalence testing and Bayes factors, can be used to adequately quantify the evidence for the absence of an effect and how they can be applied in the replication setting. Using data from the Reproducibility Project: Cancer Biology, the Experimental Philosophy Replicability Project, and the Reproducibility Project: Psychology we illustrate that many original and replication studies with 'null results' are in fact inconclusive. We conclude that it is important to also replicate studies with statistically non-significant results, but that they should be designed, analyzed, and interpreted appropriately.

Design differences and variation in results between randomised trials and non-randomised emulations: meta-analysis of RCT-DUPLICATE data.

Objective: To explore how design emulation and population differences relate to variation in results between randomised controlled trials (RCT) and non-randomised real world evidence (RWE) studies, based on the RCT-DUPLICATE initiative (Randomised, Controlled Trials Duplicated Using Prospective Longitudinal Insurance Claims: Applying Techniques of Epidemiology). Design: Meta-analysis of RCT-DUPLICATE data. Data sources: Trials included in RCT-DUPLICATE, a demonstration project that emulated 32 randomised controlled trials using three real world data sources: Optum Clinformatics Data Mart, 2004-19; IBM MarketScan, 2003-17; and subsets of Medicare parts A, B, and D, 2009-17. Eligibility criteria for selecting studies: Trials where the primary analysis resulted in a hazard ratio; 29 RCT-RWE study pairs from RCT-DUPLICATE. Results: Differences and variation in effect sizes between the results from randomised controlled trials and real world evidence studies were investigated. Most of the heterogeneity in effect estimates between the RCT-RWE study pairs in this sample could be explained by three emulation differences in the meta-regression model: treatment started in hospital (which does not appear in health insurance claims data), discontinuation of some baseline treatments at randomisation (which would have been an unusual care decision in clinical practice), and delayed onset of drug effects (which would be under-reported in real world clinical practice because of the relatively short persistence of the treatment). Adding the three emulation differences to the meta-regression reduced heterogeneity from 1.9 to almost 1 (absence of heterogeneity). Conclusions: This analysis suggests that a substantial proportion of the observed variation between results from randomised controlled trials and real world evidence studies can be attributed to differences in design emulation.

DESIGN DIFFERENCES EXPLAIN VARIATION IN RESULTS BETWEEN RANDOMIZED TRIALS AND THEIR NON-RANDOMIZED EMULATIONS.

Objectives: While randomized controlled trials (RCTs) are considered a standard for evidence on the efficacy of medical treatments, non-randomized real-world evidence (RWE) studies using data from health insurance claims or electronic health records can provide important complementary evidence. The use of RWE to inform decision-making has been questioned because of concerns regarding confounding in non-randomized studies and the use of secondary data. RCT-DUPLICATE was a demonstration project that emulated the design of 32 RCTs with non-randomized RWE studies. We sought to explore how emulation differences relate to variation in results between the RCT-RWE study pairs. Methods: We include all RCT-RWE study pairs from RCT-DUPLICATE where the measure of effect was a hazard ratio and use exploratory meta-regression methods to explain differences and variation in the effect sizes between the results from the RCT and the RWE study. The considered explanatory variables are related to design and population differences. Results: Most of the observed variation in effect estimates between RCT-RWE study pairs in this sample could be explained by three emulation differences in the meta-regression model: (i) in-hospital start of treatment (not observed in claims data), (ii) discontinuation of certain baseline therapies at randomization (not part of clinical practice), (iii) delayed onset of drug effects (missed by short medication persistence in clinical practice). Conclusions: This analysis suggests that a substantial proportion of the observed variation between results from RCTs and RWE studies can be attributed to design emulation differences. (238 words).

Using an expert survey and user feedback to construct PRECHECK: A checklist to evaluate preprints on COVID-19 and beyond.

Background: The quality of COVID-19 preprints should be considered with great care, as their contents can influence public policy. Surprisingly little has been done to calibrate the public's evaluation of preprints and their contents. The PRECHECK project aimed to generate a tool to teach and guide scientifically literate non-experts to critically evaluate preprints, on COVID-19 and beyond. Methods: To create a checklist, we applied a four-step procedure consisting of an initial internal review, an external review by a pool of experts (methodologists, meta-researchers/experts on preprints, journal editors, and science journalists), a final internal review, and a Preliminary implementation stage. For the external review step, experts rated the relevance of each element of the checklist on five-point Likert scales, and provided written feedback. After each internal review round, we applied the checklist on a small set of high-quality preprints from an online list of milestone research works on COVID-19 and low-quality preprints, which were eventually retracted, to verify whether the checklist can discriminate between the two categories. Results: At the external review step, 26 of the 54 contacted experts responded. The final checklist contained four elements (Research question, study type, transparency and integrity, and limitations), with 'superficial' and 'deep' evaluation levels. When using both levels, the checklist was effective at discriminating a small set of high- and low-quality preprints. Its usability for assessment and discussion of preprints was confirmed in workshops with Bachelors students in Psychology and Medicine, and science journalists. Conclusions: We created a simple, easy-to-use tool for helping scientifically literate non-experts navigate preprints with a critical mind and facilitate discussions within, for example, a beginner-level lecture on research methods. We believe that our checklist has potential to help guide decisions about the quality of preprints on COVID-19 in our target audience and that this extends beyond COVID-19.

Face-to-face panel meetings versus remote evaluation of fellowship applications: simulation study at the Swiss National Science Foundation.

OBJECTIVES: To trial a simplified, time and cost-saving method for remote evaluation of fellowship applications and compare this with existing panel review processes by analysing concordance between funding decisions, and the use of a lottery-based decision method for proposals of similar quality. DESIGN: The study involved 134 junior fellowship proposals for postdoctoral research ('Postdoc.Mobility'). The official method used two panel reviewers who independently scored the application, followed by triage and discussion of selected applications in a panel. Very competitive/uncompetitive proposals were directly funded/rejected without discussion. The simplified procedure used the scores of the two panel members, with or without the score of an additional, third expert. Both methods could further use a lottery to decide on applications of similar quality close to the funding threshold. The same funding rate was applied, and the agreement between the two methods analysed. SETTING: Swiss National Science Foundation (SNSF). PARTICIPANTS: Postdoc.Mobility panel reviewers and additional expert reviewers. PRIMARY OUTCOME MEASURE: Per cent agreement between the simplified and official evaluation method with 95% CIs. RESULTS: The simplified procedure based on three reviews agreed in 80.6% (95% CI: 73.9% to 87.3%) of applicants with the official funding outcome. The agreement was 86.6% (95% CI: 80.6% to 91.8%) when using the two reviews of the panel members. The agreement between the two methods was lower for the group of applications discussed in the panel (64.2% and 73.1%, respectively), and higher for directly funded/rejected applications (range: 96.7%-100%). The lottery was used in 8 (6.0%) of 134 applications (official method), 19 (14.2%) applications (simplified, three reviewers) and 23 (17.2%) applications (simplified, two reviewers). With the simplified procedure, evaluation costs could have been halved and 31 hours of meeting time saved for the two 2019 calls. CONCLUSION: Agreement between the two methods was high. The simplified procedure could represent a viable evaluation method for the Postdoc.Mobility early career instrument at the SNSF.

Gender and other potential biases in peer review: cross-sectional analysis of 38 250 external peer review reports.

OBJECTIVES: To examine whether the gender of applicants and peer reviewers and other factors influence peer review of grant proposals submitted to a national funding agency. SETTING: Swiss National Science Foundation (SNSF). DESIGN: Cross-sectional analysis of peer review reports submitted from 2009 to 2016 using linear mixed effects regression models adjusted for research topic, applicant's age, nationality, affiliation and calendar period. PARTICIPANTS: External peer reviewers. PRIMARY OUTCOME MEASURE: Overall score on a scale from 1 (worst) to 6 (best). RESULTS: Analyses included 38 250 reports on 12 294 grant applications from medicine, architecture, biology, chemistry, economics, engineering, geology, history, linguistics, mathematics, physics, psychology and sociology submitted by 26 829 unique peer reviewers. In univariable analysis, male applicants received more favourable evaluation scores than female applicants (+0.18 points; 95% CI 0.14 to 0.23), and male reviewers awarded higher scores than female reviewers (+0.11; 95% CI 0.08 to 0.15). Applicant-nominated reviewers awarded higher scores than reviewers nominated by the SNSF (+0.53; 95% CI 0.50 to 0.56), and reviewers from outside of Switzerland more favourable scores than reviewers affiliated with Swiss institutions (+0.53; 95% CI 0.49 to 0.56). In multivariable analysis, differences between male and female applicants were attenuated (+0.08; 95% CI 0.04 to 0.13) whereas results changed little for source of nomination and affiliation of reviewers. The gender difference increased after September 2011, when new evaluation forms were introduced (p=0.033 from test of interaction). CONCLUSIONS: Peer review of grant applications at SNSF might be prone to biases stemming from different applicant and reviewer characteristics. The SNSF abandoned the nomination of peer reviewers by applicants. The new form introduced in 2011 may inadvertently have given more emphasis to the applicant's track record. We encourage other funders to conduct similar studies, in order to improve the evidence base for rational and fair research funding.

Validation of discrete time-to-event prediction models in the presence of competing risks.

Clinical prediction models play a key role in risk stratification, therapy assignment and many other fields of medical decision making. Before they can enter clinical practice, their usefulness has to be demonstrated using systematic validation. Methods to assess their predictive performance have been proposed for continuous, binary, and time-to-event outcomes, but the literature on validation methods for discrete time-to-event models with competing risks is sparse. The present paper tries to fill this gap and proposes new methodology to quantify discrimination, calibration, and prediction error (PE) for discrete time-to-event outcomes in the presence of competing risks. In our case study, the goal was to predict the risk of ventilator-associated pneumonia (VAP) attributed to Pseudomonas aeruginosa in intensive care units (ICUs). Competing events are extubation, death, and VAP due to other bacteria. The aim of this application is to validate complex prediction models developed in previous work on more recently available validation data.

Dynamic clinical prediction models for discrete time-to-event data with competing risks-A case study on the OUTCOMEREA database.

The development of clinical prediction models requires the selection of suitable predictor variables. Techniques to perform objective Bayesian variable selection in the linear model are well developed and have been extended to the generalized linear model setting as well as to the Cox proportional hazards model. Here, we consider discrete time-to-event data with competing risks and propose methodology to develop a clinical prediction model for the daily risk of acquiring a ventilator-associated pneumonia (VAP) attributed to P. aeruginosa (PA) in intensive care units. The competing events for a PA VAP are extubation, death, and VAP due to other bacteria. Baseline variables are potentially important to predict the outcome at the start of ventilation, but may lose some of their predictive power after a certain time. Therefore, we use a landmark approach for dynamic Bayesian variable selection where the set of relevant predictors depends on the time already spent at risk. We finally determine the direct impact of a variable on each competing event through cause-specific variable selection.