Community consensus on core open science practices to monitor in biomedicine.

The state of open science needs to be monitored to track changes over time and identify areas to create interventions to drive improvements. In order to monitor open science practices, they first need to be well defined and operationalized. To reach consensus on what open science practices to monitor at biomedical research institutions, we conducted a modified 3-round Delphi study. Participants were research administrators, researchers, specialists in dedicated open science roles, and librarians. In rounds 1 and 2, participants completed an online survey evaluating a set of potential open science practices, and for round 3, we hosted two half-day virtual meetings to discuss and vote on items that had not reached consensus. Ultimately, participants reached consensus on 19 open science practices. This core set of open science practices will form the foundation for institutional dashboards and may also be of value for the development of policy, education, and interventions.

Epidemiology and reporting characteristics of preclinical systematic reviews.

In an effort to better utilize published evidence obtained from animal experiments, systematic reviews of preclinical studies are increasingly more common-along with the methods and tools to appraise them (e.g., SYstematic Review Center for Laboratory animal Experimentation [SYRCLE's] risk of bias tool). We performed a cross-sectional study of a sample of recent preclinical systematic reviews (2015-2018) and examined a range of epidemiological characteristics and used a 46-item checklist to assess reporting details. We identified 442 reviews published across 43 countries in 23 different disease domains that used 26 animal species. Reporting of key details to ensure transparency and reproducibility was inconsistent across reviews and within article sections. Items were most completely reported in the title, introduction, and results sections of the reviews, while least reported in the methods and discussion sections. Less than half of reviews reported that a risk of bias assessment for internal and external validity was undertaken, and none reported methods for evaluating construct validity. Our results demonstrate that a considerable number of preclinical systematic reviews investigating diverse topics have been conducted; however, their quality of reporting is inconsistent. Our study provides the justification and evidence to inform the development of guidelines for conducting and reporting preclinical systematic reviews.

The Automated Systematic Search Deduplicator (ASySD): a rapid, open-source, interoperable tool to remove duplicate citations in biomedical systematic reviews.

BACKGROUND: Researchers performing high-quality systematic reviews search across multiple databases to identify relevant evidence. However, the same publication is often retrieved from several databases. Identifying and removing such duplicates ("deduplication") can be extremely time-consuming, but failure to remove these citations can lead to the wrongful inclusion of duplicate data. Many existing tools are not sensitive enough, lack interoperability with other tools, are not freely accessible, or are difficult to use without programming knowledge. Here, we report the performance of our Automated Systematic Search Deduplicator (ASySD), a novel tool to perform automated deduplication of systematic searches for biomedical reviews. METHODS: We evaluated ASySD's performance on 5 unseen biomedical systematic search datasets of various sizes (1845-79,880 citations). We compared the performance of ASySD with EndNote's automated deduplication option and with the Systematic Review Assistant Deduplication Module (SRA-DM). RESULTS: ASySD identified more duplicates than either SRA-DM or EndNote, with a sensitivity in different datasets of 0.95 to 0.99. The false-positive rate was comparable to human performance, with a specificity of > 0.99. The tool took less than 1 h to identify and remove duplicates within each dataset. CONCLUSIONS: For duplicate removal in biomedical systematic reviews, ASySD is a highly sensitive, reliable, and time-saving tool. It is open source and freely available online as both an R package and a user-friendly web application.

Screening for in vitro systematic reviews: a comparison of screening methods and training of a machine learning classifier.

OBJECTIVE: Existing strategies to identify relevant studies for systematic review may not perform equally well across research domains. We compare four approaches based on either human or automated screening of either title and abstract or full text, and report the training of a machine learning algorithm to identify in vitro studies from bibliographic records. METHODS: We used a systematic review of oxygen-glucose deprivation (OGD) in PC-12 cells to compare approaches. For human screening, two reviewers independently screened studies based on title and abstract or full text, with disagreements reconciled by a third. For automated screening, we applied text mining to either title and abstract or full text. We trained a machine learning algorithm with decisions from 2000 randomly selected PubMed Central records enriched with a dataset of known in vitro studies. RESULTS: Full-text approaches performed best, with human (sensitivity: 0.990, specificity: 1.000 and precision: 0.994) outperforming text mining (sensitivity: 0.972, specificity: 0.980 and precision: 0.764). For title and abstract, text mining (sensitivity: 0.890, specificity: 0.995 and precision: 0.922) outperformed human screening (sensitivity: 0.862, specificity: 0.998 and precision: 0.975). At our target sensitivity of 95% the algorithm performed with specificity of 0.850 and precision of 0.700. CONCLUSION: In this in vitro systematic review, human screening based on title and abstract erroneously excluded 14% of relevant studies, perhaps because title and abstract provide an incomplete description of methods used. Our algorithm might be used as a first selection phase in in vitro systematic reviews to limit the extent of full text screening required.

The ARRIVE guidelines 2.0: Updated guidelines for reporting animal research.

Reproducible science requires transparent reporting. The ARRIVE guidelines (Animal Research: Reporting of In Vivo Experiments) were originally developed in 2010 to improve the reporting of animal research. They consist of a checklist of information to include in publications describing in vivo experiments to enable others to scrutinise the work adequately, evaluate its methodological rigour, and reproduce the methods and results. Despite considerable levels of endorsement by funders and journals over the years, adherence to the guidelines has been inconsistent, and the anticipated improvements in the quality of reporting in animal research publications have not been achieved. Here, we introduce ARRIVE 2.0. The guidelines have been updated and information reorganised to facilitate their use in practice. We used a Delphi exercise to prioritise and divide the items of the guidelines into 2 sets, the "ARRIVE Essential 10," which constitutes the minimum requirement, and the "Recommended Set," which describes the research context. This division facilitates improved reporting of animal research by supporting a stepwise approach to implementation. This helps journal editors and reviewers verify that the most important items are being reported in manuscripts. We have also developed the accompanying Explanation and Elaboration (E&E) document, which serves (1) to explain the rationale behind each item in the guidelines, (2) to clarify key concepts, and (3) to provide illustrative examples. We aim, through these changes, to help ensure that researchers, reviewers, and journal editors are better equipped to improve the rigour and transparency of the scientific process and thus reproducibility.

Reporting animal research: Explanation and elaboration for the ARRIVE guidelines 2.0.

Improving the reproducibility of biomedical research is a major challenge. Transparent and accurate reporting is vital to this process; it allows readers to assess the reliability of the findings and repeat or build upon the work of other researchers. The ARRIVE guidelines (Animal Research: Reporting In Vivo Experiments) were developed in 2010 to help authors and journals identify the minimum information necessary to report in publications describing in vivo experiments. Despite widespread endorsement by the scientific community, the impact of ARRIVE on the transparency of reporting in animal research publications has been limited. We have revised the ARRIVE guidelines to update them and facilitate their use in practice. The revised guidelines are published alongside this paper. This explanation and elaboration document was developed as part of the revision. It provides further information about each of the 21 items in ARRIVE 2.0, including the rationale and supporting evidence for their inclusion in the guidelines, elaboration of details to report, and examples of good reporting from the published literature. This document also covers advice and best practice in the design and conduct of animal studies to support researchers in improving standards from the start of the experimental design process through to publication.

Animal models of chemotherapy-induced peripheral neuropathy: A machine-assisted systematic review and meta-analysis.

We report a systematic review and meta-analysis of research using animal models of chemotherapy-induced peripheral neuropathy (CIPN). We systematically searched 5 online databases in September 2012 and updated the search in November 2015 using machine learning and text mining to reduce the screening for inclusion workload and improve accuracy. For each comparison, we calculated a standardised mean difference (SMD) effect size, and then combined effects in a random-effects meta-analysis. We assessed the impact of study design factors and reporting of measures to reduce risks of bias. We present power analyses for the most frequently reported behavioural tests; 337 publications were included. Most studies (84%) used male animals only. The most frequently reported outcome measure was evoked limb withdrawal in response to mechanical monofilaments. There was modest reporting of measures to reduce risks of bias. The number of animals required to obtain 80% power with a significance level of 0.05 varied substantially across behavioural tests. In this comprehensive summary of the use of animal models of CIPN, we have identified areas in which the value of preclinical CIPN studies might be increased. Using both sexes of animals in the modelling of CIPN, ensuring that outcome measures align with those most relevant in the clinic, and the animal's pain contextualised ethology will likely improve external validity. Measures to reduce risk of bias should be employed to increase the internal validity of studies. Different outcome measures have different statistical power, and this can refine our approaches in the modelling of CIPN.

Effectiveness of biomaterial-based combination strategies for spinal cord repair - a systematic review and meta-analysis of preclinical literature.

STUDY DESIGN: Systematic review and meta-analysis of preclinical literature. OBJECTIVES: To assess the effects of biomaterial-based combination (BMC) strategies for the treatment of Spinal Cord Injury (SCI), the effects of individual biomaterials in the context of BMC strategies, and the factors influencing their efficacy. To assess the effects of different preclinical testing paradigms in BMC strategies. METHODS: We performed a systematic literature search of Embase, Web of Science and PubMed. All controlled preclinical studies describing an in vivo or in vitro model of SCI that tested a biomaterial in combination with at least one other regenerative strategy (cells, drugs, or both) were included. Two review authors conducted the study selection independently, extracted study characteristics independently and assessed study quality using a modified CAMARADES checklist. Effect size measures were combined using random-effects models and heterogeneity was explored using meta-regression with tau2, I2 and R2 statistics. We tested for small-study effects using funnel plot-based methods. RESULTS: 134 publications were included, testing over 100 different BMC strategies. Overall, treatment with BMC therapies improved locomotor recovery by 25.3% (95% CI, 20.3-30.3; n = 102) and in vivo axonal regeneration by 1.6 SD (95% CI 1.2-2 SD; n = 117) in comparison with injury only controls. CONCLUSION: BMC strategies improve locomotor outcomes after experimental SCI. Our comprehensive study highlights gaps in current knowledge and provides a foundation for the design of future experiments.

Reproducibility of preclinical animal research improves with heterogeneity of study samples.

Single-laboratory studies conducted under highly standardized conditions are the gold standard in preclinical animal research. Using simulations based on 440 preclinical studies across 13 different interventions in animal models of stroke, myocardial infarction, and breast cancer, we compared the accuracy of effect size estimates between single-laboratory and multi-laboratory study designs. Single-laboratory studies generally failed to predict effect size accurately, and larger sample sizes rendered effect size estimates even less accurate. By contrast, multi-laboratory designs including as few as 2 to 4 laboratories increased coverage probability by up to 42 percentage points without a need for larger sample sizes. These findings demonstrate that within-study standardization is a major cause of poor reproducibility. More representative study samples are required to improve the external validity and reproducibility of preclinical animal research and to prevent wasting animals and resources for inconclusive research.

The bench is closer to the bedside than we think: Uncovering the ethical ties between preclinical researchers in translational neuroscience and patients in clinical trials.

Millions of people worldwide currently suffer from serious neurological diseases and injuries for which there are few, and often no, effective treatments. The paucity of effective interventions is, no doubt, due in large part to the complexity of the disorders, as well as our currently limited understanding of their pathophysiology. The bleak picture for patients, however, is also attributable to avoidable impediments stemming from quality concerns in preclinical research that often escape detection by research regulation efforts. In our essay, we connect the dots between these concerns about the quality of preclinical research and their potential ethical impact on the patients who volunteer for early trials of interventions informed by it. We do so in hopes that a greater appreciation among preclinical researchers of these serious ethical consequences can lead to a greater commitment within the research community to adopt widely available tools and measures that can help to improve the quality of research.

The ARRIVE guidelines 2.0: updated guidelines for reporting animal research.

Reproducible science requires transparent reporting. The ARRIVE guidelines (Animal Research: Reporting of In Vivo Experiments) were originally developed in 2010 to improve the reporting of animal research. They consist of a checklist of information to include in publications describing in vivo experiments to enable others to scrutinise the work adequately, evaluate its methodological rigour, and reproduce the methods and results. Despite considerable levels of endorsement by funders and journals over the years, adherence to the guidelines has been inconsistent, and the anticipated improvements in the quality of reporting in animal research publications have not been achieved. Here, we introduce ARRIVE 2.0. The guidelines have been updated and information reorganised to facilitate their use in practice. We used a Delphi exercise to prioritise and divide the items of the guidelines into 2 sets, the 'ARRIVE Essential 10,' which constitutes the minimum requirement, and the 'Recommended Set,' which describes the research context. This division facilitates improved reporting of animal research by supporting a stepwise approach to implementation. This helps journal editors and reviewers verify that the most important items are being reported in manuscripts. We have also developed the accompanying Explanation and Elaboration document, which serves (1) to explain the rationale behind each item in the guidelines, (2) to clarify key concepts, and (3) to provide illustrative examples. We aim, through these changes, to help ensure that researchers, reviewers, and journal editors are better equipped to improve the rigour and transparency of the scientific process and thus reproducibility.

The ARRIVE guidelines 2.0: Updated guidelines for reporting animal research.

Reproducible science requires transparent reporting. The ARRIVE guidelines (Animal Research: Reporting of In Vivo Experiments) were originally developed in 2010 to improve the reporting of animal research. They consist of a checklist of information to include in publications describing in vivo experiments to enable others to scrutinise the work adequately, evaluate its methodological rigour, and reproduce the methods and results. Despite considerable levels of endorsement by funders and journals over the years, adherence to the guidelines has been inconsistent, and the anticipated improvements in the quality of reporting in animal research publications have not been achieved. Here, we introduce ARRIVE 2.0. The guidelines have been updated and information reorganised to facilitate their use in practice. We used a Delphi exercise to prioritise and divide the items of the guidelines into 2 sets, the "ARRIVE Essential 10," which constitutes the minimum requirement, and the "Recommended Set," which describes the research context. This division facilitates improved reporting of animal research by supporting a stepwise approach to implementation. This helps journal editors and reviewers verify that the most important items are being reported in manuscripts. We have also developed the accompanying Explanation and Elaboration document, which serves (1) to explain the rationale behind each item in the guidelines, (2) to clarify key concepts, and (3) to provide illustrative examples. We aim, through these changes, to help ensure that researchers, reviewers, and journal editors are better equipped to improve the rigour and transparency of the scientific process and thus reproducibility.

The ARRIVE guidelines 2.0: Updated guidelines for reporting animal research.

Reproducible science requires transparent reporting. The ARRIVE guidelines (Animal Research: Reporting of In Vivo Experiments) were originally developed in 2010 to improve the reporting of animal research. They consist of a checklist of information to include in publications describing in vivo experiments to enable others to scrutinise the work adequately, evaluate its methodological rigour, and reproduce the methods and results. Despite considerable levels of endorsement by funders and journals over the years, adherence to the guidelines has been inconsistent, and the anticipated improvements in the quality of reporting in animal research publications have not been achieved. Here, we introduce ARRIVE 2.0. The guidelines have been updated and information reorganised to facilitate their use in practice. We used a Delphi exercise to prioritise and divide the items of the guidelines into 2 sets, the "ARRIVE Essential 10," which constitutes the minimum requirement, and the "Recommended Set," which describes the research context. This division facilitates improved reporting of animal research by supporting a stepwise approach to implementation. This helps journal editors and reviewers verify that the most important items are being reported in manuscripts. We have also developed the accompanying Explanation and Elaboration document, which serves (1) to explain the rationale behind each item in the guidelines, (2) to clarify key concepts, and (3) to provide illustrative examples. We aim, through these changes, to help ensure that researchers, reviewers, and journal editors are better equipped to improve the rigour and transparency of the scientific process and thus reproducibility.

Design of Meta-Analysis Studies.

Any given research claim can be made with a degree of confidence that a phenomenon is present, with an estimate of the precision of the observed effects and a prediction of the extent to which the findings might hold true under different experimental or real-world conditions. In some situations, the certainty and precision obtained from a single study are sufficient reliably to inform future research decisions. However, in other situations greater certainty is required. This might be the case where a substantial research investment is planned, a pivotal claim is to be made or the launch of a clinical trial programme is being considered. Under these circumstances, some form of summary of findings across studies may be helpful.Summary estimates can describe findings from exploratory (observational) or hypothesis testing experiments, but importantly, the creation of such summaries is, in itself, observational rather than experimental research. The process is therefore particularly at risk from selective identification of literature to be included, and this can be addressed using systematic search strategies and pre-specified criteria for inclusion and exclusion against which possible contributing data will be assessed. This characterises a systematic review (in contrast to nonsystematic or narrative reviews). In meta-analysis, there is an attempt to provide a quantitative summary of such research findings.

Olfactory Ensheathing Cell Transplantation in Experimental Spinal Cord Injury: Effect size and Reporting Bias of 62 Experimental Treatments: A Systematic Review and Meta-Analysis.

Olfactory ensheathing cell (OEC) transplantation is a candidate cellular treatment approach for human spinal cord injury (SCI) due to their unique regenerative potential and autologous origin. The objective of this study was, through a meta-epidemiologic approach, (i) to assess the efficacy of OEC transplantation on locomotor recovery after traumatic experimental SCI and (ii) to estimate the likelihood of reporting bias and/or missing data. A study protocol was finalized before data collection. Embedded into a systematic review and meta-analysis, we conducted a literature research of databases including PubMed, EMBASE, and ISI Web of Science from 1949/01 to 2014/10 with no language restrictions, screened by two independent investigators. Studies were included if they assessed neurobehavioral improvement after traumatic experimental SCI, administrated no combined interventions, and reported the number of animals in the treatment and control group. Individual effect sizes were pooled using a random effects model. Details regarding the study design were extracted and impact of these on locomotor outcome was assessed by meta-regression. Missing data (reporting bias) was determined by Egger regression and Funnel-plotting. The primary study outcome assessed was improvement in locomotor function at the final time point of measurement. We included 49 studies (62 experiments, 1,164 animals) in the final analysis. The overall improvement in locomotor function after OEC transplantation, measured using the Basso, Beattie, and Bresnahan (BBB) score, was 20.3% (95% CI 17.8-29.5). One missing study was imputed by trim and fill analysis, suggesting only slight publication bias and reducing the overall effect to a 19.2% improvement of locomotor activity. Dose-response ratio supports neurobiological plausibility. Studies were assessed using a 9-point item quality score, resulting in a median score of 5 (interquartile range [IQR] 3-5). In conclusion, OEC transplantation exerts considerable beneficial effects on neurobehavioral recovery after traumatic experimental SCI. Publication bias was minimal and affirms the translational potential of efficacy, but safety cannot be adequately assessed. The data justify OECs as a cellular substrate to develop and optimize minimally invasive and safe cellular transplantation paradigms for the lesioned spinal cord embedded into state-of-the-art Phase I/II clinical trial design studies for human SCI.

Correction: Risk of Bias in Reports of In Vivo Research: A Focus for Improvement.

[This corrects the article DOI: 10.1371/journal.pbio.1002273.].

Risk of Bias in Reports of In Vivo Research: A Focus for Improvement.

The reliability of experimental findings depends on the rigour of experimental design. Here we show limited reporting of measures to reduce the risk of bias in a random sample of life sciences publications, significantly lower reporting of randomisation in work published in journals of high impact, and very limited reporting of measures to reduce the risk of bias in publications from leading United Kingdom institutions. Ascertainment of differences between institutions might serve both as a measure of research quality and as a tool for institutional efforts to improve research quality.

Cardiac Stem Cell Treatment in Myocardial Infarction: A Systematic Review and Meta-Analysis of Preclinical Studies.

RATIONALE: Cardiac stem cells (CSC) therapy has been clinically introduced for cardiac repair after myocardial infarction (MI). To date, there has been no systematic overview and meta-analysis of studies using CSC therapy for MI. OBJECTIVE: Here, we used meta-analysis to establish the overall effect of CSCs in preclinical studies and assessed translational differences between and within large and small animals in the CSC therapy field. In addition, we explored the effect of CSC type and other clinically relevant parameters on functional outcome to better predict and design future (pre)clinical studies using CSCs for MI. METHODS AND RESULTS: A systematic search was performed, yielding 80 studies. We determined the overall effect of CSC therapy on left ventricular ejection fraction and performed meta-regression to investigate clinically relevant parameters. We also assessed the quality of included studies and possible bias. The overall effect observed in CSC-treated animals was 10.7% (95% confidence interval 9.4-12.1; P<0.001) improvement in ejection fraction compared with placebo controls. Interestingly, CSC therapy had a greater effect in small animals compared with large animals (P<0.001). Meta-regression indicated that cell type was a significant predictor for ejection fraction improvement in small animals. Minor publication bias was observed in small animal studies. CONCLUSIONS: CSC treatment resulted in significant improvement of ejection fraction in preclinical animal models of MI compared with placebo. There was a reduction in the magnitude of effect in large compared with small animal models. Although different CSC types have overlapping culture characteristics, we observed a significant difference in their effect in post-MI animal studies.

Risk of bias reporting in the recent animal focal cerebral ischaemia literature.

BACKGROUND: Findings from in vivo research may be less reliable where studies do not report measures to reduce risks of bias. The experimental stroke community has been at the forefront of implementing changes to improve reporting, but it is not known whether these efforts are associated with continuous improvements. Our aims here were firstly to validate an automated tool to assess risks of bias in published works, and secondly to assess the reporting of measures taken to reduce the risk of bias within recent literature for two experimental models of stroke. METHODS: We developed and used text analytic approaches to automatically ascertain reporting of measures to reduce risk of bias from full-text articles describing animal experiments inducing middle cerebral artery occlusion (MCAO) or modelling lacunar stroke. RESULTS: Compared with previous assessments, there were improvements in the reporting of measures taken to reduce risks of bias in the MCAO literature but not in the lacunar stroke literature. Accuracy of automated annotation of risk of bias in the MCAO literature was 86% (randomization), 94% (blinding) and 100% (sample size calculation); and in the lacunar stroke literature accuracy was 67% (randomization), 91% (blinding) and 96% (sample size calculation). DISCUSSION: There remains substantial opportunity for improvement in the reporting of animal research modelling stroke, particularly in the lacunar stroke literature. Further, automated tools perform sufficiently well to identify whether studies report blinded assessment of outcome, but improvements are required in the tools to ascertain whether randomization and a sample size calculation were reported.

Identification and characterization of outcome measures reported in animal models of epilepsy: Protocol for a systematic review of the literature-A TASK2 report of the AES/ILAE Translational Task Force of the ILAE.

Current antiseizure therapy is ineffective in approximately one third of people with epilepsy and is often associated with substantial side effects. In addition, most current therapeutic paradigms offer treatment, but not cure, and no therapies are able to modify the underlying disease, that is, can prevent or halt the process of epileptogenesis or alleviate the cognitive and psychiatric comorbidities. Preclinical research in the field of epilepsy has been extensive, but unfortunately, not all the animal models being used have been validated for their predictive value. The overall goal of TASK2 of the AES/ILAE Translational Task Force is to organize and coordinate systematic reviews on selected topics regarding animal research in epilepsy. Herein we describe our strategy. In the first part of the paper we provide an overview of the usefulness of systematic reviews and meta-analysis for preclinical research and explain the essentials for their conduct. Then we describe in detail the protocol for a first systematic review, which will focus on the identification and characterization of outcome measures reported in animal models of epilepsy. The specific goals of this study are to define systematically the phenotypic characteristics of the most commonly used animal models, and to effectively compare these with the manifestations of human epilepsy. This will provide epilepsy researchers with detailed information on the strengths and weaknesses of epilepsy models, facilitating their refinement and future research. Ultimately, this could lead to a refined use of relevant models for understanding the mechanism(s) of the epilepsies and developing novel therapies.