RESUMEN
OBJECTIVE: Existing strategies to identify relevant studies for systematic review may not perform equally well across research domains. We compare four approaches based on either human or automated screening of either title and abstract or full text, and report the training of a machine learning algorithm to identify in vitro studies from bibliographic records. METHODS: We used a systematic review of oxygen-glucose deprivation (OGD) in PC-12 cells to compare approaches. For human screening, two reviewers independently screened studies based on title and abstract or full text, with disagreements reconciled by a third. For automated screening, we applied text mining to either title and abstract or full text. We trained a machine learning algorithm with decisions from 2000 randomly selected PubMed Central records enriched with a dataset of known in vitro studies. RESULTS: Full-text approaches performed best, with human (sensitivity: 0.990, specificity: 1.000 and precision: 0.994) outperforming text mining (sensitivity: 0.972, specificity: 0.980 and precision: 0.764). For title and abstract, text mining (sensitivity: 0.890, specificity: 0.995 and precision: 0.922) outperformed human screening (sensitivity: 0.862, specificity: 0.998 and precision: 0.975). At our target sensitivity of 95% the algorithm performed with specificity of 0.850 and precision of 0.700. CONCLUSION: In this in vitro systematic review, human screening based on title and abstract erroneously excluded 14% of relevant studies, perhaps because title and abstract provide an incomplete description of methods used. Our algorithm might be used as a first selection phase in in vitro systematic reviews to limit the extent of full text screening required.
Asunto(s)
Algoritmos , Minería de Datos , Humanos , Minería de Datos/métodos , Proyectos de Investigación , Aprendizaje Automático , GlucosaRESUMEN
Increasing the reproducibility and trustworthiness of biomedical research requires engaging stakeholders from all levels in an institutional setting. The QUEST Center for Responsible Research aims to develop and implement new approaches to improve the culture and practice of research, tailored to the needs of these stakeholders. Members of the QUEST Center organised a brainstorm to reflect on the challenges and new opportunities encountered in implementing different projects through QUEST and share the lessons that working groups have learned over the first five years. The authors informally surveyed and interviewed working groups where relevant and highlight common themes that have influenced the success of many projects, including top-down and bottom-up engagement, managing expectations, the availability of expertise, ensuring sustainability, and considering incentives. The commentary authors conclude by encouraging the research community to view initiatives that promote reproducibility not as a one-size-fits-all undertaking, but rather as an opportunity to unite stakeholders and customise drivers of cultural change.
Asunto(s)
Investigación Biomédica , Reproducibilidad de los ResultadosRESUMEN
STUDY DESIGN: Systematic review and meta-analysis of preclinical literature. OBJECTIVES: To assess the effects of biomaterial-based combination (BMC) strategies for the treatment of Spinal Cord Injury (SCI), the effects of individual biomaterials in the context of BMC strategies, and the factors influencing their efficacy. To assess the effects of different preclinical testing paradigms in BMC strategies. METHODS: We performed a systematic literature search of Embase, Web of Science and PubMed. All controlled preclinical studies describing an in vivo or in vitro model of SCI that tested a biomaterial in combination with at least one other regenerative strategy (cells, drugs, or both) were included. Two review authors conducted the study selection independently, extracted study characteristics independently and assessed study quality using a modified CAMARADES checklist. Effect size measures were combined using random-effects models and heterogeneity was explored using meta-regression with tau2, I2 and R2 statistics. We tested for small-study effects using funnel plot-based methods. RESULTS: 134 publications were included, testing over 100 different BMC strategies. Overall, treatment with BMC therapies improved locomotor recovery by 25.3% (95% CI, 20.3-30.3; n = 102) and in vivo axonal regeneration by 1.6 SD (95% CI 1.2-2 SD; n = 117) in comparison with injury only controls. CONCLUSION: BMC strategies improve locomotor outcomes after experimental SCI. Our comprehensive study highlights gaps in current knowledge and provides a foundation for the design of future experiments.
Asunto(s)
Traumatismos de la Médula Espinal , Regeneración de la Medula Espinal , Animales , Humanos , Traumatismos de la Médula Espinal/terapia , Materiales Biocompatibles/uso terapéutico , Modelos Animales de Enfermedad , Procedimientos NeuroquirúrgicosRESUMEN
A spectre is haunting biomedical research: It appears that a substantial fraction of published research results cannot be reproduced, while spectacularly successful novel treatments developed in experimental models of disease too often fail in clinical trials. A reproducibility crisis has been proclaimed, and bench-to-bedside translation appears to be lost in a "valley of death". Both predicaments, non-reproducibility and translational roadblocks, are connected: Why should we expect to successfully "trans-late" results to humans, if already "cis-lation"-that is, the generalization from one experimental setting to an identical or fairly similar one-often fails?
Asunto(s)
Investigación Biomédica , Investigación Biomédica Traslacional , Investigación Biomédica/métodos , Humanos , Modelos Teóricos , Reproducibilidad de los ResultadosRESUMEN
The replicability of research results has been a cause of increasing concern to the scientific community. The long-held belief that experimental standardization begets replicability has also been recently challenged, with the observation that the reduction of variability within studies can lead to idiosyncratic, lab-specific results that cannot be replicated. An alternative approach is to, instead, deliberately introduce heterogeneity, known as "heterogenization" of experimental design. Here, we explore a novel perspective in the heterogenization program in a meta-analysis of variability in observed phenotypic outcomes in both control and experimental animal models of ischemic stroke. First, by quantifying interindividual variability across control groups, we illustrate that the amount of heterogeneity in disease state (infarct volume) differs according to methodological approach, for example, in disease induction methods and disease models. We argue that such methods may improve replicability by creating diverse and representative distribution of baseline disease state in the reference group, against which treatment efficacy is assessed. Second, we illustrate how meta-analysis can be used to simultaneously assess efficacy and stability (i.e., mean effect and among-individual variability). We identify treatments that have efficacy and are generalizable to the population level (i.e., low interindividual variability), as well as those where there is high interindividual variability in response; for these, latter treatments translation to a clinical setting may require nuance. We argue that by embracing rather than seeking to minimize variability in phenotypic outcomes, we can motivate the shift toward heterogenization and improve both the replicability and generalizability of preclinical research.
Asunto(s)
Experimentación Animal/normas , Proyectos de Investigación/normas , Animales , Conducta Animal/fisiología , Isquemia Encefálica/metabolismo , Humanos , Metaanálisis como Asunto , Modelos Animales , Fenotipo , Estándares de Referencia , Reproducibilidad de los Resultados , Proyectos de Investigación/tendencias , Accidente Cerebrovascular/fisiopatologíaRESUMEN
In an effort to better utilize published evidence obtained from animal experiments, systematic reviews of preclinical studies are increasingly more common-along with the methods and tools to appraise them (e.g., SYstematic Review Center for Laboratory animal Experimentation [SYRCLE's] risk of bias tool). We performed a cross-sectional study of a sample of recent preclinical systematic reviews (2015-2018) and examined a range of epidemiological characteristics and used a 46-item checklist to assess reporting details. We identified 442 reviews published across 43 countries in 23 different disease domains that used 26 animal species. Reporting of key details to ensure transparency and reproducibility was inconsistent across reviews and within article sections. Items were most completely reported in the title, introduction, and results sections of the reviews, while least reported in the methods and discussion sections. Less than half of reviews reported that a risk of bias assessment for internal and external validity was undertaken, and none reported methods for evaluating construct validity. Our results demonstrate that a considerable number of preclinical systematic reviews investigating diverse topics have been conducted; however, their quality of reporting is inconsistent. Our study provides the justification and evidence to inform the development of guidelines for conducting and reporting preclinical systematic reviews.
Asunto(s)
Revisión de la Investigación por Pares/métodos , Revisión de la Investigación por Pares/normas , Proyectos de Investigación/normas , Experimentación Animal/normas , Animales , Sesgo , Lista de Verificación/normas , Evaluación Preclínica de Medicamentos/métodos , Evaluación Preclínica de Medicamentos/normas , Investigación Empírica , Métodos Epidemiológicos , Epidemiología/tendencias , Humanos , Revisión de la Investigación por Pares/tendencias , Publicaciones , Reproducibilidad de los Resultados , Proyectos de Investigación/tendenciasRESUMEN
Preclinical research is a vital step in the drug discovery pipeline and more generally in helping to better understand human disease aetiology and its management. Systematic reviews (SRs) can be powerful in summarising and appraising this evidence concerning a specific research question, to highlight areas of improvements, areas for further research and areas where evidence may be sufficient to take forward to other research domains, for instance clinical trial. Guidance and tools for preclinical research synthesis remain limited despite their clear utility. We aimed to create an online end-to-end platform primarily for conducting SRs of preclinical studies, that was flexible enough to support a wide variety of experimental designs, was adaptable to different research questions, would allow users to adopt emerging automated tools and support them during their review process using best practice. In this article, we introduce the Systematic Review Facility (https://syrf.org.uk), which was launched in 2016 and designed to support primarily preclinical SRs from small independent projects to large, crowdsourced projects. We discuss the architecture of the app and its features, including the opportunity to collaborate easily, to efficiently manage projects, to screen and annotate studies for important features (metadata), to extract outcome data into a secure database, and tailor these steps to each project. We introduce how we are working to leverage the use of automation tools and allow the integration of these services to accelerate and automate steps in the systematic review workflow.
RESUMEN
BACKGROUND: Meta-analysis of preclinical data is used to evaluate the consistency of findings and to inform the design and conduct of future studies. Unlike clinical meta-analysis, preclinical data often involve many heterogeneous studies reporting outcomes from a small number of animals. Here, we review the methodological challenges in preclinical meta-analysis in estimating and explaining heterogeneity in treatment effects. METHODS: Assuming aggregate-level data, we focus on two topics: (1) estimation of heterogeneity using commonly used methods in preclinical meta-analysis: method of moments (DerSimonian and Laird; DL), maximum likelihood (restricted maximum likelihood; REML) and Bayesian approach; (2) comparison of univariate versus multivariable meta-regression for adjusting estimated treatment effects for heterogeneity. Using data from a systematic review on the efficacy of interleukin-1 receptor antagonist in animals with stroke, we compare these methods, and explore the impact of multiple covariates on the treatment effects. RESULTS: We observed that the three methods for estimating heterogeneity yielded similar estimates for the overall effect, but different estimates for between-study variability. The proportion of heterogeneity explained by a covariate is estimated larger using REML and the Bayesian method as compared with DL. Multivariable meta-regression explains more heterogeneity than univariate meta-regression. CONCLUSIONS: Our findings highlight the importance of careful selection of the estimation method and the use of multivariable meta-regression to explain heterogeneity. There was no difference between REML and the Bayesian method and both methods are recommended over DL. Multiple meta-regression is worthwhile to explain heterogeneity by more than one variable, reducing more variability than any univariate models and increasing the explained proportion of heterogeneity.
RESUMEN
Conventional vaccine design has been based on trial-and-error approaches, which have been generally successful. However, there have been some major failures in vaccine development and we still do not have highly effective licensed vaccines for tuberculosis, HIV, respiratory syncytial virus, and other major infections of global significance. Approaches at rational vaccine design have been limited by our understanding of the immune response to vaccination at the molecular level. Tools now exist to undertake in-depth analysis using systems biology approaches, but to be fully realized, studies are required in humans with intensive blood and tissue sampling. Methods that support this intensive sampling need to be developed and validated as feasible. To this end, we describe here a detailed approach that was applied in a study of 15 healthy adults, who were immunized with hepatitis B vaccine. Sampling included ~350 mL of blood, 12 microbiome samples, and lymph node fine needle aspirates obtained over a ~7-month period, enabling comprehensive analysis of the immune response at the molecular level, including single cell and tissue sample analysis. Samples were collected for analysis of immune phenotyping, whole blood and single cell gene expression, proteomics, lipidomics, epigenetics, whole blood response to key immune stimuli, cytokine responses, in vitro T cell responses, antibody repertoire analysis and the microbiome. Data integration was undertaken using different approaches-NetworkAnalyst and DIABLO. Our results demonstrate that such intensive sampling studies are feasible in healthy adults, and data integration tools exist to analyze the vast amount of data generated from a multi-omics systems biology approach. This will provide the basis for a better understanding of vaccine-induced immunity and accelerate future rational vaccine design.
Asunto(s)
Vacunas contra Hepatitis B/inmunología , Virus de la Hepatitis B/fisiología , Hepatitis B/diagnóstico , Monitorización Inmunológica/métodos , Vacunación/métodos , Adulto , Anciano , Anciano de 80 o más Años , Femenino , Hepatitis B/inmunología , Humanos , Masculino , Persona de Mediana Edad , Estudios Prospectivos , Biología de Sistemas , Resultado del TratamientoRESUMEN
Stroke is a significant cause of mortality and morbidity for which there are limited treatment options. Virtually all drug interventions that have been successful preclinically in experimental stroke have failed to translate to an effective treatment in the clinical setting. In this review, we examine one of the factors likely contributing to this lack of translation, the failure of preclinical studies to consider fully the advanced age and comorbidities (eg, hypertension or diabetes) present in most patients with stroke. Age and comorbidities affect the likelihood of suffering a stroke, disease progression and the response to treatment. Analysing data from preclinical systematic reviews of interventions for ischaemic stroke we show that only 11.4% of studies included an aged or comorbid model, with hypertension being the most frequent. The degree of protection (% reduction in infarct volume) varied depending on the comorbidity and the type of intervention. We consider reasons for the lack of attention to comorbid and aged animals in stroke research and discuss the value of testing a potential therapy in models representing a range of comorbidities that affect patients with stroke. These models can help establish any limits to a treatment's efficacy and inform the design of clinical trials in appropriate patient populations.
RESUMEN
We report a systematic review and meta-analysis of research using animal models of chemotherapy-induced peripheral neuropathy (CIPN). We systematically searched 5 online databases in September 2012 and updated the search in November 2015 using machine learning and text mining to reduce the screening for inclusion workload and improve accuracy. For each comparison, we calculated a standardised mean difference (SMD) effect size, and then combined effects in a random-effects meta-analysis. We assessed the impact of study design factors and reporting of measures to reduce risks of bias. We present power analyses for the most frequently reported behavioural tests; 337 publications were included. Most studies (84%) used male animals only. The most frequently reported outcome measure was evoked limb withdrawal in response to mechanical monofilaments. There was modest reporting of measures to reduce risks of bias. The number of animals required to obtain 80% power with a significance level of 0.05 varied substantially across behavioural tests. In this comprehensive summary of the use of animal models of CIPN, we have identified areas in which the value of preclinical CIPN studies might be increased. Using both sexes of animals in the modelling of CIPN, ensuring that outcome measures align with those most relevant in the clinic, and the animal's pain contextualised ethology will likely improve external validity. Measures to reduce risk of bias should be employed to increase the internal validity of studies. Different outcome measures have different statistical power, and this can refine our approaches in the modelling of CIPN.
Asunto(s)
Antineoplásicos/efectos adversos , Enfermedades del Sistema Nervioso Periférico/inducido químicamente , Crianza de Animales Domésticos , Animales , Antineoplásicos/administración & dosificación , Conducta Animal , Modelos Animales de Enfermedad , Vías de Administración de Medicamentos , Evaluación de Resultado en la Atención de Salud , Sesgo de Publicación , Publicaciones , Factores de RiesgoRESUMEN
Introduction: Globally, stroke is the second leading cause of death. Despite the burden of illness and death, few acute interventions are available to patients with ischemic stroke. Over 1,000 potential neuroprotective therapeutics have been evaluated in preclinical models. It is important to use robust evidence synthesis methods to appropriately assess which therapies should be translated to the clinical setting for evaluation in human studies. This protocol details planned methods to conduct a systematic review to identify and appraise eligible studies and to use a network meta-analysis to synthesize available evidence to answer the following questions: in preclinical in vivo models of focal ischemic stroke, what are the relative benefits of competing therapies tested in combination with the gold standard treatment alteplase in (i) reducing cerebral infarction size, and (ii) improving neurobehavioural outcomes? Methods: We will search Ovid Medline and Embase for articles on the effects of combination therapies with alteplase. Controlled comparison studies of preclinical in vivo models of experimentally induced focal ischemia testing the efficacy of therapies with alteplase versus alteplase alone will be identified. Outcomes to be extracted include infarct size (primary outcome) and neurobehavioural measures. Risk of bias and construct validity will be assessed using tools appropriate for preclinical studies. Here we describe steps undertaken to perform preclinical network meta-analysis to synthesise all evidence for each outcome and obtain a comprehensive ranking of all treatments. This will be a novel use of this evidence synthesis approach in stroke medicine to assess pre-clinical therapeutics. Combining all evidence to simultaneously compare mutliple therapuetics tested preclinically may provide a rationale for the clinical translation of therapeutics for patients with ischemic stroke. Dissemination: Review findings will be submitted to a peer-reviewed journal and presented at relevant scientific meetings to promote knowledge transfer. Registration: PROSPERO number to be submitted following peer review.
Asunto(s)
Modelos Animales de Enfermedad , Accidente Cerebrovascular/terapia , Animales , Humanos , Metaanálisis como Asunto , Metaanálisis en Red , Proyectos de InvestigaciónRESUMEN
Low translational yield for stroke may reflect the focus of discovery science on rodents rather than humans. Just how little is known about human neuronal ischaemic responses is confirmed by systematic review and meta-analysis revealing that data for the most commonly used SH-SY5Y human cells comprises only 84 papers. Oxygen-glucose deprivation, H2O2, hypoxia, glucose-deprivation and glutamate excitotoxicity yielded - 58, - 61, - 29, - 45 and - 49% injury, respectively, with a dose-response relationship found only for H2O2 injury (R2 = 29.29%, p < 0.002). Heterogeneity (I2 = 99.36%, df = 132, p < 0.0001) was largely attributable to the methods used to detect injury (R2 = 44.77%, p < 0.000) with cell death assays detecting greater injury than survival assays (- 71 vs - 47%, R2 = 28.64%, p < 0.000). Seventy-four percent of publications provided no description of differentiation status, but in the 26% that did, undifferentiated cells were susceptible to greater injury (R2 = 4.13%, p < 0.047). One hundred and sixty-nine interventions improved average survival by 34.67% (p < 0.0001). Eighty-eight comparisons using oxygen-glucose deprivation found both benefit and harm, but studies using glutamate and H2O2 injury reported only improvement. In studies using glucose deprivation, intervention generally worsened outcome. There was insufficient data to rank individual interventions, but of the studies reporting greatest improvement (> 90% effect size), 7/13 were of herbal medicine constituents (24.85% of the intervention dataset). We conclude that surprisingly little is known of the human neuronal response to ischaemic injury, and that the large impact of methodology on outcome indicates that further model validation is required. Lack of evidence for randomisation, blinding or power analysis suggests that the intervention data is at substantial risk of bias.
Asunto(s)
Isquemia/patología , Isquemia/fisiopatología , Bases de Datos Bibliográficas/estadística & datos numéricos , Humanos , Neuroblastoma/patologíaRESUMEN
[This corrects the article DOI: 10.1371/journal.pone.0184122.].
RESUMEN
BACKGROUND: Findings from in vivo research may be less reliable where studies do not report measures to reduce risks of bias. The experimental stroke community has been at the forefront of implementing changes to improve reporting, but it is not known whether these efforts are associated with continuous improvements. Our aims here were firstly to validate an automated tool to assess risks of bias in published works, and secondly to assess the reporting of measures taken to reduce the risk of bias within recent literature for two experimental models of stroke. METHODS: We developed and used text analytic approaches to automatically ascertain reporting of measures to reduce risk of bias from full-text articles describing animal experiments inducing middle cerebral artery occlusion (MCAO) or modelling lacunar stroke. RESULTS: Compared with previous assessments, there were improvements in the reporting of measures taken to reduce risks of bias in the MCAO literature but not in the lacunar stroke literature. Accuracy of automated annotation of risk of bias in the MCAO literature was 86% (randomization), 94% (blinding) and 100% (sample size calculation); and in the lacunar stroke literature accuracy was 67% (randomization), 91% (blinding) and 96% (sample size calculation). DISCUSSION: There remains substantial opportunity for improvement in the reporting of animal research modelling stroke, particularly in the lacunar stroke literature. Further, automated tools perform sufficiently well to identify whether studies report blinded assessment of outcome, but improvements are required in the tools to ascertain whether randomization and a sample size calculation were reported.
Asunto(s)
Isquemia Encefálica/complicaciones , Infarto de la Arteria Cerebral Media/complicaciones , Accidente Cerebrovascular/complicaciones , Animales , Sesgo , Modelos Animales de Enfermedad , Humanos , RiesgoRESUMEN
Animal models are commonly used in brain ageing research. Amongst these, models where rodents are exposed to d-galactose are held to recapitulate a number of features of ageing including neurobehavioral and neurochemical changes. However, results from animal studies are often inconsistent. To better understand the characteristics of the model and effects of d-galactose on neurobehavioral and neurochemical outcomes in rodents we performed a systematic review and meta-analysis. We applied random-effects meta-analysis to evaluate the effect of study features. Our results give an overview of the characteristics of the d-galactose rodent ageing model, including neurobehavioral and neurochemical outcomes. We found that few studies took measures to reduce risks of bias, and substantial heterogeneity in the reported effects of d-galactose in included studies. This highlights the need for improvements in the use of the d-galactose rodent ageing model if it is to provide useful in the development of drugs to treat human ageing.
Asunto(s)
Envejecimiento/efectos de los fármacos , Encéfalo/efectos de los fármacos , Galactosa/farmacología , Estrés Oxidativo/efectos de los fármacos , Animales , Encéfalo/fisiología , Cognición/efectos de los fármacos , Trastornos del Conocimiento/inducido químicamente , Galactosa/efectos adversos , Humanos , Modelos AnimalesRESUMEN
BACKGROUND: Patients undergoing endovascular therapy for acute ischemic stroke may require general anesthesia to undergo the procedure. At present, there is little clinical evidence to guide the choice of anesthetic in this acute setting. The clinical implications of experimental studies demonstrating anesthetic neuroprotection are poorly understood. Here, the authors evaluated the impact of anesthetic treatment on neurologic outcome in experimental stroke. METHODS: Controlled studies of anesthetics in stroke using the filament occlusion model were identified in electronic databases up to December 15, 2015. The primary outcome measures, infarct volume, and neurologic deficit score were used to calculate the normalized mean difference for each comparison. Meta-analysis of normalized mean difference values provided estimates of neuroprotection and contributions of predefined factors: study quality, the timing of treatment, and the duration of ischemia. RESULTS: In 80 retrieved publications anesthetic treatment reduced neurologic injury by 28% (95% CI, 24 to 32%; P < 0.0001). Internal validity was high: publication bias enhanced the effect size by 4% or less, effect size increased with study quality (P = 0.0004), and approximately 70% of studies were adequately powered. Apart from study quality, no predefined factor influenced neuroprotection. Neuroprotection failed in animals with comorbidities. Neuroprotection by anesthetics was associated with prosurvival mechanisms. CONCLUSIONS: Anesthetic neuroprotection is a robust finding in studies using the filament occlusion model of ischemic stroke and should be assumed to influence outcomes in studies using this model. Neuroprotection failed in female animals and animals with comorbidities, suggesting that the results in young male animals may not reflect human stroke.
Asunto(s)
Anestésicos/farmacología , Neuroprotección/efectos de los fármacos , Accidente Cerebrovascular/prevención & control , Animales , Modelos Animales de Enfermedad , Ratones , RatasRESUMEN
Interleukin-1 receptor antagonist (IL-1 RA) is an anti-inflammatory protein used clinically to treat rheumatoid arthritis and is considered a promising candidate therapy for stroke. Here, we sought to update the existing systematic review and meta-analysis of IL-1 RA in models of ischaemic stroke, published in 2009, to assess efficacy, the range of circumstances in which efficacy has been tested and whether the data appear to be confounded due to reported study quality and publication bias. We included 25 sources of data, 11 of which were additional to the original review. Overall, IL-1 RA reduced infarct volume by 36.2 % (95 % confidence interval 31.6-40.7, n = 76 comparisons from 1283 animals). Assessments for publication bias suggest 30 theoretically missing studies which reduce efficacy to 21.9 % (17.3-26.4). Efficacy was higher where IL-1 RA was administered directly into the ventricles rather than peripherally, and studies not reporting allocation concealment during the induction of ischaemia reported larger treatment effects. The preclinical data supporting IL-1 RA as a candidate therapy for ischaemic stroke have improved. The reporting of measures to reduce the risk of bias has improved substantially in this update, and studies now include the use of animals with relevant co-morbidities.