Evaluating Reproducibility and Transparency in Emergency Medicine Publications.

INTRODUCTION: We aimed to assess the reproducibility of empirical research by determining the availability of components required for replication of a study, including materials, raw data, analysis scripts, protocols, and preregistration. METHODS: We used the National Library of Medicine catalog to identify MEDLINE-indexed emergency medicine (EM) journals. Thirty journals met the inclusion criteria. From January 1, 2014-December 31, 2018, 300 publications were randomly sampled using a PubMed search. Additionally, we included four high-impact general medicine journals, which added 106 publications. Two investigators were blinded for independent extraction. Extracted data included statements regarding the availability of materials, data, analysis scripts, protocols, and registration. RESULTS: After the search, we found 25,473 articles, from which we randomly selected 300. Of the 300, only 287 articles met the inclusion criteria. Additionally, we added 106 publications from high-impact journals of which 77 met the inclusion criteria. Together, 364 publications were included, of which 212 articles contained empirical data to analyze. Of the eligible empirical articles, 2.49%, (95% confidence interval [CI], 0.33% to 4.64%] provided a material statement, 9.91% (95% CI, 5.88% to 13.93%) provided a data statement, 0 provided access to analysis scripts, 25.94% (95% CI, 20.04% to 31.84%) linked the protocol, and 39.15% (95% CI, 32.58% to 45.72%) were preregistered. CONCLUSION: Studies in EM lack indicators required for reproducibility. The majority of studies fail to report factors needed to reproduce research to ensure credibility. Thus, an intervention is required and can be achieved through the collaboration of researchers, peer reviewers, funding agencies, and journals.

Evaluation of Spin in the Abstracts of Emergency Medicine Randomized Controlled Trials.

STUDY OBJECTIVE: We aim to investigate spin in emergency medicine abstracts, using a sample of randomized controlled trials from high-impact-factor journals with statistically nonsignificant primary endpoints. METHODS: This study investigated spin in abstracts of emergency medicine randomized controlled trials from emergency medicine literature, with studies from 2013 to 2017 from the top 5 emergency medicine journals and general medical journals. Investigators screened records for inclusion and extracted data for spin. We considered evidence of spin if trial authors focused on statistically significant results, interpreted statistically nonsignificant results as equivalent or noninferior, used favorable rhetoric in the interpretation of nonsignificant results, or claimed benefit of an intervention despite statistically nonsignificant results. RESULTS: Of 772 abstracts screened, 114 randomized controlled trials reported statistically nonsignificant primary endpoints. Spin was found in 50 of 114 abstracts (44.3%). Industry-funded trials were more likely to have evidence of spin in the abstract (unadjusted odds ratio 3.4; 95% confidence interval 1.1 to 11.9). In the abstracts' results, evidence of spin was most often due to authors' emphasizing a statistically significant subgroup analysis (n=9). In the abstracts' conclusions, spin was most often due to authors' claiming they accomplished an objective that was not a prespecified endpoint (n=14). CONCLUSION: Spin was prevalent in the selected randomized controlled trial, emergency medicine abstracts. Authors most commonly incorporated spin into their reports by focusing on statistically significant results for secondary outcomes or subgroup analyses when the primary outcome was statistically nonsignificant. Spin was more common in studies that had some component of industry funding.

A review of research efforts to address the 2008 ACEP guideline for mild traumatic brain injury.

BACKGROUND: The objective of this study was to evaluate the temporal relationship between clinical practice guideline development and subsequent research performed, with the goal of providing more data on areas of sparse evidence that serve to underlie guideline recommendations. We aimed to assess the quality of current research efforts to address the American College of Emergency Physicians guideline and to provide suggestions for future research of mild traumatic brain injury. METHODS: We identified clinical practice guideline recommendations with low levels of underlying evidence and searched ClinicalTrials.gov and the World Health Organization's International Clinical Trial Registry Portal to determine whether subsequent research has reflected an effort to address guideline recommendations. RESULTS: Few currently registered clinical trials attempt to focus on clinical practice guideline recommendations where the basis of evidence is weak, and even fewer might benefit future iterations of the guideline due to multiple problems in study design and reporting. CONCLUSIONS: The amount of research dedicated to investigation of mild traumatic brain injury continues to be sparse and of poor quality. Study results should always be posted, even if the null hypothesis is confirmed. Efforts to improve the evidence base of a guideline should be realized by designing studies that directly assess and speak to the questions posed by guideline authors.

Variable methodological quality and use found in systematic reviews referenced in STEMI clinical practice guidelines.

BACKGROUND: The objective of this study was to assess the methodological quality and clarity of reporting of the systematic reviews (SRs) supporting clinical practice guideline (CPG) recommendations in the management of ST-elevation myocardial infarction (STEMI) across international CPGs. METHODS: We searched 13 guideline clearinghouses including the National Guideline Clearinghouse and Guidelines International Network (GIN). To meet inclusion criteria CPGs must be pertinent to the management of STEMI, endorsed by a governing body or national organization, and written in English. We retrieved SRs from the reference sections using a combination of keywords and hand searching. Two investigators scored eligible SRs using AMSTAR and PRISMA. RESULTS: We included four CPGs. We extracted 71 unique SRs. These SRs received AMSTAR scores ranging from 1 (low) to 9 (high) on an 11-point scale. All CPGs consistently underperformed in areas including disclosure of funding sources, risk of bias, and publication bias according to AMSTAR. PRISMA checklist completeness ranged from 44% to 96%. The PRISMA scores indicated that SRs did not provide a full search strategy, study protocol and registration, assessment of publication bias or report funding sources. Only one SR was referenced in all four CPGs. All CPGs omitted a large subset of available SRs cited by other guidelines. CONCLUSIONS: Our study demonstrates the variable quality of SRs used to establish recommendations within guidelines included in our sample. Although guideline developers have acknowledged this variability, it remains a significant finding that needs to be addressed further. FUNDING: This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.