Correlation analysis of financial conflicts of interest and favourability of results or conclusions in addiction medicine systematic reviews and meta-analysis.

OBJECTIVE: To quantify conflicts of interest, assess the accuracy of authors self-reporting them, and examine the association between conflicts of interest and favourability of results and discussions in addiction medicine systematic reviews. DESIGN: A search was performed on Medline (Ovid) from January 2016 to 25 April 2020 to locate systematic reviews and meta-analyses focused on treatments of addiction disorders using a systematic search strategy. Data were extracted from each systematic review, including conflict of interest statements, authorship characteristics and the favourability of the results/conclusion sections. A search algorithm was used to identify any undisclosed conflicts of interest on the Open Payments Database (Dollars for Docs), Dollars for Profs, Google Patents/United States Patent and Trade Office, and prior conflict of interest statements in other published works from these authors. RESULTS: The search identified 127 systematic reviews, representing 665 unique authors. Of the 127 studies, 81 reported no authors with conflicts of interest, 28 with 1 or more conflict, and 18 had no conflict of interest statement. Additional non-disclosed conflicts of interest were found for 34 authors. There were 69 reviews that had at least one author with a conflict of interest. Of the 69 reviews, 14 (20.3%) reported favourable results and 26 (37.7%) reported favourable discussion/conclusions with no statistically significant association. A subanalysis was performed on publications with only US authors (51) with 35 (68.9%) having at least 1 conflict of interest. US authored studies that had a conflict of interest favoured the results (p = <0.001) and discussion/conclusion (p = 0.018) more often. CONCLUSION: Although multiple undisclosed financial conflicts of interest were found, there was no correlation with the favourability of the results or discussion/conclusions across all addiction medicine systematic reviews. Further research needs to be done on US-based publications and encourage disclosure systems worldwide to provide more accurate reporting.

A review of reproducible and transparent research practices in urology publications from 2014 to2018.

BACKGROUND: Reproducibility is essential for the integrity of scientific research. Reproducibility is measured by the ability of different investigators to replicate the outcomes of an original publication using the same materials and procedures. Unfortunately, reproducibility is not currently a standard being met by most scientific research. METHODS: For this review, we sampled 300 publications in the field of urology to assess for 14 indicators of reproducibility including material availability, raw data availability, analysis script availability, pre-registration information, links to protocols, and if the publication was available free to the public. Publications were also assessed for statements about conflicts of interest and funding sources. RESULTS: Of the 300 sample publications, 171 contained empirical data available for analysis of reproducibility. Of the 171 articles with empirical data to analyze, 0.58% provided links to protocols, 4.09% provided access to raw data, 3.09% provided access to materials, and 4.68% were pre-registered. None of the studies provided analysis scripts. Our review is cross-sectional in nature, including only PubMed indexed journals-published in English-and within a finite time period. Thus, our results should be interpreted in light of these considerations. CONCLUSION: Current urology research does not consistently provide the components needed to reproduce original studies. Collaborative efforts from investigators and journal editors are needed to improve research quality while minimizing waste and patient risk.

Transparent and Reproducible Research Practices in the Surgical Literature.

INTRODUCTION: Previous studies have established a baseline of minimal reproducibility in the social science and biomedical literature. Clinical research is especially deficient in factors of reproducibility. Surgical journals contain fewer clinical trials than non-surgical areas of medicine, suggesting that it should be easier to reproduce the outcomes of surgical literature. METHODS: In this study, we evaluated a broad range of indicators related to transparency and reproducibility in a random sample of 387 articles published in Surgery journals between 2014 and 2018. RESULTS: A small minority of our sample made available their materials (5.3%, 95% C.I. 2.4%-8.2%), protocols (1.2%, 0-2.5%), data (2.5%, 0.7%-4.2%), or analysis scripts (0.04%). Four studies were adequately pre-registered. No studies were explicit replications of previous literature. Most studies (58%), declined to provide a funding statement, while conflicts of interest were declared in a small fraction (9.3%). Most have not been cited by systematic reviews (83%) or meta-analyses (87%), and most were only accessible to paying subscribers (59%). CONCLUSIONS: The transparency of the surgical literature could improve with adherence to baseline standards of reproducibility.

Evaluation of conflicts of interest among systematic review authors on pharmacological therapies for alcohol use disorder: A cross-sectional study.

Introduction and aims: The high prevalence and economic burden of alcohol use disorder (AUD) requires methodologically sound research to guide treatment decisions. Systematic reviews (SR) are fundamental to clinical decision making as they collate results of all studies for a given topic and provide summaries of the clinical evidence. Bias resulting from industry relationships can compromise the validity of SRs. Thus, we investigated financial conflicts of interests (FCOI) within SRs involving the pharmacologic treatment of AUD. Design and Methods: We conducted a systematic search of PubMed, Embase, and the Cochrane Database to retrieve SRs published between September 2016 and December 2019. Screening and data extraction were performed in duplicate, blinded fashion-collecting characteristics of the reviews including the authors' FCOI statements. The results and discussion sections were rated as favorable or unfavorable toward the treatment drug. A stepwise method was used to identify undisclosed FCOIs for all SR authors. Results: Nineteen (of 51) authors representing 7 (of 12) SRs were found to have FCOI. Among reviews with conflicted authors, 3 of 7 (42.9%) results sections were favorable toward the treatment drug, whereas 6 of 7 (85.7%) conclusions were rated as favorable. Discussion and Conclusions: More than one-third of SR authors and nearly two-thirds of studies were found to have FCOIs. Our investigation shows that financial ties are frequent among SR authors on AUD pharmacotherapies and that discussion sections often favored the drug for which the sponsor provided payments.

Evaluating the reproducibility of research in obstetrics and gynecology.

OBJECTIVE: Reproducibility is a core tenet of scientific research. A reproducible study is one where the results can be recreated by using the same methodology and materials as the original researchers. Unfortunately, reproducibility is not a standard to which the majority of research is currently adherent. METHODS: Our cross-sectional survey evaluated 300 trials in the field of Obstetrics and Gynecology. Our primary objective was to identify nine indicators of reproducibility and transparency. These indicators include availability of data, analysis scripts, pre-registration information, study protocols, funding source, conflict of interest statements and whether or not the study was available via Open Access. RESULTS: Of the 300 trials in our sample, 208 contained empirical data that could be assessed for reproducibility. None of the trials in our sample provided a link to their protocols or provided a statement on availability of materials. None were replication studies. Just 10.58% provided a statement regarding their data availability, while only 5.82% provided a statement on preregistration. 25.85% failed to report the presence or absence of conflicts of interest and 54.08% did not state the origin of their funding. CONCLUSION: In the studies we examined, research in the field of Obstetrics and Gynecology is not consistently reproducible and frequently lacks conflict of interest disclosure. Consequences of this could be far-reaching and include increased research waste, widespread acceptance of misleading results and erroneous conclusions guiding clinical decision-making.

Evaluation of indicators supporting reproducibility and transparency within cardiology literature.

OBJECTIVES: It has been suggested that biomedical research is facing a reproducibility issue, yet the extent of reproducible research within the cardiology literature remains unclear. Thus, our main objective was to assess the quality of research published in cardiology journals by assessing for the presence of eight indicators of reproducibility and transparency. METHODS: Using a cross-sectional study design, we conducted an advanced search of the National Library of Medicine catalogue for publications in cardiology journals. We included publications published between 1 January 2014 and 31 December 2019. After the initial list of eligible cardiology publications was generated, we searched for full-text PDF versions using Open Access, Google Scholar and PubMed. Using a pilot-tested Google Form, a random sample of 532 publications were assessed for the presence of eight indicators of reproducibility and transparency. RESULTS: A total of 232 eligible publications were included in our final analysis. The majority of publications (224/232, 96.6%) did not provide access to complete and unmodified data sets, all 229/232 (98.7%) failed to provide step-by-step analysis scripts and 228/232 (98.3%) did not provide access to complete study protocols. CONCLUSIONS: The presentation of studies published in cardiology journals would make reproducing study outcomes challenging, at best. Solutions to increase the reproducibility and transparency of publications in cardiology journals is needed. Moving forward, addressing inadequate sharing of materials, raw data and key methodological details might help to better the landscape of reproducible research within the field.

Evaluating Reproducibility and Transparency in Emergency Medicine Publications.

INTRODUCTION: We aimed to assess the reproducibility of empirical research by determining the availability of components required for replication of a study, including materials, raw data, analysis scripts, protocols, and preregistration. METHODS: We used the National Library of Medicine catalog to identify MEDLINE-indexed emergency medicine (EM) journals. Thirty journals met the inclusion criteria. From January 1, 2014-December 31, 2018, 300 publications were randomly sampled using a PubMed search. Additionally, we included four high-impact general medicine journals, which added 106 publications. Two investigators were blinded for independent extraction. Extracted data included statements regarding the availability of materials, data, analysis scripts, protocols, and registration. RESULTS: After the search, we found 25,473 articles, from which we randomly selected 300. Of the 300, only 287 articles met the inclusion criteria. Additionally, we added 106 publications from high-impact journals of which 77 met the inclusion criteria. Together, 364 publications were included, of which 212 articles contained empirical data to analyze. Of the eligible empirical articles, 2.49%, (95% confidence interval [CI], 0.33% to 4.64%] provided a material statement, 9.91% (95% CI, 5.88% to 13.93%) provided a data statement, 0 provided access to analysis scripts, 25.94% (95% CI, 20.04% to 31.84%) linked the protocol, and 39.15% (95% CI, 32.58% to 45.72%) were preregistered. CONCLUSION: Studies in EM lack indicators required for reproducibility. The majority of studies fail to report factors needed to reproduce research to ensure credibility. Thus, an intervention is required and can be achieved through the collaboration of researchers, peer reviewers, funding agencies, and journals.

Evaluation of Risks of Bias in Addiction Medicine Randomized Controlled Trials.

AIMS: Perhaps the most important step when designing and conducting randomized controlled trials (RCTs) in addiction is to put methodological safeguards in place to minimize the likelihood for bias to affect trial outcomes. In this study, we applied the revised Cochrane risk of bias tool (ROB 2) to RCTs of drug, alcohol or tobacco interventions. METHODS: We searched for trials published in 15 addiction medicine journals over a 7-year period. Our primary endpoint is the risk of bias of included studies. We conducted a sensitivity analysis of publicly funded trials. RESULTS: Overall, included RCTs were most often at high risk of bias per our judgments (244/487, 50.1%). However, significant proportions of included RCTs were at low risk of bias (123/487, 25.3%) or some concerns for bias (120/497, 24.6%). RCTs with behavioral modification interventions (19/44, 43.2%) and alcohol interventions (80/150, 53.3%) had the highest proportion of high-risk judgments. In a sensitivity analysis of publicly funded RCTs), 195/386 (50.5%) were at high risk of bias. CONCLUSIONS: Approximately half of included drug, alcohol or tobacco RCTs in our sample were judged to be at high risk of bias with the most common reason being a lack of proper blinding or proper description of blinding. Key action items to reduce bias in future addiction RCTs include adequate randomization, blinding and inclusion of a trial registry number and protocol.

An evaluation of the practice of transparency and reproducibility in addiction medicine literature.

BACKGROUND: Credible research emphasizes transparency, openness, and reproducibility. These characteristics are fundamental to promoting and maintaining research integrity. The aim of this study was to evaluate the current state of reproducibility in the field of addiction science. DESIGN: Cross-sectional design. MEASUREMENTS: The National Library of Medicine catalog was searched for all journals using the subject terms tag: Substance-Related Disorders [ST]. Journals were then searched via PubMed to identify publications from January 1, 2014, to December 31, 2018; 300 publications were randomly selected from among those identified. A pilot-tested Google form containing reproducibility/transparency characteristics was used for data extraction in a duplicated and blinded fashion by two investigators. FINDINGS: Slightly more than half of the publications were open access (152/300, 50.70%). Few publications had pre-registration (7/244, 2.87%), material availability (2/237, 0.84%), protocol availability (3/244, 1.23%), data availability (28/244, 11.48%), or analysis script availability (2/244, 0.82%). Most publications provided a conflict of interest statement (221/293, 75.43%) and funding sources (268/293, 91.47%). One replication study was reported (1/244, 0.4%). CONCLUSION: Our study found that current practices that promote transparency and reproducibility are lacking, thus, there is room for improvement. In particular, investigators should pre-register studies prior to commencement. Researchers should also make the materials, data, and analysis script publicly available. Further, individuals should be transparent about funding sources for the project and financial conflicts of interest. Research stakeholders should work together toward improvements on these matters. With such protections, the field of addiction medicine can better disseminate the information necessary to treat patients.

Analysis of practices to promote reproducibility and transparency in anaesthesiology research.

INTRODUCTION: Reliable, high-quality research is essential to the field of anaesthesiology. Reproducibility and transparency have been investigated in the biomedical domain and in the social sciences, with both lacking to provide necessary information to reproduce the study findings. In this study, we investigated 14 indicators of reproducibility in anaesthesiology research. METHODS: We used the National Library of Medicine catalogue to search for all anaesthesiology journals that are MEDLINE indexed and provide English texts. PubMed was searched with the list of journals to identify all publications from January 1, 2014 to December 31, 2018. We randomly sampled 450 publications that fit the inclusion criteria for our analysis. Data extraction was then conducted in a blinded, duplicate fashion using a pilot-tested Google form. RESULTS: The PubMed search of these journals identified 171 441 publications, with 28 310 being within the time frame. From the 450 publications sampled, 444 full-text publications were accessible. The majority of publications analysed did not have a statement regarding availability of data (164/188), analysis scripts (187/188), or study materials (160/188). CONCLUSIONS: Anaesthesiology research needs to improve indicators of reproducibility and transparency. By making research publicly available and improving accessibility to detailed study components, primary research can be reproduced in subsequent studies and help contribute to the development of new practice guidelines.

An analysis of key indicators of reproducibility in radiology.

BACKGROUND: Given the central role of radiology in patient care, it is important that radiological research is grounded in reproducible science. It is unclear whether there is a lack of reproducibility or transparency in radiologic research. PURPOSE: To analyze published radiology literature for the presence or lack of key indicators of reproducibility. METHODS: This cross-sectional retrospective study was performed by conducting a search of the National Library of Medicine (NLM) for publications contained within journals in the field of radiology. Our inclusion criteria were being MEDLINE indexed, written in English, and published from January 1, 2014, to December 31, 2018. We randomly sampled 300 publications for this study. A pilot-tested Google form was used to record information from the publications regarding indicators of reproducibility. Following peer-review, we extracted data from an additional 200 publications in an attempt to reproduce our initial results. The additional 200 publications were selected from the list of initially randomized publications. RESULTS: Our initial search returned 295,543 records, from which 300 were randomly selected for analysis. Of these 300 records, 294 met inclusion criteria and 6 did not. Among the empirical publications, 5.6% (11/195, [3.0-8.3]) contained a data availability statement, 0.51% (1/195) provided clear documented raw data, 12.0% (23/191, [8.4-15.7]) provided a materials availability statement, 0% provided analysis scripts, 4.1% (8/195, [1.9-6.3]) provided a pre-registration statement, 2.1% (4/195, [0.4-3.7]) provided a protocol statement, and 3.6% (7/195, [1.5-5.7]) were pre-registered. The validation study of the 5 key indicators of reproducibility-availability of data, materials, protocols, analysis scripts, and pre-registration-resulted in 2 indicators (availability of protocols and analysis scripts) being reproduced, as they fell within the 95% confidence intervals for the proportions from the original sample. However, materials' availability and pre-registration proportions from the validation sample were lower than what was found in the original sample. CONCLUSION: Our findings demonstrate key indicators of reproducibility are missing in the field of radiology. Thus, the ability to reproduce studies contained in radiology publications may be problematic and may have potential clinical implications.

Reproducible and transparent research practices in published neurology research.

BACKGROUND: The objective of this study was to evaluate the nature and extent of reproducible and transparent research practices in neurology publications. METHODS: The NLM catalog was used to identify MEDLINE-indexed neurology journals. A PubMed search of these journals was conducted to retrieve publications over a 5-year period from 2014 to 2018. A random sample of publications was extracted. Two authors conducted data extraction in a blinded, duplicate fashion using a pilot-tested Google form. This form prompted data extractors to determine whether publications provided access to items such as study materials, raw data, analysis scripts, and protocols. In addition, we determined if the publication was included in a replication study or systematic review, was preregistered, had a conflict of interest declaration, specified funding sources, and was open access. RESULTS: Our search identified 223,932 publications meeting the inclusion criteria, from which 400 were randomly sampled. Only 389 articles were accessible, yielding 271 publications with empirical data for analysis. Our results indicate that 9.4% provided access to materials, 9.2% provided access to raw data, 0.7% provided access to the analysis scripts, 0.7% linked the protocol, and 3.7% were preregistered. A third of sampled publications lacked funding or conflict of interest statements. No publications from our sample were included in replication studies, but a fifth were cited in a systematic review or meta-analysis. CONCLUSIONS: Currently, published neurology research does not consistently provide information needed for reproducibility. The implications of poor research reporting can both affect patient care and increase research waste. Collaborative intervention by authors, peer reviewers, journals, and funding sources is needed to mitigate this problem.

Assessment of transparent and reproducible research practices in the psychiatry literature.

BACKGROUND: Reproducibility is a cornerstone of scientific advancement; however, many published works may lack the core components needed for study reproducibility. AIMS: In this study, we evaluate the state of transparency and reproducibility in the field of psychiatry using specific indicators as proxies for these practices. METHODS: An increasing number of publications have investigated indicators of reproducibility, including research by Harwicke et al, from which we based the methodology for our observational, cross-sectional study. From a random 5-year sample of 300 publications in PubMed-indexed psychiatry journals, two researchers extracted data in a duplicate, blinded fashion using a piloted Google form. The publications were examined for indicators of reproducibility and transparency, which included availability of: materials, data, protocol, analysis script, open-access, conflict of interest, funding and online preregistration. RESULTS: This study ultimately evaluated 296 randomly-selected publications with a 3.20 median impact factor. Only 107 were available online. Most primary authors originated from USA, UK and the Netherlands. The top three publication types were cohort studies, surveys and clinical trials. Regarding indicators of reproducibility, 17 publications gave access to necessary materials, four provided in-depth protocol and one contained raw data required to reproduce the outcomes. One publication offered its analysis script on request; four provided a protocol availability statement. Only 107 publications were publicly available: 13 were registered in online repositories and four, ten and eight publications included their hypothesis, methods and analysis, respectively. Conflict of interest was addressed by 177 and reported by 31 publications. Of 185 publications with a funding statement, 153 publications were funded and 32 were unfunded. CONCLUSIONS: Currently, Psychiatry research has significant potential to improve adherence to reproducibility and transparency practices. Thus, this study presents a reference point for the state of reproducibility and transparency in Psychiatry literature. Future assessments are recommended to evaluate and encourage progress.

An Evaluation of Nephrology Literature for Transparency and Reproducibility Indicators: Cross-Sectional Review.

INTRODUCTION: Reproducibility is critical to diagnostic accuracy and treatment implementation. Concurrent with clinical reproducibility, research reproducibility establishes whether the use of identical study materials and methodologies in replication efforts permits researchers to arrive at similar results and conclusions. In this study, we address this gap by evaluating nephrology literature for common indicators of transparent and reproducible research. METHODS: We searched the National Library of Medicine catalog to identify 36 MEDLINE-indexed, English-language nephrology journals. We randomly sampled 300 publications published between January 1, 2014, and December 31, 2018. RESULTS: Our search yielded 28,835 publications, of which we randomly sampled 300 publications. Of the 300 publications, 152 (50.7%) were publicly available, whereas 143 (47.7%) were restricted through paywall and 5 (1.7%) were inaccessible. Of the remaining 295 publications, 123 were excluded because they lack empirical data necessary for reproducibility. Of the 172 publications with empirical data, 43 (25%) reported data availability statements and 4 (2.3%) analysis scripts. Of the 71 publications analyzed for preregistration and protocol availability, 0 (0.0%) provided links to a protocol and 8 (11.3%) were preregistered. CONCLUSION: Our study found that reproducible and transparent research practices are infrequently used by the nephrology research community. Greater efforts should be made by both funders and journals. In doing so, an open science culture may eventually become the norm rather than the exception.

The fragility of randomized trial outcomes underlying management of dyspepsia and Helicobacter pylori infections.

AIM: The fragility index is calculated by changing one outcome event to a nonevent within a trial until the associated P value exceeds 0.05. In this study, we assessed the robustness, risk of bias (RoB), and power of randomized controlled trials that underlie recommendations set forth by the American College of Gastroenterology (ACG) on managing dyspepsia and Helicobacter pylori infections. METHODS: All citations referenced in the guidelines were screened for inclusion criteria. The fragility indexes for eligible trials were then calculated. The likelihood and sources of bias in the included trials were evaluated by the Cochrane 'RoB' Tool 2.0. RESULTS: The median fragility index for the 52 trials was three events. Five studies (9.6%) resulted in a fragility index of 0 when statistical analysis was applied. For the 52 trials, 12 (23.1%) were at a low RoB, 15 (28.8%) had some concerns, and 25 (48.1%) were at a high RoB. High RoB was most commonly due to bias of selection in the reported result (15.5%). CONCLUSION: A median of three events was needed to nullify statistical significance in 52 trials that underpin guideline recommendations on the management of dyspepsia and H. pylori infections. In addition, concerns for RoB were found for these trials.

An assessment of transparency and reproducibility-related research practices in otolaryngology.

OBJECTIVES/HYPOTHESIS: Clinical research serves as the foundation for evidence-based patient care, and reproducibility of results is consequently critical. We sought to assess the transparency and reproducibility of research studies in otolaryngology by evaluating a random sample of publications in otolaryngology journals between 2014 and 2018. STUDY DESIGN: Review of published literature for reproducible and transparent research practices. METHODS: We used the National Library of Medicine catalog to identify otolaryngology journals that met the inclusion criteria (available in the English language and indexed in MEDLINE). From these journals, we extracted a random sample of 300 publications using a PubMed search for records published between January 1, 2014 and December 31, 2018. Specific indicators of reproducible and transparent research practices were evaluated in a blinded, independent, and duplicate manner using a pilot-tested Google form. RESULTS: Our initial search returned 26,498 records, from which 300 were randomly selected for analysis. Of these 300 records, 286 met inclusion criteria and 14 did not. Among the empirical studies, 2% (95% confidence interval [CI]: 0.4%-3.5%) of publications indicated that raw data were available, 0.6% (95% CI: 0.3%-1.6%) reported an analysis script, 5.3% (95% CI: 2.7%-7.8%) were linked to an accessible research protocol, and 3.9% (95% CI: 1.7%-6.1%) were preregistered. None of the publications had a clear statement claiming to replicate, or to be a replication of, another study. CONCLUSIONS: Inadequate reproducibility practices exist in otolaryngology. Nearly all studies in our analysis lacked a data or material availability statement, did not link to an accessible protocol, and were not preregistered. Taking steps to improve reproducibility would likely improve patient care. LEVEL OF EVIDENCE: NA Laryngoscope, 130: 1894-1901, 2020.

Financial Conflicts of Interest Among Authors of Clinical Practice Guidelines for Routine Screening Mammography.

PURPOSE: Financial conflicts of interest (FCOIs) may influence or undermine the credibility of clinical practice guidelines or society recommendations. Given the wide regard of such publications, understanding the prevalence and extent of FCOIs among their authors is essential. METHODS: The most current guidelines containing recommendations for breast cancer screening from the US Preventive Services Task Force, American Cancer Society, American College of Obstetricians and Gynecologists, International Agency for Research on Cancer, ACR, and American College of Physicians were retrieved from their respective organizational websites. Industry payments received by authors were then extracted using CMS Open Payments database (OPD), and the values and types of these payments were evaluated. Finally, financial disclosures were compared with open payments. RESULTS: Among a total of 43 authors and 7 guideline documents, 14 authors (33%) received at least one industry payment according to OPD payment records, whereas a majority of 29 authors (67%) had none. The median total payment from all sources across all breast imaging guidelines was $0 (interquartile range, $0-$84). Four authors (9%) declared at least one significant FCOI, five (12%) received more than $5,000 from a single company in a single year, and one author had a significant FCOI (2%) identified from OPD records but not disclosed within the guideline document. CONCLUSIONS: These findings suggest that FCOIs likely have little to no influence on the adoption of consensus recommendations regarding routine screening mammography for all cohorts of women.

Do oncology researchers adhere to reproducible and transparent principles? A cross-sectional survey of published oncology literature.

OBJECTIVES: As much as 50%-90% of research is estimated to be irreproducible, costing upwards of $28 billion in USA alone. Reproducible research practices are essential to improving the reproducibility and transparency of biomedical research, such as including preregistering studies, publishing a protocol, making research data and metadata publicly available, and publishing in open access journals. Here we report an investigation of key reproducible or transparent research practices in the published oncology literature. DESIGN: We performed a cross-sectional analysis of a random sample of 300 oncology publications published from 2014 to 2018. We extracted key reproducibility and transparency characteristics in a duplicative fashion by blinded investigators using a pilot tested Google Form. PRIMARY OUTCOME MEASURES: The primary outcome of this investigation is the frequency of key reproducible or transparent research practices followed in published biomedical and clinical oncology literature. RESULTS: Of the 300 publications randomly sampled, 296 were analysed for reproducibility characteristics. Of these 296 publications, 194 contained empirical data that could be analysed for reproducible and transparent research practices. Raw data were available for nine studies (4.6%). Five publications (2.6%) provided a protocol. Despite our sample including 15 clinical trials and 7 systematic reviews/meta-analyses, only 7 included a preregistration statement. Less than 25% (65/194) of publications provided an author conflict of interest statement. CONCLUSION: We found that key reproducibility and transparency characteristics were absent from a random sample of published oncology publications. We recommend required preregistration for all eligible trials and systematic reviews, published protocols for all manuscripts, and deposition of raw data and metadata in public repositories.

The fragility of statistically significant results in otolaryngology randomized trials.

OBJECTIVES: The American Academy of Otolaryngology-Head and Neck Surgery regards randomized controlled trials as class A evidence. A novel method to determine the robustness of outcomes in trials is the fragility index. This index represents the number of patients whose status would have to change from a non-event to an event to make a statistically significant result non-significant. METHODS: Investigators included otolaryngology journals listed in the top 10 of one or both of Google Scholar Metrics and Clarivate Analytics' Journal rankings. For inclusion, a randomized controlled trial needed to report a one-to-one random assignment of participants to condition, contain two parallel arms or have used a two-by-two factorial design, and report at least one statistically significant dichotomous outcome. RESULTS: Sixty-nine trials met inclusion criteria. The median fragility index was three events (interquartile range 1-7.5). Median sample size was 72 (interquartile range 50-102.5). Modest correlations were observed between fragility index and total sample size (r = 0.27) and fragility index and event rate (r = 0.46). Investigators found no correlation between fragility index and impact factor or Science Citation Index. In 39% (27/69) of trials, the number lost to follow-up was equal to or greater than the fragility index. CONCLUSION: A median fragility index of 3 indicates that three people, on average, are needed to alter the outcomes in otolaryngology trials. This indicates that the results of two-group randomized controlled trials reporting binary endpoints published in otolaryngology journals may frequently be fragile.