Assessing transparency practices in dental randomized controlled trials.

BACKGROUND: To evaluate transparency practices in randomized controlled trials (RCTs) in dentistry. METHODS: This meta-research study included RCTs in dentistry regardless of topic, methods, or level of detail reported. Only studies in English were considered. We searched PubMed for RCTs in dentistry published in English from December 31, 2016, to December 31, 2021. The screening was performed in duplicate, and data extracted included journal and author details, dental specialty, protocol registration, data and code sharing, conflict of interest declaration, and funding information. A descriptive analysis of the data was performed. We generated maps illustrating the reporting of transparency items by country of the corresponding author and a heat table reflecting reporting levels by dental specialty. RESULTS: A total of 844 RCTs were included. Only 12.86% of studies reported any information about data and code sharing. Protocol registration was reported for 50.36% of RCTs. Conflict of interest (83.41%) and funding (71.68%) declarations were present in most studies. Conflicts of interest and funding were consistently reported regardless of country or specialty, while data and code sharing had a low level of reporting across specialties, as well as low dissemination across the world. Protocol registration exhibited considerable variability. CONCLUSIONS: Considering the importance of RCTs for evidence-based dentistry, it is crucial that everyone who participates in the scientific production and dissemination process actively and consistently promotes adherence to transparent scientific standards, particularly registration of protocols, and sharing of data and code.

Evaluating strategies to recruit health researchers to participate in online survey research.

BACKGROUND: Engaging researchers as research subjects is key to informing the development of effective and relevant research practices. It is important to understand how best to engage researchers as research subjects. METHODS: A 24 factorial experiment, as part of a Multiphase Optimization Strategy, was performed to evaluate effects of four recruitment strategy components on participant opening of an emailed survey link and survey completion. Participants were members of three US-based national health research consortia. A stratified simple random sample was used to assign potential survey participants to one of 16 recruitment scenarios. Recruitment strategy components were intended to address both intrinsic and extrinsic sources of motivation, including: $50 gift, $1,000 raffle, altruistic messaging, and egoistic messaging. Multivariable generalized linear regression analyses adjusting for consortium estimated component effects on outcomes. Potential interactions among components were tested. Results are reported as adjusted odds ratios (aOR) with 95% confidence intervals (95% CI). RESULTS: Surveys were collected from June to December 2023. A total of 418 participants were included from the consortia, with final analytical sample of 400 eligible participants. Out of the final sample, 82% (341) opened the survey link and 35% (147) completed the survey. Altruistic messaging increased the odds of opening the survey (aOR 2.02, 95% CI: 1.35-2.69, p = 0.033), while egoistic messaging significantly reduced the odds of opening the survey (aOR 0.56, 95%CI 0.38-0.75, p = 0.08). The receipt of egoistic messaging increased the odds of completing the survey once opened (aOR 1.81, 95%CI: 1.39-2.23, p < 0.05). There was a significant negative interaction effect between the altruistic appeal and egoistic messaging strategies for survey completion outcome. Monetary incentives did not a have a significant impact on survey completion. CONCLUSION: Intrinsic motivation is likely to be a greater driver of health researcher participation in survey research than extrinsic motivation. Altruistic and egoistic messaging may differentially impact initial interest and survey completion and when combined may lead to improved rates of recruitment, but not survey completion. Further research is needed to determine how to best optimize message content and whether the effects observed are modified by survey burden.

Navigating PROSPERO4animals: 10 top tips for efficient pre-registration of your animal systematic review protocol.

Systematic reviews are an essential tool in identifying knowledge gaps and synthesizing evidence from in vivo animal research to improve human health. The review process follows an explicit and systematic methodology to minimize bias, but is not immune to biases or methodological flaws. Pre-registering a systematic review protocol has several benefits, including avoiding unplanned duplication of reviews, reducing reporting biases, and providing structure throughout the review process. It also helps to align the opinions of review team members and can shield researchers from post-hoc critique. PROSPERO4animals is the international prospective register of systematic reviews (PROSPERO) for the preregistration of systematic review of animal studies. As administrators, here we provide 10 tips to facilitate pre-registration in PROSPERO4animals. These tips address common difficulties that both beginners and experienced researchers may face when pre-registering their systematic review protocols. This article aims to help authors write and register a detailed systematic review protocol on PROSPERO4animals.

Is the commercial determinants conversation confined to the health sciences? Potentially, and that's a problem.

The commercial determinants of health (CDoH) are attracting increased interest and are of great importance when discussing how trade affects health. Through a citation analysis of recent foundational CDoH documents (a Lancet paper series and an Oxford University textbook), we find that fully 71% of all citations reference the health sciences. The health sciences may be well suited to documenting the specific pathways of how commercial (by)products and practices harm human health. However, to operationalize upstream solutions for mitigating these harms, our citation analysis suggests that the field can engage political scientists, economists, sociologists, the trade law and business, as well as advocates in civil society and journalism, more so than it currently does. With CDoH explicitly referring to the interaction between commerce and health, CDoH researchers might be uniquely positioned to get health on the agenda of others, which requires that CDoH methods, datasets, evidence reviews, and proposed interventions are drawn from the widest possible range of sources.

Reporting quality of scoping reviews in endodontics: A meta-research study.

OBJECTIVES: To evaluate the reporting quality of Scoping Reviews (ScRs) in endodontics according to the PRISMA Extension Checklist for Scoping Reviews (PRISMA-ScR) and to analyse their association with a range of publication and methodological/reporting characteristics. METHODS: Pubmed, Scopus, and Web of Science databases were searched up to 31 January 2024 to identify scoping reviews in the field of endodontics. An additional search was performed in three leading endodontic journals. Study selection and appraising the quality of the studies was carried out independently by two reviewers. Each of the 20 PRISMA-ScR items were allocated a score of either 0, 0.5 or 1 to reflect the completeness of the reporting. An item-specific and overall percentage reporting quality score was calculated and reported through descriptive statistics across a range of publication, as well as methodological/reporting characteristics. A univariable and multivariable quantile regression was performed to identify the effect of publication and methodological/reporting characteristics (year of publication, journal, inclusion of an appropriate reporting guideline, and study registration) on the overall percentage reporting quality score. Association of reporting quality score with publication characteristics was then investigated. RESULTS: A total of 40 ScRs were identified and included for appraisal. Most of the studies were published from 2021 onwards. The overall median reporting quality score was 86%. The most frequent items not included in the studies were: a priori protocol registration (22/40 compliant; 55%), and reporting of funding (16/40 compliant; 40%). Other key elements that were inadequately reported were the abstract (7/40 compliant; 18%), the rationale and justification of the ScR (21/40 compliant; 52%) and the objectives of the study (18/40 compliant; 45%). Studies that adhered to appropriate reporting guidelines were associated with greater reporting quality scores (ß-coefficient: 10; 95%CI: 1.1, 18.9; p = .03). ScRs with protocols registered a priori had significantly greater reporting quality scores (ß-coefficient: 12.5; 95%CI: 6.1, 18.9; p < .001), compared with non-registered reviews. CONCLUSIONS: The reporting quality of the ScRs in endodontics varied and was greater when the ScR protocols were registered a priori and when the authors adhered to reporting guidelines.

Publication bias impacts on effect size, statistical power, and magnitude (Type M) and sign (Type S) errors in ecology and evolutionary biology.

Collaborative efforts to directly replicate empirical studies in the medical and social sciences have revealed alarmingly low rates of replicability, a phenomenon dubbed the 'replication crisis'. Poor replicability has spurred cultural changes targeted at improving reliability in these disciplines. Given the absence of equivalent replication projects in ecology and evolutionary biology, two inter-related indicators offer the opportunity to retrospectively assess replicability: publication bias and statistical power. This registered report assesses the prevalence and severity of small-study (i.e., smaller studies reporting larger effect sizes) and decline effects (i.e., effect sizes decreasing over time) across ecology and evolutionary biology using 87 meta-analyses comprising 4,250 primary studies and 17,638 effect sizes. Further, we estimate how publication bias might distort the estimation of effect sizes, statistical power, and errors in magnitude (Type M or exaggeration ratio) and sign (Type S). We show strong evidence for the pervasiveness of both small-study and decline effects in ecology and evolution. There was widespread prevalence of publication bias that resulted in meta-analytic means being over-estimated by (at least) 0.12 standard deviations. The prevalence of publication bias distorted confidence in meta-analytic results, with 66% of initially statistically significant meta-analytic means becoming non-significant after correcting for publication bias. Ecological and evolutionary studies consistently had low statistical power (15%) with a 4-fold exaggeration of effects on average (Type M error rates = 4.4). Notably, publication bias reduced power from 23% to 15% and increased type M error rates from 2.7 to 4.4 because it creates a non-random sample of effect size evidence. The sign errors of effect sizes (Type S error) increased from 5% to 8% because of publication bias. Our research provides clear evidence that many published ecological and evolutionary findings are inflated. Our results highlight the importance of designing high-power empirical studies (e.g., via collaborative team science), promoting and encouraging replication studies, testing and correcting for publication bias in meta-analyses, and adopting open and transparent research practices, such as (pre)registration, data- and code-sharing, and transparent reporting.

Assessing computational reproducibility in Behavior Research Methods.

Psychological science has thrived thanks to new methods and innovative practices. Journals, including Behavior Research Methods (BRM), continue to support the dissemination and evaluation of research assets including data, software/hardware, statistical code, and databases of stimuli. However, such research assets rarely allow for computational reproducibility, meaning they are difficult to reuse. Therefore, in this preregistered report, we explore how BRM's authors and BRM structures shape the landscape of functional research assets. Our broad research questions concern: (1) How quickly methods and analytical techniques reported in BRM can be used and developed further by other scientists; (2) Whether functionality has improved following changes to BRM journal policy in support of computational reproducibility; (3) Whether we can disentangle such policy changes from changes in reproducibility over time. We randomly sampled equal numbers of papers (N = 204) published in BRM before and after the implementation of policy changes. Pairs of researchers recorded how long it took to ensure assets (data, software/hardware, statistical code, and materials) were fully operational. They also coded the completeness and reusability of the assets. While improvements were observed in all measures, only changes to completeness were altered significantly following the policy changes (d = .37). The effects varied between different types of research assets, with data sets from surveys/experiments showing the largest improvements in completeness and reusability. Perhaps more importantly, changes to policy do appear to have improved the life span of research products by reducing natural decline. We conclude with a discussion of how, in the future, research and policy might better support computational reproducibility within and beyond psychological science.

Transparency in Infectious Disease Research: Meta-research Survey of Specialty Journals.

BACKGROUND: Infectious diseases carry large global burdens and have implications for society at large. Therefore, reproducible, transparent research is extremely important. METHODS: We evaluated transparency indicators (code and data sharing, registration, and conflict and funding disclosures) in the 5340 PubMed Central Open Access articles published in 2019 or 2021 in the 9 most cited specialty journals in infectious diseases using the text-mining R package, rtransparent. RESULTS: A total of 5340 articles were evaluated (1860 published in 2019 and 3480 in 2021 [of which 1828 were on coronavirus disease 2019, or COVID-19]). Text mining identified code sharing in 98 (2%) articles, data sharing in 498 (9%), registration in 446 (8%), conflict of interest disclosures in 4209 (79%), and funding disclosures in 4866 (91%). There were substantial differences across the 9 journals: 1%-9% for code sharing, 5%-25% for data sharing, 1%-31% for registration, 7%-100% for conflicts of interest, and 65%-100% for funding disclosures. Validation-corrected imputed estimates were 3%, 11%, 8%, 79%, and 92%, respectively. There were no major differences between articles published in 2019 and non-COVID-19 articles in 2021. In 2021, non-COVID-19 articles had more data sharing (12%) than COVID-19 articles (4%). CONCLUSIONS: Data sharing, code sharing, and registration are very uncommon in infectious disease specialty journals. Increased transparency is required.

Screening for in vitro systematic reviews: a comparison of screening methods and training of a machine learning classifier.

OBJECTIVE: Existing strategies to identify relevant studies for systematic review may not perform equally well across research domains. We compare four approaches based on either human or automated screening of either title and abstract or full text, and report the training of a machine learning algorithm to identify in vitro studies from bibliographic records. METHODS: We used a systematic review of oxygen-glucose deprivation (OGD) in PC-12 cells to compare approaches. For human screening, two reviewers independently screened studies based on title and abstract or full text, with disagreements reconciled by a third. For automated screening, we applied text mining to either title and abstract or full text. We trained a machine learning algorithm with decisions from 2000 randomly selected PubMed Central records enriched with a dataset of known in vitro studies. RESULTS: Full-text approaches performed best, with human (sensitivity: 0.990, specificity: 1.000 and precision: 0.994) outperforming text mining (sensitivity: 0.972, specificity: 0.980 and precision: 0.764). For title and abstract, text mining (sensitivity: 0.890, specificity: 0.995 and precision: 0.922) outperformed human screening (sensitivity: 0.862, specificity: 0.998 and precision: 0.975). At our target sensitivity of 95% the algorithm performed with specificity of 0.850 and precision of 0.700. CONCLUSION: In this in vitro systematic review, human screening based on title and abstract erroneously excluded 14% of relevant studies, perhaps because title and abstract provide an incomplete description of methods used. Our algorithm might be used as a first selection phase in in vitro systematic reviews to limit the extent of full text screening required.

Evaluation of 'implications for research' statements in systematic reviews of interventions in advanced cancer patients - a meta-research study.

BACKGROUND: Implications for research (IfR) sections are an important part of systematic reviews (SRs) to inform health care researchers and policy makers. PRISMA 2020 recommends reporting IfR, while Cochrane Reviews require a separate chapter on IfR. However, it is unclear to what extent SRs discuss IfR. We aimed i) to assess whether SRs include an IfR statement and ii) to evaluate which elements informed IfR statements. METHODS: We conducted a meta-research study based on SRs of interventions in advanced cancer patients from a previous project (CRD42019134904). As suggested in the Cochrane Handbook, we assessed if the following predefined variables were referred to in IfR statements: patient, intervention, control, outcome (PICO) and study design; concepts underlying Grading of Recommendations, Assessment, Development and Evaluation (GRADE) domains: risk of bias, inconsistency, indirectness, imprecision, publication bias. Data were independently extracted by three reviewers after piloting the data extraction form. Discrepancies were resolved in weekly in-depth discussions. RESULTS: We included 261 SRs. The majority evaluated a pharmacological intervention (n = 244, 93.5%); twenty-nine were Cochrane Reviews (11.1%). Four out of five SRs included an IfR statement (n = 210, 80.5%). IfR statements commonly addressed 'intervention' (n = 121, 57.6%), 'patient ' (n = 113, 53.8%), and 'study design' (n = 107, 51.0%). The most frequent PICO and study design combinations were 'patient and intervention ' (n = 71, 33.8%) and 'patient, intervention and study design ' (n = 34, 16.2%). Concepts underlying GRADE domains were rarely used for informing IfR recommendations: 'risk of bias ' (n = 2, 1.0%), and 'imprecision ' (n = 1, 0.5%), 'inconsistency ' (n = 1, 0.5%). Additional elements informing IfR were considerations on cost effectiveness (n = 9, 4.3%), reporting standards (n = 4, 1.9%), and individual patient data meta-analysis (n = 4, 1.9%). CONCLUSION: Although about 80% of SRs included an IfR statement, the reporting of PICO elements varied across SRs. Concepts underlying GRADE domains were rarely used to derive IfR. Further work needs to assess the generalizability beyond SRs in advanced cancer patients. We suggest that more specific guidance on which and how IfR elements to report in SRs of interventions needs to be developed. Utilizing PICO elements and concepts underlying GRADE according to the Cochrane Handbook to state IfR seems to be a reasonable approach in the interim. REGISTRATION: CRD42019134904.

One-Third of Systematic Reviews in Rehabilitation Applied the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) System to Evaluate Certainty of Evidence: A Meta-Research Study.

OBJECTIVE: To determine how many systematic reviews (SRs) of the literature in rehabilitation assess the certainty of evidence (CoE) and how many apply the Grading of Recommendations Assessment, Development and Evaluation (GRADE) system to do this. DATA SOURCES: For this meta-research study, we searched PubMed and Cochrane Database of Systematic Reviews databases for SRs on rehabilitation published in 2020. STUDY SELECTION AND DATA EXTRACTION: Two reviewers independently selected the SRs and extracted the data. Reporting characteristics and appropriate use of the GRADE system were assessed. DATA SYNTHESIS: The search retrieved 827 records: 29% (239/827) SRs evaluated CoE, 68% (163/239) of which applied the GRADE system. GRADE was used by SRs of randomized controlled trials (RCTs, 88%; 144/163), non-randomized intervention studies (NRIS, 2%; 3/163), and both RCT and NRIS (10%; 16/163). In the latter case, a separate GRADE assessment according to the study design was not provided in 75% (12/16). The reasons for GRADE judgment were reported in 82% (134/163) of SRs. CONCLUSIONS: One-third of SRs in rehabilitation assessed CoE with the GRADE system. GRADE assessment was presented transparently by most SRs. Journal editors and funders should encourage the uptake of the GRADE system when considering SRs in rehabilitation for publication. The authors should pre-define GRADE assessment in a registered and/or published protocol.

Transparency of clinical practice guideline funding: a cross-sectional analysis of the German AWMF registry.

BACKGROUND: While reporting of individual conflicts of interest is formalised, it is unclear to what extent the funding of clinical practice guidelines (CPGs) is formally reported. The aim of this study is to explore the accuracy and comprehensiveness of reporting on funding in German CPGs. METHODS: We searched for CPGs in the registry of the Association of the Scientific Medical Societies in Germany in July 2020. Information on guideline funding was categorised by two reviewers independently and discrepancies were clarified by discussion with a third reviewer. Accuracy and comprehensiveness of reporting on funding was assessed using the German Instrument for Methodological Guideline Appraisal (DELBI). RESULTS: We included 507 CPGs published between 2015 and 2020 in the main analysis. 23/507 (4.5%) of the CPGs achieved the highest DELBI score by including information on funding sources, expenses and the amount of funding provided, as well as a statement on the independence of the guideline authors from the funding institution(s). CPGs with more rigorous methodological requirements (systematic review of the literature and/or structured consensus-building) received higher DELBI scores. CONCLUSION: German CPGs do not communicate their funding transparently. Transparency of CPG funding could be achieved by making it mandatory to publish information for all guidelines. For that purpose, a standardised form and guidance should be developed.

Recognizing the value of meta-research and making it easier to find.

Meta-research is a bourgeoning field studying topics with significant relevance to health sciences librarianship, such as research reproducibility, peer review, and open access. As a discipline that studies research itself and the practices of researchers, meta-research spans disciplines and encompasses a broad spectrum of topics and methods. The breadth of meta-research presents a significant challenge for identifying published meta-research studies. Introducing a subject heading for meta-research in the controlled vocabularies of literature databases has the potential to increase the visibility of meta-research, further advance the field, and expand its impact on research practices. Given the relatively recent designation of meta-research as a field and its expanding use as a term, now is the time to develop appropriate indexing vocabulary. We seek to call attention to the value of meta-research for health sciences librarianship, describe the challenges of identifying meta-research literature with currently available key terms, and highlight the need to establish controlled vocabulary specific to meta-research.

Do the clinical practice guidelines for paediatric dentistry meet the quality standards? A meta-research and quality appraisal using the AGREE II tool.

BACKGROUND: Clinical practice guidelines (CPGs) enhance health care and aid clinicians' decisions. AIM: To evaluate the quality of clinical guidelines in paediatric dentistry using the AGREE II tool. DESIGN: PubMed, EMBASE, Scopus, LIVIVO, Lilacs, international guidelines websites, scientific societies, and gray literature were searched until September 2021. We included paediatric dental clinical guidelines and excluded drafts or guidelines for patients with special needs. Two independent reviewers performed quality assessment using the APPRAISAL OF GUIDELINES FOR RESEARCH & EVALUATION II (AGREE II) instrument. We calculated the mean overall domain scores (95% confidence interval) for each guideline. We used regression analysis to correlate the score of overall assessment and the six domains of AGREE II with guideline characteristics. RESULTS: Forty-four guidelines were included in this study. Highest mean score was for Domain 4 (Clarity of Presentation; 58%, 95% CI: 50.8-64.9), whereas the lowest was for Domain 5 (Applicability; 16%, 95% CI: 10.8-21.4). The reporting quality was improved in Domains 1-5 with reporting checklists (p < .001), whereas that of Domain 6 was improved by decreasing years since publication (p = .047). CONCLUSION: Paediatric dental guidelines do not comply with the methodological quality standard, especially in Domain 5 (Applicability). The AGREE reporting checklist should be implemented with a system to evaluate the certainty of evidence for future guidelines.

Cross-Sectional Evaluation of Open Science Practices at Imaging Journals: A Meta-Research Study.

Objective: To evaluate open science policies of imaging journals, and compliance to these policies in published articles. Methods: From imaging journals listed we extracted open science policy details: protocol registration, reporting guidelines, funding, ethics and conflicts of interest (COI), data sharing, and open access publishing. The 10 most recently published studies from each journal were assessed to determine adherence to these policies. We calculated the proportion of open science policies into an Open Science Score (OSS) for all journals and articles. We evaluated relationships between OSS and journal/article level variables. Results: 82 journals/820 articles were included. The OSS of journals and articles was 58.3% and 31.8%, respectively. Of the journals, 65.9% had registration and 78.1% had reporting guideline policies. 79.3% of journals were members of COPE, 81.7% had plagiarism policies, 100% required disclosure of funding, and 97.6% required disclosure of COI and ethics approval. 81.7% had data sharing policies and 15.9% were fully open access. 7.8% of articles had a registered protocol, 8.4% followed a reporting guideline, 77.4% disclosed funding, 88.7% disclosed COI, and 85.6% reported ethics approval. 12.3% of articles shared their data. 51% of articles were available through open access or as a preprint. OSS was higher for journal with DOAJ membership (80% vs 54.2%; P < .0001). Impact factor was not correlated with journal OSS. Knowledge synthesis articles has a higher OSS scores (44.5%) than prospective/retrospective studies (32.6%, 30.0%, P < .0001). Conclusion: Imaging journals endorsed just over half of open science practices considered; however, the application of these practices at the article level was lower.

Heterogeneity of outcomes in randomized controlled trials on implant prosthodontic therapy is hindering comparative effectiveness research: meta-research study.

BACKGROUND: Consistency in outcomes across clinical trials allows for comparing and combining results from different studies. A core outcome set (COS), representing a minimally agreed standardized group of outcomes that should be monitored and measured through research in a specific field of medicine, is not yet available for trials in implant prosthodontic (dental implant) therapy. This meta-research study aimed to analyze outcomes used in clinical trials on implant prosthodontic therapy. METHODS: We searched the Cochrane Oral Health Group (COHG) register to identify systematic reviews of interventions in implant prosthodontic therapy published by October 2023. From the randomized controlled trials (RCTs) included in the relevant reviews, we extracted data on the characteristics of the included trials and the outcomes used. We categorized outcomes into domains. RESULTS: From 182 systematic reviews in the COHG register, we included 11 systematic reviews on dental implant therapy. The reviews included 117 unique RCTs with 4725 participants, published from 1995 to 2020, which analyzed 74 different outcomes. Using different definitions, implant failure was analyzed in 73 RCTs. Seventeen RCTs did not define implant failure. Failure was most often (30 RCTs) followed up for one year. Only one RCT assessed implant failure after five years. Trials used 17 definitions of implant failure, while 17 trials did not report on the criteria of implant failure. Complications were analyzed in 48 RCTs, although they were not clearly defined in 12 RCTs. Failure of prosthodontic supra-structure was analyzed in 74 RCTs, with definitions of failure and criteria not clearly defined in 44 RCTs. Trials considered adverse events, peri-implant tissue health, patient attitudes, and other outcomes, including cost, aesthetics, or procedure duration. These outcomes were often different between trials. Twenty-six outcomes were used only once per study. CONCLUSIONS: Clinical trials in implant prosthodontics used different outcomes, different definitions of outcomes and used different times to monitor them. Standardization of outcomes is necessary to allow comparability and evidence synthesis about the effectiveness of implant prosthodontic therapy.

Low statistical power and overestimated anthropogenic impacts, exacerbated by publication bias, dominate field studies in global change biology.

Field studies are essential to reliably quantify ecological responses to global change because they are exposed to realistic climate manipulations. Yet such studies are limited in replicates, resulting in less power and, therefore, potentially unreliable effect estimates. Furthermore, while manipulative field experiments are assumed to be more powerful than non-manipulative observations, it has rarely been scrutinized using extensive data. Here, using 3847 field experiments that were designed to estimate the effect of environmental stressors on ecosystems, we systematically quantified their statistical power and magnitude (Type M) and sign (Type S) errors. Our investigations focused upon the reliability of field experiments to assess the effect of stressors on both ecosystem's response magnitude and variability. When controlling for publication bias, single experiments were underpowered to detect response magnitude (median power: 18%-38% depending on effect sizes). Single experiments also had much lower power to detect response variability (6%-12% depending on effect sizes) than response magnitude. Such underpowered studies could exaggerate estimates of response magnitude by 2-3 times (Type M errors) and variability by 4-10 times. Type S errors were comparatively rare. These observations indicate that low power, coupled with publication bias, inflates the estimates of anthropogenic impacts. Importantly, we found that meta-analyses largely mitigated the issues of low power and exaggerated effect size estimates. Rather surprisingly, manipulative experiments and non-manipulative observations had very similar results in terms of their power, Type M and S errors. Therefore, the previous assumption about the superiority of manipulative experiments in terms of power is overstated. These results call for highly powered field studies to reliably inform theory building and policymaking, via more collaboration and team science, and large-scale ecosystem facilities. Future studies also require transparent reporting and open science practices to approach reproducible and reliable empirical work and evidence synthesis.

Replacing bar graphs of continuous data with more informative graphics: are we making progress?

Recent work has raised awareness about the need to replace bar graphs of continuous data with informative graphs showing the data distribution. The impact of these efforts is not known. The present observational meta-research study examined how often scientists in different fields use various graph types, and assessed whether visualization practices have changed between 2010 and 2020. We developed and validated an automated screening tool, designed to identify bar graphs of counts or proportions, bar graphs of continuous data, bar graphs with dot plots, dot plots, box plots, violin plots, histograms, pie charts, and flow charts. Papers from 23 fields (approximately 1000 papers/field per year) were randomly selected from PubMed Central and screened (n=227998). F1 scores for different graphs ranged between 0.83 and 0.95 in the internal validation set. While the tool also performed well in external validation sets, F1 scores were lower for uncommon graphs. Bar graphs are more often used incorrectly to display continuous data than they are used correctly to display counts or proportions. The proportion of papers that use bar graphs of continuous data varies markedly across fields (range in 2020: 4-58%), with high rates in biochemistry and cell biology, complementary and alternative medicine, physiology, genetics, oncology and carcinogenesis, pharmacology, microbiology and immunology. Visualization practices have improved in some fields in recent years. Fewer than 25% of papers use flow charts, which provide information about attrition and the risk of bias. The present study highlights the need for continued interventions to improve visualization and identifies fields that would benefit most.

Association of Accuracy, Conclusions, and Reporting Completeness With Acceptance by Radiology Conferences and Journals.

BACKGROUND: Preferential publication of studies with positive findings can lead to overestimation of diagnostic test accuracy (i.e. publication bias). Understanding the contribution of the editorial process to publication bias could inform interventions to optimize the evidence guiding clinical decisions. PURPOSE/HYPOTHESIS: To evaluate whether accuracy estimates, abstract conclusion positivity, and completeness of abstract reporting are associated with acceptance to radiology conferences and journals. STUDY TYPE: Meta-research. POPULATION: Abstracts submitted to radiology conferences (European Society of Gastrointestinal and Abdominal Radiology (ESGAR) and International Society for Magnetic Resonance in Medicine (ISMRM)) from 2008 to 2018 and manuscripts submitted to radiology journals (Radiology, Journal of Magnetic Resonance Imaging [JMRI]) from 2017 to 2018. Primary clinical studies evaluating sensitivity and specificity of a diagnostic imaging test in humans with available editorial decisions were included. ASSESSMENT: Primary variables (Youden's index [YI > 0.8 vs. <0.8], abstract conclusion positivity [positive vs. neutral/negative], number of reported items on the Standards for Reporting of Diagnostic Accuracy Studies [STARD] for Abstract guideline) and confounding variables (prospective vs. retrospective/unreported, sample size, study duration, interobserver agreement assessment, subspecialty, modality) were extracted. STATISTICAL TESTS: Multivariable logistic regression to obtain adjusted odds ratio (OR) as a measure of the association between the primary variables and acceptance by radiology conferences and journals; 95% confidence intervals (CIs) and P-values were obtained; the threshold for statistical significance was P < 0.05. RESULTS: A total of 1000 conference abstracts (500 ESGAR and 500 ISMRM) and 1000 journal manuscripts (505 Radiology and 495 JMRI) were included. Conference abstract acceptance was not significantly associated with YI (adjusted OR = 0.97 for YI > 0.8; CI = 0.70-1.35), conclusion positivity (OR = 1.21 for positive conclusions; CI = 0.75-1.90) or STARD for Abstracts adherence (OR = 0.96 per unit increase in reported items; CI = 0.82-1.18). Manuscripts with positive abstract conclusions were less likely to be accepted by radiology journals (OR = 0.45; CI = 0.24-0.86), while YI (OR = 0.85; CI = 0.56-1.29) and STARD for Abstracts adherence (OR = 1.06; CI = 0.87-1.30) showed no significant association. Positive conclusions were present in 86.7% of submitted conference abstracts and 90.2% of journal manuscripts. DATA CONCLUSION: Diagnostic test accuracy studies with positive findings were not preferentially accepted by the evaluated radiology conferences or journals. EVIDENCE LEVEL: 3 TECHNICAL EFFICACY: Stage 2.

Results publications are inadequately linked to trial registrations: An automated pipeline and evaluation of German university medical centers.

BACKGROUND/AIMS: Informed clinical guidance and health policy relies on clinicians, policymakers, and guideline developers finding comprehensive clinical evidence and linking registrations and publications of the same clinical trial. To support the finding and linking of trial evidence, the World Health Organization, the International Committee of Medical Journal Editors, and the Consolidated Standards of Reporting Trials ask researchers to provide the trial registration number in their publication and a reference to the publication in the registration. This practice costs researchers minimal effort and makes evidence synthesis more thorough and efficient. Nevertheless, trial evidence appears inadequately linked, and the extent of trial links in Germany remains unquantified. This cross-sectional study aims to evaluate links between registrations and publications across clinical trials conducted by German university medical centers and registered in ClinicalTrials.gov or the German Clinical Trials Registry. Secondary aims are to develop an automated pipeline that can be applied to other cohorts of trial registrations and publications, and to provide stakeholders, from trialists to registries, with guidance to improve trial links. METHODS: We used automated strategies to download and extract data from trial registries, PubMed, and results publications for a cohort of registered, published trials conducted across German university medical centers and completed between 2009 and 2017. We implemented regular expressions to detect and classify publication identifiers in registrations, and trial registration numbers in publication metadata, abstracts, and full-texts. RESULTS: In breach of long-standing guidelines, 75% (1,418) of trials failed to reference trial registration numbers in both the abstract and full-text of the journal article in which the results were published. Furthermore, 50% (946) of trial registrations did not contain links to their results publications. Seventeen percent (327) of trials had no links, so that associating registration and publication required manual searching and screening. Overall, trials in ClinicalTrials.gov were better linked than those in the German Clinical Trials Registry; PubMed and registry infrastructures appear to drive this difference. Trial registration numbers were more likely to be transferred to PubMed metadata from abstracts for ClinicalTrials.gov trials than for German Clinical Trials Registry trials. Most (78%, 662/849) ClinicalTrials.gov registrations with a publication link were automatically indexed from PubMed metadata, which is not possible in the German Clinical Trials Registry. CONCLUSIONS: German university medical centers have not comprehensively linked trial registrations and publications, despite established recommendations. This shortcoming threatens the quality of evidence synthesis and medical practice, and burdens researchers with manually searching and linking trial data. Researchers could easily improve this by copy-and-pasting references between their trial registrations and publications. Other stakeholders could build on this practice, for example, PubMed could capture additional trial registration numbers using automated strategies (like those developed in this study), and the German Clinical Trials Registry could automatically index publications from PubMed.