Establishing Institutional Scores With the Rigor and Transparency Index: Large-scale Analysis of Scientific Reporting Quality.

BACKGROUND: Improving rigor and transparency measures should lead to improvements in reproducibility across the scientific literature; however, the assessment of measures of transparency tends to be very difficult if performed manually. OBJECTIVE: This study addresses the enhancement of the Rigor and Transparency Index (RTI, version 2.0), which attempts to automatically assess the rigor and transparency of journals, institutions, and countries using manuscripts scored on criteria found in reproducibility guidelines (eg, Materials Design, Analysis, and Reporting checklist criteria). METHODS: The RTI tracks 27 entity types using natural language processing techniques such as Bidirectional Long Short-term Memory Conditional Random Field-based models and regular expressions; this allowed us to assess over 2 million papers accessed through PubMed Central. RESULTS: Between 1997 and 2020 (where data were readily available in our data set), rigor and transparency measures showed general improvement (RTI 2.29 to 4.13), suggesting that authors are taking the need for improved reporting seriously. The top-scoring journals in 2020 were the Journal of Neurochemistry (6.23), British Journal of Pharmacology (6.07), and Nature Neuroscience (5.93). We extracted the institution and country of origin from the author affiliations to expand our analysis beyond journals. Among institutions publishing >1000 papers in 2020 (in the PubMed Central open access set), Capital Medical University (4.75), Yonsei University (4.58), and University of Copenhagen (4.53) were the top performers in terms of RTI. In country-level performance, we found that Ethiopia and Norway consistently topped the RTI charts of countries with 100 or more papers per year. In addition, we tested our assumption that the RTI may serve as a reliable proxy for scientific replicability (ie, a high RTI represents papers containing sufficient information for replication efforts). Using work by the Reproducibility Project: Cancer Biology, we determined that replication papers (RTI 7.61, SD 0.78) scored significantly higher (P<.001) than the original papers (RTI 3.39, SD 1.12), which according to the project required additional information from authors to begin replication efforts. CONCLUSIONS: These results align with our view that RTI may serve as a reliable proxy for scientific replicability. Unfortunately, RTI measures for journals, institutions, and countries fall short of the replicated paper average. If we consider the RTI of these replication studies as a target for future manuscripts, more work will be needed to ensure that the average manuscript contains sufficient information for replication attempts.

Barriers and Facilitators to Model Replication Within Health Economics.

BACKGROUND: Model replication is important because it enables researchers to check research integrity and transparency and, potentially, to inform the model conceptualization process when developing a new or updated model. OBJECTIVE: The aim of this study was to evaluate the replicability of published decision analytic models and to identify the barriers and facilitators to replication. METHODS: Replication attempts of 5 published economic modeling studies were made. The replications were conducted using only publicly available information within the manuscripts and supplementary materials. The replicator attempted to reproduce the key results detailed in the paper, for example, the total cost, total outcomes, and if applicable, incremental cost-effectiveness ratio reported. Although a replication attempt was not explicitly defined as a success or failure, the replicated results were compared for percentage difference to the original results. RESULTS: In conducting the replication attempts, common barriers and facilitators emerged. For most case studies, the replicator needed to make additional assumptions when recreating the model. This was often exacerbated by conflicting information being presented in the text and the tables. Across the case studies, the variation between original and replicated results ranged from -4.54% to 108.00% for costs and -3.81% to 0.40% for outcomes. CONCLUSION: This study demonstrates that although models may appear to be comprehensively reported, it is often not enough to facilitate a precise replication. Further work is needed to understand how to improve model transparency and in turn increase the chances of replication, thus ensuring future usability.

CONSORT-TM: Text classification models for assessing the completeness of randomized controlled trial publications.

Objective: To develop text classification models for determining whether the checklist items in the CONSORT reporting guidelines are reported in randomized controlled trial publications. Materials and Methods: Using a corpus annotated at the sentence level with 37 fine-grained CONSORT items, we trained several sentence classification models (PubMedBERT fine-tuning, BioGPT fine-tuning, and in-context learning with GPT-4) and compared their performance. To address the problem of small training dataset, we used several data augmentation methods (EDA, UMLS-EDA, text generation and rephrasing with GPT-4) and assessed their impact on the fine-tuned PubMedBERT model. We also fine-tuned PubMedBERT models limited to checklist items associated with specific sections (e.g., Methods) to evaluate whether such models could improve performance compared to the single full model. We performed 5-fold cross-validation and report precision, recall, F1 score, and area under curve (AUC). Results: Fine-tuned PubMedBERT model that takes as input the sentence and the surrounding sentence representations and uses section headers yielded the best overall performance (0.71 micro-F1, 0.64 macro-F1). Data augmentation had limited positive effect, UMLS-EDA yielding slightly better results than data augmentation using GPT-4. BioGPT fine-tuning and GPT-4 in-context learning exhibited suboptimal results. Methods-specific model yielded higher performance for methodology items, other section-specific models did not have significant impact. Conclusion: Most CONSORT checklist items can be recognized reasonably well with the fine-tuned PubMedBERT model but there is room for improvement. Improved models can underpin the journal editorial workflows and CONSORT adherence checks and can help authors in improving the reporting quality and completeness of their manuscripts.

Overviews of reviews in the cardiovascular field underreported critical methodological and transparency characteristics: a methodological study based on the Preferred Reporting Items for Overviews of Reviews (PRIOR) statement.

OBJECTIVES: This study aimed to evaluate the epidemiology, reporting characteristics, and adherence to the Preferred Reporting Items for Overviews of Reviews (PRIOR) statement of overviews of reviews (overviews) of interventions in the cardiovascular field. STUDY DESIGN AND SETTING: MEDLINE, Scopus, and the Cochrane Database of Systematic Reviews were searched from January 1, 2000, to October 15, 2020. An updated search was performed in MEDLINE, Epistemonikos, and Google Scholar up to August 25, 2022. Overviews of interventions published in English and primarily considering populations, interventions, and outcomes pertinent to the cardiovascular field were eligible. Study selection, data extraction, and PRIOR adherence assessment were performed by two authors independently. RESULTS: We analyzed 96 overviews. Almost half (43/96 [45%]) were published between 2020 and 2022 and contained a median of 15 systematic reviews (SRs) (interquartile range, 9-28). The commonest title terminology was "overview of (systematic) reviews" (38/96 [40%]). Methods for handling SR overlap were reported in 24/96 (25%), methods for assessing primary study overlap in 18/96 (19%), handling of discrepant data in 11/96 (11%), and methods for methodological quality or risk of bias assessment of the primary studies within SRs in 23/96 (24%). Authors included data sharing statements in 28/96 (29%), complete funding disclosure in 43/96 (45%), protocol registration in 43/96 (45%), and conflict of interest statement in 82/96 (85%) overviews. CONCLUSION: Insufficient reporting was identified in methodological characteristics unique in overviews' conduct and most transparency markers. Adoption of PRIOR from the research community could ameliorate overviews' reporting.

Increased transparency was required when reporting imputation of primary outcome data in clinical trials.

OBJECTIVES: The aim of this study is to explore the transparency of reporting primary outcome data within randomized controlled trials in the presence of missing data. STUDY DESIGN AND SETTING: A cohort examination of randomized controlled trials published in the four major medical journals (NEJM, JAMA, BMJ, Lancet) in 2013 and the first quarter of 2018. Data were extracted on reporting quality, the number of randomized participants, and the number of participants included within the primary outcome analysis with observed or imputed data. RESULTS: Ninety-one of 159 (57%) studies analyzed from 2013 and 19 of 46 (41%) from 2018 included imputed data within the primary outcome analysis. Of these, only 13 of 91 (14%) studies from 2013 and 1 of 19 (5%) studies from 2018 explicitly reported the number of imputed values in the CONSORT diagram. Results' tables included levels of imputed data in 12 of 91 (13%) studies in 2013 and 4 of 19 (21%) in 2018. Consequently, identification of imputed data was a time-consuming task requiring extensive cross-referencing of all manuscript elements. CONCLUSION: Imputed primary outcome data are poorly reported. Participant flow diagrams frequently reported participant status which does not necessarily correlate to availability of data. We recommended that the number of imputed values are explicitly reported within CONSORT flow diagrams to increase transparency.

Critical Appraisal of Qualitative Studies of Muslim Females' Perceptions of Physical Activity Barriers and Facilitators.

Muslim women's perceptions of cultural, religious, and secular determinants of physical activity have been studied for many years, with information typically acquired through focus groups or interviews. Multiple reviews synthesizing the research have been published, however, individual studies have not been scrutinized for their quality/rigor. Therefore, I critically appraised the quality of the body of qualitative research studies that utilized focus groups to identify Muslim women's perceptions of physical activity barriers and facilitators. I utilized 26 items from the Consolidated Criteria for Reporting Qualitative Research (COREQ) to assess the quality of 56 papers published between 1987 and 2016. Using crosstabulations, I also examined associations between paper quality (low vs. high) and binary categorical variables for impact factor, maximum paper length allowed, publication year, and database the paper was indexed. Overall, papers averaged only 10.5 of 26 COREQ reporting criteria and only two out of 26 items were reported in more than 75% of the papers. Paper quality was not associated with impact factor and length. High quality papers were more likely published more recently (i.e., 2011 or later) and in journals indexed in the PubMed database compared to low quality papers. There is contention among qualitative researchers about standardizing reporting criteria, and while the trend in quality appears to be improving, journal reviewers and editors ought to hold authors to greater accountability in reporting.