Best practices for data management and sharing in experimental biomedical research.

Effective data management is crucial for scientific integrity and reproducibility, a cornerstone of scientific progress. Well-organized and well-documented data enable validation and building on results. Data management encompasses activities including organization, documentation, storage, sharing, and preservation. Robust data management establishes credibility, fostering trust within the scientific community and benefiting researchers' careers. In experimental biomedicine, comprehensive data management is vital due to the typically intricate protocols, extensive metadata, and large datasets. Low-throughput experiments, in particular, require careful management to address variations and errors in protocols and raw data quality. Transparent and accountable research practices rely on accurate documentation of procedures, data collection, and analysis methods. Proper data management ensures long-term preservation and accessibility of valuable datasets. Well-managed data can be revisited, contributing to cumulative knowledge and potential new discoveries. Publicly funded research has an added responsibility for transparency, resource allocation, and avoiding redundancy. Meeting funding agency expectations increasingly requires rigorous methodologies, adherence to standards, comprehensive documentation, and widespread sharing of data, code, and other auxiliary resources. This review provides critical insights into raw and processed data, metadata, high-throughput versus low-throughput datasets, a common language for documentation, experimental and reporting guidelines, efficient data management systems, sharing practices, and relevant repositories. We systematically present available resources and optimal practices for wide use by experimental biomedical researchers.

Author Correction: 'Stealth' corporate innovation: an emerging threat for therapeutic drug development.

An amendment to this paper has been published and can be accessed via a link at the top of the paper.

In defense of quantitative metrics in researcher assessments.

Qualitative assessments of researchers are resource-intensive, untenable in nonmeritocratic settings, and error-prone. Although often derided, quantitative metrics could help improve research practices if they are rigorous, field-adjusted, and centralized.

Gender imbalances among top-cited scientists across scientific disciplines over time through the analysis of nearly 5.8 million authors.

We evaluated how the gender composition of top-cited authors within different subfields of research has evolved over time. We considered 9,071,122 authors with at least 5 full papers in Scopus as of September 1, 2022. Using a previously validated composite citation indicator, we identified the 2% top-cited authors for each of 174 science subfields (Science-Metrix classification) in 4 separate publication age cohorts (first publication pre-1992, 1992 to 2001, 2002 to 2011, and post-2011). Using NamSor, we assigned 3,784,507 authors as men and 2,011,616 as women (for 36.1% gender assignment uncertain). Men outnumbered women 1.88-fold among all authors, decreasing from 3.93-fold to 1.36-fold over time. Men outnumbered women 3.21-fold among top-cited authors, decreasing from 6.41-fold to 2.28-fold over time. In the youngest (post-2011) cohort, 32/174 (18%) subfields had > = 50% women, 97/174 (56%) subfields had > = 30% women, and 3 subfields had = <10% women among the top-cited authors. Gender imbalances in author numbers decreased sharply over time in both high-income countries (including the United States of America) and other countries, but the latter had little improvement in gender imbalances for top-cited authors. In random samples of 100 women and 100 men from the youngest (post-2011) cohort, in-depth assessment showed that most were currently (April 2023) working in academic environments. 32 women and 44 men had some faculty appointment, but only 2 women and 2 men were full professors. Our analysis shows large heterogeneity across scientific disciplines in the amelioration of gender imbalances with more prominent imbalances persisting among top-cited authors and slow promotion pathways even for the most-cited young scientists.

Variability in excess deaths across countries with different vulnerability during 2020-2023.

Excess deaths provide total impact estimates of major crises, such as the COVID-19 pandemic. We evaluated excess death trajectories across countries with accurate death registration and population age structure data and assessed relationships with vulnerability indicators. Using the Human Mortality Database on 34 countries, excess deaths were calculated for 2020-2023 (to week 29, 2023) using 2017-2019 as reference, with adjustment for 5 age strata. Countries were divided into less and more vulnerable; the latter had per capita nominal GDP < $30,000, Gini > 0.35 for income inequality and/or at least ≥2.5% of their population living in poverty. Excess deaths (as proportion of expected deaths, p%) were inversely correlated with per capita GDP (r = -0.60), correlated with proportion living in poverty (r = 0.66), and modestly correlated with income inequality (r = 0.45). Incidence rate ratio for deaths was 1.062 (95% CI, 1.038-1.087) in more versus less vulnerable countries. Excess deaths started deviating in the two groups after the first wave. Between-country heterogeneity diminished gradually within each group. Less vulnerable countries had mean p% = -0.8% and 0.4% in 0-64 and >65-y-old strata. More vulnerable countries had mean p% = 7.0% and 7.2%, respectively. Lower death rates were seen in children of age 0-14 y during 2020-2023 versus prepandemic years. While the pandemic hit some countries earlier than others, country vulnerability dominated eventually the cumulative impact. Half the analyzed countries witnessed no substantial excess deaths versus prepandemic levels, while the others suffered major death tolls.

Massive covidization of research citations and the citation elite.

Massive scientific productivity accompanied the COVID-19 pandemic. We evaluated the citation impact of COVID-19 publications relative to all scientific work published in 2020 to 2021 and assessed the impact on scientist citation profiles. Using Scopus data until August 1, 2021, COVID-19 items accounted for 4% of papers published, 20% of citations received to papers published in 2020 to 2021, and >30% of citations received in 36 of the 174 disciplines of science (up to 79.3% in general and internal medicine). Across science, 98 of the 100 most-cited papers published in 2020 to 2021 were related to COVID-19; 110 scientists received ≥10,000 citations for COVID-19 work, but none received ≥10,000 citations for non-COVID-19 work published in 2020 to 2021. For many scientists, citations to their COVID-19 work already accounted for more than half of their total career citation count. Overall, these data show a strong covidization of research citations across science, with major impact on shaping the citation elite.

Prognostic Biomarkers in Kidney Transplantation: A Systematic Review and Critical Appraisal.

SIGNIFICANCE STATEMENT: Why are there so few biomarkers accepted by health authorities and implemented in clinical practice, despite the high and growing number of biomaker studies in medical research ? In this meta-epidemiological study, including 804 studies that were critically appraised by expert reviewers, the authors have identified all prognostic kidney transplant biomarkers and showed overall suboptimal study designs, methods, results, interpretation, reproducible research standards, and transparency. The authors also demonstrated for the first time that the limited number of studies challenged the added value of their candidate biomarkers against standard-of-care routine patient monitoring parameters. Most biomarker studies tended to be single-center, retrospective studies with a small number of patients and clinical events. Less than 5% of the studies performed an external validation. The authors also showed the poor transparency reporting and identified a data beautification phenomenon. These findings suggest that there is much wasted research effort in transplant biomarker medical research and highlight the need to produce more rigorous studies so that more biomarkers may be validated and successfully implemented in clinical practice. BACKGROUND: Despite the increasing number of biomarker studies published in the transplant literature over the past 20 years, demonstrations of their clinical benefit and their implementation in routine clinical practice are lacking. We hypothesized that suboptimal design, data, methodology, and reporting might contribute to this phenomenon. METHODS: We formed a consortium of experts in systematic reviews, nephrologists, methodologists, and epidemiologists. A systematic literature search was performed in PubMed, Embase, Scopus, Web of Science, and Cochrane Library between January 1, 2005, and November 12, 2022 (PROSPERO ID: CRD42020154747). All English language, original studies investigating the association between a biomarker and kidney allograft outcome were included. The final set of publications was assessed by expert reviewers. After data collection, two independent reviewers randomly evaluated the inconsistencies for 30% of the references for each reviewer. If more than 5% of inconsistencies were observed for one given reviewer, a re-evaluation was conducted for all the references of the reviewer. The biomarkers were categorized according to their type and the biological milieu from which they were measured. The study characteristics related to the design, methods, results, and their interpretation were assessed, as well as reproducible research practices and transparency indicators. RESULTS: A total of 7372 publications were screened and 804 studies met the inclusion criteria. A total of 1143 biomarkers were assessed among the included studies from blood ( n =821, 71.8%), intragraft ( n =169, 14.8%), or urine ( n =81, 7.1%) compartments. The number of studies significantly increased, with a median, yearly number of 31.5 studies (interquartile range [IQR], 23.8-35.5) between 2005 and 2012 and 57.5 (IQR, 53.3-59.8) between 2013 and 2022 ( P < 0.001). A total of 655 studies (81.5%) were retrospective, while 595 (74.0%) used data from a single center. The median number of patients included was 232 (IQR, 96-629) with a median follow-up post-transplant of 4.8 years (IQR, 3.0-6.2). Only 4.7% of studies were externally validated. A total of 346 studies (43.0%) did not adjust their biomarker for key prognostic factors, while only 3.1% of studies adjusted the biomarker for standard-of-care patient monitoring factors. Data sharing, code sharing, and registration occurred in 8.8%, 1.1%, and 4.6% of studies, respectively. A total of 158 studies (20.0%) emphasized the clinical relevance of the biomarker, despite the reported nonsignificant association of the biomarker with the outcome measure. A total of 288 studies assessed rejection as an outcome. We showed that these rejection studies shared the same characteristics as other studies. CONCLUSIONS: Biomarker studies in kidney transplantation lack validation, rigorous design and methodology, accurate interpretation, and transparency. Higher standards are needed in biomarker research to prove the clinical utility and support clinical use.

Evidence base for yearly respiratory virus vaccines: Current status and proposed improved strategies.

Annual vaccination is widely recommended for influenza and SARS-CoV-2. In this essay, we analyse and question the prevailing policymaking approach to these respiratory virus vaccines, especially in the United States. Every year, licensed influenza vaccines are reformulated to include specific strains expected to dominate in the season ahead. Updated vaccines are rapidly manufactured and approved without further regulatory requirement of clinical data. Novel vaccines (i.e. new products) typically undergo clinical trials, though generally powered for clinically unimportant outcomes (e.g. lab-confirmed infections, regardless of symptomatology or antibody levels). Eventually, the current and future efficacy of influenza and COVID-19 vaccines against hospitalization or death carries considerable uncertainty. The emergence of highly transmissible SARS-CoV-2 variants and waning vaccine-induced immunity led to plummeting vaccine effectiveness, at least against symptomatic infection, and booster doses have since been widely recommended. No further randomized trials were performed for clinically important outcomes for licensed updated boosters. In both cases, annual vaccine effectiveness estimates are generated by observational research, but observational studies are particularly susceptible to confounding and bias. Well-conducted experimental studies, particularly randomized trials, are necessary to address persistent uncertainties about influenza and COVID-19 vaccines. We propose a new research framework which would render results relevant to the current or future respiratory viral seasons. We demonstrate that experimental studies are feasible by adopting a more pragmatic approach and provide strategies on how to do so. When it comes to implementing policies that seriously impact people's lives, require substantial public resources and/or rely on widespread public acceptance, high evidence standards are desirable.

A novel framework to assess haematology and oncology registration trials: The THEOREMM project.

BACKGROUND: Methodological limitations affect a significant number of oncology and haematology trials, raising concerns about the applicability of their results. For example, a suboptimal control arm or limited access to best care upon progression may skew the trial results toward a benefit in the experimental arm. Beyond the fact that such limitations do not prevent drugs reaching the market, other assessment tools, such as those developed by professional societies-ESMO-MCBS and ASCO Value Framework-do not integrate these important shortcomings. METHODS: We propose creating a novel framework with the scope of assessing registration cancer clinical trials in haematology and oncology (randomized or single arm)-that is trials leading to a marketing authorization. The main steps of the methods are (1) assembling a scientific board; (2) defining the scope, goal and methods through pre-specified, pre-registered and protocolized methodology; (3) preregistration of the protocol; (4) conducting a scoping review of limitations and biases affecting oncology trials and assessing existing scores or methods; (5) developing a list of features to be included and assessed within the framework; (6) assessing each feature through a questionnaire sent to highly cited haematologists and oncologists involved in clinical trials; and (7) finalizing the first version of framework. RESULTS: Not applicable. CONCLUSIONS: Our proposal emerged in response to the lack of consideration for key limitations in current trial assessments. The goal is to create a framework specifically designed to assess single trials leading to marketing authorization in the field of oncology and haematogy.

Large language models for science and medicine.

Large language models (LLMs) are a type of machine learning model that learn statistical patterns over text, such as predicting the next words in a sequence of text. Both general purpose and task-specific LLMs have demonstrated potential across diverse applications. Science and medicine have many data types that are highly suitable for LLMs, such as scientific texts (publications, patents and textbooks), electronic medical records, large databases of DNA and protein sequences and chemical compounds. Carefully validated systems that can understand and reason across all these modalities may maximize benefits. Despite the inevitable limitations and caveats of any new technology and some uncertainties specific to LLMs, LLMs have the potential to be transformative in science and medicine.

Effectiveness of a fourth SARS-CoV-2 vaccine dose in previously infected individuals from Austria.

INTRODUCTION: Evidence is limited on the effectiveness of a fourth vaccine dose against coronavirus disease 2019 (COVID-19) in populations with prior severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infections. We estimated the risk of COVID-19 deaths and SARS-CoV-2 infections according to vaccination status in previously infected individuals in Austria. METHODS: This is a nationwide retrospective observational study. We calculated age and gender adjusted Cox proportional hazard ratios (HRs) of COVID-19 deaths (primary outcome) and SARS-CoV-2 infections (secondary outcome) from 1 November to 31 December 2022, primarily comparing individuals with four versus three vaccine doses. Relative vaccine effectiveness (rVE) was calculated as (1-HR) X 100. RESULTS: Among 3,986,312 previously infected individuals, 281,291 (7,1%) had four and 1,545,242 (38.8%) had three vaccinations at baseline. We recorded 69 COVID-19 deaths and 89,056 SARS-CoV-2 infections. rVE for four versus three vaccine doses was -24% (95% CI: -120 to 30) against COVID-19 deaths, and 17% (95% CI: 14-19) against SARS-CoV-2 infections. This latter effect rapidly diminished over time and infection risk with four vaccinations was higher compared to less vaccinated individuals during extended follow-up until June 2023. Adjusted HR (95% CI) for all-cause mortality for four versus three vaccinations was 0.79 (0.74-0.85). DISCUSSION: In previously infected individuals, a fourth vaccination was not associated with COVID-19 death risk, but with transiently reduced risk of SARS-CoV-2 infections and reversal of this effect in longer follow-up. All-cause mortality data suggest healthy vaccinee bias.

Correction: Empirical assessment of published effect sizes and power in the recent cognitive neuroscience and psychology literature.

[This corrects the article DOI: 10.1371/journal.pbio.2000797.].

Assessment of transparency indicators across the biomedical literature: How open is open?

Recent concerns about the reproducibility of science have led to several calls for more open and transparent research practices and for the monitoring of potential improvements over time. However, with tens of thousands of new biomedical articles published per week, manually mapping and monitoring changes in transparency is unrealistic. We present an open-source, automated approach to identify 5 indicators of transparency (data sharing, code sharing, conflicts of interest disclosures, funding disclosures, and protocol registration) and apply it across the entire open access biomedical literature of 2.75 million articles on PubMed Central (PMC). Our results indicate remarkable improvements in some (e.g., conflict of interest [COI] disclosures and funding disclosures), but not other (e.g., protocol registration and code sharing) areas of transparency over time, and map transparency across fields of science, countries, journals, and publishers. This work has enabled the creation of a large, integrated, and openly available database to expedite further efforts to monitor, understand, and promote transparency and reproducibility in science.

Ten (not so) simple rules for clinical trial data-sharing.

Clinical trial data-sharing is seen as an imperative for research integrity and is becoming increasingly encouraged or even required by funders, journals, and other stakeholders. However, early experiences with data-sharing have been disappointing because they are not always conducted properly. Health data is indeed sensitive and not always easy to share in a responsible way. We propose 10 rules for researchers wishing to share their data. These rules cover the majority of elements to be considered in order to start the commendable process of clinical trial data-sharing: Rule 1: Abide by local legal and regulatory data protection requirementsRule 2: Anticipate the possibility of clinical trial data-sharing before obtaining fundingRule 3: Declare your intent to share data in the registration stepRule 4: Involve research participantsRule 5: Determine the method of data accessRule 6: Remember there are several other elements to shareRule 7: Do not proceed aloneRule 8: Deploy optimal data management to ensure that the data shared is usefulRule 9: Minimize risksRule 10: Strive for excellence.

An empirical comparison of statistical methods for multiple cut-off diagnostic test accuracy meta-analysis of the Edinburgh postnatal depression scale (EPDS) depression screening tool using published results vs individual participant data.

BACKGROUND: Selective reporting of results from only well-performing cut-offs leads to biased estimates of accuracy in primary studies of questionnaire-based screening tools and in meta-analyses that synthesize results. Individual participant data meta-analysis (IPDMA) of sensitivity and specificity at each cut-off via bivariate random-effects models (BREMs) can overcome this problem. However, IPDMA is laborious and depends on the ability to successfully obtain primary datasets, and BREMs ignore the correlation between cut-offs within primary studies. METHODS: We compared the performance of three recent multiple cut-off models developed by Steinhauser et al., Jones et al., and Hoyer and Kuss, that account for missing cut-offs when meta-analyzing diagnostic accuracy studies with multiple cut-offs, to BREMs fitted at each cut-off. We used data from 22 studies of the accuracy of the Edinburgh Postnatal Depression Scale (EPDS; 4475 participants, 758 major depression cases). We fitted each of the three multiple cut-off models and BREMs to a dataset with results from only published cut-offs from each study (published data) and an IPD dataset with results for all cut-offs (full IPD data). We estimated pooled sensitivity and specificity with 95% confidence intervals (CIs) for each cut-off and the area under the curve. RESULTS: Compared to the BREMs fitted to the full IPD data, the Steinhauser et al., Jones et al., and Hoyer and Kuss models fitted to the published data produced similar receiver operating characteristic curves; though, the Hoyer and Kuss model had lower area under the curve, mainly due to estimating slightly lower sensitivity at lower cut-offs. When fitting the three multiple cut-off models to the full IPD data, a similar pattern of results was observed. Importantly, all models had similar 95% CIs for sensitivity and specificity, and the CI width increased with cut-off levels for sensitivity and decreased with an increasing cut-off for specificity, even the BREMs which treat each cut-off separately. CONCLUSIONS: Multiple cut-off models appear to be the favorable methods when only published data are available. While collecting IPD is expensive and time consuming, IPD can facilitate subgroup analyses that cannot be conducted with published data only.

Principles for good scholarship in systematic reviews.

Many sources document problems that jeopardize the trustworthiness of systematic reviews. This is a major concern given their potential to influence patient care and impact people's lives. Responsibility for producing trustworthy conclusions on the evidence in systematic reviews is borne primarily by authors who need the necessary training and resources to correctly report on the current knowledge base. Peer reviewers and editors are also accountable; they must ensure that systematic reviews are accurate by demonstrating proper methods. To support all these stakeholders, we attempt to distill the sprawling guidance that is currently available in our recent co-publication about best tools and practices for systematic reviews. We specifically address how to meet methodological conduct standards applicable to key components of systematic reviews. In this complementary invited review, we place these standards in the context of good scholarship principles for systematic review development. Our intention is to reach a broad audience and potentially improve the trustworthiness of evidence syntheses published in the developmental medicine literature and beyond.

Transparency in Infectious Disease Research: Meta-research Survey of Specialty Journals.

BACKGROUND: Infectious diseases carry large global burdens and have implications for society at large. Therefore, reproducible, transparent research is extremely important. METHODS: We evaluated transparency indicators (code and data sharing, registration, and conflict and funding disclosures) in the 5340 PubMed Central Open Access articles published in 2019 or 2021 in the 9 most cited specialty journals in infectious diseases using the text-mining R package, rtransparent. RESULTS: A total of 5340 articles were evaluated (1860 published in 2019 and 3480 in 2021 [of which 1828 were on coronavirus disease 2019, or COVID-19]). Text mining identified code sharing in 98 (2%) articles, data sharing in 498 (9%), registration in 446 (8%), conflict of interest disclosures in 4209 (79%), and funding disclosures in 4866 (91%). There were substantial differences across the 9 journals: 1%-9% for code sharing, 5%-25% for data sharing, 1%-31% for registration, 7%-100% for conflicts of interest, and 65%-100% for funding disclosures. Validation-corrected imputed estimates were 3%, 11%, 8%, 79%, and 92%, respectively. There were no major differences between articles published in 2019 and non-COVID-19 articles in 2021. In 2021, non-COVID-19 articles had more data sharing (12%) than COVID-19 articles (4%). CONCLUSIONS: Data sharing, code sharing, and registration are very uncommon in infectious disease specialty journals. Increased transparency is required.

Recommendations on data sharing in HIV drug resistance research.

• Human immunodeficiency virus (HIV) drug resistance has implications for antiretroviral treatment strategies and for containing the HIV pandemic because the development of HIV drug resistance leads to the requirement for antiretroviral drugs that may be less effective, less well-tolerated, and more expensive than those used in first-line regimens. • HIV drug resistance studies are designed to determine which HIV mutations are selected by antiretroviral drugs and, in turn, how these mutations affect antiretroviral drug susceptibility and response to future antiretroviral treatment regimens. • Such studies collectively form a vital knowledge base essential for monitoring global HIV drug resistance trends, interpreting HIV genotypic tests, and updating HIV treatment guidelines. • Although HIV drug resistance data are collected in many studies, such data are often not publicly shared, prompting the need to recommend best practices to encourage and standardize HIV drug resistance data sharing. • In contrast to other viruses, sharing HIV sequences from phylogenetic studies of transmission dynamics requires additional precautions as HIV transmission is criminalized in many countries and regions. • Our recommendations are designed to ensure that the data that contribute to HIV drug resistance knowledge will be available without undue hardship to those publishing HIV drug resistance studies and without risk to people living with HIV.