Open peer review urgently requires evidence: A call to action.

Open Peer Review is gaining prominence in attention and use, but to responsibly open up peer review, there is an urgent need for additional evidence. Here, we propose a preliminary research agenda and issue a call to action.

Designing and implementing a research integrity promotion plan: Recommendations for research funders.

Various stakeholders in science have put research integrity high on their agenda. Among them, research funders are prominently placed to foster research integrity by requiring that the organizations and individual researchers they support make an explicit commitment to research integrity. Moreover, funders need to adopt appropriate research integrity practices themselves. To facilitate this, we recommend that funders develop and implement a Research Integrity Promotion Plan (RIPP). This Consensus View offers a range of examples of how funders are already promoting research integrity, distills 6 core topics that funders should cover in a RIPP, and provides guidelines on how to develop and implement a RIPP. We believe that the 6 core topics we put forward will guide funders towards strengthening research integrity policy in their organization and guide the researchers and research organizations they fund.

Incorporating equity, diversity, and inclusiveness into the Hong Kong Principles.

In this response to Labib and Evans, authors of the Hong Kong Principles look forward to collaborating with those from the broad research integrity community to ensure that issues of equity, diversity and inclusion will become part of the ecosystem of research integrity.

The methodological quality of 176,620 randomized controlled trials published between 1966 and 2018 reveals a positive trend but also an urgent need for improvement.

Many randomized controlled trials (RCTs) are biased and difficult to reproduce due to methodological flaws and poor reporting. There is increasing attention for responsible research practices and implementation of reporting guidelines, but whether these efforts have improved the methodological quality of RCTs (e.g., lower risk of bias) is unknown. We, therefore, mapped risk-of-bias trends over time in RCT publications in relation to journal and author characteristics. Meta-information of 176,620 RCTs published between 1966 and 2018 was extracted. The risk-of-bias probability (random sequence generation, allocation concealment, blinding of patients/personnel, and blinding of outcome assessment) was assessed using a risk-of-bias machine learning tool. This tool was simultaneously validated using 63,327 human risk-of-bias assessments obtained from 17,394 RCTs evaluated in the Cochrane Database of Systematic Reviews (CDSR). Moreover, RCT registration and CONSORT Statement reporting were assessed using automated searches. Publication characteristics included the number of authors, journal impact factor (JIF), and medical discipline. The annual number of published RCTs substantially increased over 4 decades, accompanied by increases in authors (5.2 to 7.8) and institutions (2.9 to 4.8). The risk of bias remained present in most RCTs but decreased over time for allocation concealment (63% to 51%), random sequence generation (57% to 36%), and blinding of outcome assessment (58% to 52%). Trial registration (37% to 47%) and the use of the CONSORT Statement (1% to 20%) also rapidly increased. In journals with a higher impact factor (>10), the risk of bias was consistently lower with higher levels of RCT registration and the use of the CONSORT Statement. Automated risk-of-bias predictions had accuracies above 70% for allocation concealment (70.7%), random sequence generation (72.1%), and blinding of patients/personnel (79.8%), but not for blinding of outcome assessment (62.7%). In conclusion, the likelihood of bias in RCTs has generally decreased over the last decades. This optimistic trend may be driven by increased knowledge augmented by mandatory trial registration and more stringent reporting guidelines and journal requirements. Nevertheless, relatively high probabilities of bias remain, particularly in journals with lower impact factors. This emphasizes that further improvement of RCT registration, conduct, and reporting is still urgently needed.

The Hong Kong Principles for assessing researchers: Fostering research integrity.

For knowledge to benefit research and society, it must be trustworthy. Trustworthy research is robust, rigorous, and transparent at all stages of design, execution, and reporting. Assessment of researchers still rarely includes considerations related to trustworthiness, rigor, and transparency. We have developed the Hong Kong Principles (HKPs) as part of the 6th World Conference on Research Integrity with a specific focus on the need to drive research improvement through ensuring that researchers are explicitly recognized and rewarded for behaviors that strengthen research integrity. We present five principles: responsible research practices; transparent reporting; open science (open research); valuing a diversity of types of research; and recognizing all contributions to research and scholarly activity. For each principle, we provide a rationale for its inclusion and provide examples where these principles are already being adopted.

What do Retraction Notices Reveal About Institutional Investigations into Allegations Underlying Retractions?

Academic journal publications may be retracted following institutional investigations that confirm allegations of research misconduct. Retraction notices can provide insight into the role institutional investigations play in the decision to retract a publication. Through a content analysis of 7,318 retraction notices published between 1927 and 2019 and indexed by the Web of Science, we found that most retraction notices (73.7%) provided no information about institutional investigations that may have led to retractions. A minority of the retraction notices (26.3%) mentioned an institutional investigation either by journal authorities (12.1%), research performing organizations (10.3%), joint institutions (1.9%), research integrity and ethics governing bodies (1.0%), third-party institutions (0.5%), unspecified institutions (0.4%), or research funding organizations (0.1%). Comparing retraction notices issued before and after the introduction of retraction guidelines by the Committee on Publication Ethics (COPE) in 2009 revealed that those published after the guidelines' publication were more likely to report investigations by journal authorities. Comparing retraction notices from different disciplines revealed that those from social sciences and the humanities were more likely to disclose investigations by research performing organizations than those from biomedical and natural sciences. Based on these findings, we suggest that the COPE retraction guidelines in the future make it mandatory to disclose in retraction notices institutional investigations leading to retractions.

The Contribution of Moral Case Deliberation to Teaching RCR to PhD Students.

Teaching responsible conduct of research (RCR) to PhD students is crucial for fostering responsible research practice. In this paper, we show how the use of Moral Case Deliberation-a case reflection method used in the Amsterdam UMC RCR PhD course-is particularity valuable to address three goals of RCR education: (1) making students aware of, and internalize, RCR principles and values, (2) supporting reflection on good conduct in personal daily practice, and (3) developing students' dialogical attitude and skills so that they can deliberate on RCR issues when they arise. What makes this method relevant for RCR education is the focus on values and personal motivations, the structured reflection on real experiences and dilemmas and the cultivation of participants' dialogical skills. During these structured conversations, students reflect on the personal motives that drive them to adhere to the principles of good science, thereby building connections between those principles and their personal values and motives. Moreover, by exploring personal questions and dilemmas related to RCR, they learn how to address these with colleagues and supervisors. The reflection on personal experiences with RCR issues and questions combined with the study of relevant normative frameworks, support students to act responsibly and to pursue RCR in their day-to-day research practice in spite of difficulties and external constraints.

Co-creating Research Integrity Education Guidelines for Research Institutions.

To foster research integrity (RI), research institutions should develop a continuous RI education approach, addressing various target groups. To support institutions to achieve this, we developed RI education guidelines together with RI experts and research administrators, exploring similarities and differences in recommendations across target groups, as well as recommendations about RI education using approaches other than formal RI training. We used an iterative co-creative process. We conducted four half-day online co-creation workshops with 16 participants in total, which were informed by the RI education evidence-base. In the first two workshops, participants generated ideas for guidelines' content, focusing on different target groups and various approaches to RI education. Based on this content we developed first drafts of the guidelines. Participants in the third and fourth workshop refined those drafts. We next organized a working group which further prioritized, reorganized, and optimized the content of the guidelines. We developed four guidelines on RI education focusing on (a) bachelor, master and PhD students; (b) post-doctorate and senior researchers; (c) other RI stakeholders; as well as (d) continuous RI education. Across guidelines, we recommend mandatory RI training; follow-up refresher training; informal discussions about RI; appropriate rewards and incentives for active participation in RI education; and evaluation of RI educational events. Our work provides experience-based co-created guidance to research institutions on what to consider when developing a successful RI education strategy. Each guideline is offered as a distinct, publicly available tool in our toolbox ( www.sops4ri.eu/toolbox ) which institutions can access, adapt and implement to meet their institution-specific RI education needs.Trial registration https://osf.io/zej5b .

Stakeholders' Experiences of Research Integrity Support in Universities: A Qualitative Study in Three European Countries.

Fostering research integrity (RI) increasingly focuses on normative guidance and supportive measures within institutions. To be successful, the implementation of support should be informed by stakeholders' experiences of RI support. This study aims to explore experiences of RI support in Dutch, Spanish and Croatian universities. In total, 59 stakeholders (Netherlands n = 25, Spain n = 17, Croatia n = 17) participated in 16 focus groups in three European countries. Global themes on RI support experiences were identified by thematic analysis. Themes identified were: 'RI governance and institutional implementation', 'RI roles and structures', 'RI education and supervision', and 'Infrastructure, technology and tools supporting daily practice'. Experiences of support differed between countries in relation to: the efforts to translate norms into practice; the extent to which RI oversight was a responsibility of RE structures, or separate RI structures; and the availability of support close to research practice, such as training, responsible supervision, and adequate tools and infrastructure. The study reinforces the importance of a whole institutional approach to RI, embedded within local jurisdictions, rules, and practices. A whole institutional approach puts the emphasis of responsibility on institutions rather than individual researchers. When such an approach is lacking, some stakeholders look for intervention by authorities, such as funders, outside of the university.

Important Topics for Fostering Research Integrity by Research Performing and Research Funding Organizations: A Delphi Consensus Study.

To foster research integrity (RI), it is necessary to address the institutional and system-of-science factors that influence researchers' behavior. Consequently, research performing and research funding organizations (RPOs and RFOs) could develop comprehensive RI policies outlining the concrete steps they will take to foster RI. So far, there is no consensus on which topics are important to address in RI policies. Therefore, we conducted a three round Delphi survey study to explore which RI topics to address in institutional RI policies by seeking consensus from research policy experts and institutional leaders. A total of 68 RPO and 52 RFO experts, representing different disciplines, countries and genders, completed one, two or all rounds of the study. There was consensus among the experts on the importance of 12 RI topics for RPOs and 11 for RFOs. The topics that ranked highest for RPOs concerned education and training, supervision and mentoring, dealing with RI breaches, and supporting a responsible research process (e.g. through quality assurance). The highest ranked RFO topics concerned dealing with breaches of RI, conflicts of interest, and setting expectations on RPOs (e.g. about educating researchers about RI). Together with the research policy experts and institutional leaders, we developed a comprehensive overview of topics important for inclusion in the RI policies of RPOs and RFOs. The topics reflect preference for a preventative approach to RI, coupled with procedures for dealing with RI breaches. RPOs and RFOs should address each of these topics in order to support researchers in conducting responsible research.

What Research Institutions Can Do to Foster Research Integrity.

In many countries attention for fostering research integrity started with a misconduct case that got a lot of media exposure. But there is an emerging consensus that questionable research practices are more harmful due to their high prevalence. QRPs have in common that they can help to make study results more exciting, more positive and more statistically significant. That makes them tempting to engage in. Research institutions have the duty to empower their research staff to steer away from QRPs and to explain how they realize that in a Research Integrity Promotion Plan. Avoiding perverse incentives in assessing researchers for career advancement is an important element in that plan. Research institutions, funding agencies and journals should make their research integrity policies as evidence-based as possible. The dilemmas and distractions researchers face are real and universal. We owe it to society to collaborate and to do our utmost best to prevent QRPs and to foster research integrity.

Researchers' Perceptions of a Responsible Research Climate: A Multi Focus Group Study.

The research climate plays a key role in fostering integrity in research. However, little is known about what constitutes a responsible research climate. We investigated academic researchers' perceptions on this through focus group interviews. We recruited researchers from the Vrije Universiteit Amsterdam and the Amsterdam University Medical Center to participate in focus group discussions that consisted of researchers from similar academic ranks and disciplinary fields. We asked participants to reflect on the characteristics of a responsible research climate, the barriers they perceived and which interventions they thought fruitful to improve the research climate. Discussions were recorded and transcribed at verbatim. We used inductive content analysis to analyse the focus group transcripts. We conducted 12 focus groups with 61 researchers in total. We identified fair evaluation, openness, sufficient time, integrity, trust and freedom to be mentioned as important characteristics of a responsible research climate. Main perceived barriers were lack of support, unfair evaluation policies, normalization of overwork and insufficient supervision of early career researchers. Possible interventions suggested by the participants centered around improving support, discussing expectations and improving the quality of supervision. Some of the elements of a responsible research climate identified by participants are reflected in national and international codes of conduct, such as trust and openness. Although it may seem hard to change the research climate, we believe that the realisation that the research climate is suboptimal should provide the impetus for change informed by researchers' experiences and opinions.

Determinants of Citation in Epidemiological Studies on Phthalates: A Citation Analysis.

Citing of previous publications is an important factor in knowledge development. Because of the great amount of publications available, only a selection of studies gets cited, for varying reasons. If the selection of citations is associated with study outcome this is called citation bias. We will study determinants of citation in a broader sense, including e.g. study design, journal impact factor or the funding source of the publication. As a case study we assess which factors drive citation in the human literature on phthalates, specifically the metabolite mono(2-ethylhexyl) phthalate (MEHP). A systematic literature search identified all relevant publications on human health effect of MEHP. Data on potential determinants of citation were extracted in duplo. Specialized software was used to create a citation network, including all potential citation pathways. Random effect logistic regression was used to assess whether these determinants influence the likelihood of citation. 112 Publications on MEHP were identified, with 5684 potential citation pathways of which 551 were actual citations. Reporting of a harmful point estimate, journal impact factor, authority of the author, a male corresponding author, research performed in North America and self-citation were positively associated with the likelihood of being cited. In the literature on MEHP, citation is mostly driven by a number of factors that are not related to study outcome. Although the identified determinants do not necessarily give strong indications of bias, it shows selective use of published literature for a variety of reasons.

The effects of industry funding and positive outcomes in the interpretation of clinical trial results: a randomized trial among Dutch psychiatrists.

BACKGROUND: Most studies are inclined to report positive rather than negative or inconclusive results. It is currently unknown how clinicians appraise the results of a randomized clinical trial. For example, how does the study funding source influence the appraisal of an RCT, and do positive findings influence perceived credibility and clinical relevance? This study investigates whether psychiatrists' appraisal of a scientific abstract is influenced by industry funding disclosures and a positive outcome. METHODS: Dutch psychiatrists were randomized to evaluate a scientific abstract describing a fictitious RCT for a novel antipsychotic drug. Four different abstracts were created reporting either absence or presence of industry funding disclosure as well as a positive or a negative outcome. Primary outcomes were the perceived credibility and clinical relevance of the study results (10-point Likert scale). Secondary outcomes were the assessment of methodological quality and interest in reading the full article. RESULTS: Three hundred ninety-five psychiatrists completed the survey (completion rate 45%). Industry funding disclosure was found not to influence perceived credibility (Mean Difference MD 0.12; 95% CI - 0.28 to 0.47, p?) nor interpretation of its clinical relevance (MD 0.14; 95% CI - 0.54 to 0.27, p?). A negative outcome was perceived as more credible than a positive outcome (MD 0.81 points; 95% Confidence Interval (CI) 0.43 to 1.18, p?), but did not affect clinical relevance scores (MD -0.14; 95% CI - 0.54 to 0.27). CONCLUSIONS: In this study, industry funding disclosure was not associated with the perceived credibility nor judgement of clinical relevance of a fictional RCT by psychiatrists. Positive study outcomes were found to be less credible compared to negative outcomes, but industry funding had no significant effects. Psychiatrists may underestimate the influence of funding sources on research results. The fact that physicians indicated negative outcomes to be more credible may point to more awareness of existing publication bias in the scientific literature.

Systematic review search strategies are poorly reported and not reproducible: a cross-sectional metaresearch study.

OBJECTIVES: To determine the reproducibility of biomedical systematic review search strategies. STUDY DESIGN AND SETTING: A cross-sectional reproducibility study was conducted on a random sample of 100 systematic reviews indexed in MEDLINE in November 2021. The primary outcome measure is the percentage of systematic reviews for which all database searches can be reproduced, operationalized as fulfilling six key Preferred Reporting Items for Systematic reviews and Meta-Analyses literature search extension (PRISMA-S) reporting guideline items and having all database searches reproduced within 10% of the number of original results. Key reporting guideline items included database name, multi-database searching, full search strategies, limits and restrictions, date(s) of searches, and total records. RESULTS: The 100 systematic review articles contained 453 database searches. Only 22 (4.9%) database searches reported all six PRISMA-S items. Forty-seven (10.4%) database searches could be reproduced within 10% of the number of results from the original search; six searches differed by more than 1,000% between the originally reported number of results and the reproduction. Only one systematic review article provided the necessary search details to be fully reproducible. CONCLUSION: Systematic review search reporting is poor. To correct this will require a multifaceted response from authors, peer reviewers, journal editors, and database providers.

Are open science practices in dentistry associated with higher Altmetric scores and citation rates?

AIM: Open science, a set of principles and practices, aims to make scientific research more accessible and accountable, benefiting scientists and society. This study evaluated whether adopting open science practices (OSPs) correlates with higher citation rates and Altmetric scores. METHODS: A random sample of randomised clinical trials (RCTs) on dental caries published between 2000 and 2022 was selected. A systematic PubMed search identified relevant RCTs, and data on OSPs - study registration, open methodology, open software, open scripts, open analysis plan, open data, open peer review, and open access (OA) - were manually collected by two independent assessors. The Robot Reviewer tool automatically evaluated the risk of bias (RoB). Outcomes included the total number of citations and the Altmetric Attention Score. Associations between OSPs, RoB, and other explanatory variables with the outcomes were assessed using binomial negative regression analysis, and expressed as Incidence Rate Ratio (IRR; α =0.05). RESULTS: In total, 323 papers were analysed. At least one OSP was adopted in 57.5 % (n = 186) of the articles, dropping to 39.6 % (n = 128) without OA. Papers with protocol registration (IRR: 1.45; 95 % CI: 1.15, 1.82) and OA publication (IRR: 1.24; 95 % CI: 1.01, 1.53) had higher citation rates. Conversely, papers in full OA journals had fewer citations (IRR: 0.67; 95 % CI: 0.52, 0.87). After adjusting for RoB, low-risk studies showed higher citation rates (IRR: 1.48; 95 % CI: 1.14, 1.91), while OA lost significance. For Altmetric scores, registered and OA manuscripts showed higher scores (IRR: 3.74; 95 % CI: 2.00, 7.01; IRR: 1.69; 95 % CI: 1.04, 2.75), with registration remaining significant after adjusting for RoB and impact factor (IRR: 3.71; 95 % CI: 1.97-6.99). CONCLUSION: The adoption of OSPs demonstrated a partial correlation with citation rates and Altmetric scores in RCTs on dental caries; however, these effects are complex and seem more related to the journal's impact factor. CLINICAL SIGNIFICANCE: The citations and the attention to clinical trials in dentistry, which could drive clinical decision-making and the elaboration of policies and recommendations, seem to be driven more by the journal's prestige than by the adoption of OSPs.