Structured peer review: pilot results from 23 Elsevier journals.

Background: Reviewers rarely comment on the same aspects of a manuscript, making it difficult to properly assess manuscripts' quality and the quality of the peer review process. The goal of this pilot study was to evaluate structured peer review implementation by: 1) exploring whether and how reviewers answered structured peer review questions, 2) analysing reviewer agreement, 3) comparing that agreement to agreement before implementation of structured peer review, and 4) further enhancing the piloted set of structured peer review questions. Methods: Structured peer review consisting of nine questions was piloted in August 2022 in 220 Elsevier journals. We randomly selected 10% of these journals across all fields and IF quartiles and included manuscripts that received two review reports in the first 2 months of the pilot, leaving us with 107 manuscripts belonging to 23 journals. Eight questions had open-ended fields, while the ninth question (on language editing) had only a yes/no option. The reviews could also leave Comments-to-Author and Comments-to-Editor. Answers were independently analysed by two raters, using qualitative methods. Results: Almost all the reviewers (n = 196, 92%) provided answers to all questions even though these questions were not mandatory in the system. The longest answer (Md 27 words, IQR 11 to 68) was for reporting methods with sufficient details for replicability or reproducibility. The reviewers had the highest (partial) agreement (of 72%) for assessing the flow and structure of the manuscript, and the lowest (of 53%) for assessing whether interpretation of the results was supported by data, and for assessing whether the statistical analyses were appropriate and reported in sufficient detail (52%). Two thirds of the reviewers (n = 145, 68%) filled out the Comments-to-Author section, of which 105 (49%) resembled traditional peer review reports. These reports contained a Md of 4 (IQR 3 to 5) topics covered by the structured questions. Absolute agreement regarding final recommendations (exact match of recommendation choice) was 41%, which was higher than what those journals had in the period from 2019 to 2021 (31% agreement, P = 0.0275). Conclusions: Our preliminary results indicate that reviewers successfully adapted to the new review format, and that they covered more topics than in their traditional reports. Individual question analysis indicated the greatest disagreement regarding the interpretation of the results and the conducting and the reporting of statistical analyses. While structured peer review did lead to improvement in reviewer final recommendation agreements, this was not a randomized trial, and further studies should be performed to corroborate this. Further research is also needed to determine whether structured peer review leads to greater knowledge transfer or better improvement of manuscripts.

Paying reviewers and regulating the number of papers may help fix the peer-review process.

The exponential increase in the number of submissions, further accelerated by generative AI, and the decline in the availability of experts are burdening the peer review process. This has led to high unethical desk rejection rates, a growing appeal for the publication of unreviewed preprints, and a worrying proliferation of predatory journals. The idea of monetarily compensating peer reviewers has been around for many years; maybe, it is time to take it seriously as one way to save the peer review process. Here, I argue that paying reviewers, when done in a fair and transparent way, is a viable solution. Like the case of professional language editors, part-time or full-time professional reviewers, managed by universities or for-profit companies, can be an integral part of modern peer review. Being a professional reviewer could be financially attractive to retired senior researchers and to researchers who enjoy evaluating papers but are not motivated to do so for free. Moreover, not all produced research needs to go through peer review, and thus persuading researchers to limit submissions to their most novel and useful research could also help bring submission volumes to manageable levels. Overall, this paper reckons that the problem is not the peer review process per se but rather its function within an academic ecosystem dominated by an unhealthy culture of 'publish or perish'. Instead of reforming the peer review process, academia has to look for better science dissemination schemes that promote collaboration over competition, engagement over judgement, and research quality and sustainability over quantity.

The Peer Review Process: Past, Present, and Future.

The peer review process is a fundamental aspect of modern scientific paper publishing, underpinning essential quality control. First conceptualised in the 1700s, it is an iterative process that aims to elevate scientific literature to the highest standards whilst preventing publication of scientifically unsound, potentially misleading, and even plagiarised information. It is widely accepted that the peer review of scientific papers is an irreplaceable and fundamental aspect of the research process. However, the rapid growth of research and technology has led to a huge increase in the number of publications. This has led to increased pressure on the peer review system. There are several established peer review methodologies, ranging from single and double blind to open and transparent review, but their implementation across journals and research fields varies greatly. Some journals are testing entirely novel approaches (such as collaborative reviews), whilst others are piloting changes to established methods. Given the unprecedented growth in publication numbers, and the ensuing burden on journals, editors, and reviewers, it is imperative to improve the quality and efficiency of the peer review process. Herein we evaluate the peer review process, from its historical origins to current practice and future directions.

Assessing ChatGPT's ability to emulate human reviewers in scientific research: A descriptive and qualitative approach.

BACKGROUND: ChatGPT is an AI platform whose relevance in the peer review of scientific articles is steadily growing. Nonetheless, it has sparked debates over its potential biases and inaccuracies. This study aims to assess ChatGPT's ability to qualitatively emulate human reviewers in scientific research. METHODS: We included the first submitted version of the latest twenty original research articles published by the 3rd of July 2023, in a high-profile medical journal. Each article underwent evaluation by a minimum of three human reviewers during the initial review stage. Subsequently, three researchers with medical backgrounds and expertise in manuscript revision, independently and qualitatively assessed the agreement between the peer reviews generated by ChatGPT version GPT-4 and the comments provided by human reviewers for these articles. The level of agreement was categorized into complete, partial, none, or contradictory. RESULTS: 720 human reviewers' comments were assessed. There was a good agreement between the three assessors (Overall kappa >0.6). ChatGPT's comments demonstrated complete agreement in terms of quality and substance with 48 (6.7 %) human reviewers' comments, partially agreed with 92 (12.8 %), identifying issues necessitating further elaboration or recommending supplementary steps to address concerns, had no agreement with a significant 565 (78.5 %), and contradicted 15 (2.1 %). ChatGPT comments on methods had the lowest proportion of complete agreement (13 comments, 3.6 %), while general comments on the manuscript displayed the highest proportion of complete agreement (17 comments, 22.1 %). CONCLUSION: ChatGPT version GPT-4 has a limited ability to emulate human reviewers within the peer review process of scientific research.

Peer review of the pesticide risk assessment of the active substance lenacil.

The conclusions of the European Food Safety Authority (EFSA) following the peer review of the initial risk assessments carried out by the competent authorities of the rapporteur Member State, Belgium, and co-rapporteur Member State, Austria, for the pesticide active substance lenacil are reported. The context of the peer review was that required by Commission Implementing Regulation (EU) No 844/2012, as amended by Commission Implementing Regulation (EU) No 2018/1659. The conclusions were reached on the basis of the evaluation of the representative uses of lenacil as a herbicide on sugar and fodder beet (field use). The reliable end points, appropriate for use in regulatory risk assessment, are presented. Missing information identified as being required by the regulatory framework is listed. Concerns are reported where identified.

Cryptic late-onset myocarditis after coronavirus vaccination.

As we enter a new era of mRNA-based therapeutics, evidence on genetic or environmental factors that might predispose to unknown off-target side effects, gains in importance. Among these factors, exercise appears likely to have influenced otherwise cryptic cases of early-onset postvaccination myocarditis. And the existence of a distinct late-onset myocarditis is now being recognized. Here, three case-history reports suggest crypticity (the author's own case), unless provoked by a preexisting cardiac morbidity (one case), or by immune checkpoint blockade to enhance anticancer autoimmunity (several cases). These reports are supported by noninvasive fluorodeoxyglucose-based cardiac scan comparisons of multiple vaccinated and unvaccinated subjects. In pre-pandemic decades, applications for funds by the leading innovator in mRNA-based therapeutics seldom gained peer-review approval. Thus, at the start of the pandemic, the meager data on such side effects could justify only emergency approval. We must do better.

Assessing the impact of trained Radiologist Assistants in a busy emergency teleradiology practice: a comprehensive evaluation.

PURPOSE: This study aims to study the feasibility and usefulness of trained Radiologist Assistants in a busy emergency teleradiology practice. METHOD: This is a retrospective study over a 21-month period (January 2021 to September 2022). The study analysed archived data from 247118 peer review studies performed by Radiologist Assistants (RAs) out of a total case volume of 828526 and evaluated the rate of discrepancies, the study types commonly noted to have discrepancies, and the severity of errors. These missed findings were brought to the attention of the radiologists for approval and further decision-making. RESULTS: Peer review by RAs was performed on 30% (n = 247118) of the total volume 828526 studies reported, and yielded additional findings including but not limited to fractures (218; 23%), hemorrhage,(94; 10%) pulmonary thromboembolism, (n = 104; 11%), Calculus (n = 75; 8%) lesion (n = 66; 5%), appendicitis(n = 50; 4%) and others. These were brought to the attention of the radiologist, who agreed in 97% (1279 out of 1318) of cases, and communicated the same to the referring facility, with an addended report. CONCLUSION: Trained RAs can provide value to the peer review program of a busy teleradiology practice and decrease errors. This may be useful to meet the ongoing radiologist shortages.

Unlocking Peer Review: Elevating Scholarly Writing and Research Competence in Urology Residency.

PURPOSE OF REVIEW: It is incumbent upon training programs to set the foundation for evidence-based practices and to create opportunities for trainees to develop into academic leaders. As dedicated resident research time and funding have declined in recent years, residency programs and the field at large will need to create new ways to incorporate scholarly activity into residency curricula. RECENT FINDINGS: Literature across specialties demonstrates barriers to resident involvement including lack of time, cost, and absent scholarly mentorship. Peer review stands as a ready-made solution that can be formalized into a collaborative relationship with journals. A formal relationship between professional societies, academic journals, and residencies can facilitate the use of peer review as a teaching tool for residency programs.

How to be a good reviewer: A step-by-step guide for approaching peer review of a scientific manuscript.

Objectives: The peer review process is critical to maintaining quality, reliability, novelty, and innovation in the scientific literature. However, the teaching of scientific peer review is rarely a component of formal scientific or clinical training, and even the most experienced peer reviewers express interest in continuing education. The objective of this review article is to summarize the collective perspectives of experienced journal editors about how to be a good reviewer in a step-by-step guide that can serve as a resource for the performance of peer review of a scientific manuscript. Methods: This is a narrative review. Results: A review of the history and an overview of the modern-day peer review process are provided with attention to the role played by the reviewer, including important reasons for involvement in scientific peer review. The general components of a scientific peer review are described, and a model for how to structure a peer review report is provided. These concepts are also summarized in a reviewer checklist that can be used in real-time to develop and double-check one's reviewer report before submitting it. Conclusions: Peer review is a critically important service for maintaining quality in the scientific literature. Peer review of a scientific manuscript and the associated reviewer's report should assess specific details related to the accuracy, validity, novelty, and interpretation of a study's results. We hope that this article will serve as a resource and guide for reviewers of all levels of experience in the performance of peer review of a scientific manuscript.

Goals of Morbidity and Mortality meetings in paediatric acute care. A qualitative case study.

Morbidity and Mortality meetings are conducted in varied clinical contexts including paediatrics. Widely cited as an educational or quality improvement tool, the reality is more complex. In this qualitative study, the aim was to explore the perceived goals of the paediatric acute care Morbidity and Morbidity meeting. This study used semi-structured interviews and observation within a qualitative case study methodology. Data were collected in a large paediatric quaternary hospital. Analysis generated themes related to meeting observations and the participant's interpretation of meeting goals. A total of 44 interviews were conducted with 14 nurses, 29 doctors, and 1 allied health professional. Thirty-two meetings in six clinical departments were observed. Two themes were developed: complex and nuanced goals; and tensions and contest between and within goals. Meeting goals to evaluate care, learn, support, adhere, and change and respond were sometimes in competition and had varied interpretations. Morbidity and Mortality meetings in this setting are valued and occupy a complex role which reaches beyond identification of measurable patient safety interventions. Understanding goals more fully can lead to optimised conduct and meaningful measurement of efficacy. The strength in these meetings may be the way they promote an embedded safety culture, and an informed and skilled workforce.

Scientific and Regulatory Policy Committee Points to Consider: Review of the United States Food and Drug Administration (FDA) Guidance on Pathology Peer Review in Nonclinical Toxicology Studies.

In December 2021, the United States Food and Drug Administration (FDA) issued the final guidance for industry titled Pathology Peer Review in Nonclinical Toxicology Studies: Questions and Answers. The stated purpose of the FDA guidance is to provide information to sponsors, applicants, and nonclinical laboratory personnel regarding the management and conduct of histopathology peer review as part of nonclinical toxicology studies conducted in compliance with good laboratory practice (GLP) regulations. On behalf of and in collaboration with global societies of toxicologic pathology and the Society of Quality Assurance, the Scientific and Regulatory Policy Committee (SRPC) of the Society of Toxicologic Pathology (STP) initiated a review of this FDA guidance. The STP has previously published multiple papers related to the scientific conduct of a pathology peer review of nonclinical toxicology studies and appropriate documentation practices. The objectives of this review are to provide an in-depth analysis and summary interpretation of the FDA recommendations and share considerations for the conduct of pathology peer review in nonclinical toxicology studies that claim compliance to GLP regulations. In general, this working group is in agreement with the recommendations from the FDA guidance that has added clear expectations for pathology peer review preparation, conduct, and documentation.

Fostering student authorship skills in synthetic biology.

Women and racial minorities are underrepresented in the synthetic biology community. Developing a scholarly identity by engaging in a scientific community through writing and communication is an important component for STEM retention, particularly for underrepresented individuals. Several excellent pedagogical tools have been developed to teach scientific literacy and to measure competency in reading and interpreting scientific literature. However, fewer tools exist to measure learning gains with respect to writing, or that teach the more abstract processes of peer review and scientific publishing, which are essential for developing scholarly identity and publication currency. Here we describe our approach to teaching scientific writing and publishing to undergraduate students within a synthetic biology course. Using gold standard practices in project-based learning, we created a writing project in which students became experts in a specific application area of synthetic biology with relevance to an important global problem or challenge. To measure learning gains associated with our learning outcomes, we adapted and expanded the Student Attitudes, Abilities, and Beliefs (SAAB) concept inventory to include additional questions about the process of scientific writing, authorship, and peer review. Our results suggest the project-based approach was effective in achieving the learning objectives with respect to writing and peer reviewed publication, and resulted in high student satisfaction and student self-reported learning gains. We propose that these educational practices could contribute directly to the development of scientific identity of undergraduate students as synthetic biologists, and will be useful in creating a more diverse synthetic biology research enterprise.

A reflection on the use of virtual nominal group technique in health policy and research priority consensus studies.

BACKGROUND: Nominal group technique (NGT) is widely used in healthcare research to facilitate decision-making and consensus-building. Traditional NGT requires face-to-face interaction and its limitations include potential biases, logistical challenges and high costs. The COVID-19 pandemic necessitated a shift to virtual methods, which led to the development of virtual nominal group technique (vNGT). Aim To reflect on the use of vNGT, particularly in the context of Ghader et al (2023 ), to understand its effectiveness in overcoming the limitations of traditional NGT and on its applicability in pandemic-affected scenarios. METHOD: This paper reviews and discusses literature on the use and effectiveness of NGT compared to other consensus-building methods and examines the emergence of vNGT in overcoming the limitations of traditional NGT. The authors also reflect on their use of vNGT in a study to identify cardiovascular research priorities in the UAE and provide details of its implementation. DISCUSSION: vNGT bridges the gap between the localised nature of traditional NGT and the global reach of the Delphi technique. It allows for the inclusion of diverse participants, saves costs and offers time-efficiency. The study demonstrated vNGT's adaptability, with participants engaging in idea generation, discussion and prioritisation using online tools. However, challenges persist with vNGT, including reduced accessibility for certain demographics, timing issues across time zones and technical difficulties. CONCLUSION: vNGT successfully integrates the interactive, consensus-building aspects of NGT with the broad reach of Delphi. It can be valuable in research and decision-making, especially in an era of increased remote collaboration. IMPLICATIONS FOR PRACTICE: vNGT can significantly impact healthcare research and policy formulation by enabling more inclusive, cost-effective and timely consensus-building processes. However, considerations for accessibility and technical support are crucial for its wider adoption and effectiveness.

Navigating the journey as a peer reviewer: a roadmap to success.

Peer review is an essential step in scientific progress and clinical improvement, providing opportunity for research to be critically evaluated and improved by one's colleagues. Pharmacists from all job settings are called to serve as peer reviewers in the ever-growing publication landscape of the profession. Despite challenges to engagement such as time and compensation, peer review provides considerable professional development for both authors and reviewers alike. This article will serve as a practical guide for peer reviewers, discussing best practices as well as the handling of different situations that may arise during the process.

Quality management system in forensic science: An African perspective.

Critical issues in forensic science quality management have emerged in recent decades. The debate on accrediting quality management systems of forensic laboratories is relevant to the African context. Neuteboom, Ross, Bugeja, Willis, Roux, and Lothridge (2022) have conducted a comprehensive survey exploring critical issues in their article "Quality Management in Forensic Science: A Closer Inspection." Their work is a crucial foundation for our discussion, urging the African forensic community to engage in more in-depth conversations. This letter briefly describes the survey, discussing embracing the Sydney Declaration (SD) for Forensic Sciences and issues of quality management systems comprising standards, accreditation, and potential regulation, and highlights the issue of cognitive competency from an African perspective. This underscores the urgent need for critical dialogue, emphasizing that the time for action is now, and urges practitioners, particularly in Africa, to enhance quality management systems to deliver superior forensic products.

Evaluating Peer Review of Palliative Radiation Plans at a Canadian Tertiary Care Cancer Center.

INTRODUCTION: Peer review (PR) of palliative-intent radiation plans is an important but understudied component of quality assurance. This retrospective review aims to improve our understanding of palliative PR by examining the characteristics of reviewed plans and peer feedback along with the associated time burden of two different types of PR processes. METHODS: This single-institution, quality assurance project assessed palliative PR between 2018 and 2020. Initially, the PR involved a multi-disciplinary team PR. Subsequently, it transitioned to independent PR by a single physician. Characteristics of reviewed plans and feedback on PR were captured and abstracted. Time requirements of PR were based on self-reported estimates and attendance records. RESULTS: A total of 1942 cases were reviewed, representing 85.7% (1942/2266) of all palliative-intent plans between 2018 and 2020. A total of 41.1% (n=799) were simple (2D/3D) radiation plans while 56.0% (n=1087) were complex (volumetric modulated arc therapy (VMAT) or tomotherapy) plans. Approximately one-third (30.4%, n=590) of all plans were stereotactic treatments. The rate of any peer feedback was 2.3% (n=45), while the rate of a specific recommended or implemented change was 1.2% (n=24) and 0.9% (n=18), respectively. PR before the start of treatment was associated with more frequent recommended (p=0.005) and implemented changes (p=0.008). Most other factors, including plan complexity and use of stereotactic radiation, were not predictive in this analysis. Comparing the independent versus team PR approach, there was no significant difference in recommended or implemented changes. The mean±standard deviation (SD) staff time required per plan reviewed was 36±6 and 37±6 minutes, including 21±6 and 10±6 minutes of physician time, for team and independent PR, respectively. CONCLUSION: This work highlights the high frequency of complex and stereotactic radiation in the palliative setting, along with the importance of timely PR and the potential benefit of reviewing even simple, 2D/3D radiation plans. Additionally, from a process perspective, our work showed that independent PR may require less dedicated physician time.

Evaluation and Comparison of Academic Impact and Disruptive Innovation Level of Medical Journals: Bibliometric Analysis and Disruptive Evaluation.

BACKGROUND: As an important platform for researchers to present their academic findings, medical journals have a close relationship between their evaluation orientation and the value orientation of their published research results. However, the differences between the academic impact and level of disruptive innovation of medical journals have not been examined by any study yet. OBJECTIVE: This study aims to compare the relationships and differences between the academic impact, disruptive innovation levels, and peer review results of medical journals and published research papers. We also analyzed the similarities and differences in the impact evaluations, disruptive innovations, and peer reviews for different types of medical research papers and the underlying reasons. METHODS: The general and internal medicine Science Citation Index Expanded (SCIE) journals in 2018 were chosen as the study object to explore the differences in the academic impact and level of disruptive innovation of medical journals based on the OpenCitations Index of PubMed open PMID-to-PMID citations (POCI) and H1Connect databases, respectively, and we compared them with the results of peer review. RESULTS: First, the correlation coefficients of the Journal Disruption Index (JDI) with the Journal Cumulative Citation for 5 years (JCC5), Journal Impact Factor (JIF), and Journal Citation Indicator (JCI) were 0.677, 0.585, and 0.621, respectively. The correlation coefficient of the absolute disruption index (Dz) with the Cumulative Citation for 5 years (CC5) was 0.635. However, the average difference in the disruptive innovation and academic influence rankings of journals reached 20 places (about 17.5%). The average difference in the disruptive innovation and influence rankings of research papers reached about 2700 places (about 17.7%). The differences reflect the essential difference between the two evaluation systems. Second, the top 7 journals selected based on JDI, JCC5, JIF, and JCI were the same, and all of them were H-journals. Although 8 (8/15, 53%), 96 (96/150, 64%), and 880 (880/1500, 58.67%) of the top 0.1%, top 1%, and top 10% papers selected based on Dz and CC5, respectively, were the same. Third, research papers with the "changes clinical practice" tag showed only moderate innovation (4.96) and impact (241.67) levels but had high levels of peer-reviewed recognition (6.00) and attention (2.83). CONCLUSIONS: The results of the study show that research evaluation based on innovative indicators is detached from the traditional impact evaluation system. The 3 evaluation systems (impact evaluation, disruptive innovation evaluation, and peer review) only have high consistency for authoritative journals and top papers. Neither a single impact indicator nor an innovative indicator can directly reflect the impact of medical research for clinical practice. How to establish an integrated, comprehensive, scientific, and reasonable journal evaluation system to improve the existing evaluation system of medical journals still needs further research.

Development of a standardized peer review and oversight process in cases of suspected child physical abuse.

Peer review of medical opinions provided in cases of suspected child physical abuse is generally considered to be best practice for pediatricians engaged in this field. However, there are no published standardized guidelines on how pediatricians should undertake physical abuse peer review including case selection and process. Due to the high-stakes nature in the field of child abuse pediatrics, rigorous quality assurance practices and oversight mechanisms are essential to safeguard children, families, health care providers, and intersecting systems. The Suspected Child Abuse and Neglect program at The Hospital for Sick Children, Toronto, Canada developed a structured peer review process for cases of suspected physical abuse. Included in the process is an approach for the evaluation of institutional complaints received related to a child abuse pediatrician's medical opinion. This quality assurance process is presented so that other child abuse pediatricians and programs may replicate or adapt the protocol for their own local context.

So .You Want to be a Peer Reviewer?

Peer review is essential to preserving the integrity of the scientific publication process. Peer reviewers must adhere to the norms of the peer review process, including confidentiality, avoiding actual or apparent conflicts of interest, timeliness, constructiveness, and thoroughness. This mini review will discuss some of the different formats in which peer review might occur, as well as advantages and disadvantages of each. The topics then shift to providing advice for prospective reviewers, as well as a suggested format for use in writing a review.