Science peer review for the 21st century: Assessing scientific consensus for decision-making while managing conflict of interests, reviewer and process bias.

ABSTRACT

Science peer review plays an important role in the advancement and acceptance of scientific information, particularly when used to support decision-making. A model for science peer review is proposed here using a large, multi-tiered case study to engage a broader segment of the scientific community to support decision making on science matters, and to incorporate many of the design advantages of the two common forms of peer review (journal peer review, science advisory panels). This peer review consisted of a two-tiered structure consisting of seven panels (five review panels in Tier 1, two review panels in Tier 2), which focused on safety data for a modified risk tobacco product (MRTP). Experts from all over the world were invited to apply to one or more positions on seven peer review panels. 66 peer reviewers were selected from available applicants using objective metrics of their expertise, and for some panels based upon a consideration of panel diversity with respect to demographic parameters (e.g., geographic region, sector of employment, years of experience). All peer reviewers participated anonymously in which a third-party auditor was used to provide independent verification of their expertise. Peer reviewers were provided electronic links to all review material which included access to publications, reports, omics data, and histopathology slides, with topic-specific panels focusing on topic-specific components of the review package. Peer reviews consisted either of single-round, or multi-round (e.g., modified Delphi) format. Peer reviewer responses to the charge questions were collected via an online survey system, and were assembled into a database. Responses in the database were subject to analyses to assess the degree of favorability (i.e., supportive of the review material), degree of consensus, reproducibility of replicate panels, hidden sources of bias, and outlier response patterns. Conclusions: By careful consideration of science peer review design elements we have shown that 1) panel participation can be broadened to include scientists who would otherwise not participate; 2) panel diversity can be managed in an unbiased manner without adverse impacts to panel expertise; 3) results obtained from independent concurrent panels are shown to be reproducible; and 4) there are benefits of collecting input from expert panels via a structured format (i.e., survey) to support characterization of consensus, identification of hidden sources of bias, and identification of potential outlier participants.