Is novel research worth doing? Evidence from peer review at 49 journals.

There are long-standing concerns that peer review, which is foundational to scientific institutions like journals and funding agencies, favors conservative ideas over novel ones. We investigate the association between novelty and the acceptance of manuscripts submitted to a large sample of scientific journals. The data cover 20,538 manuscripts submitted between 2013 and 2018 to the journals Cell and Cell Reports and 6,785 manuscripts submitted in 2018 to 47 journals published by the Institute of Physics Publishing. Following previous work that found that a balance of novel and conventional ideas predicts citation impact, we measure the novelty and conventionality of manuscripts by the atypicality of combinations of journals in their reference lists, taking the 90th percentile most atypical combination as "novelty" and the 50th percentile as "conventionality." We find that higher novelty is consistently associated with higher acceptance; submissions in the top novelty quintile are 6.5 percentage points more likely than bottom quintile ones to get accepted. Higher conventionality is also associated with acceptance (+16.3% top-bottom quintile difference). Disagreement among peer reviewers was not systematically related to submission novelty or conventionality, and editors select strongly for novelty even conditional on reviewers' recommendations (+7.0% top-bottom quintile difference). Manuscripts exhibiting higher novelty were more highly cited. Overall, the findings suggest that journal peer review favors novel research that is well situated in the existing literature, incentivizing exploration in science and challenging the view that peer review is inherently antinovelty.

A simulation-based analysis of the impact of rhetorical citations in science.

Authors of scientific papers are usually encouraged to cite works that meaningfully influenced their research (substantive citations) and avoid citing works that had no meaningful influence (rhetorical citations). Rhetorical citations are assumed to degrade incentives for good work and benefit prominent papers and researchers. Here, we explore if rhetorical citations have some plausibly positive effects for science and disproportionately benefit the less prominent papers and researchers. We developed a set of agent-based models where agents can cite substantively and rhetorically. Agents first choose papers to read based on their expected quality, become influenced by those that are sufficiently good, and substantively cite them. Next, agents fill any remaining slots in their reference lists with rhetorical citations that support their narrative, regardless of whether they were actually influential. We then turned agents' ability to cite rhetorically on-and-off to measure its effects. Enabling rhetorical citing increased the correlation between paper quality and citations, increased citation churn, and reduced citation inequality. This occurred because rhetorical citing redistributed some citations from a stable set of elite-quality papers to a more dynamic set with high-to-moderate quality and high rhetorical value. Increasing the size of reference lists, often seen as an undesirable trend, amplified the effects. Overall, rhetorical citing may help deconcentrate attention and make it easier to displace established ideas.

Author-suggested reviewers rate manuscripts much more favorably: A cross-sectional analysis of the neuroscience section of PLOS ONE.

Peer review is an important part of science, aimed at providing expert and objective assessment of a manuscript. Because of many factors, including time constraints, unique expertise needs, and deference, many journals ask authors to suggest peer reviewers for their own manuscript. Previous researchers have found differing effects about this practice that might be inconclusive due to sample sizes. In this article, we analyze the association between author-suggested reviewers and review invitation, review scores, acceptance rates, and subjective review quality using a large dataset of close to 8K manuscripts from 46K authors and 21K reviewers from the journal PLOS ONE's Neuroscience section. We found that all-author-suggested review panels increase the chances of acceptance by 20 percent points vs all-editor-suggested panels while agreeing to review less often. While PLOS ONE has since ended the practice of asking for suggested reviewers, many others still use them and perhaps should consider the results presented here.

Conservatism Gets Funded? A Field Experiment on the Role of Negative Information in Novel Project Evaluation.

The evaluation and selection of novel projects lies at the heart of scientific and technological innovation, and yet there are persistent concerns about bias, such as conservatism. This paper investigates the role that the format of evaluation, specifically information sharing among expert evaluators, plays in generating conservative decisions. We executed two field experiments in two separate grant-funding opportunities at a leading research university, mobilizing 369 evaluators from seven universities to evaluate 97 projects, resulting in 761 proposal-evaluation pairs and more than $250,000 in awards. We exogenously varied the relative valence (positive and negative) of others' scores and measured how exposures to higher and lower scores affect the focal evaluator's propensity to change their initial score. We found causal evidence of a negativity bias, where evaluators lower their scores by more points after seeing scores more critical than their own rather than raise them after seeing more favorable scores. Qualitative coding of the evaluators' justifications for score changes reveals that exposures to lower scores were associated with greater attention to uncovering weaknesses, whereas exposures to neutral or higher scores were associated with increased emphasis on nonevaluation criteria, such as confidence in one's judgment. The greater power of negative information suggests that information sharing among expert evaluators can lead to more conservative allocation decisions that favor protecting against failure rather than maximizing success.

The wisdom of polarized crowds.

As political polarization in the United States continues to rise1-3, the question of whether polarized individuals can fruitfully cooperate becomes pressing. Although diverse perspectives typically lead to superior team performance on complex tasks4,5, strong political perspectives have been associated with conflict, misinformation and a reluctance to engage with people and ideas beyond one's echo chamber6-8. Here, we explore the effect of ideological composition on team performance by analysing millions of edits to Wikipedia's political, social issues and science articles. We measure editors' online ideological preferences by how much they contribute to conservative versus liberal articles. Editor surveys suggest that online contributions associate with offline political party affiliation and ideological self-identity. Our analysis reveals that polarized teams consisting of a balanced set of ideologically diverse editors produce articles of a higher quality than homogeneous teams. The effect is most clearly seen in Wikipedia's political articles, but also in social issues and even science articles. Analysis of article 'talk pages' reveals that ideologically polarized teams engage in longer, more constructive, competitive and substantively focused but linguistically diverse debates than teams of ideological moderates. More intense use of Wikipedia policies by ideologically diverse teams suggests institutional design principles to help unleash the power of polarization.