A discipline-wide investigation of the replicability of Psychology papers over the past two decades.

Conjecture about the weak replicability in social sciences has made scholars eager to quantify the scale and scope of replication failure for a discipline. Yet small-scale manual replication methods alone are ill-suited to deal with this big data problem. Here, we conduct a discipline-wide replication census in science. Our sample (N = 14,126 papers) covers nearly all papers published in the six top-tier Psychology journals over the past 20 y. Using a validated machine learning model that estimates a paper's likelihood of replication, we found evidence that both supports and refutes speculations drawn from a relatively small sample of manual replications. First, we find that a single overall replication rate of Psychology poorly captures the varying degree of replicability among subfields. Second, we find that replication rates are strongly correlated with research methods in all subfields. Experiments replicate at a significantly lower rate than do non-experimental studies. Third, we find that authors' cumulative publication number and citation impact are positively related to the likelihood of replication, while other proxies of research quality and rigor, such as an author's university prestige and a paper's citations, are unrelated to replicability. Finally, contrary to the ideal that media attention should cover replicable research, we find that media attention is positively related to the likelihood of replication failure. Our assessments of the scale and scope of replicability are important next steps toward broadly resolving issues of replicability.

Non-White scientists appear on fewer editorial boards, spend more time under review, and receive fewer citations.

Disparities continue to pose major challenges in various aspects of science. One such aspect is editorial board composition, which has been shown to exhibit racial and geographical disparities. However, the literature on this subject lacks longitudinal studies quantifying the degree to which the racial composition of editors reflects that of scientists. Other aspects that may exhibit racial disparities include the time spent between the submission and acceptance of a manuscript and the number of citations a paper receives relative to textually similar papers, but these have not been studied to date. To fill this gap, we compile a dataset of 1,000,000 papers published between 2001 and 2020 by six publishers, while identifying the handling editor of each paper. Using this dataset, we show that most countries in Asia, Africa, and South America (where the majority of the population is ethnically non-White) have fewer editors than would be expected based on their share of authorship. Focusing on US-based scientists reveals Black as the most underrepresented race. In terms of acceptance delay, we find, again, that papers from Asia, Africa, and South America spend more time compared to other papers published in the same journal and the same year. Regression analysis of US-based papers reveals that Black authors suffer from the greatest delay. Finally, by analyzing citation rates of US-based papers, we find that Black and Hispanic scientists receive significantly fewer citations compared to White ones doing similar research. Taken together, these findings highlight significant challenges facing non-White scientists.

A network-based normalized impact measure reveals successful periods of scientific discovery across discipline.

The impact of a scientific publication is often measured by the number of citations it receives from the scientific community. However, citation count is susceptible to well-documented variations in citation practices across time and discipline, limiting our ability to compare different scientific achievements. Previous efforts to account for citation variations often rely on a priori discipline labels of papers, assuming that all papers in a discipline are identical in their subject matter. Here, we propose a network-based methodology to quantify the impact of an article by comparing it with locally comparable research, thereby eliminating the discipline label requirement. We show that the developed measure is not susceptible to discipline bias and follows a universal distribution for all articles published in different years, offering an unbiased indicator for impact across time and discipline. We then use the indicator to identify science-wide high impact research in the past half century and quantify its temporal production dynamics across disciplines, helping us identifying breakthroughs from diverse, smaller disciplines, such as geosciences, radiology, and optics, as opposed to citation-rich biomedical sciences. Our work provides insights into the evolution of science and paves a way for fair comparisons of the impact of diverse contributions across many fields.

Ranking mobility and impact inequality in early academic careers.

How difficult is it for an early career academic to climb the ranks of their discipline? We tackle this question with a comprehensive bibliometric analysis of 57 disciplines, examining the publications of more than 5 million authors whose careers started between 1986 and 2008. We calibrate a simple random walk model over historical data of ranking mobility, which we use to 1) identify which strata of academic impact rankings are the most/least mobile and 2) study the temporal evolution of mobility. By focusing our analysis on cohorts of authors starting their careers in the same year, we find that ranking mobility is remarkably low for the top- and bottom-ranked authors and that this excess of stability persists throughout the entire period of our analysis. We further observe that mobility of impact rankings has increased over time, and that such rise has been accompanied by a decline of impact inequality, which is consistent with the negative correlation that we observe between such two quantities. These findings provide clarity on the opportunities of new scholars entering the academic community, with implications for academic policymaking.

A gender perspective on the global migration of scholars.

Although considerable progress toward gender equality in science has been made in recent decades, female researchers continue to face significant barriers in the academic labor market. International mobility has been increasingly recognized as a strategy for scientists to expand their professional networks, and that could help narrow the gender gap in academic careers. Using bibliometric data on over 33 million Scopus publications, we provide a global and dynamic view of gendered patterns of transnational scholarly mobility, as measured by volume, distance, diversity, and distribution, from 1998 to 2017. We find that, while female researchers continued to be underrepresented among internationally mobile researchers and migrate over shorter distances, this gender gap was narrowing at a faster rate than the gender gap in the population of general active researchers. Globally, the origin and destination countries of both female and male mobile researchers became increasingly diversified, which suggests that scholarly migration has become less skewed and more globalized. However, the range of origin and destination countries continued to be narrower for women than for men. While the United States remained the leading academic destination worldwide, the shares of both female and male scholarly inflows to that country declined from around 25% to 20% over the study period, partially due to the growing relevance of China. This study offers a cross-national measurement of gender inequality in global scholarly migration that is essential for promoting gender-equitable science policies and for monitoring the impact of such interventions.

Intersectional inequalities in science.

The US scientific workforce is primarily composed of White men. Studies have demonstrated the systemic barriers preventing women and other minoritized populations from gaining entry to science; few, however, have taken an intersectional perspective and examined the consequences of these inequalities on scientific knowledge. We provide a large-scale bibliometric analysis of the relationship between intersectional identities, topics, and scientific impact. We find homophily between identities and topic, suggesting a relationship between diversity in the scientific workforce and expansion of the knowledge base. However, topic selection comes at a cost to minoritized individuals for whom we observe both between- and within-topic citation disadvantages. To enhance the robustness of science, research organizations should provide adequate resources to historically underfunded research areas while simultaneously providing access for minoritized individuals into high-prestige networks and topics.

The narrowing of literature use and the restricted mobility of papers in the sciences.

It is a matter of debate whether a shrinking proportion of scholarly literature is getting most of the citations over time. It is also less well understood how a narrowing use of literature would affect the circulation of ideas in the sciences. Here, I show that the utilization of scientific literature follows dual tendencies over time: while a larger proportion of literature is cited at least a few times, citations are also concentrated more at the top of the citation distribution. Parallel to the latter trend, a paper's future importance increasingly depends on its past citation performance. A random network model shows that the citation concentration is directly related to the greater stability of citation performance. The presented evidence suggests that the growing heterogeneity of citation impact restricts the mobility of research articles that do not gain attention early on. While concentration grows from the beginning of the studied period in 1970, citation dispersion manifests itself significantly only from the mid-1990s, when the popularity of freshly published papers also increased. Most likely, advanced information technologies to disseminate papers are behind both of these latter trends.

Gendered citation patterns among the scientific elite.

Diversity in science is necessary to improve innovation and increase the capacity of the scientific workforce. Despite decades-long efforts to increase gender diversity, however, women remain a small minority in many fields, especially in senior positions. The dearth of elite women scientists, in turn, leaves fewer women to serve as mentors and role models for young women scientists. To shed light on gender disparities in science, we study prominent scholars who were elected to the National Academy of Sciences. We construct author citation networks that capture the structure of recognition among scholars' peers. We identify gender disparities in the patterns of peer citations and show that these differences are strong enough to accurately predict the scholar's gender. In contrast, we do not observe disparities due to prestige, with few significant differences in the structure of citations of scholars affiliated with high-ranked and low-ranked institutions. These results provide further evidence that a scholar's gender plays a role in the mechanisms of success in science.

Dynamics of cross-platform attention to retracted papers.

Retracted papers often circulate widely on social media, digital news, and other websites before their official retraction. The spread of potentially inaccurate or misleading results from retracted papers can harm the scientific community and the public. Here, we quantify the amount and type of attention 3,851 retracted papers received over time in different online platforms. Comparing with a set of nonretracted control papers from the same journals with similar publication year, number of coauthors, and author impact, we show that retracted papers receive more attention after publication not only on social media but also, on heavily curated platforms, such as news outlets and knowledge repositories, amplifying the negative impact on the public. At the same time, we find that posts on Twitter tend to express more criticism about retracted than about control papers, suggesting that criticism-expressing tweets could contain factual information about problematic papers. Most importantly, around the time they are retracted, papers generate discussions that are primarily about the retraction incident rather than about research findings, showing that by this point, papers have exhausted attention to their results and highlighting the limited effect of retractions. Our findings reveal the extent to which retracted papers are discussed on different online platforms and identify at scale audience criticism toward them. In this context, we show that retraction is not an effective tool to reduce online attention to problematic papers.

Flat teams drive scientific innovation.

With teams growing in all areas of scientific and scholarly research, we explore the relationship between team structure and the character of knowledge they produce. Drawing on 89,575 self-reports of team member research activity underlying scientific publications, we show how individual activities cohere into broad roles of 1) leadership through the direction and presentation of research and 2) support through data collection, analysis, and discussion. The hidden hierarchy of a scientific team is characterized by its lead (or L) ratio of members playing leadership roles to total team size. The L ratio is validated through correlation with imputed contributions to the specific paper and to science as a whole, which we use to effectively extrapolate the L ratio for 16,397,750 papers where roles are not explicit. We find that, relative to flat, egalitarian teams, tall, hierarchical teams produce less novelty and more often develop existing ideas, increase productivity for those on top and decrease it for those beneath, and increase short-term citations but decrease long-term influence. These effects hold within person-the same person on the same-sized team produces science much more likely to disruptively innovate if they work on a flat, high-L-ratio team. These results suggest the critical role flat teams play for sustainable scientific advance and the training and advancement of scientists.

Slowed canonical progress in large fields of science.

In many academic fields, the number of papers published each year has increased significantly over time. Policy measures aim to increase the quantity of scientists, research funding, and scientific output, which is measured by the number of papers produced. These quantitative metrics determine the career trajectories of scholars and evaluations of academic departments, institutions, and nations. Whether and how these increases in the numbers of scientists and papers translate into advances in knowledge is unclear, however. Here, we first lay out a theoretical argument for why too many papers published each year in a field can lead to stagnation rather than advance. The deluge of new papers may deprive reviewers and readers the cognitive slack required to fully recognize and understand novel ideas. Competition among many new ideas may prevent the gradual accumulation of focused attention on a promising new idea. Then, we show data supporting the predictions of this theory. When the number of papers published per year in a scientific field grows large, citations flow disproportionately to already well-cited papers; the list of most-cited papers ossifies; new papers are unlikely to ever become highly cited, and when they do, it is not through a gradual, cumulative process of attention gathering; and newly published papers become unlikely to disrupt existing work. These findings suggest that the progress of large scientific fields may be slowed, trapped in existing canon. Policy measures shifting how scientific work is produced, disseminated, consumed, and rewarded may be called for to push fields into new, more fertile areas of study.

"Value-Conditioned Interest": Secularity, Institutional Support, and the Sociology of Religion in Graduate Departments.

Although religion once played a central role in sociological inquiry, today it has fallen from prominence and now occupies a marginal space in the field. Sociologists of religion suggest that this neglect can be explained by the unusual irreligiosity of academia. However, some are hopeful that changes to graduate training and other institutional interventions can overcome this propensity and encourage more sociological engagement with religion, one of the most influential forces in contemporary society. Drawing on a new dataset of 473 sociology graduate students in the top 25 departments in the United States, we assess how personal secularity and departmental support for religious inquiry predict sociological engagement with religion. Personal secularity is a strong predictor of the decision to study religion and of the overall perception of the relevance of religion in contemporary society. Coming from a department where religion is discussed and faculty pursue research on religion predicts the perception of religion's relevance for the secular majority. Our findings contribute to the understanding of sociological knowledge creation and how individual and institutional proclivities-together-shape what we deem important enough to study.

Improving data access democratizes and diversifies science.

The foundation of the scientific method rests on access to data, and yet such access is often restricted or costly. We investigate how improved data access shifts the quantity, quality, and diversity of scientific research. We examine the impact of reductions in cost and sharing restrictions for satellite imagery data from NASA's Landsat program (the longest record of remote-sensing observations of the Earth) on academic science using a sample of about 24,000 Landsat publications by over 34,000 authors matched to almost 3,000 unique study locations. Analyses show that improved access had a substantial and positive effect on the quantity and quality of Landsat-enabled science. Improved data access also democratizes science by disproportionately helping scientists from the developing world and lower-ranked institutions to publish using Landsat data. This democratization in turn increases the geographic and topical diversity of Landsat-enabled research. Scientists who start using Landsat data after access is improved tend to focus on previously understudied regions close to their home location and introduce novel research topics. These findings suggest that policies that improve access to valuable scientific data may promote scientific progress, reduce inequality among scientists, and increase the diversity of scientific research.

Historical comparison of gender inequality in scientific careers across countries and disciplines.

There is extensive, yet fragmented, evidence of gender differences in academia suggesting that women are underrepresented in most scientific disciplines and publish fewer articles throughout a career, and their work acquires fewer citations. Here, we offer a comprehensive picture of longitudinal gender differences in performance through a bibliometric analysis of academic publishing careers by reconstructing the complete publication history of over 1.5 million gender-identified authors whose publishing career ended between 1955 and 2010, covering 83 countries and 13 disciplines. We find that, paradoxically, the increase of participation of women in science over the past 60 years was accompanied by an increase of gender differences in both productivity and impact. Most surprisingly, though, we uncover two gender invariants, finding that men and women publish at a comparable annual rate and have equivalent career-wise impact for the same size body of work. Finally, we demonstrate that differences in publishing career lengths and dropout rates explain a large portion of the reported career-wise differences in productivity and impact, although productivity differences still remain. This comprehensive picture of gender inequality in academia can help rephrase the conversation around the sustainability of women's careers in academia, with important consequences for institutions and policy makers.

Mentorship and protégé success in STEM fields.

Einstein believed that mentors are especially influential in a protégé's intellectual development, yet the link between mentorship and protégé success remains a mystery. We marshaled genealogical data on nearly 40,000 scientists who published 1,167,518 papers in biomedicine, chemistry, math, or physics between 1960 and 2017 to investigate the relationship between mentorship and protégé achievement. In our data, we find groupings of mentors with similar records and reputations who attracted protégés of similar talents and expected levels of professional success. However, each grouping has an exception: One mentor has an additional hidden capability that can be mentored to their protégés. They display skill in creating and communicating prizewinning research. Because the mentor's ability for creating and communicating celebrated research existed before the prize's conferment, protégés of future prizewinning mentors can be uniquely exposed to mentorship for conducting celebrated research. Our models explain 34-44% of the variance in protégé success and reveals three main findings. First, mentorship strongly predicts protégé success across diverse disciplines. Mentorship is associated with a 2×-to-4× rise in a protégé's likelihood of prizewinning, National Academy of Science (NAS) induction, or superstardom relative to matched protégés. Second, mentorship is significantly associated with an increase in the probability of protégés pioneering their own research topics and being midcareer late bloomers. Third, contrary to conventional thought, protégés do not succeed most by following their mentors' research topics but by studying original topics and coauthoring no more than a small fraction of papers with their mentors.

Three dimensions of scientific impact.

The growing popularity of bibliometric indexes (whose most famous example is the h index by J. E. Hirsch [J. E. Hirsch, Proc. Natl. Acad. Sci. U.S.A. 102, 16569-16572 (2005)]) is opposed by those claiming that one's scientific impact cannot be reduced to a single number. Some even believe that our complex reality fails to submit to any quantitative description. We argue that neither of the two controversial extremes is true. By assuming that some citations are distributed according to the rich get richer rule (success breeds success, preferential attachment) while some others are assigned totally at random (all in all, a paper needs a bibliography), we have crafted a model that accurately summarizes citation records with merely three easily interpretable parameters: productivity, total impact, and how lucky an author has been so far.

Scientific rewards for biomedical specialization are large and persistent.

BACKGROUND: While specialization plays an essential role in how scientific research is pursued, we understand little about its effects on a researcher's impact and career. In particular, the extent to which one specializes within their chosen fields likely has complex relationships with productivity, career stage, and eventual impact. Here, we develop a novel and fine-grained approach for measuring a researcher's level of specialization at each point in their career and apply it to the publication data of almost 30,000 established biomedical researchers to measure the effect that specialization has on the impact of a researcher's publications. RESULTS: Using a within-researcher, panel-based econometric framework, we arrive at several important results. First, there are significant scientific rewards for specialization-25% more citations per standard deviation increase in specialization. Second, these benefits are much higher early in a researcher's career-as large as 75% per standard deviation increase in specialization. Third, rewards are higher for researchers who publish few papers relative to their peers. Finally, we find that, all else equal, researchers who make large changes in their research direction see generally increased impact. CONCLUSIONS: The extent to which one specializes, particularly at the early stages of a biomedical research career, appears to play a significant role in determining the citation-based impact of their publications. When this measure of impact is, implicitly or explicitly, an input into decision-making processes within the scientific system (for example, for job opportunities, promotions, or invited talks), these findings lead to some important implications for the system-level organization of scientific research and the incentives that exist therein. We propose several mechanisms within modern scientific systems that likely lead to the scientific rewards we observe and discuss them within the broader context of reward structures in biomedicine and science more generally.

Shorter distances between papers over time are due to more cross-field references and increased citation rate to higher-impact papers.

The exponential increase in the number of scientific publications raises the question of whether the sciences are expanding into a fractured structure, making cross-field communication difficult. On the other hand, scientists may be motivated to learn extensively across fields to enhance their innovative capacity, and this may offset the negative effects of fragmentation. Through an investigation of the distances within and clustering of cross-sectional citation networks, this study presents evidence that fields of science become more integrated over time. The average citation distance between papers published in the same year decreased from ∼5.33 to 3.18 steps between 1950 and 2018. This observation is attributed to the growth of cross-field communication throughout the entire period as well as the growing importance of high-impact papers to bridge networks in the same year. Three empirical findings support this conclusion. First, distances decreased between almost all disciplines throughout the time period. Second, inequality in the number of citations received by papers increased, and, as a consequence, the shortest paths in the network depend more on high-impact papers later in the period. Third, the dispersion of connections between fields increased continually. Moreover, these changes did not entail a lower level of clustering of citations. Both within- and cross-field citations show a similar rate of slowly growing clustering values in all years. The latter findings suggest that domain-spanning scholarly communication is partly enabled by new fields that connect disciplines.

Productivity, prominence, and the effects of academic environment.

Faculty at prestigious institutions produce more scientific papers, receive more citations and scholarly awards, and are typically trained at more-prestigious institutions than faculty with less prestigious appointments. This imbalance is often attributed to a meritocratic system that sorts individuals into more-prestigious positions according to their reputation, past achievements, and potential for future scholarly impact. Here, we investigate the determinants of scholarly productivity and measure their dependence on past training and current work environments. To distinguish the effects of these environments, we apply a matched-pairs experimental design to career and productivity trajectories of 2,453 early-career faculty at all 205 PhD-granting computer science departments in the United States and Canada, who together account for over 200,000 publications and 7.4 million citations. Our results show that the prestige of faculty's current work environment, not their training environment, drives their future scientific productivity, while current and past locations drive prominence. Furthermore, the characteristics of a work environment are more predictive of faculty productivity and impact than mechanisms representing preferential selection or retention of more-productive scholars by more-prestigious departments. These results identify an environmental mechanism for cumulative advantage, in which an individual's past successes are "locked in" via placement into a more prestigious environment, which directly facilitates future success. The scientific productivity of early-career faculty is thus driven by where they work, rather than where they trained for their doctorate, indicating a limited role for doctoral prestige in predicting scientific contributions.

Establishing Institutional Scores With the Rigor and Transparency Index: Large-scale Analysis of Scientific Reporting Quality.

BACKGROUND: Improving rigor and transparency measures should lead to improvements in reproducibility across the scientific literature; however, the assessment of measures of transparency tends to be very difficult if performed manually. OBJECTIVE: This study addresses the enhancement of the Rigor and Transparency Index (RTI, version 2.0), which attempts to automatically assess the rigor and transparency of journals, institutions, and countries using manuscripts scored on criteria found in reproducibility guidelines (eg, Materials Design, Analysis, and Reporting checklist criteria). METHODS: The RTI tracks 27 entity types using natural language processing techniques such as Bidirectional Long Short-term Memory Conditional Random Field-based models and regular expressions; this allowed us to assess over 2 million papers accessed through PubMed Central. RESULTS: Between 1997 and 2020 (where data were readily available in our data set), rigor and transparency measures showed general improvement (RTI 2.29 to 4.13), suggesting that authors are taking the need for improved reporting seriously. The top-scoring journals in 2020 were the Journal of Neurochemistry (6.23), British Journal of Pharmacology (6.07), and Nature Neuroscience (5.93). We extracted the institution and country of origin from the author affiliations to expand our analysis beyond journals. Among institutions publishing >1000 papers in 2020 (in the PubMed Central open access set), Capital Medical University (4.75), Yonsei University (4.58), and University of Copenhagen (4.53) were the top performers in terms of RTI. In country-level performance, we found that Ethiopia and Norway consistently topped the RTI charts of countries with 100 or more papers per year. In addition, we tested our assumption that the RTI may serve as a reliable proxy for scientific replicability (ie, a high RTI represents papers containing sufficient information for replication efforts). Using work by the Reproducibility Project: Cancer Biology, we determined that replication papers (RTI 7.61, SD 0.78) scored significantly higher (P<.001) than the original papers (RTI 3.39, SD 1.12), which according to the project required additional information from authors to begin replication efforts. CONCLUSIONS: These results align with our view that RTI may serve as a reliable proxy for scientific replicability. Unfortunately, RTI measures for journals, institutions, and countries fall short of the replicated paper average. If we consider the RTI of these replication studies as a target for future manuscripts, more work will be needed to ensure that the average manuscript contains sufficient information for replication attempts.