Exploring a formal approach to selecting studies for replication: A feasibility study in social neuroscience.

Replication of published results is crucial for ensuring the robustness and self-correction of research, yet replications are scarce in many fields. Replicating researchers will therefore often have to decide which of several relevant candidates to target for replication. Formal strategies for efficient study selection have been proposed, but none have been explored for practical feasibility - a prerequisite for validation. Here we move one step closer to efficient replication study selection by exploring the feasibility of a particular selection strategy that estimates replication value as a function of citation impact and sample size (Isager, van 't Veer, & Lakens, 2021). We tested our strategy on a sample of fMRI studies in social neuroscience. We first report our efforts to generate a representative candidate set of replication targets. We then explore the feasibility and reliability of estimating replication value for the targets in our set, resulting in a dataset of 1358 studies ranked on their value of prioritising them for replication. In addition, we carefully examine possible measures, test auxiliary assumptions, and identify boundary conditions of measuring value and uncertainty. We end our report by discussing how future validation studies might be designed. Our study demonstrates the importance of investigating how to implement study selection strategies in practice. Our sample and study design can be extended to explore the feasibility of other formal study selection strategies that have been proposed.

Community consensus on core open science practices to monitor in biomedicine.

The state of open science needs to be monitored to track changes over time and identify areas to create interventions to drive improvements. In order to monitor open science practices, they first need to be well defined and operationalized. To reach consensus on what open science practices to monitor at biomedical research institutions, we conducted a modified 3-round Delphi study. Participants were research administrators, researchers, specialists in dedicated open science roles, and librarians. In rounds 1 and 2, participants completed an online survey evaluating a set of potential open science practices, and for round 3, we hosted two half-day virtual meetings to discuss and vote on items that had not reached consensus. Ultimately, participants reached consensus on 19 open science practices. This core set of open science practices will form the foundation for institutional dashboards and may also be of value for the development of policy, education, and interventions.

The relationship between parenting engagement and academic performance.

Gender differences in research productivity have been well documented. One frequent explanation of these differences is disproportionate child-related responsibilities for women. However, changing social dynamics around parenting has led to fathers taking an increasingly active role in parenting. This demands a more nuanced approach to understanding the relationship between parenting and productivity for both men and women. To gain insight into associations between parent roles, partner type, research productivity, and research impact, we conducted a global survey that targeted 1.5 million active scientists; we received viable responses from 10,445 parents (< 1% response rate), thus providing a basis for exploratory analyses that shed light on associations between parenting models and research outcomes, across men and women. Results suggest that the gendered effect observed in production may be related by differential engagement in parenting: men who serve in lead roles suffer similar penalties for parenting engagement, but women are more likely to serve in lead roles and to be more engaged across time and tasks, therefore suffering a higher penalty. Taking a period of parental leave is associated with higher levels of productivity; however, the productivity advantage dissipates after six months for the US-sample, and at 12-months for the non-US sample. These results suggest that parental engagement is a more powerful variable to explain gender differences in academic productivity than the mere existence of children, and that policies should factor these labor differentials into account.

A research intelligence approach to assess the research impact of the Dutch university medical centres.

BACKGROUND: The way in which research impact is evaluated and assessed has long been under debate. In recent years the focus is moving away from the use of numerical indicators, towards an emphasis on narratives. The Dutch university medical centres (UMCs) have a long-standing tradition of using bibliometric indicators. Because of the declining interest in indicators alone, this study was designed to repurpose bibliometrics to answer specific strategic questions. In this article we discuss the strategic and policy-based questions, the methodology we used in uncovering relevant information and conclusions we draw from the analyses we performed. The aim of this article is to inform a broader audience about the potential applications of bibliometric information to support a new form of research intelligence. METHODS: In this study we used a curated set of publications from the UMCs. We performed different bibliometric analyses and used bibliometric visualization tools to shed light on research focus, open science practices, collaboration, societal impact and scientific impact. RESULTS: The analyses allowed us to visualize and contextualize the research focus of the UMCs as a whole, but also to show specific focus areas of each UMC. The UMCs are active in the full spectrum of biomedical research, and at the same time are very complementary to each other. Furthermore, we were able to show the development of open access of UMC publications over time, to support the national mission. Visualizing collaboration is a powerful way of showing both the international orientation and the regional and national engine function of UMCs in research. We were able to assess societal impact by looking at the different channels in which publications find their way to societally relevant sources such as news media, policy documents and guidelines. Finally, we assessed scientific impact and put this into an international perspective. CONCLUSIONS: Research intelligence is able to transform bibliometric information by interpretation and annotation into highly relevant insights that can be used for several different strategic purposes and for research impact assessment in general.

Changing publication practices and the typification of the journal article in science and technology studies.

In this article, we study the development of the STS journal article format since the 1980s. Our analysis is based on quantitative data that suggest that the diversity of various journal publication types has diminished over the past four decades, while the format of research articles has become increasingly typified. We contextualize these historical shifts in qualitative terms, drawing on a set of 76 interviews with STS scholars and other stakeholders in scholarly publishing. Here, we first portray the STS publication culture of the 1980s and early 1990s. We then contrast this with an analysis of publishing practices today, which are characterized by a much more structured research process that is largely organized around the production of typified journal articles. Whereas earlier studies have often emphasized the importance of rhetorical persuasion strategies as drivers in the development of scholarly communication formats, our analysis highlights a complementary and historically novel set of shaping factors, namely, increasingly quantified research (self-)assessment practices in the context of a projectification of academic life. We argue that reliance on a highly structured publication format is a distinct strategy for making STS scholarship 'doable' in the sense of facilitating the planning ability and daily conduct of research across a variety of levels - including the writing process, collaboration with peers, attracting funding, and interaction with journals. We conclude by reflecting on the advantages and downsides of the typification of journal articles for STS.

Assessing publication rates from medical students' mandatory research projects in the Netherlands: a follow-up study of 10 cohorts of medical students.

OBJECTIVES: The medical field is facing a clinician-scientist shortage. Medical schools could foster the clinician-scientist workforce by offering students research opportunities. Most medical schools offer elective research programmes. Subsequently, a subset of doctors graduates without any research experience. Mandatory research projects may be more sufficient to develop clinician-scientist, but take more supervision and curricular time. There is limited insight in the scientific outcomes of mandatory research experiences. This study aims to examine publication rates of a mandatory research experience, identify factors associated with publication, and includes postgraduate research engagement. DESIGN AND SETTING: Prospective follow-up study involving 10 cohorts of medical students' mandatory research projects from Leiden University Medical Center. PARTICIPANTS: All medical students who conducted their research project between 2008 and 2018 (n=2329) were included. MAIN OUTCOME MEASURE: Publication rates were defined as peer-reviewed scientific publications, including research papers, reviews, and published meeting abstracts. Postgraduate research engagement was defined as research participation and dissemination of research at scientific conferences or in journals. RESULTS: In total, 644 (27.7%) of all mandatory research experiences resulted in publication, with students mainly as first (n=984, 42.5%) or second author (n=587, 25.3%) and above world average citation impact (mean normalised journal score 1.29, mean normalised citation score 1.23). Students who conducted their research in an academic centre (adjusted OR 2.82; 95% CI 2.10 to 3.77), extended their research (adjusted OR 1.73; 95% CI 1.35 to 2.20), were involved in an excellency track (adjusted OR 2.08; 95% CI 1.44 to 3.01), or conducted clinical (adjusted OR 2.08; 95% CI 1.15 to 3.74) or laboratory (adjusted OR 2.16; 95% CI 1.16 to 4.01) research published their research more often. Later as junior doctors, this group significantly more often disseminate their research results at scientific conferences (adjusted OR 1.89; 95% CI 1.11 to 3.23) or in journals (adjusted OR 1.98; 95% CI 1.14 to 3.43). CONCLUSIONS: Our findings suggest that a significant subset of hands-on mandatory research projects with flexible learning pathways result in tangible research output with proper impact and that such successful experiences can be considered as diving board towards a research-oriented career.

A scientometric overview of CORD-19.

As the COVID-19 pandemic unfolds, researchers from all disciplines are coming together and contributing their expertise. CORD-19, a dataset of COVID-19 and coronavirus publications, has been made available alongside calls to help mine the information it contains and to create tools to search it more effectively. We analyse the delineation of the publications included in CORD-19 from a scientometric perspective. Based on a comparison to the Web of Science database, we find that CORD-19 provides an almost complete coverage of research on COVID-19 and coronaviruses. CORD-19 contains not only research that deals directly with COVID-19 and coronaviruses, but also research on viruses in general. Publications from CORD-19 focus mostly on a few well-defined research areas, in particular: coronaviruses (primarily SARS-CoV, MERS-CoV and SARS-CoV-2); public health and viral epidemics; molecular biology of viruses; influenza and other families of viruses; immunology and antivirals; clinical medicine. CORD-19 publications that appeared in 2020, especially editorials and letters, are disproportionately popular on social media. While we fully endorse the CORD-19 initiative, it is important to be aware that CORD-19 extends beyond research on COVID-19 and coronaviruses.

Open Access uptake by universities worldwide.

The implementation of policies promoting the adoption of an open science (OS) culture must be accompanied by indicators that allow monitoring the uptake of such policies and their potential effects on research publishing and sharing practices. This study presents indicators of open access (OA) at the institutional level for universities worldwide. By combining data from Web of Science, Unpaywall and the Leiden Ranking disambiguation of institutions, we track OA coverage of universities' output for 963 institutions. This paper presents the methodological challenges, conceptual discrepancies and limitations and discusses further steps needed to move forward the discussion on fostering OA and OS practices and policies.

The Ethics Ecosystem: Personal Ethics, Network Governance and Regulating Actors Governing the Use of Social Media Research Data.

This paper examines the consequences of a culture of "personal ethics" when using new methodologies, such as the use of social media (SM) sites as a source of data for research. Using SM research as an example, this paper explores the practices of a number of actors and researchers within the "Ethics Ecosystem" which as a network governs ethically responsible research behaviour. In the case of SM research, the ethical use of this data is currently in dispute, as even though it is seemingly publically available, concerns relating to privacy, vulnerability, potential harm and consent blur the lines of responsible ethical research behaviour. The findings point to the dominance of a personal, bottom-up, researcher-led, 'ethical barometer' for making decisions regarding the permissibility of using SM data. We show that the use of different barometers by different researchers can lead to wide disparities in ethical practice - disparities which are compounded by the lack of firm guidelines for responsible practice of SM research. This has widespread consequences on the development of shared norms and understandings at all levels, and by all actors within the Ethics Ecosystem, and risks inconsistencies in their approaches to ethical decision-making. This paper argues that this governance of ethical behaviour by individual researchers perpetuates a negative cycle of academic practice that is dependent on subjective judgements by researchers themselves, rather than governed by more formalised academic institutions such as the research ethics committee and funding council guidelines.

Scientific activity by medical students: the relationship between academic publishing during medical school and publication careers after graduation.

INTRODUCTION: Engagement of clinicians in research is important for the integration of science and clinical practice. However, at this moment, there is a shortage of clinician-scientists. Success experiences can stimulate student interest in a research career. Conducting actual research leading to publication is a potential method to gain success experience. This study assessed whether publication as a medical student is associated with publication after graduation. We determined whether medical students in the Netherlands who are involved in research, as measured by publication in international journals before graduation: 1) are more likely to publish, 2) publish a greater number of papers, and 3) have higher citation impact scores after graduation. METHODS: We matched 2005-2008 MD graduates (with rare names, n = 4145 in total) from all eight Dutch university medical centres to their publications indexed in the Web of Science and published between 6 years before and 6 years after graduation. For sensitivity analysis we performed both automatic assignment on the whole group and manual assignment on a 10% random sample. RESULTS: Students who had published before graduation: 1) were 1.9 times as likely to publish, 2) published more papers, and 3) had a slightly higher citation impact after graduation. DISCUSSION: Medical students who conducted research leading to a publication before graduation were more likely to be scientifically active after graduation. While this is not a causal relationship per se, these results cautiously suggest that successful early involvement in research could influence the long-term scientific activity of clinicians.

The Psychology of Morality: A Review and Analysis of Empirical Studies Published From 1940 Through 2017.

We review empirical research on (social) psychology of morality to identify which issues and relations are well documented by existing data and which areas of inquiry are in need of further empirical evidence. An electronic literature search yielded a total of 1,278 relevant research articles published from 1940 through 2017. These were subjected to expert content analysis and standardized bibliometric analysis to classify research questions and relate these to (trends in) empirical approaches that characterize research on morality. We categorize the research questions addressed in this literature into five different themes and consider how empirical approaches within each of these themes have addressed psychological antecedents and implications of moral behavior. We conclude that some key features of theoretical questions relating to human morality are not systematically captured in empirical research and are in need of further investigation.

A researcher's perspective on the quantity of osteosarcoma in vitro studies.

Osteosarcoma is a primary malignant bone tumour, for which no improvement in survival rate has been made since the nineteen seventies. We set out to systemically identify the in vitro studies performed in the past two decades describing potential future therapies. Strikingly, we obtained a total of 5282 PubMed hits on this subject. The amount of publications has increased almost exponentially over the past few years. Studies from Chinese institutes are mainly responsible for this huge increase, accounting for 53% of the publications in 2015. Approximately 1/3 of all drugs described in the past three years could be classified as traditional medicine. Furthermore, it struck our attention that even though in such studies multiple cell lines are essential to represent the heterogeneity in patients, many studies were performed with only one or two cell lines, i.e. U-2 OS or MG-63. These cells are fast growing, facilitating rapid experimental application but also boosting drug responsiveness. This probably explains why so many in vitro studies have been published for this relatively rare disease. Furthermore, it illustrates the current publication pressure, especially in China.

Bibliometric analysis of output and impact based on CRIS data: a case study on the registered output of a Dutch university.

In this study we combine the registered output of a whole university in the Netherlands with data retrieved from the Web of Science. The initial research question was: is it possible to show the impact of the university in its' full broadness, taking into account the variety of disciplines covered in the research profile of the university? In order to answer this question, we analyzed the output of the university as registered in the CRIS system METIS, over the years 2004-2009. The registration covers a wide variety of scholarly outputs, and these are all taken into account in the analysis. In the study we conduct analyses on the coverage of the output of the university, both from the perspective of the output itself, towards the Web of Science ("external"), as well as from the Web of Science perspective itself ("internal"). This provides us with the necessary information to be able to draw clear conclusions on the validity of the usage of standard bibliometric methodologies in the research assessment of universities with such a research profile.

A Longitudinal Analysis of Publications on Maternal Mortality.

BACKGROUND: The fifth Millennium Development Goal formulated by the WHO in 2000 aimed to reduce global maternal mortality by 75% in 2015. We studied the extent to which medical research has supported this by studying maternal mortality. We performed a bibliometric analysis of the literature on maternal mortality and of the development of this literature over time. METHODS: We searched for publications on maternal mortality in the Web of Science database in the period 1994-2013. We visualised the subjects of these publications using a term map showing the most significant terms occurring in the titles and abstracts of publications on maternal mortality. RESULTS: We identified 3794 publications on maternal mortality in Web of Science. The annual number increased from 87 in 1994 to 397 in 2013. The largest number of maternal mortality publications was found in the field of Obstetrics and Gynecology, followed by the Public, Environmental, and Occupational Health field (increase from 1994 until 2013 of 300% and 700%, respectively). In both fields, the number of maternal mortality publications has increased at a much higher rate than the overall number of publications in the field. CONCLUSIONS: In line with the focus of the fifth Millennium Development Goal on reducing maternal mortality, during the past 20 years, there has been a steady increase in the amount of attention paid to maternal mortality in the medical literature. This is largely driven by an increase, mainly in recent years, in public health research on maternal mortality.

UK Doubles Its "World-Leading" Research in Life Sciences and Medicine in Six Years: Testing the Claim?

BACKGROUND: The UK, like some other countries, carries out a periodic review of research quality in universities and the most recent Research Excellence Framework (REF) reported a doubling (103% increase) in its "world leading" or so-called "4*" research outputs in the areas of life sciences and medicine between 2008 and 2014. This is a remarkable improvement in six years and if validated internationally could have profound implications for health sciences. METHODS: We compared the reported changes in 4* quality to bibliometric measures of quality for the 56,639 articles submitted to the RAE 2008 and the 50,044 articles submitted to the REF 2014 to Panel A, which assesses the life sciences, including medicine. FINDINGS: UK research submitted to the RAE and REF was of better quality than worldwide research on average. While we found evidence for some increase in the quality of top UK research articles, a 10-25% increase in the top 10%ile papers, depending upon the metrics used, we could not find evidence to support a 103% increase in quality. Instead we found that as compared to the RAE, the REF results implied a lower citation %ile threshold for declaring a 4*. INTERPRETATION: There is a wide discrepancy between bibliometric indices and peer-review panel judgements between the RAE 2008 and REF 2014. It is possible that the changes in the funding regime between 2008 and 2014 that significantly increased the financial premium for 4* articles may have influenced research quality evaluation. For the advancement of science and health, evaluation of research quality requires consistency and validity - the discrepancy noted here calls for a closer examination of mass peer-review methods like the REF.

[Biomedical research in the Netherlands: high quality due to cooperation between University Medical Centers and non-academic large teaching hospitals]. / Medisch-wetenschappelijk onderzoek in Nederland: hoge kwaliteit door samenwerking UMC's en opleidingsziekenhuizen.

OBJECTIVE: Most of the biomedical research is performed in University Medical Centers (UMC's). Increasingly, however, biomedical research is also done in non-academic large teaching hospitals, united in the Organization for Topclinical Hospitals (STZ) in the Netherlands. The objective of this study was to compare citation scores of biomedical publications from UMC's and STZ hospitals. DESIGN: Bibliometric analysis. METHOD: The Center for Science and Technology Studies of the University of Leiden, the Netherlands, annually analyzes the volume and quality (reflected by normalized citation scores) of the publications of all UMC's in the Netherland. Recently, also for STZ hospitals a similar analysis has been performed. RESULTS: Research publications from UMC's in the Netherland have normalized mean citation scores that are far above the mean world average. The normalized mean citation score of publications from STZ hospitals is lower when research is done independent of a UMC, whereas research that is a combined effort of UMC's and STZ hospitals has a very high mean normalized citation score. CONCLUSION: The Netherlands produces a relatively large volume of biomedical research and publications. Based on citation analysis research done in collaboration between UMC's and STZ hospitals has a very high quality. As most STZ hospitals mostly collaborate with a neighbouring UMC, the formation of research networks that overlap with existing teaching and training networks, could provided the necessary infrastructure for further stimulating this collaborative research.

Discussing some basic critique on Journal Impact Factors: revision of earlier comments.

In this study the issue of the validity of the argument against the applied length of citation windows in Journal Impact Factors calculations is critically re-analyzed. While previous studies argued against the relatively short citation window of 1-2 years, this study shows that the relative short term citation impact measured in the window underlying the Journal Impact Factor is a good predictor of the citation impact of the journals in the next years to come. Possible exceptions to this observation relate to journals with relatively low numbers of publications, and the citation impact related to publications in the year of publication. The study focuses on five Journal Subject Categories from the science and social sciences, on normal articles published in these journals, in the 2 years 2000 and 2004.

The "Mendel syndrome" in science: durability of scientific literature and its effects on bibliometric analysis of individual scientists.

The obsolescence and "durability" of scientific literature have been important elements of debate during many years, especially regarding the proper calculation of bibliometric indicators. The effects of "delayed recognition" on impact indicators have importance and are of interest not only to bibliometricians but also among research managers and scientists themselves. It has been suggested that the "Mendel syndrome" is a potential drawback when assessing individual researchers through impact measures. If publications from particular researchers need more time than "normal" to be properly acknowledged by their colleagues, the impact of these researchers may be underestimated with common citation windows. In this paper, we answer the question whether the bibliometric indicators for scientists can be significantly affected by the Mendel syndrome. Applying a methodology developed previously for the classification of papers according to their durability (Costas et al., J Am Soc Inf Sci Technol 61(8):1564-1581, 2010a; J Am Soc Inf Sci Technol 61(2):329-339, 2010b), the scientific production of 1,064 researchers working at the Spanish Council for Scientific Research (CSIC) in three different research areas has been analyzed. Cases of potential "Mendel syndrome" are rarely found among researchers and these cases do not significantly outperform the impact of researchers with a standard pattern of reception in their citations. The analysis of durability could be included as a parameter for the consideration of the citation windows used in the bibliometric analysis of individuals.

On the correlation between bibliometric indicators and peer review: reply to Opthof and Leydesdorff.

Opthof and Leydesdorff (Scientometrics, 2011) reanalyze data reported by Van Raan (Scientometrics 67(3):491-502, 2006) and conclude that there is no significant correlation between on the one hand average citation scores measured using the CPP/FCSm indicator and on the other hand the quality judgment of peers. We point out that Opthof and Leydesdorff draw their conclusions based on a very limited amount of data. We also criticize the statistical methodology used by Opthof and Leydesdorff. Using a larger amount of data and a more appropriate statistical methodology, we do find a significant correlation between the CPP/FCSm indicator and peer judgment.

Severe language effect in university rankings: particularly Germany and France are wronged in citation-based rankings.

We applied a set of standard bibliometric indicators to monitor the scientific state-of-arte of 500 universities worldwide and constructed a ranking on the basis of these indicators (Leiden Ranking 2010). We find a dramatic and hitherto largely underestimated language effect in the bibliometric, citation-based measurements of research performance when comparing the ranking based on all Web of Science (WoS) covered publications and on only English WoS covered publications, particularly for Germany and France.