Assessing publication rates from medical students' mandatory research projects in the Netherlands: a follow-up study of 10 cohorts of medical students.

OBJECTIVES: The medical field is facing a clinician-scientist shortage. Medical schools could foster the clinician-scientist workforce by offering students research opportunities. Most medical schools offer elective research programmes. Subsequently, a subset of doctors graduates without any research experience. Mandatory research projects may be more sufficient to develop clinician-scientist, but take more supervision and curricular time. There is limited insight in the scientific outcomes of mandatory research experiences. This study aims to examine publication rates of a mandatory research experience, identify factors associated with publication, and includes postgraduate research engagement. DESIGN AND SETTING: Prospective follow-up study involving 10 cohorts of medical students' mandatory research projects from Leiden University Medical Center. PARTICIPANTS: All medical students who conducted their research project between 2008 and 2018 (n=2329) were included. MAIN OUTCOME MEASURE: Publication rates were defined as peer-reviewed scientific publications, including research papers, reviews, and published meeting abstracts. Postgraduate research engagement was defined as research participation and dissemination of research at scientific conferences or in journals. RESULTS: In total, 644 (27.7%) of all mandatory research experiences resulted in publication, with students mainly as first (n=984, 42.5%) or second author (n=587, 25.3%) and above world average citation impact (mean normalised journal score 1.29, mean normalised citation score 1.23). Students who conducted their research in an academic centre (adjusted OR 2.82; 95% CI 2.10 to 3.77), extended their research (adjusted OR 1.73; 95% CI 1.35 to 2.20), were involved in an excellency track (adjusted OR 2.08; 95% CI 1.44 to 3.01), or conducted clinical (adjusted OR 2.08; 95% CI 1.15 to 3.74) or laboratory (adjusted OR 2.16; 95% CI 1.16 to 4.01) research published their research more often. Later as junior doctors, this group significantly more often disseminate their research results at scientific conferences (adjusted OR 1.89; 95% CI 1.11 to 3.23) or in journals (adjusted OR 1.98; 95% CI 1.14 to 3.43). CONCLUSIONS: Our findings suggest that a significant subset of hands-on mandatory research projects with flexible learning pathways result in tangible research output with proper impact and that such successful experiences can be considered as diving board towards a research-oriented career.

Scientific output of Dutch medical students.

BACKGROUND: In medical curricula, considerable effort is put into scientific education and research training. The output of these efforts, however, is not known. AIM: To assess the number of students who published at least one scientific paper during the course of their medical studies. METHODS: Names and initials of all students who received their medical degree in 2006 or 2007 in one of the six participating university medical centers in the Netherlands were searched in the Web of Science database using a well-validated algorithm. RESULTS: Of the 2973 students, 14.5% had published at least one scientific paper during the last 3 years of their medical studies. These papers were of good quality, as the average number of citations per paper is above the average for papers published in their field. CONCLUSION: Based on the results of our survey, we conclude that medical students in the Netherlands are productively involved in research during their studies. Due to publication delay, the true number of papers is likely to be higher than we counted, but the bibliometric index chosen gives a robust estimate of the effect on the output of the scientific climate of a medical school. Using such an index on a larger scale may stimulate medical faculties to recognize and cultivate academic talent among their students.

Open Access uptake by universities worldwide.

The implementation of policies promoting the adoption of an open science (OS) culture must be accompanied by indicators that allow monitoring the uptake of such policies and their potential effects on research publishing and sharing practices. This study presents indicators of open access (OA) at the institutional level for universities worldwide. By combining data from Web of Science, Unpaywall and the Leiden Ranking disambiguation of institutions, we track OA coverage of universities' output for 963 institutions. This paper presents the methodological challenges, conceptual discrepancies and limitations and discusses further steps needed to move forward the discussion on fostering OA and OS practices and policies.

Scientific activity by medical students: the relationship between academic publishing during medical school and publication careers after graduation.

INTRODUCTION: Engagement of clinicians in research is important for the integration of science and clinical practice. However, at this moment, there is a shortage of clinician-scientists. Success experiences can stimulate student interest in a research career. Conducting actual research leading to publication is a potential method to gain success experience. This study assessed whether publication as a medical student is associated with publication after graduation. We determined whether medical students in the Netherlands who are involved in research, as measured by publication in international journals before graduation: 1) are more likely to publish, 2) publish a greater number of papers, and 3) have higher citation impact scores after graduation. METHODS: We matched 2005-2008 MD graduates (with rare names, n = 4145 in total) from all eight Dutch university medical centres to their publications indexed in the Web of Science and published between 6 years before and 6 years after graduation. For sensitivity analysis we performed both automatic assignment on the whole group and manual assignment on a 10% random sample. RESULTS: Students who had published before graduation: 1) were 1.9 times as likely to publish, 2) published more papers, and 3) had a slightly higher citation impact after graduation. DISCUSSION: Medical students who conducted research leading to a publication before graduation were more likely to be scientifically active after graduation. While this is not a causal relationship per se, these results cautiously suggest that successful early involvement in research could influence the long-term scientific activity of clinicians.

Is the randomized controlled drug trial in Europe lagging behind the USA?

AIMS: Performance of randomized controlled drug trials (drugRCTs) adds to the scientific output, scientific knowledge, scientific training and up-to-date status of healthcare and may drive economy. The purpose of this study was to benchmark Europe's position on drugRCTs relative to the rest of the world, and to identify factors that may drive this performance. METHODS: The number of scientific publications on drugRCTs, indexed in PubMed and Thomson Scientific/Web of Science database over the period 1995-2004, was used as a proxy measure for the quantitative drugRCT output. The international citation impact of these publications was used as a proxy measure for the qualitative drugRCT output. RESULTS: Country's origin of 103 211 publications was determined. After adjustment for population size, the number of drugRCT publications from Europe, USA and Australia/Japan was 102, 124 and 44 publications per million inhabitants, respectively. The proportional increase in publication output from 1995 until 2004 was lower in Europe compared with the USA and Australia/Japan (29.1, 40.1 and 63.4%, respectively). The number of citations per publication was 4.9 in Europe, 7.0 in the USA and 3.4 in Australia/Japan. Within Europe, the UK, Germany and Italy produced most publications. Country-specific factors associated with publication output in Europe were the number of pharmaceutical companies with headquarters in a country (R(2) = 0.71, P < 0.001), national R&D expenditures by pharmaceutical companies (R(2) = 0.63, P < 0.001) and health-related R&D expenditures by national governments (R(2) = 0.22, P = 0.052). CONCLUSIONS: When adjusted for population size, quantitative and qualitative performance of drugRCTs in Europe lags behind the USA but is ahead of Australia/Japan. Several factors appear to explain the differences, among which are the number of headquarters of pharmaceutical companies in a country, the research expenditures by pharmaceutical companies, as well as health-related R&D expenditures of a country. To enhance and strengthen Europe's position, researchers may strengthen their collaborations with local pharmaceutical companies, and national governments could increase their budgets for medical research funding.

A researcher's perspective on the quantity of osteosarcoma in vitro studies.

Osteosarcoma is a primary malignant bone tumour, for which no improvement in survival rate has been made since the nineteen seventies. We set out to systemically identify the in vitro studies performed in the past two decades describing potential future therapies. Strikingly, we obtained a total of 5282 PubMed hits on this subject. The amount of publications has increased almost exponentially over the past few years. Studies from Chinese institutes are mainly responsible for this huge increase, accounting for 53% of the publications in 2015. Approximately 1/3 of all drugs described in the past three years could be classified as traditional medicine. Furthermore, it struck our attention that even though in such studies multiple cell lines are essential to represent the heterogeneity in patients, many studies were performed with only one or two cell lines, i.e. U-2 OS or MG-63. These cells are fast growing, facilitating rapid experimental application but also boosting drug responsiveness. This probably explains why so many in vitro studies have been published for this relatively rare disease. Furthermore, it illustrates the current publication pressure, especially in China.

Bibliometric analysis of output and impact based on CRIS data: a case study on the registered output of a Dutch university.

In this study we combine the registered output of a whole university in the Netherlands with data retrieved from the Web of Science. The initial research question was: is it possible to show the impact of the university in its' full broadness, taking into account the variety of disciplines covered in the research profile of the university? In order to answer this question, we analyzed the output of the university as registered in the CRIS system METIS, over the years 2004-2009. The registration covers a wide variety of scholarly outputs, and these are all taken into account in the analysis. In the study we conduct analyses on the coverage of the output of the university, both from the perspective of the output itself, towards the Web of Science ("external"), as well as from the Web of Science perspective itself ("internal"). This provides us with the necessary information to be able to draw clear conclusions on the validity of the usage of standard bibliometric methodologies in the research assessment of universities with such a research profile.

UK Doubles Its "World-Leading" Research in Life Sciences and Medicine in Six Years: Testing the Claim?

BACKGROUND: The UK, like some other countries, carries out a periodic review of research quality in universities and the most recent Research Excellence Framework (REF) reported a doubling (103% increase) in its "world leading" or so-called "4*" research outputs in the areas of life sciences and medicine between 2008 and 2014. This is a remarkable improvement in six years and if validated internationally could have profound implications for health sciences. METHODS: We compared the reported changes in 4* quality to bibliometric measures of quality for the 56,639 articles submitted to the RAE 2008 and the 50,044 articles submitted to the REF 2014 to Panel A, which assesses the life sciences, including medicine. FINDINGS: UK research submitted to the RAE and REF was of better quality than worldwide research on average. While we found evidence for some increase in the quality of top UK research articles, a 10-25% increase in the top 10%ile papers, depending upon the metrics used, we could not find evidence to support a 103% increase in quality. Instead we found that as compared to the RAE, the REF results implied a lower citation %ile threshold for declaring a 4*. INTERPRETATION: There is a wide discrepancy between bibliometric indices and peer-review panel judgements between the RAE 2008 and REF 2014. It is possible that the changes in the funding regime between 2008 and 2014 that significantly increased the financial premium for 4* articles may have influenced research quality evaluation. For the advancement of science and health, evaluation of research quality requires consistency and validity - the discrepancy noted here calls for a closer examination of mass peer-review methods like the REF.

Severe language effect in university rankings: particularly Germany and France are wronged in citation-based rankings.

We applied a set of standard bibliometric indicators to monitor the scientific state-of-arte of 500 universities worldwide and constructed a ranking on the basis of these indicators (Leiden Ranking 2010). We find a dramatic and hitherto largely underestimated language effect in the bibliometric, citation-based measurements of research performance when comparing the ranking based on all Web of Science (WoS) covered publications and on only English WoS covered publications, particularly for Germany and France.

The "Mendel syndrome" in science: durability of scientific literature and its effects on bibliometric analysis of individual scientists.

The obsolescence and "durability" of scientific literature have been important elements of debate during many years, especially regarding the proper calculation of bibliometric indicators. The effects of "delayed recognition" on impact indicators have importance and are of interest not only to bibliometricians but also among research managers and scientists themselves. It has been suggested that the "Mendel syndrome" is a potential drawback when assessing individual researchers through impact measures. If publications from particular researchers need more time than "normal" to be properly acknowledged by their colleagues, the impact of these researchers may be underestimated with common citation windows. In this paper, we answer the question whether the bibliometric indicators for scientists can be significantly affected by the Mendel syndrome. Applying a methodology developed previously for the classification of papers according to their durability (Costas et al., J Am Soc Inf Sci Technol 61(8):1564-1581, 2010a; J Am Soc Inf Sci Technol 61(2):329-339, 2010b), the scientific production of 1,064 researchers working at the Spanish Council for Scientific Research (CSIC) in three different research areas has been analyzed. Cases of potential "Mendel syndrome" are rarely found among researchers and these cases do not significantly outperform the impact of researchers with a standard pattern of reception in their citations. The analysis of durability could be included as a parameter for the consideration of the citation windows used in the bibliometric analysis of individuals.

Towards a new crown indicator: an empirical analysis.

We present an empirical comparison between two normalization mechanisms for citation-based indicators of research performance. These mechanisms aim to normalize citation counts for the field and the year in which a publication was published. One mechanism is applied in the current so-called crown indicator of our institute. The other mechanism is applied in the new crown indicator that our institute is currently exploring. We find that at high aggregation levels, such as at the level of large research institutions or at the level of countries, the differences between the two mechanisms are very small. At lower aggregation levels, such as at the level of research groups or at the level of journals, the differences between the two mechanisms are somewhat larger. We pay special attention to the way in which recent publications are handled. These publications typically have very low citation counts and should therefore be handled with special care.

On the correlation between bibliometric indicators and peer review: reply to Opthof and Leydesdorff.

Opthof and Leydesdorff (Scientometrics, 2011) reanalyze data reported by Van Raan (Scientometrics 67(3):491-502, 2006) and conclude that there is no significant correlation between on the one hand average citation scores measured using the CPP/FCSm indicator and on the other hand the quality judgment of peers. We point out that Opthof and Leydesdorff draw their conclusions based on a very limited amount of data. We also criticize the statistical methodology used by Opthof and Leydesdorff. Using a larger amount of data and a more appropriate statistical methodology, we do find a significant correlation between the CPP/FCSm indicator and peer judgment.

Self-citations at the meso and individual levels: effects of different calculation methods.

This paper focuses on the study of self-citations at the meso and micro (individual) levels, on the basis of an analysis of the production (1994-2004) of individual researchers working at the Spanish CSIC in the areas of Biology and Biomedicine and Material Sciences. Two different types of self-citations are described: author self-citations (citations received from the author him/herself) and co-author self-citations (citations received from the researchers' co-authors but without his/her participation). Self-citations do not play a decisive role in the high citation scores of documents either at the individual or at the meso level, which are mainly due to external citations. At micro-level, the percentage of self-citations does not change by professional rank or age, but differences in the relative weight of author and co-author self-citations have been found. The percentage of co-author self-citations tends to decrease with age and professional rank while the percentage of author self-citations shows the opposite trend. Suppressing author self-citations from citation counts to prevent overblown self-citation practices may result in a higher reduction of citation numbers of old scientists and, particularly, of those in the highest categories. Author and co-author self-citations provide valuable information on the scientific communication process, but external citations are the most relevant for evaluative purposes. As a final recommendation, studies considering self-citations at the individual level should make clear whether author or total self-citations are used as these can affect researchers differently.

Bibliometric analysis of psychotherapy research: performance assessment and position in the journal landscape.

The authors provide an overview of advanced bibliometric methods for (a) an objective and transparent assessment of journal performance and (b) positioning of a journal in relation to other journals. These methods are applied to Psychotherapy Research, an international journal within the field of clinical psychology. In the first analysis, the authors focus on journal performance in an international comparative perspective (i.e., the performance of the journal in relation to all other journals in the same field of science) and introduce a novel type of journal impact factor. In the second analysis, the authors position the journal on the basis of total citation relations among all relevant journals, including those outside the specific field of science to which the journal belongs. A multitude of interdisciplinary relations between the journal under investigation and many other journals is revealed. The investigators discuss briefly the potential of such a "journal citation mapping" for unraveling interdisciplinary developments and "interfaces" between different fields of science.