How academic sabbaticals are used and how they contribute to research - a small-scale study of the University of Cambridge using interviews and analysis of administrative data.

Background: Academic sabbaticals are seen as an important aspect of academic life and require considerable resources, however, little research has been done into how they are used and whether their effects can be measured. We explored these issues at the University of Cambridge. Methods: A mixed method approach including 24 interviews with academics, eight interviews with administrators; alongside analysis of administrative and publication data between 2010 and 2019. Results: Academics underline the importance of sabbaticals in providing uninterrupted time for research that is used to think, explore new ideas, master new techniques, develop new collaborations, draw together previous work, set work in a wider context, and provide personal discretion in research direction. They also highlight sabbaticals' contributions in allowing the beneficial effects of combining teaching and research, while mitigating some of the disadvantages. However, it is difficult to detect the effect of sabbaticals on publications using a time series approach. Conclusions: Sabbaticals provide manifold contributions to academic research at the University of Cambridge; however, detecting and quantifying this contribution, and extending these findings requires wider and more detailed investigation.

From COVID-19 research to vaccine application: why might it take 17 months not 17 years and what are the wider lessons?

It is often said that it takes 17 years to move medical research from bench to bedside. In a coronavirus disease (COVID-19) world, such time-lags feel intolerable. In these extraordinary circumstances could years be made into months? If so, could those lessons be used to accelerate medical research when the crisis eases?To measure time-lags in health and biomedical research as well as to identify ways of reducing them, we developed and published (in 2015) a matrix consisting of overlapping tracks (or stages/phases) in the translation from discovery research to developed products, policies and practice. The matrix aids analysis by highlighting the time and actions required to develop research (and its translation) both (1) along each track and (2) from one track to another, e.g. from the discovery track to the research-in-humans track. We noted four main approaches to reducing time-lags, namely increasing resources, working in parallel, starting or working at risk, and improving processes.Examining these approaches alongside the matrix helps interpret the enormous global effort to develop a vaccine for the 2019 novel coronavirus SARS-CoV-2, the causative agent of COVID-19. Rapid progress in the discovery/basic and human research tracks is being made through a combination of large-scale funding, work being conducted in parallel (between different teams globally and through working in overlapping tracks), working at greater (but proportionate) risk to safety than usual, and adopting various new processes. The overlapping work of some of the teams involves continuing animal research whilst entering vaccine candidates into Phase I trials alongside planning their Phase II trials. The additional funding available helps to reduce some of the usual financial risks in moving so quickly. Going forward through the increasingly large human trials for safety, dosage and efficacy, it will be vital to overlap work in parallel in the often challenging public policy and clinical tracks. Thus, regulatory and reimbursement bodies are beginning and preparing rapid action to pull vaccines proving to be safe and effective through to extraordinarily rapid application to the general population. Monitoring the development of a COVID-19 vaccine using the matrix (modified as necessary) could help identify which of the approaches speeding development and deployment could be usefully applied more widely in the future.

100 Metrics to Assess and Communicate the Value of Biomedical Research: An Ideas Book.

Biomedical research affects society in many ways. It has been shown to improve health, create jobs, add to our knowledge, and foster new collaborations. Despite the complexity of modern research, many of the metrics used to evaluate the impacts of research still focus on the traditional, often academic, part of the research pathway, covering areas such as the amount of grant funding received and the number of peer-reviewed publications. In response to increasing expectations of accountability and transparency, the Association of American Medical Colleges (AAMC), in collaboration with RAND Europe, undertook a project to help communicate the wider value of biomedical research. The initiative developed resources to support academic medical centers in evaluating the outcomes and impacts of their research using approaches relevant to various stakeholders, including patients, providers, administrators, and legislators. This study presents 100 ideas for metrics that can be used to assess and communicate the value of biomedical research. The list is not comprehensive, and the metrics are not fully developed, but they should serve to stimulate and broaden thinking about how academic medical centers can communicate the value of their research to a broad range of stakeholders.

How do NIHR peer review panels use bibliometric information to support their decisions?

Bibliometrics is widely used as an evaluation tool to assist prospective R&D decision-making. In the UK, for example, the National Institute for Health Research (NIHR) has employed bibliometric analysis alongside wider information in several awarding panels for major funding schemes. In this paper, we examine various aspects of the use of bibliometric information by members of these award selection panels, based on interviews with ten panel members from three NIHR panels, alongside analysis of the information provided to those panels. The aim of the work is to determine what influence bibliometrics has on their decision-making, to see which types of bibliometric measures they find more and less useful, and to identify the challenges they have when using these data. We find that panel members broadly support the use of bibliometrics in panel decision-making, and that the data are primarily used in the initial individual assessment of candidates, playing a smaller role in the selection panel meeting. Panel members felt that the most useful measures of performance are normalised citation scores and the number or proportion of papers in the most highly cited X% (e.g. 5, 10%) for the field. Panel members expressed concerns around the comparability of bibliometrics between fields, but the discussion suggested this largely represents a lack of understanding of bibliometric techniques, confirming that effective background information is important. Based on the evidence around panel behaviour and concerns, we set out guidance around providing bibliometrics to research funding panels.

Assessing and Communicating the Value of Biomedical Research: Results From a Pilot Study.

PURPOSE: Assessing the impact of research requires an approach that is sensitive both to the context of the research and the perspective of the stakeholders trying to understand its benefits. Here, the authors report on a pilot that applied such an approach to research conducted at the Collaborative Center for Health Equity (CCHE) of the University of Wisconsin School of Medicine and Public Health. METHOD: The pilot assessed the academic impact of CCHE's work; the networks between CCHE and community partners; and the reach of CCHE's programs, including an attempt to estimate return on investment (ROI). Data included bibliometrics, findings from a stakeholder survey and in-depth interviews, and financial figures. RESULTS: The pilot illustrated how CCHE programs increase the capacity of community partners to advocate for their communities and engage with researchers to ensure that research benefits the community. The results illustrate the reach of CCHE's programs into the community. The authors produced an estimate of the ROI for one CCHE program targeting childhood obesity, and values ranged from negative to positive. CONCLUSIONS: The authors experienced challenges using novel assessment techniques at a small scale including the lack of comparator groups and the scarcity of cost data for estimating ROI. This pilot demonstrated the value of research from a variety of perspectives-from academic to community. It illustrates how metrics beyond grant income and publications can capture the outputs of an academic health center in a way that may better align with the aims of the center and stakeholders.

What do we know about grant peer review in the health sciences?

Background: Peer review decisions award an estimated >95% of academic medical research funding, so it is crucial to understand how well they work and if they could be improved. Methods: This paper summarises evidence from 105 papers identified through a literature search on the effectiveness and burden of peer review for grant funding. Results: There is a remarkable paucity of evidence about the efficiency of peer review for funding allocation, given its centrality to the modern system of science. From the available evidence, we can identify some conclusions around the effectiveness and burden of peer review. The strongest evidence around effectiveness indicates a bias against innovative research. There is also fairly clear evidence that peer review is, at best, a weak predictor of future research performance, and that ratings vary considerably between reviewers. There is some evidence of age bias and cronyism. Good evidence shows that the burden of peer review is high and that around 75% of it falls on applicants. By contrast, many of the efforts to reduce burden are focused on funders and reviewers/panel members. Conclusions: We suggest funders should acknowledge, assess and analyse the uncertainty around peer review, even using reviewers' uncertainty as an input to funding decisions. Funders could consider a lottery element in some parts of their funding allocation process, to reduce both burden and bias, and allow better evaluation of decision processes. Alternatively, the distribution of scores from different reviewers could be better utilised as a possible way to identify novel, innovative research. Above all, there is a need for open, transparent experimentation and evaluation of different ways to fund research. This also requires more openness across the wider scientific community to support such investigations, acknowledging the lack of evidence about the primacy of the current system and the impossibility of achieving perfection.

Understanding the relative valuation of research impact: a best-worst scaling experiment of the general public and biomedical and health researchers.

OBJECTIVES: (1) To test the use of best-worst scaling (BWS) experiments in valuing different types of biomedical and health research impact, and (2) to explore how different types of research impact are valued by different stakeholder groups. DESIGN: Survey-based BWS experiment and discrete choice modelling. SETTING: The UK. PARTICIPANTS: Current and recent UK Medical Research Council grant holders and a representative sample of the general public recruited from an online panel. RESULTS: In relation to the study's 2 objectives: (1) we demonstrate the application of BWS methodology in the quantitative assessment and valuation of research impact. (2) The general public and researchers provided similar valuations for research impacts such as improved life expectancy, job creation and reduced health costs, but there was less agreement between the groups on other impacts, including commercial capacity development, training and dissemination. CONCLUSIONS: This is the second time that a discrete choice experiment has been used to assess how the general public and researchers value different types of research impact, and the first time that BWS has been used to elicit these choices. While the 2 groups value different research impacts in different ways, we note that where they agree, this is generally about matters that are seemingly more important and associated with wider social benefit, rather than impacts occurring within the research system. These findings are a first step in exploring how the beneficiaries and producers of research value different kinds of impact, an important consideration given the growing emphasis on funding and assessing research on the basis of (potential) impact. Future research should refine and replicate both the current study and that of Miller et al in other countries and disciplines.

A global call for action to include gender in research impact assessment.

Global investment in biomedical research has grown significantly over the last decades, reaching approximately a quarter of a trillion US dollars in 2010. However, not all of this investment is distributed evenly by gender. It follows, arguably, that scarce research resources may not be optimally invested (by either not supporting the best science or by failing to investigate topics that benefit women and men equitably). Women across the world tend to be significantly underrepresented in research both as researchers and research participants, receive less research funding, and appear less frequently than men as authors on research publications. There is also some evidence that women are relatively disadvantaged as the beneficiaries of research, in terms of its health, societal and economic impacts. Historical gender biases may have created a path dependency that means that the research system and the impacts of research are biased towards male researchers and male beneficiaries, making it inherently difficult (though not impossible) to eliminate gender bias. In this commentary, we - a group of scholars and practitioners from Africa, America, Asia and Europe - argue that gender-sensitive research impact assessment could become a force for good in moving science policy and practice towards gender equity. Research impact assessment is the multidisciplinary field of scientific inquiry that examines the research process to maximise scientific, societal and economic returns on investment in research. It encompasses many theoretical and methodological approaches that can be used to investigate gender bias and recommend actions for change to maximise research impact. We offer a set of recommendations to research funders, research institutions and research evaluators who conduct impact assessment on how to include and strengthen analysis of gender equity in research impact assessment and issue a global call for action.

Returns on Research Funded Under the NIHR Health Technology Assessment (HTA) Programme: Economic Analysis and Case Studies.

The Department of Health (England) requested that RAND Europe conduct an economic analysis of the impact of the HTA Programme. This article describes the results of that work, which consisted of analysis of the potential economic benefits of a sample of HTA funded studies and comparison to programme costs, supplemented by a set of short case studies exploring the impacts of the HTA Programme on policy and practice. Based on our analysis, if 12 per cent of the potential net benefit of implementing the findings of this sample of 10 studies for one year was realised, it would cover the cost of the HTA Programme from 1993 to 2012. Drawing on the case studies and the economic analysis, we have also made a number of observations that could help ensure that the HTA Programme maximises the likelihood of findings being adopted.

A 'DECISIVE' Approach to Research Funding: Lessons from Three Retrosight Studies.

The Retrosight approach consists of looking at research that was conducted in the past and, using Payback case studies, tracing that research through to the present day to understand both the extent to which the research has had impacts, within academia and more widely, and how these impacts came about. RAND Europe has conducted three studies based on this approach in different research fields: arthritis research, cardiovascular research and mental health research. Each drew out a set of observations and recommendations for policymakers and research funders in those research fields. By reviewing and comparing the findings of the three studies, we have identified eight lessons which combine to provide a "DECISIVE" approach to biomedical and health research funding: Different skills: Fund researchers with more than just research skills-individuals are key when it comes to translation of research into wider impact. Engaged: Support your researchers to engage with non-academic stakeholders to help their work have a wider impact. Clinical: For greater impact on patient care within 10-20 years, fund clinical rather than basic research. Impact on society: If you want to have a wider impact, don't just fund for academic excellence. Size: Bigger isn't necessarily better when it comes to the size of a research grant. International: For high academic impact, fund researchers who collaborate internationally and support them to do so. Variety: Simple metrics will only capture some of the impact of your research. Expectations: Most broader social and economic impact will come from just a few projects.

Mapping the Global Mental Health Research Funding System.

This study maps the global funding of mental health research between 2009 and 2014. It builds from the bottom up a picture of who the major funders are, what kinds of research they support and how their strategies relate to one another. It uses the funding acknowledgements on journal papers as a starting point for this. The study also looks to the future, considering some of the areas of focus, challenges and opportunities which may shape the field in the coming few years.

The impact of the National Institute for Health Research Health Technology Assessment programme, 2003-13: a multimethod evaluation.

BACKGROUND: The National Institute for Health Research (NIHR) Health Technology Assessment (HTA) programme supports research tailored to the needs of NHS decision-makers, patients and clinicians. This study reviewed the impact of the programme, from 2003 to 2013, on health, clinical practice, health policy, the economy and academia. It also considered how HTA could maintain and increase its impact. METHODS: Interviews (n = 20): senior stakeholders from academia, policy-making organisations and the HTA programme. Bibliometric analysis: citation analysis of publications arising from HTA programme-funded research. Researchfish survey: electronic survey of all HTA grant holders. Payback case studies (n = 12): in-depth case studies of HTA programme-funded research. RESULTS: We make the following observations about the impact, and routes to impact, of the HTA programme: it has had an impact on patients, primarily through changes in guidelines, but also directly (e.g. changing clinical practice); it has had an impact on UK health policy, through providing high-quality scientific evidence - its close relationships with the National Institute for Health and Care Excellence (NICE) and the National Screening Committee (NSC) contributed to the observed impact on health policy, although in some instances other organisations may better facilitate impact; HTA research is used outside the UK by other HTA organisations and systematic reviewers - the programme has an impact on HTA practice internationally as a leader in HTA research methods and the funding of HTA research; the work of the programme is of high academic quality - the Health Technology Assessment journal ensures that the vast majority of HTA programme-funded research is published in full, while the HTA programme still encourages publication in other peer-reviewed journals; academics agree that the programme has played an important role in building and retaining HTA research capacity in the UK; the HTA programme has played a role in increasing the focus on effectiveness and cost-effectiveness in medicine - it has also contributed to increasingly positive attitudes towards HTA research both within the research community and the NHS; and the HTA focuses resources on research that is of value to patients and the UK NHS, which would not otherwise be funded (e.g. where there is no commercial incentive to undertake research). The programme should consider the following to maintain and increase its impact: providing targeted support for dissemination, focusing resources when important results are unlikely to be implemented by other stakeholders, particularly when findings challenge vested interests; maintaining close relationships with NICE and the NSC, but also considering other potential users of HTA research; maintaining flexibility and good relationships with researchers, giving particular consideration to the Technology Assessment Report (TAR) programme and the potential for learning between TAR centres; maintaining the academic quality of the work and the focus on NHS need; considering funding research on the short-term costs of the implementation of new health technologies; improving the monitoring and evaluation of whether or not patient and public involvement influences research; improve the transparency of the priority-setting process; and continuing to monitor the impact and value of the programme to inform its future scientific and administrative development.

UK Doubles Its "World-Leading" Research in Life Sciences and Medicine in Six Years: Testing the Claim?

BACKGROUND: The UK, like some other countries, carries out a periodic review of research quality in universities and the most recent Research Excellence Framework (REF) reported a doubling (103% increase) in its "world leading" or so-called "4*" research outputs in the areas of life sciences and medicine between 2008 and 2014. This is a remarkable improvement in six years and if validated internationally could have profound implications for health sciences. METHODS: We compared the reported changes in 4* quality to bibliometric measures of quality for the 56,639 articles submitted to the RAE 2008 and the 50,044 articles submitted to the REF 2014 to Panel A, which assesses the life sciences, including medicine. FINDINGS: UK research submitted to the RAE and REF was of better quality than worldwide research on average. While we found evidence for some increase in the quality of top UK research articles, a 10-25% increase in the top 10%ile papers, depending upon the metrics used, we could not find evidence to support a 103% increase in quality. Instead we found that as compared to the RAE, the REF results implied a lower citation %ile threshold for declaring a 4*. INTERPRETATION: There is a wide discrepancy between bibliometric indices and peer-review panel judgements between the RAE 2008 and REF 2014. It is possible that the changes in the funding regime between 2008 and 2014 that significantly increased the financial premium for 4* articles may have influenced research quality evaluation. For the advancement of science and health, evaluation of research quality requires consistency and validity - the discrepancy noted here calls for a closer examination of mass peer-review methods like the REF.

How long does biomedical research take? Studying the time taken between biomedical and health research and its translation into products, policy, and practice.

BACKGROUND: The time taken, or 'time lags', between biomedical/health research and its translation into health improvements is receiving growing attention. Reducing time lags should increase rates of return to such research. However, ways to measure time lags are under-developed, with little attention on where time lags arise within overall timelines. The process marker model has been proposed as a better way forward than the current focus on an increasingly complex series of translation 'gaps'. Starting from that model, we aimed to develop better methods to measure and understand time lags and develop ways to identify policy options and produce recommendations for future studies. METHODS: Following reviews of the literature on time lags and of relevant policy documents, we developed a new approach to conduct case studies of time lags. We built on the process marker model, including developing a matrix with a series of overlapping tracks to allow us to present and measure elements within any overall time lag. We identified a reduced number of key markers or calibration points and tested our new approach in seven case studies of research leading to interventions in cardiovascular disease and mental health. Finally, we analysed the data to address our study's key aims. RESULTS: The literature review illustrated the lack of agreement on starting points for measuring time lags. We mapped points from policy documents onto our matrix and thus highlighted key areas of concern, for example around delays before new therapies become widely available. Our seven completed case studies demonstrate we have made considerable progress in developing methods to measure and understand time lags. The matrix of overlapping tracks of activity in the research and implementation processes facilitated analysis of time lags along each track, and at the cross-over points where the next track started. We identified some factors that speed up translation through the actions of companies, researchers, funders, policymakers, and regulators. Recommendations for further work are built on progress made, limitations identified and revised terminology. CONCLUSIONS: Our advances identify complexities, provide a firm basis for further methodological work along and between tracks, and begin to indicate potential ways of reducing lags.

Understanding factors associated with the translation of cardiovascular research: a multinational case study approach.

BACKGROUND: Funders of health research increasingly seek to understand how best to allocate resources in order to achieve maximum value from their funding. We built an international consortium and developed a multinational case study approach to assess benefits arising from health research. We used that to facilitate analysis of factors in the production of research that might be associated with translating research findings into wider impacts, and the complexities involved. METHODS: We built on the Payback Framework and expanded its application through conducting co-ordinated case studies on the payback from cardiovascular and stroke research in Australia, Canada and the United Kingdom. We selected a stratified random sample of projects from leading medical research funders. We devised a series of innovative steps to: minimize the effect of researcher bias; rate the level of impacts identified in the case studies; and interrogate case study narratives to identify factors that correlated with achieving high or low levels of impact. RESULTS: Twenty-nine detailed case studies produced many and diverse impacts. Over the 15 to 20 years examined, basic biomedical research has a greater impact than clinical research in terms of academic impacts such as knowledge production and research capacity building. Clinical research has greater levels of wider impact on health policies, practice, and generating health gains. There was no correlation between knowledge production and wider impacts. We identified various factors associated with high impact. Interaction between researchers and practitioners and the public is associated with achieving high academic impact and translation into wider impacts, as is basic research conducted with a clinical focus. Strategic thinking by clinical researchers, in terms of thinking through pathways by which research could potentially be translated into practice, is associated with high wider impact. Finally, we identified the complexity of factors behind research translation that can arise in a single case. CONCLUSIONS: We can systematically assess research impacts and use the findings to promote translation. Research funders can justify funding research of diverse types, but they should not assume academic impacts are proxies for wider impacts. They should encourage researchers to consider pathways towards impact and engage potential research users in research processes.

Mental Health Retrosight: Understanding the Returns From Research (Lessons From Schizophrenia): Policy Report.

This study examines the impacts arising from neuroscience and mental health research going back 20-25 years, and identifies attributes of the research, researchers or research setting that are associated with translation into patient benefit, in the particular case of schizophrenia. The study combined two methods: forward-tracing case studies to examine where scientific advances of 20 years ago have led to impact today; and backward-tracing perspectives to identify the research antecedents of today's interventions in schizophrenia. These research and impact trails are followed principally in Canada, the UK and the USA. The headline findings are as follows: The case studies and perspectives support the view that mental health research has led to a diverse and beneficial range of academic, health, social and economic impacts over the 20 years since the research was undertaken.Clinical research has had a larger impact on patient care than basic research has over the 20 years since the research was undertaken.Those involved in mental health research who work across boundaries are associated with wider health and social benefits.Committed individuals, motivated by patient need, who effectively champion research agendas and/or translation into practice are key in driving the development and implementation of interventions.This study provides an overview of the methods and presents the full set of findings, with the policy provocations they raise, and an emerging research agenda. It has been written for funders of biomedical and health research and health services, health researchers, and policymakers in those fields. It will also be of interest to those involved in research and impact evaluation.

In Search of the Holy Grail: Understanding Research Success.

This article considers the continuing challenges facing research funders when trying to allocate research money. It focuses on the area of research policy in mental health research funding, with a particular emphasis on funding for schizophrenia research, and provides an overview of research policy in the last 20-25 years. It then goes on to consider what approaches funders could take to build an evidence base to support future decisions about funding.

Strengthening Research Portfolio Evaluation at the Medical Research Council: Developing a Survey for the Collection of Information About Research Outputs.

The Medical Research Council (MRC) wished to better understand the wider impact of MRC research output on society and the economy. The MRC wanted to compare the strengths of different types of funding and areas of research and identify the good news stories and successes it can learn from. As an initial step in this process RAND Europe: (1) examined the range of output and outcome information MRC already collected; and (2) used that analysis to suggest how data collection could be improved. This article outlines the approach taken to the second part of this exercise and focuses on the development of a new survey instrument to support the MRC's data collection approach. Readers should bear in mind that some later stages of survey development and implementation were conducted exclusively by the MRC and are not reported here.

The answer is 17 years, what is the question: understanding time lags in translational research.

This study aimed to review the literature describing and quantifying time lags in the health research translation process. Papers were included in the review if they quantified time lags in the development of health interventions. The study identified 23 papers. Few were comparable as different studies use different measures, of different things, at different time points. We concluded that the current state of knowledge of time lags is of limited use to those responsible for R&D and knowledge transfer who face difficulties in knowing what they should or can do to reduce time lags. This effectively 'blindfolds' investment decisions and risks wasting effort. The study concludes that understanding lags first requires agreeing models, definitions and measures, which can be applied in practice. A second task would be to develop a process by which to gather these data.

Project Retrosight: Understanding the Returns from Cardiovascular and Stroke Research: The Policy Report.

This project explores the impacts arising from cardiovascular and stroke research funded 15-20 years ago and attempts to draw out aspects of the research, researcher or environment that are associated with high or low impact. The project is a case study-based review of 29 cardiovascular and stroke research grants, funded in Australia, Canada and UK between 1989 and 1993. The case studies focused on the individual grants but considered the development of the investigators and ideas involved in the research projects from initiation to the present day. Grants were selected through a stratified random selection approach that aimed to include both high- and low-impact grants. The key messages are as follows: 1) The cases reveal that a large and diverse range of impacts arose from the 29 grants studied. 2) There are variations between the impacts derived from basic biomedical and clinical research. 3) There is no correlation between knowledge production and wider impacts 4) The majority of economic impacts identified come from a minority of projects. 5) We identified factors that appear to be associated with high and low impact. This article presents the key observations of the study and an overview of the methods involved. It has been written for funders of biomedical and health research and health services, health researchers, and policy makers in those fields. It will also be of interest to those involved in research and impact evaluation.