ISRIA statement: ten-point guidelines for an effective process of research impact assessment.

As governments, funding agencies and research organisations worldwide seek to maximise both the financial and non-financial returns on investment in research, the way the research process is organised and funded is becoming increasingly under scrutiny. There are growing demands and aspirations to measure research impact (beyond academic publications), to understand how science works, and to optimise its societal and economic impact. In response, a multidisciplinary practice called research impact assessment is rapidly developing. Given that the practice is still in its formative stage, systematised recommendations or accepted standards for practitioners (such as funders and those responsible for managing research projects) across countries or disciplines to guide research impact assessment are not yet available.In this statement, we propose initial guidelines for a rigorous and effective process of research impact assessment applicable to all research disciplines and oriented towards practice. This statement systematises expert knowledge and practitioner experience from designing and delivering the International School on Research Impact Assessment (ISRIA). It brings together insights from over 450 experts and practitioners from 34 countries, who participated in the school during its 5-year run (from 2013 to 2017) and shares a set of core values from the school's learning programme. These insights are distilled into ten-point guidelines, which relate to (1) context, (2) purpose, (3) stakeholders' needs, (4) stakeholder engagement, (5) conceptual frameworks, (6) methods and data sources, (7) indicators and metrics, (8) ethics and conflicts of interest, (9) communication, and (10) community of practice.The guidelines can help practitioners improve and standardise the process of research impact assessment, but they are by no means exhaustive and require evaluation and continuous improvement. The prima facie effectiveness of the guidelines is based on the systematised expert and practitioner knowledge of the school's faculty and participants derived from their practical experience and research evidence. The current knowledge base has gaps in terms of the geographical and scientific discipline as well as stakeholder coverage and representation. The guidelines can be further strengthened through evaluation and continuous improvement by the global research impact assessment community.

Estimating the returns to United Kingdom publicly funded musculoskeletal disease research in terms of net value of improved health outcomes.

BACKGROUND: Building on an approach applied to cardiovascular and cancer research, we estimated the economic returns from United Kingdom public- and charitable-funded musculoskeletal disease (MSD) research that arise from the net value of the improved health outcomes in the United Kingdom. METHODS: To calculate the economic returns from MSD-related research in the United Kingdom, we estimated (1) the public and charitable expenditure on MSD-related research in the United Kingdom between 1970 and 2013; (2) the net monetary benefit (NMB), derived from the health benefit in quality adjusted life years (QALYs) valued in monetary terms (using a base-case value of a QALY of £25,000) minus the cost of delivering that benefit, for a prioritised list of interventions from 1994 to 2013; (3) the proportion of NMB attributable to United Kingdom research; and (4) the elapsed time between research funding and health gain. The data collected from these four key elements were used to estimate the internal rate of return (IRR) from MSD-related research investments on health benefits. We analysed the uncertainties in the IRR estimate using a one-way sensitivity analysis. RESULTS: Expressed in 2013 prices, total expenditure on MSD-related research from 1970 to 2013 was £3.5 billion, and for the period used to estimate the rate of return, 1978-1997, was £1.4 billion. Over the period 1994-2013 the key interventions analysed produced 871,000 QALYs with a NMB of £16 billion, allowing for the net NHS costs resulting from them and valuing a QALY at £25,000. The proportion of benefit attributable to United Kingdom research was 30% and the elapsed time between funding and impact of MSD treatments was 16 years. Our best estimate of the IRR from MSD-related research was 7%, which is similar to the 9% for CVD and 10% for cancer research. CONCLUSIONS: Our estimate of the IRR from the net health gain to public and charitable funding of MSD-related research in the United Kingdom is substantial, and justifies the research investments made between 1978 and 1997. We also demonstrated the applicability of the approach previously used in assessing the returns from cardiovascular and cancer research. Inevitably, with a study of this kind, there are a number of important assumptions and caveats that we highlight, and these can inform future research.

Review of Computerised Cognitive Behavioural Therapies: Products and Outcomes for People with Mental Health Needs.

Common mental health problems, such as depression, anxiety, and other disorders, affect many people and incur increasing costs to individuals, employers, and government. Several challenges have been identified in the provision of public services for people with common mental health problems. A previous study by RAND Europe suggested that providing access to online mental health assessment and support and building on computerised cognitive behavioural therapy (cCBT) interventions could help reach out to the general population and, in particular, those less likely to seek help elsewhere. This study explores online platforms and mobile applications that offer cognitive behavioural therapy (CBT) for people with mental health needs. Overall, this review shows that a variety of cCBT products exist which differ both in terms of the conditions products aim to address and the ways in which platforms are designed. The tools offer support through a varying number of modules (or lessons) clustered around specific issues that need to be addressed. The number of modules offered by the tools differs slightly by condition: platforms for anxiety disorders and insomnia are on average lengthier, with a median number of modules of 9 and 8 respectively, compared to those for depression in which the median is 6.5. The majority of tools use a linear structure and offer at least some additional guidance, although the intensity of this extra support is typically low.

Does a biomedical research centre affect patient care in local hospitals?

BACKGROUND: Biomedical research can have impacts on patient care at research-active hospitals. We qualitatively evaluated the impact of the Oxford Biomedical Research Centre (Oxford BRC), a university-hospital partnership, on the effectiveness and efficiency of healthcare in local hospitals. Effectiveness and efficiency are conceptualised in terms of impacts perceived by clinicians on the quality, quantity and costs of patient care they deliver. METHODS: First, we reviewed documentation from Oxford BRC and literature on the impact of research activity on patient care. Second, we interviewed leaders of the Oxford BRC's research to identify the direct and indirect impacts they expected their activity would have on local hospitals. Third, this information was used to inform interviews with senior clinicians responsible for patient care at Oxford's acute hospitals to discover what impacts they observed from research generally and from Oxford BRC's research work specifically. We compared and contrasted the results from the two sets of interviews using a qualitative approach. Finally, we identified themes emerging from the senior clinicians' responses, and compared them with an existing taxonomy of mechanisms through which quality of healthcare may be affected in research-active settings. RESULTS: We were able to interview 17 research leaders at the Oxford BRC and 19 senior clinicians at Oxford's acute hospitals. The research leaders identified a wide range of beneficial impacts that they expected might be felt at local hospitals as a result of their research activity. They expected the impact of their research activity on patient care to be generally positive. The senior clinicians responsible for patient care at those hospitals presented a more mixed picture, identifying many positive impacts, but also a smaller number of negative impacts, from research activity, including that of the Oxford BRC. We found the existing taxonomy of benefit types to be helpful in organising the findings, and propose modifications to further improve its usefulness. CONCLUSIONS: Impacts from research activity on the effectiveness and efficiency of patient care at the local acute hospitals, as perceived by senior clinicians, were more often beneficial than harmful. The Oxford BRC contributed to those impacts.

Understanding the relative valuation of research impact: a best-worst scaling experiment of the general public and biomedical and health researchers.

OBJECTIVES: (1) To test the use of best-worst scaling (BWS) experiments in valuing different types of biomedical and health research impact, and (2) to explore how different types of research impact are valued by different stakeholder groups. DESIGN: Survey-based BWS experiment and discrete choice modelling. SETTING: The UK. PARTICIPANTS: Current and recent UK Medical Research Council grant holders and a representative sample of the general public recruited from an online panel. RESULTS: In relation to the study's 2 objectives: (1) we demonstrate the application of BWS methodology in the quantitative assessment and valuation of research impact. (2) The general public and researchers provided similar valuations for research impacts such as improved life expectancy, job creation and reduced health costs, but there was less agreement between the groups on other impacts, including commercial capacity development, training and dissemination. CONCLUSIONS: This is the second time that a discrete choice experiment has been used to assess how the general public and researchers value different types of research impact, and the first time that BWS has been used to elicit these choices. While the 2 groups value different research impacts in different ways, we note that where they agree, this is generally about matters that are seemingly more important and associated with wider social benefit, rather than impacts occurring within the research system. These findings are a first step in exploring how the beneficiaries and producers of research value different kinds of impact, an important consideration given the growing emphasis on funding and assessing research on the basis of (potential) impact. Future research should refine and replicate both the current study and that of Miller et al in other countries and disciplines.

A global call for action to include gender in research impact assessment.

Global investment in biomedical research has grown significantly over the last decades, reaching approximately a quarter of a trillion US dollars in 2010. However, not all of this investment is distributed evenly by gender. It follows, arguably, that scarce research resources may not be optimally invested (by either not supporting the best science or by failing to investigate topics that benefit women and men equitably). Women across the world tend to be significantly underrepresented in research both as researchers and research participants, receive less research funding, and appear less frequently than men as authors on research publications. There is also some evidence that women are relatively disadvantaged as the beneficiaries of research, in terms of its health, societal and economic impacts. Historical gender biases may have created a path dependency that means that the research system and the impacts of research are biased towards male researchers and male beneficiaries, making it inherently difficult (though not impossible) to eliminate gender bias. In this commentary, we - a group of scholars and practitioners from Africa, America, Asia and Europe - argue that gender-sensitive research impact assessment could become a force for good in moving science policy and practice towards gender equity. Research impact assessment is the multidisciplinary field of scientific inquiry that examines the research process to maximise scientific, societal and economic returns on investment in research. It encompasses many theoretical and methodological approaches that can be used to investigate gender bias and recommend actions for change to maximise research impact. We offer a set of recommendations to research funders, research institutions and research evaluators who conduct impact assessment on how to include and strengthen analysis of gender equity in research impact assessment and issue a global call for action.

Treatment for Dementia: Learning from Breakthroughs for Other Conditions.

The past few decades have seen a number of medical breakthroughs that enabled the effective treatment of a range of conditions, transforming them from fatal into manageable ones. Examples include certain cancers and HIV. Conversely, progress on dementia has been limited. There are currently no treatments that will cure or even alter the progressive course of dementia, despite ongoing research investigating new therapies and care options. The UK Department of Health is interested in the potential to learn from other disease areas to better understand the particular social, economic, political, legislative and scientific contexts that have contributed to accelerating progress and breakthroughs in treatment. Such learning could helpfully inform dementia research and innovation efforts, and help identify levers for supportive policy development. This project analysed breakthroughs in the treatment of four selected conditions of ill health and seeks to identify potentially transferable lessons for the dementia context. Using evidence review and key informant interviews we sought to identify the series of "events" that eventually led to a given breakthrough, and the key milestones in the process that have helped improve understanding and potential for treatment. We also aimed to capture the temporal and causal relationships between "notable" events looking at a variety of factors implicated in the breakthrough pathway. The focus of this work was on political, economic, social, scientific and technological, and legal, regulatory and environmental factors.

A 'DECISIVE' Approach to Research Funding: Lessons from Three Retrosight Studies.

The Retrosight approach consists of looking at research that was conducted in the past and, using Payback case studies, tracing that research through to the present day to understand both the extent to which the research has had impacts, within academia and more widely, and how these impacts came about. RAND Europe has conducted three studies based on this approach in different research fields: arthritis research, cardiovascular research and mental health research. Each drew out a set of observations and recommendations for policymakers and research funders in those research fields. By reviewing and comparing the findings of the three studies, we have identified eight lessons which combine to provide a "DECISIVE" approach to biomedical and health research funding: Different skills: Fund researchers with more than just research skills-individuals are key when it comes to translation of research into wider impact. Engaged: Support your researchers to engage with non-academic stakeholders to help their work have a wider impact. Clinical: For greater impact on patient care within 10-20 years, fund clinical rather than basic research. Impact on society: If you want to have a wider impact, don't just fund for academic excellence. Size: Bigger isn't necessarily better when it comes to the size of a research grant. International: For high academic impact, fund researchers who collaborate internationally and support them to do so. Variety: Simple metrics will only capture some of the impact of your research. Expectations: Most broader social and economic impact will come from just a few projects.

Mapping the Global Mental Health Research Funding System.

This study maps the global funding of mental health research between 2009 and 2014. It builds from the bottom up a picture of who the major funders are, what kinds of research they support and how their strategies relate to one another. It uses the funding acknowledgements on journal papers as a starting point for this. The study also looks to the future, considering some of the areas of focus, challenges and opportunities which may shape the field in the coming few years.

The impact of the National Institute for Health Research Health Technology Assessment programme, 2003-13: a multimethod evaluation.

BACKGROUND: The National Institute for Health Research (NIHR) Health Technology Assessment (HTA) programme supports research tailored to the needs of NHS decision-makers, patients and clinicians. This study reviewed the impact of the programme, from 2003 to 2013, on health, clinical practice, health policy, the economy and academia. It also considered how HTA could maintain and increase its impact. METHODS: Interviews (n = 20): senior stakeholders from academia, policy-making organisations and the HTA programme. Bibliometric analysis: citation analysis of publications arising from HTA programme-funded research. Researchfish survey: electronic survey of all HTA grant holders. Payback case studies (n = 12): in-depth case studies of HTA programme-funded research. RESULTS: We make the following observations about the impact, and routes to impact, of the HTA programme: it has had an impact on patients, primarily through changes in guidelines, but also directly (e.g. changing clinical practice); it has had an impact on UK health policy, through providing high-quality scientific evidence - its close relationships with the National Institute for Health and Care Excellence (NICE) and the National Screening Committee (NSC) contributed to the observed impact on health policy, although in some instances other organisations may better facilitate impact; HTA research is used outside the UK by other HTA organisations and systematic reviewers - the programme has an impact on HTA practice internationally as a leader in HTA research methods and the funding of HTA research; the work of the programme is of high academic quality - the Health Technology Assessment journal ensures that the vast majority of HTA programme-funded research is published in full, while the HTA programme still encourages publication in other peer-reviewed journals; academics agree that the programme has played an important role in building and retaining HTA research capacity in the UK; the HTA programme has played a role in increasing the focus on effectiveness and cost-effectiveness in medicine - it has also contributed to increasingly positive attitudes towards HTA research both within the research community and the NHS; and the HTA focuses resources on research that is of value to patients and the UK NHS, which would not otherwise be funded (e.g. where there is no commercial incentive to undertake research). The programme should consider the following to maintain and increase its impact: providing targeted support for dissemination, focusing resources when important results are unlikely to be implemented by other stakeholders, particularly when findings challenge vested interests; maintaining close relationships with NICE and the NSC, but also considering other potential users of HTA research; maintaining flexibility and good relationships with researchers, giving particular consideration to the Technology Assessment Report (TAR) programme and the potential for learning between TAR centres; maintaining the academic quality of the work and the focus on NHS need; considering funding research on the short-term costs of the implementation of new health technologies; improving the monitoring and evaluation of whether or not patient and public involvement influences research; improve the transparency of the priority-setting process; and continuing to monitor the impact and value of the programme to inform its future scientific and administrative development.

How long does biomedical research take? Studying the time taken between biomedical and health research and its translation into products, policy, and practice.

BACKGROUND: The time taken, or 'time lags', between biomedical/health research and its translation into health improvements is receiving growing attention. Reducing time lags should increase rates of return to such research. However, ways to measure time lags are under-developed, with little attention on where time lags arise within overall timelines. The process marker model has been proposed as a better way forward than the current focus on an increasingly complex series of translation 'gaps'. Starting from that model, we aimed to develop better methods to measure and understand time lags and develop ways to identify policy options and produce recommendations for future studies. METHODS: Following reviews of the literature on time lags and of relevant policy documents, we developed a new approach to conduct case studies of time lags. We built on the process marker model, including developing a matrix with a series of overlapping tracks to allow us to present and measure elements within any overall time lag. We identified a reduced number of key markers or calibration points and tested our new approach in seven case studies of research leading to interventions in cardiovascular disease and mental health. Finally, we analysed the data to address our study's key aims. RESULTS: The literature review illustrated the lack of agreement on starting points for measuring time lags. We mapped points from policy documents onto our matrix and thus highlighted key areas of concern, for example around delays before new therapies become widely available. Our seven completed case studies demonstrate we have made considerable progress in developing methods to measure and understand time lags. The matrix of overlapping tracks of activity in the research and implementation processes facilitated analysis of time lags along each track, and at the cross-over points where the next track started. We identified some factors that speed up translation through the actions of companies, researchers, funders, policymakers, and regulators. Recommendations for further work are built on progress made, limitations identified and revised terminology. CONCLUSIONS: Our advances identify complexities, provide a firm basis for further methodological work along and between tracks, and begin to indicate potential ways of reducing lags.

Estimating the returns to UK publicly funded cancer-related research in terms of the net value of improved health outcomes.

BACKGROUND: Building on an approach developed to assess the economic returns to cardiovascular research, we estimated the economic returns from UK public and charitable funded cancer-related research that arise from the net value of the improved health outcomes. METHODS: To assess these economic returns from cancer-related research in the UK we estimated: 1) public and charitable expenditure on cancer-related research in the UK from 1970 to 2009; 2) net monetary benefit (NMB), that is, the health benefit measured in quality adjusted life years (QALYs) valued in monetary terms (using a base-case value of a QALY of GB£25,000) minus the cost of delivering that benefit, for a prioritised list of interventions from 1991 to 2010; 3) the proportion of NMB attributable to UK research; 4) the elapsed time between research funding and health gain; and 5) the internal rate of return (IRR) from cancer-related research investments on health benefits. We analysed the uncertainties in the IRR estimate using sensitivity analyses to illustrate the effect of some key parameters. RESULTS: In 2011/12 prices, total expenditure on cancer-related research from 1970 to 2009 was £15 billion. The NMB of the 5.9 million QALYs gained from the prioritised interventions from 1991 to 2010 was £124 billion. Calculation of the IRR incorporated an estimated elapsed time of 15 years. We related 17% of the annual NMB estimated to be attributable to UK research (for each of the 20 years 1991 to 2010) to 20 years of research investment 15 years earlier (that is, for 1976 to 1995). This produced a best-estimate IRR of 10%, compared with 9% previously estimated for cardiovascular disease research. The sensitivity analysis demonstrated the importance of smoking reduction as a major source of improved cancer-related health outcomes. CONCLUSIONS: We have demonstrated a substantive IRR from net health gain to public and charitable funding of cancer-related research in the UK, and further validated the approach that we originally used in assessing the returns from cardiovascular research. In doing so, we have highlighted a number of weaknesses and key assumptions that need strengthening in further investigations. Nevertheless, these cautious estimates demonstrate that the returns from past cancer research have been substantial, and justify the investments made during the period 1976 to 1995.

Understanding factors associated with the translation of cardiovascular research: a multinational case study approach.

BACKGROUND: Funders of health research increasingly seek to understand how best to allocate resources in order to achieve maximum value from their funding. We built an international consortium and developed a multinational case study approach to assess benefits arising from health research. We used that to facilitate analysis of factors in the production of research that might be associated with translating research findings into wider impacts, and the complexities involved. METHODS: We built on the Payback Framework and expanded its application through conducting co-ordinated case studies on the payback from cardiovascular and stroke research in Australia, Canada and the United Kingdom. We selected a stratified random sample of projects from leading medical research funders. We devised a series of innovative steps to: minimize the effect of researcher bias; rate the level of impacts identified in the case studies; and interrogate case study narratives to identify factors that correlated with achieving high or low levels of impact. RESULTS: Twenty-nine detailed case studies produced many and diverse impacts. Over the 15 to 20 years examined, basic biomedical research has a greater impact than clinical research in terms of academic impacts such as knowledge production and research capacity building. Clinical research has greater levels of wider impact on health policies, practice, and generating health gains. There was no correlation between knowledge production and wider impacts. We identified various factors associated with high impact. Interaction between researchers and practitioners and the public is associated with achieving high academic impact and translation into wider impacts, as is basic research conducted with a clinical focus. Strategic thinking by clinical researchers, in terms of thinking through pathways by which research could potentially be translated into practice, is associated with high wider impact. Finally, we identified the complexity of factors behind research translation that can arise in a single case. CONCLUSIONS: We can systematically assess research impacts and use the findings to promote translation. Research funders can justify funding research of diverse types, but they should not assume academic impacts are proxies for wider impacts. They should encourage researchers to consider pathways towards impact and engage potential research users in research processes.

Mental Health Retrosight: Understanding the Returns From Research (Lessons From Schizophrenia): Policy Report.

This study examines the impacts arising from neuroscience and mental health research going back 20-25 years, and identifies attributes of the research, researchers or research setting that are associated with translation into patient benefit, in the particular case of schizophrenia. The study combined two methods: forward-tracing case studies to examine where scientific advances of 20 years ago have led to impact today; and backward-tracing perspectives to identify the research antecedents of today's interventions in schizophrenia. These research and impact trails are followed principally in Canada, the UK and the USA. The headline findings are as follows: The case studies and perspectives support the view that mental health research has led to a diverse and beneficial range of academic, health, social and economic impacts over the 20 years since the research was undertaken.Clinical research has had a larger impact on patient care than basic research has over the 20 years since the research was undertaken.Those involved in mental health research who work across boundaries are associated with wider health and social benefits.Committed individuals, motivated by patient need, who effectively champion research agendas and/or translation into practice are key in driving the development and implementation of interventions.This study provides an overview of the methods and presents the full set of findings, with the policy provocations they raise, and an emerging research agenda. It has been written for funders of biomedical and health research and health services, health researchers, and policymakers in those fields. It will also be of interest to those involved in research and impact evaluation.

Investigating Time Lags and Attribution in the Translation of Cancer Research: A Case Study Approach.

In 2012, RAND Europe and the Health Economics Research Group (Brunel University) were commissioned by the Wellcome Trust, Cancer Research UK, the National Institute for Health Research and the Academy of Medical Science to conduct a study of the returns to the public/charitable investment in cancer-related research. This study built on previous work published in the 2008 "What's it worth?" report that estimated the economic returns to medical research in terms of spillover benefits and health gain. The 2008 study was extensively quoted and cited as a clear justification for the economic importance of medical research and appears to have played a role in achieving the protection of the medical science budget in the recent public expenditure cuts. This cancer study used a similar approach to that used in the previous study, but with some methodological developments. One of the methodological developments was the inclusion of case studies to examine the validity and variability of the estimates on elapsed time between funding and health gains, and the amount of health gains that can be attributed to UK research. This study provides the full text of the five case studies conducted as well as some discussion of observations emerging across the case study set.

Supporting the Development of a New Health R&D Strategy: A Rapid Review of International Theory and Practice for Norway's HelseOmsorg21.

The HelseOmsorg21 initiative was set up by the Ministry of Health and Care Services to develop a new research and innovation strategy for health and care services in Norway. The HelseOmsorg21 Strategy Group, through the Research Council of Norway which is providing the secretariat for the strategy development, asked RAND Europe to support the strategic review process. RAND Europe's role was to conduct a series of rapid evidence reviews around the recommendations arising from the five working groups that comprise the initiative. The reviews were conducted around networks and collaboration, data linkage and exchange, culture, values and leadership, and incentives for innovation, while capacity building was a recurrent theme throughout. This study presents the rapid evidence reviews, summarising relevant literature and highlighting international examples of particularly relevant or innovative approaches. The issues and ideas identified around each theme are then pulled together in a suggested conceptual representation of the Norwegian health and care research system.

International comparative performance of mental health research, 1980-2011.

Scientific understanding of mental illness, mental health and their neurobiological and psychosocial underpinnings has greatly increased in the last three decades. Yet, little is known about the landscape of this knowledge and how and where it is evolving. This paper provides a bibliometric assessment of mental health research (MHR) outputs from 1980 to 2011. MHR papers were retrieved using three strategies: from key mental health journals; using US National Library of Medicine Medical Subject Heading (MeSH) keywords; and from additional journals in which mental health topics accounted for over 75% of papers. The number of papers per year increased over time in absolute terms and as a proportion of total medical output. The US's proportion of world publication output dropped from 60% in 1980 to 42% in 2011, while the EU increased its share from 27% to 40%. Countries with greater research intensity in mental health generally had higher citation impact, such as the US, UK, Canada and the Netherlands. MHR also became more collaborative: 3% of all MHR papers published in 1980 were the result of international collaboration compared to 22% in 2011. We conclude by noting that the rise in MHR appears to be due to funding and that bibliometrics can help highlight the potential drivers of variation in performance of MHR systems. The paper provides an analytical basis for benchmarking MHR trends in the future.

Project Retrosight: Understanding the Returns from Cardiovascular and Stroke Research: The Policy Report.

This project explores the impacts arising from cardiovascular and stroke research funded 15-20 years ago and attempts to draw out aspects of the research, researcher or environment that are associated with high or low impact. The project is a case study-based review of 29 cardiovascular and stroke research grants, funded in Australia, Canada and UK between 1989 and 1993. The case studies focused on the individual grants but considered the development of the investigators and ideas involved in the research projects from initiation to the present day. Grants were selected through a stratified random selection approach that aimed to include both high- and low-impact grants. The key messages are as follows: 1) The cases reveal that a large and diverse range of impacts arose from the 29 grants studied. 2) There are variations between the impacts derived from basic biomedical and clinical research. 3) There is no correlation between knowledge production and wider impacts 4) The majority of economic impacts identified come from a minority of projects. 5) We identified factors that appear to be associated with high and low impact. This article presents the key observations of the study and an overview of the methods involved. It has been written for funders of biomedical and health research and health services, health researchers, and policy makers in those fields. It will also be of interest to those involved in research and impact evaluation.

An investigation of interpretive bias in insomnia: an analog study comparing normal and poor sleepers.

STUDY OBJECTIVES: Cognitive theories state that psychological disorders are associated with, and are possibly maintained by, interpretive biases, which are tendencies to make threatening interpretations of ambiguous stimuli. Recent models of insomnia have highlighted the importance of cognitive processes. The aim of this study was to empirically evaluate whether an interpretive bias is present in poor sleepers. DESIGN: A mixed-design analysis of covariance was employed with group (normal sleepers vs poor sleepers) as a between-subjects variable and sentence type (insomnia-related vs anxiety related) as a within-subjects variable. The dependent variables were the extent to which participants interpreted insomnia-related and anxiety-related sentences as having a threatening meaning. Sleepiness was used as a covariate. SETTING: Treatment and research clinic at a university department of psychiatry. PARTICIPANTS: Forty-one normal and 34 poor sleepers. MEASUREMENTS AND RESULTS: A set of ambiguous scenarios were administered to participants who gave open-ended and forced-choice interpretations of the scenarios. Each scenario could be interpreted in a threat (insomnia or anxiety)-related or neutral manner. Even after controlling for sleepiness, poor sleepers were found to make significantly more threat-related interpretations of ambiguous scenarios than did normal sleepers. CONCLUSIONS: These findings suggest that there is a bias toward threat-related interpretations among poor sleepers and that the exploration of biased interpretations may be an important avenue for future research among individuals who meet full diagnostic criteria for insomnia.