ISRIA statement: ten-point guidelines for an effective process of research impact assessment.

As governments, funding agencies and research organisations worldwide seek to maximise both the financial and non-financial returns on investment in research, the way the research process is organised and funded is becoming increasingly under scrutiny. There are growing demands and aspirations to measure research impact (beyond academic publications), to understand how science works, and to optimise its societal and economic impact. In response, a multidisciplinary practice called research impact assessment is rapidly developing. Given that the practice is still in its formative stage, systematised recommendations or accepted standards for practitioners (such as funders and those responsible for managing research projects) across countries or disciplines to guide research impact assessment are not yet available.In this statement, we propose initial guidelines for a rigorous and effective process of research impact assessment applicable to all research disciplines and oriented towards practice. This statement systematises expert knowledge and practitioner experience from designing and delivering the International School on Research Impact Assessment (ISRIA). It brings together insights from over 450 experts and practitioners from 34 countries, who participated in the school during its 5-year run (from 2013 to 2017) and shares a set of core values from the school's learning programme. These insights are distilled into ten-point guidelines, which relate to (1) context, (2) purpose, (3) stakeholders' needs, (4) stakeholder engagement, (5) conceptual frameworks, (6) methods and data sources, (7) indicators and metrics, (8) ethics and conflicts of interest, (9) communication, and (10) community of practice.The guidelines can help practitioners improve and standardise the process of research impact assessment, but they are by no means exhaustive and require evaluation and continuous improvement. The prima facie effectiveness of the guidelines is based on the systematised expert and practitioner knowledge of the school's faculty and participants derived from their practical experience and research evidence. The current knowledge base has gaps in terms of the geographical and scientific discipline as well as stakeholder coverage and representation. The guidelines can be further strengthened through evaluation and continuous improvement by the global research impact assessment community.

A global call for action to include gender in research impact assessment.

Global investment in biomedical research has grown significantly over the last decades, reaching approximately a quarter of a trillion US dollars in 2010. However, not all of this investment is distributed evenly by gender. It follows, arguably, that scarce research resources may not be optimally invested (by either not supporting the best science or by failing to investigate topics that benefit women and men equitably). Women across the world tend to be significantly underrepresented in research both as researchers and research participants, receive less research funding, and appear less frequently than men as authors on research publications. There is also some evidence that women are relatively disadvantaged as the beneficiaries of research, in terms of its health, societal and economic impacts. Historical gender biases may have created a path dependency that means that the research system and the impacts of research are biased towards male researchers and male beneficiaries, making it inherently difficult (though not impossible) to eliminate gender bias. In this commentary, we - a group of scholars and practitioners from Africa, America, Asia and Europe - argue that gender-sensitive research impact assessment could become a force for good in moving science policy and practice towards gender equity. Research impact assessment is the multidisciplinary field of scientific inquiry that examines the research process to maximise scientific, societal and economic returns on investment in research. It encompasses many theoretical and methodological approaches that can be used to investigate gender bias and recommend actions for change to maximise research impact. We offer a set of recommendations to research funders, research institutions and research evaluators who conduct impact assessment on how to include and strengthen analysis of gender equity in research impact assessment and issue a global call for action.

Impact of clinical and health services research projects on decision-making: a qualitative study.

BACKGROUND: This article reports on the impact assessment experience of a funding program of non-commercial clinical and health services research. The aim was to assess the level of implementation of results from a subgroup of research projects (on respiratory diseases), and to detect barriers (or facilitators) in the translation of new knowledge to informed decision-making. METHODS: A qualitative study was performed. The sample consisted of six projects on respiratory diseases funded by the Agency for Health Quality and Assessment of Catalonia between 1996 and 2004. Semi-structured interviews to key informants including researchers and healthcare decision-makers were carried out. Interviews were recorded, transcribed verbatim and analysed on an individual (key informant) and group (project) basis. In addition, the differences between achieved and expected impacts were described. RESULTS: Twenty-three semi-structured interviews were conducted. Most participants indicated changes in health services or clinical practice had resulted from research. The channels used to transfer new knowledge were mainly conventional ones, but also in less explicit ways, such as with the involvement of local scientific societies, or via debates and discussions with colleagues and local leaders. The barriers and facilitators identified were mostly organizational (in research management, and clinical and healthcare practice), although there were also some related to the nature of the research as well as personal factors. Both the expected and achieved impacts enabled the identification of the gaps between what is expected and what is truly achieved. CONCLUSIONS: In this study and according to key informants, the impact of these research projects on decision-making can be direct (the application of a finding or innovation) or indirect, contributing to a more complex change in clinical practice and healthcare organization, both having other contextual factors. The channels used to transfer this new knowledge to clinical practice are complex. Local scientific societies and the relationships between researchers and decision-makers can play a very important role. Specifically, the relationships between managers and research teams and the mutual knowledge of their activity have shown to be effective in applying research funding to practice and decision-making. Finally the facilitating factors and barriers identified by the respondents are closely related to the idiosyncrasy of the human relations between the different stakeholders involved.

[Information, knowledge and healthcare practice: professionals participation as the key element of the gear]. / Información, conocimiento y práctica sanitaria: la participación de los profesionales como pieza clave del engranaje.

This article analyzes the role of ICT within the complicated gear between information, knowledge and healthcare practices, which particular focus on two specific cases: the digitalization process of the healthcare system and the application of knowledge into the healthcare practices. In both cases, international and local experiences suggest, and sometimes demonstrate the importance of the participation, capacity-building and empowerment of healthcare practitioners for the generation, transfer and use of information and knowledge empowered by the digital tools which should bring into the system better performance, more efficacy, efficiency, equity, equality, security, quality.

Impact of a priority system on patients in waiting lists for knee arthroplasty.

RATIONALE, AIMS AND OBJECTIVES: The Agency for Health Quality and Assessment of Catalonia (AQuAS) developed a system for the prioritization of patients on non-urgent waiting lists for cataract as well as knee and hip arthroplasty. The aim was to evaluate the impact of the priority system's application for primary knee arthroplasty (PKA). METHODS: A retrospective evaluative study with data from the Data Tracking and Management Registry of the CatSalut Waiting Lists for PKA of hospitals in the public hospital network in the period 2003-2009. A description of the characteristics of patients on waiting lists was made and the association between priority scoring and waiting time and the order of operation analysed. Finally, waiting times were simulated that patients would have experienced if being operated on strictly according to a first in first out system or one of priority scoring, to compare them with real waiting times. RESULTS: The number of people included on waiting lists for PKA was 67403. 67% had a priority score. The distribution of the priority score was negatively skewed with an average score of 70 points. The association with the priority score for the waiting time as well as the order of the operations performed was practically null. CONCLUSIONS: The study concludes that, globally, the prioritization system for PKA has been implemented but had no effect on the prioritization of patients based on their severity. Nevertheless, in some centres, a moderate correlation between the order of operations performed and the priority score was identified.

Developing a universal tool for the prioritization of patients waiting for elective surgery.

The objective was to elaborate a priority scoring system for patients on waiting lists for elective surgery to be implemented in the Catalan public health system. This tool should ideally be universal (for all patients and across the entire region) with common criteria and weights (for all surgical procedures), simple and user-friendly. A tool based on a point-count linear scale ranging from 0 (lowest priority) to 100 (highest priority) was developed. Patients are scored in three major dimensions: clinical and functional impairment, expected benefit, and social role, which include 8 criteria (with their weights): disease severity (23%), pain (or other main symptoms) (14%), rate of disease progression (15%), difficulty in doing activities of daily life (14%), probability and degree of improvement (12%), being dependent with no caregiver (5%), limitation to care for one's dependents (if that be the case) (8%), and limitations in the ability to work, study or seek for employment (9%). As in previous studies developed in Canada, New Zealand and Catalonia, the tool obtained is mainly based on severity and need. The success of this tool depends very much on implementation mechanisms. Furthermore, prior to implementation, a definition of specificities in the selected criteria for the most frequent surgical procedures is advised.

Characteristics of Spanish articles of "scientific quality" cited in clinical practice guidelines on mental health.

INTRODUCTION: The study aims to illustrate the impact of Spanish research in clinical decision making. To this end, we analysed the characteristics of the most significant Spanish publications cited in clinical practice guidelines (CPG) on mental health. MATERIAL AND METHODS: We conducted a descriptive qualitative study on the characteristics of ten articles cited in Spanish CPG on mental health, and selected for their "scientific quality". We analysed the content of the articles on the basis of the following characteristics: topics, study design, research centres, scientific and practical relevance, type of funding, and area or influence of the reference to the content of the guidelines. RESULTS: Among the noteworthy studies, some basic science studies, which have examined the establishment of genetic associations in the pathogenesis of mental illness are included, and others on the effectiveness of educational interventions. The content of those latter had more influence on the GPC, because they were cited in the summary of the scientific evidence or in the recommendations. Some of the outstanding features in the selected articles are the sophisticated designs (experimental or analytical), and the number of study centres, especially in international collaborations. Debate or refutation of previous findings on controversial issues may have also contributed to the extensive citation of work. CONCLUSIONS: The inclusion of studies in the CPG is not a sufficient condition of "quality", but their description can be instructive for the design of future research or publications.