ISRIA statement: ten-point guidelines for an effective process of research impact assessment.

As governments, funding agencies and research organisations worldwide seek to maximise both the financial and non-financial returns on investment in research, the way the research process is organised and funded is becoming increasingly under scrutiny. There are growing demands and aspirations to measure research impact (beyond academic publications), to understand how science works, and to optimise its societal and economic impact. In response, a multidisciplinary practice called research impact assessment is rapidly developing. Given that the practice is still in its formative stage, systematised recommendations or accepted standards for practitioners (such as funders and those responsible for managing research projects) across countries or disciplines to guide research impact assessment are not yet available.In this statement, we propose initial guidelines for a rigorous and effective process of research impact assessment applicable to all research disciplines and oriented towards practice. This statement systematises expert knowledge and practitioner experience from designing and delivering the International School on Research Impact Assessment (ISRIA). It brings together insights from over 450 experts and practitioners from 34 countries, who participated in the school during its 5-year run (from 2013 to 2017) and shares a set of core values from the school's learning programme. These insights are distilled into ten-point guidelines, which relate to (1) context, (2) purpose, (3) stakeholders' needs, (4) stakeholder engagement, (5) conceptual frameworks, (6) methods and data sources, (7) indicators and metrics, (8) ethics and conflicts of interest, (9) communication, and (10) community of practice.The guidelines can help practitioners improve and standardise the process of research impact assessment, but they are by no means exhaustive and require evaluation and continuous improvement. The prima facie effectiveness of the guidelines is based on the systematised expert and practitioner knowledge of the school's faculty and participants derived from their practical experience and research evidence. The current knowledge base has gaps in terms of the geographical and scientific discipline as well as stakeholder coverage and representation. The guidelines can be further strengthened through evaluation and continuous improvement by the global research impact assessment community.

A global call for action to include gender in research impact assessment.

Global investment in biomedical research has grown significantly over the last decades, reaching approximately a quarter of a trillion US dollars in 2010. However, not all of this investment is distributed evenly by gender. It follows, arguably, that scarce research resources may not be optimally invested (by either not supporting the best science or by failing to investigate topics that benefit women and men equitably). Women across the world tend to be significantly underrepresented in research both as researchers and research participants, receive less research funding, and appear less frequently than men as authors on research publications. There is also some evidence that women are relatively disadvantaged as the beneficiaries of research, in terms of its health, societal and economic impacts. Historical gender biases may have created a path dependency that means that the research system and the impacts of research are biased towards male researchers and male beneficiaries, making it inherently difficult (though not impossible) to eliminate gender bias. In this commentary, we - a group of scholars and practitioners from Africa, America, Asia and Europe - argue that gender-sensitive research impact assessment could become a force for good in moving science policy and practice towards gender equity. Research impact assessment is the multidisciplinary field of scientific inquiry that examines the research process to maximise scientific, societal and economic returns on investment in research. It encompasses many theoretical and methodological approaches that can be used to investigate gender bias and recommend actions for change to maximise research impact. We offer a set of recommendations to research funders, research institutions and research evaluators who conduct impact assessment on how to include and strengthen analysis of gender equity in research impact assessment and issue a global call for action.

Impact of clinical and health services research projects on decision-making: a qualitative study.

BACKGROUND: This article reports on the impact assessment experience of a funding program of non-commercial clinical and health services research. The aim was to assess the level of implementation of results from a subgroup of research projects (on respiratory diseases), and to detect barriers (or facilitators) in the translation of new knowledge to informed decision-making. METHODS: A qualitative study was performed. The sample consisted of six projects on respiratory diseases funded by the Agency for Health Quality and Assessment of Catalonia between 1996 and 2004. Semi-structured interviews to key informants including researchers and healthcare decision-makers were carried out. Interviews were recorded, transcribed verbatim and analysed on an individual (key informant) and group (project) basis. In addition, the differences between achieved and expected impacts were described. RESULTS: Twenty-three semi-structured interviews were conducted. Most participants indicated changes in health services or clinical practice had resulted from research. The channels used to transfer new knowledge were mainly conventional ones, but also in less explicit ways, such as with the involvement of local scientific societies, or via debates and discussions with colleagues and local leaders. The barriers and facilitators identified were mostly organizational (in research management, and clinical and healthcare practice), although there were also some related to the nature of the research as well as personal factors. Both the expected and achieved impacts enabled the identification of the gaps between what is expected and what is truly achieved. CONCLUSIONS: In this study and according to key informants, the impact of these research projects on decision-making can be direct (the application of a finding or innovation) or indirect, contributing to a more complex change in clinical practice and healthcare organization, both having other contextual factors. The channels used to transfer this new knowledge to clinical practice are complex. Local scientific societies and the relationships between researchers and decision-makers can play a very important role. Specifically, the relationships between managers and research teams and the mutual knowledge of their activity have shown to be effective in applying research funding to practice and decision-making. Finally the facilitating factors and barriers identified by the respondents are closely related to the idiosyncrasy of the human relations between the different stakeholders involved.

The quality of gender and sex integration in scientific articles resulting from health research funded by the Brazilian Ministry of Health 2004-2016.

Aim: Integrating sex and gender into health research is critical to contributing to an ethical and more responsible science to address significant knowledge gaps, resulting in higher-quality evidence for all. Subject and methods: Using the Essential Metrics for Assessing Sex and Gender Integration in Health Research Proposals Involving Human Participants, we evaluate the quality of the integration of sex and gender in the 350 scientific articles produced by 144 health studies funded by the Department of Science and Technology of the Brazilian Ministry of Health between 2004 and 2016. Results: The results show that clinical research articles are the type of studies that most frequently report on sex differences, while population and public health research articles most frequently report on gender differences. Analysis of the quality of sex and gender integration reveals low levels of qualification in the items of the literature review and research objectives (section 1) and participant recruitment and retention (section 2). However, the data collection tools, data analysis, and knowledge translation (section 3) items were rated as excellent and good. Conclusion: Funding agencies and public institutions should recognize the importance of the integration of sex and gender at all stages of the research process, for instance, through awareness and training for researchers and reviewers, clear requirements, and the possibility to use metrics in the evaluations process.

Double incontinence in a cohort of nulliparous pregnant women.

AIMS: To estimate the frequency of double incontinence, and to identify associated risk factors during pregnancy and postpartum in previously continent nulliparous women. METHODS: A cohort study in healthy, nulliparous, continent pregnant women, attending the public healthcare system of Catalonia (Spain) was designed. The field work was conducted during the control visits of pregnancy, at the time of delivery, and in the postpartum. Double incontinence was defined as a situation in which participants reported simultaneously urinary incontinence (UI) and anal incontinence with the help of a self-administered questionnaire. Prevalence rates, and their corresponding confidence intervals (95% CI), were calculated, as well as the impact on daily life. Multivariable logistic regressions in pregnancy and postpartum were estimated to assess the association of potential risk factors. RESULTS: The prevalence rate of double incontinence during pregnancy was 8.6% (95% CI: 7.0-10.3). Age over 35 years, and family history of UI were associated to a higher risk of double incontinence. After delivery, the prevalence rate decreased to 3.5% (95% CI: 2.4-4.6); only 6.7% of women with double incontinence in pregnancy had a persistency of their symptoms in early postpartum. Instrumental vaginal deliveries carried 2.2 times more risk of double incontinence than spontaneous ones. Episiotomy implied a higher risk for double incontinence. CONCLUSIONS: Symptoms of double incontinence are prevalent during first pregnancy; age and other intrinsic factors may favor the occurrence of double incontinence throughout gestation, while instrumental delivery and episiotomy increase the risk of double incontinence in the postpartum period.

Healthcare professionals' opinions, barriers and facilitators towards low-value clinical practices in the hospital setting.

OBJECTIVE: To explore healthcare professionals' opinions about low-value practices, identify practices of this kind possibly present in the hospital and barriers and facilitators to reduce them. Low-value practices include those with little or no clinical benefit that may harm patients or lead to a waste of resources. METHOD: Using a mixed methodology, we carried out a survey and two focus groups in a tertiary hospital. In the survey, we assessed doctors' agreement, subjective adherence and perception of usefulness of 134 recommendations to reduce low-value practices from local and international initiatives. We also identified low-value practices possibly present in the hospital. In the focus groups with professionals from surgical and medical fields, using a phenomenological approach, we identified additional low-value practices, barriers and facilitators to reduce them. RESULTS: 169 doctors of 25 specialties participated (response rate: 7%-100%). Overall agreement with recommendations, subjective adherence and usefulness were 83%, 90% and 70%, respectively. Low-value practices form 22 recommendations (16%) were considered as possibly present in the hospital. In the focus groups, the professionals identified seven more. Defensive medicine and scepticism due to contradictory evidence were the main barriers. Facilitators included good leadership and coordination between professionals. CONCLUSIONS: High agreement with recommendations to reduce low-value practices and high perception of usefulness reflect great awareness of low-value care in the hospital. However, there are several barriers to reduce them. Interventions to reduce low-value practices should foster confidence in decision-making processes between professionals and patients and provide trusted evidence.

[Exhaustivity and critical tone of the news in print media reporting medical innovations]. / Exhaustividad y tono crítico de las noticias en la prensa escrita que informan de una innovación médica.

OBJECTIVE: To quantify how exhaustive and critical were stories reporting medical innovations published in print media and to analyze the characteristics that may be related. METHOD: Content analysis of the newspapers stories related to the discovery, introduction or improvement of a medical innovation through a questionnaire with ten criteria that allows calculating an overall score of exhaustiveness. The critical view was also included. RESULTS: We analyzed 204 newspapers stories that on average obtained a comprehensiveness score of 4.5. Were optimistic 70% of the stories. The most valued criteria were: level of detail of the explanation of the innovation and the correct differentiation between facts and opinions. While the worst-valued criteria were: disclosure of financial conflicts of interest and the quantification of harms. The variables author, length of the story and classification of the innovation were related to both the comprehensiveness score and the critical view. The comprehensiveness score was also related to the pathology, number of sources of information and the critical tone of the story, while the critical view was also related to the newspapers diffusion and the relevance of the news. CONCLUSIONS: The analyzed stories presented inaccuracies, biases or an excess of optimism (either intentional or involuntary). Some aspects of the stories discussed in more detail would provide solutions to many of the identified shortcomings.

[Medical indications for acupuncture: Systematic review]. / Indicaciones médicas de la acupuntura: revisión sistemática.

Acupuncture is a medical procedure with a very wide range of indications according to the WHO. However the indications require robust scientific evidence to support them. We have conducted a systematic review (2010-2015) in order to define in which pathologies acupuncture can be an effective strategy, STRICTA criteria that aim to set up acupuncture clinical trials standard criteria were defined in 2010. Only systematic reviews and meta-analyses of good or very good methodological quality according to SIGN criteria were selected. Its main objective was to evaluate the effectiveness of acupuncture in the management of any disease. Most of the final 31 selected reviews focus on chronic pain-related diseases, mainly in the disciplines of Neurology, Orthopaedics and Rheumatology. Current evidence supports the use of acupuncture in the treatment of headaches, migraines, back pain, cervical pain and osteoarthritis. The remaining pathologies still require further good quality studies.

[Development of a Conceptual Framework for the Assessment of Chronic Care in the Spanish National Health System]. / Desarrollo de un marco conceptual para la evaluación de la atención a la cronicidad en el sistema nacional de salud.

The aim was to develop a conceptual framework for the assessment of new healthcare initiatives on chronic diseases within the Spanish National Health System. A comprehensive literature review between 2002 and 2013, including systematic reviews, meta-analysis, and reports with evaluation frameworks and/or assessment of initiatives was carried out; integrated care initiatives established in Catalonia were studied and described; and semistructured interviews with key stakeholders were performed. The scope and conceptual framework were defined by using the brainstorming approach.Of 910 abstracts identified, a total of 116 studies were included. They referred to several conceptual frameworks and/or assessment indicators at a national and international level. An overall of 24 established chronic care initiatives were identified (9 integrated care initiatives); 10 in-depth interviews were carried out. The proposed conceptual framework envisages: 1)the target population according to complexity levels; 2)an evaluation approach of the structure, processes, and outcomes considering the health status achieved, the recovery process and the maintenance of health; and 3)the dimensions or attributes to be assessed. The proposed conceptual framework will be helpful has been useful to develop indicators and implement them with a community-based and result-oriented approach and a territorial or population-based perspective within the Spanish Health System. This will be essential to know which are the most effective strategies, what are the key elements that determine greater success and what are the groups of patients who can most benefit.

El objetivo del trabajo fue desarrollar un marco conceptual para la evaluación de nuevos programas de gestión clínica y asistencial de carácter integrado en el Sistema Nacional de Salud para la atención a las personas con enfermedades crónicas. Se realizó una revisión exhaustiva de la literatura entre 2002-2013 incluyendo revisiones sistemáticas, metaanálisis e informes con modelos de evaluación y/o evaluaciones de programas. Se estudiaron y describieron programas locales de atención a la cronicidad implementados en Cataluña y se realizaron entrevistas semiestructuradas con expertos clave en el ámbito catalán. El alcance y el marco conceptual se definieron mediante la técnica del brainstorming. De 910 resúmenes identificados, se incluyeron 116 documentos que se referían a marcos conceptuales e indicadores de evaluación a nivel español e internacional. Se identificaron 24 programas de atención a la cronicidad (9 de carácter integrado). El marco conceptual propuesto contempló: 1) la población diana según niveles de complejidad; 2) un enfoque de evaluación de la estructura, procesos y resultados teniendo en cuenta el estado de salud conseguido, el proceso de recuperación y mantenimiento de la salud; y 3) las dimensiones o atributos que se deben evaluar. El marco conceptual propuesto permite desarrollar indicadores e implementarlos con un enfoque comunitario, orientados a los resultados y tener una visión territorial o poblacional, que será imprescindible para saber cuál es la estrategia más efectiva, cuáles son los elementos que determinan un mayor éxito y cuáles son los grupos de pacientes que más se pueden beneficiar.

Impact of a priority system on patients in waiting lists for knee arthroplasty.

RATIONALE, AIMS AND OBJECTIVES: The Agency for Health Quality and Assessment of Catalonia (AQuAS) developed a system for the prioritization of patients on non-urgent waiting lists for cataract as well as knee and hip arthroplasty. The aim was to evaluate the impact of the priority system's application for primary knee arthroplasty (PKA). METHODS: A retrospective evaluative study with data from the Data Tracking and Management Registry of the CatSalut Waiting Lists for PKA of hospitals in the public hospital network in the period 2003-2009. A description of the characteristics of patients on waiting lists was made and the association between priority scoring and waiting time and the order of operation analysed. Finally, waiting times were simulated that patients would have experienced if being operated on strictly according to a first in first out system or one of priority scoring, to compare them with real waiting times. RESULTS: The number of people included on waiting lists for PKA was 67403. 67% had a priority score. The distribution of the priority score was negatively skewed with an average score of 70 points. The association with the priority score for the waiting time as well as the order of the operations performed was practically null. CONCLUSIONS: The study concludes that, globally, the prioritization system for PKA has been implemented but had no effect on the prioritization of patients based on their severity. Nevertheless, in some centres, a moderate correlation between the order of operations performed and the priority score was identified.

Developing a universal tool for the prioritization of patients waiting for elective surgery.

The objective was to elaborate a priority scoring system for patients on waiting lists for elective surgery to be implemented in the Catalan public health system. This tool should ideally be universal (for all patients and across the entire region) with common criteria and weights (for all surgical procedures), simple and user-friendly. A tool based on a point-count linear scale ranging from 0 (lowest priority) to 100 (highest priority) was developed. Patients are scored in three major dimensions: clinical and functional impairment, expected benefit, and social role, which include 8 criteria (with their weights): disease severity (23%), pain (or other main symptoms) (14%), rate of disease progression (15%), difficulty in doing activities of daily life (14%), probability and degree of improvement (12%), being dependent with no caregiver (5%), limitation to care for one's dependents (if that be the case) (8%), and limitations in the ability to work, study or seek for employment (9%). As in previous studies developed in Canada, New Zealand and Catalonia, the tool obtained is mainly based on severity and need. The success of this tool depends very much on implementation mechanisms. Furthermore, prior to implementation, a definition of specificities in the selected criteria for the most frequent surgical procedures is advised.

Characteristics of Spanish articles of "scientific quality" cited in clinical practice guidelines on mental health.

INTRODUCTION: The study aims to illustrate the impact of Spanish research in clinical decision making. To this end, we analysed the characteristics of the most significant Spanish publications cited in clinical practice guidelines (CPG) on mental health. MATERIAL AND METHODS: We conducted a descriptive qualitative study on the characteristics of ten articles cited in Spanish CPG on mental health, and selected for their "scientific quality". We analysed the content of the articles on the basis of the following characteristics: topics, study design, research centres, scientific and practical relevance, type of funding, and area or influence of the reference to the content of the guidelines. RESULTS: Among the noteworthy studies, some basic science studies, which have examined the establishment of genetic associations in the pathogenesis of mental illness are included, and others on the effectiveness of educational interventions. The content of those latter had more influence on the GPC, because they were cited in the summary of the scientific evidence or in the recommendations. Some of the outstanding features in the selected articles are the sophisticated designs (experimental or analytical), and the number of study centres, especially in international collaborations. Debate or refutation of previous findings on controversial issues may have also contributed to the extensive citation of work. CONCLUSIONS: The inclusion of studies in the CPG is not a sufficient condition of "quality", but their description can be instructive for the design of future research or publications.

Urinary and anal incontinence during pregnancy and postpartum: incidence, severity, and risk factors.

OBJECTIVE: To estimate frequency and severity and to identify risk factors of urinary incontinence (UI) and anal incontinence during pregnancy and after delivery in previously continent nulliparous women. METHODS: We designed a cohort study of healthy, continent, nulliparous pregnant women attending public health care services. The field work was conducted during the control visits of the three trimesters of pregnancy, at the time of delivery, and postpartum. A self-administered questionnaire was used to assess UI (validated and adapted) and anal incontinence. Frequency of UI and anal incontinence and their confidence intervals (95% CIs) were calculated. The correlations between the severity of UI and the degree of effect on daily life were also estimated. Multivariable Cox models were applied to estimate hazard ratios for both incontinences (urinary/anal) during pregnancy and postpartum. RESULTS: The cumulative incidence rate during pregnancy was 39.1% (95% CI 36.3-41.9) for UI and 10.3% (95% CI 8.3-12.3) for anal incontinence. The correlation between severity of UI and effect on daily life was moderate. Age, baseline body mass index, and family history of UI were significantly associated with the occurrence of UI during pregnancy, while age and excess weight gain during pregnancy were associated with the occurrence of anal incontinence during pregnancy. Postpartum, the identified risk factors for both incontinences were incontinence during pregnancy and vaginal delivery. CONCLUSION: The occurrence of UI and anal incontinence during the postpartum period is related to the presence of incontinence in pregnancy, and vaginal delivery increases the risk of persistent incontinence. Some risk factors for both incontinences during pregnancy and postpartum are related to lifestyles and obstetric practices.