Creating a developmental framework for evaluating RRI implementation in research organisations.

The evaluation of Responsible Research and Innovation (RRI) actions presents important challenges for the key stakeholders engaged in the process of RRI implementation, such as policy makers, programme managers, and researchers. While there is a considerable body of literature on the conceptualization of responsibility in research and a plethora of attempts to practice RRI, there is a need for increased attention to the monitoring and evaluation of case studies of RRI implementations in research organisations, in particular regarding their structural change effects. This paper aims to discuss a contextualised developmental framework for evaluating RRI implementation in research organisations, with a specific focus on achieving structural change through tailor-made action plans. The framework, developed through RRI evaluation work in the field of biosciences, adopts a systemic and process-oriented perspective, encompassing participatory, anticipatory, reflexive, and responsive dimensions. Concrete empirical examples from bioscience organizations are provided to illustrate how the framework relates to specific conditions, experiences, and solutions, demonstrating how conceptual insights have emerged from real-life practices and data analysis. While the framework was initially customized for the specific contexts of six bioscience research organizations, it holds potential for broader relevance and applicability in addressing challenges related to RRI design, implementation, and evaluation.

Understanding the Athena SWAN award scheme for gender equality as a complex social intervention in a complex system: analysis of Silver award action plans in a comparative European perspective.

BACKGROUND: Given the complex mix of structural, cultural and institutional factors that produce barriers for women in science, an equally complex intervention is required to understand and address them. The Athena SWAN Award Scheme for Gender Equality has become a widespread means to address barriers for women's advancement and leadership in the United Kingdom, Ireland, Australia, the United States of America and Canada, while the European Commission is exploring the introduction of a similar award scheme across Europe. METHODS: This study analyses the design and implementation of 16 departmental Athena SWAN Silver Action Plans in Medical Sciences at one of the world's leading universities in Oxford, United Kingdom. Data pertaining to the design and implementation of gender equality interventions were extracted from the action plans, analysed thematically, coded using categories from the 2015 Athena SWAN Charter Awards Handbook and synthesised against a typology of gender equality interventions in the European Research Area. The results were further analysed against the complexity research literature framework, where research organisations are perceived as dynamic systems that adapt, interact and co-evolve with other systems. RESULTS: Athena SWAN is a complex contextually embedded system of action planning within the context of universities. It depends on a multitude of contextual variables that relate in complex, non-linear ways and dynamically adapt to constantly moving targets and new emergent conditions. Athena SWAN Silver Action Plans conform to the key considerations of complexity - (1) multiple actions and areas of intervention with a focus on the complex system being embedded in local dynamics, (2) the non-linearity of interventions and the constantly emerging conditions, and (3) impact in terms of contribution to change, improved conditions to foster change and the increased probability that change can occur. CONCLUSIONS: To enact effective sustainable structural and cultural change for gender equality, it is necessary to acknowledge and operationalise complexity as a frame of reference. Athena SWAN is the single most comprehensive and systemic gender equality scheme in Europe. It can be further strengthened by promoting the integration of sex and gender analysis in research and education. Gender equality policies in the wider European Research Area can benefit from exploring Athena SWAN's contextually embedded systemic approach to dynamic action planning and inclusive focus on all genders and categories of staff and students.

Developing a conceptual evaluation framework for gender equality interventions in research and innovation.

In this article, we discuss the development of a conceptual evaluation framework to design and assess gender equality interventions and their effects in research and innovation. The conceptual framework presented herewith embraces the complexity, gender-sensitive and theory-based evaluation approaches ensuring that design and evaluation of gender equality interventions consider the complex systems that constitute the context in which the interventions operate. The evaluation framework offers a non-linear concept, where the notion of contribution - not attribution - to achieve impact is central to the integration of team, organizational and system factors in policy design and evaluation. The paper opens the "black box" to address the question of how and why a policy intervention works and in which context and discusses a systematic process on how to approach the interwoven linkages between input, implementation and effects in gender equality interventions in research and innovation, accounting for context sensitivity and methodological pluralism. The evaluation framework may serve as reference for researchers, evaluators, policymakers and other stakeholders in designing and assessing gender equality interventions, and in further developing their evidence, and theoretical and methodological base.

Women Underrepresentation in R&I: A Sector Program Assessment of the Contribution of Gender Equality Policies in Research and Innovation.

Despite decades of efforts to achieve gender equality in research and innovation (R&I), all EU member states still face remarkable difficulties in driving forward the development of their innovation system while at the same time improving gender equality by using all the available research potential. In this paper we focus on the development of the share of women researchers in four national innovation systems, i.e. in Austria, Denmark, Hungary and Spain in the time period 2005-2015. The four selected cases represent countries with significant differences in their innovation capacity, gender regimes and progress of gender equality in R&I. A qualitative comparative analysis (QCA) is carried out to conduct a sector program evaluation based on statistical data and qualitative studies to understand the dynamic development of the proportion of women researchers. The study aims to provide insights into the aggregated gender equality interventions and policies implemented in the four countries studied and their contributions to the development of the proportion of women scientists at the structural level. The analysis reveals that the development of the share of women researchers during the studied period has been particularly influenced by contextual factors, namely the relative size of the business enterprise sector and the share of women among holders of tertiary education. While this is the case, it is found that gender equality interventions need to be more widespread and more effectively designed to be a strong contributing factor to an increasing representation of women in R&I.

Analysing facilitating and hindering factors for implementing gender equality interventions in R&I: Structures and processes.

This article analyses the facilitating and hindering factors that have affected the implementation of gender equality interventions in research and innovation in Europe. It applies the evaluation framework developed in the EFFORTI project that recognizes the complexity of evaluating gender equality interventions in R&I, the importance of factoring in context to any sound evaluation as well as the need to distinguish between the design and implementation of interventions in evaluations. It is based on the analysis of 19 empirical case studies carried out throughout Europe and focuses on those structural and procedural factors that have either facilitated or hindered the implementation process of these interventions. Findings include how the governance framework; top-management commitment; bottom-up participation; framing synergies with other initiatives, strategies for tackling resistance; resources; sustainability of actions; gender competence, experience and knowledge and transparency, targets, standards and monitoring; and accessible data and information all contributed to the successful implementation of the interventions.

Pragmatic ex-ante evaluation using an innovative conceptual framework: The case of a high-tech entrepreneurship program for women.

Program designers at local project level usually design interventions under conditions of scarce resources and bounded rationality lacking sufficient information about the process and impact mechanisms of the complex social programs they design and implement. This paper proposes a model for pragmatic ex ante evaluation using an innovative conceptual framework for practitioners working in the field to improve gender equality in research and innovation. The presented and applied model is pertinent to the theory-oriented evaluation tradition using a logic frame and a theory of change approach adapted to the special requirements of ex-ante evaluation. The model is illustrated by the case study of a program for the promotion of women entrepreneurship in the STEM fields. It is concluded that in a setting with limited resources and information, ex-ante evaluations should aim at defining clear conditions of proportionality and need to take into account the extent to which contextual factors hinder or facilitate the implementation of a program, and the specific institutional and system mechanisms at play, in order to enable proactive risk management from the beginning.

Structural Transformation to Attain Responsible BIOSciences (STARBIOS2): Protocol for a Horizon 2020 Funded European Multicenter Project to Promote Responsible Research and Innovation.

BACKGROUND: Promoting Responsible Research and Innovation (RRI) is a major strategy of the "Science with and for Society" work program of the European Union's Horizon 2020 Framework Programme for Research and Innovation. RRI aims to achieve a better alignment of research and innovation with the values, needs, and expectations of society. The RRI strategy includes the "keys" of public engagement, open access, gender, ethics, and science education. The Structural Transformation to Attain Responsible BIOSciences (STARBIOS2) project promotes RRI in 6 European research institutions and universities from Bulgaria, Germany, Italy, Slovenia, Poland, and the United Kingdom, in partnership with a further 6 institutions from Brazil, Denmark, Italy, South Africa, Sweden, and the United States. OBJECTIVE: The project aims to attain RRI structural change in 6 European institutions by implementing action plans (APs) and developing APs for 3 non-European institutions active in the field of biosciences; use the implementation of APs as a learning process with a view to developing a set of guidelines on the implementation of RRI; and develop a sustainable model for RRI in biosciences. METHODS: The project comprises interrelated research and implementation designed to achieve the aforementioned specific objectives. The project is organized into 6 core work packages and 5 supporting work packages. The core work packages deal with the implementation of institutional APs in 6 European institutions based on the structural change activation model. The supporting work packages include technical assistance, learning process on RRI-oriented structural change, monitoring and assessment, communication and dissemination, and project management. RESULTS: The project is funded by Horizon 2020 and will run for 4 years (May 2016-April 2020). As of June 2018, the initial phase has been completed. The participating institutions have developed and approved APs and commenced their implementation. An observation tool has been launched by the Technical Assistance Team to collect information from the implementation of APs; the Evaluation & Assessment team has started monitoring the advancement of the project. As part of the communication and dissemination strategy, a project website, a Facebook page, and a Twitter account have been launched and are updated periodically. The International Scientific Advisory Committee has been formed to advise on the reporting and dissemination of the project's results. CONCLUSIONS: In the short term, we anticipate that the project will have a considerable impact on the organizational processes and structures, improving the RRI uptake in the participating institutions. In the medium term, we expect to make RRI-oriented organizational change scalable across Europe by developing guidelines on RRI implementation and an RRI model in biosciences. In the long term, we expect that the project would help increase the ability of research institutions to make discoveries and innovations in better alignment with societal needs and values. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): DERR1-10.2196/11745.