Exploring the impact of evaluation on learning and health innovation sustainability: protocol for a realist synthesis.

BACKGROUND: Within the Learning Health System (LHS) model, learning routines, including evaluation, allow for continuous incremental change to take place. Within these learning routines, evaluation assists in problem identification, data collection, and data transformation into contextualized information, which is then re-applied to the LHS environment. Evaluation that catalyzes learning and improvement may also contribute to health innovation sustainability. However, there is little consensus as to why certain evaluations seem to support learning and sustainability, while others impede it. This realist synthesis seeks to understand the contextual factors and underlying mechanisms or drivers that best support health systems learning and sustainable innovation. METHODS: This synthesis will be guided by Pawson and colleagues' 2005 and Emmel and colleagues' 2018 guidelines for conducting realist syntheses. The review process will encompass five steps: (1) scoping the review, (2) building theories, (3) identifying the evidence, (4) evidence selection and appraisal, and (5) data extraction and synthesis. An Expert Committee comprised of leaders in evaluation, innovation, sustainability, and realist methodology will guide this synthesis. Review findings will be reported using the RAMESES guidelines. DISCUSSION: The use of a realist review will allow for exploration and theorizing about the contextual factors and underlying mechanisms that make evaluations 'work' (or 'not work') to support learning and sustainability. Depending on results, we will attempt to synthesize findings into a series of recommendations for evaluations with the intention to support health systems learning and sustainability. Finalized results will be presented at national and international conferences, as well as disseminated via a peer-reviewed publication. SYSTEMATIC REVIEW REGISTRATION: This realist synthesis protocol has been registered with PROSPERO ( https://www.crd.york.ac.uk/prospero/  ID 382690).

Strategies used throughout the scaling-up process of eConsult - Multiple case study of four Canadian Provinces.

BACKGROUND: eConsult is a model of asynchronous communication connecting primary care providers to specialists to discuss patient care. This study aims to analyze the scaling-up process and identify strategies used to support scaling-up efforts in four provinces in Canada. METHODS: We conducted a multiple case study with four cases (ON, QC, MB, NL). Data collection methods included document review (n = 93), meeting observations (n = 65) and semi-structured interviews (n = 40). Each case was analyzed based on Milat's framework. RESULTS: The first scaling-up phase was marked by the rigorous evaluation of eConsult pilot projects and the publication of over 90 scientific papers. In the second phase, provinces implemented provincial multi-stakeholder committees, institutionalized the evaluation, and produced documents detailing the scaling-up plan. During the third phase, efforts were made to lead proofs of concept, obtain the endorsement of national and provincial organizations, and mobilize alternate sources of funding. The last phase was mainly observed in Ontario, where the creation of a provincial governance structure and strategies were put in place to monitor the service and manage changes. CONCLUSIONS: Various strategies need to be used throughout the scaling-up process. The process remains challenging and lengthy because health systems lack clear processes to support innovation scaling-up.

Mapping the Newcomer Journey for More Equitable Population Health: Insights from an Ontario Health Team.

The COVID-19 pandemic has magnified systemic vulnerabilities and made the global and Canadian newcomer experience even more fragile. In 2022, the Kitchener, Waterloo, Wellesley, Wilmot and Woolwich (KW4) Ontario Health Team launched a journey-mapping initiative with the aim to better understand newcomers' lived experiences with regard to their health and wellness within the first two years of their arrival in the region. We interviewed 17 newcomers from 11 different countries. The outcomes of this project are helping to inform a people-centred integrated health system approach toward service redesign and the creation of technological solutions to improve newcomers' abilities to self-navigate local services toward more equitable population health outcomes.

Scaling-Up eConsult: Promising Strategies to Address Enabling Factors in Four Jurisdictions in Canada.

BACKGROUND: Effective healthcare innovations are often not scaled up beyond their initial local context. Lack of practical knowledge on how to move from local innovations to large-system improvement hinders innovation and learning capacity in health systems. Studying scale-up processes can lead to a better understanding of how to facilitate the scale-up of interventions. eConsult is a digital health innovation that aims to connect primary care professionals with specialists through an asynchronous electronic consultation. The recent implementation of eConsult in the public health systems of four Canadian jurisdictions provides a unique opportunity to identify different enabling strategies and related factors that promote the scaling up of eConsult across jurisdictions. METHODS: We conducted a narrative case study in four Canadian provinces, Quebec, Ontario, Manitoba, and Newfoundland & Labrador, over a 3-year period (2018-2021). We observed provincial eConsult committee meetings (n=65) and national eConsult forums (n=3), and we reviewed internal documents (n=93). We conducted semi-structured interviews with key actors in each jurisdiction (eg, researchers, primary care professionals, specialists, policy-makers, and patient partners) (n=40). We conducted thematic analysis guided by the literature on factors and strategies used to scale up innovations. RESULTS: We identified a total of 31 strategies related to six key enabling factors to scaling up eConsult, including: (1) multi-actor engagement; (2) relative advantage; (3) knowledge transfer; (4) strong evidence base; (5) physician leadership; and (6) resource acquisition (eg, human, material, and financial resources). More commonly used strategies, such as leveraging research infrastructure and bringing together various actors, were used to address multiple enabling factors. CONCLUSION: Actors used various strategies to scale up eConsult within their respective contexts, and these helped address six key factors that seemed to be essential to the scale-up of eConsult.

How Openness Serves Innovation in Healthcare? Comment on "What Managers Find Important for Implementation of Innovations in the Healthcare Sector - Practice Through Six Management Perspectives".

The recent study of which enabling factors can facilitate the specific step of moving from idea generation to implementation in healthcare supports that managing innovation is a context-driven process that goes through six categories of change. While this research provides a general and rather comprehensives overview of what successful innovation work needs, it does not offer deeper insights into how categories of change can be operated in the context of accelerated openness in healthcare. I use the concepts of open innovation and open strategy to trying better understand how openness, in terms of greater inclusion and transparency, may or may not serve healthcare innovation through three theoretical questions: to whom, how and when to open up to foster innovation? Whilst diversity of knowledge, actors and systems are growing drivers of innovation, strategizing openness for more deliberate and impactful inclusion and transparency in healthcare management is key to coproducing better health.

Appropriating integrated performance management tools in healthcare: a sociomaterial work story.

PURPOSE: The purpose of this paper is to explore the appropriation of control rooms based on value-based integrated performance management tools implemented in all publicly funded health organizations in Quebec (Canada) as a form of legitimate sociomaterial work. DESIGN/METHODOLOGY/APPROACH: Multi-site organizational ethnographic case studies in two Integrated health and social services centers, with narrative process analysis of triangulated qualitative data collected through non-participant observation (163 h), individual semi-structured interviews (N = 34), and document review (N = 143). FINDINGS: Three types of legitimate sociomaterial work are accomplished when actors appropriate control rooms: 1) reformulating performance management work; 2) disrupting accountability work and; 3) effecting value-based integrated performance management. Each actor (tools, institutions and people) follows recurrent institutional work-paths: tools consistently engage in disruptive work; institutions consistently engage in maintaining work, and people consistently engage in creation work. The study reveals the potential of performance management tools as "effective integrators" of the technological, managerial, policy and delivery levels of data-driven health system performance and improvement. PRACTICAL IMPLICATIONS: This paper draws on theoretically informed empirical insights to develop actionable knowledge around how to better design, implement and adapt tool-driven health system change: 1) Packaging the three agents of data-driven system change in health care: tools, institutions, people; 2) Redefining the search for performance in health care in the context of value creation, and; 3) Strengthening clinical and managerial relevance in health performance management practice. ORIGINALITY/VALUE: The authors aim to stimulate new and original scholarship around the under-theorized concept of sociomaterial work, challenging theoretical, ontological and practical conceptions of work in healthcare organizations and beyond.

Control rooms in publicly-funded health systems: Reviving value in healthcare governance.

BACKGROUND: As part of reforms in 2015, the Ministry of Health and Social Services in Quebec, Canada mandated the national implementation of control rooms, making health system actors accountable for implementing value-based performance management. OBJECTIVE: To explore how do organizational actors appropriate control rooms as managerial tools to influence value-based performance in health systems. DESIGN: Multi-site organizational ethnographic case studies (N = 2) and narrative process analysis of triangulated qualitative data collected through non-participatory observations (179.5 h), individual semi-structured interviews (N = 34), and document review (N = 143). RESULTS: The process of appropriating control rooms plays a crucial role in achieving value-based performance management. Appropriating unfolds along three paths (cognitive, structural, technical) over three phases (implementing, testing, adapting). Implementing control rooms both produces and emerges from improvement capacities within healthcare organizations. Testing tools reveals that incompatibilities between tools, structures and values give rise to value-driven distributed clinical leadership. Adapting tools relies on the adaptability of organizations towards the value system driving the tools, rather than on the adaptability of tools to organizational design. CONCLUSION: There is no "one-size-fits-all" framework to design and support the successful appropriation of control rooms towards achieving value-based performance. However, we believe that consideration for the three distinct phases of appropriation and leveraging the right mechanism to support each phase is a first important step in reviving value in healthcare governance.

Lessons from mandated implementation of a performance management system.

PURPOSE: Lean-inspired approaches and performance management systems are being implemented in public healthcare organisations internationally. However, the literature is inconclusive regarding the benefits of these management tools and there is a lack of knowledge regarding processes for large-scale implementation of these tools. This article aims to describe the implementation process and to better understand how this process influences the mandated performance management system. DESIGN/METHODOLOGY/APPROACH: This research is based on a comparative case study of three healthcare organisations in Canada. Data consist documents, non-participant observation and semi-structured interviews with key actors (n = 30). Analysis is based on a sociotechnical approach to management tools that considers organisational context, and the tool's technical substrate, theory of action and managerial philosophy. FINDINGS: Results show that despite a standardised national mandate, the tool as implemented varied between organisations in terms of technical substrate and managerial philosophy. These variations are explained by the flexibility of the technical substrate, the lack of clarity of the managerial philosophy, and some contextual elements. Successful implementation may rest upon high hybridization of the tool on these different dimensions. A precise and prescribed technical substrate is not sufficient to guarantee implementation of a managerial philosophy. PRACTICAL IMPLICATIONS: Mandated implementation of management tools may be more successful if it is explicit on the managerial philosophy, the technical substrate and the link between the two, and if it provides some leeway to adapt both to the organisational context. ORIGINALITY/VALUE: This is one of the few studies to describe and analyse the process involved in mandated large-scale implementation of performance management systems in public healthcare organisations.

Learning from health system reform trajectories in seven Canadian provinces.

In publicly funded health systems, reform efforts have proliferated to adapt to increasingly complex demands. In Canada, prior research (Lazar et al., 2013, Paradigm Freeze: Why is it so Hard to Reform Health Care in Canada?, McGill-Queen's Press) found that reforms at the end of the 20th century failed to change the fundamentals of the Canadian system based on physician independence and assured universal coverage only for medical and hospital services. This paper focuses on reforms since the turn of the millennium to explore the transformative capacities developed in seven provinces within this system architecture. Longitudinal case studies, based on scientific and grey literature, and interviews with key informants, trace the patterns of reform in each province and reveal five objectives that, to varying degrees, preoccupied reformers: (1) address chronic disease, (2) align health system actors with provincial objectives, (3) shift from hospital to community-based care, (4) integrate physicians, and (5) develop improvement capacities. The range of strategies adopted to achieve these objectives in different provinces is compared to identify emerging pathways of reform and extract lessons for future reformers. We find significant cross-learning between provinces, but also note an emergent dimension to reforms, where multiple strategies aggregate through time to create unique patterns, presenting their own set of possibilities and limitations for the future.

How to Work Collaboratively Within the Health System: Workshop Summary and Facilitator Reflection.

Effectiveness in health services research requires development of specific knowledge and skills for working in partnership with health system decision-makers. In an initial effort to frame capacity-building activities for researchers, we designed a workshop on working collaboratively within the health system. The workshop, based on recent research exploring health system experience and perspectives on research collaborations, was trialed at the annual Canadian Health Services and Policy Research (CAHSPR) conference in May 2019. Participants reported positive evaluations of the workshop. However, further efforts should target health services researchers that may not be as motivated to develop skills in collaborative research. Additional attention to equipping researchers with the skills needed to work in partnerships is recommended, including approaches and materials that avoid oversimplification of complex challenges.

Key factors for national spread and scale-up of an eConsult innovation.

BACKGROUND: Expanding healthcare innovations from the local to national level is a complex pursuit requiring careful assessment of all relevant factors. In this study (a component of a larger eConsult programme of research), we aimed to identify the key factors involved in the spread and scale-up of a successful regional eConsult model across Canada. METHODS: We conducted a constant comparative thematic analysis of stakeholder discussions captured during a full-day National eConsult Forum meeting held in Ottawa, Canada, on 11 December 2017. Sixty-four participants attended, representing provincial and territorial governments, national organisations, healthcare providers, researchers and patients. Proceedings were recorded, transcribed and underwent qualitative analysis using the Framework for Applied Policy Research. RESULTS: This study identified four main themes that were critical to support the intentional efforts to spread and scale-up eConsult across Canada, namely (1) identifying population care needs and access problems, (2) engaging stakeholders who were willing to roll up their sleeves and take action, (3) building on current strategies and policies, and (4) measuring and communicating outcomes. CONCLUSIONS: Efforts to promote innovation in healthcare are more likely to succeed if they are based on an understanding of the forces that drive the spread and scale-up of innovation. Further research is needed to develop and strengthen the conceptual and applied foundations of the spread and scale-up of healthcare innovations, especially in the context of emergent learning health systems across Canada and beyond.

The unpredictable journeys of spreading, sustaining and scaling healthcare innovations: a scoping review.

Innovation has the potential to improve the quality of care and health service delivery, but maximising the reach and impact of innovation to achieve large-scale health system transformation remains understudied. Interest is growing in three processes of the innovation journey within health systems, namely the spread, sustainability and scale-up (3S) of innovation. Recent reviews examine what we know about these processes. However, there is little research on how to support and operationalise the 3S. This study aims to improve our understanding of the 3S of healthcare innovations. We focus specifically on the definitions of the 3S, the mechanisms that underpin them, and the conditions that either enable or limit their potential. We conducted a scoping review, systematically investigating six bibliographic databases to search, screen and select relevant literature on the 3S of healthcare innovations. We screened 641 papers, then completed a full-text review of 112 identified as relevant based on title and abstract. A total of 24 papers were retained for analysis. Data were extracted and synthesised through descriptive and inductive thematic analysis. From this, we develop a framework of actionable guidance for health system actors aiming to leverage the 3S of innovation across five key areas of focus, as follows: (1) focus on the why, (2) focus on perceived-value and feasibility, (3) focus on what people do, rather than what they should be doing, (4) focus on creating a dialogue between policy and delivery, and (5) focus on inclusivity and capacity building. While there is no standardised approach to foster the 3S of healthcare innovations, a variety of practical frameworks and tools exist to support stakeholders along this journey.

Scaling up eConsult for access to specialists in primary healthcare across four Canadian provinces: study protocol of a multiple case study.

BACKGROUND: Canada has been referred to as the land of 'perpetual pilot projects'. Effective innovations often remain small in scale, with limited impact on health systems. Several innovations have been developed in Canada to tackle important challenges such as poor access to services and excessive wait times - one of the most promising innovations that has been piloted is eConsult, which is a model of asynchronous communication that allows primary care providers to electronically consult with specialists regarding their patients' medical issues. eConsult pilot projects have been shown to reduce wait times for specialist care, prevent unnecessary referrals and reduce health system costs. eConsult has been spread throughout Ontario as well as to certain regions in Manitoba, Quebec, and Newfoundland and Labrador. Our aim is to understand and support the scale-up process of eConsult in Ontario, Quebec, Manitoba, and Newfoundland and Labrador. Our specific objectives are to (1) describe the main components of eConsult relevant to the scale-up process in each province; (2) understand the eConsult scale-up process in each province and compare across provinces; (3) identify policy issues and strategies to scaling up eConsult in each province; and (4) foster cross-level and cross-jurisdictional learning on scaling up eConsult. METHODS: We will conduct a qualitative multiple case study to investigate the scaling up of eConsult in four Canadian provinces using a grey literature review, key stakeholder interviews (10 interviews/province), non-participant observations, focus groups and deliberative dialogues. We will identify the main components of eConsult to be scaled up using logic models (obj. 1). Scaling up processes will be analysed using strategies adapted from process research (obj. 2). Policy issues and strategies to scale-up eConsult will be analysed thematically (obj. 3). Finally, a symposium will foster pan-Canadian learning on the process of scaling up eConsult (obj. 4). DISCUSSION: This study will likely increase learning and support evidence-based policy-making across participating provinces and may improve the capacity for a pan-Canadian scale-up of eConsult, including in provinces where eConsult has not yet been implemented. This work is essential to inform how similar innovations can reshape our health systems in the evolving information age.